From [pandas-docs](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.concat.html):
> The current default of sorting is deprecated and will change to not-sorting in a future version of pandas.
>
> Explicitly pass sort=True to silence the warning and sort. > Explicitly pass sort=False to silence the warning and not sort.
Since Lean/QuantConnect data from a symbol can be found in Slice.Ticks, Slice.Bars and Slice.QuoteBars, information for all of this members must be used in order present all information in the pandas.DataFrame.
- Moves PandasData class to its own file
- Refactors PandasData class to deal with list of tick
- Other minor changes requested by peer-review
- Adds unit test for PandasConverter's tick handling
In the previous implementation, each one of bars.Select statement causes an extra enumeration of the bars enumerable. To make matters worse, this is all invoked 4*N times, where N is the number of data points.
Another bottleneck was related to the way we concatenate pandas.DataFrames: we would add a new data frame to an existing data frame individually. Now, we collect all data frames and concatenate them in one operation.
With this new implementation, we have accomplshed a reduction of 90% of memory usage in a history request of 1000 symbols.
With simple commands, we can use Lean indicators in QuantBook. It fetchs the historical data from the symbol, calculates the indicator and saves the output in a pandas.DataFrame.
History requests should not return a dictionary with a dataframe, but a multi-index dataframe.
It is more common to work with multi-index dataframes rather than multi-column.
AlexCatarino