41 Commits

Author SHA1 Message Date
Alexandre Catarino 75a5f267f1 Adds Support to List of Symbol and String (Ticker)
Adds case in the mapper method to handled list type.
2020-07-04 00:11:19 +01:00
Jared f9cad3dc0c Merge pull request #4451 from QuantConnect/bug-pandasdata-misaligned-series-indexes-crash
Fixes alignment of Series objects created in PandasData causing crash
2020-05-26 14:02:17 -07:00
Gerardo Salazar 833c0540e0 Address review - short circuit the PandasData field/property type check 2020-05-25 09:46:58 -07:00
AlexCatarino 7bb7d6487f Fixes PandasData Wrapper Support for Query Method
query/eval methods needs to look for a scope variable at a higher level since the wrapper classes are children of pandas classes.
2020-05-23 02:19:52 +01:00
Gerardo Salazar bb2947d941 Fixes alignment of Series objects created in PandasData causing crash
* When a BaseData instance has Nullable fields, the number of data
    points per Series is inconsistent, and results in a Series with
    a length different from the other Series we produce, resulting
    in an error "ValueError: cannot handle a non-unique multi-index!"
    when we were constructing the final DataFrame.
2020-05-22 15:57:16 -07:00
AlexCatarino 2ab7eefd63 PandasData Speed Improvement
Use the original `pandas.Series` to create `Series` objects before `DataFrame` (wrapper version) creation.
It improves the speed because it avoids unnecessary and expensive index wrapping operations of the `DataFrame` creation.
2020-05-21 15:55:00 +01:00
AlexCatarino 34c337e766 Addresses Peer-Review 2020-05-18 16:28:19 +01:00
AlexCatarino e9eba27e6e Implements New PandasData.Remapper
In this implementation, we dynamically create new classes that wraps key functions and properties. The wrappers will map/convert any parameter that are convertible to the string representation of Symbol.ID before they are used by the original function/property.
2020-05-15 23:30:19 +01:00
Martin-Molinero 85cc7b13dc Revert "Adds pandas Methods (Part I)" 2020-05-06 20:16:04 -03:00
Martin-Molinero 31ed6e349b Revert "Adds pandas Methods (Part II)" 2020-05-06 20:15:41 -03:00
Martin-Molinero 3d9a24ff5c Revert "Adds pandas Methods (Part III)" 2020-05-06 20:15:17 -03:00
Martin-Molinero 07ef201f01 Revert "Adds pandas Methods (Part IV)" 2020-05-06 20:14:57 -03:00
Martin-Molinero 07bf7f48e4 Revert "Adds pandas Methods (Part V)" 2020-05-06 20:14:17 -03:00
AlexCatarino c1baf7da1a Addresses Peer-Review
`first_valid_index´ has been removed for now as it doesn't have a satifactory init test. It will be added in a future commit.
2020-05-06 14:28:59 +01:00
AlexCatarino 1538d0f134 Adds pandas Methods (Part V)
Methods: 'eq', 'eval', 'explode', 'ffill', 'fillna', 'filter', 'first', 'floordiv', 'ftypes', 'ge'
2020-05-06 02:05:20 +01:00
AlexCatarino b7b54ffa0b Adds pandas Methods (Part IV)
Methods: 'diff', 'div', 'divide', 'drop', 'drop_duplicates', 'droplevel', 'dropna', 'dtypes', 'duplicated'

`BackwardsCompatibilityDataFrame_binary_operator` replaces `BackwardsCompatibilityDataFrame_add` to handle all operations (more to be added in future commits)
2020-05-05 21:43:12 +01:00
AlexCatarino 02d90c4fb0 Adds pandas Methods (Part III)
Methods: 'columns', 'combine', 'combine_first', 'compound', 'copy', 'corr', 'corrwith', 'count', 'cov', 'cummax', 'cummin', 'cumprod', 'cumsum'
2020-05-05 17:15:30 +01:00
AlexCatarino a14c41bb97 Addresses Peer-Review
Removes `bool` because it doesn't return a pandas object with an index.
2020-05-05 15:24:34 +01:00
AlexCatarino 321603926a Adds pandas Methods (Part II)
Methods: 'asfreq', 'asof', 'assign', 'astype', 'at', 'at_time', 'axes', 'between_time', 'bfill', 'bool', 'clip', 'clip_lower', 'clip_upper'.
2020-05-05 12:40:14 +01:00
AlexCatarino ff32b8e15e Adds pandas Methods (Part I)
Methods: 'abs', 'add', 'add_prefix', 'add_suffix', 'agg', 'aggregate', 'align', 'all', 'any', 'append', 'apply', 'applymap'.
2020-05-05 01:32:49 +01:00
AlexCatarino a892173710 Fixes Not Wrapped pandas.Dataframe
`Remapper.__getitem__`  was not returns a `Remapper` object when the result was `pandas.DataFrame`. It is needed for a sequence of `.loc.` calls.
Refactors `Remapper._self_mapper` to handle tuples where the key can be found in both first and second position.

Update unit tests that should test `Symbol` object as key, but were using `str(Symbol)`.
2020-05-01 00:25:46 +01:00
AlexCatarino 0beaab873f Implements Wrappers for __str__ and __repr__ in PandasData.Remapper 2020-02-11 16:54:51 +00:00
Martin Molinero 19edc94f0f Improve pandas Symbol key support
- Add support for Symbol key access for pandas ix and iloc results
- Wrapp pdf merge, join, concat method results
- Wrapp pandas.concat method result with `Remapper`
- Adding unit tests
2019-11-20 13:48:42 -03:00
AlexCatarino b97d11d37a Addresses Reviews
- Covers another level on inheritance of Market Data by using `Type.IsAssignableFrom`
- Caches the list of `MethodInfo` for custom data types to avoid redefining that list.
2019-10-22 15:14:04 +01:00
AlexCatarino 0cca6869f7 Fixes Support for Custom Data
When custom data classes inherited from market data classes such as `TradeBar`, it created duplicate entries. Therefore, we need to exclude the common properties in the private field `PandasData._members`.
2019-10-22 15:14:04 +01:00
Martin Molinero 1cb4856557 Add support for Symbol as key for Pandas dataframe
- Improve user experience allowing users to use `Symbol` instance as key
for pandas data frame
2019-10-01 18:43:38 -03:00
Martin Molinero 6fabe30317 Add backwards compatibility index.levels[0]
- Add backwards compatibility shim for the `df.index.levels[0]` use
case. Reverting broken use cases and adding unit tests.
2019-09-24 10:34:35 -03:00
Martin Molinero b6b7720a1e Add more backwards compatible cases
- Moving mapper from C# to Python since some cases did not work when
implemented in C#
- Small changes to `PandasDataFrameHistoryAlgorithm` which runs till the
end with no errors
- Adding more backwards compatible unit tests
2019-09-24 10:34:35 -03:00
Martin Molinero 6a2f5b09f2 Improvements
- Add new dependency to documentation, docker files, python setup readme
- Adding backwards compatibility pandas DataFrame unit tests
- Adding missing cases for backwards compatility shim
2019-09-24 10:34:35 -03:00
Martin Molinero 8dbbbb618f Pandas and CustomData mapping fixes
- Add Pandas backwards compatibility shim
- Adding `MappingExtensions` which will remove data type from the
`Symbol.ID.Symbol` value to resolve the `MapFile`
- `SecurityIdentifier.TryParse()` will throw when given an invalid
`SecurityType`
2019-09-24 10:34:35 -03:00
Michael Handschuh d709d1c4e0 Update Common to respect CA1304 and CA1305
Updates all occurrences of parsing/ToString-ing to go through the new
StringExtensions methods that use CultureInfo.InvariantCulture

See #3045
2019-09-12 14:13:24 -04:00
Martin Molinero 103bf638b5 Adding Pandas MultiIndex cache
- Adding a cache for the `Pandas.MultiIndex`, giving a significant
performance improvement.
- Adding new `ListComparer` class. Adding unit tests.
2019-01-11 14:00:32 -03:00
Martin Molinero 3d918bae1b PandasData will convert DateTime once
- `PandasData` will convert DateTime to python once and share it among
consumers giving a performance improvement.
2019-01-10 11:45:19 -03:00
Juan José D'Ambrosio 4ace8cb255 Lean Visualizer C# toolbox project and Python Wrapper working 2018-04-27 11:49:52 -03:00
AlexCatarino 7707c166dd Refactors PandasConverter and PandasData
Since Lean/QuantConnect data from a symbol can be found in Slice.Ticks, Slice.Bars and Slice.QuoteBars, information for all of this members must be used in order present all information in the pandas.DataFrame.
2018-03-17 19:03:52 +00:00
AlexCatarino 720c1f86d7 Minor improvements
- Rename variable `_type` for `_customDataType` for clarity
- `_baseDataProperties` as static member
2017-12-04 21:42:02 +00:00
AlexCatarino 3a6b18a470 Fixes PandasData support to C# custom data
Previous implementation didn't account for different subclasses that a custom ata class could inherit from.
2017-12-04 21:38:39 +00:00
AlexCatarino de2e877ce5 Fixes decimal to double convertion in PandasData 2017-12-04 21:38:39 +00:00
AlexCatarino 03eb342c8b Implements C# custom data support to PandasData
The previous custom data implementation did not account to custom data created in C# like FxcmVolume or DailyFx
2017-12-04 21:38:39 +00:00
AlexCatarino 52bded5a6a Minor changes
- We were excluding Exchange and Suspicious of quote ticks when they can be set with security types other than equity
- Custom data was not expecting non-numerical properties
2017-11-17 16:18:57 +00:00
AlexCatarino 812906b0c3 Refactors PandasConverter
- Moves PandasData class to its own file
- Refactors PandasData class to deal with list of tick
- Other minor changes requested by peer-review
- Adds unit test for PandasConverter's tick handling
2017-11-17 16:09:44 +00:00