77 Commits

Author SHA1 Message Date
Colton Sellers 8edc89a4ff Feature Python Option Filtering Support (#4667)
* Support List and OptionFilterUniverse for Py filter

* Regression algorithm for testing

* Unit Tests

* Fix for process

* Tighten filters to reduce load on automated testing

* Address review v2
2020-09-02 21:38:16 -03:00
Colton Sellers 34676f0c46 Bug 4668 python register indicator (#4670)
* Convert to timespan if possible

* Fix to support both timespans and custom consolidators

* Adjust tests for new wrapper requirements

* Cleanup
2020-08-29 02:53:32 -03:00
Colton Sellers b8674731a5 Feature 2456 custom Python consolidator support (#4637)
* DataConsolidator Wrapper for Python Consolidators

* Regression Unit Test

* Refactor Regression test

* Bad test fix

* pre review

* self review

* Add RegisterIndicator for Python Consolidator

* Python base class for consolidators

* Modify regression algo to register indicator

* unit test - attach event

* Test fix

* Fix test python imports

* Add license header file and null check

Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
2020-08-25 17:26:55 -03:00
Alexandre Catarino 75a5f267f1 Adds Support to List of Symbol and String (Ticker)
Adds case in the mapper method to handled list type.
2020-07-04 00:11:19 +01:00
Jared f9cad3dc0c Merge pull request #4451 from QuantConnect/bug-pandasdata-misaligned-series-indexes-crash
Fixes alignment of Series objects created in PandasData causing crash
2020-05-26 14:02:17 -07:00
AlexCatarino 7bb7d6487f Fixes PandasData Wrapper Support for Query Method
query/eval methods needs to look for a scope variable at a higher level since the wrapper classes are children of pandas classes.
2020-05-23 02:19:52 +01:00
Gerardo Salazar bb2947d941 Fixes alignment of Series objects created in PandasData causing crash
* When a BaseData instance has Nullable fields, the number of data
    points per Series is inconsistent, and results in a Series with
    a length different from the other Series we produce, resulting
    in an error "ValueError: cannot handle a non-unique multi-index!"
    when we were constructing the final DataFrame.
2020-05-22 15:57:16 -07:00
AlexCatarino 34c337e766 Addresses Peer-Review 2020-05-18 16:28:19 +01:00
AlexCatarino f1b1474462 Adds unit test for all DataFrame and Series
- Adds unit tests for parameterless methods
- Adds unit tests for operations between two DataFrames or Series

- Adds unit tests for 'add_prefix', 'add_suffix', 'agg', 'aggregate', 'align', 'apply', 'applymap', 'asfreq', 'asof', 'assign', 'astype', 'at_time', 'axes', 'between_time', 'clip_lower', 'clip_upper', 'columns', 'combine', 'combine_first', 'corrwith', 'drop', 'droplevel', 'dtypes',
'eval', 'explode', 'fillna', 'filter', 'first', 'ftypes', 'get_value_index', 'get_value_column', 'groupby', 'isin', 'items', 'iterrows' and 'T'. (Ref.: #4365, #4368, #4370, #4372, #4373, #4377, #4380)
2020-05-15 23:31:38 +01:00
Martin-Molinero 85cc7b13dc Revert "Adds pandas Methods (Part I)" 2020-05-06 20:16:04 -03:00
Martin-Molinero 31ed6e349b Revert "Adds pandas Methods (Part II)" 2020-05-06 20:15:41 -03:00
Martin-Molinero 3d9a24ff5c Revert "Adds pandas Methods (Part III)" 2020-05-06 20:15:17 -03:00
Martin-Molinero 07ef201f01 Revert "Adds pandas Methods (Part IV)" 2020-05-06 20:14:57 -03:00
Martin-Molinero 07bf7f48e4 Revert "Adds pandas Methods (Part V)" 2020-05-06 20:14:17 -03:00
AlexCatarino 1538d0f134 Adds pandas Methods (Part V)
Methods: 'eq', 'eval', 'explode', 'ffill', 'fillna', 'filter', 'first', 'floordiv', 'ftypes', 'ge'
2020-05-06 02:05:20 +01:00
AlexCatarino b7b54ffa0b Adds pandas Methods (Part IV)
Methods: 'diff', 'div', 'divide', 'drop', 'drop_duplicates', 'droplevel', 'dropna', 'dtypes', 'duplicated'

`BackwardsCompatibilityDataFrame_binary_operator` replaces `BackwardsCompatibilityDataFrame_add` to handle all operations (more to be added in future commits)
2020-05-05 21:43:12 +01:00
AlexCatarino 02d90c4fb0 Adds pandas Methods (Part III)
Methods: 'columns', 'combine', 'combine_first', 'compound', 'copy', 'corr', 'corrwith', 'count', 'cov', 'cummax', 'cummin', 'cumprod', 'cumsum'
2020-05-05 17:15:30 +01:00
AlexCatarino a14c41bb97 Addresses Peer-Review
Removes `bool` because it doesn't return a pandas object with an index.
2020-05-05 15:24:34 +01:00
AlexCatarino 321603926a Adds pandas Methods (Part II)
Methods: 'asfreq', 'asof', 'assign', 'astype', 'at', 'at_time', 'axes', 'between_time', 'bfill', 'bool', 'clip', 'clip_lower', 'clip_upper'.
2020-05-05 12:40:14 +01:00
AlexCatarino ff32b8e15e Adds pandas Methods (Part I)
Methods: 'abs', 'add', 'add_prefix', 'add_suffix', 'agg', 'aggregate', 'align', 'all', 'any', 'append', 'apply', 'applymap'.
2020-05-05 01:32:49 +01:00
Martin-Molinero b66df2bf55 Merge pull request #4365 from QuantConnect/bug-4298-pandas-data-remapper
Fixes Not Wrapped pandas.Dataframe
2020-05-04 17:27:09 -03:00
Martin Molinero 439b32dc4d Address reviews 2020-05-04 16:58:47 -03:00
AlexCatarino ca9f0cc2d3 Refactors PandasConverterTests Unit Tests
Use `TestCase` for BackwardsCompatibilityDataFrame to avoid code duplication and test the three possible pandas.Dataframe index options.
2020-05-04 20:07:43 +01:00
AlexCatarino a892173710 Fixes Not Wrapped pandas.Dataframe
`Remapper.__getitem__`  was not returns a `Remapper` object when the result was `pandas.DataFrame`. It is needed for a sequence of `.loc.` calls.
Refactors `Remapper._self_mapper` to handle tuples where the key can be found in both first and second position.

Update unit tests that should test `Symbol` object as key, but were using `str(Symbol)`.
2020-05-01 00:25:46 +01:00
AlexCatarino 7573286a70 Updates DockerfileFoundation For Python Packages
Adds new and updates Python Packages:
Closes #2707 (mplfinance)
Closes #3541 (CuffLinks IpyWidgets)
Closes #3975 (Jax)
Closes #3982 (gluonts astropy gplearn tigramite ssm auto_ks)
Closes #3983 (H2O AutoML)
Closes #4050 (neural-tangent)
Closes #4125 (CNTK)
Closes #4129 (updates)
Closes #4170 (NLTK Vader Sentiments)
Closes #4246 (riskparity pyrb)
Closes #4256 (Copulae)
Closes #4282 (featuretools)
Closes #4331 (keras-rl pennylane)
2020-04-29 00:28:36 +01:00
Martin Molinero 8e71e48c07 PythonSlice
- AlgorithmPythonWrapper will directly call base OnFrameworkData()
  implementation skipping going through python and it's overhead
- Small performance improvement for adding Tick data points into a Ticks
  collection
- For python always wrap slice with PythonSlice, so that slice.Get()
  works even when no custom data is present, adding test.
2020-04-27 19:03:03 -03:00
Martin Molinero 886e5e6445 Test fixes and updates 2020-04-23 09:59:31 -03:00
Jared 8718ff663f Merge pull request #4093 from QuantConnect/feature-library-update
Add new python libraries
2020-03-06 13:56:09 -08:00
Martin Molinero 6eb08334cd Revert GetBuyingPower removal
- Fix bug where OptioMarginModel would return 0 initial margin required
2020-02-24 14:06:16 -03:00
Martin Molinero c73da247e5 Add new python libraries
- Adding tslearn=0.3.0, tweepy=3.8.0, dtw-python==1.0.4,
pywavelets=1.1.1, umap-learn=0.3.10.
- Update hmmlearn==0.2.3
2020-02-12 15:12:34 -03:00
AlexCatarino 8333374e4d Add Unit Tests 2020-02-11 19:37:33 +00:00
Martin Molinero 6eb45baa59 Refactor GetMaximumOrderQuantityForTargetBuyingPower
- Refactoring `GetMaximumOrderQuantityForTargetValue` into
`GetMaximumOrderQuantityForTargetBuyingPower` which algorithm uses
margin
- Reducing code duplication
- Adding unit tests
- Fixing futures margin model. Adding intra day logic.
2020-02-04 18:41:15 -03:00
Martin Molinero 936af7df7b Refactor BuyingPowerModel
- Remove unrequired `GetBuyingPower`
- Making `BuyingPowerModel.GetMaintenanceMarginRequirement` protected
instead of public
- Adding `GetMaximumOrderQuantityForDeltaBuyingPower` to replace
public `GetMaintenanceMarginRequirement` and improve API experience for
consumers like the `DefaultMarginCallModel`
- Adding new unit tests
2020-02-04 18:41:15 -03:00
Martin Molinero 19edc94f0f Improve pandas Symbol key support
- Add support for Symbol key access for pandas ix and iloc results
- Wrapp pdf merge, join, concat method results
- Wrapp pandas.concat method result with `Remapper`
- Adding unit tests
2019-11-20 13:48:42 -03:00
AlexCatarino c7e4d9798b Changes NLTK Data Location and Adds mlfinlab Package
- Use `/usr/share/nltk_data` instead of `/root/nltk_data`.
- Adds test for NLTK.
- Tidies the root directory of the docker image
- Adds support to mlfinlab
2019-11-12 22:30:04 +00:00
Jared 19fa8664bd Merge pull request #3748 from QuantConnect/bug-3745-fix-history-request-for-cboe
Fixes failing CBOE History request by removing duplicate property
2019-10-23 17:04:00 -07:00
Martin Molinero a5dc1774a7 Add SecurityCacheProvider
- Adding `SecurityCacheProvider` this class allows for two different
`Security` to share the same data type cache through different instance
of `SecurityCache`. This is used to directly access custom data types
through their underlying in a peformant maner
- Some small improvements
2019-10-22 13:46:57 -03:00
AlexCatarino b97d11d37a Addresses Reviews
- Covers another level on inheritance of Market Data by using `Type.IsAssignableFrom`
- Caches the list of `MethodInfo` for custom data types to avoid redefining that list.
2019-10-22 15:14:04 +01:00
AlexCatarino 0cca6869f7 Fixes Support for Custom Data
When custom data classes inherited from market data classes such as `TradeBar`, it created duplicate entries. Therefore, we need to exclude the common properties in the private field `PandasData._members`.
2019-10-22 15:14:04 +01:00
Martin Molinero 27ce9489a4 Check if Python OnMarginCall is defined
- Will check if python OnMarginCall is defined and throw is returns null
or empty list
- Adding unit tests
2019-10-16 11:49:12 -03:00
AlexCatarino 63cc139aef Adds New Python Packages
- creme: https://github.com/creme-ml/creme
- scikit-multiflow: https://scikit-multiflow.github.io/scikit-multiflow/index.html
- PuLP: https://pypi.org/project/PuLP/
- PyMC3: https://docs.pymc.io
- PyPortfolioOpt: https://pypi.org/project/pyportfolioopt/
- gensim: https://radimrehurek.com/gensim/
- Fasttext: https://fasttext.cc/

- Upgrades Numpy and Wrapt
- Changes installation order to  facilitate package dependency solving
- We get the latest odo package from Blaze due to a incompatibility with pandas 0.23.4
2019-10-10 00:01:14 +01:00
Martin Molinero 1cb4856557 Add support for Symbol as key for Pandas dataframe
- Improve user experience allowing users to use `Symbol` instance as key
for pandas data frame
2019-10-01 18:43:38 -03:00
Michael Handschuh 362826988f Improve DynamicSecurityData usability
Adds IRegisteredSecurityDataTypesProvider to track all the data types
registered in the algorithm. Using this data, we can detect if it's
possible that we'll eventually have a property of a certain type name.
For example, consider I wish to use security.Data.TradeBar but we haven't
received any trade bars yet. Before this change a KeyNotFoundException
would be raised, but since we can determine that we expect to have trade
bars, we can detect this and return an empty list when we haven't received
any data yet. This also removes the need to constantly do a HasData<T>()
check before accessing the dynamic members.

Closes #3620
2019-09-30 19:06:21 -04:00
Martin Molinero e887c9f6b4 Normalize wrapt version being used 2019-09-24 10:34:35 -03:00
Gerardo Salazar 4b0ed03b4f Added new regression algorithms
* Deleted regression algorithms because they tested behavior similar to
other existing regression algorithms
* Fixed new bug in regression algorithm due to AddData changes
* Added unit tests for wrapt version and package existence

* Fix issue where data would be set to raw normalization mode
2019-09-24 10:34:35 -03:00
Martin Molinero 6fabe30317 Add backwards compatibility index.levels[0]
- Add backwards compatibility shim for the `df.index.levels[0]` use
case. Reverting broken use cases and adding unit tests.
2019-09-24 10:34:35 -03:00
Martin Molinero b6b7720a1e Add more backwards compatible cases
- Moving mapper from C# to Python since some cases did not work when
implemented in C#
- Small changes to `PandasDataFrameHistoryAlgorithm` which runs till the
end with no errors
- Adding more backwards compatible unit tests
2019-09-24 10:34:35 -03:00
AlexCatarino 35d32ed9f4 More Tests for PR#3484 2019-09-24 10:34:35 -03:00
Martin Molinero 6a2f5b09f2 Improvements
- Add new dependency to documentation, docker files, python setup readme
- Adding backwards compatibility pandas DataFrame unit tests
- Adding missing cases for backwards compatility shim
2019-09-24 10:34:35 -03:00
Martin Molinero 8dbbbb618f Pandas and CustomData mapping fixes
- Add Pandas backwards compatibility shim
- Adding `MappingExtensions` which will remove data type from the
`Symbol.ID.Symbol` value to resolve the `MapFile`
- `SecurityIdentifier.TryParse()` will throw when given an invalid
`SecurityType`
2019-09-24 10:34:35 -03:00