* Store traded securities subscription data configs to disk
* Delegate traded subscriptions storage to the data monitor
* Add unit tests
* Address peer review
* Minor changes
Python Virtual Environments / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* New Fundamental Data
* Minor CIK lookup fix
* Handle live mode & delete unexisting properties
* Minor coarse fundamental adjustment
* Add fundamental history support
* Fix unit tests
* Performance improvements
* Fixes
* Minor regression algorithm fix
* Improvements. Add FundamentalUniverseSelectionModel
* Change default values
* Fix unit test
* Minor tweaks
* Fix unit test
* Minor error handling improvement
* Fix rebase
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
* Downloader data provider fixes
- Fix stored data time zone, which is expected in data time zone
- Minor logging improvements
- LeanData Parse handles etf universes path. Adding unit test
- LeanDataWrite will only map if there's something to map to
* Fix download data provider synchronization
* Synchronization improvements after more testing
- KeyStringSynchronizer will handle reentrancy. Adding unit tests
- Define a unique Synchronizare for downloading data, since not related
to disk reading & writting
Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Refactor alpha statistics
- Refactor alpha statistics, cleaning up and simplifying no longer required calculations and scoring
- Adding new InsightEvaluator abstraction, adding C# & PY regression
algorithms
* Optimization backtest result json converter update
* Address reviews
- Remove IAlphaHandler, move insight storage responsability to IResultHandler
and centralizing insight collection on the QCAlgorithm.Insights to be
reused by the framework models
- Fix portfolio turnover single day backtests and duplicate time
sampling handling. Updating regression algorithms
* Add InsightCollection tests and minor fixes
* Adding more & improved tests
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
* Data monitor (FIRST TEST)
* Calculate data request frequency in data monitor
* Detecting universe data usage in data monitor
* Json data monitor report generation
* Data monitor refactor
* Log and store data monitor report
* Data monitor report tests
* Clean up
* Clean up
* Address changes request
* Address changes request
* Wire data monitor only for backtest, not for live or research
* Minor tweaks
* Some minor tweaks
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Refactor chain providers
- ChainProviders will use quote, open interest and trade information,
which ever is first. Updating regression algorithm changing contract
being added.
- Add base BacktestingChainProvider for shared logic, reducing
duplication.
- ChainProviders will now use the given IDataCacheProvider and
IMapFileProvider
- If Providers are called for a date the exchange is closed they will
search for the previous available date where the exchange is open.
Adding unit tests.
* Address reviews. Logging improvements
Build & Test Lean / build (push) Has been cancelled
- Ensure data is downloaded only once with the ApiDataProvider. Adding
unit test
- Log invalid securityType ApiDataProvider requests once
- Assert Factor file and map file providers are compatible with the
ApiDataProvider
- Adding new DataPermissionManager that will own the existing
datachannelProvider. Will assert configurations before added to the
data feed. Adding unit tests
This commit is squashed from iterative development:
- More consistent method naming
- Storage root path updated to be absolute and include algorithm name
- Storage root path created only if object store is actually used
- Implemented XML save/load
- Added missing unit tests
- Replaced Log.Trace with Log.Error calls
- Added the object store name logging in Engine.Main
- Read storage root from config
- Create algorithm storage root folder in Initialize
- Remove empty folder in Dispose
- Added null checks in all methods
- Added missing XML parameter docs
- make Initialize and Dispose virtual
- make AlgorithmStorageRoot protected
The IObjectStore abstraction provides algorithms with a persistent
storage mechanism. While the algorithm is running, data is maintained
in memory as a dictionary of raw bytes (string -> byte[]). This ensures
we avoid any reference type shenanigans. Periodically, the data in the
object store is persisted and additionally, when the algorithm shuts
down, the object store's data will again be persisted. This ensures that
when the algorithm starts up again, it will have access to any state
that has been saved into the object store.
A great use case for IObjectStore is saving a compute heavy model.
For example, computing the weights of a deep neural network is very
CPU intensive, but after the weights are computed, evaluation is fairly
quick. An initial backtest can be used to solved for the network's weights
and then subsequent backtests or even in live mode, the weights will be
available to the algorithm provided they were saved into the object store.
Also, some libraries require a file path to load model data. The object
store provides a `GetFilePath(key)` method which will copy the data for
the provided key to the disk and return that path so the library can load
the model data.
The term 'alpha' is used to describe the entire algorithm. Therefore, 'alpha'
produces insights. From this we have things like IAlphaModel, which is the model
defining how insights are produced. We have IAlphaHandler, which defines how the
insights from a single 'alpha' (the algorithm) are managed, analyzed, and stored.
Types closer to the individual prediction level, such as InsightDirection, or
InsightScore relate directly to exactly 1 insight. The distinction between the
two became more clear as we developed the insights API, and from that effort it
was decided to harmonize alpha/insight terminology across the various QC systems.
Refactored IDataProvider interface to return stream. The IDataProvider
Fetch method now only takes a key. The IDataProvider
has been reshuffled to be at the bottom of the LeanDataStack. It provides
data to the rest of the Lean stack. The default implementation of IDataProvider reads
data from disc.
All IDataCacheProviders now have constructors which take IDataProviders
and use them to find data on disc.
Renamed DataCacheProvider to ZipDataCacheProvider
Added comments to IDataProvider and it's implementations
Added comments to IDataCacheProvider and it's implementations
The DefaultFileProvider is now configured as by default as the file-provider in config.json and passed in the the engine through the leanAlgorithmHandlers object.
The FileProvider is now passed into the TextSubscriptionDataSourceReader and ZipEntryNameSubscriptionDataSourceReader. Most of the changes in this commit are just getting the FileProvider to those two methods.
The command queue provides external messaging into the algorithm.
It yields ICommand instances which are runnable against the algorithm.
This allows a user to exercise some control/change over the algorithm's
behavior while it's still running.
To show the pattern, I've implemented the Order, Liquidate, and Quit commands
Martin-Molinero