5 Commits

Author SHA1 Message Date
Jhonathan Abreu fb3409d8a2 Custom data sourced from the object store (#7493)
Benchmarks / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Support sourcing custom data from ObjectStore

* Add Python regression algorithm

* Wire engine to pass ObjectStore down to the stream readers

* Minor changes

* Minor unit tests fixes

* Remove unused SetupHandlerParameters.ObjectStore

* Minor changes

* Minor changes

* Support single-file zipped data to be sourced from object store

* Assert object store custom data history requests in regression algorithms

* Add custom object store data live data feed unit test

* Add multi-file object store custom data regression algorithms

* Minor fix
2023-10-02 15:54:57 -03:00
Martin-Molinero d739873daa Add aggregator reader (#5942)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Add BaseDataCollectionAggregatorReader

- Add enumerable for BaseDataCollection
- Add BaseDataCollectionAggregatorReader
- Clean up CollectionSubscriptionDataSourceReader

* Address reviews

- Add new FileFormat that will be handled by the new Reader
- Adding unit tests for the new BaseDataCollectionAggregatorReader
- Some DataSourceReader duplication cleaning up

* Minor adjustment after some live trading tests
2021-09-27 17:25:23 -03:00
Colton Sellers ab6027723f Refactor Api (#5251)
Build & Test Lean / build (push) Has been cancelled
* Use lean data key as param for request

* key -> filePath rename and some cleanup

* Refactor

* Add Organizations Endpoints

* Add some organization api wrapper objects

* Address namespace issue

* Reorganize Api Test into seperate files using one ApiTestBase

* Add Organization tests

* Use capitalized "API" test namespace to reduce amount of file changes

* Add License to test base

* Update /data endpoint functions and response objects

* Update ApiDataProvider Logic

* Handle deserialization of organization products

* Simplify converter

* Only throw for equity requests when not subscribed to map/factor files

* Add missing header

* Make arguement exception

* Api adjustments

* Add Zip factor and map file providers

- Common project will now reference Compression project and not the other way
  round.
- Adding Zip FactorFile and MapFile providers

* Refactor FactorFileProvider to use DataProvider to fetch files

* Use resulting MinimumDate in construction of FactorFile

* Nit FactorFile comments and arrangement

* Refactor MapFileProviders to use DataProvider for fetching files

* Refactor ZipFileProvider

* Clean up

* Refactor Backtesting Future/Option chain providers to use dataprovider

* Fixes for data/ endpoints and test adjustments

* Response objects adjustments/cleanups

* ApiDateProvider fixes and testing

* Add LocalZipFactorFileTests

* Update ApiDataProvider download test to verify stream is not null

* Implement posting of agreement summary and signed time

* Mark all Api related tests as explicit and document details on running

* Clarify default token on ApiTestBase

* Adjust summary

* Update Api responses for QCC, except org products which are sold in USD

* Implement cache expiration for zip MapFile and FactorFiles. Adding unit tests

* Fix multiple markets for ZipFactorFile provider

* Use Symbol as cache key

* Api.cs review

* Dispose of factorFileStream after reading

* Use zip.EntryFileNames

* Address a few reviews

* Few more fixes

* Address Api Review

* Add Job Org id to config

* Minor tweaks

* Compare with invariant culture

* Fixes Option Universe selection

* ZipEntryNameSubscriptionDataSourceReader will use IDataProvider

* Fix research

* Fix null reference exception

* Make duplicate log debug

Co-authored-by: Martin-Molinero <martin@quantconnect.com>
2021-06-07 11:37:44 -03:00
Martin Molinero 3a5a2c582f Lazy OnEndOfDay ScheduledEvent
- Only add OnEndOfDay ScheduledEvent if the algorithm implements the
method. Adding unit tests
- Avoid creating a new baseData instance at
`SubscriptionDataSourceReader`
- Adding static `FineFundamental` instance since creating new ones is
expensive
2019-12-19 19:07:16 -03:00
Martin Molinero c9a189ae3d Add Tiingo news data
- Add Tiingo news data
- Add `IndexSubscriptionDataSourceReader` that will handle data source
which use and index file
- Add `BaseSubscriptionDataSourceReader` to avoid code duplication
- Adjustments at `LiveCustomDataSubscriptionEnumeratorFactory` so that
custom data in a collection format does not emit old data
- Adding `TiingoNewsJsonConverter`
- Adding unit tests
2019-10-01 00:26:36 -03:00