Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Do not send internal SecurityChanges to Algorithm
- Will not send internal security changes to the Algorithm by default.
Following custom security changes filter pattern. Updating regression
algorithms to assert behavior.
- The universe member will know wether it was added with internal
configurations or not
* Address reviews use a separate collection for internals
* Refactor solution. Adding security changes constructor class
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* CompositeDataQueueHandler handles duplicate configs
- CompositeDataQueueHandler will handle duplicate subscriptions sharing
the same config. Adding unit tests
* Refactor bug solution after review
- LiveSubscriptionEnumerator will not long perform any symbol mapping
but will just trigger a new subscription call when remapped.
- CompositeDHQ will handling symbol mapping and keep track of these
mapped configs to trigger unsubscribe accordengly. Adding unit tests
* Adjust solution after review
- To avoid breaking uniqueness of SubscriptionDataConfig will be adding
a private `mapped` property that will change equality of mapped
configs to avoid them clashing. Adding unit tests
* Rename IDQH subscription extension methods
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Minor performance improvements
* CompositeDataQueueHandler is UniverseProvider
- CompositeDataQueueHandler implements the UniverseProvider interface.
Adding unit tests
* initial commit
* Follow IDQH implementation
* Expect a list of data handlers from LiveNodePacket
* Return null if can not subscribe
* refctor to add check for subscription
* initialze null
* Add tests
* Check subscribe retuns null/not-null
* cleanup
* Read all required IDQH credentails to job
* Use CDQH to handle all IDQH instances
* constructor abstraction to call from setjob
* use flag
* remove redundant because derived will call initialize on it
* abstract and initialize from setjob
* handle null enumerators
* get creds from data handlers
* handle single data handler value from data-queue-handler
* Fix to support a json array
* Fix missed constructor call
* change access modifier to access from Tests files
* Add test to get brokerageFactory from dataQueueHandler
* Fix init flag to handle all conditions
* Add docs
* initialize from setjob
* Check if websocket open before using
* change defination of initialzie to include tradier
* clean up
* change defination
* fix wrong api key name
* return empty enumerator
* check websocket open before sending request
* check connection before subscribing
* fix to include more cases
* check websocket open before sending request
* Minor refactoring
* reafctor and use IsConnected
* remove unused
* clean up
* Fix test cases
* reverse change
* include config changes
* connect to websocket from setjob
* check websocket connection from setjob
* clean up
* include condition for IDQH that are not brokerage
* Address review
* Add market check condition before subscribe
* Remove deprecated
* Minor fix for deserializing data queue handler
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* Continuous Futures Refactor. Live Mappings
- Adding support for live mappings. LiveTradingDataFeed will handle any
symbol mapping at the configuration layer and resubscribe through the
IDQH
- Refactoring continuous futures adding ContinuousFutureUniverse that
will select the currently mapped security
* Minor fixes
- Remove addition of configurations in UniverseSelection step, leave
resposability for universe.
- LiveTradingDF future unit test will only assert slice data for non
internal feeds.
- ContinuousContractUniverse will respect internal option interest
subscription
* Address review
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Add tick type to data downloader get (price data) call in order for api's with rate limits to handle not making an api call at all.
* Added DTO object to pass into data downloader get
* Added missing header for new file
* Address review. Minor tweaks
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* Fix for python enumerable data history request
- Fix for python enumerable type data history request. Adding unit test.
* Add methods for adding data points into a baseDataCollection
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* Create generic writing for LeanDataWriter, + notes on todos
* Make Options Daily/Hourly data store by year
* Refactor Generic Write
* Permit hour and daily resolutions for options
* Refactor writer to merge when needed with other files
* Cleanup redundancies, run write tasks in parallel
* Make needed classes/vars available
* Update tests to reflect new naming convention for daily hourly options data
* Add Byte[] overloads for ZipData functions in compression
* Implemented Store() for ZipDataCacheProvider
* Have LeanDataWriter use a DataCacheProvider
* ZipDataCacheProvider cleanup
* ZipDataCacheProvider tweaks, doesn't support storing non-zips
* Test adjustments
* Update LeanDataWriter to use Write instead of SaveDailyHourly/SaveMinuteSecond
* Implement tests to verify DownloadAndSave behavior
* Nit cleanup on DownloadAndSave tests
* Fix for options daily/hourly underlying equity subscription read
* Add daily/hourly options data and regressions
* Add missing open interest for hourly
* Fix writing of OpenInterest Daily/Hourly data
* Update data
* Fix Date typo in regression
* Use daily algorithm to test delisting
* Revisions part 1
* Expand test for DataCacheProviders; refactor DiskDataCacheProvider
* nit - test adjustments
* ZipDataCacheProvider test setup refactor
* Adjust multithreaded read/write test; fixes for ZipDataCacheProvider
* Move DiskDataCacheProvider to its own file and add write test
* Remove _appendToZips; always overwrite entry or create zip
* Add mapping regression for daily options
* nit - add license to regression
* Fix Tick write case where more than one data point for a DateTime
* Fix data issue
* Address review
* Tweaks for tests
* Stop Store() early if no entry name is given
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Add BaseDataCollectionAggregatorReader
- Add enumerable for BaseDataCollection
- Add BaseDataCollectionAggregatorReader
- Clean up CollectionSubscriptionDataSourceReader
* Address reviews
- Add new FileFormat that will be handled by the new Reader
- Adding unit tests for the new BaseDataCollectionAggregatorReader
- Some DataSourceReader duplication cleaning up
* Minor adjustment after some live trading tests
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Alpha holdings state
- Alpha result packet will optionally provide the algorithms portfolio
state
* Rename
* Convert AlphaStreamsPortfolio to data source
* Improvements on AlphaStreams algorithm
* Fix regression tests
* Add unit tests for EW AS PCM and fixing bugs
* Protobuf AlphaStreamsPortfoliot staState
- Protobuf AlphaStreamsPortfolioState. Adding unit tests
- Add variable TPV tests for EW ASPCM
* Add alpha license to Organization response
* Improvements EW AS PCM respects free portfolio value
* Fixes
- Update tests expected statistics results affected by MHDB custom data timezone fix
- Fix for Extensions.IsCustomDataType
* Fixed and adding more regression tests
- Adding support and regression test with alpha consumer with different account currency
- Adding support and regression test of a universe adding custom data
types
- Add support and regression test for algorithm alpha consumer with existing holdings
* Add AlphaStreamsOrderEvent data type
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* Adds support for ETF constituent universes
* Adds filtering for universe data if it doesn't match the
universe subscription type
* Update mapping for ALL underlying Symbols if
`Symbol.UpdateMappedSymbol(...)` is called. Required to support
constituent ETF universes that might have mapping events
* Delistings of composite constituent universe Symbol will result in
removal of universe securities.
* Added regression algorithms for ETF constituent mappings (C#/Python),
along with data required to run locally
* Refactor universe delistings in SubscriptionSynchronizer -
big thank you to @Martin-Molinero :)
* Address review: update regression algorithms and add explanatory comments
* Address review: add additional checks to delisting regression algorithms
* Adds new regression algorithm testing the addition of a universe
without calling AddEquity() and asserts same behavior
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* Fix SubscriptionDataReader auxiliary data out of order
- Fix for SubscriptionDataReader emitting auxiliary data out of order
due to sparse that and enumerator refresh logic. Adding regression
algorithm
- Minor tweaks for unit tests failing on and off
* Fix for SDR internal subscriptions
- Fix for SubscriptionDataReader enumerator refresh for internal
subscriptions which were ignored. Adding regression test
- Updating custom data regression algorithms affected by the issue
- DropboxBaseDataUniverse was emitting a custom data point being end time
- UnlinkedTraderBarIconicType was emitting a single data point
of the underlying SPY minute data when if should of emitted
all data points
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Improve python exception parsing
- Improve python exception parsing adding support for line shift. Adding
unit tests
* PythonException revert change
* Centralized and normalize algorithm runtime handling
* Adding support for C# line and file exception report
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* WIP removal of custom data references & tests updates
* Regression algos updated and python algorithms moved to DataSource repos
* Fixes failing unit tests
* Add "LiveDataTypes" field to LiveNodePacket
* Adds Initialize() to IDataChannelProvider
* Adds new extension method to convert
HistoryRequest -> SubscriptionDataConfig
* Address review: Add protobuf definitions for Iconic data types
* Address review: re-adds DynamicSecurityDataAlgorithm as regression algo
* Small adjustments to variable naming and documentation
* Move test files to respective DataSource repos
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
- Add new CompositeDataProvider that will wrap multiple instance. Allows
using for example the ApiDataProvider and a DownloaderDataProvider.
- Adjusting ProductJsonConverter after recent Modules addition
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Add new DownloaderDataProvider
- Moving LeanDataWrite and IDataDownloader to common project
- Add new BaseDataDownloaderDataProvider with concurrency helper method
- Add new DownloaderDataProvider which will use a IDataDownloader or
IBrokerage implementation as data source
* Add support for downloading symbol chains data
* Use DataProvider for fetching Shortable and FutureMargins files
* Use data provider for AllShortableSymbols
* Refactor of NeedToDownload logic
* Adjustments
* nit - cleanup
* Expand test cases
* Create the directory if it doesn't exist
Build & Test Lean / build (push) Has been cancelled
- Ensure data is downloaded only once with the ApiDataProvider. Adding
unit test
- Log invalid securityType ApiDataProvider requests once
- Assert Factor file and map file providers are compatible with the
ApiDataProvider
Build & Test Lean / build (push) Has been cancelled
* Use lean data key as param for request
* key -> filePath rename and some cleanup
* Refactor
* Add Organizations Endpoints
* Add some organization api wrapper objects
* Address namespace issue
* Reorganize Api Test into seperate files using one ApiTestBase
* Add Organization tests
* Use capitalized "API" test namespace to reduce amount of file changes
* Add License to test base
* Update /data endpoint functions and response objects
* Update ApiDataProvider Logic
* Handle deserialization of organization products
* Simplify converter
* Only throw for equity requests when not subscribed to map/factor files
* Add missing header
* Make arguement exception
* Api adjustments
* Add Zip factor and map file providers
- Common project will now reference Compression project and not the other way
round.
- Adding Zip FactorFile and MapFile providers
* Refactor FactorFileProvider to use DataProvider to fetch files
* Use resulting MinimumDate in construction of FactorFile
* Nit FactorFile comments and arrangement
* Refactor MapFileProviders to use DataProvider for fetching files
* Refactor ZipFileProvider
* Clean up
* Refactor Backtesting Future/Option chain providers to use dataprovider
* Fixes for data/ endpoints and test adjustments
* Response objects adjustments/cleanups
* ApiDateProvider fixes and testing
* Add LocalZipFactorFileTests
* Update ApiDataProvider download test to verify stream is not null
* Implement posting of agreement summary and signed time
* Mark all Api related tests as explicit and document details on running
* Clarify default token on ApiTestBase
* Adjust summary
* Update Api responses for QCC, except org products which are sold in USD
* Implement cache expiration for zip MapFile and FactorFiles. Adding unit tests
* Fix multiple markets for ZipFactorFile provider
* Use Symbol as cache key
* Api.cs review
* Dispose of factorFileStream after reading
* Use zip.EntryFileNames
* Address a few reviews
* Few more fixes
* Address Api Review
* Add Job Org id to config
* Minor tweaks
* Compare with invariant culture
* Fixes Option Universe selection
* ZipEntryNameSubscriptionDataSourceReader will use IDataProvider
* Fix research
* Fix null reference exception
* Make duplicate log debug
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
Build & Test Lean / build (push) Has been cancelled
* Fix FutureOptionMarginModel Margin Requirements
- Fixes FuturesOptionsMarginModel margin requirements being zero due to
underlying being null on model creation. Adding unit test
* Address reviews
- SecurityService will set underlying when creating a new security when
provided
Build & Test Lean / build (push) Has been cancelled
* Use UniverseSettings.DataNormalizationMode for securities added in Algorithm
* Stop SubscriptionUtils from forcing Adjusted mode
* Return behavior to original and add comments
* nit typo
* Add regression
* Add unit test that verifies DataNormalizationMode can be altered manually by security
* Cleanup and add Py version of regression
* Refactor to queue up start date changes to log on dispose
* Fix typo in warning collection used
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Collect and log symbols that had start date adjusted because of factor files
* Only post the message if the set has values
* Modify message to be [Symbol, Date] combo for more information on log
* Set a hard limit on the warning set to keep it from growing unnecessarily large
* Enforce a hard limit and improve message if full
* Enforce limit in log message because of possible threading adding past max
* Cleanup
* Add futures regression reproducing the issue
* Cleanup futures regression
* Add Options regression
* Add IRegressionAlgorithmDefinition
* Add IRegressionAlgorithmDefinition
* Adjust regressions to compare to expiration date; delisting time is not always correct
* Refactor subscription enumerator filtering of AuxData to top of the stack
* Update broken tests to reflect the split/dividends/delisting subscriptions
* Update regression statistics because of Delisting EOD instead of at time
* Let custom data configurations bypass filter
* Adjust expected count, now that we are letting aux data through to history requests
* another small adjustment
* Use ExpectedBarCount in error message
* Add filter test for both cases
* Add some clarifying comments
* Verify we did recieve data in the regressions
* Make _shouldEmitAuxiliaryData private readonly
* Filter out aux data for history requests
* Remove option to not include aux data in subscriptions
* Refactor filtering to be more explicit for each piece of data; fixes universe selection aux data
* Cleanup comments after removed var
* Refactor order of filtering for performance reasons.
* Reconcile duplicated code
* Add License header
* CS0219 Fixes: Value assigned, but never used
* CA1507: Use nameof in place of string literals
* CS0108 : Hides Inherited Member; Use new keyword to overwrite formally
* CS0114: Hides inherited member; use override keyword
* CS0168: Variable is declared but never used
* Tests CS1062; using obsolete implicit Symbol -> String; fix via .ToString()
* CS0472: Non Nullable Obj getting Null Checked
* CS0067 Member not used; ignore all cases for future use
* CS00162 : Unreachable code; either removed or ignored for debugging and test cases
* CS0169 Remove non-used fields; ignore those that may be used in future
* CS0414; Field is assigned but never used.
* CS0618; Obsolete properties and members; Only fixes simple ones, rest will have to broken up
* CS0649; Field never assigned too
* CS0659 & CS0661 ; Overwrite operators and equals but not hashcode; I don't really override it but just call base
* Small comment fix
* Cleanup pragma statement
`LiveOptionChainProvider.FindOptionContracts` handles the following data format:
`SPY 2021 03 26 190 000 C P 0 612 360000000`
where both existing `OptionRight.Call` and `OptionRight.Put` are declared in the same line.
* Add support for Index SecurityType 🚀 (#5364)
* Add Index SecurityType 🚀
* Extend SecurityIdentifier & Lean Data classes with Index support
* Add Index SecurityType 🚀
* Extend SecurityIdentifier & Lean Data classes with Index support
* Fixes
* Added index cross basic template demonstration
* WIP: Prototype index security type for LEAN as non tradable asset
* Re-adds Index entries to MHDB after rebase
* First steps to getting Index Options running
* Looks at any instance where we pattern match for an option type
and replaces it with a generic call to `.IsOption()` for easier
extensibility in the future for additional option security types
* Adds IndexOption security and misc. classes
* Misc. changes, mainly related to any sort of special casing of
equity options and made index options take the same path
* Enables index options data for backtesting
* Adds new index options market hours to MHDB
* Misc. bug fixes for index options
* WIP: add live support for index options and indexes
* Use OptionMarginModel for Index Options because they both use the
same calculation for margin requirements
* Fixes contract not found errors on SPX index options and SPX index in IB
* Turns out index options' last trading day is the day before expiry,
which IB was expecting the last trading day.
* Add index option test cases (temp)
* LiveOptionChainProvider fix, use Symbol vs. ticker
* Description updates to regression algorithms
* Fixes bug in live trading for indexes and index options
* Adds overridable minimum price variation symbol property
* Adds variable sized minimum price variation for index options
* Adjusts symbol properties for index options
* Misc. bug fixes
* Fixes option assignment simulation for European options
* Updates index options regression algorithms (WIP)
* Fixes bug where index option exercise would trade index underlying
* Fixes bugs where SecurityType.Index was getting flagged as tradable
* Regression algorithms updates and addresses review
* Misc. style fixes and refactoring + a few bug fixes
* Updates regression algorithms to run without runtime errors
* Adds data for regression algos
* Sets DefaultOptionStyle on Canonical and support index options
* Update regression algos statistics
* Removes bad line in regression algorithm causing build to fail
* Minor tweaks
* Address review add comment about quoteBar parse scale
Co-authored-by: Balamurali Pandranki <balamurali@live.com>
Co-authored-by: Jared Broad <jaredbroad@gmail.com>
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
Martin-Molinero