630 Commits

Author SHA1 Message Date
Alexandre Catarino c5e5de3416 Fixes Index Option Assignment Simulation (#6697)
Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Cash settlements do not open a position for the underlying, so we don't need to calculate the cost of closing the position.
2022-10-19 10:45:55 -03:00
Martin-Molinero 39735941ec FillForward delisting handling (#6692)
Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
- In some rare cases the FillForwardEnumerator was missing to register
  the delisting date of an asset because it was the first data it got.
  Adding unit test reproducing issue
2022-10-14 20:29:40 -03:00
Martin-Molinero fc6835cace Normalize time in universe behavior (#6693)
Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Normalize time in universe behavior

- Normalize Option & Future chain universe behavior regarding their
  assets time in universe. They will now respect the universe settings
  time in universe value. Adding new regression algorithms asserting the
  behavior

* Address reviews & cleanup
2022-10-14 15:13:59 -03:00
Martin-Molinero 59eae963be Fix order cancel using order ticket (#6640)
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
- If cancel request failed, allow user to retry. Adding unit test
  reproducing the issue
2022-09-19 18:36:51 -03:00
Martin-Molinero 3f42101f6d Update to python38 and libraries (#6558)
* Update to python38

- Update to python38
- Update libraries

* Create latest virtual environment. Add CI unit tests

* Further update python packages

* Virtual env fixes

- Refactor python virtual env setup
- Adding CI unit tests for python packages and virtual envs

* Adding more python packages

- Adding NBeats, Ax-platform, Riskfolio-Lib

* Rename pomegranate venv
2022-08-22 13:54:02 -03:00
Jhonathan Abreu 539011274c Support extended market hours for futures (#6522)
Python Virtual Environments / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Support extended market hours in AddFuture()

* Support extended market hours in AddFutureContract()

* Add C# regression algorithm

* Add Python regression algorithm

* Add regression algorithm for future contracts

* Add regression algorithm checking market hour ranges

* Fixed future regression algorithms to use extended market hours

* Fixed future regression algorithms to use extended market hours

* Fixed future regression algorithms to use extended market hours

* Fixed AddFutureOptionContractFromFutureChainRegressionAlgorithm to use extended market hours

* Update future market hours to include extended in market hours database

* Fixed AddFutureOptionContractDataStreamingRegressionAlgorithm to use extended market hours

* Fixed AddFutureOptionContractFromFutureChainRegressionAlgorithm to use extended market hours

* Fixed AddFutureContractWithContinuousRegressionAlgorithm to use extended market hours

* Fixed BasicTemplateContinuousFutureAlgorithm to use extended market hours

* Fixed BasicTemplateFuturesAlgorithm to use extended market hours

* Fix BasicTemplateFuturesDailyAlgorithm to use extended market hours

* Fixed BasicTemplateFuturesFrameworkAlgorithm to use extended market hours

* Fixed BasicTemplateFuturesHistoryAlgorithm to use extended market hours

* Fixed ContinuousBackMonthRawFutureRegressionAlgorithm to use extended market hours

* Fixed ContinuousFutureBackMonthRegressionAlgorithm to use extended market hours

* Fixed ContinuousFutureHistoryRegressionAlgorithm to use extended market hours

* Fixed ContinuousFutureLimitIfTouchedOrderRegressionAlgorithm to use extended market hours

* Fixed ContinuousFutureRegressionAlgorithm to use extended market hours

* Fixed DelistedFutureLiquidateRegressionAlgorithm to use extended market hours

* Fixed AutomaticIndicatorWarmupDataTypeRegressionAlgorithm to use extended market hours

* Fixed ConsolidateRegressionAlgorithm to use extended market hours

* Fixed DelistingFutureOptionRegressionAlgorithm to use extended market hours

* Fixed EqualWeightingPortfolioConstructionModelFutureRegressionAlgorithm to use extended market hours

* Fixed FutureContractsExtendedMarketHoursRegressionAlgorithm to use extended market hours

* Fixed FutureMarketOpenAndCloseRegressionAlgorithm to use extended market hours

* Fixed FutureMarketOpenConsolidatorRegressionAlgorithm to use extended market hours

* Fixed FutureOptionBuySellCallIntradayRegressionAlgorithm to use extended market hours

* Fixed FutureOptionCallITMExpiryRegressionAlgorithm to use extended market hours

* Fixed FutureOptionCallITMGreeksExpiryRegressionAlgorithm to use extended market hours

* Fixed FutureOptionCallOTMExpiryRegressionAlgorithm to use extended market hours

* Fixed FutureOptionDailyRegressionAlgorithm to use extended market hours

* Fixed FutureOptionHourlyRegressionAlgorithm to use extended market hours

* Fixed FutureOptionMultipleContractsInDifferentContractMonthsWithSameUnderlyingFutureRegressionAlgorithm to use extended market hours

* Fixed FutureOptionPutITMExpiryRegressionAlgorithm to use extended market hours

* Fixed FutureOptionPutOTMExpiryRegressionAlgorithm to use extended market hours

* Fixed FutureOptionShortCallITMExpiryRegressionAlgorithm to use extended market hours

* Fixed FutureOptionShortCallOTMExpiryRegressionAlgorithm to use extended market hours

* Fixed FutureOptionShortPutITMExpiryRegressionAlgorithm to use extended market hours

* Fixed FutureOptionShortPutOTMExpiryRegressionAlgorithm to use extended market hours

* Fixed FuturesAndFuturesOptionsExpiryTimeAndLiquidationRegressionAlgorithm to use extended market hours

* Fixed FuturesExpiredContractRegression to use extended market hours

* Fixed FutureSharingTickerRegressionAlgorithm to use extended market hours

* Fixed HistoryWithDifferentContinuousContractDepthOffsetsRegressionAlgorithm to use extended market hours

* Fixed HistoryWithDifferentDataMappingModeRegressionAlgorithm to use extended market hours

* Fixed HistoryWithDifferentDataNormalizationModeRegressionAlgorithm to use extended market hours

* Fixed LimitOrdersAreFilledAfterHoursForFuturesRegressionAlgorithm to use extended market hours

* Fixed OpenInterestFuturesRegressionAlgorithm to use extended market hours

* Fixed RegisterIndicatorRegressionAlgorithm to use extended market hours

* Fixed SetHoldingsFutureRegressionAlgorithm to use extended market hours

* Fixed WarmupFutureRegressionAlgorithm to use extended market hours

* Fixed AddFutureOptionSingleOptionChainSelectedInUniverseFilterRegressionAlgorithm to use extended market hours

* Fixed AlgorithmHistoryTests to use extended market hours for futures

* Fixed AlgorithmTradingTests to use extended market hours for futures

* Fixed BrokerageSetupHandlerTests to use extended market hours for futures

* Fixed TimeRulesTests to use extended market hours for futures

* Fixed FutureOptionMarginBuyingPowerModelTests to use extended market hours for futures

* Fixed FutureMarginBuyingPowerModelTests to use extended market hours for futures

* Fixed FileSystemDataFeedTests to use extended market hours for futures

* Fixed QuantBookHistoryTests to use extended market hours for futures

* Split BasicTemplateContinuousFutureAlgorithm to have an extended market version

* Fixed FutureMarketOpenAndCloseRegressionAlgorithm to use extended market hours

* Split BasicTemplateFuturesAlgorithm to have an extended market version

* Split BasicTemplateFuturesAlgorithm to have an extended market version

* Split BasicTemplateFuturesFrameworkAlgorithm to have an extended market version

* Split BasicTemplateFuturesHistoryAlgorithm to have an extended market version

* Revert AddFutureContractWithContinuousRegressionAlgorithm

* Revert AddFutureOptionContractDataStreamingRegressionAlgorithm and added data

* Revert AddFutureOptionContractFromFutureChainRegressionAlgorithm

* Revert AddFutureOptionSingleOptionChainSelectedInUniverseFilterRegressionAlgorithm

* Revert ConsolidateRegressionAlgorithm

* Revert Algorithm.CSharp/ContinuousBackMonthRawFutureRegressionAlgorithm.cs

* Revert ContinuousFutureBackMonthRegressionAlgorithm

* Revert ContinuousFutureHistoryRegressionAlgorithm

* Revert ContinuousFutureLimitIfTouchedOrderRegressionAlgorithm

* Revert ContinuousFutureRegressionAlgorithm

* Revert Algorithm.CSharp/DelistedFutureLiquidateRegressionAlgorithm.cs

* Revert EqualWeightingPortfolioConstructionModelFutureRegressionAlgorithm

* Split FutureMarketOpenAndCloseRegressionAlgorithm to have an extended market version

* Split FutureMarketOpenConsolidatorRegressionAlgorithm to have an extended market version

* Revert FutureOptionBuySellCallIntradayRegressionAlgorithm

* Revert FutureOptionCallITMExpiryRegressionAlgorithm

* Revert FutureOptionDailyRegressionAlgorithm

* Revert FutureOptionPutITMExpiryRegressionAlgorithm

* Revert FutureSharingTickerRegressionAlgorithm

* Revert FuturesAndFuturesOptionsExpiryTimeAndLiquidationRegressionAlgorithm

* Revert FuturesExpiredContractRegression

* Revert HistoryWithDifferentContinuousContractDepthOffsetsRegressionAlgorithm

* Revert HistoryWithDifferentDataMappingModeRegressionAlgorithm

* Revert HistoryWithDifferentDataNormalizationModeRegressionAlgorithm

* Revert OpenInterestFuturesRegressionAlgorithm

* Revert RegisterIndicatorRegressionAlgorithm

* Revert SetHoldingsFutureRegressionAlgorithm

* Revert WarmupFutureRegressionAlgorithm

* Revert AutomaticIndicatorWarmupDataTypeRegressionAlgorithm

* Some cleanup

* Address changes request

* Address changes request

* Add more Class III Milk data to fix DelistingFutureOptionDailyRegressionAlgorithm
2022-08-18 10:56:44 -03:00
Martin-Molinero 37dca9d458 Remove log packet aggregation (#6538)
Regression Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
- Remove log packet aggregation
- Clean up Queue api usage
2022-08-11 19:06:11 -03:00
Martin-Molinero 3f7d339463 Fix live result Handler null reference (#6521)
Regression Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
- Fix live result handler null reference, that could happen if the
  algorithm explodes mid initialization. Adding unit test
2022-08-03 19:57:03 -03:00
Martin-Molinero 3f2bcc1784 Add missing CancellationTokenSource dispose (#6512)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
- Add missing CancellationTokenSource disposals to avoid memory leaks
2022-08-01 11:12:01 -03:00
Martin-Molinero a9073396bd Add WarmUpResolution pass through (#6487)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
- Add new WarmUpResoltuion pass through version for friendly
user experience. Adding unit test
2022-07-18 13:47:04 -03:00
Martin-Molinero 38772a5a89 Do not apply split during live warmup (#6488)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
- Do not apply split during live warmup. Adding unit tests asserting the
  behavior for live dividends and splits during warmup and non warmup
2022-07-15 18:17:26 -03:00
Martin-Molinero 7540af454c Warmup resolution respected (#6467)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Respect warmup resolution given

- The data feed will respect the warmup resolution given and override
  the resolution used by the algorithm when adding a subscription.
  Updating regression algorithm to keep previous statistics. Adding new
  regression algorithm asserting the desired behavior

* Testing improvements

- Add more unit tests and regresion test
- Add missing data for crypto
- Fix bug with FFed data crossing after the end time of the warmup
  request

* Add more Warmup resolution regression algorithms

- Adding more warmup resolution regression algorithms, using
  Settings.WarmupResolution and an option selection case

* Add more warmup regression tests

- Adding more warmup regression tests.
- Will no longer skip universe selection subscriptions from warmup
  resolution enforcement. Updating regression algorithms data points

* Fix bug with data rounding

- Fix data rounding bug when warmup resolution is set to a different
  value than the original configuration. Updating regression algorithms
  to assert the expected behavior

* Address reviews

- Revert regression algorithms changes to use Resolution during warmup.
  Updating their stats.
- Adding new regression algorithms asserting the behavior warming up
  using a timespan and no warmup resolution
- Fix bug where data used to warmup the 'normal' enumerator will make it
  through into the warmup time span. Updating tests

* Address reviews

- Add missing comments, explaning warmup algorithms time span
  calculations.
- Revert changes in existing `WarmupOptionTimeSpanRegressionAlgorithm`
  to reduce diff to minimum
- Adding new warmup unit tests asseting algorithm warmup start time, for
  different combinations of bar count, timespan, resolution
2022-07-15 13:05:06 -03:00
Martin-Molinero e6f8a7535e Fix live future and option universe selection daily resolution (#6482)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Fix live future and option universe selection daily resolution

- Adding unit test reproducing issue for option and future universe
  chain using daily resolution, since was rounding down in UTC TZ the
  date might not change and it would re emit the same data point

* Address reviews

- Add more test cases
- Adding locks around RegisteredSecurityDataTypesProvider to allow it to
  be used by multiple threads at the same time without exploding
2022-07-13 19:10:13 -03:00
Martin-Molinero 778d436b94 Removed mapped futures contract is non tradable (#6479)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Minor fixes for mapped futures contract

- Future security was never able to be non tradable
- Skip remove future contracts from live holdings. Adding unit test
  reproducing issue and regression algorithm

* Fix removal of continuous futures

- The canonical continuous future would leave behind an OpenInterest
  subscription when removed because of a different in the way the
  configurations were built, this will now be centralized in a single
  method. Adding unit tests reproducing issue
- FillForwarding setting of the continuous future was not being
  respected. Updating algorithm reproducing the issue to assert the
  behavior failing in master

* Address reviews

- Minor refactors addressing reviews
2022-07-13 15:36:24 -03:00
Martin-Molinero 632833caaa Minor live related testing changes (#6481)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
- The history provider manager will log the underlying instances it's using.
- Allow brokerage test suites to reuse the algorithm stubs class
2022-07-13 15:26:45 -03:00
Martin-Molinero 6200988ad5 Fix result handler live holdings (#6478)
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Fix result handler live holdings

- Live result handler live holdings will include future assets
  associated with continuous futures. Adding unit tests

* Fixed for live holdings price rounding

- Live holdings will use the SPDB minimum price variation for rounding.
  Adding unit tests
2022-07-12 18:31:35 -03:00
Martin-Molinero 15e399c96b Live T-1 selection warmup (#6395)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Live T-1 selection warmup

- Only use BaseDataCollection class
- Reuse collection enumerator
- Remove FuturesChainUniverseSubscriptionEnumeratorFactory
- Universe selection will use Cache providers
- Add null reference check
- Add more tests
- Fixes for warmup fill forwarding. Add more tests

* Address reviews. Add missing comments
2022-06-10 17:25:20 -03:00
Martin-Molinero a3f4dff299 Refactor chain providers (#6394)
* Refactor chain providers

- ChainProviders will use quote, open interest and trade information,
  which ever is first. Updating regression algorithm changing contract
  being added.
- Add base BacktestingChainProvider for shared logic, reducing
  duplication.
- ChainProviders will now use the given IDataCacheProvider and
  IMapFileProvider
- If Providers are called for a date the exchange is closed they will
  search for the previous available date where the exchange is open.
  Adding unit tests.

* Address reviews. Logging improvements
2022-06-10 11:24:09 -03:00
Martin-Molinero cbfdd85cef Live Internal Subscription Start Time (#6392)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
- Live Trading internal subscription start time will be after warmup.
  Adding unit tests.
2022-06-09 17:28:32 -03:00
Jhonathan Abreu 5fa9dbad3d Fix Engine not running algorithm when calling Error() from Initialize() (#6379)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
2022-06-06 19:33:27 -03:00
Martin-Molinero 3a00b6963c Avoid sending expired symbols to IDQH (#6378)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
- Avoid sending expired symbols to the IDQH. This can happen during
  warmup period. Adding unit tests
2022-06-06 19:30:04 -03:00
Martin-Molinero e0b9a2735f Warmup fixes (#6293)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Add regression test reproducing issue

* WIP

* Update regression algorithms

* Improvements

* Cleanup and more fixes

- Clean up BaseDataExchange

* RealTimeHandler fixes

* Address reviews

* More comments, tests and minor tweaks

* Avoid false test failures

* Fixes

* Minor improvements

* Scheduled Event start time fix

* Add warmup option regression algorithm and fix

* Add WarmupFutureRegressionAlgorithm

* Normalize backtesting and live future selection
2022-05-31 10:03:56 -03:00
Martin-Molinero 284b26a9bd Performance improvements and fixes (#6330)
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Reuse GUID

* Reduce Lean task usage

* Reduce CPU usage

* Address reviews
2022-05-13 15:04:09 -03:00
Martin-Molinero 64125668db Catch expected test exception (#6317)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
2022-05-02 12:44:18 -03:00
Martin-Molinero b3c2303111 Zip data cache provider improvements (#6296)
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
- Throw on failure to store. Add unit test
- Fix handle leak
2022-04-22 13:52:30 -03:00
Ronit Jain 49562c712f Extract IB brokerage files and dependencies (#6283)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* make public safe to expose test properties

* extract ib files

* remove dependencies

* refactor to use ib reference as string

* remove un-used dependencies

* remove redundant refrences
2022-04-07 14:48:10 -03:00
quantify-cflynn a374ce9b6e Update Tick.cs Time parsing (#6280)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Update Tick.cs Time parsing

* Adjust equity tick millisecond read

* Fix tick sub millisecond precision

- DateTime.AddMilliseconds will rount to the nearest integer. So instead
  use AddTicks

* Avoid sporadic test failure

Co-authored-by: Martin-Molinero <martin@quantconnect.com>
2022-04-04 17:36:57 -03:00
Ronit Jain ad34429014 Refactor changes required for brokerages extraction (#6279)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* set price currency for open orders from brokerage setup handler, remove setters from brokerage

* order already has price currency being set by brokerage transcation handler

* remove as overwritten by BrokerageSetupHandler.GetOpenOrders

* refactor because can't use internal setters in private repos

* refactor, use ctor to set attributes

* remove redundant broker dependency

(cherry picked from commit 57b071eb174ca9fe857021ebffa7a546834e86f7)

* check limit price for limit order instead for checking price

* add price agrument for MarketOrder

* remove, not required anymore

* AddUnrequestedSecurity should not return null

* assert string in not null cases
2022-04-04 17:01:36 -03:00
Ronit Jain 15066ae5e1 Feature improve regression tests (#6245)
* add data count properties

* 'add history count property

* assert data counts

* update missing override

* consider override/virtual cases

* implement data count

* add message handler for regression tests

* use regression test message handler

* set algorithm manager for regression test message handler

* update data count

* check if stats are present, check if algo manager is not null

* update

* add c# algo

* make same as c# algo

* use new line

* logic shifted to RegressionTestMessageHandler

* cleanup

* auto cleanup

* skip non deterministic data count

* change data count

* use inheritance

* improve stats

* update couht

* add sma indicator to c# and customSMA to python

* call base method before executing further

* skip test

* revert to original

* add duplicate sma

* skip regression test
2022-03-15 16:51:15 -03:00
Ronit Jain f86926bf7a Fixes HistoryProviderManager slice merging by using algortime for all slices (#6226)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* remove un-used

* use utc time for slice sync

* use utc time

* refactor

* add regression test

* use utc time

* use utc time

* make utctime required parameter

* add utcTime in slice creation

* assert warm up complete

* check if algorithm is still warmingup

* use exchange tz
2022-02-18 20:43:33 -03:00
Martin-Molinero 306298a16f Update HistoryProviderManagerTests.cs
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
2022-02-15 12:34:00 -03:00
Adalyat Nazirov e8160f33d5 Binance extraction (#6193)
* extract binance brokerage

* remove binance tools from main toolbox app

* modify OrderTestParameters constructor

* fix BinanceBrokerage ref config.json

* update symbol properties

* move BinanceFeeModel tests to the right place
2022-02-15 12:05:47 -03:00
Ronit Jain 2d644d7879 Feature handle multiple history providers (#6187)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* initial commit

* get history from data providers

* merge history from history providers

* merge slice

* Add merge function in slice

* Update test

* merge aux data

* add data points in _data

* append new data points in original list

* Add test suite for HistoryProviderManager

* reduce complexity

* setup once

* add fake history provider

* doesn't count aux data

* add tests for options

* style changes

* add custom data

* merge rawDataList

* use array of history providers

* optimize

* fix formatting error

* add comment

* add tests

* simplify

* use list

* use abstraction

* split tests

* add test

* use abstraction to create generic enumertor class

* refactor

* accept list of type T

* sync history slices

* use SubscriptionDataReaderHistoryProvider for live

* rename test file

* return empty

* add tests

* cleanup

* address reviews

* optimize

* Fix method definition

* use initial time for basedata

* refactor

* re-use collection

* inherit HistoryProviderBase

* always return HistoryProviderManager

* add tests

* update rawDataList

* reset composer

* consider null elements

* add tests for binary search method

* Follow lean coding style

* revert

* remove binary search method

* convert to field
2022-02-14 11:58:00 -03:00
Martin-Molinero 1aaaa20c61 Fix daily auxiliary data points emission time (#6186)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* Use GC server mode for tests

* Fix daily auxiliary data points emission time

- Due to fillforwarding, in some cases with daily resolution symbol
  change events (generically any auxiliary data) would arrive late.
  Updating regression test to reproduce the issue. Adding unit test
- Some refactoring and logging improvements

* Address reviews

* Remove old xml docs param
2022-02-04 21:06:45 -03:00
Ronit Jain e5c709ee29 Extract zerodha brokerage out of LEAN (#6163)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Remove zerodha brokerage

* use different brokerage for tests

* remove zerodha files with conflict

* remove redundant dependecies
2022-01-24 18:43:12 -03:00
Martin-Molinero ca787d0a25 Add support for live price scaling (#6104)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* Add support for live price scaling

- Add support for live trading price scaling for continuous futures.
  Adding unit tests

* Move live price scale application. Updating unit tests

- Live trading application of price scaling will happen before fill
  forwarding and updating securities real time price. Updating unit
  tests to reproduce issue
2022-01-24 18:02:24 -03:00
Ricardo Andrés Marino Rojas 472f78cc53 Remove Quandl from LEAN (#6110)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* Remove Quandl from LEAN

* Nit changes and CustomLiveDataFeedTests.cs

* Resolve conflicts

* Remove files related with Quandl

* Fix bug

* Fix QuantBookHistoryTests.cs

* Fix bug

* Fix bug

* Fix unit tests

* Try fix regression tests

* Nit changes

* Fix bug

* Some of the requested changes

* The missing changes

* Requested changes

* Nit changes

* Revert "Nit changes"

This reverts commit 9800bc5c34f3ac20e30bea7a92dd4a9867213bb5.

* Nit changes

* Fix bug

* Requested changes

* Missing file using Quandl to be removed

* Nit changes

* Not applied nit change

* Nit change

* Nit change

* Add nasdaq-auth-code parameter in config.json

* Remove 'quandl-auth-token' from config.json

Co-authored-by: Martin-Molinero <martin@quantconnect.com>
2022-01-12 12:10:20 -03:00
Martin-Molinero 4b94f50754 Option selection improvements (#6144)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
- Zip entries will be sourced from cache provider
- Option underlying will use SubscriptionDataSource to fetch it's data.
  Fixing bug where it would let through an old data point, or miss
  sending data through.
2022-01-10 11:18:28 -03:00
Martin-Molinero 5bdc60b137 Fix for warmup history requests when internal subscriptions present (#6146)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
2022-01-10 11:10:44 -03:00
Martin-Molinero 7a753bfa3f Live mapped subscription will clone the underlying (#6141)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
- Live subscription enumerator will clone the underlying data set when
  live mapping is being done. To avoid issues where IDQH implementations
  could reuse a data point with same configurations. Adjusting unit test
  to reproduce issue
2022-01-05 13:45:52 -03:00
Martin-Molinero 55cb3bdaff ApiDataProvider Support Future map & factor files (#6132)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
- Add support for the ApiDataProvider to handle future map and factor
  files downloads. Adding unit test
2021-12-27 21:52:51 -03:00
Martin-Molinero 68e2a9170a Do not send internal SecurityChanges to Algorithm (#6118)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Do not send internal SecurityChanges to Algorithm

- Will not send internal security changes to the Algorithm by default.
  Following custom security changes filter pattern. Updating regression
  algorithms to assert behavior.
- The universe member will know wether it was added with internal
  configurations or not

* Address reviews use a separate collection for internals

* Refactor solution. Adding security changes constructor class
2021-12-21 20:24:32 -03:00
Alexandre Catarino 0e1cc288a6 Change Default Market of CFD to OANDA (#6107)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Oanda is currently the only provider.
2021-12-13 17:58:57 -03:00
Martin-Molinero 3b826535c7 Fix for ConcurrentDictionary Thread Safety (#6105)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
- ConcurrentDictionary with OrderBy is not thread safe, adding thread
  safe extension method. Adding unit tests
2021-12-13 10:45:58 -03:00
Martin-Molinero f42d7bb3a2 Align universe security and configuration TZ (#6102)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* Align universe security and configuration TZ

- Make sure universe security and configuration tz are aligned always.
  Adding unit test reproducing issue

* Address review UserDefinedUniverse will use UTC TZ
2021-12-09 15:53:56 -03:00
Martin-Molinero 87bd0d7792 CompositeDataQueueHandler handles duplicate configs (#6101)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* CompositeDataQueueHandler handles duplicate configs

- CompositeDataQueueHandler will handle duplicate subscriptions sharing
  the same config. Adding unit tests

* Refactor bug solution after review

- LiveSubscriptionEnumerator will not long perform any symbol mapping
  but will just trigger a new subscription call when remapped.
- CompositeDHQ will handling symbol mapping and keep track of these
  mapped configs to trigger unsubscribe accordengly. Adding unit tests

* Adjust solution after review

- To avoid breaking uniqueness of SubscriptionDataConfig will be adding
  a private `mapped` property that will change equality of mapped
  configs to avoid them clashing. Adding unit tests

* Rename IDQH subscription extension methods
2021-12-09 14:53:04 -03:00
Martin-Molinero dd27a382f7 Composite data queue handler IUniverseProvider (#6097)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Minor performance improvements

* CompositeDataQueueHandler is UniverseProvider

- CompositeDataQueueHandler implements the UniverseProvider interface.
  Adding unit tests
2021-12-06 10:27:18 -03:00
Ronit Jain 264c3c8374 Composite IDQH - Support multiple live data feeds (#6047)
* initial commit

* Follow IDQH implementation

* Expect a list of data handlers from LiveNodePacket

* Return null if can not subscribe

* refctor to add check for subscription

* initialze null

* Add tests

* Check subscribe retuns null/not-null

* cleanup

* Read all required IDQH credentails to job

* Use CDQH to handle all IDQH instances

* constructor abstraction to call from setjob

* use flag

* remove redundant because derived will call initialize on it

* abstract and initialize from setjob

* handle null enumerators

* get creds from data handlers

* handle single data handler value from data-queue-handler

* Fix to support a json array

* Fix missed constructor call

* change access modifier to access from Tests files

* Add test to get brokerageFactory from dataQueueHandler

* Fix init flag to handle all conditions

* Add docs

* initialize from setjob

* Check if websocket open before using

* change defination of initialzie to include tradier

* clean up

* change defination

* fix wrong api key name

* return empty enumerator

* check websocket open before sending request

* check connection before subscribing

* fix to include more cases

* check websocket open before sending request

* Minor refactoring

* reafctor and use IsConnected

* remove unused

* clean up

* Fix test cases

* reverse change

* include config changes

* connect to websocket from setjob

* check websocket connection from setjob

* clean up

* include condition for IDQH that are not brokerage

* Address review

* Add market check condition before subscribe

* Remove deprecated

* Minor fix for deserializing data queue handler

Co-authored-by: Martin-Molinero <martin@quantconnect.com>
2021-12-03 16:18:41 -03:00
Martin-Molinero 5f434f2fa5 SetupHandlers will exit if errors on initialize (#6092)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
- SetupHandlers will exit right away if detect errors on initialize.
  Adding unit tests
2021-12-02 19:24:51 -03:00
Martin-Molinero b4e95209f6 Fix for IB Option position race condition (#6090)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
- Fix for IB option position update race condition, where the BTH would
  incorrectly assume it should trigger an early option assignment. Adding unit tests
2021-12-02 17:09:50 -03:00