2471 Commits

Author SHA1 Message Date
Martin-Molinero 37dca9d458 Remove log packet aggregation (#6538)
Regression Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
- Remove log packet aggregation
- Clean up Queue api usage
2022-08-11 19:06:11 -03:00
Martin-Molinero 3f7d339463 Fix live result Handler null reference (#6521)
Regression Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
- Fix live result handler null reference, that could happen if the
  algorithm explodes mid initialization. Adding unit test
2022-08-03 19:57:03 -03:00
Martin-Molinero 3f2bcc1784 Add missing CancellationTokenSource dispose (#6512)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
- Add missing CancellationTokenSource disposals to avoid memory leaks
2022-08-01 11:12:01 -03:00
Martin-Molinero ef531e09bd Add Lean commands (#6513)
Regression Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Add Lean commands

- Add Lean commands to common project

* Address reviews

- Address reviews, refactor command interfaces and classes

* Rename FileCommandQueueHandler to FileCommandHandler
2022-07-28 16:44:40 -03:00
Martin-Molinero c43f472503 Fill model will use internal configurations (#6494)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
- Fill model will take into account internal configurations, this is
  specially relevant for continuous future mapped contracts. Updating
  regression algorithms and unit tests.
- Allowing internal configurations to update trade builder information
2022-07-19 19:00:35 -03:00
Martin-Molinero 38772a5a89 Do not apply split during live warmup (#6488)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
- Do not apply split during live warmup. Adding unit tests asserting the
  behavior for live dividends and splits during warmup and non warmup
2022-07-15 18:17:26 -03:00
Martin-Molinero 7540af454c Warmup resolution respected (#6467)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Respect warmup resolution given

- The data feed will respect the warmup resolution given and override
  the resolution used by the algorithm when adding a subscription.
  Updating regression algorithm to keep previous statistics. Adding new
  regression algorithm asserting the desired behavior

* Testing improvements

- Add more unit tests and regresion test
- Add missing data for crypto
- Fix bug with FFed data crossing after the end time of the warmup
  request

* Add more Warmup resolution regression algorithms

- Adding more warmup resolution regression algorithms, using
  Settings.WarmupResolution and an option selection case

* Add more warmup regression tests

- Adding more warmup regression tests.
- Will no longer skip universe selection subscriptions from warmup
  resolution enforcement. Updating regression algorithms data points

* Fix bug with data rounding

- Fix data rounding bug when warmup resolution is set to a different
  value than the original configuration. Updating regression algorithms
  to assert the expected behavior

* Address reviews

- Revert regression algorithms changes to use Resolution during warmup.
  Updating their stats.
- Adding new regression algorithms asserting the behavior warming up
  using a timespan and no warmup resolution
- Fix bug where data used to warmup the 'normal' enumerator will make it
  through into the warmup time span. Updating tests

* Address reviews

- Add missing comments, explaning warmup algorithms time span
  calculations.
- Revert changes in existing `WarmupOptionTimeSpanRegressionAlgorithm`
  to reduce diff to minimum
- Adding new warmup unit tests asseting algorithm warmup start time, for
  different combinations of bar count, timespan, resolution
2022-07-15 13:05:06 -03:00
Martin-Molinero 07aa5625e1 Avoid division by zero error in backtesting result handler (#6484)
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Minor improvements

- Avoid division by zero error in backtesting result handler. Race
  condition where the SetAlgorithm implementation would still be running
  and we try to loop through the update method
- Improvement nullreference exception message
- Cleanup resolution to timespan extension method

* Address reviews

- Cleanup data config fill forward conditional statement for tick
  resolution
2022-07-14 12:54:55 -03:00
Martin-Molinero 31fc7ac236 Do not generate Equity Quote Daily data (#6483)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
- Fix for so the random data generator does not try to create daily
  equity quote data
2022-07-14 10:20:34 -03:00
Martin-Molinero e6f8a7535e Fix live future and option universe selection daily resolution (#6482)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Fix live future and option universe selection daily resolution

- Adding unit test reproducing issue for option and future universe
  chain using daily resolution, since was rounding down in UTC TZ the
  date might not change and it would re emit the same data point

* Address reviews

- Add more test cases
- Adding locks around RegisteredSecurityDataTypesProvider to allow it to
  be used by multiple threads at the same time without exploding
2022-07-13 19:10:13 -03:00
Martin-Molinero 778d436b94 Removed mapped futures contract is non tradable (#6479)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Minor fixes for mapped futures contract

- Future security was never able to be non tradable
- Skip remove future contracts from live holdings. Adding unit test
  reproducing issue and regression algorithm

* Fix removal of continuous futures

- The canonical continuous future would leave behind an OpenInterest
  subscription when removed because of a different in the way the
  configurations were built, this will now be centralized in a single
  method. Adding unit tests reproducing issue
- FillForwarding setting of the continuous future was not being
  respected. Updating algorithm reproducing the issue to assert the
  behavior failing in master

* Address reviews

- Minor refactors addressing reviews
2022-07-13 15:36:24 -03:00
Martin-Molinero 632833caaa Minor live related testing changes (#6481)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
- The history provider manager will log the underlying instances it's using.
- Allow brokerage test suites to reuse the algorithm stubs class
2022-07-13 15:26:45 -03:00
Martin-Molinero 6200988ad5 Fix result handler live holdings (#6478)
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Fix result handler live holdings

- Live result handler live holdings will include future assets
  associated with continuous futures. Adding unit tests

* Fixed for live holdings price rounding

- Live holdings will use the SPDB minimum price variation for rounding.
  Adding unit tests
2022-07-12 18:31:35 -03:00
Martin-Molinero 826f2cbbf9 Minor refactor messaging handler (#6472)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
2022-07-11 16:24:26 -03:00
Martin-Molinero 8c0b60cd2f Refactor work queue scheduling (#6465)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
- Remove thread in charge of sorting and updating the queues weight.
  This will be handled by each worker thread instead. Avoid cases where
  the work count is so high that the sorting thread in master would not
  be able to update all the weights, causing data to be enqueued beyond
  the max prefetch limit. Adding benchmark algorithm reproducing issue
2022-07-06 12:13:32 -03:00
Martin-Molinero 64f0a9e0a2 Fix for ApiDataProvider (#6461)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
- Avoid race condition while downloading data.
- Reused http client for downloads
2022-07-05 18:43:58 -03:00
Martin-Molinero 3f407cd80f Smaller live warmup history requests (#6455)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
- Smaller live warmup history requests, will keep track of the last point
  we got from the file based enumerator and start our history enumeration from this point
2022-07-05 13:15:23 -03:00
Martin-Molinero 18eec48a3d Avoid PythonNet runtime stash on shutdown (#6441)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Can't relesase the GIL after shutdown

* Bump version to pythonnet 2.0.16
2022-06-27 13:04:01 -03:00
Martin-Molinero 299cb79832 OnWarmupFinished always called (#6413)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* OnWarmupFinished always called

- Always call OnWarmupFinished. Adding regression tests

* Assert method called with counter
2022-06-20 14:24:12 -03:00
Martin-Molinero a8e7a8b27e Reconcile the Market Opening for Futures and Scheduled Events (#6397)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
- Changes taken from https://github.com/QuantConnect/Lean/pull/6227
- Rebased + a few more tweaks and added tests
2022-06-15 13:47:00 -03:00
Martin-Molinero 15e399c96b Live T-1 selection warmup (#6395)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Live T-1 selection warmup

- Only use BaseDataCollection class
- Reuse collection enumerator
- Remove FuturesChainUniverseSubscriptionEnumeratorFactory
- Universe selection will use Cache providers
- Add null reference check
- Add more tests
- Fixes for warmup fill forwarding. Add more tests

* Address reviews. Add missing comments
2022-06-10 17:25:20 -03:00
Martin-Molinero a3f4dff299 Refactor chain providers (#6394)
* Refactor chain providers

- ChainProviders will use quote, open interest and trade information,
  which ever is first. Updating regression algorithm changing contract
  being added.
- Add base BacktestingChainProvider for shared logic, reducing
  duplication.
- ChainProviders will now use the given IDataCacheProvider and
  IMapFileProvider
- If Providers are called for a date the exchange is closed they will
  search for the previous available date where the exchange is open.
  Adding unit tests.

* Address reviews. Logging improvements
2022-06-10 11:24:09 -03:00
Martin-Molinero cbfdd85cef Live Internal Subscription Start Time (#6392)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
- Live Trading internal subscription start time will be after warmup.
  Adding unit tests.
2022-06-09 17:28:32 -03:00
Jhonathan Abreu 5fa9dbad3d Fix Engine not running algorithm when calling Error() from Initialize() (#6379)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
2022-06-06 19:33:27 -03:00
Martin-Molinero 3a00b6963c Avoid sending expired symbols to IDQH (#6378)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
- Avoid sending expired symbols to the IDQH. This can happen during
  warmup period. Adding unit tests
2022-06-06 19:30:04 -03:00
Martin-Molinero cde9be8d16 Fix 'OnWarmupFinished' callback (#6377)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
- Fix missing 'OnWarmupFinished' callback for python. Adding regression
  algorithms
2022-06-06 11:56:02 -03:00
Martin-Molinero 0c0ee829a2 Warmup state minor fixes (#6367)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Warmup minor fixes

- Result handler fix
- Centralize and normalize status update during warmup

* Address reviews

* Minor adjustments

* Add console message during warmup period

* Fix warmup percentage update
2022-06-02 18:20:02 -03:00
Martin-Molinero e0b9a2735f Warmup fixes (#6293)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Add regression test reproducing issue

* WIP

* Update regression algorithms

* Improvements

* Cleanup and more fixes

- Clean up BaseDataExchange

* RealTimeHandler fixes

* Address reviews

* More comments, tests and minor tweaks

* Avoid false test failures

* Fixes

* Minor improvements

* Scheduled Event start time fix

* Add warmup option regression algorithm and fix

* Add WarmupFutureRegressionAlgorithm

* Normalize backtesting and live future selection
2022-05-31 10:03:56 -03:00
Martin-Molinero 895dfedf81 User Account currency symbol for fees and capacity (#6359)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* User Account currency symbol for fees and capacity

* Update expected currency symbol stats
2022-05-27 16:36:44 -03:00
Martin-Molinero 00dd3dbd2c Bump to pythonNet 2.0.15 (#6342)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
2022-05-18 13:09:45 -03:00
Martin-Molinero cb2062debd Expand GetSecondUnevenWait usages (#6340)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Reduce Live CPU usage

* Address selfreview
2022-05-16 20:12:22 -03:00
Martin-Molinero 284b26a9bd Performance improvements and fixes (#6330)
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Reuse GUID

* Reduce Lean task usage

* Reduce CPU usage

* Address reviews
2022-05-13 15:04:09 -03:00
Martin-Molinero eb995b86ea Update to pythonNet 2.0.14 (#6322)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Update to pythonNet 2.0.14

* Add unit test reproducing issue
2022-05-06 17:29:55 -03:00
Martin-Molinero b7a21ce4be Update to net6 (#6311)
* Update to net6

* Bump pythonNet to 2.0.13
2022-05-03 11:45:55 -03:00
Martin-Molinero b9d3d99917 Bump pythonNet version 2.0.12 (#6310)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Updates after pythonNet rebase

* Bump pythonNet version 2.0.12

* Fix exception types being thrown
2022-05-02 14:38:50 -03:00
Martin-Molinero 813412afb2 Persist and order runtime statistics (#6314)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
2022-04-29 20:59:40 -03:00
Ricardo Andrés Marino Rojas 6d181cc452 Fix bug with SetBenchmark() using custom data (#6304)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* First commit

* Solve the easy case

* Solve bugs

* Solve bugs

* Requested changes

* Requested changes
2022-04-27 13:58:10 -03:00
Martin-Molinero b3c2303111 Zip data cache provider improvements (#6296)
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
- Throw on failure to store. Add unit test
- Fix handle leak
2022-04-22 13:52:30 -03:00
Ronit Jain 49562c712f Extract IB brokerage files and dependencies (#6283)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* make public safe to expose test properties

* extract ib files

* remove dependencies

* refactor to use ib reference as string

* remove un-used dependencies

* remove redundant refrences
2022-04-07 14:48:10 -03:00
maciek231 73fa0461ed Added checking if algorithm is stopped in initialize method (#6282)
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Added checking if algorithm is stopped in initialize method

* Quit on initialize adjusment

- Minor adjustments to solution. Adding more regression algorithms

Co-authored-by: Martin-Molinero <martin@quantconnect.com>
2022-04-06 19:56:18 -03:00
Ronit Jain ad34429014 Refactor changes required for brokerages extraction (#6279)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* set price currency for open orders from brokerage setup handler, remove setters from brokerage

* order already has price currency being set by brokerage transcation handler

* remove as overwritten by BrokerageSetupHandler.GetOpenOrders

* refactor because can't use internal setters in private repos

* refactor, use ctor to set attributes

* remove redundant broker dependency

(cherry picked from commit 57b071eb174ca9fe857021ebffa7a546834e86f7)

* check limit price for limit order instead for checking price

* add price agrument for MarketOrder

* remove, not required anymore

* AddUnrequestedSecurity should not return null

* assert string in not null cases
2022-04-04 17:01:36 -03:00
Martin-Molinero 06c05c4274 Fully reset security cache (#6277)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
- Fully reset the security cache when it's finally removed from the algorithm.
   Adding regression algorithm reproducing issue
- Updating regression algorithms which would trade based on data
  previously available
2022-04-01 15:00:29 -03:00
maciek231 58e89872c1 Fix HttpClient request timeout (#6273)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Fix HttpClient request timeout

* Update LiveOptionChainProvider.cs

Remove unrequired accepted encoding specification

Co-authored-by: maciej.tromiczak <maciej.tromiczak@96volt.com>
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
2022-03-30 17:34:02 -03:00
Ronit Jain 15066ae5e1 Feature improve regression tests (#6245)
* add data count properties

* 'add history count property

* assert data counts

* update missing override

* consider override/virtual cases

* implement data count

* add message handler for regression tests

* use regression test message handler

* set algorithm manager for regression test message handler

* update data count

* check if stats are present, check if algo manager is not null

* update

* add c# algo

* make same as c# algo

* use new line

* logic shifted to RegressionTestMessageHandler

* cleanup

* auto cleanup

* skip non deterministic data count

* change data count

* use inheritance

* improve stats

* update couht

* add sma indicator to c# and customSMA to python

* call base method before executing further

* skip test

* revert to original

* add duplicate sma

* skip regression test
2022-03-15 16:51:15 -03:00
Ronit Jain f86926bf7a Fixes HistoryProviderManager slice merging by using algortime for all slices (#6226)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* remove un-used

* use utc time for slice sync

* use utc time

* refactor

* add regression test

* use utc time

* use utc time

* make utctime required parameter

* add utcTime in slice creation

* assert warm up complete

* check if algorithm is still warmingup

* use exchange tz
2022-02-18 20:43:33 -03:00
Jasper van Merle c598a8d260 Add support for array history-provider values to QuantBook (#6219)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
2022-02-17 11:24:20 -03:00
Martin-Molinero e63bfc9127 Adjust virtual position in margin accounts (#6214)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
- The BrokerTransactionHandler will also adjust virtual positions for
  margin accounts when fees are in base currency and the asset is a
  crypto or forex pair. Adding new regression tests reproducing issue
2022-02-16 15:25:47 -03:00
Ronit Jain 2d644d7879 Feature handle multiple history providers (#6187)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* initial commit

* get history from data providers

* merge history from history providers

* merge slice

* Add merge function in slice

* Update test

* merge aux data

* add data points in _data

* append new data points in original list

* Add test suite for HistoryProviderManager

* reduce complexity

* setup once

* add fake history provider

* doesn't count aux data

* add tests for options

* style changes

* add custom data

* merge rawDataList

* use array of history providers

* optimize

* fix formatting error

* add comment

* add tests

* simplify

* use list

* use abstraction

* split tests

* add test

* use abstraction to create generic enumertor class

* refactor

* accept list of type T

* sync history slices

* use SubscriptionDataReaderHistoryProvider for live

* rename test file

* return empty

* add tests

* cleanup

* address reviews

* optimize

* Fix method definition

* use initial time for basedata

* refactor

* re-use collection

* inherit HistoryProviderBase

* always return HistoryProviderManager

* add tests

* update rawDataList

* reset composer

* consider null elements

* add tests for binary search method

* Follow lean coding style

* revert

* remove binary search method

* convert to field
2022-02-14 11:58:00 -03:00
Martin-Molinero 459f60603b Fees is base currency subtracted from quote currency (#6188)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* WIP

* Fix fees in base currency not being subtracted

- Fix fees in base currency not being subtracted from the quote currency
  for crypto cash accounts. Updating regression tests to assert
  portfolio, cashbook state and holdings state.

* Fix unit test race condition
2022-02-07 14:57:40 -03:00
Martin-Molinero 1aaaa20c61 Fix daily auxiliary data points emission time (#6186)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* Use GC server mode for tests

* Fix daily auxiliary data points emission time

- Due to fillforwarding, in some cases with daily resolution symbol
  change events (generically any auxiliary data) would arrive late.
  Updating regression test to reproduce the issue. Adding unit test
- Some refactoring and logging improvements

* Address reviews

* Remove old xml docs param
2022-02-04 21:06:45 -03:00