2457 Commits

Author SHA1 Message Date
Martin-Molinero 8c0b60cd2f Refactor work queue scheduling (#6465)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
- Remove thread in charge of sorting and updating the queues weight.
  This will be handled by each worker thread instead. Avoid cases where
  the work count is so high that the sorting thread in master would not
  be able to update all the weights, causing data to be enqueued beyond
  the max prefetch limit. Adding benchmark algorithm reproducing issue
2022-07-06 12:13:32 -03:00
Martin-Molinero 64f0a9e0a2 Fix for ApiDataProvider (#6461)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
- Avoid race condition while downloading data.
- Reused http client for downloads
2022-07-05 18:43:58 -03:00
Martin-Molinero 3f407cd80f Smaller live warmup history requests (#6455)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
- Smaller live warmup history requests, will keep track of the last point
  we got from the file based enumerator and start our history enumeration from this point
2022-07-05 13:15:23 -03:00
Martin-Molinero 18eec48a3d Avoid PythonNet runtime stash on shutdown (#6441)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Can't relesase the GIL after shutdown

* Bump version to pythonnet 2.0.16
2022-06-27 13:04:01 -03:00
Martin-Molinero 299cb79832 OnWarmupFinished always called (#6413)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* OnWarmupFinished always called

- Always call OnWarmupFinished. Adding regression tests

* Assert method called with counter
2022-06-20 14:24:12 -03:00
Martin-Molinero a8e7a8b27e Reconcile the Market Opening for Futures and Scheduled Events (#6397)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
- Changes taken from https://github.com/QuantConnect/Lean/pull/6227
- Rebased + a few more tweaks and added tests
2022-06-15 13:47:00 -03:00
Martin-Molinero 15e399c96b Live T-1 selection warmup (#6395)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Live T-1 selection warmup

- Only use BaseDataCollection class
- Reuse collection enumerator
- Remove FuturesChainUniverseSubscriptionEnumeratorFactory
- Universe selection will use Cache providers
- Add null reference check
- Add more tests
- Fixes for warmup fill forwarding. Add more tests

* Address reviews. Add missing comments
2022-06-10 17:25:20 -03:00
Martin-Molinero a3f4dff299 Refactor chain providers (#6394)
* Refactor chain providers

- ChainProviders will use quote, open interest and trade information,
  which ever is first. Updating regression algorithm changing contract
  being added.
- Add base BacktestingChainProvider for shared logic, reducing
  duplication.
- ChainProviders will now use the given IDataCacheProvider and
  IMapFileProvider
- If Providers are called for a date the exchange is closed they will
  search for the previous available date where the exchange is open.
  Adding unit tests.

* Address reviews. Logging improvements
2022-06-10 11:24:09 -03:00
Martin-Molinero cbfdd85cef Live Internal Subscription Start Time (#6392)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
- Live Trading internal subscription start time will be after warmup.
  Adding unit tests.
2022-06-09 17:28:32 -03:00
Jhonathan Abreu 5fa9dbad3d Fix Engine not running algorithm when calling Error() from Initialize() (#6379)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
2022-06-06 19:33:27 -03:00
Martin-Molinero 3a00b6963c Avoid sending expired symbols to IDQH (#6378)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
- Avoid sending expired symbols to the IDQH. This can happen during
  warmup period. Adding unit tests
2022-06-06 19:30:04 -03:00
Martin-Molinero cde9be8d16 Fix 'OnWarmupFinished' callback (#6377)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
- Fix missing 'OnWarmupFinished' callback for python. Adding regression
  algorithms
2022-06-06 11:56:02 -03:00
Martin-Molinero 0c0ee829a2 Warmup state minor fixes (#6367)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Warmup minor fixes

- Result handler fix
- Centralize and normalize status update during warmup

* Address reviews

* Minor adjustments

* Add console message during warmup period

* Fix warmup percentage update
2022-06-02 18:20:02 -03:00
Martin-Molinero e0b9a2735f Warmup fixes (#6293)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Add regression test reproducing issue

* WIP

* Update regression algorithms

* Improvements

* Cleanup and more fixes

- Clean up BaseDataExchange

* RealTimeHandler fixes

* Address reviews

* More comments, tests and minor tweaks

* Avoid false test failures

* Fixes

* Minor improvements

* Scheduled Event start time fix

* Add warmup option regression algorithm and fix

* Add WarmupFutureRegressionAlgorithm

* Normalize backtesting and live future selection
2022-05-31 10:03:56 -03:00
Martin-Molinero 895dfedf81 User Account currency symbol for fees and capacity (#6359)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* User Account currency symbol for fees and capacity

* Update expected currency symbol stats
2022-05-27 16:36:44 -03:00
Martin-Molinero 00dd3dbd2c Bump to pythonNet 2.0.15 (#6342)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
2022-05-18 13:09:45 -03:00
Martin-Molinero cb2062debd Expand GetSecondUnevenWait usages (#6340)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Reduce Live CPU usage

* Address selfreview
2022-05-16 20:12:22 -03:00
Martin-Molinero 284b26a9bd Performance improvements and fixes (#6330)
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Reuse GUID

* Reduce Lean task usage

* Reduce CPU usage

* Address reviews
2022-05-13 15:04:09 -03:00
Martin-Molinero eb995b86ea Update to pythonNet 2.0.14 (#6322)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Update to pythonNet 2.0.14

* Add unit test reproducing issue
2022-05-06 17:29:55 -03:00
Martin-Molinero b7a21ce4be Update to net6 (#6311)
* Update to net6

* Bump pythonNet to 2.0.13
2022-05-03 11:45:55 -03:00
Martin-Molinero b9d3d99917 Bump pythonNet version 2.0.12 (#6310)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Updates after pythonNet rebase

* Bump pythonNet version 2.0.12

* Fix exception types being thrown
2022-05-02 14:38:50 -03:00
Martin-Molinero 813412afb2 Persist and order runtime statistics (#6314)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
2022-04-29 20:59:40 -03:00
Ricardo Andrés Marino Rojas 6d181cc452 Fix bug with SetBenchmark() using custom data (#6304)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* First commit

* Solve the easy case

* Solve bugs

* Solve bugs

* Requested changes

* Requested changes
2022-04-27 13:58:10 -03:00
Martin-Molinero b3c2303111 Zip data cache provider improvements (#6296)
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
- Throw on failure to store. Add unit test
- Fix handle leak
2022-04-22 13:52:30 -03:00
Ronit Jain 49562c712f Extract IB brokerage files and dependencies (#6283)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* make public safe to expose test properties

* extract ib files

* remove dependencies

* refactor to use ib reference as string

* remove un-used dependencies

* remove redundant refrences
2022-04-07 14:48:10 -03:00
maciek231 73fa0461ed Added checking if algorithm is stopped in initialize method (#6282)
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Added checking if algorithm is stopped in initialize method

* Quit on initialize adjusment

- Minor adjustments to solution. Adding more regression algorithms

Co-authored-by: Martin-Molinero <martin@quantconnect.com>
2022-04-06 19:56:18 -03:00
Ronit Jain ad34429014 Refactor changes required for brokerages extraction (#6279)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* set price currency for open orders from brokerage setup handler, remove setters from brokerage

* order already has price currency being set by brokerage transcation handler

* remove as overwritten by BrokerageSetupHandler.GetOpenOrders

* refactor because can't use internal setters in private repos

* refactor, use ctor to set attributes

* remove redundant broker dependency

(cherry picked from commit 57b071eb174ca9fe857021ebffa7a546834e86f7)

* check limit price for limit order instead for checking price

* add price agrument for MarketOrder

* remove, not required anymore

* AddUnrequestedSecurity should not return null

* assert string in not null cases
2022-04-04 17:01:36 -03:00
Martin-Molinero 06c05c4274 Fully reset security cache (#6277)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
- Fully reset the security cache when it's finally removed from the algorithm.
   Adding regression algorithm reproducing issue
- Updating regression algorithms which would trade based on data
  previously available
2022-04-01 15:00:29 -03:00
maciek231 58e89872c1 Fix HttpClient request timeout (#6273)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Fix HttpClient request timeout

* Update LiveOptionChainProvider.cs

Remove unrequired accepted encoding specification

Co-authored-by: maciej.tromiczak <maciej.tromiczak@96volt.com>
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
2022-03-30 17:34:02 -03:00
Ronit Jain 15066ae5e1 Feature improve regression tests (#6245)
* add data count properties

* 'add history count property

* assert data counts

* update missing override

* consider override/virtual cases

* implement data count

* add message handler for regression tests

* use regression test message handler

* set algorithm manager for regression test message handler

* update data count

* check if stats are present, check if algo manager is not null

* update

* add c# algo

* make same as c# algo

* use new line

* logic shifted to RegressionTestMessageHandler

* cleanup

* auto cleanup

* skip non deterministic data count

* change data count

* use inheritance

* improve stats

* update couht

* add sma indicator to c# and customSMA to python

* call base method before executing further

* skip test

* revert to original

* add duplicate sma

* skip regression test
2022-03-15 16:51:15 -03:00
Ronit Jain f86926bf7a Fixes HistoryProviderManager slice merging by using algortime for all slices (#6226)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* remove un-used

* use utc time for slice sync

* use utc time

* refactor

* add regression test

* use utc time

* use utc time

* make utctime required parameter

* add utcTime in slice creation

* assert warm up complete

* check if algorithm is still warmingup

* use exchange tz
2022-02-18 20:43:33 -03:00
Jasper van Merle c598a8d260 Add support for array history-provider values to QuantBook (#6219)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
2022-02-17 11:24:20 -03:00
Martin-Molinero e63bfc9127 Adjust virtual position in margin accounts (#6214)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
- The BrokerTransactionHandler will also adjust virtual positions for
  margin accounts when fees are in base currency and the asset is a
  crypto or forex pair. Adding new regression tests reproducing issue
2022-02-16 15:25:47 -03:00
Ronit Jain 2d644d7879 Feature handle multiple history providers (#6187)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* initial commit

* get history from data providers

* merge history from history providers

* merge slice

* Add merge function in slice

* Update test

* merge aux data

* add data points in _data

* append new data points in original list

* Add test suite for HistoryProviderManager

* reduce complexity

* setup once

* add fake history provider

* doesn't count aux data

* add tests for options

* style changes

* add custom data

* merge rawDataList

* use array of history providers

* optimize

* fix formatting error

* add comment

* add tests

* simplify

* use list

* use abstraction

* split tests

* add test

* use abstraction to create generic enumertor class

* refactor

* accept list of type T

* sync history slices

* use SubscriptionDataReaderHistoryProvider for live

* rename test file

* return empty

* add tests

* cleanup

* address reviews

* optimize

* Fix method definition

* use initial time for basedata

* refactor

* re-use collection

* inherit HistoryProviderBase

* always return HistoryProviderManager

* add tests

* update rawDataList

* reset composer

* consider null elements

* add tests for binary search method

* Follow lean coding style

* revert

* remove binary search method

* convert to field
2022-02-14 11:58:00 -03:00
Martin-Molinero 459f60603b Fees is base currency subtracted from quote currency (#6188)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* WIP

* Fix fees in base currency not being subtracted

- Fix fees in base currency not being subtracted from the quote currency
  for crypto cash accounts. Updating regression tests to assert
  portfolio, cashbook state and holdings state.

* Fix unit test race condition
2022-02-07 14:57:40 -03:00
Martin-Molinero 1aaaa20c61 Fix daily auxiliary data points emission time (#6186)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* Use GC server mode for tests

* Fix daily auxiliary data points emission time

- Due to fillforwarding, in some cases with daily resolution symbol
  change events (generically any auxiliary data) would arrive late.
  Updating regression test to reproduce the issue. Adding unit test
- Some refactoring and logging improvements

* Address reviews

* Remove old xml docs param
2022-02-04 21:06:45 -03:00
Ricardo Andrés Marino Rojas a675aca7e5 Refactor GetFilePath() (#6164)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* Refactor `GetFilePath()`
Add also useful methods to use with this one

* Nit changes

* Requested changes

* Requested changes

* Restore SaveString()

* Nit changes

* Address self review

* Test improvements

* Adjust example KerasNeuralNetworkAlgorithm

* Minor tweak for KerasNeuralNetworkAlgorithm.py

Co-authored-by: Martin-Molinero <martin@quantconnect.com>
2022-02-04 16:58:25 -03:00
Martin-Molinero fc0b2f3fa4 Fix for Add & Remove option contract case (#6172)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
2022-01-28 14:13:21 -03:00
Martin-Molinero c4a2d6eef4 Crypto base currency fees handled correctly (#6166)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Binance fees deducted from fill quantity accordengly

- For Binance cash accounts while buying, if fees are from the base
  currency of leans virtual position, we need to deduct the fee from the
  fill quantity, else we can end with a position bigger that it actually
  is and not be able to liquidate

* Refactor solution

- Refactor solution into a more generic approach solving fees in base
  currency at the BrokerageTransactionHandler level, covering all
  brokerages that require it. Adding regression algorithm reproducing
  issue.
- Update Bitfinex and Binance fee models to correctly reflact reality

* Log fill quantity adjusment once
2022-01-28 13:02:58 -03:00
Martin-Molinero b698641c90 Minor tweak for ApiDataProvider to support India (#6169)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
- Minor tweaks for the ApiDataProvider to better support India market
2022-01-24 21:27:22 -03:00
Martin-Molinero ca787d0a25 Add support for live price scaling (#6104)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* Add support for live price scaling

- Add support for live trading price scaling for continuous futures.
  Adding unit tests

* Move live price scale application. Updating unit tests

- Live trading application of price scaling will happen before fill
  forwarding and updating securities real time price. Updating unit
  tests to reproduce issue
2022-01-24 18:02:24 -03:00
Ricardo Andrés Marino Rojas 472f78cc53 Remove Quandl from LEAN (#6110)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* Remove Quandl from LEAN

* Nit changes and CustomLiveDataFeedTests.cs

* Resolve conflicts

* Remove files related with Quandl

* Fix bug

* Fix QuantBookHistoryTests.cs

* Fix bug

* Fix bug

* Fix unit tests

* Try fix regression tests

* Nit changes

* Fix bug

* Some of the requested changes

* The missing changes

* Requested changes

* Nit changes

* Revert "Nit changes"

This reverts commit 9800bc5c34f3ac20e30bea7a92dd4a9867213bb5.

* Nit changes

* Fix bug

* Requested changes

* Missing file using Quandl to be removed

* Nit changes

* Not applied nit change

* Nit change

* Nit change

* Add nasdaq-auth-code parameter in config.json

* Remove 'quandl-auth-token' from config.json

Co-authored-by: Martin-Molinero <martin@quantconnect.com>
2022-01-12 12:10:20 -03:00
Adalyat Nazirov 0c26d42561 Feature 2839 black scholes data generator (#6135)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* replace to local functions as they are more performant

* fix random generator upper bound

Next() includes minValue, but not maxValue, so we increment it +1

* introduce abstract layers

* refactoring

* fix tets

* adapt tests

* fixup

* implement blackschole price model for options

* use risk free rate

* use ql price model

* wip

* change interface

* fix

* tidy up the code

* wip

* iterate groupped symbols

* wip

* wip

* fix

* allow symbol of different types

* improve settings

* wip

* iterate full range

* fix issue with negative option

* fix

* fixup

* use StandardDeviationOfReturnsVolatilityModel

* re-use existing tick types per security type

* parametrize underlying security type

* use default option style

* dynamic option price model

* fix enumeration

* test

* fix unit tests

* refactor code

* remove unused file

* minor tweaks and refactoring

* rename symbol generator class

* fix interface

* add comments

* more comments and unit tests

* more tests

* add disclaimer

* more tests

* more comments and tests

* split tests into different files

* tidy up the code

* tidy up the code; more tests

* refactor TickGenerator => use security price directly on each iteration

* remove dupe; reuse main constructor

* use SecurityManager, refactor code

* bugfix: save ticks in history array

* check volatility warm up & tests

* more unit tests

* describe volatility period span in settings

* rename command line option

* Minor adjusments. Address review

- Use Lean log handler instead of writting directly to console
- Rename BlackShcolesPriceGenerator to generically OptionPriceModelPriceGenerator
- Minor format clean up & standarization
- Add support for specifying the option chain size

* Rename TickGenerator private fields

* Fix unit tests

* fix tests class name

* Support tickers being specified

Co-authored-by: Martin-Molinero <martin@quantconnect.com>
2022-01-10 17:21:03 -03:00
Martin-Molinero 4b94f50754 Option selection improvements (#6144)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
- Zip entries will be sourced from cache provider
- Option underlying will use SubscriptionDataSource to fetch it's data.
  Fixing bug where it would let through an old data point, or miss
  sending data through.
2022-01-10 11:18:28 -03:00
Martin-Molinero 7a753bfa3f Live mapped subscription will clone the underlying (#6141)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
- Live subscription enumerator will clone the underlying data set when
  live mapping is being done. To avoid issues where IDQH implementations
  could reuse a data point with same configurations. Adjusting unit test
  to reproduce issue
2022-01-05 13:45:52 -03:00
Martin-Molinero 55cb3bdaff ApiDataProvider Support Future map & factor files (#6132)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
- Add support for the ApiDataProvider to handle future map and factor
  files downloads. Adding unit test
2021-12-27 21:52:51 -03:00
Martin-Molinero 10bb627fc2 Update to pythonNet 2.0.11 (#6131)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
2021-12-27 15:49:24 -03:00
Martin-Molinero 68e2a9170a Do not send internal SecurityChanges to Algorithm (#6118)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Do not send internal SecurityChanges to Algorithm

- Will not send internal security changes to the Algorithm by default.
  Following custom security changes filter pattern. Updating regression
  algorithms to assert behavior.
- The universe member will know wether it was added with internal
  configurations or not

* Address reviews use a separate collection for internals

* Refactor solution. Adding security changes constructor class
2021-12-21 20:24:32 -03:00
Martin-Molinero 4fdd60d146 Use 64 when saving zips update dotnetZip (#6114)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
2021-12-20 16:39:06 -03:00
Martin-Molinero abbb50e209 Minor improvements for data reading and caching (#6113)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* Minor improvements for data reading and caching

* Address reviews

* Adjust zip cache error check
2021-12-16 19:58:37 -03:00