Commit Graph

760 Commits

Author SHA1 Message Date
Martin-Molinero 5c078cd2a9 Remove 'data-directory' usage (#7241)
Regression Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
2023-05-12 18:24:12 -03:00
Ricardo Andrés Marino Rojas 4e56af487c Fix bug in RandomDataGenerator for Option data at Minute resolution (#7237)
Python Virtual Environments / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Fix the bug and add unit tests
- The bug was raised by different conditions.

First, the tick created in tick generator for Open Interest was not of type Open Interest.

Second, TickAggregator.cs was made to use daily resolution for OpenInterest always.

Third, the RandomValueGenerator, generated a random friday as expiration date for the option created but this one needed first two underlying data points, so if the expiration date was before the start date plus 3 days, RandomDataGenerator just generated OpenInterest data for the option.

* Add docs
2023-05-11 12:54:45 -03:00
Martin-Molinero 642fe96fde Update Binance Symbol Properties DB (#7040)
Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
- Update Binance SPDB
- Minot fix for ExchangeInfoUpdater
2023-03-03 19:41:53 -03:00
Martin-Molinero 38f5a9c60c Bump to pythonNet 2.0.18 (#7036)
Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
2023-03-03 13:11:03 -03:00
Martin-Molinero c1c4114dca Newtonsoft & Protobuf & RecyclableMemoryStream Updates (#7002)
* Dependency updates

- Update Newtonsoft to 13.0.2, protobuf to 3.2.0, RecyclableMemoryStream
  to 2.3.1

* Downgrade protobuf to 3.1.33
2023-02-23 12:11:01 -03:00
Jhonathan Abreu 5675e71a32 Handle prices getting stuck on data generation (#6995)
* Handle prices getting stuck on data generation

* Add more unit tests
2023-02-21 13:09:31 -03:00
Martin-Molinero 7178be4f55 Remove Yahoo auxiliary data generator (#6993)
- Remove yahoo auxiliary data generator and failing tests
2023-02-20 19:23:31 -03:00
Martin-Molinero 61fb997f07 Delete unsupported AlgoSeek Option Converter (#6992)
- Delete unsupported algoseek option converter
2023-02-20 19:21:15 -03:00
Jhonathan Abreu 38c5ef3e2b Add CME Ether futures to symbols database (#6884)
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
* Add Ether futures to symbols database

* Change ETH futures exchange time zone to US/Central

* Add new contracts FuturesExpiryFuntions

* Remove BTIC Micro BTC and BTIC Micro Ether securities

* Add Ether futures IB fees and price multiplier entry

* Update Ethereum price multiplier
2023-02-09 18:42:19 -03:00
Alexandre Catarino e8b95b80e1 Polygon Handles Data Subscriptions with One Security Type (#6855)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
* Handles Data Subscriptions with One Security Type

Polygon sells data subscriptions for security types separately, so we should not test whether the users have all three subscriptions, Equity, Forex, and Crypto. Users just need the type they will use.

Adds new crypto markets, and unit tests for them. Adapt unit test.

* Fixes ToolBox Downloader

Adds `--api-key` argument.
Closes #6687

* Address reviews

Co-authored-by: Martin-Molinero <martin@quantconnect.com>
2023-01-12 17:37:29 -03:00
Martin-Molinero ee683933a3 Feature Perpetual Crypto Futures (#6807)
Python Virtual Environments / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
* WIP

* Add base currency cash

* Symbol properties and data processing

* Add basic template algorithm

* Add hourly crypto future algorithm

* Minor fixes after live trading testing

* CoinApiDataQueueHandler CryptoFuture support

* Address reviews

* Fix regression algorithms after update
2022-12-30 13:58:11 -03:00
Alexandre Catarino bc513ce0e7 Changes Crumb Location from Yahoo Downloader (#6812)
Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Yahoo has changed the location from the crumb which is no longer located in "CrumbStore",
2022-12-20 10:33:52 -03:00
Mathieu Paquette 60028ad48a fix(ToolBox/IQFeed): increase default buffer size (#6762)
Python Virtual Environments / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
fixes #6761
2022-11-25 10:40:52 -03:00
Ryan Russell ac0cdb20f5 docs: formatted readability fix (#6627)
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-16 17:32:44 -03:00
Martin-Molinero 19f3d8b305 Fix python memory leak (#6579)
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
* Fix python memory leak

- Fix python memory leak. Adding unit test

* Update pythonnet to 2.0.17
2022-08-24 18:24:07 -03:00
Martin-Molinero b7df632294 Correctly handle Polygon authentication sequence (#6485)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
- Correctly handler Polygon data queue handler authentication sequence
2022-07-18 13:40:21 -03:00
Martin-Molinero 31fc7ac236 Do not generate Equity Quote Daily data (#6483)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
- Fix for so the random data generator does not try to create daily
  equity quote data
2022-07-14 10:20:34 -03:00
Martin-Molinero 18eec48a3d Avoid PythonNet runtime stash on shutdown (#6441)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Can't relesase the GIL after shutdown

* Bump version to pythonnet 2.0.16
2022-06-27 13:04:01 -03:00
Ronit Jain 60c9162b11 Extract Gdax brokerage files and dependencies (#6391)
* use research-object-store-name key from config while creating object store for research notebooks

* initial removal

* remove reference

* move test to private repo containing downloaders
2022-06-09 12:32:27 -03:00
Ronit Jain c11e24c5b2 Extract Oanda brokerage files and dependencies (#6383)
* use research-object-store-name key from config while creating object store for research notebooks

* remove oanda files
2022-06-07 14:07:24 -03:00
Ronit Jain c162e75bbc Extract bitfinex brokerage files and dependencies (#6382)
* use research-object-store-name key from config while creating object store for research notebooks

* remove bitfinex files

* remove redundant ref

* move test to private repo
2022-06-07 11:07:28 -03:00
Martin-Molinero 0fbbc0e612 Revert "Coarse fine flag improvement (#6349)" (#6353)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
This reverts commit fb90f118c9.
2022-05-25 16:51:33 -03:00
Martin-Molinero fb90f118c9 Coarse fine flag improvement (#6349)
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Coarse fine flag improvement

- Coarse will check global data folder for fine existance if it's
  different than the given path to check

* Update readme.md
2022-05-24 19:46:37 -03:00
Martin-Molinero 00dd3dbd2c Bump to pythonNet 2.0.15 (#6342)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
2022-05-18 13:09:45 -03:00
Martin-Molinero eb995b86ea Update to pythonNet 2.0.14 (#6322)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Update to pythonNet 2.0.14

* Add unit test reproducing issue
2022-05-06 17:29:55 -03:00
Martin-Molinero b7a21ce4be Update to net6 (#6311)
* Update to net6

* Bump pythonNet to 2.0.13
2022-05-03 11:45:55 -03:00
Martin-Molinero b9d3d99917 Bump pythonNet version 2.0.12 (#6310)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Updates after pythonNet rebase

* Bump pythonNet version 2.0.12

* Fix exception types being thrown
2022-05-02 14:38:50 -03:00
Ronit Jain 49562c712f Extract IB brokerage files and dependencies (#6283)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* make public safe to expose test properties

* extract ib files

* remove dependencies

* refactor to use ib reference as string

* remove un-used dependencies

* remove redundant refrences
2022-04-07 14:48:10 -03:00
kulaj 29e426c2b6 Add localization for CSV export delimiters in CoarseUniverseGeneratorProgram (#6228) (#6275)
* Add localization for CSV export delimiters (#6228)

* Use invariant string decimal for coarse

Co-authored-by: Martin-Molinero <martin@quantconnect.com>
2022-04-01 18:12:47 -03:00
Adalyat Nazirov 18b99338fc Implement Binance US exchange (#6222)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Binance.US base changes

* biniance exchange info update

* Binance US uses same fee rates as main Binance

* binance us brokerage model tests

* use base implementation of GetBuyingPowerModel method
2022-02-22 13:32:59 -03:00
Ronit Jain f129ab1a09 Feature implement ExchangeInfoDownloader (#6213)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* add gdax exchange info downloader

* add downloader method to program]

* fetch currency description

* change definition to include headers

* use extension method to make request

* remove log from test

* replace WebRequest

* cleanup

* use relevant name

* implement IExchangeInfoDownloader for bitfinex, initial commit

* add default values

* use default market value

* use correct attribute for lotsize

* don't skip missing values

* handle multiple downloaders

* add gdax and bitfinex exchange downloader

* follow LEAN data directory structure

* update SPDB

* order tickers

* order tickers

* add exchange info downloader test template

* delete files

* update SPDB

* use currency mapping

* update bitfinex symbols

* update currency mapping

* sort result after old currency symbols are used

* use market of the respective brokerage

* no more unknown symbol

* change minimum order size value

* direct conversion possible

* update bitfinex symbols

* change user-agent

* add test for indirect conversion

* update stats
2022-02-22 13:00:54 -03:00
Ronit Jain f86926bf7a Fixes HistoryProviderManager slice merging by using algortime for all slices (#6226)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* remove un-used

* use utc time for slice sync

* use utc time

* refactor

* add regression test

* use utc time

* use utc time

* make utctime required parameter

* add utcTime in slice creation

* assert warm up complete

* check if algorithm is still warmingup

* use exchange tz
2022-02-18 20:43:33 -03:00
Adalyat Nazirov e8160f33d5 Binance extraction (#6193)
* extract binance brokerage

* remove binance tools from main toolbox app

* modify OrderTestParameters constructor

* fix BinanceBrokerage ref config.json

* update symbol properties

* move BinanceFeeModel tests to the right place
2022-02-15 12:05:47 -03:00
Ronit Jain 2d644d7879 Feature handle multiple history providers (#6187)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* initial commit

* get history from data providers

* merge history from history providers

* merge slice

* Add merge function in slice

* Update test

* merge aux data

* add data points in _data

* append new data points in original list

* Add test suite for HistoryProviderManager

* reduce complexity

* setup once

* add fake history provider

* doesn't count aux data

* add tests for options

* style changes

* add custom data

* merge rawDataList

* use array of history providers

* optimize

* fix formatting error

* add comment

* add tests

* simplify

* use list

* use abstraction

* split tests

* add test

* use abstraction to create generic enumertor class

* refactor

* accept list of type T

* sync history slices

* use SubscriptionDataReaderHistoryProvider for live

* rename test file

* return empty

* add tests

* cleanup

* address reviews

* optimize

* Fix method definition

* use initial time for basedata

* refactor

* re-use collection

* inherit HistoryProviderBase

* always return HistoryProviderManager

* add tests

* update rawDataList

* reset composer

* consider null elements

* add tests for binary search method

* Follow lean coding style

* revert

* remove binary search method

* convert to field
2022-02-14 11:58:00 -03:00
Martin-Molinero 9a355c9be5 Adjust INR/USD micro future scale to usd dollars (#6203)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
2022-02-11 17:01:01 -03:00
Ricardo Andrés Marino Rojas eb55311052 Feature 5157 micro futures update (#6190)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Update MHDB and SPDB

Update MHDB and SPDB with micro futures

* Fix bugs

* Add more FutureExpiryFunctions

* Requested changes

* Nit changes

* Nit changes

- Fix some future values in SPDB
- Test LastFriday() method
- Apparently Micro CHF/USD Futures(MSF) refers to the micro of Swiss Franc Futures (6S). The same happens with Micro JPY/USD (MJY) Futures and Japanese Yen Futures (6J)

* Fix bugs

* Micro futures MultipleFactor

* Fix some futures MultipleFactor
2022-02-10 17:45:07 -03:00
Martin-Molinero 9fd50a302e Update or remove SharpZipLib dependency (#6180)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
2022-02-02 13:28:24 -03:00
Ronit Jain e5c709ee29 Extract zerodha brokerage out of LEAN (#6163)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Remove zerodha brokerage

* use different brokerage for tests

* remove zerodha files with conflict

* remove redundant dependecies
2022-01-24 18:43:12 -03:00
Ricardo Andrés Marino Rojas 0946bfc2fb Enable users to use symbol tickers when using Toolkit (#6158)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* If the market ticker has a ":" the user can use the symbol ticker

* Nit change
2022-01-19 15:42:21 -03:00
Adalyat Nazirov 2529ba124d FTX API endpoint is cinfigurable (#6026)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* specify enpoint url of ftx api (can be FTX pr FTX.US)

* more metadata for ftxus

* tidy up code

* tests

* more tests

* fix tests

* fix us fee rates

* add account tier

* update symbol props

* typo

* typo-2

* update symbol properties

* use FTXUS fee model

* minor tweaks
2022-01-12 13:51:18 -03:00
Ricardo Andrés Marino Rojas 472f78cc53 Remove Quandl from LEAN (#6110)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* Remove Quandl from LEAN

* Nit changes and CustomLiveDataFeedTests.cs

* Resolve conflicts

* Remove files related with Quandl

* Fix bug

* Fix QuantBookHistoryTests.cs

* Fix bug

* Fix bug

* Fix unit tests

* Try fix regression tests

* Nit changes

* Fix bug

* Some of the requested changes

* The missing changes

* Requested changes

* Nit changes

* Revert "Nit changes"

This reverts commit 9800bc5c34f3ac20e30bea7a92dd4a9867213bb5.

* Nit changes

* Fix bug

* Requested changes

* Missing file using Quandl to be removed

* Nit changes

* Not applied nit change

* Nit change

* Nit change

* Add nasdaq-auth-code parameter in config.json

* Remove 'quandl-auth-token' from config.json

Co-authored-by: Martin-Molinero <martin@quantconnect.com>
2022-01-12 12:10:20 -03:00
Adalyat Nazirov 0c26d42561 Feature 2839 black scholes data generator (#6135)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* replace to local functions as they are more performant

* fix random generator upper bound

Next() includes minValue, but not maxValue, so we increment it +1

* introduce abstract layers

* refactoring

* fix tets

* adapt tests

* fixup

* implement blackschole price model for options

* use risk free rate

* use ql price model

* wip

* change interface

* fix

* tidy up the code

* wip

* iterate groupped symbols

* wip

* wip

* fix

* allow symbol of different types

* improve settings

* wip

* iterate full range

* fix issue with negative option

* fix

* fixup

* use StandardDeviationOfReturnsVolatilityModel

* re-use existing tick types per security type

* parametrize underlying security type

* use default option style

* dynamic option price model

* fix enumeration

* test

* fix unit tests

* refactor code

* remove unused file

* minor tweaks and refactoring

* rename symbol generator class

* fix interface

* add comments

* more comments and unit tests

* more tests

* add disclaimer

* more tests

* more comments and tests

* split tests into different files

* tidy up the code

* tidy up the code; more tests

* refactor TickGenerator => use security price directly on each iteration

* remove dupe; reuse main constructor

* use SecurityManager, refactor code

* bugfix: save ticks in history array

* check volatility warm up & tests

* more unit tests

* describe volatility period span in settings

* rename command line option

* Minor adjusments. Address review

- Use Lean log handler instead of writting directly to console
- Rename BlackShcolesPriceGenerator to generically OptionPriceModelPriceGenerator
- Minor format clean up & standarization
- Add support for specifying the option chain size

* Rename TickGenerator private fields

* Fix unit tests

* fix tests class name

* Support tickers being specified

Co-authored-by: Martin-Molinero <martin@quantconnect.com>
2022-01-10 17:21:03 -03:00
Ronit Jain 0e298edcb2 use compression library (#6142)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
2022-01-05 15:19:41 -03:00
Martin-Molinero 10bb627fc2 Update to pythonNet 2.0.11 (#6131)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
2021-12-27 15:49:24 -03:00
Martin-Molinero 4fdd60d146 Use 64 when saving zips update dotnetZip (#6114)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
2021-12-20 16:39:06 -03:00
Martin-Molinero abbb50e209 Minor improvements for data reading and caching (#6113)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* Minor improvements for data reading and caching

* Address reviews

* Adjust zip cache error check
2021-12-16 19:58:37 -03:00
Ronit Jain 264c3c8374 Composite IDQH - Support multiple live data feeds (#6047)
* initial commit

* Follow IDQH implementation

* Expect a list of data handlers from LiveNodePacket

* Return null if can not subscribe

* refctor to add check for subscription

* initialze null

* Add tests

* Check subscribe retuns null/not-null

* cleanup

* Read all required IDQH credentails to job

* Use CDQH to handle all IDQH instances

* constructor abstraction to call from setjob

* use flag

* remove redundant because derived will call initialize on it

* abstract and initialize from setjob

* handle null enumerators

* get creds from data handlers

* handle single data handler value from data-queue-handler

* Fix to support a json array

* Fix missed constructor call

* change access modifier to access from Tests files

* Add test to get brokerageFactory from dataQueueHandler

* Fix init flag to handle all conditions

* Add docs

* initialize from setjob

* Check if websocket open before using

* change defination of initialzie to include tradier

* clean up

* change defination

* fix wrong api key name

* return empty enumerator

* check websocket open before sending request

* check connection before subscribing

* fix to include more cases

* check websocket open before sending request

* Minor refactoring

* reafctor and use IsConnected

* remove unused

* clean up

* Fix test cases

* reverse change

* include config changes

* connect to websocket from setjob

* check websocket connection from setjob

* clean up

* include condition for IDQH that are not brokerage

* Address review

* Add market check condition before subscribe

* Remove deprecated

* Minor fix for deserializing data queue handler

Co-authored-by: Martin-Molinero <martin@quantconnect.com>
2021-12-03 16:18:41 -03:00
bmello4688 1d8243ecac Add tick type to data downloader get (price data) call in order for a… (#6057)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Add tick type to data downloader get (price data) call in order for api's with rate limits to handle not making an api call at all.

* Added DTO object to pass into data downloader get

* Added missing header for new file

* Address review. Minor tweaks

Co-authored-by: Martin-Molinero <martin@quantconnect.com>
2021-11-22 12:00:16 -03:00
Martin-Molinero b8b0d18993 Continuous futures (#6034)
* Continuous Future Contracts

* Mapping approach

* Tweaks WIP

* Live mapping

* Live mapping

- Add support for live mapping, refreshing mapfiles
- Fix future expiration functions
- Adding unit tests

* Update moq test package

* Continuous futures price scaling

* Refactor price factors scaling

* Factor file related renames

* Address reviews
2021-11-15 14:44:30 -03:00
Martin-Molinero d17865b2cf CoinApi read previous date file (#6019)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
- If available, CoinApi converter will read previous date files to
  rescue initial midnight ticks for the processing date. Also make sure
  to drop ticks from tomorrow
2021-10-25 20:26:12 -03:00