7 Commits

Author SHA1 Message Date
JosueNina a11a66b09e Log warning when downloader returns no data for requested date range (#9344) 2026-04-07 13:07:31 -03:00
Roman Yavnikov 0212acf927 Feat: support downloading single contract in DownloaderProvider (#9285)
API Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Report Generator Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Syntax Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
* feat: improve symbol parsing for futures/options in downloader

* refactor: symbol parsing and unify ticker handling

* refactor: symbol parsing and loading in BaseDataDownloadConfig
2026-02-13 13:45:10 -03:00
Roman Yavnikov 87a65451ea Feature: Universe Data Downloader (#8612)
* feat: parsing of universe file path for different security types

* fix: market position in parsing of universe file
refactor: add example of universes file path in xml description method

* test:feat: parsing of universe path

* rename: return variable in ParseUniversePath

* feat: extension to create history request internally

* feat: entity  DataUniverseDownloaderGetParameters

* config: downloaderDataProvider project

* feat: create DataUniverseDownloadConfig in download project

* feat: run universe downloader in DownloaderProvider

* test:fix: missing using
test:remove: parameter ctor of DataDownloadConfig

* refactor: run Universe downloader in Main

* feat: GetUniverseFileName by processingDate

* refactor: CreateDataUniverseDownloaderGetParameters with using Start and EndDate
refactor: exception message in DataUniverseDownloadConfig

* feat: new out argument in TryParsePath's LeanData

* feat: RunUniverseDownloader in DownloadDataProvider

* feat: DerivativeUniverseData to write csv file
feat: extension to download universe data with different parameters

* remove: log warn which path when get universe file path

* refactor: DataUniverseDownloaderGetParameters
feat: run Download Universes file in DownloaderDataProvider internally

* refactor: overload TryParsePath only with 4 parameters

* remove: editconfig file in DownloaderDataProvider

* remove: validation on null parameters

* rename: typo DateType to DataType

* refactor: use endDate in ctor DataUniverseDownloaderGetParameters
refactor: use UnderlyingSymbol from base class Symbol property
remove: CreateDataUniverseDownloaderGetParameters()
refactor: put LoadSymbol in base class
clean/remove: DataUniverseDownloadConfig

* refactor: use Time.EachTradeableDay in CreateDataDownloaderGetParameters
remove: CheckMarketOpenStatus

* fix: RunUniverseDownloader with EndUtc in DownloaderDataProvider

* refactor: GetUniverseFullFilePath

* remove:  validate historyData like Any

* fix: use date from parsing path in DownloaderDataProvider

* feat: download QuoteBar for Universe file
feat: handle QuoteBar in DerivativeUniverseData
refactor: filter option in final dictionary

* revert: CreateHistoryRequest() in DownloaderExtensions

* feat: use optional parameter SecurityExchangeHours in DataUniverseDownloaderGetParameters
refactor: DataUniverseDownloaderGetParameters
refactor: use Log.Debug instead Log.Trace to prevent spamming

* fix: ordering of download universe data
revert: parameter ctor in DataDownloadConfig
revert:test: use parameter ctor instead of Config.Set

* clean: extra `usings`

* fix: Get DataType in DataDownloadConfig
feat: new ctor in BaseDataDownloadConfig
2025-03-12 19:38:18 -03:00
Roman Yavnikov 86fcc40c3e Feature: Generic brokerage downloader wrapper (#8235)
Python Virtual Environments / build (push) Has been cancelled
API Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Report Generator Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* feat: new constructor of AlgorithmNodePacket

* refactor: extract JobQueue configuration

* remove: not used `using` in IDataDownloader

* feat: create BrokerageDataDownloader

* Revert "refactor: extract JobQueue configuration"

This reverts commit 5778936b712befd6e367bee3decc4a3565ca69af.

* Revert "feat: new constructor of AlgorithmNodePacket"

This reverts commit d7a565ff761243ae1391bd45fd10c93221f0329e.

* feat: new config `data-download-brokerage` in DataDownloadProvider

* refactor: initialize in BrokerageDataDownloader

* remove: not used `using` in Program's DataDownloadProvider

* remove: not used ref on QuantConnect.Queue proj

* refactor: use default market based on SecurityType

* refactor: MarketName in DataDownloadConfig struct
test:feat: validate MarketName

* feat: support Canonical Symbols in BrokerageDataDownloader

* remove: not used command arguments

* feat: init CacheProvider of IOptionChainProvider in Downloader

* feat: add brokerage message event in BrokerageDataDownloader
2024-07-31 19:02:13 -03:00
Roman Yavnikov 9fad556903 Remove: log ETA calculation (#7994)
API Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Report Generator Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
* remove: ETA in data download provider

* remove: log of ETA calculation

* remove: test with ETA downloader
2024-05-03 14:25:10 -03:00
Roman Yavnikov ef23d7b6fe Add Group Symbol and Total Time Execution Logging for Downloading Canonical Data (#7984)
* refactor: modifier static to parse enum and ignore warning

* feat: group symbol when download cannonicial data
feat: log total time execution of download provider
test: right keeping of cannonicial data

* feat: parameter ctor in DataDownloadConfig

* test:feat: additional assertion of wrote data in file

* feat: static FilterGroupDownloadDataBySymbol

* refactor: extract downloader folder path

* try: map file false

* feat: get rid of dataCacheProvider in DownloaderDataProvider

* refactor: get rid mapFile flag

* CI test fix

---------

Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
2024-05-01 13:39:34 -03:00
Roman Yavnikov 74d95f1611 Implement Download Project in Lean (#7975)
* feat: init Download proj

* feat: update docker file to copy download proj

* feat: parsing in cmd arguments

* feat: DataDownloadConfig model
feat: add config file
refactor: logic of parsing config

* feat: overrid ToString in DataDownloaderGetParameters

* remove: not used argument command

* feat: calculate downloading % (without ETA time)

* fix: dockfile copy path for DownloaderDataProvider proj

* feat: additional Property in .csproj

* remove: DataProviderFullName prop from DataDownloadConfig

* remove: extra package (ref from Configuration proj)

* feat: ETA calculation of downloading data

* Minor tweak

* rename: namespace of downloaderDataProvider to generic style format

* test: ETA from DownloaderDataProvider

* fix: calculate ETA like real downloading process

---------

Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
2024-04-25 19:53:58 -03:00