* Retry transient empty/failed remote custom-data downloads
A remote custom-data source can intermittently answer with a transient failure or
a successful-but-empty body (e.g. an HTTP 200 with no content from a proxied
endpoint). Previously a single such response made RemoteFileSubscriptionStreamReader
treat it as the file contents: an empty byte[] passed the 'bytes != null' check and
was written/cached as a valid (empty) file, so the whole subscription silently
yielded no data and, with caching enabled, reused the empty file.
Download with a bounded retry that treats an empty response or an exception as
transient, and only persist/cache a non-empty result. On HTTP errors a WebException
is still thrown after the retries are exhausted, preserving existing behavior.
Adds tests: retry recovers after empty responses, and an empty download is not cached.
* Update RemoteFileSubscriptionStreamReader.cs
* task: remove extraneous defaults from Isolator/MonitorTask
- ExecuteWithTimeLimit -- the only user of private MonitorTask --
already defines the defaults for memoryCap & sleepInterval
* fix: typos
* Align OnWarmupFinished time to StartDate when ScheduledUniverse skips midnight
* Align algorithm time to StartDate before OnWarmupFinished fires
* Apply warmup time alignment fix to LiveSynchronizer
* Minor fix
* Skip warmup pulse if algorithm not locked
* Fix OnWarmupFinished timing in live trading
Regression Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Report Generator Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Syntax Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
* Fix DataManager skipping invalid data type subscriptions
* Update regression algorithms
* Use effective resolution when looking up subscription data types
* Initial backtest analyzer commit
* Changes for cleanup
* Minor fix
* Add class for analysis result context
* Fix more models
* More fixes
* More fixes
* More fixes
* More fixes
* Cleanup
* Cleanup
* Add documentation
* Cleanup and remove some classes
* Write backtest analysis to results file
* Cleanup
* Minor fixes
* Minor fix
* Minor cleanup
* Implement json deserialization for backtest analysis result
* Rename analyzer classes to generalize
Remove "backtest" references from names
* Add weight to analyses
* Run analyses in order by weight
* Cleanup
* Cleanup
* Update Issue text
* Set sub-test weights to 0
* Update test weights
* Add xml docs
* Disable result analysis on regression tests
* Refactor analysis naming
* Renaming cleanup
* Simplify analysis results context
* Log when analyzer reaches max tests or time limit
* Sample down equity curve to match daily benchmark
* Minor changes
* Minor fixes
---------
Co-authored-by: Derek Melchin <melchin.derek@gmail.com>
* Use universe settings when adding derivatives
Use resolution, fill forward and extended market hours settings from universe settings when adding derivative securities (options, index options, futures and future options)
* Minor python syntax check fix
* Minor change
* Default resolution from universe settings for common lean types securities
* Minor test fix
* Cleanup
* feat: add CanonicalDataDownloaderDecorator for chain support
Introduced CanonicalDataDownloaderDecorator to enable automatic resolution and parallel downloading of all contracts for canonical symbols (options and futures chains). Updated Program.cs and DownloaderDataProvider to use this decorator, ensuring seamless data retrieval for both canonical and non-canonical symbols. Refactored initialization logic to handle chain providers within the decorator and removed redundant code from Program.cs.
* feat: improve canonical symbol error handling in data downloader
* refactor: data downloader selection in DownloaderDataProvider
* refactor: CanonicalDataDownloaderDecorator construction
* feat: limit parallelism and improve contract data error handling
* refactor: canonical data downloader and provider usage
* remove: CanonicalNotSupportedException and refactor handling
* refactor: data downloader selection with selector class
Introduce DataDownloaderSelector to choose the correct IDataDownloader implementation based on data type, using CanonicalDataDownloaderDecorator only when needed. Update Program.cs and DownloaderDataProvider to use the selector, remove redundant initialization logic, and ensure proper resource disposal. This improves flexibility, correctness, and resource management for data downloads, especially for custom and canonical data types.
* fix: wrong selector condition
* refactor: change default log handler; add parallelism config to downloader
- Set ConsoleLogHandler as the default log handler in code and config
- Make contract download parallelism configurable via downloader-thread-count (default 4)
- Track and log number of processed contracts in CanonicalDataDownloaderDecorator
- Add error logging for missing universe data
- Improve logging clarity and code readability
* feat: prevent duplicate contract downloads in canonical chains
Introduce ContractDownloadParameters to uniquely identify contract/tick type/resolution combinations and cache them in CanonicalDataDownloaderDecorator, avoiding redundant downloads when contracts are shared across canonical symbol chains. Add date range optimization for contract downloads and unit tests for the new class.
* Revert "feat: prevent duplicate contract downloads in canonical chains"
This reverts commit 44386c4c788904531c473c60c2a173e2354561ec.
* feat: clamp contract date ranges, deduplicate downloads
Added AdjustDateRangeForContract to CanonicalDataDownloaderDecorator to clamp start/end dates based on contract expiry and security type. Introduced _contractsCache to prevent duplicate downloads across canonical chains and _processedContracts for logging. Updated contract retrieval logic for deduplication. Added NUnit tests to verify date range adjustments for futures and options.
* refactor: canonical contract date range adjustment logic
* test:feat: add test for non-option/future contract date adjustment
* feat: make look-back periods for canonical symbols configurable
* refactor: remove Lazy from CanonicalDataDownloaderDecorator
Revert "refactor: remove Lazy from CanonicalDataDownloaderDecorator"
This reverts commit 909c1299406688a19352273f8d0b5b3ea99c0fb9.
refactor: Inject IMapFileProvider into downloader selectors
Revert "refactor: Inject IMapFileProvider into downloader selectors"
This reverts commit 0add95269809680e425bfa6080e465bf16377e56.
Reapply "refactor: remove Lazy from CanonicalDataDownloaderDecorator"
This reverts commit 9e3c4bed078868a3c8dfedcab0e52d6dced1daad.
* refactor: data downloader dependency injection
* refactor: provider initialization in data downloader classes
* refactor: DataDownloaderSelector dependencies, add tests
- Require explicit IDataProvider in DataDownloaderSelector constructor, removing default and fallback logic
- Change default log handler to ConsoleLogHandler in Program.cs
- Pass DownloaderDataProvider as IDataProvider to DataDownloaderSelector
- Add DataDownloaderSelector unit tests to verify correct downloader selection and decorator usage
* feat: ensure factorFileProvider is initialized when defaulted
* fix: missed dataProvider in DataDownloaderSelector ctor
* refactor: improve IFactorFileProvider resolution in DataDownloaderSelector
Build & Test Lean / build (push) Has been cancelled
Syntax Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Report Generator Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Implement indicator-based option price model
This model uses IV and Greeks indicators to implement Lean's own option pricing model
* Minor fixes
* Address peer review
* Minor tests fixes
* Make the indicator based price model the default for options
* Address peer review
* Cleanup and minor changes
* Support indicators configuration for new pricing model
* Some cleanup
* Add QL option price model example algorithm
* Return lean models from static helpers
* Minor tests fixes
* Minor test fixes
* Address peer review
* Cleanup
* Fix unit tests
* Move QL models to OptionPriceModels.QuantLib.*
* Add forward tree helper method
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Report Generator Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Syntax Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
* Enhance order request processing by grouping requests and adding concurrency tests
* Minor adjustment
---------
Co-authored-by: Adrian Tkacz <adrian.tkacz@creatdy.com>
Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
API Tests / build (push) Has been cancelled
Report Generator Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Syntax Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
* Add a temporary benchmark stats sample in live
Allow trades to be written to result file on first day of the deployment before the first daily sample is done
* Enable trades results streaming
* Minor changes
* Make Trade.Id a Guid
* Cleanup
* Sample temporary charts once per hour
* Minor change
* Minor fix
* Minor fix
* Minor fix
* Add fix for potential race condition
* feat: add MappedSynchronizingHistoryProvider base class
Introduces an abstract class for history providers that handle symbol mapping and time-aligned data slices. Uses IMapFileProvider to resolve ticker changes, provides an abstract method for mapped history retrieval, and overrides GetHistory to synchronize results. Enables nullable reference types and adds documentation.
* Minor tweaks
---------
Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
Report Generator Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
API Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Syntax Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* Add total performance statistics to live result files
* Truncate closed trades in live results
* Avoid adding totalPerformance to live minute result file
* Deprecated Trade.Symbol in favor of new Trade.Symbols
* Fixes for Trade serialization
* Add trades json serialization tests
* Cleanup
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
API Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Report Generator Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Syntax Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
* Add SeedInitialPrices algorithm setting
This is true by default and indicates that the engine will seed initial prices right after the security is added or selected
* Update regression algorithms
* Update regression algorithms
* Update regression algorithms
* Refactor default securities seeding
* Minor fix
* Minro fixes
* Cleanup
* Updated and add regression algorithms
* Address peer review
* Centralize logic to get last known data for multiple securities
* Some cleanup
* Minor build fix
* Minor fixes
* More logic centralization
* Some more cleanup
* Cleanup
* Update regression algorithms and minor fixes
* Update regression algorithms
* Minor fix
* More minor fixes
* Update regression algorithms
* Cleanup
* Minor test fix
* Address peer review
* Minor fix and performance improvement
* Fix to seed open interest data
* Minor test fixes
* Address peer review
* Minor change
* Minor revert
* Minor fixes and improvements
* Disable initial seeding by default
* Minor fixes
* Cleanup
* Cleanup
* Minor fix
* Refactor user define universe handling
- Normalize user define universe additions and removals to behave like
other subscriptions without requiting special handling
* Minor fixes
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Report Generator Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Syntax Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
* Fix performance degradation on expirying assets
- Options would expire and get removed, when the universe removed the
security it would never be reseted on universe selection, causing
leaks
- Minor improvement for composer assembly loading
- Minor improvement for loading exception interepreters, so it uses
composer
* Minor regression test fix
* Add new benchmark algorithm
Regression Tests / build (push) Has been cancelled
API Tests / build (push) Has been cancelled
Report Generator Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Syntax Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
* Use Futures Bank Holidays For Expirations
* Minor improvements
- Futures will be stored by their contract month, not expiry
* Delete dairy future products
* Minor test fixes
JosueNina