792 Commits

Author SHA1 Message Date
Martin-Molinero 62d63010ab Convert daily resolution market orders to MarketOnClose/MarketOnOpen (#9534)
Python Virtual Environments / build (push) Has been cancelled
API Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Report Generator Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Syntax Tests / build (push) Has been cancelled
* Convert daily resolution market orders to MarketOnClose/MarketOnOpen

A market order placed intraday (e.g. through a scheduled event) on an
asset subscribed only at daily resolution has no fresh intraday price to
fill against, so it was filling at the stale previous daily close. This
is common when mixing daily resolution assets with minute resolution
assets or intraday scheduled events.

QCAlgorithm.MarketOrder now converts these orders so they fill at a real
daily open/close instead of the stale previous close:
 - market closed (any resolution): MarketOnOpen, as before
 - market open, daily-only subscription: MarketOnClose (today's close),
   or MarketOnOpen (next open) when already within the MarketOnClose
   submission buffer near the close

Assets with intraday data are left untouched, and the conversion is only
applied in backtesting; in live trading an open-market market order fills
at the real current price.

Adds DailyResolutionMarketOrderConversionRegressionAlgorithm covering the
MarketOnClose and MarketOnOpen conversion paths plus a minute resolution
asset that is correctly left as a regular market order.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Minor fix

* Reword conversion warning: "current market price" instead of "real current price"

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Update regression stats affected by daily market order conversion

Daily-resolution market orders placed intraday are now converted to
MarketOnClose/MarketOnOpen so they fill at a real daily open/close
instead of the stale previous close. This shifts the affected fills:

 - IndexOptionCall{ITM,OTM}ExpiryDaily: the SPX option entry, placed one
   minute after the open, now fills at the daily close. Same economics,
   one extra data point and a new order list hash.
 - AllShortableSymbols (C# + Python): an intraday order's type changed
   from Market to a converted order; identical End Equity, new hash.
 - ResolutionSwitchingAlgorithm sampling test: the RemoveSecurity
   liquidation (fired at 15:50) previously filled at the stale previous
   close; it now converts, shifting the equity/performance samples.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Revert expected data point count change

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 13:08:57 -03:00
JosueNina 3f5eefd879 Fix IResultHandler.RuntimeError not stopping the algorithm (#9444)
Python Virtual Environments / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
API Tests / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Report Generator Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Syntax Tests / build (push) Has been cancelled
* Fix IResultHandler.RuntimeError not stopping the algorithm

* Minor fix
2026-05-04 10:46:43 -03:00
Martin-Molinero 387b56bdd6 Minor fix for schedule universe to always be 8 am ny (#9427)
API Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Report Generator Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Syntax Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
- Avoid schedule universe running 7 or 8 NY depending on daylight
  savings, now will always run 8 am NY. Adding unit tests
2026-04-21 11:26:44 -03:00
Roman Yavnikov d1337eae2c Feat: CanonicalDataDownloaderDecorator for chain support (#9299)
* feat: add CanonicalDataDownloaderDecorator for chain support

Introduced CanonicalDataDownloaderDecorator to enable automatic resolution and parallel downloading of all contracts for canonical symbols (options and futures chains). Updated Program.cs and DownloaderDataProvider to use this decorator, ensuring seamless data retrieval for both canonical and non-canonical symbols. Refactored initialization logic to handle chain providers within the decorator and removed redundant code from Program.cs.

* feat: improve canonical symbol error handling in data downloader

* refactor: data downloader selection in DownloaderDataProvider

* refactor: CanonicalDataDownloaderDecorator construction

* feat: limit parallelism and improve contract data error handling

* refactor: canonical data downloader and provider usage

* remove: CanonicalNotSupportedException and refactor handling

* refactor: data downloader selection with selector class

Introduce DataDownloaderSelector to choose the correct IDataDownloader implementation based on data type, using CanonicalDataDownloaderDecorator only when needed. Update Program.cs and DownloaderDataProvider to use the selector, remove redundant initialization logic, and ensure proper resource disposal. This improves flexibility, correctness, and resource management for data downloads, especially for custom and canonical data types.

* fix: wrong selector condition

* refactor: change default log handler; add parallelism config to downloader

- Set ConsoleLogHandler as the default log handler in code and config
- Make contract download parallelism configurable via downloader-thread-count (default 4)
- Track and log number of processed contracts in CanonicalDataDownloaderDecorator
- Add error logging for missing universe data
- Improve logging clarity and code readability

* feat: prevent duplicate contract downloads in canonical chains

Introduce ContractDownloadParameters to uniquely identify contract/tick type/resolution combinations and cache them in CanonicalDataDownloaderDecorator, avoiding redundant downloads when contracts are shared across canonical symbol chains. Add date range optimization for contract downloads and unit tests for the new class.

* Revert "feat: prevent duplicate contract downloads in canonical chains"

This reverts commit 44386c4c788904531c473c60c2a173e2354561ec.

* feat: clamp contract date ranges, deduplicate downloads

Added AdjustDateRangeForContract to CanonicalDataDownloaderDecorator to clamp start/end dates based on contract expiry and security type. Introduced _contractsCache to prevent duplicate downloads across canonical chains and _processedContracts for logging. Updated contract retrieval logic for deduplication. Added NUnit tests to verify date range adjustments for futures and options.

* refactor: canonical contract date range adjustment logic

* test:feat: add test for non-option/future contract date adjustment

* feat: make look-back periods for canonical symbols configurable

* refactor: remove Lazy from CanonicalDataDownloaderDecorator

Revert "refactor: remove Lazy from CanonicalDataDownloaderDecorator"

This reverts commit 909c1299406688a19352273f8d0b5b3ea99c0fb9.

refactor: Inject IMapFileProvider into downloader selectors

Revert "refactor: Inject IMapFileProvider into downloader selectors"

This reverts commit 0add95269809680e425bfa6080e465bf16377e56.

Reapply "refactor: remove Lazy from CanonicalDataDownloaderDecorator"

This reverts commit 9e3c4bed078868a3c8dfedcab0e52d6dced1daad.

* refactor: data downloader dependency injection

* refactor: provider initialization in data downloader classes

* refactor: DataDownloaderSelector dependencies, add tests

- Require explicit IDataProvider in DataDownloaderSelector constructor, removing default and fallback logic
- Change default log handler to ConsoleLogHandler in Program.cs
- Pass DownloaderDataProvider as IDataProvider to DataDownloaderSelector
- Add DataDownloaderSelector unit tests to verify correct downloader selection and decorator usage

* feat: ensure factorFileProvider is initialized when defaulted

* fix: missed dataProvider in DataDownloaderSelector ctor

* refactor: improve IFactorFileProvider resolution in DataDownloaderSelector
2026-02-23 15:58:15 +02:00
Adrian Tkacz c55a064fe0 Fix combo order queue affinity to prevent duplicate brokerage submissions (#9293)
API Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Report Generator Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Syntax Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
* Enhance order request processing by grouping requests and adding concurrency tests

* Minor adjustment

---------

Co-authored-by: Adrian Tkacz <adrian.tkacz@creatdy.com>
Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
2026-02-16 11:44:39 -03:00
Martin-Molinero 4b6643312e Allow emtpy BNFCR cash through (#9247)
Report Generator Tests / build (push) Has been cancelled
API Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
Syntax Tests / build (push) Has been cancelled
2026-01-28 10:53:19 -03:00
JosueNina 609deb2b03 Logging only once price rounding warnings (#9238) 2026-01-26 12:54:33 -03:00
Jhonathan Abreu 26584c2fd0 Stop and delete unused threads in tests (#9219)
API Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Report Generator Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Syntax Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
2026-01-13 13:13:37 -03:00
JosueNina 0854ab82da Default option filter now includes weeklies to prevent empty chains (#9162)
* Return weekly contracts if no standard contracts exist

* Fix unit and regression tests

* Centralize default expiration type flags

* Add ExcludeWeeklys() method

* Mark IncludeWeeklys() as obsolete since weeklies are now default
2025-12-31 10:11:46 -03:00
Martin-Molinero 2f23c89307 Avoid FF enumerator error warning (#9122) 2025-12-10 18:18:08 -03:00
Jhonathan Abreu 683bfe007a Avoid emitting overlapping fill forwarded data (#9107)
Syntax Tests / build (push) Has been cancelled
API Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
Report Generator Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Avoid emitting overlapping fill forward data

* Cleanup
2025-12-05 10:05:40 -04:00
Jhonathan Abreu 16fdf61903 Track securities addition and removal notifications (#9097)
* Track security addition and removal sent notifications

* Minor regression algorithm fixes

* Fix regression algorithms data points count

* Add description to new regression algorithms

* Simplify tracking removal notifications

* Cleanup
2025-11-28 10:37:31 -04:00
Jhonathan Abreu c81f5d7d1a Seed securities by default (#9045)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
API Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Report Generator Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Syntax Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
* Add SeedInitialPrices algorithm setting

This is true by default and indicates that the engine will seed initial prices right after the security is added or selected

* Update regression algorithms

* Update regression algorithms

* Update regression algorithms

* Refactor default securities seeding

* Minor fix

* Minro fixes

* Cleanup

* Updated and add regression algorithms

* Address peer review

* Centralize logic to get last known data for multiple securities

* Some cleanup

* Minor build fix

* Minor fixes

* More logic centralization

* Some more cleanup

* Cleanup

* Update regression algorithms and minor fixes

* Update regression algorithms

* Minor fix

* More minor fixes

* Update regression algorithms

* Cleanup

* Minor test fix

* Address peer review

* Minor fix and performance improvement

* Fix to seed open interest data

* Minor test fixes

* Address peer review

* Minor change

* Minor revert

* Minor fixes and improvements

* Disable initial seeding by default

* Minor fixes

* Cleanup

* Cleanup

* Minor fix
2025-11-18 13:05:56 -04:00
Martin-Molinero 4961844f82 Refactor user define universe handling (#9088)
* Refactor user define universe handling

- Normalize user define universe additions and removals to behave like
  other subscriptions without requiting special handling

* Minor fixes
2025-11-18 12:29:05 -03:00
Martin-Molinero afca923cd9 Fix history provider equity skip (#9090) 2025-11-18 12:28:17 -03:00
JosueNina 7a5ff8054b Prevent adding zero quantity currencies to CashBook (#9076)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Syntax Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
API Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Report Generator Tests / build (push) Has been cancelled
* Prevent adding zero quantity currencies to CashBook

* Solve review comments

* Solve new review comments

* Add unit test
2025-11-10 09:33:38 -03:00
Martin-Molinero 6c5b577b46 Update HistoryProviderManagerTests.cs
Remove no longer valid unit test
2025-11-03 09:36:59 -03:00
JosueNina 9c7d633b41 Throw exceptions when ObjectStore limits are exceeded in Research mode (#9060)
API Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Report Generator Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Syntax Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
* Throw exception in Research mode when storage limits are exceeded

* Solve review comments
2025-10-27 10:16:25 -05:00
Martin-Molinero 6811b93d6a Add performance tracking tool (#9063)
- Add optional performance tracking tool
2025-10-27 11:31:14 -03:00
Martin-Molinero 6916636e8e Remove Future expiry date from data & use Bank Holidays For Expirations (#8580)
Regression Tests / build (push) Has been cancelled
API Tests / build (push) Has been cancelled
Report Generator Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Syntax Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
* Use Futures Bank Holidays For Expirations

* Minor improvements

- Futures will be stored by their contract month, not expiry

* Delete dairy future products

* Minor test fixes
2025-10-24 17:18:20 -03:00
Martin-Molinero 9ab9fe5f2b Python performance improvement (#9052)
Research Regression Tests / build (push) Has been cancelled
Syntax Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
API Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Report Generator Tests / build (push) Has been cancelled
* Minor tweak to cache liquidation fees

* Result handler cleanup

* Update pythonnet version to 2.0.49
2025-10-21 12:34:52 -03:00
Ryan Crowley b27e9e67dd ProcessSplitSymbols crashes during live/paper trading warmup with InvalidOperationException #9029 (#9030)
API Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Report Generator Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Syntax Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
* Fix ProcessSplitSymbols crash during live mode warmup

Fixes InvalidOperationException when CancelOpenOrders is called during warmup in live/paper trading mode.

When historical splits are replayed during live mode warmup, ProcessSplitSymbols attempts to cancel open orders for option contracts, but CancelOpenOrders throws an exception during warmup.

The fix adds an early return when algorithm.LiveMode && algorithm.IsWarmingUp. This is safe because in live/paper trading, current positions already reflect historical splits.

Includes unit tests validating the fix.

* Some improvements and more tests

- Ignore splits during warmup for backtesting and live trading. Adding a
  regression algorithm reproducing issue

* Minor fix

---------

Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
2025-10-13 13:21:09 -03:00
Martin-Molinero 8ee7a190c6 Minor fix for negative rolling window index (#9026)
* Minor fix for negative rolling window index

- Minor fix for negative rolling window index when partially full,
  adding unit test

* Reduce RefreshesSymbolProperties test runtime
2025-10-10 17:47:58 -03:00
Jhonathan Abreu ee7be1a233 Enable concurrency for live paper brokerage (#8972)
API Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Report Generator Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Syntax Tests / build (push) Has been cancelled
* Enable concurrency for paper brokerage.

Implement PaperBrokerageTransactionHandler to handle concurrency for live paper trading transaction handler

* Remove PaperTradingBrokerageTransaction handler

Favor existing BacktestingTransactionHandler

* Minor fixes and cleanup

* Minor unit tests fix
2025-09-15 09:25:49 -04:00
Jhonathan Abreu 5644520545 Refactor some unit and regression tests for speed improvements (#8970)
API Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Report Generator Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Syntax Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
* Reduce history unit tests duration

* Unit and regression tests speed improvements

* Minor change

* Fix unit test race condition
2025-09-12 10:15:09 -04:00
Jhonathan Abreu d82d70dacf Support asynchronous non-market orders (#8946)
API Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Report Generator Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Syntax Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
* Add 'asynchronous' parameter to trading api

* Add regression algorithms

* More fixes for asynchronous orders

* Fix failing unit tests

* Fix failing unit tests

* Add multiple orders requests in transaction handler

Make each transaction thread handle requests corresponding to the same order

* Refactor order state machine in BrokerageTransactionHandler

Now using a single dictionary to hold orders and their state, either pending for submission, open or closed.

* Revert: remove complete orders from new state machine in BTH

* Add order ticket to new BTH open orders state machine

* Run syn in Backtesting transaction handler for new orders submission

* Remove OpenOrderState.SubmissionPending property

* Add Security to OpenOrderState

* Minor fix

* Cleanup

* Some cleanup in BTH

* Cleanup

* Minor fix

* Minor unit test fix

* Minor fix

* Make Brokerage.ConcurrencyEnable property virtual

* Minor change

* Cleanup

* Add 'asynchronous' parameter to SetHoldings api

* Improve regression algorithms
2025-09-10 08:29:50 -04:00
JosueNina a7265c8a41 Place orders only when the algorithm is running (#8917)
* Place orders only when the algorithm is running

* Solve review comments

* Split unit test into separate cases

* Move the algorithm assignment
2025-08-11 11:48:17 -03:00
Jhonathan Abreu 546afd2a61 Async orders in execution models (#8872)
Syntax Tests / build (push) Has been cancelled
API Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Report Generator Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
* Changed default async to true and passed target.Tag

* Make execution models place orders asynchronously if specified

* Add unit tests

* Execution model default to asynchronous orders.

Also, minor fixes for tickets remaining fill quantity potential race conditions

* Add SecurityHolding.UnrealizedQuantity property

It gets the holding quantity the security will have once all open orders are filled.
Added for thread safety reasons when execution models place asynchronous orders and need to calculate the actual quantity needed to reach the target of there are open orders

* Some cleanup

* Adjust projected holdings quantity on splits

* Minor fix

* More changes and cleanup

* Minor fix

* Improvements for thread safety

* Add IOrderProvider.GetProjectedHoldings to get projected holdings atomically

* Minor unit tests fix

* Add ProjectedHoldings DTO class

* Address peer review

---------

Co-authored-by: arthiondaena <arthiondaena@gmail.com>
2025-07-21 13:11:03 -04:00
Jhonathan Abreu 2e9cc39c89 Rename Controls.StoragePermission -> Controls.StorageAccess (#8862)
* Refactor Controls.StoragePermissions format for the API

* Minor changes

* Rename Controls.StoragePermission -> Controls.StorageAccess

* Minor fix
2025-07-09 11:11:27 -04:00
Jhonathan Abreu 09f0239101 Support concurrent transaction handlers (#8857)
API Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Report Generator Tests / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Syntax Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
* Allow multiple concurrent transaction handlers

Also support multiple producers in BrokerageConcurrentMessageHandler

* Minor change

* Minor change

* Add new maximum-transaction-threads config to json file

* Minor naming changes

* Address peer review and add unit tests

* Minor fix

* Minor fix and unit tests

* Address peer review
2025-07-07 18:38:09 -04:00
Jhonathan Abreu 4602a9d222 Fill-forward last warmup data point to normal data feed (#8831)
* Fill-forward last warmup data point to normal data feed

* Minor clean up

* Minor changes and unit tests

* Address peer review

* Minor changes

* Improve unit tests
2025-06-17 12:07:58 -04:00
Martin-Molinero 890fcd114e Fix Symbol EOD internal SecurityChanges handling (#8824)
- Fix Symbol EOD internal SecurityChanges handling. Adding unit test and
  regression test
2025-06-12 19:18:15 -03:00
JosueNina dbdf014a4c Add BNFCR as stable coin without pair (#8821)
* Add BNFCR as stable coin without pair

* Add unit test

* Skip conversion in PerformCashSync when ConversionRate is zero

* Solve review comments
2025-06-12 19:17:32 -03:00
Martin-Molinero 592118b323 Fix live warmup price scaling (#8822)
- Fix missing live warmup price scaling. Adding unit test
2025-06-10 13:30:43 -03:00
Jhonathan Abreu 9e349259b2 Conditional timeout for live fill forward (#8817)
API Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Report Generator Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Syntax Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
* Wait for data before emitting fill forwarded bars only on market open/close in live trading

* Minor fixes and more unit tests
2025-06-10 10:58:56 -04:00
JosueNina 4c7bda8e34 Fix futures history request error (#8678)
* Initial solution

* Create testCases for earlyClose and lateOpen

* Refactor GetExchangeHours and add unit tests

* Update market-hours-database and create a unit test

* Resolve review comments

* Update market-hours-database

* Parallelize the foreach loop to improve test performance
2025-04-11 13:37:14 -03:00
Jhonathan Abreu 1974884c13 Security re-addition tidy up (#8656)
* Initialize securities on re-addition

* Minor unit tests fixes

* Reset Index.ManualSetIsTradable when security is removed

* More changes to clean up and simplify

* Address security re-utilization only

* Cleanup regression algorithms
2025-04-04 16:28:15 -04:00
Ricardo Andrés Marino Rojas 980cdeaa75 Remove fill warnings from order tags (#8668)
* Remove warnings from tags and tests

* Fix bugs

* fix bugs

* Revert wrong updates in some stats

* Fix bugs
2025-04-03 17:49:21 -03:00
Jhonathan Abreu 69d2f5ae82 Futures and Future Options file-based universes (#8480)
Report Generator Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
API Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* Make FOPs selection universe file-based for backtesting

* Make FOPs option chains universe file based

* Make Future universe selection file-based like option universe

* Make Future universe selection file-based like option universe

* Abstraction cleanup

* Add FuturesChains API to QC algorithm

Also refactor future chain provider to use the new FutureUniverse instead of zip file names

* Update regression algorithms stats

* Refactor QuantBook option and future history to use new universes

* Fix failing tests

* Fix failing tests

* Fix failing tests

* Minor future chains unit test improvement

* Add futures chains DataFrame property

Also, remove IDerivativeSecurity interface from Future

* Add DataFrame property to FuturesChains class

* Add regression algorithms

* Add regression algorithms

* Replace QCAlgorithm.FutureChainProvider usages with new FuturesChain api

* Minor fixes

* Reduce number of universe files in repo

* Minor data fixes

* Regression algorithms updates

* Add implicit conversion from FuturesContract to Symbol

Modified algorithms to use futures contract objects directly instead of accessing their Symbol property.
Removed unnecessary import statements and redundant lines in various files.

* Improve resolution handling for history requests

* Changed _auxiliaryData field to lazily-initialized AuxiliaryData property

* Refactor data handling in BaseChain and TimeSliceFactory

- Added `AddData` method to `BaseChain` for adding market data
- Refactored `TimeSliceFactory` to use `BaseChain.AddData` method

* Remove specific constructors and indexers from Chain classes

Removed public indexers in `BaseChains` for getting or setting `BaseChain` instances by `ticker` or `Symbol`, which were used for Pythonnet compatibility.

* Remove chain cache logic from FuturesChainUniverse

* Refactor class and interface names for clarity

Renamed `FileBasedUniverse` to `BaseChainUniverseData` and
`IFileBasedUniverse` to `IChainUniverseData`.

* Add base class for options and futures contracts

- Introduced `BaseContract` as an abstract base class for contracts, consolidating common properties and methods.
- Removed ISymbolInterface

* Add minor fix for future options tickers parsing

Added tests

* Trigger Build

* Trigger Build

* Trigger Build

* Trigger Build

* Clean chain provider classes up

* Remove ZipEntryName other classes and unused code

Removed ZipEntryName class and references across various files.
Removed DataQueueFuturesChainUniverseDataCollectionEnumerator and DataQueueOptionChainUniverseDataCollectionEnumerator classes.
Removed OptionChainUniverseSubscriptionEnumeratorFactory class.
Removed unused code for handling OptionChainUniverse and FuturesChainUniverse in FileSystemDataFeed.cs and LiveTradingDataFeed.cs.
Removed several test files related to enumerator factories and universe data collection.

* Minor changes and cleanup

* Trigger Build

* Trigger Build

* Refactor FuturesContract data handling

Forward price data from bars and ticks stored in private fields for improved memory usage

* Fix: use universe data for market data in FuturesContract

* Update regression algorithms stats after rebase

Added HSI futures universe files

* Sort configs by internal flag

Internals go first

* Throw from option universe data filters for future options

Future options IV, Open interest and greeks are not supported for future options

* Minor changes

* Improve some regression algorithms

* Minor fix for failing unit tests

* Update FOPs universe file header

Removed greeks and IV columns.
Updated FOPs universe files: removed outdated columns.

* Minor unit test fix

* Trigger Build

* Trigger Build

* Trigger Build

* Trigger Build

* Trigger Build

* Trigger Build

* Trigger Build

* Trigger Build

* Trigger Build

* Trigger Build

* Trigger Build

* Trigger Build

* Trigger Build

* Minor fix

* Add history provider as constructor argument for chain providers

* Update new regression algorithms data points count

* Minor fix for FakeDataQueue

* Add initialize method to chain providers classes

* Minor changes

* Trigger Build

* Trigger Build

* Trigger Build

* Minor fix

* Minor fix

* Trigger Build

* Trigger Build

* Trigger Build

* Trigger Build

* Add logs to ProcessedDataProvider

* Removed test logs

* Minor fix

* Support downloading options and futures universe files from api data provider
2025-03-25 16:22:38 -04:00
Jhonathan Abreu 8149c291fc Dynamic market hours for time and date rules (#8541)
API Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
Report Generator Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Add unit test reproducing the bug

* Improve unit test

* Apply market hours database updates to instance

Make sure entries are updated instead of overridden so that consumers holding the mhdb can pick up the changes

* Improved unit tests

* Update data folder MHDB entries instead of overriding instance

* Optionally reset custom mhdb entries on reset

* Simplify mhdb reset in live trading

Cleanup and minor fixes

* Ensure symbol properties database updates are picked up by consumers

Update SPDB entries instead of overriding them, just like for the MHDB so that consumers pick up updates after refresh

* Fix for symbol properties update thread safety

* Minor improvements and cleanup

* Mhdb and spdb update logic simplification

* Move "force exchange always open" logic to mhdb

* Simplify symbol properties holder and updates

* Refactor security databases to use a common base class

Introduce BaseSecurityDatabase<T, TEntry> to encapsulate common functionality for MarketHoursDatabase and SymbolPropertiesDatabase.

* Cleanup
2025-01-31 15:00:14 -04:00
JosueNina 5accfd6be3 Update secondsTimeStep value (#8528) 2025-01-14 15:17:55 -03:00
Ricardo Andrés Marino Rojas 50f887ca64 Fix delisting time handling in SubscriptionDataReader (#8470)
The delisting time is handled by the DateChangeTimeKeeper now, so there is no need for the SubscriptionDataReader to do any special logic for it.
This was preventing the new tradable date events to be emitted after the delisting date in some cases, like when the day after delisting is not tradable

Co-authored-by: Jhonathan Abreu <jdabreu25@gmail.com>
2025-01-13 10:22:14 -04:00
Jhonathan Abreu 7ee297ab77 Add BTIC on cryptocurrencies futures back (#8503)
* Add BTIC on cryptocurrencies futures back

Add MIB (BTIC on micro bitcoin futures) and BRM (BTIC on micro ether futures) back

* Minor unit tests fix
2025-01-06 15:01:30 -04:00
Jhonathan Abreu 57ce6958dd Skip subscription to selected securities when no tradable dates (#8495)
* Skip "no tradable dates" log for options added on non-tradable dates

* Skip subscription to selected securities when no tradable dates

Skip selected securities subscriptions when there are no tradable dates available from selection time till end date

* Minor comment

* Remove "no tradable days" error log
2025-01-03 20:47:52 -03:00
Jhonathan Abreu a1d0b6cde2 Live trading consolidation synchronization (#8436)
API Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Report Generator Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
* Adjust time slice time for live trading

Adjust time slice to be driven by data, so that consolidators are update at the correct times.
Live trading uses DateTime.UtcNow which might be a few milliseconds after the latest data, causing some race conditions in consolidators scan times.

* Refactor: rounding time slice time down for past consolidators scan (live)

* Cleanup
2024-12-04 11:15:00 -04:00
Jhonathan Abreu d416580456 Adjust lower-resolution fill-forwarded strict end-timed daily bars (#8412)
Regression Tests / build (push) Has been cancelled
Report Generator Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
API Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Adjust lower-resolution fill-forwarded daily bars when strict end times is enabled

This allows to get fill-forwarded bars with unchanged time stamps

* Minor fixes

* Minor test data changes

* Fixes and comments

* Address peer review and add some fixes

* Minor fix and add regression algorithm

* Minor fix
2024-11-29 09:52:01 -04:00
Jhonathan Abreu f2100adb46 Fix price shifts around futures rollover date (#8371)
* Test

* Fix price shift on continuous future rollover

* Keep track of both data and exchange time in subscription data reader

* Fix for daily and hourly resolutions.

Add daily and hour algorithms

* Fixes for emitting new exchange dates at the beginning and end when data is not available

* Cleanup

* Minor fix

* Minor fix: detect mapping in data reader after advancing time keeper

* Minor fix

* More fixes

* More fixes

* Fixes for universe files

* More fixes for daily resolution data

* Separate DateChangeTimeKeeper to its own file

* Add DateChangeTimeKeeper unit tests

* Minor fix

* Add more DateChangeTimeKeeper unit tests

* Address peer review

* Address peer review

* Address peer review

* Detect mappings using the config NewSymbol event

* DateChangeTimeKeeper performance improvement for time zone conversions

* Clean up date change time keeper

* Cleanup

* Minor regression algo update

* Cleanup

* Cleanup

* Fix perform universe selection on sundays for continuous futures with extended market hours

* Cleanup

* Address peer review

* More cleanup

* More cleanup
2024-10-21 09:38:00 -03:00
Roman Yavnikov 6b22254924 Feature: new sorting property in SubscriptionDataSource and SortEnumerator (#8352)
API Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Report Generator Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
* feat: sort property in SubscriptionDataSource

* refactor: read lines (with sort opportunity) in TextSubscriptionDataSourceReader

* refactor: xml description of Sort property in SubscriptionDataSource

* revet: TextSubscriptionDataSourceReader changes

* feat: develop UnsortedTextSubscriptionDataSourceReader

* remove: UnsortedTextSubscriptionDataSourceReader

* feat: develop SortEnumerator
feat: use SortEnumerator in SubscriptionDataReader

* revert: TextSubscriptionDataSourceReader changes

* feat: use sort Enumerator in LiveCustomDataSubscriptionEnumeratorFactory

* test:feat: Unsorted Live Data Feed with using Object Store

* test:feat: history CustomData request with unsorted data

* refactor: develop TryWrapper on SortEnumerator

* remove: extra spacing

* refactor: Sort in SubscriptionDataSource

* feat: add RegressionAlgorithms when source return data in Descending format

* refactor: xml description of prop Sort in  SubscriptionDataSource

* refactor: SortEnumerator

* test:refactor access modifier in ExampleCustomDataWithSort

* test:refactor: increase performance LiveDataFeedSourcesDataFromObjectStore

* feat: add xml comment of regression algorithm

* test:revert: access modifier to public in internal test class

* test:refactor: use while instead of timer

* Revert "test:refactor: use while instead of timer"

This reverts commit 4fe43da8094df366f7ae4b361c97a4a14b815c32.
2024-10-07 18:16:33 -03:00
Jhonathan Abreu 0a9dc2c71c QCAlgorithm's OptionChain() api refactor (#8334)
* Fix pandas converter to handle list of data with different symbols

* Properly convert list of data into dataframe

Take into consideration data for multiple symbols in the same list

* Cleanup

* Index dataframes by symbol object instead of SID string

* Add symbol equality operator to compare against object

* Exclude "ID" from option chain dataframe

* Minor fix

* Add greeks columns directly in option chain dataframe.

Also add pass-through properties for greek values in OptionUniverse

* Some cleanup

* Minor fix

* Add new QCAlgorithm.OptionChains() method

- Use OptionChains as output
- Add DataFrame to OptionChain and OptionChains
- Rename Greeks classes
- Add ISymbolProvider for classes that have a symbol (IBaseData, OptionContract)

* Unify QCAlgorithmOptionChain API

Also refactor OptionContract to handle: (1) Actual market data and option price model data, and (2) OptionUniverse data

* Pass symbol properties to OptionUniverse option chain from algorithm

* Format OptionContract for dataframe

* Minor fix

* Add multiple option chains api regression algorithms and other minor changes

* Address peer review

Add NullGreeks class: keep ModeledGreeks as internal as possible

* Minor fix and add PandasConverter unit tests

* Peer review: Non-thread-safe Lazy for Python

* Handle Greeks unwrapping by PandasData

* PandasData cleanup

* Add data and other minor changes

* Unit test fix

* Update Pythonnet to 2.0.39

* Cleanup

* PandasData handling children class members

Address peer review

* Fix: indexing symbol conversion in pandas mapper

* Fix pandas mapper to convert string keys to symbol only when necessary

* Cleanup

* Cleanup

* Add PandasColumn python class to handle proper indexing

This allows propery hash and equality between Symbols, C# strings and Python strings

* Minor fixes

* Symbol cache improvements

* Minor fix for cache miss

* Revert PandasMapper reserved names and improvements

* Minor fix

* Revert reserved names

* Minor fix for Symbol equality operators

---------

Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
2024-10-04 12:26:15 -04:00
Martin-Molinero f2f1d06237 Improve shutdown (#8335) 2024-09-19 19:32:29 -03:00