Commit Graph

37 Commits

Author SHA1 Message Date
Jhonathan Abreu a565dfa6f0 Wait for fresh data before filling market orders on stale data (#9563)
* Wait for first session bar before filling equity market orders at open

EquityFillModel.MarketFill could fill a market order placed right after
market open using data from the previous trading date, because the first
bar of the current session has not been emitted yet. ShouldWaitForFreshData
only covered hour/daily resolutions, so minute/second orders filled on stale
prices.

Add IsWithinFirstResolutionSpanAfterMarketOpen: when the order time is within
the lowest subscribed resolution span after the open and the price is stale,
wait for the first bar instead of filling on the previous date's price.

* Share opening-bar stale-fill wait across fill models

Move IsWithinFirstResolutionSpanAfterMarketOpen to the base FillModel and add
a ShouldWaitForFreshDataOnStale sibling helper that combines it with the
existing coarse-resolution ShouldWaitForFreshData check. The base FillModel,
FutureFillModel and EquityFillModel market fills now share this single wait
decision at their stale-data guards.

ShouldWaitForFreshData is intentionally left untouched at its GetMarketFillPrice
call site, which uses it to choose the bar open vs current price and is not
gated by staleness, so fill prices for finer resolutions are unchanged. The
opening-bar helper is guarded against always-open markets, which have no
session open to wait for.

* Add regression algorithm for stale fill at market open

Reproduces the opening-bar stale fill issue: a market order placed one second
after the open while subscribed to minute resolution. Without the fix the order
fills on the previous trading date's stale price; the algorithm asserts in
OnOrderEvent that a fill never happens within the first minute after the open,
so it errors without the fix and passes with it.

Uses SPY minute data over 2013-10-07 to 2013-10-11, which is available in the
repository Data folder.

* Add unit tests for stale fill wait at market open

Cover the opening-bar stale fill scenario directly at the fill model level:
a market order placed within the first bar after the session open, while only
the previous session's stale bar is available, must wait instead of filling on
the stale price, and fills once the first session bar arrives. EquityFillModel
also asserts the boundary (orders past the first bar still fill on stale data),
and FutureFillModel covers the shared base helper from the future path.

* Generalize stale market-order fill wait to any time of day

Replace the market-open-specific wait with a generic check: a market order
that would be filled on stale data waits for fresh data when the latest
available data is more than one subscribed resolution bar behind the current
time. This no longer considers the market open explicitly; it covers the
opening bar (the first session bar has not been emitted yet) and any intraday
data gap larger than the resolution.

ShouldWaitForFreshDataOnStale now takes the latest data end time and the
current time instead of the order time, and is shared by FillModel,
FutureFillModel and EquityFillModel. Coarse resolutions (hour/daily) still
always wait; tick never waits. Internal configurations are included when
sizing the resolution bar. EquityFillModel's best-effort price helpers now
report the stale data end time so the gap can be measured.

Tests: EquityFillModelTests and FutureFillModelTests cover the market-open and
mid-session stale cases (wait then fill on fresh data) plus the within-one-bar
boundary (fill on stale). The regression algorithm is generalized to assert no
fill happens on data staler than the resolution, with orders at the open and
mid-session. Pre-existing plumbing/data-selection tests that used degenerate
timestamps were given fresh timestamps so they still exercise their original
intent.

* Add sample data and adjust regression algorithms for stale-fill wait

Add minute/daily sample data so market orders that now wait for fresh data
can fill (ES futures gap days, TWX/GOOG equities and options, SPXW weeklies,
GC futures/options copy for 2020-01-06). Adjust a few regression algorithms
to the deferred-fill behavior: cap orders in the extended-market continuous
future test, ignore daily-resolution SPY in the automatic-seed data checks,
and refresh OptionAssignmentStatistics expected constants.

* Update regression expected statistics for stale-fill wait

Regenerate ExpectedStatistics, DataPoints and AlgorithmHistoryDataPoints for
the regression algorithms affected by the wait-for-fresh-data fill change and
the added sample data: futures/options fill-timing shifts, ES data-point count
increases, and GOOG 2015-12-28 outcome changes.

* Trim SPXW sample data to expiries within filter window

The two SPXW algorithms filter with Expiration(0,7), so contracts expiring
more than a week out are never subscribed. Drop those far-dated expiries from
the 2021-01-06/08 minute files (760KB->108KB and 776KB->108KB on the quote
files). Fills, DataPoints and statistics are unchanged; both regression tests
still pass.

* Trim ES minute and GOOG option sample data to order-fill minimum

The ES minute gap-day files source no order fills (daily-resolution algos fill
from es_daily); keep only the front contract used for execution and drop the
unused back-month contracts. Trim the GOOG 2015-12-28 option file (no fill
depends on it) to the morning chain window. Regenerate the back-month futures
statistics affected by the dropped back-month bars. Full CSharp regression
suite passes (722/722).

* Use SMA gap threshold in BasicTemplateContinuousFuture for C#/Python parity

At a fast/slow SMA cross the two averages can coincide to within rounding
noise, where the C# (decimal) and Python (double) comparisons disagree,
producing different orders between languages. Require a minimum gap before
acting on a cross so both languages stay in lockstep, and update the shared
expected statistics accordingly.

* Mirror order cap in Python algorithm and update future history counts

Apply the same pre-2013-11-12/3-order cap to the Python
BasicTemplateContinuousFutureWithExtendedMarket algorithm for C#/Python parity,
and update the QuantBook future-history expected counts to reflect the added ES
sample data.

* Use SMA gap threshold in BasicTemplateContinuousFutureWithExtendedMarket for C#/Python parity

This algorithm had the same fast/slow SMA cross divergence already fixed in
BasicTemplateContinuousFutureAlgorithm (ad8fc33): at the 2013-10-29 cross the two
averages coincide to within rounding noise (C# decimal diff -1e-25, Python double
diff exactly 0.0), so the raw `_fast > _slow` / `_fast < _slow` comparisons disagree
between languages. C# fired a liquidate+rebuild that Python skipped, producing 5
orders in C# vs 3 in Python. Require a minimum 0.001 gap before acting on a cross so
both languages stay in lockstep, and regenerate the shared expected statistics
(Total Orders 5 -> 3).

* Document SMA cross threshold as a C#/Python parity workaround

Add a short note before the fast/slow SMA comparisons in both continuous-future
template algorithms clarifying that the minimum-gap threshold exists only so the
C# and Python versions take the exact same trades on the limited sample data in
the repository, where decimal vs double rounding can disagree at a cross.

* Fetch subscription configs once per equity market fill

MarketFill resolved the subscription configs twice per fill: once via the
best-effort price helpers (GetSubscribedTypes) and again via
ShouldWaitForFreshDataOnStale. Fetch them once and thread them through both
paths via optional parameters, leaving existing callers unchanged.

* Measure stale-fill wait against order submission time

ShouldWaitForFreshDataOnStale compared the latest data end time against the
security current time. Compare against the order submission time instead so the
decision to wait for fresh data reflects how stale the data is relative to when
the order was placed. Realign the stale-price warning fill test accordingly.

* Fix stale market data in SendingNewOrderFromOnOrderEvent test

The market price tick was timestamped a day before the order submission time,
so under the order-time staleness check the market orders waited for fresh data
instead of filling. Use a reference time with the tick one minute before the
order so the data is fresh and the orders fill.

* Centralize internal-inclusive subscription config lookup in fill models

ShouldWaitForFreshDataOnStale re-resolved the subscription configs through
the ShouldWaitForFreshData call it makes first, and GetMarketFillPrice did
the same. Thread the already-fetched configs through ShouldWaitForFreshData
and GetMarketFillPrice so each market fill resolves them at most once.

Add a GetSubscriptionDataConfigs(Security) helper on the base FillModel that
returns the internal-inclusive configs, and route every fill-model call site
through it to remove the duplicated lookup and repeated comment.

* Avoid list allocation in ShouldWaitForFreshData

Replace the Where(...).ToList() + All(...) with a single foreach over the
subscription configs, short-circuiting on the first non-coarse resolution.
2026-07-01 11:22:50 -04:00
JosueNina 7008d17714 Add MaxDrawdownRecovery metric (#8865)
API Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Report Generator Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Syntax Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
* Implement  a prototype of the maximum recovery time function.

* Add unit test skeletons.

* Add failing test

* Issue #4581: Implement MaxDrawdownRecoveryTime.

* Issue 4581: Add DTO for Drawdown Percentage, Drawdown Enddate, and High Value

* Issue 4581: Fix bgu for when lDrawdowns list is empty.

* Issue 4581: Change names of tests. Change name of file.

* Issue 4581: Make adjustements to flow of adding drawdowns to lDrawdowns.

* Issue 4581: Add multiple unit tests.

* Issue #4581: Change name of unit test

* Issue #4581: Add to PerformanceMetrics

* Issue #4581: Add Maximum Drawdown Recovery to PortolioStatistics class.

* Issue #4581: Add to portolfio statistics class.

* Issue #4581: Add to statistics builder.

* Issue #4581: Add report key.

* Case #4581: Convert to decimal.

* Issue #4581: Correct comment.

* Issue #4581: Correct performance metrics view model string.

* Case #4581: Correct statistics builder view model string..again.

* Issue #4581: Placed DradownDradownDateHighValueDTO at the end of the file for simpler diff.

* Issue #4581: Add 2 new tests.

* Issue #4581: Change algorithm so that when multiple maximum drawdowns occur, the longest of all recoveries is reported.

* Issue #4581: Add unit test.

* Issue #4581: Remove reportkey. Change dto name.

* Issue #4581: Change summary.

* Issue #4581: Change comment.

* Add max drawdown recovery calculation with unit tests

* Update regression algorithms with the new metric

* Solve review comments

* Update regression algorithms

* Add TryGet to safely get the key: MaximumDrawdownRecovery

* Ignore MaximumDrawdownRecovery metric in OptimizationBacktest Json

* Revert changes in Messaging

* Update regression algorithms

* Add test case: TakesLongestRecoveryAmongMultipleDrawdowns

* Use integer days for MaximumDrawdownRecovery

* Add MaximumDrawdownRecoveryReportElement

* Use more explicit names

* Rename files and variables for consistency

* Update regression algorithms

---------

Co-authored-by: Alain Schaerer <aschaerer@pcatg.com>
2025-07-17 16:32:23 -03:00
Ricardo Andrés Marino Rojas 980cdeaa75 Remove fill warnings from order tags (#8668)
* Remove warnings from tags and tests

* Fix bugs

* fix bugs

* Revert wrong updates in some stats

* Fix bugs
2025-04-03 17:49:21 -03:00
Jhonathan Abreu 69d2f5ae82 Futures and Future Options file-based universes (#8480)
Report Generator Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
API Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* Make FOPs selection universe file-based for backtesting

* Make FOPs option chains universe file based

* Make Future universe selection file-based like option universe

* Make Future universe selection file-based like option universe

* Abstraction cleanup

* Add FuturesChains API to QC algorithm

Also refactor future chain provider to use the new FutureUniverse instead of zip file names

* Update regression algorithms stats

* Refactor QuantBook option and future history to use new universes

* Fix failing tests

* Fix failing tests

* Fix failing tests

* Minor future chains unit test improvement

* Add futures chains DataFrame property

Also, remove IDerivativeSecurity interface from Future

* Add DataFrame property to FuturesChains class

* Add regression algorithms

* Add regression algorithms

* Replace QCAlgorithm.FutureChainProvider usages with new FuturesChain api

* Minor fixes

* Reduce number of universe files in repo

* Minor data fixes

* Regression algorithms updates

* Add implicit conversion from FuturesContract to Symbol

Modified algorithms to use futures contract objects directly instead of accessing their Symbol property.
Removed unnecessary import statements and redundant lines in various files.

* Improve resolution handling for history requests

* Changed _auxiliaryData field to lazily-initialized AuxiliaryData property

* Refactor data handling in BaseChain and TimeSliceFactory

- Added `AddData` method to `BaseChain` for adding market data
- Refactored `TimeSliceFactory` to use `BaseChain.AddData` method

* Remove specific constructors and indexers from Chain classes

Removed public indexers in `BaseChains` for getting or setting `BaseChain` instances by `ticker` or `Symbol`, which were used for Pythonnet compatibility.

* Remove chain cache logic from FuturesChainUniverse

* Refactor class and interface names for clarity

Renamed `FileBasedUniverse` to `BaseChainUniverseData` and
`IFileBasedUniverse` to `IChainUniverseData`.

* Add base class for options and futures contracts

- Introduced `BaseContract` as an abstract base class for contracts, consolidating common properties and methods.
- Removed ISymbolInterface

* Add minor fix for future options tickers parsing

Added tests

* Trigger Build

* Trigger Build

* Trigger Build

* Trigger Build

* Clean chain provider classes up

* Remove ZipEntryName other classes and unused code

Removed ZipEntryName class and references across various files.
Removed DataQueueFuturesChainUniverseDataCollectionEnumerator and DataQueueOptionChainUniverseDataCollectionEnumerator classes.
Removed OptionChainUniverseSubscriptionEnumeratorFactory class.
Removed unused code for handling OptionChainUniverse and FuturesChainUniverse in FileSystemDataFeed.cs and LiveTradingDataFeed.cs.
Removed several test files related to enumerator factories and universe data collection.

* Minor changes and cleanup

* Trigger Build

* Trigger Build

* Refactor FuturesContract data handling

Forward price data from bars and ticks stored in private fields for improved memory usage

* Fix: use universe data for market data in FuturesContract

* Update regression algorithms stats after rebase

Added HSI futures universe files

* Sort configs by internal flag

Internals go first

* Throw from option universe data filters for future options

Future options IV, Open interest and greeks are not supported for future options

* Minor changes

* Improve some regression algorithms

* Minor fix for failing unit tests

* Update FOPs universe file header

Removed greeks and IV columns.
Updated FOPs universe files: removed outdated columns.

* Minor unit test fix

* Trigger Build

* Trigger Build

* Trigger Build

* Trigger Build

* Trigger Build

* Trigger Build

* Trigger Build

* Trigger Build

* Trigger Build

* Trigger Build

* Trigger Build

* Trigger Build

* Trigger Build

* Minor fix

* Add history provider as constructor argument for chain providers

* Update new regression algorithms data points count

* Minor fix for FakeDataQueue

* Add initialize method to chain providers classes

* Minor changes

* Trigger Build

* Trigger Build

* Trigger Build

* Minor fix

* Minor fix

* Trigger Build

* Trigger Build

* Trigger Build

* Trigger Build

* Add logs to ProcessedDataProvider

* Removed test logs

* Minor fix

* Support downloading options and futures universe files from api data provider
2025-03-25 16:22:38 -04:00
Jhonathan Abreu 7eea365aa1 Let delistings through for internal subscriptions (#8398)
* Let delistings through for internal subscriptions

Without it, previously mapped continuous future contracts for which a position is open never received a delisting event for them to be liquidated and marked as non-tradable

* Update regression algorithm stats

* Revert/remove transaction handler initialization parameters DTO class

* Cleanup
2024-11-25 11:41:02 -03:00
Jhonathan Abreu f2100adb46 Fix price shifts around futures rollover date (#8371)
* Test

* Fix price shift on continuous future rollover

* Keep track of both data and exchange time in subscription data reader

* Fix for daily and hourly resolutions.

Add daily and hour algorithms

* Fixes for emitting new exchange dates at the beginning and end when data is not available

* Cleanup

* Minor fix

* Minor fix: detect mapping in data reader after advancing time keeper

* Minor fix

* More fixes

* More fixes

* Fixes for universe files

* More fixes for daily resolution data

* Separate DateChangeTimeKeeper to its own file

* Add DateChangeTimeKeeper unit tests

* Minor fix

* Add more DateChangeTimeKeeper unit tests

* Address peer review

* Address peer review

* Address peer review

* Detect mappings using the config NewSymbol event

* DateChangeTimeKeeper performance improvement for time zone conversions

* Clean up date change time keeper

* Cleanup

* Minor regression algo update

* Cleanup

* Cleanup

* Fix perform universe selection on sundays for continuous futures with extended market hours

* Cleanup

* Address peer review

* More cleanup

* More cleanup
2024-10-21 09:38:00 -03:00
Martin-Molinero 6c30157fab Remove universe selection on extended market dates (#8160)
- Remove universe selection on dates with extended market hours only
- Updating regression algorithms
- Expand date & time rules API to support specifying whether extended
  market hours only dates are desired or not
2024-07-08 11:58:50 -03:00
Ricardo Andrés Marino Rojas 16a737ee26 Fix most of CA1725 warnings (#8154)
* Fix CA1725 warning

* Address requested changes
2024-07-05 16:19:56 -03:00
Ricardo Andrés Marino Rojas a2b420cb0a Fix warnings part 8 (#8113)
* Fix CA1819 and CA1002 warnings

Changed the type of Languages statistic in regression tests from
Language[] to List<Language>. By doing that, the warning CA1819 was
removed but then the warning CA1002 was raised. However, this warning
was expected to be excluded from QuantConnect.Algorithm.CSharp.

* Improve implementation

* Simplify code

* Fix bugs
2024-06-28 10:10:30 -03:00
Ricardo Andrés Marino Rojas b41df5b0f2 1/4 of the changes of RegressionTestException (#8100)
Regression Tests / build (push) Has been cancelled
Report Generator Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
API Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
2024-06-24 15:23:33 -03:00
Ricardo Andrés Marino Rojas 8c33536498 Add algorithm status statistic (#8095)
* First draft of the solution

* Fix bugs

* Fix bugs
2024-06-24 14:19:32 -03:00
Ricardo Andrés Marino Rojas cce8945fe8 Api Clean up, Documentation and Standarization part two (#7964)
API Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Report Generator Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
* Add improvements

* Add improvments and unit tests

* Add XML comments

* Nit changes

* Add unit tests for OrderJsonConverter

* Improve unit tests

* Address requested changes

* Fix bugs

* Fix bugs

* Fix bugs and self-review

* Fix bugs

* Address requested changes

* Fix unit test bug

* Fix bugs

* Improve unit tests

* Solve bugs
2024-04-26 13:17:34 -03:00
Ricardo Andrés Marino Rojas ead2efe6b9 Add Starting and Ending KPI's (#7811)
* First draft of the solution

* Add missing changes

* Remove the new KPI's from report

* Fix bugs

* nit change

* Add improvements

* Fix regression tests

* Solve bugs in the regression algos

* Fix regression tests bugs

* Expand unit tests and add minor changes
2024-03-25 15:40:38 -03:00
Martin-Molinero feff802479 Standardize trade count statistic (#7827)
* Standarize trade count statistic

* Rename 'Total Trades' to 'Total Orders'
2024-03-06 14:52:34 -03:00
Jhonathan Abreu 09db45c91d Universes immediate selection (#7785)
* Make sure continuous futures universe selection is performed right away on the start of the algorithm.

This ensures the Future.Mapped is not null

* Minor changes

* Minor fixes

* Update regression algorithms stats

* Add DC futures map file

* Update DC futures map file

* Update regression algorithms stats

* Include ETFs for immediate universe selection

* Include Coarse Fundamentals for immediate universe selection

* Include Futures Chain Universe for immediate selection

* Add Option Chain Universe test

* Include custom universes for immediate selection

* Minor updates

* Include constituents universes for immediate selection

* Minor update

* Minor changes, docs and stats updates

* Minor changes

* Minor changes and clean up

* Minor regression algorithm stat update
2024-02-21 08:59:58 -04:00
Derek Melchin eefa74baaa Add Sortino ratio to statistics and report (#6698)
Regression Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
* Add Sortino ratio to statistics and report

* Adds Sortino Ratio to Report Key Statistics

* Addresses Peer-Review

Reuse `SharpeRatioReportElement` and change the template.

* Reuse Calculations Across Statistics and PortfolioStatistics

* Adds Sortino Ratio to Regression Algorithms

* Removes Sortino Ratio from Optimization Result Table

---------

Co-authored-by: Alexandre Catarino <AlexCatarino@users.noreply.github.com>
2023-12-12 21:06:13 -03:00
Ricardo Andrés Marino Rojas dcb5f8ee4e Merge branches 7501 and 7506 (#7605)
Research Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* Update Future-cme-[*] and Future-cme-ES

Acoording to `pandas_market_calendars` there were some dates in
Future-cme-[*] who wasn't early_closes, so they needed to be removed
from there. On the other hand, the early closes list of Future-cme-ES were
shifted by 1 hour according to CME webpage. Besides, there were some
missing dates.

* Update CME Future entries in MHDB

* Rebase

* nit change

* Fix unit tests

* Resume after early close/halts

* Add missing dates in MHDB and fix bugs in it

* Fix bug, add more unit tests and add docs

* fix regression algos

* address required changes

* Update failing regression test stats

After debugging the tests it was found they were failing due to the last
change on SecurityExchangeHours.IsOpen(). That method wasn't taking into
account that even if there is a late open after an early close if the
timespan is after the early close but before the late open, the market
is still close.

* enhance solution

* Update and fix bugs in MHDB

* Address required changes and update stats

* Update stats after rebase

* Nit change

* Missing update to regression test

* Use MHDB instead of USHoliday for Expiration Dates

VIX expiry function now relies completely on MHDB. However, it had to be
created an entry in MHDB for VIX since there wasn't one for it. CBOE
webpage only provided 2023 holidays so only those dates were considered
in the Holidays entry in MHDB. Therefore, some unit tests failed so it
was necessary to change also the VIX entry in FuturesExpiryFunctionsTestData.xml.

* Remove Global.cs/USHolidays class

* Use a lazy implementation

* First draft of the solution

* Use MHDB in FuturesExpiryFunctions.cs

* Remove unused class and fix indentation errors

* Fix indentation errors

* Nit changes

* Merge branches 7501 and 7506

* Merge changes in 7501 and 7506

In order to check compatibility between those branches, a new branch
was created out of branch 7501 and then it was merged with branch 7506.
2 regression tests and 8 unit tests failed, the regression tests failed on
the DataPoint stats. On the other hand, the unit tests failed since the
default parameter UseEquityHoliday was removed from
FuturesExpirtyUtilityFunctions.AddBusinessDays() and from other methods in
the same class too.

* Add missing changes

* Remove repeated good fridays

* Address minor review

---------

Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
2023-12-04 18:09:57 -03:00
Jhonathan Abreu 314a334e88 Price adjustment mode Order property (#7582)
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
* Add new Order.PriceAdjustmentMode property

* Minor fix and unit test

* Minor fix and regression algorithms' stats update

* Unit test fixes

* Minor fix

* Set order price adjustment mode to raw always for live trading
2023-11-17 16:09:03 -04:00
Martin-Molinero de10a1d669 Remove securities from algorithm collection (#7573)
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
- Once a security has been delisted or no longer selected by any
  universe it will be remove from the main algorithm securities
  collection but kept in the new Complete collection, this improves
  performance in long running backtests where security collection can
  get enumerated in different cases
- Minor refactor for security exchange local time to avoid having to set
  it explicitly
2023-11-15 17:09:43 -03:00
Ricardo Andrés Marino Rojas b4bad69772 Dynamically Adjust Risk Free Rate of Return (#7489)
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
* First attempt to solve the bug

* Enhance implementation

* Enhance implementation

* Simplify implementation

* Rebase regression stats

* Solve unit test bugs

* Review

* Update Rolling.Sharpe() method

* Update regression stats

* Update unit tests

* Update missing regression algos

* Update Rolling.cs

---------

Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
2023-10-02 13:42:28 -03:00
Martin-Molinero c33ce2f8c3 MHDB Merge Common Holidays, LateOpens & EarlyCloses (#7278)
* MHDB will merge common entry

- The MHDB will merge the market and security common entry holidays,
  early closes and late opens

* Normalize & reuse future US holidays

- Normalize & reuse future US holidays

* Update existing unit tests expected stats
2023-05-30 11:54:28 -03:00
Martin-Molinero b5ced1635e Chain Providers will filter expired (#7267)
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
- Future and option chain providers will always filter expired contracts
  from their result. Update existing tests
2023-05-23 18:33:07 -03:00
Jhonathan Abreu 0b81cf0218 Add dataMappingMode parameter to History methods (#7204)
Regression Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
* Add dataMappingMode parameter to every history api method overload

* Minor unit tests fixes

* Update regression algorithm stats

* Minor changes

* Minor changes
2023-04-21 17:24:14 -03:00
Martin-Molinero bbbab6d9a8 Refactor alpha statistics phase I (#7055)
Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Refactor alpha statistics

- Refactor alpha statistics, cleaning up and simplifying no longer required calculations and scoring
- Adding new InsightEvaluator abstraction, adding C# & PY regression
  algorithms

* Optimization backtest result json converter update

* Address reviews

- Remove IAlphaHandler, move insight storage responsability to IResultHandler
  and centralizing insight collection on the QCAlgorithm.Insights to be
  reused by the framework models
- Fix portfolio turnover single day backtests and duplicate time
  sampling handling. Updating regression algorithms

* Add InsightCollection tests and minor fixes

* Adding more & improved tests
2023-03-10 13:12:23 -03:00
Martin-Molinero c4433098c3 Refactor Framework Statistics (#7041)
* Refactor framework statistics

* Further insight chart cleanup
2023-03-03 19:40:29 -03:00
Martin-Molinero 6563d6e394 Use any resolution for chain provider (#6860)
Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Use any resolution for chain provider

- Use any data resolution available to source symbols for the file based
  chain provider. Adding unit test

* Fix selection timezone bug

- Fix universe selection timezone bug. Updating regression algorithms
2023-01-16 16:10:07 -03:00
Jhonathan Abreu 4415468d96 Update CME futures market hours (#6607)
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Update CME futures market hours

* Update regression algorithms stats

* Update regression algorithms stats

* Update expected values in unit tests

* Additional mhdb updates

* Update mhdb
2022-09-02 12:33:50 -03:00
Jhonathan Abreu 8e35155c40 Fix IB futures commission fees (#6557)
Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Adapted futures commission fees for IB

* Unit tests

* Some cleanup

* Fix exchange fees

* Update regression algorithms stats

* Update unit tests
2022-08-18 18:35:49 -03:00
Jhonathan Abreu 539011274c Support extended market hours for futures (#6522)
Python Virtual Environments / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Support extended market hours in AddFuture()

* Support extended market hours in AddFutureContract()

* Add C# regression algorithm

* Add Python regression algorithm

* Add regression algorithm for future contracts

* Add regression algorithm checking market hour ranges

* Fixed future regression algorithms to use extended market hours

* Fixed future regression algorithms to use extended market hours

* Fixed future regression algorithms to use extended market hours

* Fixed AddFutureOptionContractFromFutureChainRegressionAlgorithm to use extended market hours

* Update future market hours to include extended in market hours database

* Fixed AddFutureOptionContractDataStreamingRegressionAlgorithm to use extended market hours

* Fixed AddFutureOptionContractFromFutureChainRegressionAlgorithm to use extended market hours

* Fixed AddFutureContractWithContinuousRegressionAlgorithm to use extended market hours

* Fixed BasicTemplateContinuousFutureAlgorithm to use extended market hours

* Fixed BasicTemplateFuturesAlgorithm to use extended market hours

* Fix BasicTemplateFuturesDailyAlgorithm to use extended market hours

* Fixed BasicTemplateFuturesFrameworkAlgorithm to use extended market hours

* Fixed BasicTemplateFuturesHistoryAlgorithm to use extended market hours

* Fixed ContinuousBackMonthRawFutureRegressionAlgorithm to use extended market hours

* Fixed ContinuousFutureBackMonthRegressionAlgorithm to use extended market hours

* Fixed ContinuousFutureHistoryRegressionAlgorithm to use extended market hours

* Fixed ContinuousFutureLimitIfTouchedOrderRegressionAlgorithm to use extended market hours

* Fixed ContinuousFutureRegressionAlgorithm to use extended market hours

* Fixed DelistedFutureLiquidateRegressionAlgorithm to use extended market hours

* Fixed AutomaticIndicatorWarmupDataTypeRegressionAlgorithm to use extended market hours

* Fixed ConsolidateRegressionAlgorithm to use extended market hours

* Fixed DelistingFutureOptionRegressionAlgorithm to use extended market hours

* Fixed EqualWeightingPortfolioConstructionModelFutureRegressionAlgorithm to use extended market hours

* Fixed FutureContractsExtendedMarketHoursRegressionAlgorithm to use extended market hours

* Fixed FutureMarketOpenAndCloseRegressionAlgorithm to use extended market hours

* Fixed FutureMarketOpenConsolidatorRegressionAlgorithm to use extended market hours

* Fixed FutureOptionBuySellCallIntradayRegressionAlgorithm to use extended market hours

* Fixed FutureOptionCallITMExpiryRegressionAlgorithm to use extended market hours

* Fixed FutureOptionCallITMGreeksExpiryRegressionAlgorithm to use extended market hours

* Fixed FutureOptionCallOTMExpiryRegressionAlgorithm to use extended market hours

* Fixed FutureOptionDailyRegressionAlgorithm to use extended market hours

* Fixed FutureOptionHourlyRegressionAlgorithm to use extended market hours

* Fixed FutureOptionMultipleContractsInDifferentContractMonthsWithSameUnderlyingFutureRegressionAlgorithm to use extended market hours

* Fixed FutureOptionPutITMExpiryRegressionAlgorithm to use extended market hours

* Fixed FutureOptionPutOTMExpiryRegressionAlgorithm to use extended market hours

* Fixed FutureOptionShortCallITMExpiryRegressionAlgorithm to use extended market hours

* Fixed FutureOptionShortCallOTMExpiryRegressionAlgorithm to use extended market hours

* Fixed FutureOptionShortPutITMExpiryRegressionAlgorithm to use extended market hours

* Fixed FutureOptionShortPutOTMExpiryRegressionAlgorithm to use extended market hours

* Fixed FuturesAndFuturesOptionsExpiryTimeAndLiquidationRegressionAlgorithm to use extended market hours

* Fixed FuturesExpiredContractRegression to use extended market hours

* Fixed FutureSharingTickerRegressionAlgorithm to use extended market hours

* Fixed HistoryWithDifferentContinuousContractDepthOffsetsRegressionAlgorithm to use extended market hours

* Fixed HistoryWithDifferentDataMappingModeRegressionAlgorithm to use extended market hours

* Fixed HistoryWithDifferentDataNormalizationModeRegressionAlgorithm to use extended market hours

* Fixed LimitOrdersAreFilledAfterHoursForFuturesRegressionAlgorithm to use extended market hours

* Fixed OpenInterestFuturesRegressionAlgorithm to use extended market hours

* Fixed RegisterIndicatorRegressionAlgorithm to use extended market hours

* Fixed SetHoldingsFutureRegressionAlgorithm to use extended market hours

* Fixed WarmupFutureRegressionAlgorithm to use extended market hours

* Fixed AddFutureOptionSingleOptionChainSelectedInUniverseFilterRegressionAlgorithm to use extended market hours

* Fixed AlgorithmHistoryTests to use extended market hours for futures

* Fixed AlgorithmTradingTests to use extended market hours for futures

* Fixed BrokerageSetupHandlerTests to use extended market hours for futures

* Fixed TimeRulesTests to use extended market hours for futures

* Fixed FutureOptionMarginBuyingPowerModelTests to use extended market hours for futures

* Fixed FutureMarginBuyingPowerModelTests to use extended market hours for futures

* Fixed FileSystemDataFeedTests to use extended market hours for futures

* Fixed QuantBookHistoryTests to use extended market hours for futures

* Split BasicTemplateContinuousFutureAlgorithm to have an extended market version

* Fixed FutureMarketOpenAndCloseRegressionAlgorithm to use extended market hours

* Split BasicTemplateFuturesAlgorithm to have an extended market version

* Split BasicTemplateFuturesAlgorithm to have an extended market version

* Split BasicTemplateFuturesFrameworkAlgorithm to have an extended market version

* Split BasicTemplateFuturesHistoryAlgorithm to have an extended market version

* Revert AddFutureContractWithContinuousRegressionAlgorithm

* Revert AddFutureOptionContractDataStreamingRegressionAlgorithm and added data

* Revert AddFutureOptionContractFromFutureChainRegressionAlgorithm

* Revert AddFutureOptionSingleOptionChainSelectedInUniverseFilterRegressionAlgorithm

* Revert ConsolidateRegressionAlgorithm

* Revert Algorithm.CSharp/ContinuousBackMonthRawFutureRegressionAlgorithm.cs

* Revert ContinuousFutureBackMonthRegressionAlgorithm

* Revert ContinuousFutureHistoryRegressionAlgorithm

* Revert ContinuousFutureLimitIfTouchedOrderRegressionAlgorithm

* Revert ContinuousFutureRegressionAlgorithm

* Revert Algorithm.CSharp/DelistedFutureLiquidateRegressionAlgorithm.cs

* Revert EqualWeightingPortfolioConstructionModelFutureRegressionAlgorithm

* Split FutureMarketOpenAndCloseRegressionAlgorithm to have an extended market version

* Split FutureMarketOpenConsolidatorRegressionAlgorithm to have an extended market version

* Revert FutureOptionBuySellCallIntradayRegressionAlgorithm

* Revert FutureOptionCallITMExpiryRegressionAlgorithm

* Revert FutureOptionDailyRegressionAlgorithm

* Revert FutureOptionPutITMExpiryRegressionAlgorithm

* Revert FutureSharingTickerRegressionAlgorithm

* Revert FuturesAndFuturesOptionsExpiryTimeAndLiquidationRegressionAlgorithm

* Revert FuturesExpiredContractRegression

* Revert HistoryWithDifferentContinuousContractDepthOffsetsRegressionAlgorithm

* Revert HistoryWithDifferentDataMappingModeRegressionAlgorithm

* Revert HistoryWithDifferentDataNormalizationModeRegressionAlgorithm

* Revert OpenInterestFuturesRegressionAlgorithm

* Revert RegisterIndicatorRegressionAlgorithm

* Revert SetHoldingsFutureRegressionAlgorithm

* Revert WarmupFutureRegressionAlgorithm

* Revert AutomaticIndicatorWarmupDataTypeRegressionAlgorithm

* Some cleanup

* Address changes request

* Address changes request

* Add more Class III Milk data to fix DelistingFutureOptionDailyRegressionAlgorithm
2022-08-18 10:56:44 -03:00
Martin-Molinero c43f472503 Fill model will use internal configurations (#6494)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
- Fill model will take into account internal configurations, this is
  specially relevant for continuous future mapped contracts. Updating
  regression algorithms and unit tests.
- Allowing internal configurations to update trade builder information
2022-07-19 19:00:35 -03:00
Martin-Molinero 778d436b94 Removed mapped futures contract is non tradable (#6479)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Minor fixes for mapped futures contract

- Future security was never able to be non tradable
- Skip remove future contracts from live holdings. Adding unit test
  reproducing issue and regression algorithm

* Fix removal of continuous futures

- The canonical continuous future would leave behind an OpenInterest
  subscription when removed because of a different in the way the
  configurations were built, this will now be centralized in a single
  method. Adding unit tests reproducing issue
- FillForwarding setting of the continuous future was not being
  respected. Updating algorithm reproducing the issue to assert the
  behavior failing in master

* Address reviews

- Minor refactors addressing reviews
2022-07-13 15:36:24 -03:00
Ronit Jain d1ff914e5a fix docs (#6285)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
2022-04-08 17:44:01 -03:00
Ronit Jain 15066ae5e1 Feature improve regression tests (#6245)
* add data count properties

* 'add history count property

* assert data counts

* update missing override

* consider override/virtual cases

* implement data count

* add message handler for regression tests

* use regression test message handler

* set algorithm manager for regression test message handler

* update data count

* check if stats are present, check if algo manager is not null

* update

* add c# algo

* make same as c# algo

* use new line

* logic shifted to RegressionTestMessageHandler

* cleanup

* auto cleanup

* skip non deterministic data count

* change data count

* use inheritance

* improve stats

* update couht

* add sma indicator to c# and customSMA to python

* call base method before executing further

* skip test

* revert to original

* add duplicate sma

* skip regression test
2022-03-15 16:51:15 -03:00
Martin-Molinero de57cedc5a Make continuous future untradable (#6252)
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Make continuous futures untradable

- To match live trading behavior. Adjust continuous futures securities
  to be untradable

* Set ContinuousFuture as non tradable

- Set continuous futures as non tradable. Update regression algorithms
- Fixes for symbol capacity calculation
2022-03-14 14:23:26 -03:00
Martin-Molinero 68e2a9170a Do not send internal SecurityChanges to Algorithm (#6118)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Do not send internal SecurityChanges to Algorithm

- Will not send internal security changes to the Algorithm by default.
  Following custom security changes filter pattern. Updating regression
  algorithms to assert behavior.
- The universe member will know wether it was added with internal
  configurations or not

* Address reviews use a separate collection for internals

* Refactor solution. Adding security changes constructor class
2021-12-21 20:24:32 -03:00
Martin-Molinero d1a35e6281 Continuous Futures Refactor. Live Mappings (#6076)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* Continuous Futures Refactor. Live Mappings

- Adding support for live mappings. LiveTradingDataFeed will handle any
  symbol mapping at the configuration layer and resubscribe through the
  IDQH
- Refactoring continuous futures adding ContinuousFutureUniverse that
  will select the currently mapped security

* Minor fixes

- Remove addition of configurations in UniverseSelection step, leave
  resposability for universe.
- LiveTradingDF future unit test will only assert slice data for non
  internal feeds.
- ContinuousContractUniverse will respect internal option interest
  subscription

* Address review
2021-11-30 21:38:50 -03:00
Martin-Molinero b8b0d18993 Continuous futures (#6034)
* Continuous Future Contracts

* Mapping approach

* Tweaks WIP

* Live mapping

* Live mapping

- Add support for live mapping, refreshing mapfiles
- Fix future expiration functions
- Adding unit tests

* Update moq test package

* Continuous futures price scaling

* Refactor price factors scaling

* Factor file related renames

* Address reviews
2021-11-15 14:44:30 -03:00