Files
Jhonathan Abreu a565dfa6f0 Wait for fresh data before filling market orders on stale data (#9563)
* Wait for first session bar before filling equity market orders at open

EquityFillModel.MarketFill could fill a market order placed right after
market open using data from the previous trading date, because the first
bar of the current session has not been emitted yet. ShouldWaitForFreshData
only covered hour/daily resolutions, so minute/second orders filled on stale
prices.

Add IsWithinFirstResolutionSpanAfterMarketOpen: when the order time is within
the lowest subscribed resolution span after the open and the price is stale,
wait for the first bar instead of filling on the previous date's price.

* Share opening-bar stale-fill wait across fill models

Move IsWithinFirstResolutionSpanAfterMarketOpen to the base FillModel and add
a ShouldWaitForFreshDataOnStale sibling helper that combines it with the
existing coarse-resolution ShouldWaitForFreshData check. The base FillModel,
FutureFillModel and EquityFillModel market fills now share this single wait
decision at their stale-data guards.

ShouldWaitForFreshData is intentionally left untouched at its GetMarketFillPrice
call site, which uses it to choose the bar open vs current price and is not
gated by staleness, so fill prices for finer resolutions are unchanged. The
opening-bar helper is guarded against always-open markets, which have no
session open to wait for.

* Add regression algorithm for stale fill at market open

Reproduces the opening-bar stale fill issue: a market order placed one second
after the open while subscribed to minute resolution. Without the fix the order
fills on the previous trading date's stale price; the algorithm asserts in
OnOrderEvent that a fill never happens within the first minute after the open,
so it errors without the fix and passes with it.

Uses SPY minute data over 2013-10-07 to 2013-10-11, which is available in the
repository Data folder.

* Add unit tests for stale fill wait at market open

Cover the opening-bar stale fill scenario directly at the fill model level:
a market order placed within the first bar after the session open, while only
the previous session's stale bar is available, must wait instead of filling on
the stale price, and fills once the first session bar arrives. EquityFillModel
also asserts the boundary (orders past the first bar still fill on stale data),
and FutureFillModel covers the shared base helper from the future path.

* Generalize stale market-order fill wait to any time of day

Replace the market-open-specific wait with a generic check: a market order
that would be filled on stale data waits for fresh data when the latest
available data is more than one subscribed resolution bar behind the current
time. This no longer considers the market open explicitly; it covers the
opening bar (the first session bar has not been emitted yet) and any intraday
data gap larger than the resolution.

ShouldWaitForFreshDataOnStale now takes the latest data end time and the
current time instead of the order time, and is shared by FillModel,
FutureFillModel and EquityFillModel. Coarse resolutions (hour/daily) still
always wait; tick never waits. Internal configurations are included when
sizing the resolution bar. EquityFillModel's best-effort price helpers now
report the stale data end time so the gap can be measured.

Tests: EquityFillModelTests and FutureFillModelTests cover the market-open and
mid-session stale cases (wait then fill on fresh data) plus the within-one-bar
boundary (fill on stale). The regression algorithm is generalized to assert no
fill happens on data staler than the resolution, with orders at the open and
mid-session. Pre-existing plumbing/data-selection tests that used degenerate
timestamps were given fresh timestamps so they still exercise their original
intent.

* Add sample data and adjust regression algorithms for stale-fill wait

Add minute/daily sample data so market orders that now wait for fresh data
can fill (ES futures gap days, TWX/GOOG equities and options, SPXW weeklies,
GC futures/options copy for 2020-01-06). Adjust a few regression algorithms
to the deferred-fill behavior: cap orders in the extended-market continuous
future test, ignore daily-resolution SPY in the automatic-seed data checks,
and refresh OptionAssignmentStatistics expected constants.

* Update regression expected statistics for stale-fill wait

Regenerate ExpectedStatistics, DataPoints and AlgorithmHistoryDataPoints for
the regression algorithms affected by the wait-for-fresh-data fill change and
the added sample data: futures/options fill-timing shifts, ES data-point count
increases, and GOOG 2015-12-28 outcome changes.

* Trim SPXW sample data to expiries within filter window

The two SPXW algorithms filter with Expiration(0,7), so contracts expiring
more than a week out are never subscribed. Drop those far-dated expiries from
the 2021-01-06/08 minute files (760KB->108KB and 776KB->108KB on the quote
files). Fills, DataPoints and statistics are unchanged; both regression tests
still pass.

* Trim ES minute and GOOG option sample data to order-fill minimum

The ES minute gap-day files source no order fills (daily-resolution algos fill
from es_daily); keep only the front contract used for execution and drop the
unused back-month contracts. Trim the GOOG 2015-12-28 option file (no fill
depends on it) to the morning chain window. Regenerate the back-month futures
statistics affected by the dropped back-month bars. Full CSharp regression
suite passes (722/722).

* Use SMA gap threshold in BasicTemplateContinuousFuture for C#/Python parity

At a fast/slow SMA cross the two averages can coincide to within rounding
noise, where the C# (decimal) and Python (double) comparisons disagree,
producing different orders between languages. Require a minimum gap before
acting on a cross so both languages stay in lockstep, and update the shared
expected statistics accordingly.

* Mirror order cap in Python algorithm and update future history counts

Apply the same pre-2013-11-12/3-order cap to the Python
BasicTemplateContinuousFutureWithExtendedMarket algorithm for C#/Python parity,
and update the QuantBook future-history expected counts to reflect the added ES
sample data.

* Use SMA gap threshold in BasicTemplateContinuousFutureWithExtendedMarket for C#/Python parity

This algorithm had the same fast/slow SMA cross divergence already fixed in
BasicTemplateContinuousFutureAlgorithm (ad8fc33): at the 2013-10-29 cross the two
averages coincide to within rounding noise (C# decimal diff -1e-25, Python double
diff exactly 0.0), so the raw `_fast > _slow` / `_fast < _slow` comparisons disagree
between languages. C# fired a liquidate+rebuild that Python skipped, producing 5
orders in C# vs 3 in Python. Require a minimum 0.001 gap before acting on a cross so
both languages stay in lockstep, and regenerate the shared expected statistics
(Total Orders 5 -> 3).

* Document SMA cross threshold as a C#/Python parity workaround

Add a short note before the fast/slow SMA comparisons in both continuous-future
template algorithms clarifying that the minimum-gap threshold exists only so the
C# and Python versions take the exact same trades on the limited sample data in
the repository, where decimal vs double rounding can disagree at a cross.

* Fetch subscription configs once per equity market fill

MarketFill resolved the subscription configs twice per fill: once via the
best-effort price helpers (GetSubscribedTypes) and again via
ShouldWaitForFreshDataOnStale. Fetch them once and thread them through both
paths via optional parameters, leaving existing callers unchanged.

* Measure stale-fill wait against order submission time

ShouldWaitForFreshDataOnStale compared the latest data end time against the
security current time. Compare against the order submission time instead so the
decision to wait for fresh data reflects how stale the data is relative to when
the order was placed. Realign the stale-price warning fill test accordingly.

* Fix stale market data in SendingNewOrderFromOnOrderEvent test

The market price tick was timestamped a day before the order submission time,
so under the order-time staleness check the market orders waited for fresh data
instead of filling. Use a reference time with the tick one minute before the
order so the data is fresh and the orders fill.

* Centralize internal-inclusive subscription config lookup in fill models

ShouldWaitForFreshDataOnStale re-resolved the subscription configs through
the ShouldWaitForFreshData call it makes first, and GetMarketFillPrice did
the same. Thread the already-fetched configs through ShouldWaitForFreshData
and GetMarketFillPrice so each market fill resolves them at most once.

Add a GetSubscriptionDataConfigs(Security) helper on the base FillModel that
returns the internal-inclusive configs, and route every fill-model call site
through it to remove the duplicated lookup and repeated comment.

* Avoid list allocation in ShouldWaitForFreshData

Replace the Where(...).ToList() + All(...) with a single foreach over the
subscription configs, short-circuiting on the first non-coarse resolution.
2026-07-01 11:22:50 -04:00
..

alt tag

LEAN Data Formats / Equity

QuantConnect hosts US Equity Data (market 'USA') provided by AlgoSeek (trade and quote data from post-2007) and QuantQuote (trade data from pre-2007). Ticks are stored unfiltered. TradeBars and QuoteBars have suspicious ticks filtered out and the resulting ticks are consolidated and saved. QuantConnect has trade equity ticks as {YYYYMMDD}_trade.zip files, and quote equity ticks as {YYYYMMDD}_quote.zip.

The US equity data is in the New York timezone. Data timezones are found in the MarketHoursDatabase.json

Equity data supports the following Resolutions:

  • Tick
  • Second
  • Minute
  • Hour
  • Daily

Minute, Second Data File Format

Minute, Second files are located in the equity / usa / resolution folders. The file name uses a 8-character length date: /data/equity/usa/minute/ticker/{YYYYMMDD}_{trade|quote}.zip. Note that only post-2007 period has quote data.

The zip file contains 1 CSV file which repeats the information about the path in the file name. e.g. 20140605_aapl_minute_trade.csv.

The trade CSV contents are as follows:

Time Open High Low Close Volume
15300000 6448000 6448000 6448000 6448000 90
  • Time - Milliseconds since midnight in the timezone of the data format.
  • Open - Deci-cents Open Price for TradeBar.
  • High - Deci-cents High Price for TradeBar.
  • Low - Deci-cents Low Price for TradeBar.
  • Close - Deci-cents Close Price for TradeBar.
  • Volume - Number of shares traded in this TradeBar.

The quote CSV contents are as follows:

Time Bid Open Bid High Bid Low Bid Close Bid Size Ask Open Ask High Ask Low Ask Close Ask Size
15300000 6448000 6448000 6448000 6448000 90 6500000 6500000 6500000 6500000 100
  • Time - Milliseconds since midnight in the timezone of the data format.
  • Bid Open - Deci-cents Bid Open Price for QuoteBar.
  • Bid High - Deci-cents Bid High Price for QuoteBar.
  • Bid Low - Deci-cents Bid Low Price for QuoteBar.
  • Bid Close - Deci-cents Bid Close Price for QuoteBar.
  • Bid Size - Number of shares being bid that quoted in this QuoteBar.
  • Ask Open - Deci-cents Ask Open Price for QuoteBar.
  • Ask High - Deci-cents Ask High Price for QuoteBar.
  • Ask Low - Deci-cents Ask Low Price for QuoteBar.
  • Ask Close - Deci-cents Ask Close Price for QuoteBar.
  • Ask Size - Number of shares being ask for that quoted in this QuoteBar.

Hour and Daily File Format

Hour and Daily files are located in the /equity/usa/{hour|daily} folder. Each file contains all bars available for this ticker. e.g. /data/equity/usa/hour/aapl.zip. The zip file contains 1 CSV file named the same as the ticker (aapl.csv). Only trade bar data is available in Hour and Daily resolution.

The CSV contents are as follows:

DateTime Open High Low Close Volume
20131001 09:00 6448000 6448000 6448000 6448000 90
  • DateTime - String date "YYYYMMDD HH:MM" in the timezone of the data format.
  • Open - Deci-cents Open Price for TradeBar.
  • High - Deci-cents High Price for TradeBar.
  • Low - Deci-cents Low Price for TradeBar.
  • Close - Deci-cents Close Price for TradeBar.
  • Volume - Number of shares traded in this TradeBar.

Divide prices by 10,000 to convert deci-cents to dollars.

Tick File Format

Equity tick data is stored in files which are located in the /equity/usa/tick folder. The file name uses a 8-character length date: /data/equity/usa/tick/{ticker}/{YYYYMMDD}_{trade|quote}.zip. QuantConnect currently provides Quote equity ticks (post-2007) and Trade equity ticks.

Trade tick files are stored in files named {YYYYMMDD}_trade.zip. There is one file equity tick names named after the data: 20131008_bac_Trade_Tick.csv. The CSV contains records of each trade:

Time TradeSale Trade Volume Exchange Trade Sale Condition Suspicious
14400009.367 137450 100 D 1 0
  • Time - Milliseconds since midnight in the timezone of the data format.
  • TradeSale - Deci-cents price of the tick sale.
  • Volume - Number of shares in the sale.
  • Exchange - Location of the sale.
  • Trade Sale Condition - Notes on the sale.
  • Suspicious - Boolean indicating the tick is flagged as suspicious according to AlgoSeek's algorithms. This generally indicates the trade is far from other market prices and may be reversed. TradeBar data excludes suspicious ticks.

Quote tick files are stored in files named {YYYYMMDD}_quote.zip. There is one file equity tick names named after the data: 20131008_bac_Quote_Tick.csv. The CSV contains records of each trade:

Time Bid Sale Bid Size Ask Sale Ask Size Exchange Quote Sale Condition Suspicious
14400009.367 137450 100 0 0 D 1 0
  • Time - Milliseconds since midnight in the timezone of the data format.
  • Bid Sale - Deci-cents bid price of the bid quote tick.
  • Bid Size - Number of shares in the bid quote tick.
  • Ask Sale - Deci-cents ask price of the ask quote tick.
  • Ask Size - Number of shares in the ask quote tick.
  • Exchange - Location of the sale.
  • Quote Sale Condition - Notes on the sale.
  • Suspicious - Boolean indicating the tick is flagged as suspicious according to AlgoSeek's algorithms. This generally indicates the quote is far from other market prices and may be reversed. Each quote tick contains either bid or ask data only. QuoteBar data excludes suspicious ticks.

Tick Exchange Codes

Exchange Letter Exchange Name
A NYSE MKT Stock Exchange
B NASDAQ OMX BX Stock Exchange
C National Stock Exchange
D FINRA
I International Securities Exchange
J Direct Edge A Stock Exchange
K Direct Edge X Stock Exchange
M Chicago Stock Exchange
N New York Stock Exchange
T NASDAQ OMX Stock Exchange
P NYSE Arca SM
S Consolidated Tape System
T/Q NASDAQ Stock Exchange
W CBOE Stock Exchange
X NASDAQ OMX PSX Stock Exchange
Y BATS Y-Exchange
Z BATS Exchange

Tick Sale Conditions

Trade Ticks
Sale Condition Code Description
0 No Condition
1 A trade made without stated conditions is deemed regular way for settlement on the third business day following the transaction date.
2 A transaction which requires delivery of securities and payment on the same day the trade takes place.
4 A transaction that requires the delivery of securities on the first business day following the trade date.
8 A Sellers Option transaction gives the seller the right to deliver the security at any time within a specific period, ranging from not less than two calendar days, to not more than sixty calendar days.
10 Market Centers will have the ability to identify regular trades being reported during specific events as out of the ordinary by appending a new sale condition code Yellow Flag (Y) on each transaction reported to the UTP SIP. The new sale condition will be eligible to update all market center and consolidated statistics.
20 The transaction that constituted the trade-through was the execution of an order identified as an Intermarket Sweep Order.
40 The trade that constituted the trade-through was a single priced opening transaction by the Market Center.
80 The transaction that constituted the trade-through was a single priced closing transaction by the Market Center.
100 The trade that constituted the trade-through was a single priced reopening transaction by the Market Center.
200 The transaction that constituted the trade-through was the execution of an order at a price that was not based, directly or indirectly, on the quoted price of the security at the time of execution and for which the material terms were not reasonably determinable at the time the commitment to execute the order was made.
400 Trading in extended hours enables investors to react quickly to events that typically occur outside regular market hours, such as earnings reports. However, liquidity may be constrained during such Form T trading, resulting in wide bid-ask spreads.
800 Sold Last is used when a trade prints in sequence but is reported late or printed in conformance to the One or Two Point Rule.
1000 The transaction that constituted the trade-through was the execution by a trading center of an order for which, at the time of receipt of the order, the execution at no worse than a specified price a 'stopped order'
2000 Identifies a trade that was executed outside of regular primary market hours and is reported as an extended hours trade.
4000 Identifies a trade that takes place outside of regular market hours.
8000 An execution in two markets when the specialist or Market Maker in the market first receiving the order agrees to execute a portion of it at whatever price is realized in another market to which the balance of the order is forwarded for execution.
10000 A transaction made on the Exchange as a result of an Exchange acquisition.
20000 A trade representing an aggregate of two or more regular trades in a security occurring at the same price either simultaneously or within the same 60-second period, with no individual trade exceeding 10,000 shares.
40000 Stock-Option Trade is used to identify cash equity transactions which are related to options transactions and therefore potentially subject to cancellation if market conditions of the options leg(s) prevent the execution of the stock-option order at the price agreed upon.
80000 Sale of a large block of stock in such a manner that the price is not adversely affected.
100000 A trade where the price reported is based upon an average of the prices for transactions in a security during all or any portion of the trading day.
200000 Indicates that the trade resulted from a Market Centers crossing session.
400000 Indicates a regular market session trade transaction that carries a price that is significantly away from the prevailing consolidated or primary market value at the time of the transaction.
800000 To qualify as a NYSE AMEX Rule 155
1000000 Indicates the Official closing value as determined by a Market Center. This transaction report will contain the market center generated closing price.
2000000 A sale condition that identifies a trade based on a price at a prior point in time i.e. more than 90 seconds prior to the time of the trade report. The execution time of the trade will be the time of the prior reference price.
4000000 Indicates the Official open value as determined by a Market Center. This transaction report will contain the market
8000000 The CAP Election Trade highlights sales as a result of a sweep execution on the NYSE, whereby CAP orders have been elected and executed outside the best price bid or offer and the orders appear as repeat trades at subsequent execution prices. This indicator provides additional information to market participants that an automatic sweep transaction has occurred with repeat trades as one continuous electronic transaction.
10000000 A sale condition code that identifies a NYSE trade that has been automatically executed without the potential benefit of price improvement.
20000000 Denotes whether or not a trade is exempt (Rule 611) and when used jointly with certain Sale Conditions, will more fully describe the characteristics of a particular trade.
40000000 This flag is present in raw data, but AlgoSeek document does not describe it.
80000000 Denotes the trade is an odd lot less than a 100 shares.
Quote Ticks
Sale Condition Code Description
0 No Condition
1 This condition is used for the majority of quotes to indicate a normal trading environment.
2 This condition is used to indicate that the quote is a Slow Quote on both the Bid and Offer sides due to a Set Slow List that includes High Price securities.
4 While in this mode, auto-execution is not eligible, the quote is then considered manual and non-firm in the Bid and Offer and either or both sides can be traded through as per Regulation NMS.
8 This condition can be disseminated to indicate that this quote was the last quote for a security for that Participant.
10 This regulatory Opening Delay or Trading Halt is used when relevant news influencing the security is being disseminated. Trading is suspended until the primary market determines that an adequate publication or disclosure of information has occurred.transaction reported to the UTP SIP. The new sale condition will be eligible to update all market center and consolidated statistics.
20 This condition is used to indicate a regulatory Opening Delay or Trading Halt due to an expected news announcement, which may influence the security. An Opening Delay or Trading Halt may be continued once the news has been disseminated.
40 The condition is used to denote the probable trading range (bid and offer prices, no sizes) of a security that is not Opening Delayed or Trading Halted. The Trading Range Indication is used prior to or after the opening of a security.
80 This non-regulatory Opening Delay or Trading Halt is used when there is a significant imbalance of buy or sell orders.
100 This condition is disseminated by each individual FINRA Market Maker to signify either the last quote of the day or the premature close of an individual Market Maker for the day.
200 This quote condition indicates a regulatory Opening Delay or Trading Halt due to conditions in which a security experiences a 10 % or more change in price over a five minute period.
400 This quote condition suspends a Participant's firm quote obligation for a quote for a security.
800 This condition can be disseminated to indicate that this quote was the opening quote for a security for that Participant.
1000 This non-regulatory Opening Delay or Trading Halt is used when events relating to one security will affect the price and performance of another related security. This non-regulatory Opening Delay or Trading Halt is also used when non-regulatory halt reasons such as Order Imbalance, Order Influx and Equipment Changeover are combined with Due to Related Security on CTS.
2000 This quote condition along with zero-filled bid, offer and size fields is used to indicate that trading for a Participant is no longer suspended in a security which had been Opening Delayed or Trading Halted.
4000 This quote condition is used when matters affecting the common stock of a company affect the performance of the non-common associated securities, e.g., warrants, rights, preferred, classes, etc.
8000 This non-regulatory Opening Delay or Trading Halt is used when the ability to trade a security by a Participant is temporarily inhibited due to a systems, equipment or communications facility problem or for other technical reasons.
10000 This non-regulatory Opening Delay or Trading Halt is used to indicate an Opening Delay or Trading Halt for a security whose price may fall below $1.05, possibly leading to a sub-penny execution.
20000 This quote condition is used to indicate that an Opening Delay or a Trading Halt is to be in effect for the rest of the trading day in a security for a Participant.
40000 This quote condition is used to indicate that a Limit Up-Limit Down Price Band is applicable for a security.
80000 This quote condition is used to indicate that a Limit Up-Limit Down Price Band that is being disseminated is a republication of the latest Price Band for a security.
100000 This indicates that the market participant is in a manual mode on both the Bid and Ask. While in this mode, automated execution is not eligible on the Bid and Ask side and can be traded through pursuant to Regulation NMS requirements.
200000 For extremely active periods of short duration. While in this mode, the UTP participant will enter quotations on a “best efforts” basis.
400000 A halt condition used when there is a sudden order influx. To prevent a disorderly market, trading is temporarily suspended by the UTP participant.

See more information in the AlgoSeek whitepaper.

Trade Ticks (Pre-2007)
Exchange Sale Condition Code Description
CTS Blank or @ Regular Sale (no condition)
CTS B Average Price Trade
CTS C Cash Trade (same day clearing)
CTS E Automatic Execution
CTS F Intermarket Sweep Order
CTS G Opening/Reopening Trade Detail
CTS H Intraday Trade Detail
CTS I CAP Election Trade
CTS J Rule 127 Trade
CTS K Rule 127 trade (NYSE only) or Rule 155 trade
NYSE L Sold Last (late reporting)
NYSE N Next Day Trade (next day clearing)
NYSE O Market Center Opening Trade
NYSE R Seller
NYSE S Reserved
NYSE T Extended Hours Trade
NYSE U Extended Hours (Sold Out of Sequence)
NYSE Z Sold (out of sequence)
NYSE 4 Derivatively Priced
NYSE 5 Market Center Re-opening Prints
NYSE 6 Market Center Closing Prints
NASD @ Regular Trade
NASD A Acquisition
NASD B Bunched Trade
NASD C Cash Trade
NASD D Distribution
NASD F Intermarket Sweep
NASD G Bunched Sold Trade
NASD K Rule 155 Trade (NYSE MKT Only)
NASD L Sold Last
NASD M Market Center Close Price
NASD N Next Day
NASD O Opening Prints
NASD P Prior Reference Price
NASD Q Market Center Open Price
NASD R Seller (Long-Form Message Formats Only)
NASD S Split Trade
NASD T Form - T Trade
NASD U Extended Hours (Sold Out of Sequence)
NASD W Average Price Trade
NASD Y Yellow Flag
NASD Z Sold (Out of Sequence)
NASD 1 Stopped Stock - Regular Trade
NASD 2 Stopped Stock - Sold Last
NASD 3 Stopped Stock - Sold Last 3
NASD 4 Derivatively Priced
NASD 5 Re-opening Prints
NASD 6 Closing Prints
NASD 7 Placeholder for 611 Exempt
NASD 8 Placeholder for 611 Exempt
NASD 9 Placeholder for 611 Exempt

See more information in the QuantQuote whitepaper.