Juan José D'Ambrosio
765f2cab3b
* Add main exchange to map file minimum minimorum test * Add MainExchange to MapFilRow IEquatable implementation Also write CSV line correctly * Normalize exchanges Also add small test form MapFileRow * Exchange name normalization - Exchange name normalization. - Tick will protobuf `ExchangeCode` which will set existing `Exchange` * Include single character exchange code * Encode exchange as byte, Include exchange name as Enum * Update MapFileRow.cs to accept a byte instead a string for primary exchange * Fix issues in encoding exchanges map * Fix broken MapFile tests * Rename Main Exchange property in MapFileRow * Use BATS instead of BATSZ * Add primary exchange from map file * Move MapFilePrimaryExchangeProvider to its own file * Implement Char for primary exchange in map files * Fix broken tests after implementing char exchange into map file * Add a new constructor for MapFileRow it accepts a PrimaryExxchange object as parameter * Fix broken test WIP * Address review WIP * Handles unexpected encoded exchanges * Keep addressing review * Address review * Fix broken tests Make PrimaryExchange as default even if the IPrimaryExchangeProvider is not instantiated * Revert breaking change. * Add a check that resolved map file actually have content The issue that leads to this change was caused by a map file resolved to APC in the CoarseFundamentalTop3Algorithm. The resolver returns a not-null MapFile, but it is empty. To my best understand, the MapFile Resolver should return only existing map files in the map_files folder. But the LocalMapFileProvider returned a map file with a permtick that doesn't exist in the file system as CSV. * Set PrimaryExchange.UNKNOWN as default * Address review couple minor bugs - Use SID market instead of USA - Default ticke exchange is UNKNOWN * Update map files with latest format and latest info This include: - VXX, it was delisted, then VXXB appears and later it chnages from VXXB -> VXX - AAA delisting and a new listing with ticker AAA - FOXA ticker changue to TFCFA and later it was delisted * Update test after updating latest map files * Fix bug with factor files after updating map files * Fix bug with factor files after updating map files and revert statistics in regression tests * Update OrderHash in regression test Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
LEAN Data Formats / Equity
QuantConnect hosts US Equity Data (market 'usa') provided by QuantQuote. Ticks are stored unfiltered. TradeBars have suspicious ticks filtered out and the resulting ticks are consolidated and saved. QuantConnect has trade equity ticks only; so only saves date_trade.zip files.
The US equity data is in the New York timezone. Data timezones are found in the MarketHoursDatabase.json
Equity data supports the following Resolutions:
- Tick
- Second
- Minute
- Hour
- Daily
Minute, Second Data File Format
Minute, Second files are located in the equity / usa / resolution folders. The file name uses a 8-character length date. /data/equity/usa/minute/ticker/YYYYMMDD_trade.zip.
The zip file contains 1 CSV file which repeats the information about the path in the file name. e.g. 20140605_aapl_minute_trade.csv. Currently this filename is not used but for consistency it should follow this pattern.
The CSV contents are as follows:
| Time | Open | High | Low | Close | Volume |
|---|---|---|---|---|---|
| 15300000 | 6448000 | 6448000 | 6448000 | 6448000 | 90 |
- Time - Milliseconds since midnight in the timezone of the data format.
- Open - Deci-cents Open Price for TradeBar.
- High - Deci-cents High Price for TradeBar.
- Low - Deci-cents Low Price for TradeBar.
- Close - Deci-cents Close Price for TradeBar.
- Volume - Number of shares traded in this TradeBar.
Hour and Daily File Format
Hour and Daily files are located in the /equity/usa/{hour|daily} folder. Each file contains all bars available for this ticker. e.g. /data/equity/usa/hour/aapl.zip. The zip file contains 1 CSV file named the same as the ticker (aapl.csv). Currently this filename is not used but for consistency it should follow this pattern.
The CSV contents are as follows:
| DateTime | Open | High | Low | Close | Volume |
|---|---|---|---|---|---|
| 20131001 09:00 | 6448000 | 6448000 | 6448000 | 6448000 | 90 |
- DateTime - String date "YYYYMMDD HH:MM" in the timezone of the data format.
- Open - Deci-cents Open Price for TradeBar.
- High - Deci-cents High Price for TradeBar.
- Low - Deci-cents Low Price for TradeBar.
- Close - Deci-cents Close Price for TradeBar.
- Volume - Number of shares traded in this TradeBar.
Divide prices by 10,000 to convert deci-cents to dollars.
Tick File Format
Equity tick data is stored in files which are located in the /equity/usa/tick folder. The file name uses a 8-character length date. /data/equity/usa/tick/{ticker}/YYYYMMDD_tickType.zip. QuantConnect currently only provides Trade equity ticks.
Trade tick files are stored in files named YYYYMMDD_trade.zip. There is one file equity tick names named after the data: 20131008_bac_Trade_Tick.csv. The CSV contains records of each trade:
| Time | TradeSale | Trade Volume | Exchange | Sale Condition | Suspicious |
|---|---|---|---|---|---|
| 48754000 | 137450 | 100 | D | @ | 0 |
- Time - Milliseconds since midnight in the timezone of the data format.
- TradeSale - Deci-cents price of the tick sale.
- Volume - Number of shares in the sale.
- Exchange - Location of the sale.
- Sale Condition - Notes on the sale.
- Suspicious - Boolean indicating the tick is flagged as suspicious according to QuantQuote's algorithms. This generally indicates the trade is far from other market prices and may be reversed. TradeBar data excludes suspicious ticks.
Tick Exchange Codes
| Exchange Letter | Exchange Name |
|---|---|
| A | NYSE MKT Stock Exchange |
| B | NASDAQ OMX BX Stock Exchange |
| C | National Stock Exchange |
| D | FINRA |
| I | International Securities Exchange |
| J | Direct Edge A Stock Exchange |
| K | Direct Edge X Stock Exchange |
| M | Chicago Stock Exchange |
| N | New York Stock Exchange |
| T | NASDAQ OMX Stock Exchange |
| P | NYSE Arca SM |
| S | Consolidated Tape System |
| T/Q | NASDAQ Stock Exchange |
| W | CBOE Stock Exchange |
| X | NASDAQ OMX PSX Stock Exchange |
| Y | BATS Y-Exchange |
| Z | BATS Exchange |
Tick Sale Conditions
| Exchange | Sale Condition Code | Description |
|---|---|---|
| CTS | Blank or ‘@’ | Regular Sale (no condition) |
| CTS | ‘B’ | Average Price Trade |
| CTS | ‘C’ | Cash Trade (same day clearing) |
| CTS | ‘E’ | Automatic Execution |
| CTS | ‘F’ | Intermarket Sweep Order |
| CTS | ‘G’ | Opening/Reopening Trade Detail |
| CTS | ‘H’ | Intraday Trade Detail |
| CTS | ‘I’ | CAP Election Trade |
| CTS | ‘J’ | Rule 127 Trade |
| CTS | ‘K’ | Rule 127 trade (NYSE only) or Rule 155 trade |
| NYSE | ‘L’ | Sold Last (late reporting) |
| NYSE | ‘N’ | Next Day Trade (next day clearing) |
| NYSE | ‘O’ | Market Center Opening Trade |
| NYSE | ‘R’ | Seller |
| NYSE | ‘S’ | Reserved |
| NYSE | ‘T’ | Extended Hours Trade |
| NYSE | ‘U’ | Extended Hours (Sold Out of Sequence) |
| NYSE | ‘Z’ | Sold (out of sequence) |
| NYSE | ‘4’ | Derivatively Priced |
| NYSE | ‘5’ | Market Center Re-opening Prints |
| NYSE | ‘6’ | Market Center Closing Prints |
| NASD | ‘@’ | Regular Trade |
| NASD | ‘A’ | Acquisition |
| NASD | ‘B’ | Bunched Trade |
| NASD | ‘C’ | Cash Trade |
| NASD | ‘D’ | Distribution |
| NASD | ‘F’ | Intermarket Sweep |
| NASD | ‘G’ | Bunched Sold Trade |
| NASD | ‘K’ | Rule 155 Trade (NYSE MKT Only) |
| NASD | ‘L’ | Sold Last |
| NASD | ‘M’ | Market Center Close Price |
| NASD | ‘N’ | Next Day |
| NASD | ‘O’ | Opening Prints |
| NASD | ‘P’ | Prior Reference Price |
| NASD | ‘Q’ | Market Center Open Price |
| NASD | ‘R’ | Seller (Long-Form Message Formats Only) |
| NASD | ‘S’ | Split Trade |
| NASD | ‘T’ | Form - T Trade |
| NASD | ‘U’ | Extended Hours (Sold Out of Sequence) |
| NASD | ‘W’ | Average Price Trade |
| NASD | ‘Y’ | Yellow Flag |
| NASD | ‘Z’ | Sold (Out of Sequence) |
| NASD | ‘1’ | Stopped Stock - Regular Trade |
| NASD | ‘2’ | Stopped Stock - Sold Last |
| NASD | ‘3’ | Stopped Stock - Sold Last 3 |
| NASD | ‘4’ | Derivatively Priced |
| NASD | ‘5’ | Re-opening Prints |
| NASD | ‘6’ | Closing Prints |
| NASD | ‘7’ | Placeholder for 611 Exempt |
| NASD | ‘8’ | Placeholder for 611 Exempt |
| NASD | ‘9’ | Placeholder for 611 Exempt |
See more information in the QuantQuote whitepaper.