Juan José D'Ambrosio
765f2cab3b
* Add main exchange to map file minimum minimorum test * Add MainExchange to MapFilRow IEquatable implementation Also write CSV line correctly * Normalize exchanges Also add small test form MapFileRow * Exchange name normalization - Exchange name normalization. - Tick will protobuf `ExchangeCode` which will set existing `Exchange` * Include single character exchange code * Encode exchange as byte, Include exchange name as Enum * Update MapFileRow.cs to accept a byte instead a string for primary exchange * Fix issues in encoding exchanges map * Fix broken MapFile tests * Rename Main Exchange property in MapFileRow * Use BATS instead of BATSZ * Add primary exchange from map file * Move MapFilePrimaryExchangeProvider to its own file * Implement Char for primary exchange in map files * Fix broken tests after implementing char exchange into map file * Add a new constructor for MapFileRow it accepts a PrimaryExxchange object as parameter * Fix broken test WIP * Address review WIP * Handles unexpected encoded exchanges * Keep addressing review * Address review * Fix broken tests Make PrimaryExchange as default even if the IPrimaryExchangeProvider is not instantiated * Revert breaking change. * Add a check that resolved map file actually have content The issue that leads to this change was caused by a map file resolved to APC in the CoarseFundamentalTop3Algorithm. The resolver returns a not-null MapFile, but it is empty. To my best understand, the MapFile Resolver should return only existing map files in the map_files folder. But the LocalMapFileProvider returned a map file with a permtick that doesn't exist in the file system as CSV. * Set PrimaryExchange.UNKNOWN as default * Address review couple minor bugs - Use SID market instead of USA - Default ticke exchange is UNKNOWN * Update map files with latest format and latest info This include: - VXX, it was delisted, then VXXB appears and later it chnages from VXXB -> VXX - AAA delisting and a new listing with ticker AAA - FOXA ticker changue to TFCFA and later it was delisted * Update test after updating latest map files * Fix bug with factor files after updating map files * Fix bug with factor files after updating map files and revert statistics in regression tests * Update OrderHash in regression test Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
LEAN Data Formats
Introduction
From the beginning LEAN strived to use an open, human readible data format - independent of any specific database or file format. From this core philosophy we built LEAN to read its financial data from flat files on disk. Data compression is done in zip format; and all individual files are CSV or JSON.
When there is no activity for a security, the price is omitted to the file. Only new ticks, and price changes are recorded.
File Data Format
Although we strive to make all data formats identical it is often not possible. Below are links to dedicated documentation on the file format of the data in each asset type:
Equity | Forex | Options | Futures | Crypto
Folder Structure
Data files are separated and nested in a few predictable layers:
-
Tick, Second and Minute Financial Data:
/data/securityType/marketName/resolution/ticker/date_tradeType.zip -
Hour, Daily Financial Data:
/data/securityType/marketName/resolution/ticker.zip
The market value is used to separate different tradable assets with the same ticker. E.g. EURUSD is traded on multiple brokerages all with slightly different prices.
Core Data Types
LEAN has a few core data types which are represented in all the asset classes we support. Below are links to their implementation in LEAN.
-
TradeBar - TradeBar represents trade ticks of assets consolidated for a period. TradeBar file format is slightly different for high resolution (second, minute) and low resolution (daily, hour).
-
QuoteBar - QuoteBar represents top of book quote data consolidated over a period of time (bid and ask bar).
-
Tick - Tick data represents an individual record of trades ("trade ticks") or quote updates ("quote tick") for an asset. Tick data is instantaneous - it does not have a period.
Data Readers
All data is parsed from disk via Reader() methods. The Reader takes a single line of the file and converts it the appropriate type. i.e. TradeBar.Reader() method is a factory which returns TradeBar objects. When implementing custom data Readers are used
Other Data Formats
Theoretically LEAN can accept data in any format (database, API or flatfile). However practically we currently have reader implementations written for a flat file system.