Jhonathan Abreu
0a9dc2c71c
* Fix pandas converter to handle list of data with different symbols * Properly convert list of data into dataframe Take into consideration data for multiple symbols in the same list * Cleanup * Index dataframes by symbol object instead of SID string * Add symbol equality operator to compare against object * Exclude "ID" from option chain dataframe * Minor fix * Add greeks columns directly in option chain dataframe. Also add pass-through properties for greek values in OptionUniverse * Some cleanup * Minor fix * Add new QCAlgorithm.OptionChains() method - Use OptionChains as output - Add DataFrame to OptionChain and OptionChains - Rename Greeks classes - Add ISymbolProvider for classes that have a symbol (IBaseData, OptionContract) * Unify QCAlgorithmOptionChain API Also refactor OptionContract to handle: (1) Actual market data and option price model data, and (2) OptionUniverse data * Pass symbol properties to OptionUniverse option chain from algorithm * Format OptionContract for dataframe * Minor fix * Add multiple option chains api regression algorithms and other minor changes * Address peer review Add NullGreeks class: keep ModeledGreeks as internal as possible * Minor fix and add PandasConverter unit tests * Peer review: Non-thread-safe Lazy for Python * Handle Greeks unwrapping by PandasData * PandasData cleanup * Add data and other minor changes * Unit test fix * Update Pythonnet to 2.0.39 * Cleanup * PandasData handling children class members Address peer review * Fix: indexing symbol conversion in pandas mapper * Fix pandas mapper to convert string keys to symbol only when necessary * Cleanup * Cleanup * Add PandasColumn python class to handle proper indexing This allows propery hash and equality between Symbols, C# strings and Python strings * Minor fixes * Symbol cache improvements * Minor fix for cache miss * Revert PandasMapper reserved names and improvements * Minor fix * Revert reserved names * Minor fix for Symbol equality operators --------- Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
LEAN Data Formats
Introduction
From the beginning, LEAN has strived to use an open, human-readable data format - independent of any specific database or file format. From this core philosophy, we built LEAN to read its financial data from flat files on disk. Data compression is done in zip format, and all individual files are CSV or JSON.
When there is no activity for a security, the price is omitted from the file. Only new ticks and price changes are recorded.
File Data Format
Although we strive to make all data formats identical, it is often impossible. Below are links to dedicated documentation on the file format of the data in each asset type:
Equity | Forex | Options | Futures | Crypto
Folder Structure
Data files are separated and nested in a few predictable layers:
-
Tick, Second and Minute Financial Data:
/data/securityType/marketName/resolution/ticker/date_tradeType.zip -
Hour, Daily Financial Data:
/data/securityType/marketName/resolution/ticker.zip
The marketName value is used to separate different tradable assets with the same ticker. E.g. EURUSD is traded on multiple brokerages all with slightly different prices.
Core Data Types
LEAN has a few core data types represented in all the asset classes we support. Below are links to their implementation in LEAN.
-
TradeBar - TradeBar represents trade ticks of assets consolidated for a period. TradeBar file format is slightly different for high resolution (second, minute) and low resolution (daily, hour).
-
QuoteBar - QuoteBar represents top of book quote data consolidated over a period of time (bid and ask bar).
-
Tick - Tick data represents an individual record of trades ("trade ticks") or quote updates ("quote tick") for an asset. Tick data is instantaneous - it does not have a period.
Data Readers
All data is parsed from disk via Reader() methods. The Reader takes a single line of the file and converts it the appropriate type. i.e. TradeBar.Reader() method is a factory which returns TradeBar objects. When implementing custom data, Readers are used
Other Data Formats
Theoretically LEAN can accept data in any format (database, API or flatfile). However, in practice, we currently have reader implementations written for a flat file system.