Gerardo Salazar
4c085ff853
* Add support for Index SecurityType 🚀 (#5364) * Add Index SecurityType 🚀 * Extend SecurityIdentifier & Lean Data classes with Index support * Add Index SecurityType 🚀 * Extend SecurityIdentifier & Lean Data classes with Index support * Fixes * Added index cross basic template demonstration * WIP: Prototype index security type for LEAN as non tradable asset * Re-adds Index entries to MHDB after rebase * First steps to getting Index Options running * Looks at any instance where we pattern match for an option type and replaces it with a generic call to `.IsOption()` for easier extensibility in the future for additional option security types * Adds IndexOption security and misc. classes * Misc. changes, mainly related to any sort of special casing of equity options and made index options take the same path * Enables index options data for backtesting * Adds new index options market hours to MHDB * Misc. bug fixes for index options * WIP: add live support for index options and indexes * Use OptionMarginModel for Index Options because they both use the same calculation for margin requirements * Fixes contract not found errors on SPX index options and SPX index in IB * Turns out index options' last trading day is the day before expiry, which IB was expecting the last trading day. * Add index option test cases (temp) * LiveOptionChainProvider fix, use Symbol vs. ticker * Description updates to regression algorithms * Fixes bug in live trading for indexes and index options * Adds overridable minimum price variation symbol property * Adds variable sized minimum price variation for index options * Adjusts symbol properties for index options * Misc. bug fixes * Fixes option assignment simulation for European options * Updates index options regression algorithms (WIP) * Fixes bug where index option exercise would trade index underlying * Fixes bugs where SecurityType.Index was getting flagged as tradable * Regression algorithms updates and addresses review * Misc. style fixes and refactoring + a few bug fixes * Updates regression algorithms to run without runtime errors * Adds data for regression algos * Sets DefaultOptionStyle on Canonical and support index options * Update regression algos statistics * Removes bad line in regression algorithm causing build to fail * Minor tweaks * Address review add comment about quoteBar parse scale Co-authored-by: Balamurali Pandranki <balamurali@live.com> Co-authored-by: Jared Broad <jaredbroad@gmail.com> Co-authored-by: Martin-Molinero <martin@quantconnect.com>
LEAN Data Formats
Introduction
From the beginning LEAN strived to use an open, human readible data format - independent of any specific database or file format. From this core philosophy we built LEAN to read its financial data from flat files on disk. Data compression is done in zip format; and all individual files are CSV or JSON.
When there is no activity for a security, the price is omitted to the file. Only new ticks, and price changes are recorded.
File Data Format
Although we strive to make all data formats identical it is often not possible. Below are links to dedicated documentation on the file format of the data in each asset type:
Equity | Forex | Options | Futures | Crypto
Folder Structure
Data files are separated and nested in a few predictable layers:
-
Tick, Second and Minute Financial Data:
/data/securityType/marketName/resolution/ticker/date_tradeType.zip -
Hour, Daily Financial Data:
/data/securityType/marketName/resolution/ticker.zip
The marketName value is used to separate different tradable assets with the same ticker. E.g. EURUSD is traded on multiple brokerages all with slightly different prices.
Core Data Types
LEAN has a few core data types which are represented in all the asset classes we support. Below are links to their implementation in LEAN.
-
TradeBar - TradeBar represents trade ticks of assets consolidated for a period. TradeBar file format is slightly different for high resolution (second, minute) and low resolution (daily, hour).
-
QuoteBar - QuoteBar represents top of book quote data consolidated over a period of time (bid and ask bar).
-
Tick - Tick data represents an individual record of trades ("trade ticks") or quote updates ("quote tick") for an asset. Tick data is instantaneous - it does not have a period.
Data Readers
All data is parsed from disk via Reader() methods. The Reader takes a single line of the file and converts it the appropriate type. i.e. TradeBar.Reader() method is a factory which returns TradeBar objects. When implementing custom data Readers are used
Other Data Formats
Theoretically LEAN can accept data in any format (database, API or flatfile). However practically we currently have reader implementations written for a flat file system.