Jhonathan Abreu
e29bb2c5e0
* Initial options universe with greeks implementation * Options universe improvements * Address peer review * File based options universe fixes and improvements. - Adjust OptionUniverse start-end times and period. - Adapt unit tests and some algorithms to pass with new options universe selection. * Updated options regression algorithms stats for new universe data * Updated options regression algorithms stats for new universe data * Updated options regression algorithms stats for new universe data * Updated options regression algorithms stats for new universe data * Updated options regression algorithms stats for new universe data * Option chain provider with new options universe * Allow canonical option history requests * Address peer review * Address peer review * Fix symbols parsing in OptionUniverse * Fix universe selection subscriptions start time to not include extended market hours * Minor changes * Minor changes * Peer recommended changes and fixes * Update regression algorithm stats * Update regression algorithms stats and minor fixes * Fix option chain provider history request * Round option indicators values * Added option universe csv header property * Update regression algorithms stats * Update regression algorithms stats * Data fixes and regression algos stats update * Unit test fixes * Minor changes * Option chain handling in live trading data feed * Minor changes * Added processed data provider * Fix thread-safety violation in Slice class * Minor change * Update options filter universe API to use OptionUniverse data Add new filter methods for greeks, IV and open interest * Option filter universe api updates * Add OptionUniverse history regression algorithms * Add regression algorithms for new options filter universe api methods * Added options greeks data and updated regression algorithms * Address peer review * Address peer review * Add more assertions to new options filter api regression algorithms * Minor performance improvement. Reduce greeks binomial model steps to 140 * Minor tests updates * Greeks numerical models performance improvements * Greeks numerical models performance improvements * Revert array pool change for option pricing numerical models * Update default dividend yield provider depending on option type * [TEST] * Add helper method con calculate time till expiration * Use double in price option numerical models * Implied volatility calculation improvements - Adjust root finding method accuracy as a factor of the option price - Use BSM to get a first guess * Cleanup * Some regression algorithms and unit tests cleanup * Regression tests updates after rebasing from master * Add universe files * Self review and cleanup * Minor regression tests updates after rebase * Fix: set data time zone to same as exchange tz for options universes * Minor change * Minor change * Fix for live trading options universe selection * Keep underlying when aggregating collections in BaseDataCollectionAggregatorEnumerator * Update index options regression algorithms stats * Minor change * Address peer review * Memory usage improvements * Minor build fix * Minor changes and test fixes * Cache symbols in OptionUniverse * Cleanup * Fix index option creation in OptionUniverse * Use cached underlying SID when parsing from string * Abstract symbols cache to BaseDataCollection * Return actual underlying symbol when mapping decomposing ICO ticker * Address peer review * Minor performance improvements reduce garbage * Limit Symbols and SIDs cache size to help with memory usage * Minor fix in symbols and sid cache cleanup * Build fix * Lazily parse greeks on individual access * Cleanup and tests * Address peer review * Minor greeks fix --------- Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
LEAN Data Formats
Introduction
From the beginning, LEAN has strived to use an open, human-readable data format - independent of any specific database or file format. From this core philosophy, we built LEAN to read its financial data from flat files on disk. Data compression is done in zip format, and all individual files are CSV or JSON.
When there is no activity for a security, the price is omitted from the file. Only new ticks and price changes are recorded.
File Data Format
Although we strive to make all data formats identical, it is often impossible. Below are links to dedicated documentation on the file format of the data in each asset type:
Equity | Forex | Options | Futures | Crypto
Folder Structure
Data files are separated and nested in a few predictable layers:
-
Tick, Second and Minute Financial Data:
/data/securityType/marketName/resolution/ticker/date_tradeType.zip -
Hour, Daily Financial Data:
/data/securityType/marketName/resolution/ticker.zip
The marketName value is used to separate different tradable assets with the same ticker. E.g. EURUSD is traded on multiple brokerages all with slightly different prices.
Core Data Types
LEAN has a few core data types represented in all the asset classes we support. Below are links to their implementation in LEAN.
-
TradeBar - TradeBar represents trade ticks of assets consolidated for a period. TradeBar file format is slightly different for high resolution (second, minute) and low resolution (daily, hour).
-
QuoteBar - QuoteBar represents top of book quote data consolidated over a period of time (bid and ask bar).
-
Tick - Tick data represents an individual record of trades ("trade ticks") or quote updates ("quote tick") for an asset. Tick data is instantaneous - it does not have a period.
Data Readers
All data is parsed from disk via Reader() methods. The Reader takes a single line of the file and converts it the appropriate type. i.e. TradeBar.Reader() method is a factory which returns TradeBar objects. When implementing custom data, Readers are used
Other Data Formats
Theoretically LEAN can accept data in any format (database, API or flatfile). However, in practice, we currently have reader implementations written for a flat file system.