Ricardo Andrés Marino Rojas
dcb5f8ee4e
* Update Future-cme-[*] and Future-cme-ES Acoording to `pandas_market_calendars` there were some dates in Future-cme-[*] who wasn't early_closes, so they needed to be removed from there. On the other hand, the early closes list of Future-cme-ES were shifted by 1 hour according to CME webpage. Besides, there were some missing dates. * Update CME Future entries in MHDB * Rebase * nit change * Fix unit tests * Resume after early close/halts * Add missing dates in MHDB and fix bugs in it * Fix bug, add more unit tests and add docs * fix regression algos * address required changes * Update failing regression test stats After debugging the tests it was found they were failing due to the last change on SecurityExchangeHours.IsOpen(). That method wasn't taking into account that even if there is a late open after an early close if the timespan is after the early close but before the late open, the market is still close. * enhance solution * Update and fix bugs in MHDB * Address required changes and update stats * Update stats after rebase * Nit change * Missing update to regression test * Use MHDB instead of USHoliday for Expiration Dates VIX expiry function now relies completely on MHDB. However, it had to be created an entry in MHDB for VIX since there wasn't one for it. CBOE webpage only provided 2023 holidays so only those dates were considered in the Holidays entry in MHDB. Therefore, some unit tests failed so it was necessary to change also the VIX entry in FuturesExpiryFunctionsTestData.xml. * Remove Global.cs/USHolidays class * Use a lazy implementation * First draft of the solution * Use MHDB in FuturesExpiryFunctions.cs * Remove unused class and fix indentation errors * Fix indentation errors * Nit changes * Merge branches 7501 and 7506 * Merge changes in 7501 and 7506 In order to check compatibility between those branches, a new branch was created out of branch 7501 and then it was merged with branch 7506. 2 regression tests and 8 unit tests failed, the regression tests failed on the DataPoint stats. On the other hand, the unit tests failed since the default parameter UseEquityHoliday was removed from FuturesExpirtyUtilityFunctions.AddBusinessDays() and from other methods in the same class too. * Add missing changes * Remove repeated good fridays * Address minor review --------- Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
LEAN Data Formats
Introduction
From the beginning LEAN strived to use an open, human readable data format - independent of any specific database or file format. From this core philosophy we built LEAN to read its financial data from flat files on disk. Data compression is done in zip format; and all individual files are CSV or JSON.
When there is no activity for a security, the price is omitted to the file. Only new ticks, and price changes are recorded.
File Data Format
Although we strive to make all data formats identical it is often not possible. Below are links to dedicated documentation on the file format of the data in each asset type:
Equity | Forex | Options | Futures | Crypto
Folder Structure
Data files are separated and nested in a few predictable layers:
-
Tick, Second and Minute Financial Data:
/data/securityType/marketName/resolution/ticker/date_tradeType.zip -
Hour, Daily Financial Data:
/data/securityType/marketName/resolution/ticker.zip
The marketName value is used to separate different tradable assets with the same ticker. E.g. EURUSD is traded on multiple brokerages all with slightly different prices.
Core Data Types
LEAN has a few core data types which are represented in all the asset classes we support. Below are links to their implementation in LEAN.
-
TradeBar - TradeBar represents trade ticks of assets consolidated for a period. TradeBar file format is slightly different for high resolution (second, minute) and low resolution (daily, hour).
-
QuoteBar - QuoteBar represents top of book quote data consolidated over a period of time (bid and ask bar).
-
Tick - Tick data represents an individual record of trades ("trade ticks") or quote updates ("quote tick") for an asset. Tick data is instantaneous - it does not have a period.
Data Readers
All data is parsed from disk via Reader() methods. The Reader takes a single line of the file and converts it the appropriate type. i.e. TradeBar.Reader() method is a factory which returns TradeBar objects. When implementing custom data Readers are used
Other Data Formats
Theoretically LEAN can accept data in any format (database, API or flatfile). However practically we currently have reader implementations written for a flat file system.