Jhonathan Abreu
539011274c
* Support extended market hours in AddFuture() * Support extended market hours in AddFutureContract() * Add C# regression algorithm * Add Python regression algorithm * Add regression algorithm for future contracts * Add regression algorithm checking market hour ranges * Fixed future regression algorithms to use extended market hours * Fixed future regression algorithms to use extended market hours * Fixed future regression algorithms to use extended market hours * Fixed AddFutureOptionContractFromFutureChainRegressionAlgorithm to use extended market hours * Update future market hours to include extended in market hours database * Fixed AddFutureOptionContractDataStreamingRegressionAlgorithm to use extended market hours * Fixed AddFutureOptionContractFromFutureChainRegressionAlgorithm to use extended market hours * Fixed AddFutureContractWithContinuousRegressionAlgorithm to use extended market hours * Fixed BasicTemplateContinuousFutureAlgorithm to use extended market hours * Fixed BasicTemplateFuturesAlgorithm to use extended market hours * Fix BasicTemplateFuturesDailyAlgorithm to use extended market hours * Fixed BasicTemplateFuturesFrameworkAlgorithm to use extended market hours * Fixed BasicTemplateFuturesHistoryAlgorithm to use extended market hours * Fixed ContinuousBackMonthRawFutureRegressionAlgorithm to use extended market hours * Fixed ContinuousFutureBackMonthRegressionAlgorithm to use extended market hours * Fixed ContinuousFutureHistoryRegressionAlgorithm to use extended market hours * Fixed ContinuousFutureLimitIfTouchedOrderRegressionAlgorithm to use extended market hours * Fixed ContinuousFutureRegressionAlgorithm to use extended market hours * Fixed DelistedFutureLiquidateRegressionAlgorithm to use extended market hours * Fixed AutomaticIndicatorWarmupDataTypeRegressionAlgorithm to use extended market hours * Fixed ConsolidateRegressionAlgorithm to use extended market hours * Fixed DelistingFutureOptionRegressionAlgorithm to use extended market hours * Fixed EqualWeightingPortfolioConstructionModelFutureRegressionAlgorithm to use extended market hours * Fixed FutureContractsExtendedMarketHoursRegressionAlgorithm to use extended market hours * Fixed FutureMarketOpenAndCloseRegressionAlgorithm to use extended market hours * Fixed FutureMarketOpenConsolidatorRegressionAlgorithm to use extended market hours * Fixed FutureOptionBuySellCallIntradayRegressionAlgorithm to use extended market hours * Fixed FutureOptionCallITMExpiryRegressionAlgorithm to use extended market hours * Fixed FutureOptionCallITMGreeksExpiryRegressionAlgorithm to use extended market hours * Fixed FutureOptionCallOTMExpiryRegressionAlgorithm to use extended market hours * Fixed FutureOptionDailyRegressionAlgorithm to use extended market hours * Fixed FutureOptionHourlyRegressionAlgorithm to use extended market hours * Fixed FutureOptionMultipleContractsInDifferentContractMonthsWithSameUnderlyingFutureRegressionAlgorithm to use extended market hours * Fixed FutureOptionPutITMExpiryRegressionAlgorithm to use extended market hours * Fixed FutureOptionPutOTMExpiryRegressionAlgorithm to use extended market hours * Fixed FutureOptionShortCallITMExpiryRegressionAlgorithm to use extended market hours * Fixed FutureOptionShortCallOTMExpiryRegressionAlgorithm to use extended market hours * Fixed FutureOptionShortPutITMExpiryRegressionAlgorithm to use extended market hours * Fixed FutureOptionShortPutOTMExpiryRegressionAlgorithm to use extended market hours * Fixed FuturesAndFuturesOptionsExpiryTimeAndLiquidationRegressionAlgorithm to use extended market hours * Fixed FuturesExpiredContractRegression to use extended market hours * Fixed FutureSharingTickerRegressionAlgorithm to use extended market hours * Fixed HistoryWithDifferentContinuousContractDepthOffsetsRegressionAlgorithm to use extended market hours * Fixed HistoryWithDifferentDataMappingModeRegressionAlgorithm to use extended market hours * Fixed HistoryWithDifferentDataNormalizationModeRegressionAlgorithm to use extended market hours * Fixed LimitOrdersAreFilledAfterHoursForFuturesRegressionAlgorithm to use extended market hours * Fixed OpenInterestFuturesRegressionAlgorithm to use extended market hours * Fixed RegisterIndicatorRegressionAlgorithm to use extended market hours * Fixed SetHoldingsFutureRegressionAlgorithm to use extended market hours * Fixed WarmupFutureRegressionAlgorithm to use extended market hours * Fixed AddFutureOptionSingleOptionChainSelectedInUniverseFilterRegressionAlgorithm to use extended market hours * Fixed AlgorithmHistoryTests to use extended market hours for futures * Fixed AlgorithmTradingTests to use extended market hours for futures * Fixed BrokerageSetupHandlerTests to use extended market hours for futures * Fixed TimeRulesTests to use extended market hours for futures * Fixed FutureOptionMarginBuyingPowerModelTests to use extended market hours for futures * Fixed FutureMarginBuyingPowerModelTests to use extended market hours for futures * Fixed FileSystemDataFeedTests to use extended market hours for futures * Fixed QuantBookHistoryTests to use extended market hours for futures * Split BasicTemplateContinuousFutureAlgorithm to have an extended market version * Fixed FutureMarketOpenAndCloseRegressionAlgorithm to use extended market hours * Split BasicTemplateFuturesAlgorithm to have an extended market version * Split BasicTemplateFuturesAlgorithm to have an extended market version * Split BasicTemplateFuturesFrameworkAlgorithm to have an extended market version * Split BasicTemplateFuturesHistoryAlgorithm to have an extended market version * Revert AddFutureContractWithContinuousRegressionAlgorithm * Revert AddFutureOptionContractDataStreamingRegressionAlgorithm and added data * Revert AddFutureOptionContractFromFutureChainRegressionAlgorithm * Revert AddFutureOptionSingleOptionChainSelectedInUniverseFilterRegressionAlgorithm * Revert ConsolidateRegressionAlgorithm * Revert Algorithm.CSharp/ContinuousBackMonthRawFutureRegressionAlgorithm.cs * Revert ContinuousFutureBackMonthRegressionAlgorithm * Revert ContinuousFutureHistoryRegressionAlgorithm * Revert ContinuousFutureLimitIfTouchedOrderRegressionAlgorithm * Revert ContinuousFutureRegressionAlgorithm * Revert Algorithm.CSharp/DelistedFutureLiquidateRegressionAlgorithm.cs * Revert EqualWeightingPortfolioConstructionModelFutureRegressionAlgorithm * Split FutureMarketOpenAndCloseRegressionAlgorithm to have an extended market version * Split FutureMarketOpenConsolidatorRegressionAlgorithm to have an extended market version * Revert FutureOptionBuySellCallIntradayRegressionAlgorithm * Revert FutureOptionCallITMExpiryRegressionAlgorithm * Revert FutureOptionDailyRegressionAlgorithm * Revert FutureOptionPutITMExpiryRegressionAlgorithm * Revert FutureSharingTickerRegressionAlgorithm * Revert FuturesAndFuturesOptionsExpiryTimeAndLiquidationRegressionAlgorithm * Revert FuturesExpiredContractRegression * Revert HistoryWithDifferentContinuousContractDepthOffsetsRegressionAlgorithm * Revert HistoryWithDifferentDataMappingModeRegressionAlgorithm * Revert HistoryWithDifferentDataNormalizationModeRegressionAlgorithm * Revert OpenInterestFuturesRegressionAlgorithm * Revert RegisterIndicatorRegressionAlgorithm * Revert SetHoldingsFutureRegressionAlgorithm * Revert WarmupFutureRegressionAlgorithm * Revert AutomaticIndicatorWarmupDataTypeRegressionAlgorithm * Some cleanup * Address changes request * Address changes request * Add more Class III Milk data to fix DelistingFutureOptionDailyRegressionAlgorithm
LEAN Data Formats
Introduction
From the beginning LEAN strived to use an open, human readable data format - independent of any specific database or file format. From this core philosophy we built LEAN to read its financial data from flat files on disk. Data compression is done in zip format; and all individual files are CSV or JSON.
When there is no activity for a security, the price is omitted to the file. Only new ticks, and price changes are recorded.
File Data Format
Although we strive to make all data formats identical it is often not possible. Below are links to dedicated documentation on the file format of the data in each asset type:
Equity | Forex | Options | Futures | Crypto
Folder Structure
Data files are separated and nested in a few predictable layers:
-
Tick, Second and Minute Financial Data:
/data/securityType/marketName/resolution/ticker/date_tradeType.zip -
Hour, Daily Financial Data:
/data/securityType/marketName/resolution/ticker.zip
The marketName value is used to separate different tradable assets with the same ticker. E.g. EURUSD is traded on multiple brokerages all with slightly different prices.
Core Data Types
LEAN has a few core data types which are represented in all the asset classes we support. Below are links to their implementation in LEAN.
-
TradeBar - TradeBar represents trade ticks of assets consolidated for a period. TradeBar file format is slightly different for high resolution (second, minute) and low resolution (daily, hour).
-
QuoteBar - QuoteBar represents top of book quote data consolidated over a period of time (bid and ask bar).
-
Tick - Tick data represents an individual record of trades ("trade ticks") or quote updates ("quote tick") for an asset. Tick data is instantaneous - it does not have a period.
Data Readers
All data is parsed from disk via Reader() methods. The Reader takes a single line of the file and converts it the appropriate type. i.e. TradeBar.Reader() method is a factory which returns TradeBar objects. When implementing custom data Readers are used
Other Data Formats
Theoretically LEAN can accept data in any format (database, API or flatfile). However practically we currently have reader implementations written for a flat file system.