Balamurali Pandranki
c22a538bad
* Indian stock markets & Samco and Zerodha brokerage implementations * Build fixes & Implement multi leg orders (BracketOrder & CoverOrder) * Build fixes * Clean & refactor * Clean up & remove samco brokerage * Clean up & remove samco brokerage * Fix Nifty, BankNifty & Sensex Index futures expiry time functions * Fix Nifty, BankNifty & Sensex Index futures expiry time functions * Fix Futures Expiry Testcases * Fix Futures Expiry Testcases * Refactor Zerodha Symbol Mapper * Refactor Zerodha Symbol Mapper * Add Future symbols to symbol prop db csv * Fix Symbol Mapper context * Fix Market Hours Database * Fix OrderJsonConverter * Add Zerodha AccountBaseCurrency * Add QuantConnect License headers to new source files * cleanup config.json formatting & tick aggregator implementation * cleanup config.json formatting & tick aggregator implementation * Refactor ZerodhaBrokerageModel * Fix * Build Fixes * Refactor * Refactor Brokerage class & remove TextFieldParser * Refactor Brokerage FeeModel * Add ZerodhaOrderProperties * Add Refactor HistoryProvider * Refactor CanExecuteOrder * Refactor SymbolMapper * Refactor market names * Refactor & remove Zerodha subscription manager * Refactor & remove ZerodhaWebSocketChannels * Refactor & remove ZerodhaWebSocketChannels * Refactor ZerodhaBrokerage * Refactor symbol prop db * zerodha update downloader ticker data using symbol name * 1 Zerodha mapper class update to get instruments based on Market. 2 Zerodha Getholdings data fetch fix. * Update market-hours-database.json * Implement basic testcases for ZerodhaBrokerage * Update market-hours-database.json * Update Market.cs * 1 Fix for Zerodha subscriber and unsubscribe 2 User of CSVHelper to read instrument list * Rename Zerodha DataQueueHandler class implementation * Changes related to TradeBar and emit tick * Fix DataQueueUniverseProvider & Handle Timezone conversion in WS EmitQuotes * Emit Order Fixes * DataQueueHandler refactoring and build fix * Update config.json * Removal of IHistoryProvider impplementation * Reverting timezone logic as already taken care by BrokerImplementation using Unix time * Place, modify and cancel order implementation * fix zerodha test cases * Updating get quotes and restricting getHoldings to MIS * Testcase Fixes * Testcase Fixes * Testcase Fixes * Updating emitFillOrder * Add ProductType to ZerodhaOrderProperties * Addition of ZerodhaProduct Type property and test update * Fix for unit tests and minor changes for place and update order * Unit test fix for Zerodha * Addition of the product type and trading segment configs * PR review fixes * Addition of trading segment and product type configs * Nuget Fixes * Fix ZerodhaBrokerage DataQueueHandler * Cleanup OrderTypes & rm global.json * Update UpdateOrderRequest.cs * Removal of custom csvreader * Implementing additional method CanPerformSelection * Add LiveOptionChainProvider * Clean up and add QC license headers * Quick cleanup * Update ZerodhaBrokerage.cs * Fix OrderEvent timezone conversions * use item.Unrealised for Intraday position holdings * Refactor Option chain get instruments calls * Refactor Option chain get instruments calls * Review fixes * Remove whitespaces * Updating options strike price * Undo time stamp change * Optional gethistory * Addition of comments and minor changes * Build Fixes * Add comments for LogType enums * Options fix * Fix json name in CsvInstrument * Update tick generation in DataQueueHandler - Use Timestamp field for both trades and quotes - Fix incorrect bid/ask sizes - Avoid reading depth on each tick to find top quote - Use new Tick constructors - Remove unnecessary locking * Add missing null check in Utils.StringToDate * Order fixes - Include TriggerPending orders in GetOpenOrders - Remove unnecessary invalid check in PlaceOrder - Fix GetOrderPrice and GetOrderTriggerPrice * Zerodha account balance fetch * Removal of Futures and Options * Remove Futures & Options Support * Update ZerodhaBrokerage.cs * Update ZerodhaBrokerage.cs * Add License headers * Add License Headers * Refactor * historical TradeBar & security fee calculation fixes * fix brokerage test case * Update ZerodhaBrokerage.cs * Update ZerodhaBrokerageHistoryProviderTests.cs Co-authored-by: Aman Ray <ray.aman9876@gmail.com> Co-authored-by: Stefano Raggi <stefano.raggi67@gmail.com>
LEAN Data Formats
Introduction
From the beginning LEAN strived to use an open, human readible data format - independent of any specific database or file format. From this core philosophy we built LEAN to read its financial data from flat files on disk. Data compression is done in zip format; and all individual files are CSV or JSON.
When there is no activity for a security, the price is omitted to the file. Only new ticks, and price changes are recorded.
File Data Format
Although we strive to make all data formats identical it is often not possible. Below are links to dedicated documentation on the file format of the data in each asset type:
Equity | Forex | Options | Futures | Crypto
Folder Structure
Data files are separated and nested in a few predictable layers:
-
Tick, Second and Minute Financial Data:
/data/securityType/marketName/resolution/ticker/date_tradeType.zip -
Hour, Daily Financial Data:
/data/securityType/marketName/resolution/ticker.zip
The marketName value is used to separate different tradable assets with the same ticker. E.g. EURUSD is traded on multiple brokerages all with slightly different prices.
Core Data Types
LEAN has a few core data types which are represented in all the asset classes we support. Below are links to their implementation in LEAN.
-
TradeBar - TradeBar represents trade ticks of assets consolidated for a period. TradeBar file format is slightly different for high resolution (second, minute) and low resolution (daily, hour).
-
QuoteBar - QuoteBar represents top of book quote data consolidated over a period of time (bid and ask bar).
-
Tick - Tick data represents an individual record of trades ("trade ticks") or quote updates ("quote tick") for an asset. Tick data is instantaneous - it does not have a period.
Data Readers
All data is parsed from disk via Reader() methods. The Reader takes a single line of the file and converts it the appropriate type. i.e. TradeBar.Reader() method is a factory which returns TradeBar objects. When implementing custom data Readers are used
Other Data Formats
Theoretically LEAN can accept data in any format (database, API or flatfile). However practically we currently have reader implementations written for a flat file system.