Stefano Raggi
934128cfa0
* Binance Brokerage skeleton * Market hours * Implement Symbol Mapper - known symbols available on /api/v1/exchangeInfo - fiat currencies are pegged * Implement GetCashBalance * Implement GetAccountHoldings - there are no pre-existing currency swaps - cash balances are pulled and stored in the cashbook * Implement GetOpenOrders * Manage orders: PlaceOrder * Manage orders: UpdateOrder Update operation is not supported * Manage orders: CancelOrder * Messaging: order book * Messaging: trades * Messaging: combine streams - connect to fake /ws/open channel on init - case by channel name, but not event type * Messaging: order depth updates - ticker symbol is not enough as it pushes updates only once a second, this would be a very incomplete data stream - fetch ticker snapshot if lastUpdateId == 0 - follow Binance instructions for keeping local orderbook fresh * Messaging: user data streaming - Request userDataStream endpoint to get listenKey - keep listenkey alive - handle order close event - handle order fill event * DataDownloader: get history - we can aggregate minute candles for higher resolutions * fix data stream * Tests: FeeModel tests * Tests: base brokerage tests * Tests: download history * Tests: symbol mapper * Support StopLimit andd StopMarket orders * StopMarket orders disabled Take profit and Stop loss orders are not supported for any symbols (tested with BTCUSDT, ETHUSDT) * Tests: StopLimit order * Tests: crypto parsing * Reissue user data listen key * comment custom currency limitation * rework websocket connections * implement delayed subscription * adapt ignore message * add license banner * use better suited exception type * avoid message double parsing * support custom fee values * extract BinanceApiClient to manage the request/response between lean and binance * use api events to invoke brokerage events * do not allow to terminate session if it wasn't allocated. * update binance exchange info * tool to add or update binance exchange info * ExchangeInfo basic test * Rebase + Resharp * Binance brokerage updates - Fix sign bug in sell order fills - Fix bug in GetHistory - Remove duplicate symbol from symbol properties db * Remove unused code * Revert removal of account currency check * Update symbols properties database * Address review * Address review - Upgrade API endpoints from v1 to v3 - Updated sub/unsub for new subscription manager - Subscribe best bid/ask quotes instead of full order book - Added handling of websocket error messages - Cleanup + refactor * Update symbol properties database * Remove list from symbol mapper * Fix symbol mapper tests * Address review - Fix resubscribe after reconnect - Fix quote tick edge case * Fix EnsureCurrencyDataFeed for non-tradeable currencies * Fix check in EnsureCurrencyDataFeed * Reuse base class subscribe on reconnect Co-authored-by: Adalyat Nazirov <aenazirov@gmail.com> Co-authored-by: Martin-Molinero <martin@quantconnect.com>
LEAN Data Formats
Introduction
From the beginning LEAN strived to use an open, human readible data format - independent of any specific database or file format. From this core philosophy we built LEAN to read its financial data from flat files on disk. Data compression is done in zip format; and all individual files are CSV or JSON.
When there is no activity for a security, the price is omitted to the file. Only new ticks, and price changes are recorded.
File Data Format
Although we strive to make all data formats identical it is often not possible. Below are links to dedicated documentation on the file format of the data in each asset type:
Equity | Forex | Options | Futures | Crypto
Folder Structure
Data files are separated and nested in a few predictable layers:
-
Tick, Second and Minute Financial Data:
/data/securityType/marketName/resolution/ticker/date_tradeType.zip -
Hour, Daily Financial Data:
/data/securityType/marketName/resolution/ticker.zip
The market value is used to separate different tradable assets with the same ticker. E.g. EURUSD is traded on multiple brokerages all with slightly different prices.
Core Data Types
LEAN has a few core data types which are represented in all the asset classes we support. Below are links to their implementation in LEAN.
-
TradeBar - TradeBar represents trade ticks of assets consolidated for a period. TradeBar file format is slightly different for high resolution (second, minute) and low resolution (daily, hour).
-
QuoteBar - QuoteBar represents top of book quote data consolidated over a period of time (bid and ask bar).
-
Tick - Tick data represents an individual record of trades ("trade ticks") or quote updates ("quote tick") for an asset. Tick data is instantaneous - it does not have a period.
Data Readers
All data is parsed from disk via Reader() methods. The Reader takes a single line of the file and converts it the appropriate type. i.e. TradeBar.Reader() method is a factory which returns TradeBar objects. When implementing custom data Readers are used
Other Data Formats
Theoretically LEAN can accept data in any format (database, API or flatfile). However practically we currently have reader implementations written for a flat file system.