Gerardo Salazar
82c9b6ccb7
Build & Test Lean / build (push) Has been cancelled
* Adds processed data directory to read price data from
* Make coarse universe generator look at data directory before failing to find daily data
* Set coarse generator output of missing daily file to debug log
* Add CoarseUniverseGenerator logs
* Fixes 100 nanosecond increment lookahead bias when parsing large numbers
* Whenever we parse a number that is has precision greater than
DateTime ticks (sub-100 nanoseconds), if we have nanoseconds
between [0, 1000), excluding numbers divisible by 100,
we will have leftover nanoseconds between [0, 100) nanoseconds, but
they won't be factored in to the DateTime calculation, since casting
to `long` only takes the integer component of the number, so we lose
the extra nanoseconds that came with the decimal, and time is set to
the "floored" value without those nanoseconds.
Since .NET `DateTime` type has a limitation of only being able
to represent time in increments of 100 nanoseconds, by not
considering the sub-100 nanoseconds, we introduce a look-ahead
bias of at most 100 nanoseconds/1 tick
* Misc adjustment to make method use `decimal` instead of `double`
for increased precision when parsing large numbers
* Changes CoinAPI data converter to support processing raw files in original directory structure and file name
* Removes Market requirement from CoinAPI data converter
* Remove timeout on decompression of raw AlgoSeek futures data
* Updates SEC downloader to use HttpClient where requests were failing
* For some unknown reason, valid requests to a valid URL were
failing when using WebClient. Changing our requester to
HttpClient fixes the issue, and enables us to leverage
async capabilities where applicable.
* Added fault tolerance to index file downloads, including a
rate limit in case we've been rate limited
* Further refactoring; catches 429 errors, adds missing rategate calls
* Replace all usage of WebClient, force retry for all failures
* Adds optional config value for Benzinga News API key in downloader
* Modifies Estimize Downloader api config name and fixes directory not found bug
* Refactor Estimize to speed up processing time
* Adds ticker limits if desired
* Misc. bug fixes, performance improvements, code cleanup
* Remove debug log statements leftover from previous commit
* Add support for non-tick Index resolutions in LeanDataWriter
* Empty commit
* Empty commit
* Empty commit
* Empty commit
* Empty commit
* Empty commit
* Lower requests/second for SEC downloader, add missing rategate call
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
Lean Data ToolBox
Lean Home | Documentation | Download Lean
Introduction
Lean Engine is an open-source algorithmic trading engine built for easy strategy research, backtesting and live trading. We integrate with common data providers and brokerages so you can quickly deploy algorithmic trading strategies.
The ToolBox project is a command line program which wraps over 15 tools.
Usage
Each tool requires a different set of parameters, the only required argument is '--app=', which defines the target tool and is case insensitive.
Help information is available using the '--help' parameter.
Example: --app=YahooDownloader --tickers=SPY,AAPL --resolution=Daily --from-date=yyyyMMdd-HH:mm:ss --to-date=yyyyMMdd-HH:mm:ss
Available downloaders
- '--app='
- GDAXDownloader or GDAXDL
- CryptoiqDownloader or CDL
- DukascopyDownloader or DDL
- FxcmDownloader or FDL
- FxcmVolumeDownload or FVDL
- IBDownloader or IBDL
- KrakenDownloader or KDL
- OandaDownloader or ODL
- QuandlBitfinexDownloader or QBDL
- YahooDownloader or YDL
- IEXDownloader or IEXDL
- BitfinexDownloader or BFXDL
- BinanceDownloader or MBXDL
- PolygonDownloader or PDL
- '--from-date=yyyyMMdd-HH:mm:ss' required
- '--tickers=SPY,AAPL,etc' required, except for QuandlBitfinexDownloader (QBDL)
- '--resolution=Tick/Second/Minute/Hour/Daily/All' required, except for QuandlBitfinexDownloader (QBDL), CryptoiqDownloader (CDL). Case sensitive. Not all downloaders support all resolutions, send empty for more information.
- '--to-date=yyyyMMdd-HH:mm:ss' optional. If not provided 'DateTime.UtcNow' will be used
Available Converters
- '--app='
- AlgoSeekFuturesConverter or ASFC
- '--date=yyyyMMdd' reference date.
- AlgoSeekOptionsConverter or ASOC
- '--date=yyyyMMdd' reference date.
- CoinApiDataConverter or CADC
- '--source-dir=' path to the raw CoinAPI data.
- IVolatilityEquityConverter or IVEC
- '--source-dir=' source archived IVolatility data.
- '--source-meta-dir=' source archived IVolatility meta data.
- '--destination-dir=' directory where Lean Data is located "Lean/Data".
- '--resolution=Minute/Hour/Daily' resolution of your IVolatility data. Case insensitive.
- KaikoDataConverter or KDC
- '--market=' the exchange the data represents.
- '--tick-type=Quote/Trade' the tick type being processed. Case insensitive.
- '--source-dir=' path to the raw Kaiko data.
- NseMarketDataConverter or NMDC
- '--source-dir=' source directory of unzipped NSE data.
- '--destination-dir=' directory where Lean Data is located "Lean/Data".
- QuantQuoteConverter or QQC
- '--source-dir=' directory where your QuantQuote order is extracted.
- '--destination-dir=' directory where Lean Data is located "Lean/Data".
- '--resolution=' resolution of the QuantQuote data.
- AlgoSeekFuturesConverter or ASFC
Other tools
- '--app='
- CoarseUniverseGenerator or CUG