Build & Test Lean / build (push) Has been cancelled
* Adds processed data directory to read price data from
* Make coarse universe generator look at data directory before failing to find daily data
* Set coarse generator output of missing daily file to debug log
* Add CoarseUniverseGenerator logs
* Fixes 100 nanosecond increment lookahead bias when parsing large numbers
* Whenever we parse a number that is has precision greater than
DateTime ticks (sub-100 nanoseconds), if we have nanoseconds
between [0, 1000), excluding numbers divisible by 100,
we will have leftover nanoseconds between [0, 100) nanoseconds, but
they won't be factored in to the DateTime calculation, since casting
to `long` only takes the integer component of the number, so we lose
the extra nanoseconds that came with the decimal, and time is set to
the "floored" value without those nanoseconds.
Since .NET `DateTime` type has a limitation of only being able
to represent time in increments of 100 nanoseconds, by not
considering the sub-100 nanoseconds, we introduce a look-ahead
bias of at most 100 nanoseconds/1 tick
* Misc adjustment to make method use `decimal` instead of `double`
for increased precision when parsing large numbers
* Changes CoinAPI data converter to support processing raw files in original directory structure and file name
* Removes Market requirement from CoinAPI data converter
* Remove timeout on decompression of raw AlgoSeek futures data
* Updates SEC downloader to use HttpClient where requests were failing
* For some unknown reason, valid requests to a valid URL were
failing when using WebClient. Changing our requester to
HttpClient fixes the issue, and enables us to leverage
async capabilities where applicable.
* Added fault tolerance to index file downloads, including a
rate limit in case we've been rate limited
* Further refactoring; catches 429 errors, adds missing rategate calls
* Replace all usage of WebClient, force retry for all failures
* Adds optional config value for Benzinga News API key in downloader
* Modifies Estimize Downloader api config name and fixes directory not found bug
* Refactor Estimize to speed up processing time
* Adds ticker limits if desired
* Misc. bug fixes, performance improvements, code cleanup
* Remove debug log statements leftover from previous commit
* Add support for non-tick Index resolutions in LeanDataWriter
* Empty commit
* Empty commit
* Empty commit
* Empty commit
* Empty commit
* Empty commit
* Lower requests/second for SEC downloader, add missing rategate call
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Removes SevenZipSharp library from ToolBox
- Library removal required for update to .NET 5.0 since it's not
compatible with it and Linux is unsupported.
* Adds new extract 7z functionality to Compression project
* Refactors AlgoSeekFuturesConverter 7z extract
* Refactors DukascopyDataDownloader 7z extract
* Addresses review: code cleanup + exception type change on timeout
* Removes not needed stream of data in Dukascopy downloader
* Makes output directory non-optional
* Allow LeanDataWriter to append to zip data files
* Use the data directory provided to the writer instead of the global value
* Disregard the time-portion of an input date
* Overwrite zip entries when creating futures data files
* Minor tweak and adding unit test
Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
The implementation here was stripping out the directories and unzipping to
a flat structure. This is not required and was breaking python algorithms by
messing up their folder based imports. Also, all consumers currently utilize
the return value which is a full list of the unzipped files including the sub
directories, so no changes were needed elsewhere.
Refactored IDataProvider interface to return stream. The IDataProvider
Fetch method now only takes a key. The IDataProvider
has been reshuffled to be at the bottom of the LeanDataStack. It provides
data to the rest of the Lean stack. The default implementation of IDataProvider reads
data from disc.
All IDataCacheProviders now have constructors which take IDataProviders
and use them to find data on disc.
Renamed DataCacheProvider to ZipDataCacheProvider
Added comments to IDataProvider and it's implementations
Added comments to IDataCacheProvider and it's implementations
Fixed bug where DataCacheProvider was returning a stream without closing
the stream. Also, ZipFiles returned from ZipFileCacheProvider are
effectively disposed.
Removed date field from IDataCacheProvider interface
This interface was built around the existing implementation of the
DataCacheProvider. It's purpose is to define a method for caching data
files. Currently, most data files are either read from disc or retrieved
remotely. The IDataChceProvider returns returns a Stream and can handle
many types of data. The IDataFileCacheProvider is needed because keeping data in memory will improve performance and overcome certain File read/write limitations (such as sharing violations) currently experienced by Lean.
There are two implementations of this interface. The first is the
SingleZipEntryDataCache. This is the default implementation. It does not
cache and returns the first entry found within a ZipFile. The second is
the ZipEntryCacheProvider. This is a rework of the existing
DataCachePrivider. It caches ZipFiles for up to 10 seconds. It can
return specific entries for option and future ZipFiles. Otherwise, it
returns the first entry in a ZipFile.
Added IDataFileCacheProvider as parameter to IHistoryProvider.Initialize()
The IDataFileCacheProvider is very useful for the IHistoryProvider in that it can eliminate the need for history request to touch disc. This can greatly improve peerformance and eliminate disc bugs associated with disc read/writes. To minimize changes throughtout Lean, the default value for the IDataFileCacheProvider is null in the IHistoryProvider.Initialize method.
IDataCacheProviders are passed down the Lean stack. Each instance of a Subscription Enumerator factory decides what cache it
will use. In other words, the IDataCacheProvider is not configurable from
config.json. The IDataCacheProvider is passwed down the stack form the
Enumerator factory to the IStreamReader where it is used to retrieve data.
Added useful helper method to Unizp files into a Stream.
This new overload allows the caller to specify every piece of data used by the
zip routine, allowing specification of the entry name, destination path, and
whether or not to delete the original file.
The replaced zip implementation had issues in high performance,
multi-threaded scenarios. The exact cause of the issue was not
determined, but invalid zip were being generated when using this
method directly.
The replacement uses the built in .NET zip archive implementation
which should (in theory) have better cross-platform characteristics
than 3rd party implementations.
The unit test was also update as it made incorrect usage of the
Compression.Unzip routine. It has been replaced with a more reliable
method that unzips the first entry and returns a stream reader.
This follows the behavior of logging when the file doesn't exist as well.
This is very common in options trade data, since it's often there are no trades for a contract
Many places in the code used Log.Error(err.Message) or equivalent which
strips out all the really useful information, such as the stack trace
and inner exceptions. Using Log.Error(exception) is the correct way to
log an error as it will correctly write all the message details, also,
by passing the full Exception object we can improve the logging in this
one place and all call sites will automatically benefit from the improvements
Gerardo Salazar