* Allow LeanDataWriter to append to zip data files
* Use the data directory provided to the writer instead of the global value
* Disregard the time-portion of an input date
* Overwrite zip entries when creating futures data files
* Minor tweak and adding unit test
Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
Update test to the new crypto and equity subscriptions rules:
- Only consolidates trades
- Low resolution data are only trades.
Update and fix tests
Add missing minute sample files
Update Regression algorithms statistics
Crypto and Equities will consolidate trades by default
Daily and hourly resolution will return only trades for crypto and equities.
Update equity TAQ regression test
Add minute sample files
Updating regression algorithms
Update SpotMarket test cases
Add regression test for equity trades and quotes
- History request.
- Trades and quotes pumped into OnData.
- Subscriptions are added correctly.
Add sample data
Checks low resolution only subscribes to trade bars
The implementation here was stripping out the directories and unzipping to
a flat structure. This is not required and was breaking python algorithms by
messing up their folder based imports. Also, all consumers currently utilize
the return value which is a full list of the unzipped files including the sub
directories, so no changes were needed elsewhere.
Refactored IDataProvider interface to return stream. The IDataProvider
Fetch method now only takes a key. The IDataProvider
has been reshuffled to be at the bottom of the LeanDataStack. It provides
data to the rest of the Lean stack. The default implementation of IDataProvider reads
data from disc.
All IDataCacheProviders now have constructors which take IDataProviders
and use them to find data on disc.
Renamed DataCacheProvider to ZipDataCacheProvider
Added comments to IDataProvider and it's implementations
Added comments to IDataCacheProvider and it's implementations
The replaced zip implementation had issues in high performance,
multi-threaded scenarios. The exact cause of the issue was not
determined, but invalid zip were being generated when using this
method directly.
The replacement uses the built in .NET zip archive implementation
which should (in theory) have better cross-platform characteristics
than 3rd party implementations.
The unit test was also update as it made incorrect usage of the
Compression.Unzip routine. It has been replaced with a more reliable
method that unzips the first entry and returns a stream reader.
michael-sena