- Refactor ZipDataCacheProvider to avoid value types thread issues, it
will now use a timer. Add missing dispose calls.
- Synchronizer won't share the `SubscriptionFrontierTimeProvider`
instance since it's not thread safe and shouldn't be called
- Fix BacktestingResultHandler `_daysProcessed` thread safety
- Fixing `RealTimeScheduleEventService` thread safety
Updates usages of GetBaseDataInstance that have a config in scope to use this
new method. This method properly assigns the symbol to the newly created base
data instance. This enables some of the new flag methods to work as expected.
Updated other poor usages where we weren't setting the symbol property.
See #3618
The convention throughout the documentation and source code is to either
return null or a new, uninitialized object to indicate that the reader method
was unable to produce an instance from the previous content. It appears over
time that these filters were forgotten within the data enumerator stacks.
Refactored IDataProvider interface to return stream. The IDataProvider
Fetch method now only takes a key. The IDataProvider
has been reshuffled to be at the bottom of the LeanDataStack. It provides
data to the rest of the Lean stack. The default implementation of IDataProvider reads
data from disc.
All IDataCacheProviders now have constructors which take IDataProviders
and use them to find data on disc.
Renamed DataCacheProvider to ZipDataCacheProvider
Added comments to IDataProvider and it's implementations
Added comments to IDataCacheProvider and it's implementations
Fixed bug where DataCacheProvider was returning a stream without closing
the stream. Also, ZipFiles returned from ZipFileCacheProvider are
effectively disposed.
Removed date field from IDataCacheProvider interface
This interface was built around the existing implementation of the
DataCacheProvider. It's purpose is to define a method for caching data
files. Currently, most data files are either read from disc or retrieved
remotely. The IDataChceProvider returns returns a Stream and can handle
many types of data. The IDataFileCacheProvider is needed because keeping data in memory will improve performance and overcome certain File read/write limitations (such as sharing violations) currently experienced by Lean.
There are two implementations of this interface. The first is the
SingleZipEntryDataCache. This is the default implementation. It does not
cache and returns the first entry found within a ZipFile. The second is
the ZipEntryCacheProvider. This is a rework of the existing
DataCachePrivider. It caches ZipFiles for up to 10 seconds. It can
return specific entries for option and future ZipFiles. Otherwise, it
returns the first entry in a ZipFile.
Added IDataFileCacheProvider as parameter to IHistoryProvider.Initialize()
The IDataFileCacheProvider is very useful for the IHistoryProvider in that it can eliminate the need for history request to touch disc. This can greatly improve peerformance and eliminate disc bugs associated with disc read/writes. To minimize changes throughtout Lean, the default value for the IDataFileCacheProvider is null in the IHistoryProvider.Initialize method.
IDataCacheProviders are passed down the Lean stack. Each instance of a Subscription Enumerator factory decides what cache it
will use. In other words, the IDataCacheProvider is not configurable from
config.json. The IDataCacheProvider is passwed down the stack form the
Enumerator factory to the IStreamReader where it is used to retrieve data.
Added useful helper method to Unizp files into a Stream.
The DefaultFileProvider is now configured as by default as the file-provider in config.json and passed in the the engine through the leanAlgorithmHandlers object.
The FileProvider is now passed into the TextSubscriptionDataSourceReader and ZipEntryNameSubscriptionDataSourceReader. Most of the changes in this commit are just getting the FileProvider to those two methods.
Martin Molinero