- Deleting `ParallelRunnerController` and `ParallelRunnerWorker`.
Replacing them for `Tasks`. The consumer, `EnqueueableEnumerator` will
directly spin up a new producer Task when he is running low on items.
- Moving `Splits`, `Dividends`, `Mappings`, `Delistings`, from the
`SubscriptionDataReader` into there own enumerator, inheriting from
new `CorporateEventBaseEnumerator`
- Adding new helper `static class CorporateEventEnumeratorFactory` to
create the new enumerators
- Adding unit tests for new `CorporateEventBaseEnumerator`
- Adding some extra checks to `HourSplitRegressionAlgorithm` and
`DelistingEventsAlgorithm`
- Removing unneeded `CashBook` instance to create a new `TimeSlice`
- Adding new `TimeSliceFactory`, an instance base class that will
provide methods to create a new `TimeSlice`. Will own the `DateTimeZone`
property.
- Adding new `ISecurityPrice` and `IOptionPrice` that will provide a
reduced interface for accessing price properties and methods used when
creating a new `TimeSlice`
- The class `Subscription` will internally track each `Universe`
`SubscriptionRequest` added or removed
- Adding regression test in which two different `Universe` request the
same `SubscriptionDataConfig` and one of them removes/adds it in a
toggle fashion (fails on current master)
- `UniverseSelection` pending removals will also be tracked by
`Universe`
- `UniverseDecorator` will overwrite the `Universe` member of
`SubscriptionsRequests` at `GetSubscriptionRequests()`. This is due to
`this != this,Universe`
- Adding `Subscription` unit tests covering expected behavior
- Extracting pending removals logic from `UniverseSelection` class into
a new helper class `PendingRemovalsManager`. This new class will keep
track of the `universes` requesting to remove a security. Adding unit tests
- Completly move `DataManager` in front of `DataFeed`. Specifically
`AddSubscription()` and `RemoveSubscription()` implementations. Also
removing IDataFeed.Subscriptions
- Creating new `Synchronizer` which will consolidate and combine `TimeSlice`
streaming for both live and backtests modes. Will live in front of the
`DataManager`
- `SubscriptionSynchronizer` will be the `FrontierTimeProvider` exposed,
and owned, by the new `Synchronizer`
- Updating existing tests
Security instances will require private access to this value in order to
compute close profit.
NOTE: The extent of these changes for simply adding a constructor argument
insinuates that we're missing an abstraction to manage the construction of
these objects, such as a factor object for Security. This will need some
careful TLC in the near future.
- Removing LiveTradingDataFeed bridge. LTDF will yield return Slices
directly from the enumerator, as done in the FSDF. The objective is for
both to share the same logic here and for it to be extracted in a next
PR
- Removing now unnecessary DataFeed running thread
- Adding if (_isStopping) { break; } statement for BaseDataExchange.
This is to avoid the thread to continue looping over the enumerators when stopping. Impacts when there are many securities.
Adds the concept of universe disposal which is requested by an algorithm
through invocation of UniverseManager.Remove, which is invoked via
algorithm.RemoveSecurity. This instructs the data feed that the algorithm
has requested to completely remove the universe and any child subscriptions
from the feed. Security changes are fired for all removed securities.
Internal forex currency conversion feeds are currently added without
a corresponding universe which leads to throwing a null reference exception
when the subscription gets removed at the end of the backtest.
If we pull data and on the same time step that security gets removed,
we can still get that data in OnData(Slice) even though it was removed.
This change filters out removed securities by tracking a reference to
the subscription's disposed flag. Another change was made to wait until
the end of the time step to dispose of subscriptions.
Add IDataFeed.GetSubscription(SubscriptionDataConfig)
Provides a common abstraction for UserDefinedUniverse and the
yet-to-be-impemented ScheduledUniverse. Categorized by the fact
that they don't operate on data, but instead a fixed schedule and
any data required is fetched through external means, such as a
web request.
A large percentage of synchronization time is spent performing time zone conversions.
Previously, it was possible that we may compute the same tz conversion multiple times
on the same piece of data until it's emitted. This change moves the time zone conversion
logic onto the data feed thread right before it puts the data into enqueueable enumerator.
In addition, we're not performing the clone operation at the same moment. This removes
unnecessary computation while moving the necessary computation to the data feed thread to
lighten the load on the synchronization/algorithm thread.
Provides an abstraction point allowing us to easily substitute in a different
synchronization algorithm. This isn't 100% complete as I'm hopefully able to
remove the universe selection dependency from the synchronization implementation.
The concept of subscription frontier is only relevant to the subscription synchronizer.
This change removes the back and forth communication of the frontier data between the
data feed component and the synchronizer component
This caused a change in the CoarseUniverseTop5DollarVolumeAlgorithm. The changed values are
due to the subscriptions now correctly stopping before the turn of the new year. The exact
same number of data points are emitted and the same trades are made, but due to the subscription
stopping slightly sooner (correctly), we lose a benchmark sample which caused a slightly different
beta to be computed (along with dependent statistics).
This is in preparation for the synchronization interface. The synchronization implementation
will be in charge of managing the algorithm manager's time frontier.
Changing the default time zone after SetStartDate is set ends up with the algorithm's
UtcTime being incorrect. This can lead to incorrect start times in the data feed as
well
Previously the number of symbols that could be selected was limited by estimates of RAM usage calculated using the resolution of subscription, now it will only be limited by the physical RAM available.
This was causing unnecessary data processing with algorithms using universe selection, so we should expect some improvements in speed and memory usage.
Another less common issue fixed in this PR is unexpected price spikes on symbols with splits.
This fix only affects backtesting, no change in live trading.
Refactored IDataProvider interface to return stream. The IDataProvider
Fetch method now only takes a key. The IDataProvider
has been reshuffled to be at the bottom of the LeanDataStack. It provides
data to the rest of the Lean stack. The default implementation of IDataProvider reads
data from disc.
All IDataCacheProviders now have constructors which take IDataProviders
and use them to find data on disc.
Renamed DataCacheProvider to ZipDataCacheProvider
Added comments to IDataProvider and it's implementations
Added comments to IDataCacheProvider and it's implementations
The DefaultFileProvider is now configured as by default as the file-provider in config.json and passed in the the engine through the leanAlgorithmHandlers object.
The FileProvider is now passed into the TextSubscriptionDataSourceReader and ZipEntryNameSubscriptionDataSourceReader. Most of the changes in this commit are just getting the FileProvider to those two methods.
Martin Molinero