This will change the order in which we enumerate subscriptions so we don't need
to re-sort inside of TimeSlice.Create.
The way we currently build time slices requires that we receive the underlying data
before any derivative data. This change ensures that we enumerate the underlying equity
data before we process any derivative data, such as options.
The list of subscriptions needs to be sorted by TickType when enumerating, but this sorting was previously done in GetEnumerator (called for each data point).
This has been fixed by sorting only when adding or removing subscriptions, gaining a significant backtest speed improvement.
In addition, the storage for subscriptions was simplified to a single ConcurrentDictionary, instead of nested dictionaries.
This method was not removing the symbol dictionary entry when removing the last subscription, allowing the collection to grow excessively over time (especially with universe selection algorithms).
This PR also includes a few minor performance fixes.
An occasional NullReferenceException was thrown when enumerating and concurrently adding or removing items.
A new unit test that always throws this exception was also added.
In the SubscriptionCollection class, subscriptions are stored in nested dictionaries, keyed by Symbol and by SubscriptionDataConfig.
According to MSDN when enumerating: The order in which the items are returned is undefined.
https://msdn.microsoft.com/en-us/library/xfhwa508.aspx
The enumeration ordering for different symbols is not an issue, but the order of configs for the same symbol can cause non-deterministic backtest results (only security types with multiple data types per subscription are affected such as Crypto, Futures, Options). This random behavior was observed with the FractionalQuantityRegressionAlgorithm, so the regression statistics have also been updated.
In this PR we force a fixed priority for data types, which will guarantee that different data points for the same symbol at the same time step will always be emitted in the same order (TradeBar before QuoteBar).
Martin Molinero