Implements `DataNormalizationMode` field in `UniverseSettings` to enable the settings of a desired `DataNormalizationMode` to the securities that are chosen in Universe Selection.
Since subscriptions are enumerated in non-deterministic order, tick consolidators for multi-tick-type security types (such as Crypto, Future and Option) could end up being added to the wrong subscription, e.g. QuoteBarTickConsolidator added to a tick subscription with TickType.Trade. This was causing the Consolidated event handler to never be called.
The Resolution.Tick case is now handled properly, checking the subscription tick type.
- Adding new ISubscriptionDataConfigBuilder implemented by DataManager, exposed by
SubscriptionManager
- SecurityManager and SubscriptionManager will use new interface.
In a next PR, its intended for it to also be consumed by Universe.GetSubscriptionRequests().
- Moving LookupSubscriptionConfigDataTypes implementation into
DataManager
Refactoring previous commit so DataManager only keeps and receives a
reference to IAlgorithmSettigs, with the objective of reducing tight
coupling
Note: Investigate if IAlgorithmSettings.DataSubscriptionLimit should limit subscriptions or unique securities.
Today its using SubscriptionManagerSubscriptions.Where(x => !x.Symbol.IsCanonical()).DistinctBy(x => x.Symbol.Value).Count() @DataManager
All consolidators now clear the event handlers list when being disposed.
In addition, SubscriptionManager.RemoveConsolidator will now dispose of
the consolidator before returning. This ensures the consolidator and any
downstream indicators that were attached to it can be properly cleaned
by garbage collection.
This allows algorithms to remove consolidators at run time. This is especially
relevant with universe selection where we can add/remove securities at run time
and potentially be creating indicators as well. When the security is removed
from the universe we'll also want to remove any consolidators/indicators that
have been wired up as well.
Previously Tick resolution subscriptions only received ticks with TickType.Trade, now TickType.Quote and TickType.OpenInterest are received as well.
This PR replaces PR #1065
Futures and Options have multiple data types: QuoteBar, TradeBar and OpenInterest.
With these asset types only QuoteBarConsolidator could be used before, TradeBarConsolidator and OpenInterestConsolidator can now be used as well.
This flag will be used to speed python algorithms execution, since it avoids a wrapping operation that is only required when there is custom data in python algorithms.
Subscriptions can now be explicit as to whether or not a SubscriptionFilterEnumerator is to be applied.
Regular security price data subscriptions get filtered (user/market-hours)
Universe subscriptions don't get filtered (even if a subscription of equity price data, such as options underlying feeds)
Add dynamic data flag to config
Remove dynamic data flag from Security
Rename dynamically loaded flag to IsCustomData
Remove Security from SubscriptionDataReader ctor
This includes updating all usages of symbol as a security identifier to use the new type.
The type includes a unique field, SID, as well as the current ticker's value. This allows
for consistent addressability while also allowing the ticker to evolve over time with the
mapping changes.
Effort was made to maintain compile and runtime backwards compatibility.
Time sync:
* Data feeds are required to time sync in UTC time
* TimeSlice.Time is now in UTC
IAlgorithm
* Time is now exclusively the algorithm's local time zone
* Added UtcTime
* SetDateTime( DateTime ) accepts a UTC time and is internally converted
SubscriptionDataConfig
* Adds market and time zone as required ctor parameters
SecurityExchange
* Now passes most calls directly through to SecurityExchangeHours class
SecurityExchangeHours
* Holds market hours for each day of week (LocalMarketHours)
* Talks in terms of local times in the SecurityExchangeHours.TimeZone time zone
Data/market-hours/
* New data folder to hold market hour information
* Includes market-hours-database.csv to hold market hours per market/symbol/security (see doc in file)
* Includes holidays-usa.csv to hold holidays for 'usa' market
+ The holiday files follow the pattern 'holidays-*.csv' where * is the market
TimeKeeper
* Receives updates in UTC time
* Passes that to LocalTimeKeeper's who lazily evaluate the time in their respective time zones
* Eventually this can grow to be the sole source of time in the algorithm's scope
MISC:
* Fixes exception thrown when exiting LiveTradingDataFeed
* Fixes exception thrown when exiting FileSystemDataFeed
* Fixes exception thrown when exiting StatusPing
* Simplify FillForwardEnumerator logic with GetNextMarketOpen
* Adds many time zones, see TimeZones.cs
AlexCatarino