* Calculate both raw and adjuasted prices for backtesting
* disable second price factoring
* move and reuse method
* test coverage for new methods
* reuse scaling method
* reuse subscriptionData.Create method
* removed unused code
* regression test
* switch to aapl
* fix regression test output
* more asserts
* fix comments - reduce shortcuts and abbrevation
* more comments
* merge parameters
* reduce number of getting price factors
* fix tests
* fix tests
* fix regression tests
* calculate TotalReturn on demand
* include TotalReturn calculations
* perf tuning
* more unit tests for SubscriptionData.Create
* simplify things - store and return only raw and precalculated data
* fix regression tests; change it back
* factor equals 1 for Raw data
* small changes
* follow code style
* implement backward compatibility
* Add internal subscription manager
- Add InternalSubscriptionManager that will handle internal
Subscription. Replaces the realtime updates
- Fix thread race condition in the TimeTriggeredUniverseSubscription, we
have one thread injecting data points, the main algorithm thread, and
the base exchange is pulling from it
- Fixes for FakeDataQueue
- Adding unit tests
* Address reviews and fixes
- Internal subscription will use extended market hours
- Only sample charts accordingly
- Get api-url once
- Adding `LazyToUpper()` implementation, that will avoid the call to
`ToUpper` if the string is already upper.
- Reduce the timezone conversions at `Time.EachTradeableDayInTimeZone`
- `TotalPortfolioValue` will iterate over all securities once
- Adding new `SecurityIdentifier` cache, significant impact for
algorithms using coarse/fine data
Since we're creating a new instance of the SubscriptionDataConfig, any new
consolidators added afterwards (includes incdicators) won't receive pricing
updates because they'll be on the wrong config. They would be attached to the
config that live in SubscriptionManager.Subscriptions
In backtesting with universe selection, when symbols are be added and removed the next day, we have experienced slightly different backtesting results for the same algorithm. This was caused by invalid factors applied to equities due to the sharing of a SubscriptionDataConfig instance between two threads.
The issue was solved by calling the SubscriptionDataConfig copy constructor in Universe.GetSubscriptionRequests.
Previously Tick resolution subscriptions only received ticks with TickType.Trade, now TickType.Quote and TickType.OpenInterest are received as well.
This PR replaces PR #1065
Previously, DataNormalizationMode in history requests was always being set to the Adjusted default value, so SetDataNormalizationMode calls were not being respected in SubscriptionDataReaderHistoryProvider.GetHistory and warmup.
Subscriptions can now be explicit as to whether or not a SubscriptionFilterEnumerator is to be applied.
Regular security price data subscriptions get filtered (user/market-hours)
Universe subscriptions don't get filtered (even if a subscription of equity price data, such as options underlying feeds)
This places all security subscriptions within a universes
Subscriptions generated via calls to AddSecurity are place in a UserDefinedUniverse
UserDefinedUniverse will serve up a predetermined list of symbols on a requested interval
LiveTradingDataFeed - reworked custom enumerators to use RefreshEnumerator as wrapper for rate limitting
Add dynamic data flag to config
Remove dynamic data flag from Security
Rename dynamically loaded flag to IsCustomData
Remove Security from SubscriptionDataReader ctor
This includes updating all usages of symbol as a security identifier to use the new type.
The type includes a unique field, SID, as well as the current ticker's value. This allows
for consistent addressability while also allowing the ticker to evolve over time with the
mapping changes.
Effort was made to maintain compile and runtime backwards compatibility.
Time sync:
* Data feeds are required to time sync in UTC time
* TimeSlice.Time is now in UTC
IAlgorithm
* Time is now exclusively the algorithm's local time zone
* Added UtcTime
* SetDateTime( DateTime ) accepts a UTC time and is internally converted
SubscriptionDataConfig
* Adds market and time zone as required ctor parameters
SecurityExchange
* Now passes most calls directly through to SecurityExchangeHours class
SecurityExchangeHours
* Holds market hours for each day of week (LocalMarketHours)
* Talks in terms of local times in the SecurityExchangeHours.TimeZone time zone
Data/market-hours/
* New data folder to hold market hour information
* Includes market-hours-database.csv to hold market hours per market/symbol/security (see doc in file)
* Includes holidays-usa.csv to hold holidays for 'usa' market
+ The holiday files follow the pattern 'holidays-*.csv' where * is the market
TimeKeeper
* Receives updates in UTC time
* Passes that to LocalTimeKeeper's who lazily evaluate the time in their respective time zones
* Eventually this can grow to be the sole source of time in the algorithm's scope
MISC:
* Fixes exception thrown when exiting LiveTradingDataFeed
* Fixes exception thrown when exiting FileSystemDataFeed
* Fixes exception thrown when exiting StatusPing
* Simplify FillForwardEnumerator logic with GetNextMarketOpen
* Adds many time zones, see TimeZones.cs
Updates SubscriptionDataReader to use IStreamReader
Adds FactorFile and MapFile helpers
Adds support for Raw, Adjusted, and TotalReturn data normalization and portfolio application
Changed system to use Margin Models, removing leverage wherever possible allowing for dynamic margin models.
Created first portfolio fill model for FOREX which treats it as a currency swap through the cashbook instead of a tradable asset.
Updated the Securities/SecurityPortfolioManager to use cashbook value + holdings removing FOREX virtual positions.
Created a new benchmark system for comparing quantconnect builds.
Adalyat Nazirov