* OptionChain and OptionContract improvements
- QCAlgorithm.AddUniverse will return the added Universe instance.
- Adding new OptionChainedUniverseSelectionModel will monitor a Universe changes
and will spwan new OptionChainUniverse from it's selections. Adding
regression test Py/C#.
- Adding new OptionContractUniverse that will own option contracts and
their underlying symbol. Adding regression test
- Fix double notification for security changes, bug seen in updated
UniverseSelectionRegressionAlgorithm
- Remove UniverseSelection special handling for Option and Future chains
- Fix DataManager not removing SubscriptionDataConfigs for Subscriptions
which finished before being removed from the universe
- Refactor detection of user added Universe so that they do not get
removed after calling the UniverseSelectionModel
* Add check for option underlying price is set
* Address reviews
- Adding python regression algorithm for
`AddOptionContractFromUniverseRegressionAlgorithm`
and `AddOptionContractExpiresRegressionAlgorithm`
- Rename QCAlgorithm new api method to `AddChainedOptionUniverse`
* Fix universe refresh bug
- Fix bug where a universe selection refresh would cause option or
future chain universes from being removed. Adding regression algorithm
reproducing the issue.
* Rename new option universe Algorithm API method
- Rename new option universe Algorith API method from
AddChainedOptionUniverse to AddUniverseOptions
- Rebase and update regression test order hash because of
option expiration message changed
- Replace Enum to string operations on Options enums, since it's
expensive
- Replace Enum IsDefined check since it uses reflection
- Improve OptionFilterUniverse linq operations
- Improve performance of SecurityIdentifier properties to avoid having
to extract them from properties always
- Avoid calling ToList for already list data collections (it creates a
copy)
- Avoid iterating over securities for which we have no holdings when
calculating TPV or MarginUsed
- Adding new ReferenceWrapper for structs, value types, to avoid thread
race conditions while reading and writting. In C# reference type
assignments are atomic, so it allows us avoid using locks.
- `PortfolioTargetCollection` avoid calling `Count` on
ConcurrentDictionary directly -> has to take all locks
- `SecurityChanges` change Union for Concat since constructor will call
HashSet
- Make `DynamicSecurityData` hold lazy data objects
- `RegisteredSecurityDataTypesProvider` avoid looping over all
registered types, adding `TryGetType`
- `Security.Update()` will no call group by on data since this data is
already grouped by type. Adding `ContainsFillForwardData` will allows to
be lazy and not re loop through the data unless necessary
- `DefaultAlphaHandler` will use the `static`
`Enumerable.Empty<Insight>` instance when possible
- `SubscriptionSynchronizer` will be lazy to construct the
`universeData` dictionary which is not used in most of the times. Will
use `Count` vs `Any` -> `Count` is known by the dictionary
- For python algorithms `JobQueue` will respect `AlgorithmLocation`, was
using unexisting `"algorithm-path-python"`
- `SubscriptionSynchronizer` will call `OnSubscriptionFinished` for
subscriptions that added a data point to the packet after creating the
`TimeSlice` to avoid dropping the last data point.
- `SubscriptionSynchronizer` will emit a `TimeSlice.TimePulse` before
performing any universe selection on each time loop. This will advance
`Algorithm.Time` which will allow universe selection data time and
`Algorithm.Time` to be aligned.
- Updating Regression algorithms that were using `algorithm.Time` in the
selection method.
- Coarse selection will start from the algorithms start date (not in the
next day)
- Adding regression algorithm
- Setting SPY as the default security benchmark
- The security benchmark subscription will be added at `UniverseSelection`
as an internal subscription. Using its own dedicated Security instance
which doesn't live in the algorithms.Securities collection.
- Reducing algorithms exposure to internal subscriptions
- `TimeSliceFactory` will prioritize higher resolution bars, when same
symbol is present twice (for non-internal subscriptionst)
- Adding regression test `CustomUniverseWithBenchmarkRegressionAlgorithm`
- `TimeSliceFactory` will avoid creating empty collections
- `ExecutionModels` will check target collection count before trying to
enumerate
- Reduce calls to .`TotalPortfolioValue`
- `SecurityValues` will only be created when required
- `TimeKeeper` will use TimeZone unique Id as dictionary key. The
TimeZone hash is expensive.
- `AlgorithmManager` will avoid calling `DateTime.UtcNow`,
`ConvertFromUtc()` and `RoundDownInTimeZone()`
- `UniverseSelection` class will now keep track and remove internal
currency subscriptions through the usage of new class
`CurrencySubscriptionDataConfigManager`
- Modifying existing regression tests to assert bug 2773.
- Adding new regression test that fixes the case where a new cash is set
after initialization. This could happen during cash sync performed by
the `BrokerageTransactionHandler`.
- Adding new `CashAdded` event for the `CashBook` to solve the
issue.
- Removing unneeded `CashBook` instance to create a new `TimeSlice`
- Adding new `TimeSliceFactory`, an instance base class that will
provide methods to create a new `TimeSlice`. Will own the `DateTimeZone`
property.
- Adding new `ISecurityPrice` and `IOptionPrice` that will provide a
reduced interface for accessing price properties and methods used when
creating a new `TimeSlice`
- The class `Subscription` will internally track each `Universe`
`SubscriptionRequest` added or removed
- Adding regression test in which two different `Universe` request the
same `SubscriptionDataConfig` and one of them removes/adds it in a
toggle fashion (fails on current master)
- `UniverseSelection` pending removals will also be tracked by
`Universe`
- `UniverseDecorator` will overwrite the `Universe` member of
`SubscriptionsRequests` at `GetSubscriptionRequests()`. This is due to
`this != this,Universe`
- Adding `Subscription` unit tests covering expected behavior
- Extracting pending removals logic from `UniverseSelection` class into
a new helper class `PendingRemovalsManager`. This new class will keep
track of the `universes` requesting to remove a security. Adding unit tests
- Creating new `Synchronizer` which will consolidate and combine `TimeSlice`
streaming for both live and backtests modes. Will live in front of the
`DataManager`
- `SubscriptionSynchronizer` will be the `FrontierTimeProvider` exposed,
and owned, by the new `Synchronizer`
- Updating existing tests
Adds the concept of universe disposal which is requested by an algorithm
through invocation of UniverseManager.Remove, which is invoked via
algorithm.RemoveSecurity. This instructs the data feed that the algorithm
has requested to completely remove the universe and any child subscriptions
from the feed. Security changes are fired for all removed securities.
With the enumerator in blocking mode (backtesting), MoveNext was returning true while Current was being set to null.
This was causing a NullReferenceException in SubscriptionSynchronizer.Sync.
If we pull data and on the same time step that security gets removed,
we can still get that data in OnData(Slice) even though it was removed.
This change filters out removed securities by tracking a reference to
the subscription's disposed flag. Another change was made to wait until
the end of the time step to dispose of subscriptions.
Add IDataFeed.GetSubscription(SubscriptionDataConfig)
The fine enumerator factor accepts a 'date'. This date is that day
that the data is produced, so the emit time on that data is midnight
the following day. When we go to read the fine data we can't use the
current time because that's the 'emit time' -- instead we need to back
the time up a full day to find the correct start time. In addition,
the fine data was being read using UTC time stamps which led to even
more confusion here -- this change resolves both issues.
In addition to the above, the security.Fundamentals property wasn't being
properly set for the first time step when a security is selected due to
a bug in the subscription synchronizer not clearing out the universeData
in preparation for another loop (in the event changes != None).
Fine universes are considered special due to their chaining behavior with
the coarse universe. UniverseSelection.ApplyUniverseSelection manually
reads the fine data for symbols that have passed the initial coarse filter.
There doesn't exist a mechanism for pipe this data back to the data feed
for inclusion in the TimeSlice object. As such, this hack relies on reference
semantics to updates the BaseDataCollection to include the merged coarse and
fine fundamental data.
Pipes universe data from the data feed, through TimeSlice.Create and adds
TimeSlice.UniverseData dictionary property for read access from the algo
manager, where the data will be placed onto the correct security object.
A large percentage of synchronization time is spent performing time zone conversions.
Previously, it was possible that we may compute the same tz conversion multiple times
on the same piece of data until it's emitted. This change moves the time zone conversion
logic onto the data feed thread right before it puts the data into enqueueable enumerator.
In addition, we're not performing the clone operation at the same moment. This removes
unnecessary computation while moving the necessary computation to the data feed thread to
lighten the load on the synchronization/algorithm thread.
The underlying price data was offset one time step in the future due to the
usage of `subscription.Current` after we've performed the synchronization loop.
At this time, `subscription.Current` is guaranteed to either be null
(`EndOfStream`) or be ahead of the frontier
Provides an abstraction point allowing us to easily substitute in a different
synchronization algorithm. This isn't 100% complete as I'm hopefully able to
remove the universe selection dependency from the synchronization implementation.
The concept of subscription frontier is only relevant to the subscription synchronizer.
This change removes the back and forth communication of the frontier data between the
data feed component and the synchronizer component
This caused a change in the CoarseUniverseTop5DollarVolumeAlgorithm. The changed values are
due to the subscriptions now correctly stopping before the turn of the new year. The exact
same number of data points are emitted and the same trades are made, but due to the subscription
stopping slightly sooner (correctly), we lose a benchmark sample which caused a slightly different
beta to be computed (along with dependent statistics).
This method was adding unnecessarily two types of DataFeedPacket objects.
The first type of packet contained the coarse data just used for universe selection, the other one contained only empty packets.
Since these packets would be filtered out later anyway, there is no point in passing them to TimeSlice.Create which needs to enumerate them.
This update reverts PR #664 and includes an improved filtering solution.
It prevents consolidators and indicators from receiving inputs at a resolution higher than the data subscription resolution, typically fill-forward bars.
Volume-based indicators and consolidators will now be calculated correctly with fill-forward data.
It also fixes the RegressionAlgorithm failing test.
Before now, universe data was provided as a single subscription.
Now, you may have many subscriptions feeding one universe, so we
aggregate the universe data and then invoke universe selection.
This change also removes the restriction of a single subscription
per symbol
Martin-Molinero