- Moving `MapFile` and `FactorFile` resolution to the data feed stack so
that they do not add a performance overhead to the algorithm thread.
- Create logging string messaged only if required.
- Calculate `FactorFileRow.PriceScaleFactor` the least amount of times
- `FactorFile` will keep an ordered reversed list with the dates.
Calling `Reverse()` on the `SortedList` is expensive.
- `MapFiles` will keep first and last date, so we don't need to call
`First()` and `Last()` multiple times.
- `Liquidate` will go through all the algorithms securities only if
necessary
- `TradeBar` parsing will not call `new T` for pure `TradeBar` which is
expensive
- Removing `Lazy` hash code and security type for the
`SecurityIdentifier`, replacing for direct initialization. Accessing the
`Lazy` value adds an overhead.
- Replacing `Enum` to string for hardcoded switch statement. `Enum.ToString` is expensive.
- `DataManager` will be lazy for counting the subscriptions for
determining if its above the limit
- Adding `AlgorithmSecurityValuesProvider.GetAllValues()`, removes the
need to fetch all the security keys twice.
- During universe selection, will not try to re add already added symbol
Implements `DataNormalizationMode` field in `UniverseSettings` to enable the settings of a desired `DataNormalizationMode` to the securities that are chosen in Universe Selection.
- Adding new `IAlgorithm.AccountCurrency { get; }` that will point to the
`Portfolio.CashBook.AccountCurrency`. Setter will be added in a
following PR.
- Base `Brokerage` class will now have a `AccountCurrency { get }`
pointing to the `IAccountCurrencyProvider`. Will be used by the different
brokerages implementations.
- Modifying `IFillModel` interface removing old methods and adding new
method `Fill Fill(FillModelParameters)`. This is a breaking change.
- Adding new `PythonWrapper` property for the `FillModel` base class.
This is required due to a limitation in PythonNet:
- Given C# class T has `virtual` methods A and B. Where method A
calls method B. And given custom python class L inherits class T.
And overrides method B. When class L calls
base method A (of class T). And when method A internally calls method B.
It will call C# implementation, not the python override. This issue
is solved going back to the `PythonWrapper`. Adding unit tests.
- Adding new `Parameters` property for the `FillModel` base class that will
be set by the call to `Fill()`. The `Parameters` property will be used by
the modified `XxxxFill()` implementations
- Adding new `Fill` result object for the `Fill(FillModelParameters)`
method
- Adding new check before removing a `SubscriptionDataConfig` due to the FillModels consuming the configuration collection when determining which Price to use. WIll now only remove the `SDC` if the symbol was removed from the selecting `universe`, this will avoid the case where the symbol is never deselected and the subscription ends, which happens at the end of all executions.
- Adding unit tests showcasing retro compatibility.
- Enabling C# `CustomModelsAlgorithm` as a regression test. Python
version returns a different result due to random number generation.
- Previously when a user requested to remove a universe, they would call
`RemoveSecurity()`->`UniverseManager.Remove()`-> `Universe.Dispose()` ->
collection change notification would call `DataManager()` that would
call `RemoveSubscription()` that *would not perform any operation* since
it would return `false` through
`if(subscription.IsUniverseSelectionSubscription &&
subscription.Universe.DisposeRequested)`. Previous comment stated that
the `Subscription` would be removed by the `SubscriptionSynchronizer`
via `SubscriptionFinished()` but this called would go through the same
if statemente as above, returning false and no performing any operation.
So the `Universe` `Subscription` was never removed.
PR https://github.com/QuantConnect/Lean/pull/2039
- In this PR I'm removing the mentioned `if` statement and the call
relation between `UniverseManager.Remove()` and
`DataManager.RemoveSubscription()` since this will be covered by the
`SubscriptionSynchronizer` in the next loop.
- Modified regression test (originally added with these changes at
https://github.com/QuantConnect/Lean/pull/2039) to verify this issue.
- After removing a `Subscription` entirely, we will now also remove the
matching `SubscriptionDataConfig` from the `SubscriptionManager`
- Adding a new regression test, which fails in `master`
- Fixing previous PR change that removed `Trace` logging when adding
and removing a `Subscription`
- Fixing a bug in the `UserDefinedUniverse` where calling `RemoveMember`
would cause the `SubscriptionDataConfig` to be re added to the
`SubscriptionManager` (not the `DF`). Found this through added
regression test.
- The class `Subscription` will internally track each `Universe`
`SubscriptionRequest` added or removed
- Adding regression test in which two different `Universe` request the
same `SubscriptionDataConfig` and one of them removes/adds it in a
toggle fashion (fails on current master)
- `UniverseSelection` pending removals will also be tracked by
`Universe`
- `UniverseDecorator` will overwrite the `Universe` member of
`SubscriptionsRequests` at `GetSubscriptionRequests()`. This is due to
`this != this,Universe`
- Adding `Subscription` unit tests covering expected behavior
- Extracting pending removals logic from `UniverseSelection` class into
a new helper class `PendingRemovalsManager`. This new class will keep
track of the `universes` requesting to remove a security. Adding unit tests
- Completly move `DataManager` in front of `DataFeed`. Specifically
`AddSubscription()` and `RemoveSubscription()` implementations. Also
removing IDataFeed.Subscriptions
- Creating new `Synchronizer` which will consolidate and combine `TimeSlice`
streaming for both live and backtests modes. Will live in front of the
`DataManager`
- `SubscriptionSynchronizer` will be the `FrontierTimeProvider` exposed,
and owned, by the new `Synchronizer`
- Updating existing tests
- Adding new ISecurityService and its implementation SecurityService.
Expose by SecurityManager.
This class will expose a method for creating new securities. The
SecurityManager is exposing this new interface, calling _securityService
internally, so Future/OptionUniverseSelectionModel.cs can use it
- Replacing all usages of SecurityManager.CreateSecurity for new
ISecurityService
- Modifying `Cash.cs` and `CashBook.cs` `EnsureCurrencyDataFeeds()` to
return newly added `SubscriptionDataConfig` instead of `Security`. This
will avoid using `Security.Subscriptions` at call site.
- Moving old SecurityManager.CreateSecurity into new
SecurityServiceTests.cs
- Removed usages of algorithm.Securities.key as a parameter for the
`ManualUniverseSelectionModel()` since those securities, added through
`AddXXXX` calls will be managed by the `UserDefinedUniverse`. This was
causing for Universes to try to add the same subscription requests
- Adding new empty constructor for ManualUniverseSelectionModel,
required for Python
- ManualUniverse will return any existing SDC for the
symbol. This is for maintaining existing behavior and
preventing breaking changes: Specifically motivated by usages of
Algorithm.Securities.Keys as constructor parameter of the
ManualUniverseSelectionModel, since those Symbols added by Addxxx()
calls will already be managed by the UserDefinedUniverse
- Making some format modifications to aling with used Lean formatting
- Adding new ISubscriptionDataConfigBuilder implemented by DataManager, exposed by
SubscriptionManager
- SecurityManager and SubscriptionManager will use new interface.
In a next PR, its intended for it to also be consumed by Universe.GetSubscriptionRequests().
- Moving LookupSubscriptionConfigDataTypes implementation into
DataManager
Refactoring previous commit so DataManager only keeps and receives a
reference to IAlgorithmSettigs, with the objective of reducing tight
coupling
Note: Investigate if IAlgorithmSettings.DataSubscriptionLimit should limit subscriptions or unique securities.
Today its using SubscriptionManagerSubscriptions.Where(x => !x.Symbol.IsCanonical()).DistinctBy(x => x.Symbol.Value).Count() @DataManager
Stefano Raggi