* Add OrderRight.GetExerciseDirection(isShort) extension
Returns the OrderDirection resulting from exercise/assignment of a particular
option right
See: BUG #4731
* Fix option exercise/assignment order tags and order event messages
The algorithm manager was doing work to determine whether or not the option ended
in exercise or assignment at expiration. This decision should be left for the exercise
model to decide -- from the algorithm manager's perspective, all that matters is that
the option was expired. The DefaultExerciseModel was updated to properly track whether
the option expired with automatic assignment or exercise, dependending on whether or
not we wrote or bought the option (held liability or right, respectively). Updated unit
tests to check for order event counts and order event messages for option exercise cases.
Fixes: #4731
* Fix typo in algorithm documentation
* Update regression tests order hash
Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
* Add internal subscription manager
- Add InternalSubscriptionManager that will handle internal
Subscription. Replaces the realtime updates
- Fix thread race condition in the TimeTriggeredUniverseSubscription, we
have one thread injecting data points, the main algorithm thread, and
the base exchange is pulling from it
- Fixes for FakeDataQueue
- Adding unit tests
* Address reviews and fixes
- Internal subscription will use extended market hours
- Only sample charts accordingly
- Get api-url once
* Get rid of `previousTime` and use `time` instead in AlgorithmManager
* Refactor variable names in Backtesting and Live IResultHandler impls
* Moves shared variables to BaseResultHandler
* Modifies BacktestNodePacketTests statistics to get tests passing
* Adds new StatisticsBuilder tests
* Modifies BacktestingResultHandler tests to make them passing
- Regarding these tests, the decision was made to get them
passing so that if any behavior changes, we will know immediately.
Next commit will contain regression test changes for easy rollback.
* Removed Sample[a-zA-Z]+ methods from IResultHandler definition
* Converted Sample[a-zA-Z]+ methods from public to protected
* Updated inheritors of BaseResultHandler to use new accessibility
modifiers
* Removes useless code in ResolutionSwitchingAlgorithm
* Refactors AlgorithmManager loop
* Refactors StatisticsBuilder methods and strategy for series alignment
* Move sampling logic to the corresponding IResultHandler
* Changes benchmark resolution to Resolution.Hour
* Modifies IResultHandler to enable external sampling
* Adds BacktestResultHandler unit tests
* Adds ResolutionSwitchingAlgorithm to test misalignment
* Adds support to AlgorithmRunner to store algorithm IResultHandler
Warning: this commit breaks accurate calculations for algorithms that
only make use of `Daily` resolution data. Previously, because
the benchmark was added in Daily resolution in backtesting, any
algorithm that only made use of daily data would have an accurate
calculation for beta and various other statistics.
These changes serve to fix the statistics calculations of non-daily
resolution algorithms, with daily resolution to be revisited at a later
time.
- `AlgorithmManager` will search for `SubscriptionDataConfigs` using the
`SubcriptionDataConfigService` versus directly checking active `Subscriptions`.
In the case of warmup, subscription have not been added yet. Also will
include internal subscriptions.
- Adding unit tests
- Adding `SecurityCacheProvider` this class allows for two different
`Security` to share the same data type cache through different instance
of `SecurityCache`. This is used to directly access custom data types
through their underlying in a peformant maner
- Some small improvements
- `DynamicSecurityData` will be a view into the `SecurityCache` instance
- Custom data which has an underlying will use the underlying
`SecurityCache` data type cache instance
- Refactors for `Security` and `SecurityCache` to avoid storing twice
the same data points in the data type cache
We restrict each algorithm time loop to a pre-determined amount of time.
Exceeding this limit will cause the algorithm to immediately terminate.
This quickly becomes an issue when considering users running trainable
models that have a long initialization period that exceeds the time loop
maximum.
This change provides a mechanism through which a long-running scheduled
event is permitted to keep running and is permitted to avoid the time loop
permitted by requesting additional time. Requests for additional time are
limited according to a leaky bucket implementation whose parameters are
set via the job's controls structure. The fundamental time unit for the
algorithm is a single minute.
Here's how it works. If a scheduled event takes longer than one full wall
clock second then a request is made to the leaky bucket for one more minute.
If the scheduled event continues to take more time, it will continue to
request additional minutes. Each requested minute will prevent the algorithm's
time loop check from terminating the algorithm. When the bucket is empty and
no more minutes are available to be requested, a TimeoutException is thrown
causing a cascade that ends in the algorithm's termination and status being
flipped to RuntimeError.
Additionally, this applies equally to ALL scheduled events. While some helpers
were added with the naming of Train and TrainNow to the ScheduleManager, these
methods don't do anything special and the infrastructure doesn't otherwise
flag them as different, so this feature becomes part of the core Scheduled
Event feature set.
Further, the live scheduled events were not touched and are still pending
further discussion regarding the value added by enforcing a time restriction
when simulation time and wall clock time are equivalent.
Fixes#3319
Extracting this behavior into it's own class. We'll later extend
the functionality of the implementation to enable a training event
a mechanism for extending the current time loop maximum and/or for
flat out disabling it while the training is runnig and the leaky
bucket has capacity.
- `PortfolioTargetCollection` avoid calling `Count` on
ConcurrentDictionary directly -> has to take all locks
- `SecurityChanges` change Union for Concat since constructor will call
HashSet
- Make `DynamicSecurityData` hold lazy data objects
- `RegisteredSecurityDataTypesProvider` avoid looping over all
registered types, adding `TryGetType`
- `Security.Update()` will no call group by on data since this data is
already grouped by type. Adding `ContainsFillForwardData` will allows to
be lazy and not re loop through the data unless necessary
- `DefaultAlphaHandler` will use the `static`
`Enumerable.Empty<Insight>` instance when possible
- `SubscriptionSynchronizer` will be lazy to construct the
`universeData` dictionary which is not used in most of the times. Will
use `Count` vs `Any` -> `Count` is known by the dictionary
- For python algorithms `JobQueue` will respect `AlgorithmLocation`, was
using unexisting `"algorithm-path-python"`
Custom derivative data is now being saved into the underlying security's
cache. This makes the custom derivative data available via the underlying's
security object via underlying.Cache.Get<T> where T is the custom data type.
- `SubscriptionSynchronizer` will emit a `TimeSlice.TimePulse` before
performing any universe selection on each time loop. This will advance
`Algorithm.Time` which will allow universe selection data time and
`Algorithm.Time` to be aligned.
- Updating Regression algorithms that were using `algorithm.Time` in the
selection method.
- Coarse selection will start from the algorithms start date (not in the
next day)
- Adding regression algorithm
- Setting SPY as the default security benchmark
- The security benchmark subscription will be added at `UniverseSelection`
as an internal subscription. Using its own dedicated Security instance
which doesn't live in the algorithms.Securities collection.
- Reducing algorithms exposure to internal subscriptions
- `TimeSliceFactory` will prioritize higher resolution bars, when same
symbol is present twice (for non-internal subscriptionst)
- Adding regression test `CustomUniverseWithBenchmarkRegressionAlgorithm`
- Adding `IRealTimeHandler.OnSecurityChanged()` will be used to update
the `OnEndOfDay` security related scheduled events
- Adding `BaseRealTimeHandler.cs` to reduce code duplication in the
`Backtesting` and `LiveTrading` `RealTimeHandlers`
- Adding CSharp and Python regression tests
- Deprecating `OnEndOfDay()` callback because of two reasons, mainly
because Python does not support two methods with the same name, but also
because different assets have different market close times.
- `ScheduledEvents` set at the same time will now be deterministic
- Adding new regression test algorithm
- `SecurityPortfolioManager` will subscribe to the `CashBook.Updated`
event to invalidate the `TotalPortfolioValue
- `TimeSliceFactory` will avoid creating empty collections
- `ExecutionModels` will check target collection count before trying to
enumerate
- Reduce calls to .`TotalPortfolioValue`
- `SecurityValues` will only be created when required
- `TimeKeeper` will use TimeZone unique Id as dictionary key. The
TimeZone hash is expensive.
- `AlgorithmManager` will avoid calling `DateTime.UtcNow`,
`ConvertFromUtc()` and `RoundDownInTimeZone()`
- Adding check at `AlgorithmManager.ProcessSplitSymbols()`, will skip
splits from removed securities.
- Adding regression test which reproduces original issue in master.
- Moving `UniverseSelection.EnsureCurrencyDataFeeds` call into the
`IResultHandler` implementation through usage of the new `SetupHandlerHelper`
class, that will also set an initial conversion rate if none present.
- Adding regression test, that reproduces original issue
- `AlgorithmManager` will call `EnsureCurrencyDataFeeds()` before the
history requests are created so the conversion rate securities are also
updated during warmup.
- `EnsureCurrencyDataFeeds()` will add new `SubscriptionDataConfigs` to
the `_addedCurrencySubscriptionDataConfigs` hash set. This hash set will
be used during `UniverseSelection()` to add the subscriptions.
- Wont trigger a `UniverseSelection()` before warmup. This was causing
the data to be fetched twice and for consolidators to be updated with
old data.
- Adding a new regression test and adding new checks to existing
regression tests.
- Adding new `IAlgorithm.AccountCurrency { get; }` that will point to the
`Portfolio.CashBook.AccountCurrency`. Setter will be added in a
following PR.
- Base `Brokerage` class will now have a `AccountCurrency { get }`
pointing to the `IAccountCurrencyProvider`. Will be used by the different
brokerages implementations.
> This PR is a mechanical refactor, no behaviour changed
- Obsoleting `DataNormalizationMode`. Replacing the usaged by requesting
the `SubscriptionDataConfigs` to the new `SubscriptionDataConfigService`
> Note we still need to refresh the Security.DataNormalizationMode
property.
> This PR is a mechanical refactor, no behaviour changed
- Obsoleting `IsFillDataForward`, `Resolution`, `IsExtendedMarketHours`
`Security` configuration properties. Replacing there usages by
requesting the `SubscriptionDataConfigs` to the new
`SubscriptionDataConfigService`
Colton Sellers