Since extension methods don't play well with pythonnet, this change converts
the extensions class into a decorator class. Additionally, this ObjectStore
type is the type that gets exposed via QCAlgorithm so users can access these
methods directly without requiring the use of extension methods.
This approach has many good properties. For one, it doesn't force implementors
of IObjectStore to use a base class. Second, it maintains healthy separation of
API level concerns (such as convenient methods) from the abstraction level conerns
of IObjectStore. Setting it up in this way ensures ANY implementation of IObjectStore
will still get access to these additional methods. Another thing to note is this
prevents using a base class on QCAlgorithm's public interface. Instead, we have a
specific type that is dedicated to fulfilling API level requirements, which also
provides us flexibility in the event the API needs to be updated. If it were a subclass,
you run the risk of breaking the implementors of the subclass.
- Only add OnEndOfDay ScheduledEvent if the algorithm implements the
method. Adding unit tests
- Avoid creating a new baseData instance at
`SubscriptionDataSourceReader`
- Adding static `FineFundamental` instance since creating new ones is
expensive
This commit is squashed from iterative development:
- More consistent method naming
- Storage root path updated to be absolute and include algorithm name
- Storage root path created only if object store is actually used
- Implemented XML save/load
- Added missing unit tests
- Replaced Log.Trace with Log.Error calls
- Added the object store name logging in Engine.Main
- Read storage root from config
- Create algorithm storage root folder in Initialize
- Remove empty folder in Dispose
- Added null checks in all methods
- Added missing XML parameter docs
- make Initialize and Dispose virtual
- make AlgorithmStorageRoot protected
The IObjectStore abstraction provides algorithms with a persistent
storage mechanism. While the algorithm is running, data is maintained
in memory as a dictionary of raw bytes (string -> byte[]). This ensures
we avoid any reference type shenanigans. Periodically, the data in the
object store is persisted and additionally, when the algorithm shuts
down, the object store's data will again be persisted. This ensures that
when the algorithm starts up again, it will have access to any state
that has been saved into the object store.
A great use case for IObjectStore is saving a compute heavy model.
For example, computing the weights of a deep neural network is very
CPU intensive, but after the weights are computed, evaluation is fairly
quick. An initial backtest can be used to solved for the network's weights
and then subsequent backtests or even in live mode, the weights will be
available to the algorithm provided they were saved into the object store.
Also, some libraries require a file path to load model data. The object
store provides a `GetFilePath(key)` method which will copy the data for
the provided key to the disk and return that path so the library can load
the model data.
- Replacing `BaseData.AdjustResolution` for `DefaultResolution` and
`SupportedResolutions`
- Making `Resolution` nullable for `Algorithm.AddData` methods
- The `ISubscriptionDataConfigService` will set the default resolution
if none was provided and assert it is supported
- Fix bug with `PythonData` `IsSparseData` and `RequiresMapping`
resolution
- Adding `BaseData.AdjustResolution()` that should return a valid
resolution for the given data and security type.
This allows us to set a limitation which is useful to avoid invalid data
requests or unnecessary fill forward situations. The user will be
notified through a console message.
- Adding unit and regression test
- Updating example algorithms custom data resolution
- Some performance improvements. Wont change console color if
`SelectedOptimization` is defined
- Adding `IRealTimeHandler.OnSecurityChanged()` will be used to update
the `OnEndOfDay` security related scheduled events
- Adding `BaseRealTimeHandler.cs` to reduce code duplication in the
`Backtesting` and `LiveTrading` `RealTimeHandlers`
- Adding CSharp and Python regression tests
- Deprecating `OnEndOfDay()` callback because of two reasons, mainly
because Python does not support two methods with the same name, but also
because different assets have different market close times.
- `ScheduledEvents` set at the same time will now be deterministic
- Adding _some_ of the missing PyObject.Dispose calls. In the cases
where C# is calling the Python side.
- Note that Python calls to C# code is correctly handling the
disposure of resources.
- Removing `using QCAlgorithmFramework = QuantConnect.Algorithm.QCAlgorithm`
- Removing `QCAlgorithmFrameworkBridge`
- Removing `IsFrameworkAlgorithm`
- Making `EmitInsightBasedOnFill` private. Adding new
`IOrderEventProvider` exposing an `event` to which `QCAlgorithm` will
subscribe.
- `AccountType.Cash` algorithms will be allowed to manually trade and
emight insights manually or with alpha model.
- Classic Algorithms will emight insights based on order fills.
- To be able to update generated insights closed time, we will not
clone emitted insights.
- `InsightAnalysisContext` will update `AnalysisEndTimeUtc` when the
Insight period is closed and the period is `EndOfTimeTimeSpan`
- Adding new regression algorithm asserting on the new emitted insights
- Adding unit tests
- `LiveTradingResultHandler` will store `AlphaRunTimeStatistics`
- Making `DefaultAlphaHandler.ProcessAsynchronousEvents` virtual to
facilitate cloud changes
- Adding new `SetAccountCurrency()` for backtesting. Has to be called
before adding any `Security` or calling `SetCash()`, else will throw.
- Adding new Non account currency unit tests for `CashBuyingPower`,
`SecurityPortfolioModel`, `SecurityMarginModel`,
`SecurityPortfolioManager`, `Future/OptionMarginBuyingPowerModels`
- Adding new C# regression test using `SetAccountCurrency()`, one for
`CashBuyingPowerModel` and one for `SecurityMarginModel`
- Adding new Py and C# basic regression algorithms using
`SetAccountCurrency()`
- `Options` and `Futures` will use not use `AccountCurrency` as quote
Cash.
- `SecurityBenchmark` value will be in account currency
- `AlgorithmPythonWrapper` will now keep reference to the `OnData` and
`OnOrderEvent` `PyObject` method, giving a performance improvement,
since it does not have to resolve it in each loop.
- Moving `UniverseSelection.EnsureCurrencyDataFeeds` call into the
`IResultHandler` implementation through usage of the new `SetupHandlerHelper`
class, that will also set an initial conversion rate if none present.
- Adding regression test, that reproduces original issue
- Adding new `IAlgorithm.AccountCurrency { get; }` that will point to the
`Portfolio.CashBook.AccountCurrency`. Setter will be added in a
following PR.
- Base `Brokerage` class will now have a `AccountCurrency { get }`
pointing to the `IAccountCurrencyProvider`. Will be used by the different
brokerages implementations.
IAlgorithm exposed a means of setting the CurrentSlice but not a
means for consumers to get the value. This was because until now
all consumers were within the QCAlgorithm scope and had access to
the member variable. This change makes the CurrentSlice available
to LEAN engine code, where it's first use will be in PaperBrokerage
to detect and apply dividend distributions.
This type is just used as a container for generatd insights. Renaming in
preparation for a new InsightCollection to mirror the PortfolioTargetCollection
`AlgorithmPythonWrapper.OnFrameworkData` should call the base class method (`QCAlgorithmFramework.OnFrameworkData`) directly instead of trying to call this method from the python script
We had an issue with the data feed picking up universe/security changes
too quickly, thereby preventing user code from being able to configure the
security object properly. Specifically, users were having an issue setting
the data normalization mode of options and underlying equity securities. By
the time the user code had set the data mode, the data feed had already
created a subscription and began processing it, so the changes were never
seen in the data feed.
This change moves all security/universe changes into pending lists and at
the end of the time step applies those changes. Security objects are still
added directly to the SecurityManager for instance access, but we delay in
adding the security to the universe and the universe to the UniverseManager.
Once added to the universe manager, an event is fired and the data feed will
process the new subscriptions.
This event was not properly wrapped.
When `IAlgorithm.InsightsGenerated` is set in `IAlphaHandler`, it should be directed to the base algorithm, whereas it was set to a null variable.
Move the logic of importing the module into AlgorithmPythonWrapper where it is wrapped.
Throws an exception if the script does not have a class that inherits from either QCAlgorithm not QCAlgorithmFramework.
Adds a check for OnData being defined in the module. If not, OnData from the base class will not be called (it causes stack overflow otherwise)
The term 'alpha' is used to describe the entire algorithm. Therefore, 'alpha'
produces insights. From this we have things like IAlphaModel, which is the model
defining how insights are produced. We have IAlphaHandler, which defines how the
insights from a single 'alpha' (the algorithm) are managed, analyzed, and stored.
Types closer to the individual prediction level, such as InsightDirection, or
InsightScore relate directly to exactly 1 insight. The distinction between the
two became more clear as we developed the insights API, and from that effort it
was decided to harmonize alpha/insight terminology across the various QC systems.
The error message it was used to bypass the the error logging has been changed from `takes exactly x arguments (y given)` to `takes x positional argument but y were given`.
With this constructor, SetConverter method, that was not respecting Lean pattern, is removed. The initialization of _converter field is done once the type is loaded.
Python algorithms with custom data requires an operation that converts a dictionary key into a attribute. In the current implementation the Slice object was converted into a python dictionary. This was not optimal, since we just need to make this conversion when the value of a key in the Slice is accessed.
This implementation proposes a wrapper for the Slice object, PythonSlice, that would just perform the operation described above when needed.
This method is being added to allow algorithms to complete initialization tasks that cannot be executed during Initialize, such as cancelling existing open orders in live trading.
This method will be called only once, when the warmup task is complete.
Closes#1043
IAlgorithm.FrameworkOnData is used to pulse models with new data each time step
IAlgorithm.FrameworkOnSecuritiesChanged is used to pulse models with security changes
These two functions need to be separate to ensure that if we add an indicator during
the securities changed event that it will get the data from the current time step.
This forces us to call the securities changed event before we invoke the consolidators
for the current time step.
The engine creates schedule events that call OnEndOfDay() and OnEndOdDay(Symbol) for every security. If just one of these method overloads is defined in the python script, the call of the other will throw a exception due different parameter count.
In this case, we will disregard this exception and throw those that may happen in the method implementation body.
Michael Handschuh