- `TimeSliceFactory` will avoid creating empty collections
- `ExecutionModels` will check target collection count before trying to
enumerate
- Reduce calls to .`TotalPortfolioValue`
- `SecurityValues` will only be created when required
- `TimeKeeper` will use TimeZone unique Id as dictionary key. The
TimeZone hash is expensive.
- `AlgorithmManager` will avoid calling `DateTime.UtcNow`,
`ConvertFromUtc()` and `RoundDownInTimeZone()`
- Adding new `AlgorithmSettings` Min and Max absolute portfolio target
percentage
- Adding new `PortfolioConstructionModel.FilterInvalidInsightMagnitude()`
helper method that will be used by the `BlackLitterman` and
`MeanVariance` optiomization portfolio construction models to skip
insights with extreme magnitudes that will cause exceptions
- `PortfolioTarget.Percentage()` will now verify requested percent is
withing the settings values
- `Alpha Assets` chart will only store last data point
- Adding new `JsonRoundingConverter` that will round to 4 (number of
digits currently used for comparing alpha statistics) fractional
digits.
- Will be used for `Insights` and `ChartPoint`
- `FactorFile` will keep an ordered reversed list with the dates.
Calling `Reverse()` on the `SortedList` is expensive.
- `MapFiles` will keep first and last date, so we don't need to call
`First()` and `Last()` multiple times.
- `Liquidate` will go through all the algorithms securities only if
necessary
- `TradeBar` parsing will not call `new T` for pure `TradeBar` which is
expensive
- Removing `Lazy` hash code and security type for the
`SecurityIdentifier`, replacing for direct initialization. Accessing the
`Lazy` value adds an overhead.
- Replacing `Enum` to string for hardcoded switch statement. `Enum.ToString` is expensive.
- `DataManager` will be lazy for counting the subscriptions for
determining if its above the limit
- Adding `AlgorithmSecurityValuesProvider.GetAllValues()`, removes the
need to fetch all the security keys twice.
- During universe selection, will not try to re add already added symbol
- Using `Aggregate(lambda)` vs `Sum(lambda)` since the later is slower
due to performing an extra `Select`
- For `QCAlgorithm.Framework.OnFrameworkData()` will avoid calling
`ToArray()` on empty `Enumerables` due to its cost * the number of
calls. If the `Enumerable` is the empty instance, which is static,
will create a new empty array and return it instead.
- Replacing `SecurityIdentifier` `SecurityType` and `GetHashCode`
implementations for `Lazy` versions, that are performed just once, since
these values do not change and are used multiple times.
- For the different `DataDictionary<T>` implementations adding `this[
Symbol] get; set` since existing overload `this [string]` produces an
extra round operations `Symbol->string->Symbol` with a significant
impact.
- Adding `PortfolioTargetCollection.AddRange()` overload using an array
to avoid unnecessary convertions.
Implement a new overload to `Insight` constructor that accepts a `Func<DateTime, DateTime>` that is used to compute the `CloseTimeUtc` and `Period` after the `Insight` object is emitted (`SetPeriodAndCloseTime` method).
Adds the static Expiry class with functions that can be used to compute a future date/time (expiry) given a date/time.
Closes#3038
- Classic Algorithms will emight insights based on order fills.
- To be able to update generated insights closed time, we will not
clone emitted insights.
- `InsightAnalysisContext` will update `AnalysisEndTimeUtc` when the
Insight period is closed and the period is `EndOfTimeTimeSpan`
- Adding new regression algorithm asserting on the new emitted insights
- Adding unit tests
- `LiveTradingResultHandler` will store `AlphaRunTimeStatistics`
- Making `DefaultAlphaHandler.ProcessAsynchronousEvents` virtual to
facilitate cloud changes
- When there are or aren'tt new insights, the EqualWeightingPortfolioConstructionModel will creates a target to flatten delisted securities from the universe of expired insights.
- Helper methods were added to deal with removing expired insights and getting active ones and used in `EqualWeightingPortfolioConstructionModel`
- Adds unit test
- Updates framework algorithms
Fixes a bug where we were using the security's data resolution to compute
the insight's close time. This led a case such as insight.Period == 20days
to step 20days worth of tradable minutes (assuming minute data resolution),
yielding a close time that was very far in the future.
We also add different means of specifying an insight's period/close time:
1. Specify insight period as a TimeSpan and we compute close time
2. Specify insight period and a resolution and bar count and we compute close time
3. Specify insight close time local directly and we compute the insight period
The key here is maintaining consistency between the three different approaches
which is heavily validated with the corresponding unit tests.
Edits also made to trust the insight's close time as the analysis end time in
the case where the analysis period == insight period (extra analysis period = 0).
Given the current setup (extra analysis period == 0), this guarantees that close
and analysis end times are equivalent.
Regression statistics were updated and expectedly we get many more insights that
have completed analysis, and as such, average scores have also changed.
The insight itself defines when it closes via the algorithm framework
and properly takes into account weekends and out of market hours. The
insight analysis was not respecting the insight's stated close time,
but instead was simply doing generated time + period, which doesn't
properly take into account market hours. This causes the number of
closed insights to decrease and due to the extra time for each insight,
the values of the insights have also increased.
This identifier is used to determine the alpha model that generated.
This is NOT ideal, since it requires users to specify the value, more
thought will be givent to how we can resolve this value automatically
Adds test for surviving roundtrip copy operation.
Provides a collection type for managing insights. Internally it uses
a dictionary Symbol->List<Insight> but does NOT implement the dictionary
interface due to potentially unexpected behavior when enumerating, i.e,
different behavior when enumerating if statically known as list vs statically
known as dictionary -- not sure how python would resposne to the ambiguity,
so best to leave well enough alone :)
This type is just used as a container for generatd insights. Renaming in
preparation for a new InsightCollection to mirror the PortfolioTargetCollection
Implements `Insight.Price` method to make it easier to create new instances of `Insight` of `InsightType.Price`.
Standardize the parameter order to `Symbol`, `TimeSpan`, `InsightType`, `InsightDirection`, `Double`, `Double`.
This method is misleading at best and incorrect at worst.
Insight objects should use reference equality or compare ids to
perform equality checking. The only usage, in MacdAlphaModel,
was easily converted to not relying on this method.
The term 'alpha' is used to describe the entire algorithm. Therefore, 'alpha'
produces insights. From this we have things like IAlphaModel, which is the model
defining how insights are produced. We have IAlphaHandler, which defines how the
insights from a single 'alpha' (the algorithm) are managed, analyzed, and stored.
Types closer to the individual prediction level, such as InsightDirection, or
InsightScore relate directly to exactly 1 insight. The distinction between the
two became more clear as we developed the insights API, and from that effort it
was decided to harmonize alpha/insight terminology across the various QC systems.
As we strayed further from the serialization happy path, things started
getting unwieldy quickly. By defining a dedicated DTO to represent the
serialized type, we don't have to much around as much with json.net to
get it to transform/project our type to what we want. The projection is
handled completely within user code and json.net s used to serialize the
simple/flat result object.
This resolves#1623 where we were unable to properly deserialize a
serialized alpha string. The issue was json.net didn't know what
constructor to use. In addition, json.net didn't know it was supposed
to set using the private property setters which isn't default behavior.
When run in an algorithm this time is set by the algorithm. This constructor is added
to make testing alphas a little easier where we can't rely on the algorithm to set the
generated and close times properly.
This change fixes the non-determinism seen in the rolling averaged alpha scores. This was
caused by usage of ConcurrentDictionary coupled with the key being a new guid. The new guid
was the source of non-determinism as it caused the same conceptual alpha from backtest A to
end up in a different 'bucket' on backtest B due to a different guid. The concurrency isn't
actually needed or desirable in this context. The AlphaManager must be invoked synchronously
to avoid inconsistent/non-deterministic analysis. Given this, the collections were changed
to use HashSet<T> and now produces deterministic results independent of the alpha's id.
The IAlphaManagerExtension defines a type that needs to react to events produced
by the AlphaManager. The actual events were removed in favor of a interface to
handle the events. This removes the need to wire events and instead just pass the
extensions to the alph manager and it will handle invoking the extensions at the
appropriate time.
This change removes all charting and statistics aggregation logic from the alpha
handler and moves it into dedicated types, AlphaChartingManagerExtension and
AlphaStatisticsManagerExtension. The resulting types are highly decoupled from the
LEAN ecosystem allowing them to be easily unit tested, whereas before the logic
was embedded in a handler with many many dependencies which would be very hard to
properly unit test.
As part of this change (and in preparation for moving scoring to the alpha thread)
the resolution of SecurityValues was removed from the alpha manager. In this new
pattern, the alpha manager is pushed generated alphas and security values at each
time step.
If we make a prediction for 1 day in the future, we actually mean 1 trading day.
This change updates the alpha analysis logic to take into account the security's
market hours.
It turns out that checking if it's empty is much more expensive than enumerating
an empty collection. This is because IsEmpty acquires all the locks in order to
give a 'snapshot' answer. Enumerating the dictionary directly (not .Keys or .Values)
instead acquires a single finer grained lock at a time. IIRC, one lock will manage a
few buckets internally.
Not seeding this value cause a very heavy 0 starting value to keep the average
suppressed throughout the entire backtest, or until enough alphas are generated
to overcome the initial ema seed value.
Martin Molinero