* Adds Validate class in util with accompanying regular expressions
* Provides tests for NotificationEmail and Validate.EmailAddress
* Provides default values for NotificationEmail Subject and Message
These default values resolve the underlying issue of #3021 w/out
needing to modify external sources (such as impl of IMessagingHandler)
- For backtesting the algorithm thread will sequentially process pending
order requests
- Adding new non blocking `BusyCollection`
- Updating `UpdateOrderRegressionAlgorithm` that suffered of a race
condition: it supposed `OnOrderEvent` would be called after the call to
`Transactions.AddOrder()` was finished
- Adding `WorkerThread` class, wrapper for a worker thread that will
execute given `Actions`.
- Algorithm related code (`Construction`, `Initialization`,
`Execution` will be executed by the same `WorkerThread` instance,
this is required for `Python` debugging.
- `Alpha Assets` chart will only store last data point
- Adding new `JsonRoundingConverter` that will round to 4 (number of
digits currently used for comparing alpha statistics) fractional
digits.
- Will be used for `Insights` and `ChartPoint`
Energy futures (CL, HO, RB, NG) expire in the month before the contract month.
To handle these properly, the following changes have been implemented:
- the AlgoSeekFuturesReader has been updated to create the future symbol with the correct expiry date
- the zip entry names in futures data files now contain the full expiry date (in addition to the contract month)
- the sample data files have been updated to use the new zip entry names
- new unit tests have been added and existing ones updated
- `FactorFile` will keep an ordered reversed list with the dates.
Calling `Reverse()` on the `SortedList` is expensive.
- `MapFiles` will keep first and last date, so we don't need to call
`First()` and `Last()` multiple times.
- `Liquidate` will go through all the algorithms securities only if
necessary
- `TradeBar` parsing will not call `new T` for pure `TradeBar` which is
expensive
- Removing `Lazy` hash code and security type for the
`SecurityIdentifier`, replacing for direct initialization. Accessing the
`Lazy` value adds an overhead.
- Replacing `Enum` to string for hardcoded switch statement. `Enum.ToString` is expensive.
- `DataManager` will be lazy for counting the subscriptions for
determining if its above the limit
- Adding `AlgorithmSecurityValuesProvider.GetAllValues()`, removes the
need to fetch all the security keys twice.
- During universe selection, will not try to re add already added symbol
- `Composer` will load `QuantConnect` assemblies and keep the `Types`
which is much faster than using previous wway of getting types that uses
reflexion.
- Adding static `MapFileProvider` improving calls to
`QuantConnect.commo.Symbol.Create` or `Addxxx` which created a new
instance every time
- `Engine.StackExceptionInterpreter` will be lazy initialized, since it
takes some time because it goes through all assemblies
- Moving `MarketHoursDataBase` initialization earlier and in a `Task`
since it takes some time.
- Adding _some_ of the missing PyObject.Dispose calls. In the cases
where C# is calling the Python side.
- Note that Python calls to C# code is correctly handling the
disposure of resources.
- Deleting `ParallelRunnerController` and `ParallelRunnerWorker`.
Replacing them for `Tasks`. The consumer, `EnqueueableEnumerator` will
directly spin up a new producer Task when he is running low on items.
In the new package:
- C# decimal conversion will use C# double and python float due to the big performance impact of converting C# decimal to python decimal;
- `Composer` will scan the specified directory for binaries only once.
This gives a performance improvement:
- Regression tests execution times:
- `master`: 11:11
- `PR`: 9:56, 9:46
- Reverting the assert logic of a unit test
> We now have regression tests < 1 second as low as 0.5s!
- Fixing exception at ParallelRunnerController due to trying to double
dispose of workers and setting already disposed `_waitHandler`
- Fixing exception at ParallelRunnerWorker, disposing of handle after
thread. Will now try joining thread before aborting if cancellation was
requested.
Removing sleep and disposing queues after consumers,
I believe this should also solve original issue while saving time.
Will first Join with 1 second timeout and Abort if it failed.
Adding try catch for robustness.
This implements a finalizer to perform disposal.
Also adds ability to mark other objects for disposal when this instance
is disposed. This is useful for disposing, for example, a ZipFile that
was used to create the StreamReader instance.
Adds reference price to dividend/split
Adds GetSplitsAndDividends to FactorFile
Adds Apply methods to FactorFile/FactorFileRow
Updates factor files to include reference prices
When creating a collection of directories from where dynamic-link library can be loaded from, a user-defined path is used if the reserved key `composer-dll-directory` exists. Otherwise, the current executing directory is used.
This feature is necessary for applications, e.g.: Jupyter kernels, that use Lean/QuantConnect assemblies, since they may be called from a directory with dlls that don't need to be loaded by the `Composer`.
Existing implementation was written under the assumption that the result type
was a value type and was being read using JToken.Value<TResul>() which cases
exceptions if TResult is a non-string class type. The fix was to check the type
and use JToken.ToObject<TResult>() when the result type is not a value type.
The term 'alpha' is used to describe the entire algorithm. Therefore, 'alpha'
produces insights. From this we have things like IAlphaModel, which is the model
defining how insights are produced. We have IAlphaHandler, which defines how the
insights from a single 'alpha' (the algorithm) are managed, analyzed, and stored.
Types closer to the individual prediction level, such as InsightDirection, or
InsightScore relate directly to exactly 1 insight. The distinction between the
two became more clear as we developed the insights API, and from that effort it
was decided to harmonize alpha/insight terminology across the various QC systems.
In order to access the custom data classes, the module containing them was added to the ObjectActivator. This was unnecessary if it wasn't a custom data algorithm.
Also, this operation would not be taken into account if the custom data class were defined after the algorithm was created: this is the case for QuantBook.
We refactor how custom data is handled: a new class was added to provide a instance creation factory that creates an instance of each python custom type.
The FineFundamentalSubscriptionEnumeratorFactory used in combination with SingleEntryDataCacheProvider and DefaultDataProvider was never disposing of the underlying file stream, leaving files open.
Michael Handschuh