* Live Coarse universe refactor
- Live trading will source Coarse and Fine fundamental data directly
from disk. Updating unit tests.
* Adds ILiveDataProvider interface
* Adds wrapper for IDataQueueHandler implementations
* Replaces IDataQueueHandler with ILiveDataProvider in
LiveTradingDataFeed
* Edits IDataQueueHandler documentation
* Maintains aggregation for current IDQH impls and skips for ILDF impls
* Note: No unit test was created for this method, go back and TODO
* Protobuf Market data
- Adding protobuf support for Ticks, TradeBars and QuoteBars. Adding
unit tests.
* Adds unit tests for LiveDataAggregator changes
* Fixes bug where custom data was not handled as it was before
* Fixes race condition bug because of variable reuse in class
* Add protobuf extension serialization
* Fixes for protobuf serialization
* Refactor
* Fix OptionChainUniverse
* replace BaseDataExchange pumping ticks with consolidators
* AlpacaBrokerage
* BitfinexBrokerage
* GDAXBrokerage
* OandaBrokerage
* InteractiveBrokers
* TradierBrokerage
* FxcmBrokerage
* PaperBrokerage
* etc
* WIP fixes for existing LTDF unit tests
* Fixes more LTDF unit tests
* make IDataAggregator.Update recieving Generic BaseData rather than Tick
* Change IDataQueueHandler.Subscribe method
* Some fixes after adding new commits
* Adds protobuf (de)serialization support for Dividend and Split
* Serialize protobuf with length prefix
* Fix missing LTDF unit tests
* Adds TiingoNews protobuf definitions
* fix comments
* more fixes on IQFeedDataQueueHandler
* disallow putting ticks into enumerator directly
* ScannableEnumerator tests
* fix OandaBrokerage
* AggregationManager unit tests
* fix AlpacaBrokerage tests
* fix InteractiveBrokers
* fix FxcmBrokerage tests
* call AggregationManager.Remove method on unsubscribe
* fix GDAX existing tests
* Fixes, refactor adding more tests for AggregatorManager
* Adds BenzingaNews protobuf definitions and round trip unit test
* Adds missing TiingoNews unit test to Protobuf round trip tests
* Improve sleep sequence of LiveSynchronizer
* need start aggregating first, and then can subscribe
* More test fixes and refactor
- Refactoring AggregationManager and ScannableEnumerator so the last is
the one that owns the consolidator
- Adding pulse on the main LiveSynchronizer
* Improve performance of LEquityDataSynchronizingEnu
* Add missing Set job packet method
* Minor performance improvements
* Improvements add test timeout
- Improvements adding test timeout to find blocking test in travis
* Improve aggregationManager performance
* Testing improvements for travis
* Remove test timeouts
* More test fixes
- Adding more missing dispose calls and improving determinism
* fix IEXDataQueueHandler and tests
* Final tweaks to LTDF tests
* more AggregationManager tests
* consume and log ticks
* fix test: couldn't subscribe to Forex tickers
* change Resolution for all bar configs
* Improve RealTimeScheduleEventServiceAccuracy
* refactoring: move common code to base class
* fixed bug; unsubscribe SubscriptionDataConfig
* Small performance improvement
* Minor fixes
* Avoid Symbol serialization
* Fixes coarse selection in live mode
* Fix for live coarse
* Adds protobuf (de)serialization support for Robintrack
* Adds round-trip unit test
* Minor performance improvements
* More minor performance improvements
* pass LiveNodePacket through to OandaBrokerage
* Fixes empty list becoming null value when deserializing with protobuf
* Reverts BZ live trading exception removal and fixes tests
* Refactor WorkQueue making it abstract
* Add try catch for composer
* Adds optional data batching period to LiveFillForwardEnumerator
* Override data-queue-handler with config
* Improve PeriodCountConsolidator.Scan performance
* Move batching delay to main Synchornizer thread
* Reverts addition of Robintrack protobuf definitions
* Give priority to config history provider if set
* Add Estimize protobuffing
- Add Estimize protobuffing support. Adding unit tests
* Always dispose of data queue handler
Co-authored-by: Gerardo Salazar <gsalaz9800@gmail.com>
Co-authored-by: Adalyat Nazirov <aenazirov@gmail.com>
Since extension methods don't play well with pythonnet, this change converts
the extensions class into a decorator class. Additionally, this ObjectStore
type is the type that gets exposed via QCAlgorithm so users can access these
methods directly without requiring the use of extension methods.
This approach has many good properties. For one, it doesn't force implementors
of IObjectStore to use a base class. Second, it maintains healthy separation of
API level concerns (such as convenient methods) from the abstraction level conerns
of IObjectStore. Setting it up in this way ensures ANY implementation of IObjectStore
will still get access to these additional methods. Another thing to note is this
prevents using a base class on QCAlgorithm's public interface. Instead, we have a
specific type that is dedicated to fulfilling API level requirements, which also
provides us flexibility in the event the API needs to be updated. If it were a subclass,
you run the risk of breaking the implementors of the subclass.
This commit is squashed from iterative development:
- More consistent method naming
- Storage root path updated to be absolute and include algorithm name
- Storage root path created only if object store is actually used
- Implemented XML save/load
- Added missing unit tests
- Replaced Log.Trace with Log.Error calls
- Added the object store name logging in Engine.Main
- Read storage root from config
- Create algorithm storage root folder in Initialize
- Remove empty folder in Dispose
- Added null checks in all methods
- Added missing XML parameter docs
- make Initialize and Dispose virtual
- make AlgorithmStorageRoot protected
The IObjectStore abstraction provides algorithms with a persistent
storage mechanism. While the algorithm is running, data is maintained
in memory as a dictionary of raw bytes (string -> byte[]). This ensures
we avoid any reference type shenanigans. Periodically, the data in the
object store is persisted and additionally, when the algorithm shuts
down, the object store's data will again be persisted. This ensures that
when the algorithm starts up again, it will have access to any state
that has been saved into the object store.
A great use case for IObjectStore is saving a compute heavy model.
For example, computing the weights of a deep neural network is very
CPU intensive, but after the weights are computed, evaluation is fairly
quick. An initial backtest can be used to solved for the network's weights
and then subsequent backtests or even in live mode, the weights will be
available to the algorithm provided they were saved into the object store.
Also, some libraries require a file path to load model data. The object
store provides a `GetFilePath(key)` method which will copy the data for
the provided key to the disk and return that path so the library can load
the model data.
- Replacing `BaseData.AdjustResolution` for `DefaultResolution` and
`SupportedResolutions`
- Making `Resolution` nullable for `Algorithm.AddData` methods
- The `ISubscriptionDataConfigService` will set the default resolution
if none was provided and assert it is supported
- Fix bug with `PythonData` `IsSparseData` and `RequiresMapping`
resolution
- Adding `Security.NullLeverage` value to determine when the
`SecurityInitializer` leverage should be used or not
- Adding regression algorithm which reproduces the issue
Per review comments provided in #3743 regarding the
training/long-running scheduled events and the leaky
bucket algorithm. See the PR for more information.
- Adds PyObject overload to `ScheduleManager.TrainingNow` and `ScheduleManager.Training`
- Adds `QCAlgorithm.Train` helper method
- Adds Python algorithm showing how to use the helper method.
* Implement method precedence hack on two additional python methods
* Changed behavior of SymbolCache loading with a ticker
- Previously, we would search the cache with ticker for types that did not
require mapping.
* Added new documentation
* Code cleanup
* Change first date to be SID.DefaultDate if SecurityType isn't expected
* Modified AddData tests to meet expectations. AddData unit tests are passing
- Custom data types will know whether or not Lean should use map files
- Updating regression test with sample custom data using map files,
which can run locally
- Adding unit tests for the `SubscriptionDataReaderHistoryProvider`,
checking it mappes equities and options correctly
The `NullReferenceException` type is intended to only be thrown by the CLR.
In most cases, it should be converted to an `ArgumentException` or an
`InvalidOperationException`, depending on if the null value is a parameter
to the current method or not.
The `Exception` type should never really be thrown as it doesn't provide any
additional information or hints as to the issue. It also forces users that
would like to handle expected exceptions to catch all exceptions. These are
converted to an exception type that more accurately describes the reason for
raising the exception: `KeyNotFoundException`, `InvalidOperationException`
- Setting SPY as the default security benchmark
- The security benchmark subscription will be added at `UniverseSelection`
as an internal subscription. Using its own dedicated Security instance
which doesn't live in the algorithms.Securities collection.
- Reducing algorithms exposure to internal subscriptions
- `TimeSliceFactory` will prioritize higher resolution bars, when same
symbol is present twice (for non-internal subscriptionst)
- Adding regression test `CustomUniverseWithBenchmarkRegressionAlgorithm`
- Adding `IRealTimeHandler.OnSecurityChanged()` will be used to update
the `OnEndOfDay` security related scheduled events
- Adding `BaseRealTimeHandler.cs` to reduce code duplication in the
`Backtesting` and `LiveTrading` `RealTimeHandlers`
- Adding CSharp and Python regression tests
- Deprecating `OnEndOfDay()` callback because of two reasons, mainly
because Python does not support two methods with the same name, but also
because different assets have different market close times.
- `ScheduledEvents` set at the same time will now be deterministic
- Adding `LazyToUpper()` implementation, that will avoid the call to
`ToUpper` if the string is already upper.
- Reduce the timezone conversions at `Time.EachTradeableDayInTimeZone`
- `TotalPortfolioValue` will iterate over all securities once
- Adding new `SecurityIdentifier` cache, significant impact for
algorithms using coarse/fine data
Remove usage of DateTime.UtcNow in buying power models
In PR#3013 we added support for fee models with history, so the new changes to the GDAXFeeModel exposed this bug, breaking a couple of regression tests (issue #3044)
Update regression stats for EmitInsightCryptoCashAccountType
* Fix typos
Add missing time keeper in CashBuyingPowerModelTests
- Implementing `QCAlgorithm.AddUniverseSelectionModel`
- Adding C#/Py regression algorithms using the new API
- Modifying `ManualUniverSelectionModels` symbol, adding hash of
the selected `Symbol.Values`
- Modifying `Coarse` and `Fine` symbol, adding random GUID
- Adding `NullUniverseSelectionModel`
- Removing `using QCAlgorithmFramework = QuantConnect.Algorithm.QCAlgorithm`
- Removing `QCAlgorithmFrameworkBridge`
- Removing `IsFrameworkAlgorithm`
- Making `EmitInsightBasedOnFill` private. Adding new
`IOrderEventProvider` exposing an `event` to which `QCAlgorithm` will
subscribe.
- `AccountType.Cash` algorithms will be allowed to manually trade and
emight insights manually or with alpha model.
- Classic Algorithms will emight insights based on order fills.
- To be able to update generated insights closed time, we will not
clone emitted insights.
- `InsightAnalysisContext` will update `AnalysisEndTimeUtc` when the
Insight period is closed and the period is `EndOfTimeTimeSpan`
- Adding new regression algorithm asserting on the new emitted insights
- Adding unit tests
- `LiveTradingResultHandler` will store `AlphaRunTimeStatistics`
- Making `DefaultAlphaHandler.ProcessAsynchronousEvents` virtual to
facilitate cloud changes
- Adding new `SetAccountCurrency()` for backtesting. Has to be called
before adding any `Security` or calling `SetCash()`, else will throw.
- Adding new Non account currency unit tests for `CashBuyingPower`,
`SecurityPortfolioModel`, `SecurityMarginModel`,
`SecurityPortfolioManager`, `Future/OptionMarginBuyingPowerModels`
- Adding new C# regression test using `SetAccountCurrency()`, one for
`CashBuyingPowerModel` and one for `SecurityMarginModel`
- Adding new Py and C# basic regression algorithms using
`SetAccountCurrency()`
- `Options` and `Futures` will use not use `AccountCurrency` as quote
Cash.
- `SecurityBenchmark` value will be in account currency
- Moving `UniverseSelection.EnsureCurrencyDataFeeds` call into the
`IResultHandler` implementation through usage of the new `SetupHandlerHelper`
class, that will also set an initial conversion rate if none present.
- Adding regression test, that reproduces original issue
- Adding new `HistoryRequestFactory` class. Will provide some methods to
facilitate the creation of new `HistoryRequests`. Moving
`CreateHistoryRequest` and `GetStartTimeAlgoTz` into the new
`HistoryRequestFactory`. And consolidating `GetStartTimeAlgoTz` and
`GetStartTimeAlgoTzForSecurity`.
- Adding new `IAlgorithm.AccountCurrency { get; }` that will point to the
`Portfolio.CashBook.AccountCurrency`. Setter will be added in a
following PR.
- Base `Brokerage` class will now have a `AccountCurrency { get }`
pointing to the `IAccountCurrencyProvider`. Will be used by the different
brokerages implementations.
> This PR is a mechanical refactor, no behaviour changed
- Obsoleting `DataNormalizationMode`. Replacing the usaged by requesting
the `SubscriptionDataConfigs` to the new `SubscriptionDataConfigService`
> Note we still need to refresh the Security.DataNormalizationMode
property.
> This PR is a mechanical refactor, no behaviour changed
- Obsoleting `IsFillDataForward`, `Resolution`, `IsExtendedMarketHours`
`Security` configuration properties. Replacing there usages by
requesting the `SubscriptionDataConfigs` to the new
`SubscriptionDataConfigService`
- After removing a `Subscription` entirely, we will now also remove the
matching `SubscriptionDataConfig` from the `SubscriptionManager`
- Adding a new regression test, which fails in `master`
- Fixing previous PR change that removed `Trace` logging when adding
and removing a `Subscription`
- Fixing a bug in the `UserDefinedUniverse` where calling `RemoveMember`
would cause the `SubscriptionDataConfig` to be re added to the
`SubscriptionManager` (not the `DF`). Found this through added
regression test.
- Adding new ISecurityService and its implementation SecurityService.
Expose by SecurityManager.
This class will expose a method for creating new securities. The
SecurityManager is exposing this new interface, calling _securityService
internally, so Future/OptionUniverseSelectionModel.cs can use it
- Replacing all usages of SecurityManager.CreateSecurity for new
ISecurityService
- Modifying `Cash.cs` and `CashBook.cs` `EnsureCurrencyDataFeeds()` to
return newly added `SubscriptionDataConfig` instead of `Security`. This
will avoid using `Security.Subscriptions` at call site.
- Moving old SecurityManager.CreateSecurity into new
SecurityServiceTests.cs
- Some formatting, comments changes
- Removing SubscriptionManager from new FuturesChainUniverse constructor
- Making Option and Future properties in FutureChainUniverse and
OptionChainUniverse
- Adding new and obsoleting old constructors which received
SecurityInitializer as a parameter.
- Extracting the setting of Underlying security at UniverseSelection for
Option and Future chain universes.
Martin-Molinero