* Live Coarse universe refactor
- Live trading will source Coarse and Fine fundamental data directly
from disk. Updating unit tests.
* Adds ILiveDataProvider interface
* Adds wrapper for IDataQueueHandler implementations
* Replaces IDataQueueHandler with ILiveDataProvider in
LiveTradingDataFeed
* Edits IDataQueueHandler documentation
* Maintains aggregation for current IDQH impls and skips for ILDF impls
* Note: No unit test was created for this method, go back and TODO
* Protobuf Market data
- Adding protobuf support for Ticks, TradeBars and QuoteBars. Adding
unit tests.
* Adds unit tests for LiveDataAggregator changes
* Fixes bug where custom data was not handled as it was before
* Fixes race condition bug because of variable reuse in class
* Add protobuf extension serialization
* Fixes for protobuf serialization
* Refactor
* Fix OptionChainUniverse
* replace BaseDataExchange pumping ticks with consolidators
* AlpacaBrokerage
* BitfinexBrokerage
* GDAXBrokerage
* OandaBrokerage
* InteractiveBrokers
* TradierBrokerage
* FxcmBrokerage
* PaperBrokerage
* etc
* WIP fixes for existing LTDF unit tests
* Fixes more LTDF unit tests
* make IDataAggregator.Update recieving Generic BaseData rather than Tick
* Change IDataQueueHandler.Subscribe method
* Some fixes after adding new commits
* Adds protobuf (de)serialization support for Dividend and Split
* Serialize protobuf with length prefix
* Fix missing LTDF unit tests
* Adds TiingoNews protobuf definitions
* fix comments
* more fixes on IQFeedDataQueueHandler
* disallow putting ticks into enumerator directly
* ScannableEnumerator tests
* fix OandaBrokerage
* AggregationManager unit tests
* fix AlpacaBrokerage tests
* fix InteractiveBrokers
* fix FxcmBrokerage tests
* call AggregationManager.Remove method on unsubscribe
* fix GDAX existing tests
* Fixes, refactor adding more tests for AggregatorManager
* Adds BenzingaNews protobuf definitions and round trip unit test
* Adds missing TiingoNews unit test to Protobuf round trip tests
* Improve sleep sequence of LiveSynchronizer
* need start aggregating first, and then can subscribe
* More test fixes and refactor
- Refactoring AggregationManager and ScannableEnumerator so the last is
the one that owns the consolidator
- Adding pulse on the main LiveSynchronizer
* Improve performance of LEquityDataSynchronizingEnu
* Add missing Set job packet method
* Minor performance improvements
* Improvements add test timeout
- Improvements adding test timeout to find blocking test in travis
* Improve aggregationManager performance
* Testing improvements for travis
* Remove test timeouts
* More test fixes
- Adding more missing dispose calls and improving determinism
* fix IEXDataQueueHandler and tests
* Final tweaks to LTDF tests
* more AggregationManager tests
* consume and log ticks
* fix test: couldn't subscribe to Forex tickers
* change Resolution for all bar configs
* Improve RealTimeScheduleEventServiceAccuracy
* refactoring: move common code to base class
* fixed bug; unsubscribe SubscriptionDataConfig
* Small performance improvement
* Minor fixes
* Avoid Symbol serialization
* Fixes coarse selection in live mode
* Fix for live coarse
* Adds protobuf (de)serialization support for Robintrack
* Adds round-trip unit test
* Minor performance improvements
* More minor performance improvements
* pass LiveNodePacket through to OandaBrokerage
* Fixes empty list becoming null value when deserializing with protobuf
* Reverts BZ live trading exception removal and fixes tests
* Refactor WorkQueue making it abstract
* Add try catch for composer
* Adds optional data batching period to LiveFillForwardEnumerator
* Override data-queue-handler with config
* Improve PeriodCountConsolidator.Scan performance
* Move batching delay to main Synchornizer thread
* Reverts addition of Robintrack protobuf definitions
* Give priority to config history provider if set
* Add Estimize protobuffing
- Add Estimize protobuffing support. Adding unit tests
* Always dispose of data queue handler
Co-authored-by: Gerardo Salazar <gsalaz9800@gmail.com>
Co-authored-by: Adalyat Nazirov <aenazirov@gmail.com>
* Changed variable names of protected members in BaseResultHandler to
match existing variable naming convention
* Changed AlgorithmRunner return type
* Remove AlgorithmResults dictionary from AlgorithmRunner
* Create AlgorithmRunnerResults container class
* Modify Relative Sampling test to accept failure cases
* Misc. updates as a result of changing AlgorithmRunner return type
* Refactors AlgorithmManager loop
* Refactors StatisticsBuilder methods and strategy for series alignment
* Move sampling logic to the corresponding IResultHandler
* Changes benchmark resolution to Resolution.Hour
* Modifies IResultHandler to enable external sampling
* Adds BacktestResultHandler unit tests
* Adds ResolutionSwitchingAlgorithm to test misalignment
* Adds support to AlgorithmRunner to store algorithm IResultHandler
Warning: this commit breaks accurate calculations for algorithms that
only make use of `Daily` resolution data. Previously, because
the benchmark was added in Daily resolution in backtesting, any
algorithm that only made use of daily data would have an accurate
calculation for beta and various other statistics.
These changes serve to fix the statistics calculations of non-daily
resolution algorithms, with daily resolution to be revisited at a later
time.
- When BacktestNodePacket has the inital `CashAmount` set we will clear
all existing cash amounts and set the account currency
- Adding more unit tests
- `AlgorithmManager` will search for `SubscriptionDataConfigs` using the
`SubcriptionDataConfigService` versus directly checking active `Subscriptions`.
In the case of warmup, subscription have not been added yet. Also will
include internal subscriptions.
- Adding unit tests
We restrict each algorithm time loop to a pre-determined amount of time.
Exceeding this limit will cause the algorithm to immediately terminate.
This quickly becomes an issue when considering users running trainable
models that have a long initialization period that exceeds the time loop
maximum.
This change provides a mechanism through which a long-running scheduled
event is permitted to keep running and is permitted to avoid the time loop
permitted by requesting additional time. Requests for additional time are
limited according to a leaky bucket implementation whose parameters are
set via the job's controls structure. The fundamental time unit for the
algorithm is a single minute.
Here's how it works. If a scheduled event takes longer than one full wall
clock second then a request is made to the leaky bucket for one more minute.
If the scheduled event continues to take more time, it will continue to
request additional minutes. Each requested minute will prevent the algorithm's
time loop check from terminating the algorithm. When the bucket is empty and
no more minutes are available to be requested, a TimeoutException is thrown
causing a cascade that ends in the algorithm's termination and status being
flipped to RuntimeError.
Additionally, this applies equally to ALL scheduled events. While some helpers
were added with the naming of Train and TrainNow to the ScheduleManager, these
methods don't do anything special and the infrastructure doesn't otherwise
flag them as different, so this feature becomes part of the core Scheduled
Event feature set.
Further, the live scheduled events were not touched and are still pending
further discussion regarding the value added by enforcing a time restriction
when simulation time and wall clock time are equivalent.
Fixes#3319
I noticed some deltas in regression statistics after a change. Before this
change, we log all orders into an {AlgorithmName}.{Language}.orders.log.
This information turned out to be insufficient to identify the regression.
This change also adds daily logging of portfolio value and each security's
holding quantity/value, as well as the full cash book. This should help to
more quickly identify the root cause behind system regression test failures.
- Removing `using QCAlgorithmFramework = QuantConnect.Algorithm.QCAlgorithm`
- Removing `QCAlgorithmFrameworkBridge`
- Removing `IsFrameworkAlgorithm`
- Making `EmitInsightBasedOnFill` private. Adding new
`IOrderEventProvider` exposing an `event` to which `QCAlgorithm` will
subscribe.
- `AccountType.Cash` algorithms will be allowed to manually trade and
emight insights manually or with alpha model.
- Adding new `IAlgorithm.AccountCurrency { get; }` that will point to the
`Portfolio.CashBook.AccountCurrency`. Setter will be added in a
following PR.
- Base `Brokerage` class will now have a `AccountCurrency { get }`
pointing to the `IAccountCurrencyProvider`. Will be used by the different
brokerages implementations.
Many logs were showing the name of the first algorithm on the line:
SetUp Backtesting: User: <uid> ProjectId: <pid> AlgoId: AddRemoveSecurityRegressionAlgorithm
Instead of the name of the running algorithm.
Finalizes implementation of alpha framework, including alpha.spanner updates, many name changes, and harmonizes insight serialization with alpha streams API
The term 'alpha' is used to describe the entire algorithm. Therefore, 'alpha'
produces insights. From this we have things like IAlphaModel, which is the model
defining how insights are produced. We have IAlphaHandler, which defines how the
insights from a single 'alpha' (the algorithm) are managed, analyzed, and stored.
Types closer to the individual prediction level, such as InsightDirection, or
InsightScore relate directly to exactly 1 insight. The distinction between the
two became more clear as we developed the insights API, and from that effort it
was decided to harmonize alpha/insight terminology across the various QC systems.
The engine defines securities for each universe to properly track them within
the data feed. These securities are not tradable and have no price data associated
with them, and as such, we should not be sending history requests for these symbols.
This change removes all universe symbols from history requests.
NOTE: Requests made directly to the history provider are not filtered out, as the
filtering happens within the QCAlgorithm implementation.
When inspecting regression differences, the first thing that should be looked at
are the fills. Some of the regression algorithms log this data making it possible
to inspect whie others do not. In addition, for algorithms with many securities the
log can quickly become fills with noise from scheduled event logging.
This change aims to make it very easy to compare any regressions in orders/fills
against the most recent successful run of the specified regression algorithm.
When the algorithm finishes, the alpha statistics are merged with the backtest
result statistics. This keeps the regression testing pattern of alpha statistics
the same as the existing regular statistics.
This file is no longer written as part of the regresson test suit.
Regression logs files can now be found in the ./passed and ./regression
folders from the test's executing directory.
Often when implementing new features I find myself having to switch to master
in order to get a copy of a passing regression run log file. This change aims
to simplify that process by saving regression algorithm log files to their own
unique file and if they pass the regression test, copying that file to another
location to keep it while testing a potentially failing regression run.
The current run and most recent passed log files are placed in separate
directories to enable easy directory diffing to quickly find regression issues.
Current run:
./regression/<algorithm-name>.<language>.log
Most recent pass:
./passed/<algorithm-name>.<language>.log
ILeanManagement is meant to be a wrapper on the entire Lean engine and allow
command to be sent from ILeanManagement to the rest of Lean. In addition,
ILeanManagement can act as a hosting environment specific implementation for hosting Leaninstances.
The engine itself shouldn't concern itself with where jobs come from, so we pull the next job and acknowledge job logic out of Engine.Run
Add RamAllocation to AlgorithmNodePacket
Removed unused BaseDataFeed and IQFeedDataFeed
Add catch for NullReferenceException in stream store
ConsoleSetupHandler now throws on live jobs, use BrokerageSetupHandler
Renamed/moved FixedSizedQueue to Util\FixedSizeQueue
Adds EndTime to BaseData
Adds Period to TradeBar
Now filling data forward on days missing files/data
Some performance enhancements
Removed sorted dictonary in DataStream/AlgorithmManager
We round down fill forward data in the data stream since if we were to perform this in the data feed we would continually produce data at the same time and it would all go into the algorithm at once.
Martin-Molinero