* Calculate both raw and adjuasted prices for backtesting
* disable second price factoring
* move and reuse method
* test coverage for new methods
* reuse scaling method
* reuse subscriptionData.Create method
* removed unused code
* regression test
* switch to aapl
* fix regression test output
* more asserts
* fix comments - reduce shortcuts and abbrevation
* more comments
* merge parameters
* reduce number of getting price factors
* fix tests
* fix tests
* fix regression tests
* calculate TotalReturn on demand
* include TotalReturn calculations
* perf tuning
* more unit tests for SubscriptionData.Create
* simplify things - store and return only raw and precalculated data
* fix regression tests; change it back
* factor equals 1 for Raw data
* small changes
* follow code style
* implement backward compatibility
* Add set market price during extended market hours
- Set market prices during extended market hours for live trading.
Adding unit test
* Add assert on internal data count
* Live Coarse universe refactor
- Live trading will source Coarse and Fine fundamental data directly
from disk. Updating unit tests.
* Adds ILiveDataProvider interface
* Adds wrapper for IDataQueueHandler implementations
* Replaces IDataQueueHandler with ILiveDataProvider in
LiveTradingDataFeed
* Edits IDataQueueHandler documentation
* Maintains aggregation for current IDQH impls and skips for ILDF impls
* Note: No unit test was created for this method, go back and TODO
* Protobuf Market data
- Adding protobuf support for Ticks, TradeBars and QuoteBars. Adding
unit tests.
* Adds unit tests for LiveDataAggregator changes
* Fixes bug where custom data was not handled as it was before
* Fixes race condition bug because of variable reuse in class
* Add protobuf extension serialization
* Fixes for protobuf serialization
* Refactor
* Fix OptionChainUniverse
* replace BaseDataExchange pumping ticks with consolidators
* AlpacaBrokerage
* BitfinexBrokerage
* GDAXBrokerage
* OandaBrokerage
* InteractiveBrokers
* TradierBrokerage
* FxcmBrokerage
* PaperBrokerage
* etc
* WIP fixes for existing LTDF unit tests
* Fixes more LTDF unit tests
* make IDataAggregator.Update recieving Generic BaseData rather than Tick
* Change IDataQueueHandler.Subscribe method
* Some fixes after adding new commits
* Adds protobuf (de)serialization support for Dividend and Split
* Serialize protobuf with length prefix
* Fix missing LTDF unit tests
* Adds TiingoNews protobuf definitions
* fix comments
* more fixes on IQFeedDataQueueHandler
* disallow putting ticks into enumerator directly
* ScannableEnumerator tests
* fix OandaBrokerage
* AggregationManager unit tests
* fix AlpacaBrokerage tests
* fix InteractiveBrokers
* fix FxcmBrokerage tests
* call AggregationManager.Remove method on unsubscribe
* fix GDAX existing tests
* Fixes, refactor adding more tests for AggregatorManager
* Adds BenzingaNews protobuf definitions and round trip unit test
* Adds missing TiingoNews unit test to Protobuf round trip tests
* Improve sleep sequence of LiveSynchronizer
* need start aggregating first, and then can subscribe
* More test fixes and refactor
- Refactoring AggregationManager and ScannableEnumerator so the last is
the one that owns the consolidator
- Adding pulse on the main LiveSynchronizer
* Improve performance of LEquityDataSynchronizingEnu
* Add missing Set job packet method
* Minor performance improvements
* Improvements add test timeout
- Improvements adding test timeout to find blocking test in travis
* Improve aggregationManager performance
* Testing improvements for travis
* Remove test timeouts
* More test fixes
- Adding more missing dispose calls and improving determinism
* fix IEXDataQueueHandler and tests
* Final tweaks to LTDF tests
* more AggregationManager tests
* consume and log ticks
* fix test: couldn't subscribe to Forex tickers
* change Resolution for all bar configs
* Improve RealTimeScheduleEventServiceAccuracy
* refactoring: move common code to base class
* fixed bug; unsubscribe SubscriptionDataConfig
* Small performance improvement
* Minor fixes
* Avoid Symbol serialization
* Fixes coarse selection in live mode
* Fix for live coarse
* Adds protobuf (de)serialization support for Robintrack
* Adds round-trip unit test
* Minor performance improvements
* More minor performance improvements
* pass LiveNodePacket through to OandaBrokerage
* Fixes empty list becoming null value when deserializing with protobuf
* Reverts BZ live trading exception removal and fixes tests
* Refactor WorkQueue making it abstract
* Add try catch for composer
* Adds optional data batching period to LiveFillForwardEnumerator
* Override data-queue-handler with config
* Improve PeriodCountConsolidator.Scan performance
* Move batching delay to main Synchornizer thread
* Reverts addition of Robintrack protobuf definitions
* Give priority to config history provider if set
* Add Estimize protobuffing
- Add Estimize protobuffing support. Adding unit tests
* Always dispose of data queue handler
Co-authored-by: Gerardo Salazar <gsalaz9800@gmail.com>
Co-authored-by: Adalyat Nazirov <aenazirov@gmail.com>
- Adding new DataPermissionManager that will own the existing
datachannelProvider. Will assert configurations before added to the
data feed. Adding unit tests
- `UserDefinedUniverse` will no longer be removed as a data subscription.
- When `algorithm.AddData()` is called a universe selection data point
will be added to the `UserDefinedUniverse` subscription to trigger
selection and add the requested data.
- `DataManager` will make sure an active subscriptions
`SubscriptionDataConfig` will be present in the configuration collection
- Adding unit and regression tests
- Enable parallel workers for history requests, improving performance.
- Adding SubscriptionUtils to be used by backtesting through the FileSystemDataFeed and
SubscriptionHistoryProvider and live deployments which use the
SubscriptionHistoryProvider in the core
- Fix some timezone issues when using local
start/end time instead of UTC
- Adding missing `Enumerator.Dispose()` to be called by the worker
- Reducing the filesystem upper limit data count for coarse file to 1GB
of ram. Currently in master as is, for long backtests, will load all
data, which is roughly 9GB of ram
- Deleting `ParallelRunnerController` and `ParallelRunnerWorker`.
Replacing them for `Tasks`. The consumer, `EnqueueableEnumerator` will
directly spin up a new producer Task when he is running low on items.
- Moving `Splits`, `Dividends`, `Mappings`, `Delistings`, from the
`SubscriptionDataReader` into there own enumerator, inheriting from
new `CorporateEventBaseEnumerator`
- Adding new helper `static class CorporateEventEnumeratorFactory` to
create the new enumerators
- Adding unit tests for new `CorporateEventBaseEnumerator`
- Adding some extra checks to `HourSplitRegressionAlgorithm` and
`DelistingEventsAlgorithm`
- Removing unneeded `CashBook` instance to create a new `TimeSlice`
- Adding new `TimeSliceFactory`, an instance base class that will
provide methods to create a new `TimeSlice`. Will own the `DateTimeZone`
property.
- Adding new `ISecurityPrice` and `IOptionPrice` that will provide a
reduced interface for accessing price properties and methods used when
creating a new `TimeSlice`
- The class `Subscription` will internally track each `Universe`
`SubscriptionRequest` added or removed
- Adding regression test in which two different `Universe` request the
same `SubscriptionDataConfig` and one of them removes/adds it in a
toggle fashion (fails on current master)
- `UniverseSelection` pending removals will also be tracked by
`Universe`
- `UniverseDecorator` will overwrite the `Universe` member of
`SubscriptionsRequests` at `GetSubscriptionRequests()`. This is due to
`this != this,Universe`
- Adding `Subscription` unit tests covering expected behavior
- Extracting pending removals logic from `UniverseSelection` class into
a new helper class `PendingRemovalsManager`. This new class will keep
track of the `universes` requesting to remove a security. Adding unit tests
- Completly move `DataManager` in front of `DataFeed`. Specifically
`AddSubscription()` and `RemoveSubscription()` implementations. Also
removing IDataFeed.Subscriptions
- Creating new `Synchronizer` which will consolidate and combine `TimeSlice`
streaming for both live and backtests modes. Will live in front of the
`DataManager`
- `SubscriptionSynchronizer` will be the `FrontierTimeProvider` exposed,
and owned, by the new `Synchronizer`
- Updating existing tests
Security instances will require private access to this value in order to
compute close profit.
NOTE: The extent of these changes for simply adding a constructor argument
insinuates that we're missing an abstraction to manage the construction of
these objects, such as a factor object for Security. This will need some
careful TLC in the near future.
- Removing LiveTradingDataFeed bridge. LTDF will yield return Slices
directly from the enumerator, as done in the FSDF. The objective is for
both to share the same logic here and for it to be extracted in a next
PR
- Removing now unnecessary DataFeed running thread
- Adding if (_isStopping) { break; } statement for BaseDataExchange.
This is to avoid the thread to continue looping over the enumerators when stopping. Impacts when there are many securities.
Adds the concept of universe disposal which is requested by an algorithm
through invocation of UniverseManager.Remove, which is invoked via
algorithm.RemoveSecurity. This instructs the data feed that the algorithm
has requested to completely remove the universe and any child subscriptions
from the feed. Security changes are fired for all removed securities.
Internal forex currency conversion feeds are currently added without
a corresponding universe which leads to throwing a null reference exception
when the subscription gets removed at the end of the backtest.
If we pull data and on the same time step that security gets removed,
we can still get that data in OnData(Slice) even though it was removed.
This change filters out removed securities by tracking a reference to
the subscription's disposed flag. Another change was made to wait until
the end of the time step to dispose of subscriptions.
Add IDataFeed.GetSubscription(SubscriptionDataConfig)
Provides a common abstraction for UserDefinedUniverse and the
yet-to-be-impemented ScheduledUniverse. Categorized by the fact
that they don't operate on data, but instead a fixed schedule and
any data required is fetched through external means, such as a
web request.
A large percentage of synchronization time is spent performing time zone conversions.
Previously, it was possible that we may compute the same tz conversion multiple times
on the same piece of data until it's emitted. This change moves the time zone conversion
logic onto the data feed thread right before it puts the data into enqueueable enumerator.
In addition, we're not performing the clone operation at the same moment. This removes
unnecessary computation while moving the necessary computation to the data feed thread to
lighten the load on the synchronization/algorithm thread.
Provides an abstraction point allowing us to easily substitute in a different
synchronization algorithm. This isn't 100% complete as I'm hopefully able to
remove the universe selection dependency from the synchronization implementation.
The concept of subscription frontier is only relevant to the subscription synchronizer.
This change removes the back and forth communication of the frontier data between the
data feed component and the synchronizer component
This caused a change in the CoarseUniverseTop5DollarVolumeAlgorithm. The changed values are
due to the subscriptions now correctly stopping before the turn of the new year. The exact
same number of data points are emitted and the same trades are made, but due to the subscription
stopping slightly sooner (correctly), we lose a benchmark sample which caused a slightly different
beta to be computed (along with dependent statistics).
This is in preparation for the synchronization interface. The synchronization implementation
will be in charge of managing the algorithm manager's time frontier.
Changing the default time zone after SetStartDate is set ends up with the algorithm's
UtcTime being incorrect. This can lead to incorrect start times in the data feed as
well
Previously the number of symbols that could be selected was limited by estimates of RAM usage calculated using the resolution of subscription, now it will only be limited by the physical RAM available.
This was causing unnecessary data processing with algorithms using universe selection, so we should expect some improvements in speed and memory usage.
Another less common issue fixed in this PR is unexpected price spikes on symbols with splits.
This fix only affects backtesting, no change in live trading.
Refactored IDataProvider interface to return stream. The IDataProvider
Fetch method now only takes a key. The IDataProvider
has been reshuffled to be at the bottom of the LeanDataStack. It provides
data to the rest of the Lean stack. The default implementation of IDataProvider reads
data from disc.
All IDataCacheProviders now have constructors which take IDataProviders
and use them to find data on disc.
Renamed DataCacheProvider to ZipDataCacheProvider
Added comments to IDataProvider and it's implementations
Added comments to IDataCacheProvider and it's implementations
Adalyat Nazirov