Commit Graph

12 Commits

Author SHA1 Message Date
Martin-Molinero 9cdb4a91c5 Refactor live data feed (#4636)
* Live Coarse universe refactor

- Live trading will source Coarse and Fine fundamental data directly
  from disk. Updating unit tests.

* Adds ILiveDataProvider interface

  * Adds wrapper for IDataQueueHandler implementations

  * Replaces IDataQueueHandler with ILiveDataProvider in
    LiveTradingDataFeed

  * Edits IDataQueueHandler documentation

* Maintains aggregation for current IDQH impls and skips for ILDF impls

  * Note: No unit test was created for this method, go back and TODO

* Protobuf Market data

- Adding protobuf support for Ticks, TradeBars and QuoteBars. Adding
  unit tests.

* Adds unit tests for LiveDataAggregator changes

  * Fixes bug where custom data was not handled as it was before
  * Fixes race condition bug because of variable reuse in class

* Add protobuf extension serialization

* Fixes for protobuf serialization

* Refactor

* Fix OptionChainUniverse

* replace BaseDataExchange pumping ticks with consolidators

* AlpacaBrokerage

* BitfinexBrokerage

* GDAXBrokerage

* OandaBrokerage

* InteractiveBrokers

* TradierBrokerage

* FxcmBrokerage

* PaperBrokerage

* etc

* WIP fixes for existing LTDF unit tests

* Fixes more LTDF unit tests

* make IDataAggregator.Update recieving Generic BaseData rather than Tick

* Change IDataQueueHandler.Subscribe method

* Some fixes after adding new commits

* Adds protobuf (de)serialization support for Dividend and Split

* Serialize protobuf with length prefix

* Fix missing LTDF unit tests

* Adds TiingoNews protobuf definitions

* fix comments

* more fixes on IQFeedDataQueueHandler

* disallow putting ticks into enumerator directly

* ScannableEnumerator tests

* fix OandaBrokerage

* AggregationManager unit tests

* fix AlpacaBrokerage tests

* fix InteractiveBrokers

* fix FxcmBrokerage tests

* call AggregationManager.Remove method on unsubscribe

* fix GDAX existing tests

* Fixes, refactor adding more tests for AggregatorManager

* Adds BenzingaNews protobuf definitions and round trip unit test

* Adds missing TiingoNews unit test to Protobuf round trip tests

* Improve sleep sequence of LiveSynchronizer

* need start aggregating first, and then can subscribe

* More test fixes and refactor

- Refactoring AggregationManager and ScannableEnumerator so the last is
  the one that owns the consolidator
- Adding pulse on the main LiveSynchronizer

* Improve performance of LEquityDataSynchronizingEnu

* Add missing Set job packet method

* Minor performance improvements

* Improvements add test timeout

- Improvements adding test timeout to find blocking test in travis

* Improve aggregationManager performance

* Testing improvements for travis

* Remove test timeouts

* More test fixes

- Adding more missing dispose calls and improving determinism

* fix IEXDataQueueHandler and tests

* Final tweaks to LTDF tests

* more AggregationManager tests

* consume and log ticks

* fix test: couldn't subscribe to Forex tickers

* change Resolution for all bar configs

* Improve RealTimeScheduleEventServiceAccuracy

* refactoring: move common code to base class

* fixed bug; unsubscribe SubscriptionDataConfig

* Small performance improvement

* Minor fixes

* Avoid Symbol serialization

* Fixes coarse selection in live mode

* Fix for live coarse

* Adds protobuf (de)serialization support for Robintrack

  * Adds round-trip unit test

* Minor performance improvements

* More minor performance improvements

* pass LiveNodePacket through to OandaBrokerage

* Fixes empty list becoming null value when deserializing with protobuf

* Reverts BZ live trading exception removal and fixes tests

* Refactor WorkQueue making it abstract

* Add try catch for composer

* Adds optional data batching period to LiveFillForwardEnumerator

* Override data-queue-handler with config

* Improve PeriodCountConsolidator.Scan performance

* Move batching delay to main Synchornizer thread

* Reverts addition of Robintrack protobuf definitions

* Give priority to config history provider if set

* Add Estimize protobuffing

- Add Estimize protobuffing support. Adding unit tests

* Always dispose of data queue handler

Co-authored-by: Gerardo Salazar <gsalaz9800@gmail.com>
Co-authored-by: Adalyat Nazirov <aenazirov@gmail.com>
2020-08-18 20:21:10 -03:00
Martin Molinero e73e927c0f Enable parallel unit testing 2020-04-27 17:45:53 -03:00
Martin Molinero 1997197f47 Fix failing unit tests 2018-09-18 12:10:42 -03:00
Martin Molinero 41873b2315 Fixing unit tests 2018-08-31 16:06:41 -03:00
devalkeralia 3cc8928a0a removed the IResultHandler from the constructor for DefaultBrokerageMessageHandler 2016-07-20 19:31:14 -04:00
Michael Handschuh f3cc448ad5 Update SecurityExchange.UpdateMarketHours to use segments 2016-01-29 18:12:57 -05:00
Michael Handschuh b64cd05333 Removes old-dead brokerage msg handler tests 2016-01-19 20:28:37 -05:00
snugs 9e660ffcbe Remove usages of Symbol implicits from non-algorithm projects 2015-11-18 18:15:31 -05:00
snugs 988f2bfe0f 'TravisExclude' DefaultBrokerageMessageHandler
These tests have dependencies on TPL and wall-clock time making them
not always pass in Travis, exclude for now.
2015-11-02 13:21:26 -05:00
snugs 83678d4c9b Make DefaultBrokerageMessageHandler tests pass
Issues could arise on single threaded machines not running the thread when expected, give more time
2015-11-02 12:27:57 -05:00
snugs 12f5dfd179 Increase sleep on test 2015-10-31 18:51:43 -04:00
snugs 4521a9aea3 Improve disconnect handling logic
No need to kill the algorithm if there's no open exchanges, wait
at least 15 minutes before killing algo (same as before with IB)

This logic is now centralized via an IBrokerageMessageHandler

Includes two new BrokerageMessageType
    Disconnect
    Reconnect

These messages should be used when a disconnect happens and again when we're reconnected
to tell the message handler the current state
2015-10-29 20:18:22 -04:00