508 Commits

Author SHA1 Message Date
Stefano Raggi 5f61456df8 Set Account Base Currency from Brokerage in Live Mode (#4806)
* Add property IBrokerage.AccountBaseCurrency

* Set AccountCurrency to brokerage AccountBaseCurrency

* Remove USD AccountCurrency check

* Fix Oanda account base currency

* Fix currency symbol in CashBook.ToString()

* Fix unit tests

* Address review

* Add DebugMessage when changing account currency

* Add debug message for brokerage account base currency

* Fix currency symbol in equity chart and runtime statistics

* Update unit tests
2020-10-09 09:58:01 -03:00
Martin-Molinero 460fd626a6 Simplify BaseWebSocketsBrokerage reconnection (#4758)
* Simplify BaseWebSocketsBrokerage reconnection

- Gdax does not emit time pulse so when no subscription was present the
  DefaultConnectionHandler would trigger a reconnect. Replacing for
  directly resubscribing on reconnect.
- TPV == 0 will send warning message instead of being an exception

* Fix unit test
2020-09-25 13:13:33 -03:00
Adalyat Nazirov 75eac27795 Bug 4600 brokerage unsubscribe symbol granular (#4640)
* bitfinex unsubscribe impl

* IB implementation

* Alpaca Brokerage

* Fxcm Brokerage

* Bitfinex remake

* GDAX Brokerage

* move & rename DataQueueHandler subscription manager to better place for using in ToolBox

* performance tuning

use unified Channel class everywhere
BaseWebsocketBrokerage.GetSubscribed method relies on DQHSubscriptionManager

* TradierBrokerage implementation

* OandaBrokerage implementation

* FakeDataQueue subcription manager

* Fake SubscriptionManager for testing

* CoinApi implementation

* IEXDataQueueHandler implementation

* IQFeed Implementation

* unit tests

* fix SubscribeSymbols params

* thread safe subscribed symbols

* don't need to lock Keys prop, because it's thread safe

* accurate PolygonDataQueueHandler subscribe/unsubscribe methods

* Alpaca isnt DataQueueHAndler anymore

* remove unused variable

* remove redundant hashset

* fix coin api subscribe method

* disclaimer

* prettify code

* race condition?

* fix symbol conversion

* log if unsubscribed; change Channel Id type

* requested changes

* pass GetChannelName func as required parameter

* centralized logs

* use single name for all idqh with no difference in tick type

* change Oanda resubscribe method

* CanSubscribe doesn't change instance state - can be marked as static.

* change Oanda Subscription tracking

* style changes

* bitfinex fix

* fix spelling
2020-09-14 20:11:44 -03:00
Martin-Molinero 9cb2452025 Oanda default Forex Market (#4706)
* Oanda default forex Market

- Use Oanda as default forex Market since it has more pairs.
- Remove FXCM data add Oanda equivalente data.
- Update unit and regression tests

* Address reviews

- Revert FXCM data removal
- Remove unrequired commented code

* Fix rebase
2020-09-14 16:43:23 -03:00
Adalyat Nazirov 7bb143b215 Bug 4031 Change data depending on configuration (#4650)
* Calculate both raw and adjuasted prices for backtesting

* disable second price factoring

* move and reuse method

* test coverage for new methods

* reuse scaling method

* reuse subscriptionData.Create method

* removed unused code

* regression test

* switch to aapl

* fix regression test output

* more asserts

* fix comments - reduce shortcuts and abbrevation

* more comments

* merge parameters

* reduce number of getting price factors

* fix tests

* fix tests

* fix regression tests

* calculate TotalReturn on demand

* include TotalReturn calculations

* perf tuning

* more unit tests for SubscriptionData.Create

* simplify things - store and return only raw and precalculated data

* fix regression tests; change it back

* factor equals 1 for Raw data

* small changes

* follow code style

* implement backward compatibility
2020-09-09 18:40:19 -03:00
Adalyat Nazirov 808fa327e2 Fix duplicated history entries when contains daylight saving time change (#4700)
* regression test

* fix

* add comments

* more humanic implementation

* unit tests

* more comments
2020-09-09 13:57:40 -03:00
Martin-Molinero 98d3a98656 Inline some methods for performance (#4696)
- Add AggressiveInlining for some methods
2020-09-04 18:13:46 -03:00
Martin-Molinero 3609340281 Add internal subscriptions always (#4690)
* Add set market price during extended market hours

- Set market prices during extended market hours for live trading.
  Adding unit test

* Add assert on internal data count
2020-09-03 18:15:21 -07:00
Martin-Molinero 400a0d42d9 Add internal subscription manager (#4678)
* Add internal subscription manager

- Add InternalSubscriptionManager that will handle internal
  Subscription. Replaces the realtime updates
- Fix thread race condition in the TimeTriggeredUniverseSubscription, we
  have one thread injecting data points, the main algorithm thread, and
  the base exchange is pulling from it
- Fixes for FakeDataQueue
- Adding unit tests

* Address reviews and fixes

- Internal subscription will use extended market hours
- Only sample charts accordingly
- Get api-url once
2020-09-01 21:22:22 -03:00
Adalyat Nazirov ae11e9ce43 allow Composer to create IDataQueueHandler instances (#4677) 2020-09-01 11:23:08 -03:00
Martin-Molinero f4ed48fce2 Always use AlgorithmHandler.DataProvider (#4658)
- `UniverseSelection` and `Research` will always use
  `AlgorithmHandler.DataProvider` instance instead of
  `DefaultDataProvider`
- Remove Compression at test/app.config since the dll isn't required
- Add missing license header
2020-08-27 16:07:56 -03:00
Stefano Raggi 14dd52ba00 Add zero TotalPortfolioValue check in BrokerageSetupHandler (#4642)
* Add empty cashbook check in BrokerageSetupHandler

* Update check for zero TotalPortfolioValue

* Move check for zero TPV after setting currency conversions

* Fix zero conversion rate for account currency + add unit tests

* Fix unit test
2020-08-21 15:09:29 -03:00
Martin-Molinero 4348e830ba Revert "Add zero TotalPortfolioValue check in BrokerageSetupHandler (#4639)" (#4641)
This reverts commit c85f608444.
2020-08-19 15:30:13 -07:00
Stefano Raggi c85f608444 Add zero TotalPortfolioValue check in BrokerageSetupHandler (#4639)
* Add empty cashbook check in BrokerageSetupHandler

* Update check for zero TotalPortfolioValue
2020-08-19 12:55:43 -03:00
Martin-Molinero 9cdb4a91c5 Refactor live data feed (#4636)
* Live Coarse universe refactor

- Live trading will source Coarse and Fine fundamental data directly
  from disk. Updating unit tests.

* Adds ILiveDataProvider interface

  * Adds wrapper for IDataQueueHandler implementations

  * Replaces IDataQueueHandler with ILiveDataProvider in
    LiveTradingDataFeed

  * Edits IDataQueueHandler documentation

* Maintains aggregation for current IDQH impls and skips for ILDF impls

  * Note: No unit test was created for this method, go back and TODO

* Protobuf Market data

- Adding protobuf support for Ticks, TradeBars and QuoteBars. Adding
  unit tests.

* Adds unit tests for LiveDataAggregator changes

  * Fixes bug where custom data was not handled as it was before
  * Fixes race condition bug because of variable reuse in class

* Add protobuf extension serialization

* Fixes for protobuf serialization

* Refactor

* Fix OptionChainUniverse

* replace BaseDataExchange pumping ticks with consolidators

* AlpacaBrokerage

* BitfinexBrokerage

* GDAXBrokerage

* OandaBrokerage

* InteractiveBrokers

* TradierBrokerage

* FxcmBrokerage

* PaperBrokerage

* etc

* WIP fixes for existing LTDF unit tests

* Fixes more LTDF unit tests

* make IDataAggregator.Update recieving Generic BaseData rather than Tick

* Change IDataQueueHandler.Subscribe method

* Some fixes after adding new commits

* Adds protobuf (de)serialization support for Dividend and Split

* Serialize protobuf with length prefix

* Fix missing LTDF unit tests

* Adds TiingoNews protobuf definitions

* fix comments

* more fixes on IQFeedDataQueueHandler

* disallow putting ticks into enumerator directly

* ScannableEnumerator tests

* fix OandaBrokerage

* AggregationManager unit tests

* fix AlpacaBrokerage tests

* fix InteractiveBrokers

* fix FxcmBrokerage tests

* call AggregationManager.Remove method on unsubscribe

* fix GDAX existing tests

* Fixes, refactor adding more tests for AggregatorManager

* Adds BenzingaNews protobuf definitions and round trip unit test

* Adds missing TiingoNews unit test to Protobuf round trip tests

* Improve sleep sequence of LiveSynchronizer

* need start aggregating first, and then can subscribe

* More test fixes and refactor

- Refactoring AggregationManager and ScannableEnumerator so the last is
  the one that owns the consolidator
- Adding pulse on the main LiveSynchronizer

* Improve performance of LEquityDataSynchronizingEnu

* Add missing Set job packet method

* Minor performance improvements

* Improvements add test timeout

- Improvements adding test timeout to find blocking test in travis

* Improve aggregationManager performance

* Testing improvements for travis

* Remove test timeouts

* More test fixes

- Adding more missing dispose calls and improving determinism

* fix IEXDataQueueHandler and tests

* Final tweaks to LTDF tests

* more AggregationManager tests

* consume and log ticks

* fix test: couldn't subscribe to Forex tickers

* change Resolution for all bar configs

* Improve RealTimeScheduleEventServiceAccuracy

* refactoring: move common code to base class

* fixed bug; unsubscribe SubscriptionDataConfig

* Small performance improvement

* Minor fixes

* Avoid Symbol serialization

* Fixes coarse selection in live mode

* Fix for live coarse

* Adds protobuf (de)serialization support for Robintrack

  * Adds round-trip unit test

* Minor performance improvements

* More minor performance improvements

* pass LiveNodePacket through to OandaBrokerage

* Fixes empty list becoming null value when deserializing with protobuf

* Reverts BZ live trading exception removal and fixes tests

* Refactor WorkQueue making it abstract

* Add try catch for composer

* Adds optional data batching period to LiveFillForwardEnumerator

* Override data-queue-handler with config

* Improve PeriodCountConsolidator.Scan performance

* Move batching delay to main Synchornizer thread

* Reverts addition of Robintrack protobuf definitions

* Give priority to config history provider if set

* Add Estimize protobuffing

- Add Estimize protobuffing support. Adding unit tests

* Always dispose of data queue handler

Co-authored-by: Gerardo Salazar <gsalaz9800@gmail.com>
Co-authored-by: Adalyat Nazirov <aenazirov@gmail.com>
2020-08-18 20:21:10 -03:00
Gerardo Salazar 57cbfa2f26 Fixes stalling of LiveFillForwardEnumerator (#4620)
* Adds unit test containing previously failing test case
2020-08-10 18:58:46 -03:00
Stefano Raggi 036d8684c0 IEX DataQueueHandler updates
- Add parameterless constructor (for usage in LEAN config)
- Added config setting for IEX Cloud api key (required for history requests)
2020-07-30 17:32:00 +02:00
Martin Molinero febc4b50ea Fix QuoteBarConsolidator
- TickQuoteBarConsolidator will use previous bar. Updating unit tests
- QuoteBarConsolidator open ask and bid will match previous bar close
  bid and ask. Adding unit tests
2020-06-23 14:42:56 -03:00
Martin Molinero bcc67d2457 DataPermissionManager is used by history provider
- DataPermissionManager will be used by the history provider
- Moving interfaces from Engine to Common
2020-06-12 12:51:14 -03:00
Martin Molinero 452bf27bd9 Adding new DataPermissionManager
- Adding new DataPermissionManager that will own the existing
  datachannelProvider. Will assert configurations before added to the
  data feed. Adding unit tests
2020-06-12 12:34:10 -03:00
Gerardo Salazar 6333081f7d Address self review - Adds missing license header/removes unused imports 2020-05-22 17:20:13 -07:00
Gerardo Salazar a20ac26eb7 Fixes rounding of Time property in SubscriptionData for data that has no time delta.
* The original implementation was made with the idea in mind that bars
    have a time range that they encompass. When the `Time` property is
    rounded down, the period added to `EndTime` ensures that the data
    will not have look-ahead bias.

    However, multiple data sources do not have a time range that the data
    applies to, and is actually point-in-time. For these kinds of
    data, rounding of the data is not preferred since they are supposed
    to represent a point in time.
2020-05-22 17:13:18 -07:00
Stefano Raggi b3cf6315de Add IncludeExpired flag to IDataQueueUniverseProvider.Lookup
This flag was added to enable history requests for expired futures and options contracts.
2020-05-13 11:38:53 +02:00
Martin Molinero 269b8ba204 Remove PsychSignal data
- Remove PsychSignal data since it's no longer available
2020-05-11 18:48:13 -03:00
Martin Molinero f72bb3d513 Fix invalid round down operation
- RoundDown was being performed in exchange time zone but should be done
  in data time zone
2020-05-08 15:10:43 -03:00
Martin Molinero 63bf4e72a0 Revert "Revert "Fix market open missing FillForward bar""
This reverts commit 1ac8146eae.
2020-05-08 12:46:02 -03:00
Martin-Molinero 1ac8146eae Revert "Fix market open missing FillForward bar" 2020-05-08 11:38:59 -03:00
Martin Molinero c226914322 Value types thread safety changes
- Refactor ZipDataCacheProvider to avoid value types thread issues, it
  will now use a timer. Add missing dispose calls.
- Synchronizer won't share the `SubscriptionFrontierTimeProvider`
  instance since it's not thread safe and shouldn't be called
- Fix BacktestingResultHandler `_daysProcessed` thread safety
- Fixing `RealTimeScheduleEventService` thread safety
2020-05-01 16:58:29 -03:00
Martin-Molinero eb5d380f5a Merge pull request #4356 from QuantConnect/bug-4338-value-type-thread-safety
Adding new ReferenceWrapper for structs
2020-05-01 12:16:07 -03:00
Martin Molinero 649b9f010f Fix thread race conditions reading and writing
- Fix for thread race condition reading and writing current time loop.
  Adding unit test
2020-05-01 11:47:19 -03:00
Martin Molinero 1e221c514b Unit test race condition fixes 2020-04-30 17:19:43 -03:00
Jared c939c68f21 Merge pull request #4350 from QuantConnect/feature-4348-extend-idqh
Add Dispose and IsConnected to IDataQueueHandler
2020-04-30 11:24:50 -07:00
Martin-Molinero 5625acbd6f Merge pull request #4328 from QuantConnect/refactor-future-symbol-market
Refactor Future Market
2020-04-28 19:40:04 -03:00
Martin Molinero fa77c33fed Address reviews. Update unit tests
- Updating unit tests which were still using Market.USA for futures
2020-04-28 16:29:17 -03:00
Martin Molinero 5bc9651dc8 Fix more unit tests 2020-04-28 16:29:16 -03:00
Martin Molinero 9a8b1dc8fd Fixes for unit tests 2020-04-28 16:29:15 -03:00
Martin Molinero 149d30f020 Fixes and Improvements
- Removing spaces in MHDB.json file reducing size
- Update tests using wrong future market
2020-04-28 16:29:15 -03:00
Martin Molinero f78142fe32 Fix rebase after data update 2020-04-28 15:53:01 -03:00
Martin Molinero 075e82629a Add more unit tests
- Add more fill forward unit test including a weekend using different
combinations of resolutions
- Add required AIG second data file
- Avoid list in most common FF case
2020-04-28 15:23:07 -03:00
Martin Molinero bf1e4ae19c Fix market open FillForward
- Fix bar not being fill forward on market open
- Adding unit tests which fail in master
2020-04-28 15:23:06 -03:00
Stefano Raggi 7554babad8 Add Dispose and IsConnected to IDataQueueHandler 2020-04-28 17:39:13 +02:00
Martin Molinero 1fcd9588d6 Dispose Timers and threads in tests 2020-04-27 17:46:16 -03:00
Martin Molinero 9ee9cf9fc5 Address reviews 2020-04-27 17:46:10 -03:00
Martin Molinero e73e927c0f Enable parallel unit testing 2020-04-27 17:45:53 -03:00
Martin Molinero de05f15a12 Address review
- Adding live trading unit test
2020-04-24 19:04:41 -03:00
Martin Molinero b3970bdc3b Fixes and rebase
- FineFundamentalSubscriptionEnumeratorFactory will not emit empty fine
  points
2020-04-24 17:21:28 -03:00
Martin Molinero 886e5e6445 Test fixes and updates 2020-04-23 09:59:31 -03:00
Leonardo-Ferreira 951c973d5b Adjusted the OneTimeSetup attribute usage 2020-04-22 18:19:08 -03:00
Leonardo-Ferreira 6b40d1be6d Code clean-up 2020-04-22 18:19:08 -03:00
Leonardo-Ferreira a9b356d7ff Replaced TestFixtureTearDown 2020-04-22 18:16:58 -03:00