10490 Commits

Author SHA1 Message Date
Michael Handschuh fb2f846159 Bug 4722: Prevent Repetitive Factor File Numerical Precision Warnings (#4742)
* Add DataProviderEventArgs base class for IDataProviderEvents event args

This base class includes a Symbol property. This will empower event listeners
to make decisions based on which security (symbol) raised the event. The immediate
use case is preventing multiple numerical precision messages for the same security.
This pattern can equally be applied to other error messages that are raised each
time a security is added to a universe.

See: #BUG-4722

* Update ConcurrentSet.Add to use ISet<T>.Add returning bool

It's a very common pattern to use if (set.Add(item)) which is enabled
via bool ISet<T>.Add(item) but not enabled via void ICollectiont<T>.Add(item).
This change simples changes the default Add implementation to use the ISet<T>
overload and relegates the ICollection<T>.Add implementation to be explicit.

See: #BUG-4722

* Prevent multiple numerical precision messages for same symbol

If a security is continually added/removed from a universe, then the user will
see this message each time the security is added. This results in some spam.
This change simply remembers for which symbols we've notified the user about the
numerical precision issue.

Fixes: #BUG-4722
9347
2020-09-22 21:06:59 -03:00
Martin-Molinero d03ac0fd90 Gdax reconnection & Bitfinex minor fixes (#4748)
* Fix for BaseWebSocketsBrokerage reconnection

- `BaseWebsocketsBrokerage` will handle reconnection using the
existing `DefaultConnectionHandler` to avoid code duplication.

* Fix for Bitfinex failed Subscription calls

* Fix for bitfinex orderbook

* Fix unit test
9346
2020-09-22 20:47:41 -03:00
Martin-Molinero 56db26d16d Allow specifying email Headers (#4735)
* Allow specifying email Headers

* Address review

- Fix for Python Notification.Email use case
9328
2020-09-18 21:16:43 -03:00
Alexandre Catarino dd8dc473b3 Improve Error Message For Arithmetic Overflow In Decimal Cast of Double (#4728)
* SafeDecimalCast Throws Exception For Non-Finite Numbers

* Fixes Arithmetic Overflow Exception in QCAlgorithm.Trading Methods

Replace decimal cast for `SafeDecimalCast()`.

If the algorithm uses a non-finite number in QCAlgorithm trading methods, it will throw with an user-frieldly exception message.

* Fixes KellyCriterionProbabilityValue Calculation
9317
2020-09-18 09:17:09 -03:00
Colton Sellers b8033c496c Bug 4487 Get Fundamental for CSharp (#4703)
* Add unit tests

* Refactor Py and create C# function

* Update readme to include local

* Refactor solution; fix python cases

* Update tests

* Don't accept null selector for python; Create SelectedData class

* Fix Testing

* Pre review

* Fix tests for Travis

* Test fix V2

* Test fix V3

* Refactor quantbook and fix tests

* Sort list by date

* Move ConvertToSymbols to Python Util

* Address review

* Order dataframe columns by Security ID

* Address review V2

* header for PythonUtilTests
9312
2020-09-16 18:06:01 -03:00
Martin-Molinero 166fee311a Fix for LiveTradingResultHandler Holdings (#4719)
- LiveTradingResultHandler will not send non tradable securities in the
  holdings update
9308
2020-09-16 10:08:34 -03:00
Gerardo Salazar be8e381fba Fixes historical option data not loaded when provided underlying Symbol (#4720)
* Adds unit test to cover changes
9303
2020-09-14 20:39:48 -03:00
Adalyat Nazirov 75eac27795 Bug 4600 brokerage unsubscribe symbol granular (#4640)
* bitfinex unsubscribe impl

* IB implementation

* Alpaca Brokerage

* Fxcm Brokerage

* Bitfinex remake

* GDAX Brokerage

* move & rename DataQueueHandler subscription manager to better place for using in ToolBox

* performance tuning

use unified Channel class everywhere
BaseWebsocketBrokerage.GetSubscribed method relies on DQHSubscriptionManager

* TradierBrokerage implementation

* OandaBrokerage implementation

* FakeDataQueue subcription manager

* Fake SubscriptionManager for testing

* CoinApi implementation

* IEXDataQueueHandler implementation

* IQFeed Implementation

* unit tests

* fix SubscribeSymbols params

* thread safe subscribed symbols

* don't need to lock Keys prop, because it's thread safe

* accurate PolygonDataQueueHandler subscribe/unsubscribe methods

* Alpaca isnt DataQueueHAndler anymore

* remove unused variable

* remove redundant hashset

* fix coin api subscribe method

* disclaimer

* prettify code

* race condition?

* fix symbol conversion

* log if unsubscribed; change Channel Id type

* requested changes

* pass GetChannelName func as required parameter

* centralized logs

* use single name for all idqh with no difference in tick type

* change Oanda resubscribe method

* CanSubscribe doesn't change instance state - can be marked as static.

* change Oanda Subscription tracking

* style changes

* bitfinex fix

* fix spelling
9302
2020-09-14 20:11:44 -03:00
Colton Sellers 12b481f1ce Adjust regression algorithm (#4714)
* Adjust test to show how to legitimately change an order

* Address review

* fix comparator; check Order Time
9298
2020-09-14 17:07:14 -03:00
Martin-Molinero 9cb2452025 Oanda default Forex Market (#4706)
* Oanda default forex Market

- Use Oanda as default forex Market since it has more pairs.
- Remove FXCM data add Oanda equivalente data.
- Update unit and regression tests

* Address reviews

- Revert FXCM data removal
- Remove unrequired commented code

* Fix rebase
9297
2020-09-14 16:43:23 -03:00
Stefano Raggi d8dc03fadc Reuse symbol properties database for currency conversions (#4710)
* Remove invalid symbols from symbol properties db

* Add SymbolPropertiesDatabase.GetSymbolPropertiesList

* Remove symbol list in BitfinexSymbolMapper

* In EnsureCurrencyDataFeed fetch symbols from symbol properties database

* Remove symbol list in OandaSymbolMapper

* Remove unused code

* Address review

- Remove StringComparer.OrdinalIgnoreCase usage
- Rename KnownSymbolStrings to KnownTickers
9295
2020-09-14 15:12:17 -03:00
Gerardo Salazar 124e76cfe8 Adds deployment of packaged stubs for distribution in CI process (#4713) 9293 2020-09-11 17:18:08 -07:00
Martin-Molinero 4a0fb30df5 Fixes for OpenInterest storing (#4712)
- Fix Slice.Get OpenInterest type. Adding unit test
- Fix for SecurityCache that wasn't storing OpenInterest types
- Updateing regression tests to covere these usages
9292
2020-09-11 15:28:27 -07:00
Stefano Raggi 60b8cf76ba Bitfinex Brokerage updates (#4584)
* Upgrade Bitfinex brokerage to API v2

* Fix rebase

* Address review

- use te instead of tu messages for trades
- add missing orderMap removals
- ClientOrderId is now time-based instead of a counter
- minor cleanup

* Trigger build
9288
2020-09-10 17:14:10 -03:00
aarjaneiro 9916a9069c Update PythonPackagesTests.cs (#4673)
* Update PythonPackagesTests.cs

Detecting issues related to https://stackoverflow.com/questions/56957512/pythonnet-missing-addreference-method, which has shown up for me when using lean (https://www.quantconnect.com/forum/discussion/9054/lean-vagrant-box/p1).

* Update PythonPackagesTests.cs

Named test

* Name change of MonoTest to SanityClrInstallation 

Name suggestion by @Martin-Molinero

* Fix minor typo

Co-authored-by: Martin-Molinero <martin@quantconnect.com>
9285
2020-09-10 16:34:37 -03:00
Colton Sellers f135fb8060 Bug Backtesting Brokerage Clones (#4644)
* fix order updates

* Fix option exercise issue

* Regression changes

* Update regressions to reflect fixes

* Refactor handling of order to fully fix #2846

* Regression Algorithm for unit test

* Pre review

* Fix breaking tests

* OrderImmutability Regression Algo

* OrderImmutability Regression Algo Compile

* Address review

* Update regressions with new orderhash
9275
2020-09-09 19:44:56 -03:00
Adalyat Nazirov 7bb143b215 Bug 4031 Change data depending on configuration (#4650)
* Calculate both raw and adjuasted prices for backtesting

* disable second price factoring

* move and reuse method

* test coverage for new methods

* reuse scaling method

* reuse subscriptionData.Create method

* removed unused code

* regression test

* switch to aapl

* fix regression test output

* more asserts

* fix comments - reduce shortcuts and abbrevation

* more comments

* merge parameters

* reduce number of getting price factors

* fix tests

* fix tests

* fix regression tests

* calculate TotalReturn on demand

* include TotalReturn calculations

* perf tuning

* more unit tests for SubscriptionData.Create

* simplify things - store and return only raw and precalculated data

* fix regression tests; change it back

* factor equals 1 for Raw data

* small changes

* follow code style

* implement backward compatibility
9274
2020-09-09 18:40:19 -03:00
Stefano Raggi d7e543736f GDAX Brokerage updates (#4635)
* GDAX Brokerage updates

- Replaced fill detection from trade stream with monitor task
- Order fees for fills are now the real fees paid (previously they were calculated by the brokerage model)
- All unit and integration tests are green

* Address review

- Remove unnecessary signals
- Add "gdax-fill-monitor-timeout" config setting

* Remove user channel
9272
2020-09-09 16:18:10 -03:00
Stefano Raggi 8785af7ad6 Alpaca Brokerage updates (part 2) (#4601)
* Remove IDataQueueHandler from AlpacaBrokerage

- a new IDataQueueHandler implementation for Polygon.io will be added to the ToolBox

* Fix Alpaca Websocket connect not waiting for completion

* Add security type check in Alpaca history

* Fix merge

* Remove aggregator from AlpacaBrokerage
9269
2020-09-09 15:17:10 -03:00
Adalyat Nazirov 808fa327e2 Fix duplicated history entries when contains daylight saving time change (#4700)
* regression test

* fix

* add comments

* more humanic implementation

* unit tests

* more comments
9266
2020-09-09 13:57:40 -03:00
Martin-Molinero 778e3015c8 Revert live trading config IDQH over job packet (#4705) 9265 2020-09-09 12:03:02 -03:00
Martin-Molinero 616b2b8d52 Prioritize job packet history provider (#4701)
- Will prioritize job packet history provider value.
- Improve LTRH logging
9255
2020-09-08 18:45:36 -03:00
Martin-Molinero 98d3a98656 Inline some methods for performance (#4696)
- Add AggressiveInlining for some methods
9247
2020-09-04 18:13:46 -03:00
Derek Melchin 20791d6a9e Add schedule queuing algorithm (#4695)
* Add schedule queuing algorithm

* Add algorithm file to csproj file

* Add c# version of queuing algorithm
2020-09-04 12:34:07 -07:00
Martin-Molinero 3609340281 Add internal subscriptions always (#4690)
* Add set market price during extended market hours

- Set market prices during extended market hours for live trading.
  Adding unit test

* Add assert on internal data count
9241
2020-09-03 18:15:21 -07:00
Colton Sellers 8edc89a4ff Feature Python Option Filtering Support (#4667)
* Support List and OptionFilterUniverse for Py filter

* Regression algorithm for testing

* Unit Tests

* Fix for process

* Tighten filters to reduce load on automated testing

* Address review v2
9237
2020-09-02 21:38:16 -03:00
Martin-Molinero 718dc02173 Add finish log for volatility warmup (#4684) 9236 2020-09-02 16:54:39 -07:00
Martin-Molinero ec5ff53566 Fix key not found exception at InternalSubsManager (#4679)
* Fix key not found exception at InternalSubsManager

- Fix key not found exception
- Fix backtesting chart sampling

* Add comment about PreviousUtcSampleTime
9228
2020-09-01 22:32:21 -03:00
Martin-Molinero 400a0d42d9 Add internal subscription manager (#4678)
* Add internal subscription manager

- Add InternalSubscriptionManager that will handle internal
  Subscription. Replaces the realtime updates
- Fix thread race condition in the TimeTriggeredUniverseSubscription, we
  have one thread injecting data points, the main algorithm thread, and
  the base exchange is pulling from it
- Fixes for FakeDataQueue
- Adding unit tests

* Address reviews and fixes

- Internal subscription will use extended market hours
- Only sample charts accordingly
- Get api-url once
2020-09-01 21:22:22 -03:00
Adalyat Nazirov ae11e9ce43 allow Composer to create IDataQueueHandler instances (#4677) 2020-09-01 11:23:08 -03:00
Colton Sellers d3ccc14fcd Unit test for RegisterIndicator in Python (#4676) 9215 2020-08-31 19:11:24 -03:00
QuantConnect Server Applications 90e911f22c Adds autogenerated Python stubs via Travis for QCAlgorithm (Build 14139) (#4671)
Co-authored-by: Python Stubs Deployer <stubs-deploy@quantconnect.com>
9211
2020-08-29 13:10:37 -03:00
Colton Sellers 34676f0c46 Bug 4668 python register indicator (#4670)
* Convert to timespan if possible

* Fix to support both timespans and custom consolidators

* Adjust tests for new wrapper requirements

* Cleanup
9209
2020-08-29 02:53:32 -03:00
QuantConnect Server Applications d4ca27f93f Adds autogenerated Python stubs via Travis for QCAlgorithm (Build 14115) (#4662)
Co-authored-by: Python Stubs Deployer <stubs-deploy@quantconnect.com>
9208
2020-08-28 16:43:17 -03:00
Martin-Molinero f4ed48fce2 Always use AlgorithmHandler.DataProvider (#4658)
- `UniverseSelection` and `Research` will always use
  `AlgorithmHandler.DataProvider` instance instead of
  `DefaultDataProvider`
- Remove Compression at test/app.config since the dll isn't required
- Add missing license header
9203
2020-08-27 16:07:56 -03:00
Gerardo Salazar ce29a68f85 Deletes residual python files left over by stubs generation (#4666) 9201 2020-08-27 11:47:48 -07:00
Stefano Raggi 375f192f07 Add PolygonDataQueueHandler (#4604)
* Add PolygonDataQueueHandler

* Add history provider and downloader for Polygon

* Add aggregator to PolygonDataQueueHandler

* Address review

- Removed duplication in message classes
- Added public Subscribe/Unsubscribe methods in PolygonWebSocketClientWrapper
- Added history requests for Forex and Crypto

* Address review

- Add security type and market arguments to downloader
- Fix time zone bug in downloader

* Remove unnecessary locks

* Add Polygon history for all resolutions

- Equity: trades and quotes
- Forex: quotes only
- Crypto: trades only
9194
2020-08-27 12:09:20 -03:00
Gerardo Salazar ef9ca98bd9 Fixes bug where build was attempted on PR submission (#4661)
* Fixes potential stale commits by changing --force-with-lease to -f
9189
2020-08-26 16:05:33 -07:00
Gerardo Salazar aded66ec5b Adds Python stubs location definition for PyCharm and Visual Studio Code (#4657) 9188 2020-08-26 14:17:33 -07:00
Colton Sellers 59ce008725 typo fix (#4659) 9187 2020-08-26 14:16:42 -07:00
Mathieu Paquette 46e10c0a27 feat(ToolBox/IQFeed): remove memory pressure from GroupBy when fetching Tick (#4646)
#4645
9185
2020-08-26 17:46:34 -03:00
Gerardo Salazar 2431cfabe7 Updates .travis.yml to include build/deploy scripts for Python stubs (autocomplete) (#4656)
* Updates .travis.yml to include build/deploy scripts for py stubs

  * Adds stubs generator script for CI

* Silences curl output
9184
2020-08-26 13:19:54 -07:00
Colton Sellers b8674731a5 Feature 2456 custom Python consolidator support (#4637)
* DataConsolidator Wrapper for Python Consolidators

* Regression Unit Test

* Refactor Regression test

* Bad test fix

* pre review

* self review

* Add RegisterIndicator for Python Consolidator

* Python base class for consolidators

* Modify regression algo to register indicator

* unit test - attach event

* Test fix

* Fix test python imports

* Add license header file and null check

Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
9180
2020-08-25 17:26:55 -03:00
Alexandre Catarino a14f8f1f47 Refactors SecurityCache to Support Access to Tick of Different Type (#4621)
* Refactors SecurityCache to Support Access to Tick of Different Type

Adds two read-only lists of BaseData to save the last list of Tick of TickType.Trade and TickType.Quote. Changes methods accordingly to get and set these lists.

* Addresses Peer Review

The assignment of `_lastData` is moved to keep the current behavior of not beeing set with fill forward data.
`Reset` and `ShareTypeCacheInstance` implements the new class objects.

Adds unit tests for `Reset` and `ShareTypeCacheInstance`.

* Only Cache Non Fill-Forward Data

Undo changes that made AddData cache non fill-forward data.
9171
2020-08-24 10:37:50 -03:00
Jovad Uribe fc6ccdbc11 Feature emv indicator addition (#4591)
* Create EaseOfMovementValue.cs

Added ease of movement file

* Update EaseOfMovementValue.cs

Added calculation for EMV and return its value

* Update EaseOfMovementValue.cs

* Update EaseOfMovementValue.cs

* Update EaseOfMovementValue.cs

* Update EaseOfMovementValue.cs

Rearranged code and removed all IndicatorBases

* Update EaseOfMovementValue.cs

* Update EaseOfMovementValue.cs

Added Min and Max Indicator

* Added Tests and Compile

* Fixed Bugs and Removed Reset

* Added Current Value and revereted to Bar data

* Fixed test file and refined indicator file

* TradeBar to IBaseDataBar

* Bug fixes

* bug fix

* Switching to TradeBar and attempting to fix Volume bug

There are two bugs that I have been having trouble fixing. 1. Cannot implicitly convert decimal to int (simple fix but cannot find where bug is taking place)
2. IBaseDataBar does not contain a definition for Volume

* Update EaseOfMovementValueTests.cs

* bug fix

* added data

* updated assertion

* added reset

* Update EaseOfMovementValueTests.cs

* Update EaseOfMovementValue.cs

* Update spy_emv.txt

I had the wrong test data in. Was throwing failed test for many pull requests.

* Update EaseOfMovementValue.cs

* Update EaseOfMovementValue.cs

* Update EaseOfMovementValue.cs

* Update EaseOfMovementValue.cs

* Cleaned Data

* Bug fixes

Fixed zero division error. Used better Test Data.

* removed readonly from _previous...price

* Update EaseOfMovementValue.cs

* Update EaseOfMovementValue.cs

* Update EaseOfMovementValue.cs

* bug fix

* Test Bug Fix

* EMV data from online

* Cosmetics

* Out of bounds fix

* Update EaseOfMovementValueTests.cs

* Update spy_emv.txt

* Update spy_emv.txt

* Added changes requested

Placed constructor first, fixed nullable type if statement, set 10,000 to default argument, added SMA.

* Update EaseOfMovementValue.cs

added variables

* Update EaseOfMovementValue.cs

* Update EaseOfMovementValue.cs

* Update EaseOfMovementValue.cs

* Fixed bugs

* Changed Delta, Added Assert

Create Indicator -> Update EMV -> Assert Status. Also changed delta from 1 to 0.000001 to improve test accuracy.

* Added unit test testing the SMA
9170
2020-08-24 09:54:58 -03:00
Martin-Molinero ccd31bdfab Minor improvements (#4643)
* Minor improvements

- Make RH interval configurable
- Fix double to decimal casts
- Fix FDQ

* Revert safe decimal usage
9163
2020-08-21 21:18:29 -03:00
Stefano Raggi 14dd52ba00 Add zero TotalPortfolioValue check in BrokerageSetupHandler (#4642)
* Add empty cashbook check in BrokerageSetupHandler

* Update check for zero TotalPortfolioValue

* Move check for zero TPV after setting currency conversions

* Fix zero conversion rate for account currency + add unit tests

* Fix unit test
9157
2020-08-21 15:09:29 -03:00
Mathieu Paquette 2d46674fb7 feat(Toolbox\IQFeed): optimized download history for IQFeed (#4302)
* feat(Toolbox\IQFeed): optimized download history for IQFeed

Fixes #4301

* feat(IQFeed/ToolBox): code review comments

* feat(IQFeed/ToolBox): add support for quotes and trades

* feat(IQFeed/ToolBox): trade constructor change

* feat(IQFeed/ToolBox): always apply timezone conversion

* feat(IQFeed/ToolBox): fix interval start
9154
2020-08-21 11:26:46 -03:00
Mathieu Paquette 554b7345bb feat(IQFeed/ToolBox): add IQFeed support for docker (#4632)
* feat(IQFeed/ToolBox): add IQFeed support for docker

#4631

* feat(IQFeed/ToolBox): add IQFeed documentation for docker

#4631

* feat(IQFeed/ToolBox): code review comments

#4631
9139
2020-08-19 18:14:37 -07:00
Martin-Molinero 4348e830ba Revert "Add zero TotalPortfolioValue check in BrokerageSetupHandler (#4639)" (#4641)
This reverts commit c85f608444.
9134
2020-08-19 15:30:13 -07:00