10465 Commits

Author SHA1 Message Date
Colton Sellers 8edc89a4ff Feature Python Option Filtering Support (#4667)
* Support List and OptionFilterUniverse for Py filter

* Regression algorithm for testing

* Unit Tests

* Fix for process

* Tighten filters to reduce load on automated testing

* Address review v2
9237
2020-09-02 21:38:16 -03:00
Martin-Molinero 718dc02173 Add finish log for volatility warmup (#4684) 9236 2020-09-02 16:54:39 -07:00
Martin-Molinero ec5ff53566 Fix key not found exception at InternalSubsManager (#4679)
* Fix key not found exception at InternalSubsManager

- Fix key not found exception
- Fix backtesting chart sampling

* Add comment about PreviousUtcSampleTime
9228
2020-09-01 22:32:21 -03:00
Martin-Molinero 400a0d42d9 Add internal subscription manager (#4678)
* Add internal subscription manager

- Add InternalSubscriptionManager that will handle internal
  Subscription. Replaces the realtime updates
- Fix thread race condition in the TimeTriggeredUniverseSubscription, we
  have one thread injecting data points, the main algorithm thread, and
  the base exchange is pulling from it
- Fixes for FakeDataQueue
- Adding unit tests

* Address reviews and fixes

- Internal subscription will use extended market hours
- Only sample charts accordingly
- Get api-url once
2020-09-01 21:22:22 -03:00
Adalyat Nazirov ae11e9ce43 allow Composer to create IDataQueueHandler instances (#4677) 2020-09-01 11:23:08 -03:00
Colton Sellers d3ccc14fcd Unit test for RegisterIndicator in Python (#4676) 9215 2020-08-31 19:11:24 -03:00
QuantConnect Server Applications 90e911f22c Adds autogenerated Python stubs via Travis for QCAlgorithm (Build 14139) (#4671)
Co-authored-by: Python Stubs Deployer <stubs-deploy@quantconnect.com>
9211
2020-08-29 13:10:37 -03:00
Colton Sellers 34676f0c46 Bug 4668 python register indicator (#4670)
* Convert to timespan if possible

* Fix to support both timespans and custom consolidators

* Adjust tests for new wrapper requirements

* Cleanup
9209
2020-08-29 02:53:32 -03:00
QuantConnect Server Applications d4ca27f93f Adds autogenerated Python stubs via Travis for QCAlgorithm (Build 14115) (#4662)
Co-authored-by: Python Stubs Deployer <stubs-deploy@quantconnect.com>
9208
2020-08-28 16:43:17 -03:00
Martin-Molinero f4ed48fce2 Always use AlgorithmHandler.DataProvider (#4658)
- `UniverseSelection` and `Research` will always use
  `AlgorithmHandler.DataProvider` instance instead of
  `DefaultDataProvider`
- Remove Compression at test/app.config since the dll isn't required
- Add missing license header
9203
2020-08-27 16:07:56 -03:00
Gerardo Salazar ce29a68f85 Deletes residual python files left over by stubs generation (#4666) 9201 2020-08-27 11:47:48 -07:00
Stefano Raggi 375f192f07 Add PolygonDataQueueHandler (#4604)
* Add PolygonDataQueueHandler

* Add history provider and downloader for Polygon

* Add aggregator to PolygonDataQueueHandler

* Address review

- Removed duplication in message classes
- Added public Subscribe/Unsubscribe methods in PolygonWebSocketClientWrapper
- Added history requests for Forex and Crypto

* Address review

- Add security type and market arguments to downloader
- Fix time zone bug in downloader

* Remove unnecessary locks

* Add Polygon history for all resolutions

- Equity: trades and quotes
- Forex: quotes only
- Crypto: trades only
9194
2020-08-27 12:09:20 -03:00
Gerardo Salazar ef9ca98bd9 Fixes bug where build was attempted on PR submission (#4661)
* Fixes potential stale commits by changing --force-with-lease to -f
9189
2020-08-26 16:05:33 -07:00
Gerardo Salazar aded66ec5b Adds Python stubs location definition for PyCharm and Visual Studio Code (#4657) 9188 2020-08-26 14:17:33 -07:00
Colton Sellers 59ce008725 typo fix (#4659) 9187 2020-08-26 14:16:42 -07:00
Mathieu Paquette 46e10c0a27 feat(ToolBox/IQFeed): remove memory pressure from GroupBy when fetching Tick (#4646)
#4645
9185
2020-08-26 17:46:34 -03:00
Gerardo Salazar 2431cfabe7 Updates .travis.yml to include build/deploy scripts for Python stubs (autocomplete) (#4656)
* Updates .travis.yml to include build/deploy scripts for py stubs

  * Adds stubs generator script for CI

* Silences curl output
9184
2020-08-26 13:19:54 -07:00
Colton Sellers b8674731a5 Feature 2456 custom Python consolidator support (#4637)
* DataConsolidator Wrapper for Python Consolidators

* Regression Unit Test

* Refactor Regression test

* Bad test fix

* pre review

* self review

* Add RegisterIndicator for Python Consolidator

* Python base class for consolidators

* Modify regression algo to register indicator

* unit test - attach event

* Test fix

* Fix test python imports

* Add license header file and null check

Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
9180
2020-08-25 17:26:55 -03:00
Alexandre Catarino a14f8f1f47 Refactors SecurityCache to Support Access to Tick of Different Type (#4621)
* Refactors SecurityCache to Support Access to Tick of Different Type

Adds two read-only lists of BaseData to save the last list of Tick of TickType.Trade and TickType.Quote. Changes methods accordingly to get and set these lists.

* Addresses Peer Review

The assignment of `_lastData` is moved to keep the current behavior of not beeing set with fill forward data.
`Reset` and `ShareTypeCacheInstance` implements the new class objects.

Adds unit tests for `Reset` and `ShareTypeCacheInstance`.

* Only Cache Non Fill-Forward Data

Undo changes that made AddData cache non fill-forward data.
9171
2020-08-24 10:37:50 -03:00
Jovad Uribe fc6ccdbc11 Feature emv indicator addition (#4591)
* Create EaseOfMovementValue.cs

Added ease of movement file

* Update EaseOfMovementValue.cs

Added calculation for EMV and return its value

* Update EaseOfMovementValue.cs

* Update EaseOfMovementValue.cs

* Update EaseOfMovementValue.cs

* Update EaseOfMovementValue.cs

Rearranged code and removed all IndicatorBases

* Update EaseOfMovementValue.cs

* Update EaseOfMovementValue.cs

Added Min and Max Indicator

* Added Tests and Compile

* Fixed Bugs and Removed Reset

* Added Current Value and revereted to Bar data

* Fixed test file and refined indicator file

* TradeBar to IBaseDataBar

* Bug fixes

* bug fix

* Switching to TradeBar and attempting to fix Volume bug

There are two bugs that I have been having trouble fixing. 1. Cannot implicitly convert decimal to int (simple fix but cannot find where bug is taking place)
2. IBaseDataBar does not contain a definition for Volume

* Update EaseOfMovementValueTests.cs

* bug fix

* added data

* updated assertion

* added reset

* Update EaseOfMovementValueTests.cs

* Update EaseOfMovementValue.cs

* Update spy_emv.txt

I had the wrong test data in. Was throwing failed test for many pull requests.

* Update EaseOfMovementValue.cs

* Update EaseOfMovementValue.cs

* Update EaseOfMovementValue.cs

* Update EaseOfMovementValue.cs

* Cleaned Data

* Bug fixes

Fixed zero division error. Used better Test Data.

* removed readonly from _previous...price

* Update EaseOfMovementValue.cs

* Update EaseOfMovementValue.cs

* Update EaseOfMovementValue.cs

* bug fix

* Test Bug Fix

* EMV data from online

* Cosmetics

* Out of bounds fix

* Update EaseOfMovementValueTests.cs

* Update spy_emv.txt

* Update spy_emv.txt

* Added changes requested

Placed constructor first, fixed nullable type if statement, set 10,000 to default argument, added SMA.

* Update EaseOfMovementValue.cs

added variables

* Update EaseOfMovementValue.cs

* Update EaseOfMovementValue.cs

* Update EaseOfMovementValue.cs

* Fixed bugs

* Changed Delta, Added Assert

Create Indicator -> Update EMV -> Assert Status. Also changed delta from 1 to 0.000001 to improve test accuracy.

* Added unit test testing the SMA
9170
2020-08-24 09:54:58 -03:00
Martin-Molinero ccd31bdfab Minor improvements (#4643)
* Minor improvements

- Make RH interval configurable
- Fix double to decimal casts
- Fix FDQ

* Revert safe decimal usage
9163
2020-08-21 21:18:29 -03:00
Stefano Raggi 14dd52ba00 Add zero TotalPortfolioValue check in BrokerageSetupHandler (#4642)
* Add empty cashbook check in BrokerageSetupHandler

* Update check for zero TotalPortfolioValue

* Move check for zero TPV after setting currency conversions

* Fix zero conversion rate for account currency + add unit tests

* Fix unit test
9157
2020-08-21 15:09:29 -03:00
Mathieu Paquette 2d46674fb7 feat(Toolbox\IQFeed): optimized download history for IQFeed (#4302)
* feat(Toolbox\IQFeed): optimized download history for IQFeed

Fixes #4301

* feat(IQFeed/ToolBox): code review comments

* feat(IQFeed/ToolBox): add support for quotes and trades

* feat(IQFeed/ToolBox): trade constructor change

* feat(IQFeed/ToolBox): always apply timezone conversion

* feat(IQFeed/ToolBox): fix interval start
9154
2020-08-21 11:26:46 -03:00
Mathieu Paquette 554b7345bb feat(IQFeed/ToolBox): add IQFeed support for docker (#4632)
* feat(IQFeed/ToolBox): add IQFeed support for docker

#4631

* feat(IQFeed/ToolBox): add IQFeed documentation for docker

#4631

* feat(IQFeed/ToolBox): code review comments

#4631
9139
2020-08-19 18:14:37 -07:00
Martin-Molinero 4348e830ba Revert "Add zero TotalPortfolioValue check in BrokerageSetupHandler (#4639)" (#4641)
This reverts commit c85f608444.
9134
2020-08-19 15:30:13 -07:00
Stefano Raggi c85f608444 Add zero TotalPortfolioValue check in BrokerageSetupHandler (#4639)
* Add empty cashbook check in BrokerageSetupHandler

* Update check for zero TotalPortfolioValue
9130
2020-08-19 12:55:43 -03:00
Martin-Molinero 9cdb4a91c5 Refactor live data feed (#4636)
* Live Coarse universe refactor

- Live trading will source Coarse and Fine fundamental data directly
  from disk. Updating unit tests.

* Adds ILiveDataProvider interface

  * Adds wrapper for IDataQueueHandler implementations

  * Replaces IDataQueueHandler with ILiveDataProvider in
    LiveTradingDataFeed

  * Edits IDataQueueHandler documentation

* Maintains aggregation for current IDQH impls and skips for ILDF impls

  * Note: No unit test was created for this method, go back and TODO

* Protobuf Market data

- Adding protobuf support for Ticks, TradeBars and QuoteBars. Adding
  unit tests.

* Adds unit tests for LiveDataAggregator changes

  * Fixes bug where custom data was not handled as it was before
  * Fixes race condition bug because of variable reuse in class

* Add protobuf extension serialization

* Fixes for protobuf serialization

* Refactor

* Fix OptionChainUniverse

* replace BaseDataExchange pumping ticks with consolidators

* AlpacaBrokerage

* BitfinexBrokerage

* GDAXBrokerage

* OandaBrokerage

* InteractiveBrokers

* TradierBrokerage

* FxcmBrokerage

* PaperBrokerage

* etc

* WIP fixes for existing LTDF unit tests

* Fixes more LTDF unit tests

* make IDataAggregator.Update recieving Generic BaseData rather than Tick

* Change IDataQueueHandler.Subscribe method

* Some fixes after adding new commits

* Adds protobuf (de)serialization support for Dividend and Split

* Serialize protobuf with length prefix

* Fix missing LTDF unit tests

* Adds TiingoNews protobuf definitions

* fix comments

* more fixes on IQFeedDataQueueHandler

* disallow putting ticks into enumerator directly

* ScannableEnumerator tests

* fix OandaBrokerage

* AggregationManager unit tests

* fix AlpacaBrokerage tests

* fix InteractiveBrokers

* fix FxcmBrokerage tests

* call AggregationManager.Remove method on unsubscribe

* fix GDAX existing tests

* Fixes, refactor adding more tests for AggregatorManager

* Adds BenzingaNews protobuf definitions and round trip unit test

* Adds missing TiingoNews unit test to Protobuf round trip tests

* Improve sleep sequence of LiveSynchronizer

* need start aggregating first, and then can subscribe

* More test fixes and refactor

- Refactoring AggregationManager and ScannableEnumerator so the last is
  the one that owns the consolidator
- Adding pulse on the main LiveSynchronizer

* Improve performance of LEquityDataSynchronizingEnu

* Add missing Set job packet method

* Minor performance improvements

* Improvements add test timeout

- Improvements adding test timeout to find blocking test in travis

* Improve aggregationManager performance

* Testing improvements for travis

* Remove test timeouts

* More test fixes

- Adding more missing dispose calls and improving determinism

* fix IEXDataQueueHandler and tests

* Final tweaks to LTDF tests

* more AggregationManager tests

* consume and log ticks

* fix test: couldn't subscribe to Forex tickers

* change Resolution for all bar configs

* Improve RealTimeScheduleEventServiceAccuracy

* refactoring: move common code to base class

* fixed bug; unsubscribe SubscriptionDataConfig

* Small performance improvement

* Minor fixes

* Avoid Symbol serialization

* Fixes coarse selection in live mode

* Fix for live coarse

* Adds protobuf (de)serialization support for Robintrack

  * Adds round-trip unit test

* Minor performance improvements

* More minor performance improvements

* pass LiveNodePacket through to OandaBrokerage

* Fixes empty list becoming null value when deserializing with protobuf

* Reverts BZ live trading exception removal and fixes tests

* Refactor WorkQueue making it abstract

* Add try catch for composer

* Adds optional data batching period to LiveFillForwardEnumerator

* Override data-queue-handler with config

* Improve PeriodCountConsolidator.Scan performance

* Move batching delay to main Synchornizer thread

* Reverts addition of Robintrack protobuf definitions

* Give priority to config history provider if set

* Add Estimize protobuffing

- Add Estimize protobuffing support. Adding unit tests

* Always dispose of data queue handler

Co-authored-by: Gerardo Salazar <gsalaz9800@gmail.com>
Co-authored-by: Adalyat Nazirov <aenazirov@gmail.com>
9125
2020-08-18 20:21:10 -03:00
michael-sena a128f8bb2e Allow LeanDataWriter to create zip files for futures data (#4569)
* Allow LeanDataWriter to append to zip data files

* Use the data directory provided to the writer instead of the global value

* Disregard the time-portion of an input date

* Overwrite zip entries when creating futures data files

* Minor tweak and adding unit test

Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
9121
2020-08-18 17:44:02 -03:00
Colton Sellers 3ef6a761ad Feature research docker scripts (#4622)
* Add research docker launch script

* Internalize creation of cfg and dir to dockerfile

* Implement bash version of notebook run script

* quick fix to use docker.cfg

* update readme

* typo fix

* Review tweaks

* Tweaks
9110
2020-08-17 14:35:34 -03:00
Gerardo Salazar b866c8b893 Removes Robintrack data source (#4628) 2020-08-14 19:46:30 -03:00
Colton Sellers 64a99d1194 Feature shorcut helper for STC (#4623)
* Shorcut helper for STC

* Use name variable
2020-08-12 14:41:20 -03:00
Gerardo Salazar 57cbfa2f26 Fixes stalling of LiveFillForwardEnumerator (#4620)
* Adds unit test containing previously failing test case
9033
2020-08-10 18:58:46 -03:00
Stefano Raggi bc44e23322 Upgrade IBAutomater to v1.0.29 (#4616)
* Upgrade IBAutomater to v1.0.28

* Upgrade IBAutomater to v1.0.29
9015
2020-08-07 12:56:31 -03:00
Juan José D'Ambrosio 6c051cd816 Update symbol property database with latest crypto pairs (#4612) 8998 2020-08-06 10:23:20 -03:00
Stefano Raggi 4c04efc237 Fix KeyNotFoundException in Cash.EnsureCurrencyDataFeed (#4610) 8997 2020-08-06 09:49:27 -03:00
Colton Sellers a83388ab1a case matching compatibility improvement (#4613) 8996 2020-08-06 09:38:59 -03:00
Adalyat Nazirov 1c9b817960 Track indicator previous input per symbol individually (#4607)
* track previous input per symbol

* improve Arms Index period checks

* don't need to be thread safe due to consolidators update are sequential

* Use TryGetValue for performance

- Minor update for AdvanceDeclineIndicator to use TryGetValue to reduce
  amount of dictionary access

Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
8952
2020-08-02 20:13:27 -03:00
Colton Sellers c5c57a33c1 IDE & Docker Integration (#4576)
* Mount custom algorithms to container

attach algorithms with bash script

* VS Code Integration

* PyCharm Integration

* Simplify the run docker script

* Mono C# Debugging is broken, removed related tasks

* add OSX config for run-docker

* Update readme.md

Reflect latest changes to configuration

* Update to move pdb for debugging algos

* VS Code C# Debugging fix

* Visual Studio fixes for debugging C#

* rework of launch scripts; also allows inline args

* Documentation overhaul

* Add python directory arg

* address review

* List debugging options in config

* Undo unnecessary change

* Address review and bughunt

* remove python debugging method
8951
2020-08-02 19:45:18 -03:00
Martin-Molinero 47ec1ba8be Merge pull request #4570 from AdalyatNazirov/feature-4168-arms-index-indicator
Add Arms Index Indicator
8949
2020-07-31 17:09:45 -03:00
Adalyat Nazirov e37c9dbb1a fix CreateIndicatorName; add unit tests 2020-07-31 22:50:31 +03:00
Adalyat Nazirov c2cac7fbee remove unused using 2020-07-31 22:03:57 +03:00
Adalyat Nazirov df9c1973da fix warmup process; handle not ordered bars 2020-07-31 21:39:31 +03:00
Adalyat Nazirov bfb13e46c8 improve warming indicator up 2020-07-31 21:13:56 +03:00
Adalyat Nazirov 59286bb750 handle missing values 2020-07-31 21:13:56 +03:00
Adalyat Nazirov 939b334478 test RemoveStock method 2020-07-31 21:13:56 +03:00
Adalyat Nazirov 22155bf3ce add extension methods for new indicators 2020-07-31 21:13:56 +03:00
Adalyat Nazirov 33bb1e8984 improve performance 2020-07-31 17:53:56 +03:00
Adalyat Nazirov fb483365ae refactor class 2020-07-31 17:53:56 +03:00
Adalyat Nazirov b0fcf9f7ac fixup data 2020-07-31 17:53:56 +03:00
Adalyat Nazirov a044cd6416 fix TRIN 2020-07-31 17:53:56 +03:00