3315 Commits

Author SHA1 Message Date
Colton Sellers 8edc89a4ff Feature Python Option Filtering Support (#4667)
* Support List and OptionFilterUniverse for Py filter

* Regression algorithm for testing

* Unit Tests

* Fix for process

* Tighten filters to reduce load on automated testing

* Address review v2
2020-09-02 21:38:16 -03:00
Martin-Molinero 400a0d42d9 Add internal subscription manager (#4678)
* Add internal subscription manager

- Add InternalSubscriptionManager that will handle internal
  Subscription. Replaces the realtime updates
- Fix thread race condition in the TimeTriggeredUniverseSubscription, we
  have one thread injecting data points, the main algorithm thread, and
  the base exchange is pulling from it
- Fixes for FakeDataQueue
- Adding unit tests

* Address reviews and fixes

- Internal subscription will use extended market hours
- Only sample charts accordingly
- Get api-url once
2020-09-01 21:22:22 -03:00
Colton Sellers 34676f0c46 Bug 4668 python register indicator (#4670)
* Convert to timespan if possible

* Fix to support both timespans and custom consolidators

* Adjust tests for new wrapper requirements

* Cleanup
2020-08-29 02:53:32 -03:00
Stefano Raggi 375f192f07 Add PolygonDataQueueHandler (#4604)
* Add PolygonDataQueueHandler

* Add history provider and downloader for Polygon

* Add aggregator to PolygonDataQueueHandler

* Address review

- Removed duplication in message classes
- Added public Subscribe/Unsubscribe methods in PolygonWebSocketClientWrapper
- Added history requests for Forex and Crypto

* Address review

- Add security type and market arguments to downloader
- Fix time zone bug in downloader

* Remove unnecessary locks

* Add Polygon history for all resolutions

- Equity: trades and quotes
- Forex: quotes only
- Crypto: trades only
2020-08-27 12:09:20 -03:00
Colton Sellers b8674731a5 Feature 2456 custom Python consolidator support (#4637)
* DataConsolidator Wrapper for Python Consolidators

* Regression Unit Test

* Refactor Regression test

* Bad test fix

* pre review

* self review

* Add RegisterIndicator for Python Consolidator

* Python base class for consolidators

* Modify regression algo to register indicator

* unit test - attach event

* Test fix

* Fix test python imports

* Add license header file and null check

Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
2020-08-25 17:26:55 -03:00
Alexandre Catarino a14f8f1f47 Refactors SecurityCache to Support Access to Tick of Different Type (#4621)
* Refactors SecurityCache to Support Access to Tick of Different Type

Adds two read-only lists of BaseData to save the last list of Tick of TickType.Trade and TickType.Quote. Changes methods accordingly to get and set these lists.

* Addresses Peer Review

The assignment of `_lastData` is moved to keep the current behavior of not beeing set with fill forward data.
`Reset` and `ShareTypeCacheInstance` implements the new class objects.

Adds unit tests for `Reset` and `ShareTypeCacheInstance`.

* Only Cache Non Fill-Forward Data

Undo changes that made AddData cache non fill-forward data.
2020-08-24 10:37:50 -03:00
Martin-Molinero 9cdb4a91c5 Refactor live data feed (#4636)
* Live Coarse universe refactor

- Live trading will source Coarse and Fine fundamental data directly
  from disk. Updating unit tests.

* Adds ILiveDataProvider interface

  * Adds wrapper for IDataQueueHandler implementations

  * Replaces IDataQueueHandler with ILiveDataProvider in
    LiveTradingDataFeed

  * Edits IDataQueueHandler documentation

* Maintains aggregation for current IDQH impls and skips for ILDF impls

  * Note: No unit test was created for this method, go back and TODO

* Protobuf Market data

- Adding protobuf support for Ticks, TradeBars and QuoteBars. Adding
  unit tests.

* Adds unit tests for LiveDataAggregator changes

  * Fixes bug where custom data was not handled as it was before
  * Fixes race condition bug because of variable reuse in class

* Add protobuf extension serialization

* Fixes for protobuf serialization

* Refactor

* Fix OptionChainUniverse

* replace BaseDataExchange pumping ticks with consolidators

* AlpacaBrokerage

* BitfinexBrokerage

* GDAXBrokerage

* OandaBrokerage

* InteractiveBrokers

* TradierBrokerage

* FxcmBrokerage

* PaperBrokerage

* etc

* WIP fixes for existing LTDF unit tests

* Fixes more LTDF unit tests

* make IDataAggregator.Update recieving Generic BaseData rather than Tick

* Change IDataQueueHandler.Subscribe method

* Some fixes after adding new commits

* Adds protobuf (de)serialization support for Dividend and Split

* Serialize protobuf with length prefix

* Fix missing LTDF unit tests

* Adds TiingoNews protobuf definitions

* fix comments

* more fixes on IQFeedDataQueueHandler

* disallow putting ticks into enumerator directly

* ScannableEnumerator tests

* fix OandaBrokerage

* AggregationManager unit tests

* fix AlpacaBrokerage tests

* fix InteractiveBrokers

* fix FxcmBrokerage tests

* call AggregationManager.Remove method on unsubscribe

* fix GDAX existing tests

* Fixes, refactor adding more tests for AggregatorManager

* Adds BenzingaNews protobuf definitions and round trip unit test

* Adds missing TiingoNews unit test to Protobuf round trip tests

* Improve sleep sequence of LiveSynchronizer

* need start aggregating first, and then can subscribe

* More test fixes and refactor

- Refactoring AggregationManager and ScannableEnumerator so the last is
  the one that owns the consolidator
- Adding pulse on the main LiveSynchronizer

* Improve performance of LEquityDataSynchronizingEnu

* Add missing Set job packet method

* Minor performance improvements

* Improvements add test timeout

- Improvements adding test timeout to find blocking test in travis

* Improve aggregationManager performance

* Testing improvements for travis

* Remove test timeouts

* More test fixes

- Adding more missing dispose calls and improving determinism

* fix IEXDataQueueHandler and tests

* Final tweaks to LTDF tests

* more AggregationManager tests

* consume and log ticks

* fix test: couldn't subscribe to Forex tickers

* change Resolution for all bar configs

* Improve RealTimeScheduleEventServiceAccuracy

* refactoring: move common code to base class

* fixed bug; unsubscribe SubscriptionDataConfig

* Small performance improvement

* Minor fixes

* Avoid Symbol serialization

* Fixes coarse selection in live mode

* Fix for live coarse

* Adds protobuf (de)serialization support for Robintrack

  * Adds round-trip unit test

* Minor performance improvements

* More minor performance improvements

* pass LiveNodePacket through to OandaBrokerage

* Fixes empty list becoming null value when deserializing with protobuf

* Reverts BZ live trading exception removal and fixes tests

* Refactor WorkQueue making it abstract

* Add try catch for composer

* Adds optional data batching period to LiveFillForwardEnumerator

* Override data-queue-handler with config

* Improve PeriodCountConsolidator.Scan performance

* Move batching delay to main Synchornizer thread

* Reverts addition of Robintrack protobuf definitions

* Give priority to config history provider if set

* Add Estimize protobuffing

- Add Estimize protobuffing support. Adding unit tests

* Always dispose of data queue handler

Co-authored-by: Gerardo Salazar <gsalaz9800@gmail.com>
Co-authored-by: Adalyat Nazirov <aenazirov@gmail.com>
2020-08-18 20:21:10 -03:00
Gerardo Salazar b866c8b893 Removes Robintrack data source (#4628) 2020-08-14 19:46:30 -03:00
Stefano Raggi 4c04efc237 Fix KeyNotFoundException in Cash.EnsureCurrencyDataFeed (#4610) 2020-08-06 09:49:27 -03:00
Adalyat Nazirov ea9007601d fix date parsing: TwelveCharacter dateformat can't be parsed properly;
more String Extensions: extract substring safely
add more US formats
2020-07-31 17:53:55 +03:00
Martin Molinero db1bf686b9 Fix factor files missing file dir check 2020-07-31 10:44:57 -03:00
Michael Sena b0c21b605c Allow use of the MarketHoursDatabase class without requiring the Data folder 2020-07-30 16:16:25 +10:00
Juan José D'Ambrosio 211f1144b5 Implement NotSpecified SmartInsiderEventType 2020-07-29 21:08:52 +00:00
JJD e7682cbd37 Add Plan Re-started to SmartInsiderEventType 2020-07-29 11:24:44 -03:00
Jared 48db57d224 Merge pull request #4537 from C-SELLERS/bug-4524-Update-API-Node-Endpoints
Add Node API Functions
2020-07-28 07:30:12 -07:00
Colton Sellers dc82c15c18 Address review 2020-07-27 14:11:40 -07:00
Martin Molinero 5faf7a0b55 Fix Mono reporting double CPU usage 2020-07-16 11:40:32 -03:00
Colton Sellers 935db4ed3e Address new pricing structure when deserializing node objects 2020-07-10 14:04:41 -07:00
Colton Sellers 5188d0cb7f Documentation update 2020-07-10 12:28:38 -07:00
Jared d1624ab899 Merge pull request #4560 from QuantConnect/bug-4547-pandas-data-list-index
Adds Support to List of Symbol and String (Ticker)
2020-07-08 17:29:40 -07:00
Colton Sellers df7bc89d9b Address review 2020-07-08 14:37:58 -07:00
Jared 0f8e4122cd Merge pull request #4539 from QuantConnect/ib-brokerage-updates-2
IB Brokerage Updates
2020-07-08 14:20:26 -07:00
Martin Molinero bab635856d Create empty result packet for live trading 2020-07-08 11:47:28 -03:00
Martin Molinero aa0dfc7b3b Create empty result packet for failed inits
- Backtesting will create an empty packet for failed inits
2020-07-08 11:03:04 -03:00
Alexandre Catarino 75a5f267f1 Adds Support to List of Symbol and String (Ticker)
Adds case in the mapper method to handled list type.
2020-07-04 00:11:19 +01:00
Stefano Raggi b897f12e66 Minor log message fixes 2020-07-01 15:35:56 +02:00
Anand Vishnu ec6abfacc8 Add future contracts for NK, IN, TW and HSI 2020-06-29 17:35:40 +01:00
Colton Sellers 831e1c0b78 Address reviews v2 2020-06-24 18:02:13 -07:00
Colton Sellers 053925829c Address review 2020-06-24 17:44:15 -07:00
Colton Sellers 4263afab9f Removed all unnecessary new lines 2020-06-24 17:06:21 -07:00
Colton Sellers da89f6f3d1 RAM can be decimal, GetSKU is static for direct method calls, & commented out bug on CreateNode response wrapper. 2020-06-24 15:53:20 -07:00
C-SELLERS b28ed7c9f2 Add GetSKU helper and CreatedNode response wrapper 2020-06-24 12:22:57 -07:00
Colton Sellers b7ba768371 Addition of API Node classes 2020-06-23 16:55:18 -07:00
Colton Sellers a9a37c1647 Node & NodeList classes for API response 2020-06-23 16:53:48 -07:00
Martin Molinero c18b1af5ce Address reviews
- Add new comment expanding on the QuoteBar ask and bid fill forward behavior for new bars, which are not a fill forward bar.
2020-06-23 14:50:23 -03:00
Martin Molinero febc4b50ea Fix QuoteBarConsolidator
- TickQuoteBarConsolidator will use previous bar. Updating unit tests
- QuoteBarConsolidator open ask and bid will match previous bar close
  bid and ask. Adding unit tests
2020-06-23 14:42:56 -03:00
Anand Vishnu 79a05ebd27 Quote bar previous close and current open should be same 2020-06-23 14:42:56 -03:00
Martin Molinero 39e56ea7c0 Address reviews
- Removing the need to call MHDB.SetEntry, this will be handled by the
  data manager
2020-06-15 20:05:12 -03:00
Martin Molinero b575880739 Coarse tiingo new universe selection algorithm
- Adding an example algorithm of a custom universe selection using
  coarse data and adding tiingo news. If conditions are met will add the
  underlying and trade it
- Adding required UniversePythonWrapper
2020-06-15 20:05:08 -03:00
Martin-Molinero 47a4f875ff Merge pull request #4506 from QuantConnect/feature-data-permission-manager
Adding new DataPermissionManager
2020-06-15 15:08:18 -03:00
Alexandre Catarino 2ca4ba694e Fixes Decimal Parse of MinimumPriceVariation
`MinimumPriceVariation` values in `SymbolPropertiesDatabase` can be represented as a exponent number. `ToDecimal` method fails to parse it correctly. A new method, `ToDecimalAllowExponent` is used instead.
2020-06-13 01:38:31 +01:00
Martin Molinero bcc67d2457 DataPermissionManager is used by history provider
- DataPermissionManager will be used by the history provider
- Moving interfaces from Engine to Common
2020-06-12 12:51:14 -03:00
Martin Molinero 452bf27bd9 Adding new DataPermissionManager
- Adding new DataPermissionManager that will own the existing
  datachannelProvider. Will assert configurations before added to the
  data feed. Adding unit tests
2020-06-12 12:34:10 -03:00
Martin Molinero e782ffa4c6 Reuse existing FileAccess permissions 2020-06-09 18:13:12 -03:00
Martin Molinero 5bd11ebd9d Add permission control for Storage
- Adding `Controls.StoragePermissions` to govern permissions for storing into
  the object store. Adding unit tests
2020-06-09 17:24:00 -03:00
Martin Molinero 69194bfbe9 Minor improvements
- Makeing FactorFileRow.Parse private to avoid wrong usages
- Avoid end of stream check while unzipping
2020-06-08 14:33:52 -03:00
Martin Molinero 8d1a8d767b Remove depreceted projects
- Remove Java and VisualStudioPlugin deprecated projects
- Removing VS 15 solution file
- Nuget .nuspec fixes
2020-06-01 21:27:45 -03:00
Martin Molinero 8c0235617b Add CPU max allocation to Controls
- Add CPU max allocation to Controls
- Rename json field ram allocation from iRamAllocation to iMaxRamAllocation
2020-05-28 15:53:28 -03:00
Stefano Raggi 0a046699fc Consolidate SharpZipLib library usage 2020-05-28 11:18:44 +02:00
Jared f9cad3dc0c Merge pull request #4451 from QuantConnect/bug-pandasdata-misaligned-series-indexes-crash
Fixes alignment of Series objects created in PandasData causing crash
2020-05-26 14:02:17 -07:00