10440 Commits

Author SHA1 Message Date
Stefano Raggi c85f608444 Add zero TotalPortfolioValue check in BrokerageSetupHandler (#4639)
* Add empty cashbook check in BrokerageSetupHandler

* Update check for zero TotalPortfolioValue
9130
2020-08-19 12:55:43 -03:00
Martin-Molinero 9cdb4a91c5 Refactor live data feed (#4636)
* Live Coarse universe refactor

- Live trading will source Coarse and Fine fundamental data directly
  from disk. Updating unit tests.

* Adds ILiveDataProvider interface

  * Adds wrapper for IDataQueueHandler implementations

  * Replaces IDataQueueHandler with ILiveDataProvider in
    LiveTradingDataFeed

  * Edits IDataQueueHandler documentation

* Maintains aggregation for current IDQH impls and skips for ILDF impls

  * Note: No unit test was created for this method, go back and TODO

* Protobuf Market data

- Adding protobuf support for Ticks, TradeBars and QuoteBars. Adding
  unit tests.

* Adds unit tests for LiveDataAggregator changes

  * Fixes bug where custom data was not handled as it was before
  * Fixes race condition bug because of variable reuse in class

* Add protobuf extension serialization

* Fixes for protobuf serialization

* Refactor

* Fix OptionChainUniverse

* replace BaseDataExchange pumping ticks with consolidators

* AlpacaBrokerage

* BitfinexBrokerage

* GDAXBrokerage

* OandaBrokerage

* InteractiveBrokers

* TradierBrokerage

* FxcmBrokerage

* PaperBrokerage

* etc

* WIP fixes for existing LTDF unit tests

* Fixes more LTDF unit tests

* make IDataAggregator.Update recieving Generic BaseData rather than Tick

* Change IDataQueueHandler.Subscribe method

* Some fixes after adding new commits

* Adds protobuf (de)serialization support for Dividend and Split

* Serialize protobuf with length prefix

* Fix missing LTDF unit tests

* Adds TiingoNews protobuf definitions

* fix comments

* more fixes on IQFeedDataQueueHandler

* disallow putting ticks into enumerator directly

* ScannableEnumerator tests

* fix OandaBrokerage

* AggregationManager unit tests

* fix AlpacaBrokerage tests

* fix InteractiveBrokers

* fix FxcmBrokerage tests

* call AggregationManager.Remove method on unsubscribe

* fix GDAX existing tests

* Fixes, refactor adding more tests for AggregatorManager

* Adds BenzingaNews protobuf definitions and round trip unit test

* Adds missing TiingoNews unit test to Protobuf round trip tests

* Improve sleep sequence of LiveSynchronizer

* need start aggregating first, and then can subscribe

* More test fixes and refactor

- Refactoring AggregationManager and ScannableEnumerator so the last is
  the one that owns the consolidator
- Adding pulse on the main LiveSynchronizer

* Improve performance of LEquityDataSynchronizingEnu

* Add missing Set job packet method

* Minor performance improvements

* Improvements add test timeout

- Improvements adding test timeout to find blocking test in travis

* Improve aggregationManager performance

* Testing improvements for travis

* Remove test timeouts

* More test fixes

- Adding more missing dispose calls and improving determinism

* fix IEXDataQueueHandler and tests

* Final tweaks to LTDF tests

* more AggregationManager tests

* consume and log ticks

* fix test: couldn't subscribe to Forex tickers

* change Resolution for all bar configs

* Improve RealTimeScheduleEventServiceAccuracy

* refactoring: move common code to base class

* fixed bug; unsubscribe SubscriptionDataConfig

* Small performance improvement

* Minor fixes

* Avoid Symbol serialization

* Fixes coarse selection in live mode

* Fix for live coarse

* Adds protobuf (de)serialization support for Robintrack

  * Adds round-trip unit test

* Minor performance improvements

* More minor performance improvements

* pass LiveNodePacket through to OandaBrokerage

* Fixes empty list becoming null value when deserializing with protobuf

* Reverts BZ live trading exception removal and fixes tests

* Refactor WorkQueue making it abstract

* Add try catch for composer

* Adds optional data batching period to LiveFillForwardEnumerator

* Override data-queue-handler with config

* Improve PeriodCountConsolidator.Scan performance

* Move batching delay to main Synchornizer thread

* Reverts addition of Robintrack protobuf definitions

* Give priority to config history provider if set

* Add Estimize protobuffing

- Add Estimize protobuffing support. Adding unit tests

* Always dispose of data queue handler

Co-authored-by: Gerardo Salazar <gsalaz9800@gmail.com>
Co-authored-by: Adalyat Nazirov <aenazirov@gmail.com>
9125
2020-08-18 20:21:10 -03:00
michael-sena a128f8bb2e Allow LeanDataWriter to create zip files for futures data (#4569)
* Allow LeanDataWriter to append to zip data files

* Use the data directory provided to the writer instead of the global value

* Disregard the time-portion of an input date

* Overwrite zip entries when creating futures data files

* Minor tweak and adding unit test

Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
9121
2020-08-18 17:44:02 -03:00
Colton Sellers 3ef6a761ad Feature research docker scripts (#4622)
* Add research docker launch script

* Internalize creation of cfg and dir to dockerfile

* Implement bash version of notebook run script

* quick fix to use docker.cfg

* update readme

* typo fix

* Review tweaks

* Tweaks
9110
2020-08-17 14:35:34 -03:00
Gerardo Salazar b866c8b893 Removes Robintrack data source (#4628) 2020-08-14 19:46:30 -03:00
Colton Sellers 64a99d1194 Feature shorcut helper for STC (#4623)
* Shorcut helper for STC

* Use name variable
2020-08-12 14:41:20 -03:00
Gerardo Salazar 57cbfa2f26 Fixes stalling of LiveFillForwardEnumerator (#4620)
* Adds unit test containing previously failing test case
9033
2020-08-10 18:58:46 -03:00
Stefano Raggi bc44e23322 Upgrade IBAutomater to v1.0.29 (#4616)
* Upgrade IBAutomater to v1.0.28

* Upgrade IBAutomater to v1.0.29
9015
2020-08-07 12:56:31 -03:00
Juan José D'Ambrosio 6c051cd816 Update symbol property database with latest crypto pairs (#4612) 8998 2020-08-06 10:23:20 -03:00
Stefano Raggi 4c04efc237 Fix KeyNotFoundException in Cash.EnsureCurrencyDataFeed (#4610) 8997 2020-08-06 09:49:27 -03:00
Colton Sellers a83388ab1a case matching compatibility improvement (#4613) 8996 2020-08-06 09:38:59 -03:00
Adalyat Nazirov 1c9b817960 Track indicator previous input per symbol individually (#4607)
* track previous input per symbol

* improve Arms Index period checks

* don't need to be thread safe due to consolidators update are sequential

* Use TryGetValue for performance

- Minor update for AdvanceDeclineIndicator to use TryGetValue to reduce
  amount of dictionary access

Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
8952
2020-08-02 20:13:27 -03:00
Colton Sellers c5c57a33c1 IDE & Docker Integration (#4576)
* Mount custom algorithms to container

attach algorithms with bash script

* VS Code Integration

* PyCharm Integration

* Simplify the run docker script

* Mono C# Debugging is broken, removed related tasks

* add OSX config for run-docker

* Update readme.md

Reflect latest changes to configuration

* Update to move pdb for debugging algos

* VS Code C# Debugging fix

* Visual Studio fixes for debugging C#

* rework of launch scripts; also allows inline args

* Documentation overhaul

* Add python directory arg

* address review

* List debugging options in config

* Undo unnecessary change

* Address review and bughunt

* remove python debugging method
8951
2020-08-02 19:45:18 -03:00
Martin-Molinero 47ec1ba8be Merge pull request #4570 from AdalyatNazirov/feature-4168-arms-index-indicator
Add Arms Index Indicator
8949
2020-07-31 17:09:45 -03:00
Adalyat Nazirov e37c9dbb1a fix CreateIndicatorName; add unit tests 2020-07-31 22:50:31 +03:00
Adalyat Nazirov c2cac7fbee remove unused using 2020-07-31 22:03:57 +03:00
Adalyat Nazirov df9c1973da fix warmup process; handle not ordered bars 2020-07-31 21:39:31 +03:00
Adalyat Nazirov bfb13e46c8 improve warming indicator up 2020-07-31 21:13:56 +03:00
Adalyat Nazirov 59286bb750 handle missing values 2020-07-31 21:13:56 +03:00
Adalyat Nazirov 939b334478 test RemoveStock method 2020-07-31 21:13:56 +03:00
Adalyat Nazirov 22155bf3ce add extension methods for new indicators 2020-07-31 21:13:56 +03:00
Adalyat Nazirov 33bb1e8984 improve performance 2020-07-31 17:53:56 +03:00
Adalyat Nazirov fb483365ae refactor class 2020-07-31 17:53:56 +03:00
Adalyat Nazirov b0fcf9f7ac fixup data 2020-07-31 17:53:56 +03:00
Adalyat Nazirov a044cd6416 fix TRIN 2020-07-31 17:53:56 +03:00
Adalyat Nazirov 3f556d70ec A/D Volume Ratio: wip 2020-07-31 17:53:55 +03:00
Adalyat Nazirov 220ccbae7c A/D Ratio : wip 2020-07-31 17:53:55 +03:00
Adalyat Nazirov a6d1908aa9 TRIN 2020-07-31 17:53:55 +03:00
Adalyat Nazirov 99aa1eadd7 create A/D Volume Ratio indicator 2020-07-31 17:53:55 +03:00
Adalyat Nazirov d53f79b271 create A/D Ratio indicator class 2020-07-31 17:53:55 +03:00
Adalyat Nazirov 78d30b23cb new status 2020-07-31 17:53:55 +03:00
Adalyat Nazirov a8d6857407 test data 2020-07-31 17:53:55 +03:00
Adalyat Nazirov fad38f8ed4 [tmp] generate equity, but need generic method to parse symbol 2020-07-31 17:53:55 +03:00
Adalyat Nazirov c7091dbf6b parse Symbol if exists 2020-07-31 17:53:55 +03:00
Adalyat Nazirov ea9007601d fix date parsing: TwelveCharacter dateformat can't be parsed properly;
more String Extensions: extract substring safely
add more US formats
2020-07-31 17:53:55 +03:00
Adalyat Nazirov 5278a3dd02 too many code duplications
reuse existing GetCsvFileStream for streaming file data
2020-07-31 17:53:54 +03:00
Adalyat Nazirov ebf4d21bd9 reuse existing TradeBarIndicator instead of Generic one 2020-07-31 17:53:54 +03:00
Adalyat Nazirov c83627a89c fix BollingerBands indicator comments 2020-07-31 17:53:54 +03:00
Martin-Molinero 27e0ce7a57 Merge pull request #4586 from QuantConnect/bug-4568-iex-data-queue-handler-fix
IEX DataQueueHandler updates
8943
2020-07-31 10:52:49 -03:00
Jared 457418e7bc Merge pull request #4603 from QuantConnect/fix-factor-files-missing-file-check
Fix factor files missing file dir check
8942
2020-07-31 06:46:00 -07:00
Martin Molinero db1bf686b9 Fix factor files missing file dir check 2020-07-31 10:44:57 -03:00
Stefano Raggi bc41b4461e Add logging for missing IEX api key 2020-07-31 08:49:03 +02:00
Stefano Raggi 036d8684c0 IEX DataQueueHandler updates
- Add parameterless constructor (for usage in LEAN config)
- Added config setting for IEX Cloud api key (required for history requests)
2020-07-30 17:32:00 +02:00
Martin-Molinero d1e38b33a6 Merge pull request #4599 from QuantConnect/fix-smartinsider-intention-raw-data-read
Fix SmartInsiderTests failing test
2020-07-30 11:09:14 -03:00
Juan José D'Ambrosio a28cde3633 Fix failing test
SmartInsiderIntention.FromRawData method doesn't read the raw data as they come from SmartInsider, it needs a filtering of some columns that is performed in SmartInsiderConverter.Process method.

That's added to the failing test.
2020-07-30 13:49:05 +00:00
Martin-Molinero 2360e4c961 Merge pull request #4598 from michael-sena/bug-4597-allow-markethoursdatabase-to-be-used-without-data-folder
Allow use of the MarketHoursDatabase class without requiring the Data folder
8926
2020-07-30 10:08:09 -03:00
Martin-Molinero 839b38d904 Merge pull request #4573 from QuantConnect/alpaca-brokerage-updates
Alpaca Brokerage updates
2020-07-30 09:57:46 -03:00
Martin-Molinero 443ec46c89 Merge pull request #4590 from michael-sena/bug-4589-move-margin-files-for-sgx-and-hkfe
Move the margin files for SGX and HKFE into the correct folder
8924
2020-07-30 09:46:25 -03:00
Stefano Raggi 38ce10c723 Remove unused references to WebSocketSharpFork 2020-07-30 14:06:22 +02:00
Stefano Raggi ab8d95f627 Remove redundant logging 2020-07-30 14:06:22 +02:00