2384 Commits

Author SHA1 Message Date
Stefano Raggi 6c42a266be IB brokerage - Update option positions at contract expiration (#5880)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* IB brokerage - Update option positions at contract expiration

* Add handling of live option expiration events

* Address review

- Nits

* Refactor option notification events [WIP]

* Add early exercise+assignment unit tests

* Fix sign bug in IB ExerciseOrder

* Address review

- log unexpected position
- add unit test cases for partial exercise/assignment
2021-09-30 13:48:41 -03:00
Martin-Molinero f082a40f34 Add missing comments (#5944)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
2021-09-27 18:48:53 -03:00
Martin-Molinero d739873daa Add aggregator reader (#5942)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Add BaseDataCollectionAggregatorReader

- Add enumerable for BaseDataCollection
- Add BaseDataCollectionAggregatorReader
- Clean up CollectionSubscriptionDataSourceReader

* Address reviews

- Add new FileFormat that will be handled by the new Reader
- Adding unit tests for the new BaseDataCollectionAggregatorReader
- Some DataSourceReader duplication cleaning up

* Minor adjustment after some live trading tests
2021-09-27 17:25:23 -03:00
Martin-Molinero 0f7dfe8ec8 Fix live ETF universe selection (#5938)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
- ETF live universe selection will behave the same as Coarse selection.
  Adding unit test
2021-09-23 15:11:35 -03:00
Martin-Molinero 0931ae868a Fix multiple coarse selection algorithms (#5927)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
- Fix for live trading multiple coarse selections. Adding unit test
  reproducing issue.
2021-09-14 20:27:09 -03:00
Colton Sellers 8358c394e1 Support all security types for benchmarks (#5919)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Adjust benchmark subscription to dynamically determine best resolution

* Get types from SM

* Add red/green test set for security types
2021-09-10 16:58:31 -03:00
Martin-Molinero 9acf68ad19 Bump pythonNet to 2.0.6 (#5881)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
2021-08-26 18:01:05 -03:00
Martin-Molinero 3410832ccf Alpha holdings state (#5770)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Alpha holdings state

- Alpha result packet will optionally provide the algorithms portfolio
  state

* Rename

* Convert AlphaStreamsPortfolio to data source

* Improvements on AlphaStreams algorithm

* Fix regression tests

* Add unit tests for EW AS PCM and fixing bugs

* Protobuf AlphaStreamsPortfoliot staState

- Protobuf AlphaStreamsPortfolioState. Adding unit tests
- Add variable TPV tests for EW ASPCM

* Add alpha license to Organization response

* Improvements EW AS PCM respects free portfolio value

* Fixes

- Update tests expected statistics results affected by MHDB custom data timezone fix
- Fix for Extensions.IsCustomDataType

* Fixed and adding more regression tests

- Adding support and regression test with alpha consumer with different account currency
- Adding support and regression test of a universe adding custom data
  types
- Add support and regression test for algorithm alpha consumer with existing holdings

* Add AlphaStreamsOrderEvent data type
2021-08-19 13:56:18 -03:00
Stefano Raggi d573a0f6c7 BrokerageTransactionHandler - handle OrderStatus.New in order update/cancel (#5867)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
2021-08-19 12:01:38 -03:00
Martin-Molinero 1fdb36ee89 Improve object store logs (#5864)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
2021-08-18 10:33:34 -03:00
Gerardo Salazar cd9e451ead Adds Support for ETF Constituent Universes (#5862)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* Adds support for ETF constituent universes

  * Adds filtering for universe data if it doesn't match the
    universe subscription type

  * Update mapping for ALL underlying Symbols if
    `Symbol.UpdateMappedSymbol(...)` is called. Required to support
    constituent ETF universes that might have mapping events

  * Delistings of composite constituent universe Symbol will result in
    removal of universe securities.

  * Added regression algorithms for ETF constituent mappings (C#/Python),
    along with data required to run locally

  * Refactor universe delistings in SubscriptionSynchronizer -
    big thank you to @Martin-Molinero :)

* Address review: update regression algorithms and add explanatory comments

* Address review: add additional checks to delisting regression algorithms

  * Adds new regression algorithm testing the addition of a universe
    without calling AddEquity() and asserts same behavior
2021-08-17 19:54:29 -03:00
Martin-Molinero 46829f0f01 Dependency clean up (#5834)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
2021-08-11 12:39:00 -03:00
Martin-Molinero e92230a7fe Fix SubscriptionDataReader auxiliary data out of order (#5843)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* Fix SubscriptionDataReader auxiliary data out of order

- Fix for SubscriptionDataReader emitting auxiliary data out of order
  due to sparse that and enumerator refresh logic. Adding regression
  algorithm
- Minor tweaks for unit tests failing on and off

* Fix for SDR internal subscriptions

- Fix for SubscriptionDataReader enumerator refresh for internal
  subscriptions which were ignored. Adding regression test
- Updating custom data regression algorithms affected by the issue
	- DropboxBaseDataUniverse was emitting a custom data point being end time
	- UnlinkedTraderBarIconicType was emitting a single data point
	  of the underlying SPY minute data when if should of emitted
	  all data points
2021-08-10 18:57:23 -03:00
IlshatGaripov 893ca2b41d Makes sure to create directory for results-destination-folder (#5842)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
2021-08-09 17:39:57 -03:00
Martin-Molinero e720147e50 Algorithm Manager will break when algorithm is deleted (#5840)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
2021-08-09 15:42:13 -03:00
Martin-Molinero b77f0122b2 Improve python exception parsing (#5831)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Improve python exception parsing

- Improve python exception parsing adding support for line shift. Adding
  unit tests

* PythonException revert change

* Centralized and normalize algorithm runtime handling

* Adding support for C# line and file exception report
2021-08-06 20:11:54 -03:00
Gerardo Salazar 650ce2e2b5 Removes References To Built-in Custom Data Implementations (#5793)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* WIP removal of custom data references & tests updates

* Regression algos updated and python algorithms moved to DataSource repos

* Fixes failing unit tests

* Add "LiveDataTypes" field to LiveNodePacket

* Adds Initialize() to IDataChannelProvider

  * Adds new extension method to convert
    HistoryRequest -> SubscriptionDataConfig

* Address review: Add protobuf definitions for Iconic data types

* Address review: re-adds DynamicSecurityDataAlgorithm as regression algo

  * Small adjustments to variable naming and documentation

* Move test files to respective DataSource repos
2021-08-02 19:24:45 -03:00
Martin-Molinero ef4fea7592 Fix engine initialization (#5814)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
2021-07-30 17:33:03 -07:00
Martin-Molinero 3cf9cc7d41 Update RestSharp to 106.12.0t (#5810)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
2021-07-30 17:45:59 -03:00
Martin-Molinero 41636c94c3 Common engine initialization (#5809)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
- Create Lean common engine initialization class, which will be called
  by research and Lean launcher
- Fix backtest deserialization bug due to failing to handle decimal
  value
2021-07-29 21:14:17 -03:00
Martin-Molinero 9cf4075957 Send invalid orderEvent for non shortable order (#5772)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
- TransactionHandler will see invalid OrderEvent for an order which
  tries to short an non shortable asset. Adding unit test, updating
  regression algorithm.
- Fix Api data price path normalization, found while testing with
shortable provider. Adding unit tests
2021-07-15 21:10:46 -03:00
Martin-Molinero 1ed390c5f7 Minor tweaks (#5771)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
- Improve readme file, add nuget link, update Lean home page
- Increase 10min look back to 60 for setting currency conversion
- Replace error message for trace when minute resolution history request
  is empty
2021-07-15 13:03:46 -03:00
Jasper van Merle f5224f8be9 Logging improvements (#5767)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
2021-07-14 20:53:09 -03:00
Colton Sellers 8d11bbf0ba Update to PythonNet 2.0.5 (#5764)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
2021-07-14 11:01:57 -03:00
Martin-Molinero f9adc6b136 Rename data permissions (#5753)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
2021-07-09 22:59:14 -03:00
Martin-Molinero 5bd0675423 Fix minor race condition in brokerage message handler (#5736)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Fix minor race condition in brokerage message handler

- Fix race condition where a message could be enqueued and
wait for a new call, not being processed ASAP

* Reduce BrokerageTransactionHandler logs
2021-07-03 19:46:31 -03:00
Martin-Molinero e5a50109b4 DownloaderDataProvider improvements (#5730)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
- Ignore margin files requests. Only provided by ApiDataProvider
- Reduce noise by LeanData.TryParse() call on failure
2021-07-02 14:03:16 -03:00
Rohan Sharma 8d352b6318 feat: Improve error message from ObjectStore (#5729) 2021-07-02 10:37:37 -03:00
Martin-Molinero 55195867a9 Reduce LTRH error log to debug (#5723)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
2021-06-30 16:44:52 -03:00
Martin-Molinero 0d45bf5101 Add composite DataProvider (#5714)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
- Add new CompositeDataProvider that will wrap multiple instance. Allows
  using for example the ApiDataProvider and a DownloaderDataProvider.
- Adjusting ProductJsonConverter after recent Modules addition
2021-06-29 12:40:31 -03:00
Martin-Molinero 66552b49bd Add new DownloaderDataProvider (#5709)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Add new DownloaderDataProvider

- Moving LeanDataWrite and IDataDownloader to common project
- Add new BaseDataDownloaderDataProvider with concurrency helper method
- Add new DownloaderDataProvider which will use a IDataDownloader or
  IBrokerage implementation as data source

* Add support for downloading symbol chains data
2021-06-28 15:56:10 -03:00
Colton Sellers f47775338c Expand Lean Default Charts and Reduce Clutter (#5680)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Update charts that are not empty and not default values

* Testing

* Filter final results

* Explicitly filter alpha charts if not needed

* Use filtered charts for Live results handler

* Refactor solution to manage alpha charts internally by creator

* nit comment fixes

* Implement drawdown as default chart

* Implement Capacity Estimate for backtest default chart

* Add default values to QC.Plotting

* Adjust drawdown calculation

* Only sample "Capacity" once a day

* Implement Volume chart

* Nit fixes

* Volume Chart adjustments and fixes

* Rename to "Assets Sales Volume"

* Implement Exposure Sampling

* Store algorithm currency symbol, round capacity to nearest 1k

* Add new plots to default charts

* nit change name

* Address reviews

* Reduce duplication and clean up exposure sampling

* Address reviews

* Improve sample exposure

* Address review

* Only enumerate holdings once

* nit - comments

* Post rebase fix

* Don't need to round anymore
2021-06-22 11:19:48 -03:00
Colton Sellers fa1e97bc60 Round Capacity Estimate for Runtime Statistics (#5688)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Refactor solution to deal with rounding in CapacityEstimate

* Use DiscretelyRoundBy + adjust regressions
2021-06-21 21:06:09 -03:00
Stefano Raggi 080511fa5a Add IResultHandler.BrokerageMessage (#5681)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Add IResultHandler.BrokerageMessage

- Result handlers can receive notifications for brokerage message events

* Make LiveTradingResultHandler.RuntimeError virtual
2021-06-21 12:23:46 -03:00
Colton Sellers 74f0c89499 Option Chain Provider Symbol Map Resolve (#5664)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Use mapFileProvider to resolve symbol mapping

* Add reproducing unit test

* Only resolve mapping for types that require it

* address review
2021-06-16 19:42:38 -03:00
Martin-Molinero 0991f089d0 Reduce disk usage in the cloud (#5673)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
2021-06-16 18:43:55 -03:00
Colton Sellers 424fa36703 Organization Product ID and ApiDataProvider Fix (#5671)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Add ID to Organization Products

* Use Product ID to verify subscription
2021-06-16 15:01:53 -03:00
Martin-Molinero b7c18cbfe7 Update pythonNet to net5 (#5653)
Build & Test Lean / build (push) Has been cancelled
- Csproj clean up legacy build print logs
- Update pythonNet to net5
2021-06-11 13:51:09 -03:00
Martin-Molinero 7e2678d0f6 Normalize live holdings initialization (#5647)
Build & Test Lean / build (push) Has been cancelled
* Normalize live holdings initialization

* PaperBrokerage config based live state recovery

- Increase algorithm status storing sampling period
- Fix bug where duplicate cash entries could happen
2021-06-09 20:11:38 -03:00
Colton Sellers 0f0eeaff82 DataProvider Updates & Refactoring (#5641)
* Use DataProvider for fetching Shortable and FutureMargins files

* Use data provider for AllShortableSymbols

* Refactor of NeedToDownload logic

* Adjustments

* nit - cleanup

* Expand test cases

* Create the directory if it doesn't exist
2021-06-08 22:11:21 -03:00
Martin-Molinero e8cf002238 Live trading holding improvements (#5644)
Build & Test Lean / build (push) Has been cancelled
- Store holdings at end of algorithm
- Warmup holding market price if 0
2021-06-08 21:55:17 -03:00
Martin-Molinero a231a7d47a Update python net to 2.0.3 (#5640)
Build & Test Lean / build (push) Has been cancelled
* Update pythonNet version to 2.0.3

* Fix outputPath causing issues with nuget
2021-06-07 21:37:26 -03:00
Colton Sellers 37cce922d3 Api Data Request Path Format Issue (#5639)
Build & Test Lean / build (push) Has been cancelled
* Address some format issues for data request paths

* Add reproducing test

* Drop logging for failure, let Api take care of it
2021-06-07 20:59:39 -03:00
Martin-Molinero 42c58d6e53 Ensure data is downloaded only once (#5635)
Build & Test Lean / build (push) Has been cancelled
- Ensure data is downloaded only once with the ApiDataProvider. Adding
  unit test
- Log invalid securityType ApiDataProvider requests once
- Assert Factor file and map file providers are compatible with the
  ApiDataProvider
2021-06-07 14:26:24 -03:00
Colton Sellers ab6027723f Refactor Api (#5251)
Build & Test Lean / build (push) Has been cancelled
* Use lean data key as param for request

* key -> filePath rename and some cleanup

* Refactor

* Add Organizations Endpoints

* Add some organization api wrapper objects

* Address namespace issue

* Reorganize Api Test into seperate files using one ApiTestBase

* Add Organization tests

* Use capitalized "API" test namespace to reduce amount of file changes

* Add License to test base

* Update /data endpoint functions and response objects

* Update ApiDataProvider Logic

* Handle deserialization of organization products

* Simplify converter

* Only throw for equity requests when not subscribed to map/factor files

* Add missing header

* Make arguement exception

* Api adjustments

* Add Zip factor and map file providers

- Common project will now reference Compression project and not the other way
  round.
- Adding Zip FactorFile and MapFile providers

* Refactor FactorFileProvider to use DataProvider to fetch files

* Use resulting MinimumDate in construction of FactorFile

* Nit FactorFile comments and arrangement

* Refactor MapFileProviders to use DataProvider for fetching files

* Refactor ZipFileProvider

* Clean up

* Refactor Backtesting Future/Option chain providers to use dataprovider

* Fixes for data/ endpoints and test adjustments

* Response objects adjustments/cleanups

* ApiDateProvider fixes and testing

* Add LocalZipFactorFileTests

* Update ApiDataProvider download test to verify stream is not null

* Implement posting of agreement summary and signed time

* Mark all Api related tests as explicit and document details on running

* Clarify default token on ApiTestBase

* Adjust summary

* Update Api responses for QCC, except org products which are sold in USD

* Implement cache expiration for zip MapFile and FactorFiles. Adding unit tests

* Fix multiple markets for ZipFactorFile provider

* Use Symbol as cache key

* Api.cs review

* Dispose of factorFileStream after reading

* Use zip.EntryFileNames

* Address a few reviews

* Few more fixes

* Address Api Review

* Add Job Org id to config

* Minor tweaks

* Compare with invariant culture

* Fixes Option Universe selection

* ZipEntryNameSubscriptionDataSourceReader will use IDataProvider

* Fix research

* Fix null reference exception

* Make duplicate log debug

Co-authored-by: Martin-Molinero <martin@quantconnect.com>
2021-06-07 11:37:44 -03:00
Martin-Molinero 0335b49edf Update pythonNet to 2.0.2 (#5611) 2021-05-28 12:10:55 -03:00
Martin-Molinero 9e96aeb440 Language update. Csproj clean up (#5590)
Build & Test Lean / build (push) Has been cancelled
- Update CSharp language version to latest 9
- Remove old csproj build configurations
2021-05-24 11:10:03 -03:00
Jasper van Merle bb8bd27fe5 Support 2-leg currency conversions (#5552)
Build & Test Lean / build (push) Has been cancelled
* Apply changes from #2146 on current master

Credit for most of these changes goes to https://github.com/viliwonka.
Rebasing three years of changes didn't go so well, so I manually
applied the changes in #2146 on the current master branch.

This commit attempts to represent the changes in #2146 as closely as
possible, no cleanup has been performed yet. Despite this, some changes
were inevitable because part of #2146 has been implemented another way
in the past three years.

* Clean up

* Process review comments

* Add tests

* symbol -> potentialConversionRateSymbol

* Ensure regression tests run/get data

* Fix broken test

* Process review comments

* Process review comments

* Simplify currency conversion logic

* Performance improvement, doc updates and test rename

* Rebase fixes

* Remove obsolete properties

Co-authored-by: Martin-Molinero <martin@quantconnect.com>
2021-05-20 13:19:39 -03:00
Martin-Molinero 978da37cda Minor tweaks (#5580)
Build & Test Lean / build (push) Has been cancelled
* Minor tweaks

- Add AccountCurrency and symbol for live result packet
- Improve performance of Tick clone and reading

* Reduce unit test from 9s to 500ms
2021-05-20 12:00:42 -03:00
Colton Sellers 4bb7fc17a2 Resolve Currency ConversionRate Backup (#5575)
Build & Test Lean / build (push) Has been cancelled
* Attempt to use daily resolution request to resolve currency conversion

* Add BaseSetupHandlerTests

* Minor tweaks. Adding missing dispose call.

Co-authored-by: Martin-Molinero <martin@quantconnect.com>
2021-05-19 19:54:02 -03:00