2402 Commits

Author SHA1 Message Date
Martin-Molinero b4e95209f6 Fix for IB Option position race condition (#6090)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
- Fix for IB option position update race condition, where the BTH would
  incorrectly assume it should trigger an early option assignment. Adding unit tests
2021-12-02 17:09:50 -03:00
Martin-Molinero d1a35e6281 Continuous Futures Refactor. Live Mappings (#6076)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* Continuous Futures Refactor. Live Mappings

- Adding support for live mappings. LiveTradingDataFeed will handle any
  symbol mapping at the configuration layer and resubscribe through the
  IDQH
- Refactoring continuous futures adding ContinuousFutureUniverse that
  will select the currently mapped security

* Minor fixes

- Remove addition of configurations in UniverseSelection step, leave
  resposability for universe.
- LiveTradingDF future unit test will only assert slice data for non
  internal feeds.
- ContinuousContractUniverse will respect internal option interest
  subscription

* Address review
2021-11-30 21:38:50 -03:00
bmello4688 1d8243ecac Add tick type to data downloader get (price data) call in order for a… (#6057)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Add tick type to data downloader get (price data) call in order for api's with rate limits to handle not making an api call at all.

* Added DTO object to pass into data downloader get

* Added missing header for new file

* Address review. Minor tweaks

Co-authored-by: Martin-Molinero <martin@quantconnect.com>
2021-11-22 12:00:16 -03:00
Martin-Molinero 5f95a9ba77 Bug fix python history requests enumerable data (#6066)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* Fix for python enumerable data history request

- Fix for python enumerable type data history request. Adding unit test.

* Add methods for adding data points into a baseDataCollection
2021-11-19 14:58:48 -03:00
Colton Sellers 33599b473d Refactor Delistings Processing (#6059)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Move processing of delistings to Brokerage

* Deal with case that exchange is not open on OptionSymbol.ID.Date

* Refactor solution to use DelistingNotification event

* Adjust some regression expected liquidation time

* Mark some todos on deprecated functions

* Update expected liqudation time for Py regressions

* Update regressions that have been validated

* Use HandlePositionAssigned for assignment orders

* Update regressions

* Update some missed unit tests; remove one that is already covered by regression

* Cleanup deprecated backend functions

* nit - small cleanup adjustment

* Post rebase fix

* Address review

* Minor tweak to py regression
2021-11-17 17:43:35 -03:00
Martin-Molinero b2517cbbb4 Fix DiskDataCacheProvider. Expanding unit test (#6058)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
2021-11-15 15:57:09 -03:00
Martin-Molinero b8b0d18993 Continuous futures (#6034)
* Continuous Future Contracts

* Mapping approach

* Tweaks WIP

* Live mapping

* Live mapping

- Add support for live mapping, refreshing mapfiles
- Fix future expiration functions
- Adding unit tests

* Update moq test package

* Continuous futures price scaling

* Refactor price factors scaling

* Factor file related renames

* Address reviews
2021-11-15 14:44:30 -03:00
Martin-Molinero 57f0d17c5d Remove user plan enum (#6055)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Remove user plan enum

- Get maximum order and runtime from job controls
- Remove user plan enum definitions

* Add log for BacktestingSetupHandler maximums
2021-11-12 19:37:23 -03:00
Colton Sellers dd4da7ba95 Feature Daily/Hourly Options Support (#6017)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* Create generic writing for LeanDataWriter, + notes on todos

* Make Options Daily/Hourly data store by year

* Refactor Generic Write

* Permit hour and daily resolutions for options

* Refactor writer to merge when needed with other files

* Cleanup redundancies, run write tasks in parallel

* Make needed classes/vars available

* Update tests to reflect new naming convention for daily hourly options data

* Add Byte[] overloads for ZipData functions in compression

* Implemented Store() for ZipDataCacheProvider

* Have LeanDataWriter use a DataCacheProvider

* ZipDataCacheProvider cleanup

* ZipDataCacheProvider tweaks, doesn't support storing non-zips

* Test adjustments

* Update LeanDataWriter to use Write instead of SaveDailyHourly/SaveMinuteSecond

* Implement tests to verify DownloadAndSave behavior

* Nit cleanup on DownloadAndSave tests

* Fix for options daily/hourly underlying equity subscription read

* Add daily/hourly options data and regressions

* Add missing open interest for hourly

* Fix writing of OpenInterest Daily/Hourly data

* Update data

* Fix Date typo in regression

* Use daily algorithm to test delisting

* Revisions part 1

* Expand test for DataCacheProviders; refactor DiskDataCacheProvider

* nit - test adjustments

* ZipDataCacheProvider test setup refactor

* Adjust multithreaded read/write test; fixes for ZipDataCacheProvider

* Move DiskDataCacheProvider to its own file and add write test

* Remove _appendToZips; always overwrite entry or create zip

* Add mapping regression for daily options

* nit - add license to regression

* Fix Tick write case where more than one data point for a DateTime

* Fix data issue

* Address review

* Tweaks for tests

* Stop Store() early if no entry name is given
2021-11-11 20:05:31 -03:00
Alexandre Catarino 4ecdf14ce8 Include Security Master Url to ApiDataProvider Error Message (#6023)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
2021-10-27 14:58:15 -03:00
IlshatGaripov 2e4252c92b LiveTradingResultHandler -> changes access modifier of SetNextStatusUpdate (#6015)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
2021-10-25 11:17:35 -03:00
Martin-Molinero 58ccdee1b1 PythonNet version bump 2.0.10 (#6010)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
2021-10-22 20:27:38 -03:00
Colton Sellers 6bd2859f64 Update to PythonNet 2.0.9 (#6004)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
2021-10-21 20:06:38 -03:00
Colton Sellers cff3640f15 Update to PythonNet 2.0.8 (#5978)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
2021-10-13 20:19:06 -03:00
Martin-Molinero b5574a7986 Do not emit zero quantity Exercise Order events (#5976)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
- BrokerateTransactionHandler will not emit zero quantity Exercise order
  events. Updating unit tests
2021-10-12 19:38:37 -03:00
Martin-Molinero 1d6fed8843 Update python to 2.0.7 (#5968)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
2021-10-08 12:55:14 -03:00
Colton Sellers d2d99b1f10 Algorithm Sampling and Statistics Fixes (#5936)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Implement scheduled event sampling solution

* Use UTC time, only update daily portfolio value once a day

* For daily resolutions sample chart always

* Cleanup

* Drop resample daily all together

* Force final sample

* Regression updates

* FIx LiveResultHandler to update portfolio and benchmark values outside of sampling event

* Name the daily sampling event

* Address review pt 1

* Drop force and use reference wrapper

* Adjust tests

* Fix warning for Benchmark Timezone Misalignment and also add test

* Fix for daily resolution orders and test adjustments

* Also warn on universe settings with daily resolution

* Update missed regression

* Fix reference wrapper use

* Update regression after rebase

* Add values back in for Daylight Algo

* Have statistics builder skip day 1 performance

* Regression adjustments

* Test adjustments

* Update regression unit test

* Adjust some regressions starts to show performance values

* Add hourly algorithm for beta comparison

* Address missing Python regression changes

* Remove null comment
2021-10-05 19:31:25 -03:00
Stefano Raggi 29294cb1f4 Transaction handlers - minor updates (#5956)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
- Move debug logging of order events from Brokerage to BrokerageTransactionHandler
- Override CurrentTimeUtc in BacktestingTransactionHandler
2021-10-01 19:09:08 -03:00
Stefano Raggi 6c42a266be IB brokerage - Update option positions at contract expiration (#5880)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* IB brokerage - Update option positions at contract expiration

* Add handling of live option expiration events

* Address review

- Nits

* Refactor option notification events [WIP]

* Add early exercise+assignment unit tests

* Fix sign bug in IB ExerciseOrder

* Address review

- log unexpected position
- add unit test cases for partial exercise/assignment
2021-09-30 13:48:41 -03:00
Martin-Molinero f082a40f34 Add missing comments (#5944)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
2021-09-27 18:48:53 -03:00
Martin-Molinero d739873daa Add aggregator reader (#5942)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Add BaseDataCollectionAggregatorReader

- Add enumerable for BaseDataCollection
- Add BaseDataCollectionAggregatorReader
- Clean up CollectionSubscriptionDataSourceReader

* Address reviews

- Add new FileFormat that will be handled by the new Reader
- Adding unit tests for the new BaseDataCollectionAggregatorReader
- Some DataSourceReader duplication cleaning up

* Minor adjustment after some live trading tests
2021-09-27 17:25:23 -03:00
Martin-Molinero 0f7dfe8ec8 Fix live ETF universe selection (#5938)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
- ETF live universe selection will behave the same as Coarse selection.
  Adding unit test
2021-09-23 15:11:35 -03:00
Martin-Molinero 0931ae868a Fix multiple coarse selection algorithms (#5927)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
- Fix for live trading multiple coarse selections. Adding unit test
  reproducing issue.
2021-09-14 20:27:09 -03:00
Colton Sellers 8358c394e1 Support all security types for benchmarks (#5919)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Adjust benchmark subscription to dynamically determine best resolution

* Get types from SM

* Add red/green test set for security types
2021-09-10 16:58:31 -03:00
Martin-Molinero 9acf68ad19 Bump pythonNet to 2.0.6 (#5881)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
2021-08-26 18:01:05 -03:00
Martin-Molinero 3410832ccf Alpha holdings state (#5770)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Alpha holdings state

- Alpha result packet will optionally provide the algorithms portfolio
  state

* Rename

* Convert AlphaStreamsPortfolio to data source

* Improvements on AlphaStreams algorithm

* Fix regression tests

* Add unit tests for EW AS PCM and fixing bugs

* Protobuf AlphaStreamsPortfoliot staState

- Protobuf AlphaStreamsPortfolioState. Adding unit tests
- Add variable TPV tests for EW ASPCM

* Add alpha license to Organization response

* Improvements EW AS PCM respects free portfolio value

* Fixes

- Update tests expected statistics results affected by MHDB custom data timezone fix
- Fix for Extensions.IsCustomDataType

* Fixed and adding more regression tests

- Adding support and regression test with alpha consumer with different account currency
- Adding support and regression test of a universe adding custom data
  types
- Add support and regression test for algorithm alpha consumer with existing holdings

* Add AlphaStreamsOrderEvent data type
2021-08-19 13:56:18 -03:00
Stefano Raggi d573a0f6c7 BrokerageTransactionHandler - handle OrderStatus.New in order update/cancel (#5867)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
2021-08-19 12:01:38 -03:00
Martin-Molinero 1fdb36ee89 Improve object store logs (#5864)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
2021-08-18 10:33:34 -03:00
Gerardo Salazar cd9e451ead Adds Support for ETF Constituent Universes (#5862)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* Adds support for ETF constituent universes

  * Adds filtering for universe data if it doesn't match the
    universe subscription type

  * Update mapping for ALL underlying Symbols if
    `Symbol.UpdateMappedSymbol(...)` is called. Required to support
    constituent ETF universes that might have mapping events

  * Delistings of composite constituent universe Symbol will result in
    removal of universe securities.

  * Added regression algorithms for ETF constituent mappings (C#/Python),
    along with data required to run locally

  * Refactor universe delistings in SubscriptionSynchronizer -
    big thank you to @Martin-Molinero :)

* Address review: update regression algorithms and add explanatory comments

* Address review: add additional checks to delisting regression algorithms

  * Adds new regression algorithm testing the addition of a universe
    without calling AddEquity() and asserts same behavior
2021-08-17 19:54:29 -03:00
Martin-Molinero 46829f0f01 Dependency clean up (#5834)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
2021-08-11 12:39:00 -03:00
Martin-Molinero e92230a7fe Fix SubscriptionDataReader auxiliary data out of order (#5843)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* Fix SubscriptionDataReader auxiliary data out of order

- Fix for SubscriptionDataReader emitting auxiliary data out of order
  due to sparse that and enumerator refresh logic. Adding regression
  algorithm
- Minor tweaks for unit tests failing on and off

* Fix for SDR internal subscriptions

- Fix for SubscriptionDataReader enumerator refresh for internal
  subscriptions which were ignored. Adding regression test
- Updating custom data regression algorithms affected by the issue
	- DropboxBaseDataUniverse was emitting a custom data point being end time
	- UnlinkedTraderBarIconicType was emitting a single data point
	  of the underlying SPY minute data when if should of emitted
	  all data points
2021-08-10 18:57:23 -03:00
IlshatGaripov 893ca2b41d Makes sure to create directory for results-destination-folder (#5842)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
2021-08-09 17:39:57 -03:00
Martin-Molinero e720147e50 Algorithm Manager will break when algorithm is deleted (#5840)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
2021-08-09 15:42:13 -03:00
Martin-Molinero b77f0122b2 Improve python exception parsing (#5831)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Improve python exception parsing

- Improve python exception parsing adding support for line shift. Adding
  unit tests

* PythonException revert change

* Centralized and normalize algorithm runtime handling

* Adding support for C# line and file exception report
2021-08-06 20:11:54 -03:00
Gerardo Salazar 650ce2e2b5 Removes References To Built-in Custom Data Implementations (#5793)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* WIP removal of custom data references & tests updates

* Regression algos updated and python algorithms moved to DataSource repos

* Fixes failing unit tests

* Add "LiveDataTypes" field to LiveNodePacket

* Adds Initialize() to IDataChannelProvider

  * Adds new extension method to convert
    HistoryRequest -> SubscriptionDataConfig

* Address review: Add protobuf definitions for Iconic data types

* Address review: re-adds DynamicSecurityDataAlgorithm as regression algo

  * Small adjustments to variable naming and documentation

* Move test files to respective DataSource repos
2021-08-02 19:24:45 -03:00
Martin-Molinero ef4fea7592 Fix engine initialization (#5814)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
2021-07-30 17:33:03 -07:00
Martin-Molinero 3cf9cc7d41 Update RestSharp to 106.12.0t (#5810)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
2021-07-30 17:45:59 -03:00
Martin-Molinero 41636c94c3 Common engine initialization (#5809)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
- Create Lean common engine initialization class, which will be called
  by research and Lean launcher
- Fix backtest deserialization bug due to failing to handle decimal
  value
2021-07-29 21:14:17 -03:00
Martin-Molinero 9cf4075957 Send invalid orderEvent for non shortable order (#5772)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
- TransactionHandler will see invalid OrderEvent for an order which
  tries to short an non shortable asset. Adding unit test, updating
  regression algorithm.
- Fix Api data price path normalization, found while testing with
shortable provider. Adding unit tests
2021-07-15 21:10:46 -03:00
Martin-Molinero 1ed390c5f7 Minor tweaks (#5771)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
- Improve readme file, add nuget link, update Lean home page
- Increase 10min look back to 60 for setting currency conversion
- Replace error message for trace when minute resolution history request
  is empty
2021-07-15 13:03:46 -03:00
Jasper van Merle f5224f8be9 Logging improvements (#5767)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
2021-07-14 20:53:09 -03:00
Colton Sellers 8d11bbf0ba Update to PythonNet 2.0.5 (#5764)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
2021-07-14 11:01:57 -03:00
Martin-Molinero f9adc6b136 Rename data permissions (#5753)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
2021-07-09 22:59:14 -03:00
Martin-Molinero 5bd0675423 Fix minor race condition in brokerage message handler (#5736)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Fix minor race condition in brokerage message handler

- Fix race condition where a message could be enqueued and
wait for a new call, not being processed ASAP

* Reduce BrokerageTransactionHandler logs
2021-07-03 19:46:31 -03:00
Martin-Molinero e5a50109b4 DownloaderDataProvider improvements (#5730)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
- Ignore margin files requests. Only provided by ApiDataProvider
- Reduce noise by LeanData.TryParse() call on failure
2021-07-02 14:03:16 -03:00
Rohan Sharma 8d352b6318 feat: Improve error message from ObjectStore (#5729) 2021-07-02 10:37:37 -03:00
Martin-Molinero 55195867a9 Reduce LTRH error log to debug (#5723)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
2021-06-30 16:44:52 -03:00
Martin-Molinero 0d45bf5101 Add composite DataProvider (#5714)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
- Add new CompositeDataProvider that will wrap multiple instance. Allows
  using for example the ApiDataProvider and a DownloaderDataProvider.
- Adjusting ProductJsonConverter after recent Modules addition
2021-06-29 12:40:31 -03:00
Martin-Molinero 66552b49bd Add new DownloaderDataProvider (#5709)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Add new DownloaderDataProvider

- Moving LeanDataWrite and IDataDownloader to common project
- Add new BaseDataDownloaderDataProvider with concurrency helper method
- Add new DownloaderDataProvider which will use a IDataDownloader or
  IBrokerage implementation as data source

* Add support for downloading symbol chains data
2021-06-28 15:56:10 -03:00
Colton Sellers f47775338c Expand Lean Default Charts and Reduce Clutter (#5680)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Update charts that are not empty and not default values

* Testing

* Filter final results

* Explicitly filter alpha charts if not needed

* Use filtered charts for Live results handler

* Refactor solution to manage alpha charts internally by creator

* nit comment fixes

* Implement drawdown as default chart

* Implement Capacity Estimate for backtest default chart

* Add default values to QC.Plotting

* Adjust drawdown calculation

* Only sample "Capacity" once a day

* Implement Volume chart

* Nit fixes

* Volume Chart adjustments and fixes

* Rename to "Assets Sales Volume"

* Implement Exposure Sampling

* Store algorithm currency symbol, round capacity to nearest 1k

* Add new plots to default charts

* nit change name

* Address reviews

* Reduce duplication and clean up exposure sampling

* Address reviews

* Improve sample exposure

* Address review

* Only enumerate holdings once

* nit - comments

* Post rebase fix

* Don't need to round anymore
2021-06-22 11:19:48 -03:00