957 Commits

Author SHA1 Message Date
Martin-Molinero f9adc6b136 Rename data permissions (#5753)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
2021-07-09 22:59:14 -03:00
Martin-Molinero e5a50109b4 DownloaderDataProvider improvements (#5730)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
- Ignore margin files requests. Only provided by ApiDataProvider
- Reduce noise by LeanData.TryParse() call on failure
2021-07-02 14:03:16 -03:00
Martin-Molinero 0d45bf5101 Add composite DataProvider (#5714)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
- Add new CompositeDataProvider that will wrap multiple instance. Allows
  using for example the ApiDataProvider and a DownloaderDataProvider.
- Adjusting ProductJsonConverter after recent Modules addition
2021-06-29 12:40:31 -03:00
Martin-Molinero 66552b49bd Add new DownloaderDataProvider (#5709)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Add new DownloaderDataProvider

- Moving LeanDataWrite and IDataDownloader to common project
- Add new BaseDataDownloaderDataProvider with concurrency helper method
- Add new DownloaderDataProvider which will use a IDataDownloader or
  IBrokerage implementation as data source

* Add support for downloading symbol chains data
2021-06-28 15:56:10 -03:00
Colton Sellers 74f0c89499 Option Chain Provider Symbol Map Resolve (#5664)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Use mapFileProvider to resolve symbol mapping

* Add reproducing unit test

* Only resolve mapping for types that require it

* address review
2021-06-16 19:42:38 -03:00
Colton Sellers 424fa36703 Organization Product ID and ApiDataProvider Fix (#5671)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Add ID to Organization Products

* Use Product ID to verify subscription
2021-06-16 15:01:53 -03:00
Colton Sellers 0f0eeaff82 DataProvider Updates & Refactoring (#5641)
* Use DataProvider for fetching Shortable and FutureMargins files

* Use data provider for AllShortableSymbols

* Refactor of NeedToDownload logic

* Adjustments

* nit - cleanup

* Expand test cases

* Create the directory if it doesn't exist
2021-06-08 22:11:21 -03:00
Colton Sellers 37cce922d3 Api Data Request Path Format Issue (#5639)
Build & Test Lean / build (push) Has been cancelled
* Address some format issues for data request paths

* Add reproducing test

* Drop logging for failure, let Api take care of it
2021-06-07 20:59:39 -03:00
Martin-Molinero 42c58d6e53 Ensure data is downloaded only once (#5635)
Build & Test Lean / build (push) Has been cancelled
- Ensure data is downloaded only once with the ApiDataProvider. Adding
  unit test
- Log invalid securityType ApiDataProvider requests once
- Assert Factor file and map file providers are compatible with the
  ApiDataProvider
2021-06-07 14:26:24 -03:00
Colton Sellers ab6027723f Refactor Api (#5251)
Build & Test Lean / build (push) Has been cancelled
* Use lean data key as param for request

* key -> filePath rename and some cleanup

* Refactor

* Add Organizations Endpoints

* Add some organization api wrapper objects

* Address namespace issue

* Reorganize Api Test into seperate files using one ApiTestBase

* Add Organization tests

* Use capitalized "API" test namespace to reduce amount of file changes

* Add License to test base

* Update /data endpoint functions and response objects

* Update ApiDataProvider Logic

* Handle deserialization of organization products

* Simplify converter

* Only throw for equity requests when not subscribed to map/factor files

* Add missing header

* Make arguement exception

* Api adjustments

* Add Zip factor and map file providers

- Common project will now reference Compression project and not the other way
  round.
- Adding Zip FactorFile and MapFile providers

* Refactor FactorFileProvider to use DataProvider to fetch files

* Use resulting MinimumDate in construction of FactorFile

* Nit FactorFile comments and arrangement

* Refactor MapFileProviders to use DataProvider for fetching files

* Refactor ZipFileProvider

* Clean up

* Refactor Backtesting Future/Option chain providers to use dataprovider

* Fixes for data/ endpoints and test adjustments

* Response objects adjustments/cleanups

* ApiDateProvider fixes and testing

* Add LocalZipFactorFileTests

* Update ApiDataProvider download test to verify stream is not null

* Implement posting of agreement summary and signed time

* Mark all Api related tests as explicit and document details on running

* Clarify default token on ApiTestBase

* Adjust summary

* Update Api responses for QCC, except org products which are sold in USD

* Implement cache expiration for zip MapFile and FactorFiles. Adding unit tests

* Fix multiple markets for ZipFactorFile provider

* Use Symbol as cache key

* Api.cs review

* Dispose of factorFileStream after reading

* Use zip.EntryFileNames

* Address a few reviews

* Few more fixes

* Address Api Review

* Add Job Org id to config

* Minor tweaks

* Compare with invariant culture

* Fixes Option Universe selection

* ZipEntryNameSubscriptionDataSourceReader will use IDataProvider

* Fix research

* Fix null reference exception

* Make duplicate log debug

Co-authored-by: Martin-Molinero <martin@quantconnect.com>
2021-06-07 11:37:44 -03:00
Colton Sellers ec77cf3ca5 Delisted Warning Time Correction (#5568)
Build & Test Lean / build (push) Has been cancelled
* Use DelistingDate for Warning event

* Update test

* Add regression

* Cleanup regression

* Add issue reproducing unit test
2021-05-19 15:19:38 -03:00
Martin-Molinero 9887a05d77 Fix FutureOptionMarginModel Margin Requirements (#5570)
Build & Test Lean / build (push) Has been cancelled
* Fix FutureOptionMarginModel Margin Requirements

- Fixes FuturesOptionsMarginModel margin requirements being zero due to
  underlying being null on model creation. Adding unit test

* Address reviews

- SecurityService will set underlying when creating a new security when
  provided
2021-05-18 14:04:30 -03:00
Colton Sellers 5bf72d2432 Adjust Volume for Splits (#5525)
Build & Test Lean / build (push) Has been cancelled
* Adjust TradeBar volume for splits

* Refactor solution to centralized function `Scale`

* Expand BaseData scale/adjust/normalize tests to assert volume behavior

* Add regression algorithm

* Update regression statistics

* Address review

* NOP to trigger cloud build

* adjust QuoteBar ask and bid size

* adjust broken regression

* Update tests to include QuoteBar adjustments

* nit cleanup

* Adjust Quote Tick bid and ask sizes

* Round volume and size values to nearest int in scale()

* nit

* Adjust regressions
2021-05-17 16:47:44 -03:00
Colton Sellers ed077da501 DataNormalizationMode Applies to AddSecurity (#5528)
Build & Test Lean / build (push) Has been cancelled
* Use UniverseSettings.DataNormalizationMode for securities added in Algorithm

* Stop SubscriptionUtils from forcing Adjusted mode

* Return behavior to original and add comments

* nit typo

* Add regression

* Add unit test that verifies DataNormalizationMode can be altered manually by security

* Cleanup and add Py version of regression
2021-05-10 20:58:50 -03:00
Martin-Molinero 1958ac2935 Minor adjustments for WorkScheduler (#5533)
Build & Test Lean / build (push) Has been cancelled
2021-05-07 19:28:14 -03:00
Colton Sellers 772eb4650c Stop UniverseSelection from removing the Symbol from the cache (#5513) 2021-04-30 19:01:29 -03:00
Colton Sellers a3836a64c4 Refactor StartDateLimited Warning Logs (#5510)
* Refactor to queue up start date changes to log on dispose

* Fix typo in warning collection used

Co-authored-by: Martin-Molinero <martin@quantconnect.com>
2021-04-30 10:55:16 -03:00
Colton Sellers db61cea3db Refactor logging for SubscriptionDataReader FactorFile processing (#5508)
* Collect and log symbols that had start date adjusted because of factor files

* Only post the message if the set has values

* Modify message to be [Symbol, Date] combo for more information on log

* Set a hard limit on the warning set to keep it from growing unnecessarily large

* Enforce a hard limit and improve message if full

* Enforce limit in log message because of possible threading adding past max

* Cleanup
2021-04-29 11:02:04 -03:00
Derek Melchin ba4e2b115c Update Summary of DefaultDataProvider (#5503) 2021-04-23 11:19:46 -03:00
Colton Sellers 5b10b3b509 Refactor of Auxiliary Data Filtering (#5485)
* Add futures regression reproducing the issue

* Cleanup futures regression

* Add Options regression

* Add IRegressionAlgorithmDefinition

* Add IRegressionAlgorithmDefinition

* Adjust regressions to compare to expiration date; delisting time is not always correct

* Refactor subscription enumerator filtering of AuxData to top of the stack

* Update broken tests to reflect the split/dividends/delisting subscriptions

* Update regression statistics because of Delisting EOD instead of at time

* Let custom data configurations bypass filter

* Adjust expected count, now that we are letting aux data through to history requests

* another small adjustment

* Use ExpectedBarCount in error message

* Add filter test for both cases

* Add some clarifying comments

* Verify we did recieve data in the regressions

* Make _shouldEmitAuxiliaryData private readonly

* Filter out aux data for history requests

* Remove option to not include aux data in subscriptions

* Refactor filtering to be more explicit for each piece of data; fixes universe selection aux data

* Cleanup comments after removed var

* Refactor order of filtering for performance reasons.
2021-04-19 11:10:47 -03:00
Colton Sellers ea11fb41c9 LiveDataBasedDelistingEventProvider Fix (#5471)
* Fix race condition in DelistingEventProvider

* Remove logging and speed up test time advancement

* Reduce timeout to original
2021-04-09 20:26:09 -03:00
Colton Sellers 1c3d849ad5 Fix Warnings V2 (#5436)
* Reconcile duplicated code

* Add License header

* CS0219 Fixes: Value assigned, but never used

* CA1507: Use nameof in place of string literals

* CS0108 : Hides Inherited Member; Use new keyword to overwrite formally

* CS0114: Hides inherited member; use override keyword

* CS0168: Variable is declared but never used

* Tests CS1062; using obsolete implicit Symbol -> String; fix via .ToString()

* CS0472: Non Nullable Obj getting Null Checked

* CS0067 Member not used; ignore all cases for future use

* CS00162 : Unreachable code; either removed or ignored for debugging and test cases

* CS0169 Remove non-used fields; ignore those that may be used in future

* CS0414; Field is assigned but never used.

* CS0618; Obsolete properties and members; Only fixes simple ones, rest will have to broken up

* CS0649; Field never assigned too

* CS0659 & CS0661 ; Overwrite operators and equals but not hashcode; I don't really override it but just call base

* Small comment fix

* Cleanup pragma statement
2021-04-02 11:20:01 -07:00
Alexandre Catarino d6072c88a5 Fixes LiveOptionChainProvider.FindOptionContracts (#5434)
`LiveOptionChainProvider.FindOptionContracts` handles the following data format:
`SPY  2021 03 26 190 000 C P  0 612 360000000`
where both existing `OptionRight.Call` and `OptionRight.Put` are declared in the same line.
2021-03-29 13:39:41 -07:00
Colton Sellers e2a0873b7c Fix Lean Warnings V1 (#5408)
Cleanup all non-breaking warnings
2021-03-22 11:08:48 -07:00
Gerardo Salazar 4c085ff853 Adds Indexes and Index Options asset types (Backtesting/Live, IB only) (#5379)
* Add support for Index SecurityType  🚀 (#5364)

* Add Index SecurityType  🚀

* Extend SecurityIdentifier & Lean Data classes with Index support

* Add Index SecurityType  🚀

* Extend SecurityIdentifier & Lean Data classes with Index support

* Fixes

* Added index cross basic template demonstration

* WIP: Prototype index security type for LEAN as non tradable asset

* Re-adds Index entries to MHDB after rebase

* First steps to getting Index Options running

  * Looks at any instance where we pattern match for an option type
    and replaces it with a generic call to `.IsOption()` for easier
    extensibility in the future for additional option security types

  * Adds IndexOption security and misc. classes

  * Misc. changes, mainly related to any sort of special casing of
    equity options and made index options take the same path

* Enables index options data for backtesting

  * Adds new index options market hours to MHDB
  * Misc. bug fixes for index options
  * WIP: add live support for index options and indexes
  * Use OptionMarginModel for Index Options because they both use the
    same calculation for margin requirements

* Fixes contract not found errors on SPX index options and SPX index in IB

  * Turns out index options' last trading day is the day before expiry,
    which IB was expecting the last trading day.

* Add index option test cases (temp)

* LiveOptionChainProvider fix, use Symbol vs. ticker

  * Description updates to regression algorithms

* Fixes bug in live trading for indexes and index options

  * Adds overridable minimum price variation symbol property
  * Adds variable sized minimum price variation for index options
  * Adjusts symbol properties for index options
  * Misc. bug fixes

* Fixes option assignment simulation for European options

  * Updates index options regression algorithms (WIP)

* Fixes bug where index option exercise would trade index underlying

  * Fixes bugs where SecurityType.Index was getting flagged as tradable

* Regression algorithms updates and addresses review

  * Misc. style fixes and refactoring + a few bug fixes
  * Updates regression algorithms to run without runtime errors
  * Adds data for regression algos

* Sets DefaultOptionStyle on Canonical and support index options

* Update regression algos statistics

* Removes bad line in regression algorithm causing build to fail

* Minor tweaks

* Address review add comment about quoteBar parse scale

Co-authored-by: Balamurali Pandranki <balamurali@live.com>
Co-authored-by: Jared Broad <jaredbroad@gmail.com>
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
2021-03-12 20:46:23 -03:00
Martin-Molinero 5236fc202d Adding Symbol.Canonical (#5383)
- Adding Symbol.Canonical property which will be cached. Adding unit and
  regression tests
2021-03-11 18:38:20 -03:00
Martin-Molinero 785c706d4c Live delisting will only ask mapfile for equities (#5336)
* Live delisting will only ask mapfile for equities

* Remove unrequired log error
2021-02-24 14:14:50 -03:00
Martin-Molinero 67b3873205 Reduce live coarse selection interval check to 10 min (#5332)
* Reduce live coarse selection interval check to 10 min

- Reduce live coarse selection interval check to 10 min. Adding unit
  tests, reducing code duplication in tests.

* Adjust unit test after self review
2021-02-23 16:37:08 -03:00
Colton Sellers 43a540cbb1 OpenInterest Bug Fixes (#5207)
* Filter values that are before subscription start time; also adjust starttime for OpenInterest

* Use data EndTime for comparison

* Allow Auxiliary data through

* Fix OpenInterest DataReader Logic

* Add regression

* Address review

* Ignore open interest for time slice

- TimeSliceFactory will directly ignore open interest for determining if
  the slice has data or not. Open interest will still be available
  through the Tick collection. Reverting some of the previous commits
  changes since they are no longer required.
- HistoryRequests and SubscriptionRequest will use AlwaysOpen exchange
  for open interest requests. Adding unit test reproducing issue
- Adding `BaseDataRequest` to avoid duplication logic.

* Make OpenInterest an internal feed and ignored by default in history

- Adding unit tests

* Revert SubscriptionFilterEnumerator Start time addition

Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
2021-02-18 19:04:29 -03:00
Martin-Molinero c30929f612 Add support for live trading delisting events (#5195)
* Add support for live trading delisting events

- Adding `LiveDelistingEventProviderEnumerator` that will emit delisting
  events. Adding unit tests
- Remove unused `LiveAuxiliaryDataEnumerator`.
- Refactor the corporate event providers being used per security type

* Self review: compare date component

* Add new LiveDataBaseDelistingEventProvider for equities

- Add support for delisting events for equities. Adding unit tests

* Fix cleanup of delisting equity subscription
2021-02-17 19:53:53 -03:00
Martin-Molinero 1606b12475 Fix IB future chain being started at reset hours (#5246)
* Fix IB future chain being started at reset hours

- Fix IB future chain being started at reset hours. Adding unit test

* Refactor CanAdvanceTime into CanPerformSelection
2021-02-01 18:53:33 -03:00
Martin-Molinero 8cbdfcaf28 Fix hour resolution mapping data handling (#5233)
- In some cases hour resolution data subscription would end early
  because of remapping not being correctly handled. Adding regression
  test
2021-01-28 20:41:04 -03:00
Martin-Molinero 3ccf428498 Adjust delisting liquidation time (#5203)
* Adjust delisting liquidation time

- Adjust delisting liquidation time to 15 min before market closes.
  Adding unit tests. Updating existing.
- Handle `Statistics.CompoundingAnnualPerformance` invalid calculation
  to avoid exception.
- AlgorithmManager will not handle delisting events in live trading
- Fix bug where due to a split driven liquidation matching delisting
  date a position in the option would remain open. Reproduced by
  `BasicTemplateOptionsFrameworkAlgorithm`

* Address review

- Address review add documentation on delisting offset span
2021-01-22 14:41:29 -03:00
Martin-Molinero f175ac6c6b Fix universe selection symbol cache removal (#5193)
- Only remove symbol from cache during universe selection if there is no
  other universe using it. Adding regression test reproducing issue
2021-01-21 15:48:33 -03:00
Martin-Molinero 592d037085 Fix delisted liquidation orders being cancelled (#5169)
* Fix delisted liquidation orders being cancelled

- Place delisted liquidation orders 10 min before market closes of 10
  min before the end of the delisting warning date. Adding regression
  test and unit tests. Updating existing tests.

* Fix failing python option unit tests

* Fix bug where positions would be open delisting liquidation

* Fix universe selection and delisting

- Delisting will happen ASAP for all types. Giving priority to close
  positions on derivates first
- Fix bug in universe selection where OptionChain would remove
  underlying even if holding a position in derivate.
- Updating regression tests statistics

* Add unit test, fix unit test expected stats
2021-01-18 22:02:40 -03:00
Adalyat Nazirov 9a0d943f1d Fill forward data points until delisted date (#4973)
* fill forward data points until expiry date

* wip: test non Delisting

* handle Aux not-Delisting dp; keep order while ff'ing

* simplify test checks

* wip

* fix liquidation expected date

* change statistics

* check EndTime, because it's used for time synchronization

* clean up code; remove duplicated test
2021-01-11 16:45:09 -03:00
Martin-Molinero 7f381cb135 Universe dispose and removal (#5134)
* Universe dispose and removal

- Fix bug where in some cases the disposed universe would not get removed
from UniverseManager not allowing new universes to get added correctly,
using the same symbol. Adding regression test
- Fix bug where in some cases selected symbols would not get removed
  when the parent universe would get disposed of.
- Minor normalization on UniverseSelection call

* Address review

- Add status plots for the added regression test

* Adding regression algorithm reproducing issue 3914
2021-01-07 12:03:17 -03:00
Adalyat Nazirov 54fce3f666 FillForward enumerator does not loop infinitely if fillforward resolution is different from data resolution (#5118)
* change filename template. add fillforward resolution suffix

* replicate GH issue 5116 on master

it's easy to reproduce on FXCM market by using data resolution different from fillforward resolution.
in this cases daily vs hour/minute were added

* change priorities of Time & EndTime values.

it's necessary to calculate EndTime  properly, and then we can align Time to it.
potential end time should be also calculated using ptoper TZ. Because of we store open hours without TZ movement TZ in market-hours it's necessary to reapply TZ

fix tests

* Miss 2AM bar on Sunday of DST; FF 2AM bar on Sunday ST

* use UTC TimeZone as baseline during comparison
2021-01-06 20:18:08 -03:00
Martin-Molinero 529d0a3634 Fix low resolution cache (#5080)
* Replace cache size for MB

* Remove setting capacity

* Custom cache implementation

* Reduce locks in custom cache

* Add cache performance unit test
2020-12-28 16:27:11 -03:00
Adalyat Nazirov 1988ad1ae5 Bug 4925 daylight out of order bar (#4941)
* test

* wip

* Revert "Fix duplicated history entries when contains daylight saving time change (#4700)"

Use proper rounding down

* regression test

* remove unused parameters

* more tests

* fix name and comment

* improve regression test

* more tests: oanda market hours

* re-apply Exchange TZ to bar EndTime

* fix expected results

* we can't substract minute because it can harm algorithm on minute resolution; so we could use tick?

* rename prop: conflict with QCAlgorithm.StartDate

* do not log messages to pass travis ci log limit

* assign loghandler in AlgorithmSetupHandler

* reference to PR for more description

* due to https://github.com/QuantConnect/Lean/pull/5039 we don't need to override it manually
2020-12-28 16:24:33 -03:00
Colton Sellers 8e410fcaf1 Update old Hourly/Daily data for ApiDataProvider (#5034)
* Update old Hourly/Daily data for ApiDataProvider

* Add unit test for IsOutOfDate

* Address review

* Fix reference to static function in test
2020-12-16 19:04:06 -03:00
Martin-Molinero f39acceadc Load fine fundamental from data cache folder (#4995)
* Load fine fundamental from data cache folder

* Add daily backwards lookup for live fine
2020-12-04 18:28:24 -08:00
Gerardo Salazar eb1181f5f7 Adds Futures Options Asset Class w/ IB Support (#4928)
* Adds preliminary universe selection for Future Options

* Fixes scaling issues with Future Options

* Fixes scaling multiplying by 10000x instead of using _scaleFactor

* Fixes scaling for Tick

* Revert changes to Tick since it divides the scaling factor

* Changes stale method name to new method name after rebase

* Fixes selection bugs, adds new methods, and adds unit tests

  * Fixes bug where Equity Symbol was created for an underlying
    non-equity Symbol, resulting in equity data trying to be loaded

  * Adds unit tests covering changes to Tick, QuoteBar, TradeBar and
    LeanData

  * Adds regression test for AddUniverseOption filter contract selection
    for Future Options

* Addresses review - modifies the AddFutureOption signature

  * Adds new AddUniverseOptions method overload
  * Removes and adds a new unit test
  * Misc. modifications to account for new changes

* Fixes bug where futures were loaded using default SID Date

  * Refactors and removes unnecessary work
  * Fixes regression algorithm, which previously made no trades

* Adds future option data

  * Adds the corresponding underlying data, in this case, futures data
    to enable usage of future options data

* Replaces data with new data (ES18Z20)

  * Improves Future chain filtering and updates regression stats

* Add AddFutureOptionContract API

* Expands regression and unit tests to test in finer detail

* Adds Python regression algorithms for AddFutureOption[Contract] methods

* Adds new unit test for BacktestingOptionChainProvider

  * Fixes bug with BacktesingOptionChainProvider where we
    attempted to load the Trades option chain first, resulting
    in breakage of backwards compatibility and limitation of the
    option chain.

  * Adds new regression algorithms (Py) to Algorithm.Python project

* Adds FutureOptionMarginBuyingPowerModel

  * Modifies code paths used to select margin model
  * Adds related unit tests for margin model

* Fixes issue with unit test and MHDB/SPDB lookup for Future Options

* Preliminary regression algorithm testing ITM call/put option buying

  * Fixes bug where fee model used did not find non-US market
    options fee model. We now use the futures fee model for future
    options because IB charges the same commissions per contract
    between futures and futures options

* Adds proper regression algorithm for ITM future options expiration

* Pushing broken algorithm for review

  * Currently, algorithm does not fill forward, causing
    a single future option to not get exercised when it is delisted.

* Adds FutureOptionPutITMExpiryRegressionAlgorithm

  * Improves existing regression algorithm for call side
  * Fixes bug in existing regression algorithm
  * Adds AAPL daily data to advance enumerator for ^^^ fix

* Adds additional future option regression algorithms

  * Adds Buy OTM expiration regression algorithms
  * Adds Sell ITM/OTM expiration regression algorithms
  * Adds missing Python regression algorithms

* Adds remaining Python regression algorithms and fixes issues

  * Fixes naming issues and statistics
  * Adds short option OTM regression algorithms (Py)

* Add license header and class comments to python algorithms

  * Cleans up comments and docstrings
  * Create Buy/Sell call intraday regression algo

* Redirects future options symbol properties to futures symbol properties

  * Asserts exercise/assignment price and updates stats in regression algos
  * Adds new unit test covering changes to SecurityService

* Adds comments and fixes failing test

* Partially fixes future option mis-calculated profit/loss

* Adjusts portfolio model to calculate FOP as a no upfront pay asset class

  * Updates regression algorithm statistics

* Begin IB FOP support

* Initial support for FOP IB data streaming, live í¾‰

  * Adds additional functionality to LiveOptionChainProvider
    - Allows querying CME API to retrieve option chains for CME products
    - Ultimately, it's also the groundwork for the CME
      LiveFutureChainProvider

  * Edits IDataQueueUniverseProvider interface to provide greater
    control to implementors of it

  * Misc. bug fixes required to get FOP data streaming through IB

* Adds comments, adds missing rategate call, and cleans up code

* Force exchange for FOP and Futures when no exchange is provided

* Fixes bug with Portfolio modeling across all asset classes

* Adds LiveOptionChainProvider tests for Future Options

* IB brokerage option symbol bug fixes and improvements

* Fixes contract multiplier lookup bug

  * Fixes issue where we attempted to subscribe to IB data feed with canonical security
  * Adds ES MHDB entry

* Reverts portfolio modeling changes for Futures Options

  * Since IB eats into our account's cash balance when
    a new FOP contract is purchased, we must model by applying funds
    to our cash whenever a new purchase/sell occurs.
    If we choose to model FOPs exactly as we do with futures, we
    will end up with an invalid TotalPortfolioValue on algorithm
    restart. By all means and purposes, FOPs are modeled exactly
    the same as equity options with respect to the portfolio.

  * Adds comments clarifying portfolio modeling and clarifies
    existing portfolio modeling comments with additional context.

* Fixes IB symbol lookup for future options

  * Fixes LiveOptionChainProvider looping 5 times per option chain
    request, even on success

  * Sets OptionChainedUniverseSelectionModel to produce a canonical
    future/future option/option Symbol to avoid creating two Symbols

  * Adds GLOBEX future option symbol mapping from future -> fop

* Fixes LiveOptionChainProvider loading wrong contract option chains

  * Fixes loading of futures options ZIP files when backtesting
  * Adds a string -> decimal JSON converter
  * Additional fixes/refactoring to the LiveOptionChainProvider

* Adds tests for changes to Symbol and LeanData

  * Reverts changes to IB-symbol-map

* Fixes Value for mapped future options tickers

  * Fixes Symbol test

* Changes path of future options to future's expiry date

  * Extra changes made to remove scaling from writing CSV
  * Added method to map from FOP Globex -> FUT Globex

* Fixes MOO and MOC orders for future options

  * Note: this order type might not be supported by IB or CME.

* Bug fixes and updates unit tests

* Update regression tests and data format

* Rebase changes

* 1. Multiple bug fixes for LiveOptionChainProvider, reverts IQFeed changes
2. Address review (partial): Code reuse and cleanup

1.
  * Modifies check in
    `AddFutureOptionShort(Call|Put)ITMExpiryRegressionAlgorithm`
    to ensure no buys have negative quantity

  * Code reuse changes in IB brokerage

  * Bug fix in IB brokerage where we assigned the FOP expiry
    as the futures expiry (requires verification)

  * Doc changes and adds missing summaries/license banners
  * Disposes of HTTP client resources in LiveOptionChainProvider
  * Renames classes and adds FutureOption folder in Common/Securities

2.
  * We revert back to the quotes API for the option chain,
    since the settlement API sometimes had missing strikes.

  * Fixes future option expiry being set as future's expiry
    in LiveOptionChainProvider

  * Fixes bug where wrong option chain was selected because of bad
    expiry lookup in the futures expiries returned from CME

  * Fixes multiple looping bug in LiveOptionChainProvider
  * Adds strike price scaling for LiveOptionChainProvider

  * Reverts IQFeed changes and simplifies interface upgrade changes

  Some additional challenges we'll have to solve as part of FOPs:

    - The `OptionSymbol.IsStandard` method makes the assumption that
      weeklies contracts follow the pattern equities follows, which
      does not apply to Futures Options

    - The Subscription created in:
        `OptionChainUniverseSubscriptionEnumeratorFactory`

      ...adds a Trade config. For illiquid contracts, this
      will delay universe selection for the option symbol
      until we get a trade. However, if we add a quote config,
      the data would instead be loaded based on the first quote
      we received from the brokerage.

      But since we're currently using a trade config, illiquid
      contracts won't start streaming data until it receives a trade.

NOTE: this commit is a WIP to addressing the reviews received in the PR,
but has been committed early for efficiency in the review process

* Fixes regression algorithms and misc. bugs

  * Fixes map file lookup for non-equity options
  * Adds extra assertion at end of algorithm to ensure no holdings are
    left when the algorithm ends.

  * Adds FutureOptionSymbol, allowing all contracts through as standard
  * Changes SPDB to allow defaulting to underlying future symbol
    properties if no entry is found for the given FOP

  * Fixes calls to SPDB in SecurityService, IBBrokerage
  * Reverts AAPL daily ZIP file to fix majority of regression algorithms
  * Adds FOPs symbol properties
  * Fixes existing symbol properties for a few futures
  * Adds tests for changes to Symbol Properties Database

* Removes string SPDB lookup method

  * Updates tests and misc callees of previous method

* Updates all regression tests to use data of already expired contracts

  * Adds Futures Options Expiry Functions tests
  * Adds required futures data for 2020-01-05

* Address review (partial): Expands test coverage and fixes tests

* Set option chain tests parallelism to fixture only

* Fixes broken test for contract month delta for FuturesOptionsExpiryFunctions

* Changes delisting date logic for Futures Options

* Address review: removes duplicate code, misc code fixes

  * Bug fix in MarketHoursDatabase.GetDatabaseSymbolKey() where
    we would use the underlying's Symbol for lookup in the MHDB

  * Adds missing license banner
  * Removes Futures Options entries from MHDB
  * Adds new tests

* Adds SecurityType.FutureOption

  * Converts any underlying comparisons and uses SecurityType directly
    instead for FOP specific behavior

  * Extra code modifications to acommodate new SecurityType

* Addresses review: fixes order fee bug on exercise

  * Additional bug fixes and adding of SecurityType.FutureOption
  * Updates regression algorithms OrderListHash

* Fixes various bugs in IB live implementation

  * Fixes bug setting the right contract expiration date for FOP
    generated by LiveOptionChainProvider

  * Adds new function to FuturesOptionsExpiryFunctions

  * Clarifies parameter names better in some functions/methods

  * Fixes bugs in IB brokerage for FOPs

* Address review - code cleanup and refactor

  * Remove MappingEventProvider, SplitEventProvider, and
    DividendEventProvider for Futures Options in
    CorporateEventEnumeratorFactory

* Address review: Use MHDB key resolver in SPDB

* Makes regression tests pass and adds comment for expiry issue

* Fixes MHDB lookup on string symbol method

* Adds Futures Options greeks regression algorithm (C# only)

* Adds explanitory comment on MHDB FOP lookup

* Remove python from FutureOptionCallITMGreeksExpiryRegressionAlgorithm
2020-12-02 21:49:59 -03:00
Colton Sellers 9d4014b7db Bug 4757 Synchronize Security Additions (#4879)
* Bump time by one tick in adding subscription

* Adjust regressions

* Change removal to immediate

* Use series.AddPoint instead of directly adding it

* Adjust regression

* Add checks for fixed behavior in regressions

* Address review
2020-12-02 21:20:30 -03:00
Martin-Molinero 91e8393aac DividedEventProvider distribution computation (#4828)
* DividedEventProvider distribution computation

- Update regression algorithm which was using a different reference
  price when calculating the dividend
- Adjust divided event provider to compute distribution using factor
  file reference price, if not 0. Adding unit tests
- For equities, only emit auxiliary data points for
  TradeBar configurations, not for QuoteBars, nor internal.

* Address reviews

- Split and Dividend event provider will throw an exception when there
  is no reference price available. Updating `wm` factor file which was
  missing references price and regression algorithms using WM.
- Updating unit tests asserting new exception
2020-11-11 15:47:51 -03:00
Martin-Molinero 6ef2ead929 Do not update price scale for fillforward data & IsFillForward flag fix (#4836)
* Do not update price scale for fillforward data

- Do no update price scale for fill forward data. FillForward data
  should keep using the prev scale for which it was created. Adding unit tests
- When cloning do not lose IsFillForward flag state, affects
QuoteBars/Ticks, does not affect TradeBars since they perform a memberwise clone.
Adding unit tests

* Auxiliaries shouldn't really affect on applied price factor scale.

Despite we can receeive FillForward'ed data points, corresponding
Auxiliaries for them are not FillForward so we do meet the condition
and then refresh price factor. As a result all futher FF data points are scaled too.

* Regression algorithm to check that FillForward'ed data points arrived with last real price factor

* Add trade for regression algorithm

- Minot tweaks and adding trade for new regression algorithm.
- Updating AddOptionContractExpiresRegressionAlgorithm because it is
  using the symbol for which new data was added.

Co-authored-by: Adalyat Nazirov <aenazirov@gmail.com>
2020-10-09 18:09:30 -03:00
Martin-Molinero bfd319c91e OptionChain and OptionContract improvements (#4804)
* OptionChain and OptionContract improvements

- QCAlgorithm.AddUniverse will return the added Universe instance.
- Adding new OptionChainedUniverseSelectionModel will monitor a Universe changes
  and will spwan new OptionChainUniverse from it's selections. Adding
  regression test Py/C#.
- Adding new OptionContractUniverse that will own option contracts and
  their underlying symbol. Adding regression test
- Fix double notification for security changes, bug seen in updated
  UniverseSelectionRegressionAlgorithm
- Remove UniverseSelection special handling for Option and Future chains
- Fix DataManager not removing SubscriptionDataConfigs for Subscriptions
  which finished before being removed from the universe
- Refactor detection of user added Universe so that they do not get
  removed after calling the UniverseSelectionModel

* Add check for option underlying price is set

* Address reviews

- Adding python regression algorithm for
  `AddOptionContractFromUniverseRegressionAlgorithm`
  and `AddOptionContractExpiresRegressionAlgorithm`
- Rename QCAlgorithm new api method to `AddChainedOptionUniverse`

* Fix universe refresh bug

- Fix bug where a universe selection refresh would cause option or
  future chain universes from being removed. Adding regression algorithm
  reproducing the issue.

* Rename new option universe Algorithm API method

- Rename new option universe Algorith API method from
  AddChainedOptionUniverse to AddUniverseOptions
- Rebase and update regression test order hash because of
  option expiration message changed
2020-10-09 10:52:50 -03:00
Michael Handschuh 194ed59cbe Check collection for null (#4763)
In OptionChainUniverseDataCollectionEnumerator.IsValid we check the underlying
foir null and ensure that there are data points but we forget to check the actual
collection object for null, which it is initialized to. The collection remains
null in the case where no data is found.
2020-09-28 10:39:59 -03:00
Adalyat Nazirov 1e3a1e3c43 Historical data requests start & time fix (#4733)
* regression tests

* fix: apply the same time convertion to history request time as for data time

* ver2

* fixup

* unit tests

* do not need this conversion because RoundDownInTimeZone returns in proper TZ

* comment

* requested changes

* refactoring

* more refactoring

* fix existing test: should return Sunday if open

* more symbols

* fix existing tests: submit new btcusd data

* fix

* add Cfd symbol
2020-09-24 14:51:35 -03:00
Michael Handschuh fb2f846159 Bug 4722: Prevent Repetitive Factor File Numerical Precision Warnings (#4742)
* Add DataProviderEventArgs base class for IDataProviderEvents event args

This base class includes a Symbol property. This will empower event listeners
to make decisions based on which security (symbol) raised the event. The immediate
use case is preventing multiple numerical precision messages for the same security.
This pattern can equally be applied to other error messages that are raised each
time a security is added to a universe.

See: #BUG-4722

* Update ConcurrentSet.Add to use ISet<T>.Add returning bool

It's a very common pattern to use if (set.Add(item)) which is enabled
via bool ISet<T>.Add(item) but not enabled via void ICollectiont<T>.Add(item).
This change simples changes the default Add implementation to use the ISet<T>
overload and relegates the ICollection<T>.Add implementation to be explicit.

See: #BUG-4722

* Prevent multiple numerical precision messages for same symbol

If a security is continually added/removed from a universe, then the user will
see this message each time the security is added. This results in some spam.
This change simply remembers for which symbols we've notified the user about the
numerical precision issue.

Fixes: #BUG-4722
2020-09-22 21:06:59 -03:00