928 Commits

Author SHA1 Message Date
Martin-Molinero 66552b49bd Add new DownloaderDataProvider (#5709)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Add new DownloaderDataProvider

- Moving LeanDataWrite and IDataDownloader to common project
- Add new BaseDataDownloaderDataProvider with concurrency helper method
- Add new DownloaderDataProvider which will use a IDataDownloader or
  IBrokerage implementation as data source

* Add support for downloading symbol chains data
2021-06-28 15:56:10 -03:00
Colton Sellers 74f0c89499 Option Chain Provider Symbol Map Resolve (#5664)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Use mapFileProvider to resolve symbol mapping

* Add reproducing unit test

* Only resolve mapping for types that require it

* address review
2021-06-16 19:42:38 -03:00
Alexandre Catarino c4e6e7f8dc Changes VIXCentralContango.F9 to Nullable (#5662)
Build & Test Lean / build (push) Has been cancelled
According to the data, the contract 8 months away from the front-month contract, `F9`, can be null.
2021-06-14 16:23:28 -07:00
Gerardo Salazar 250ffa1fec Adds VIXCentralContango data, modifies cached alternative data sourcing in live mode (#5648)
Build & Test Lean / build (push) Has been cancelled
* Adds VIXCentral Contango data

  * Refactor cached alternative data in live mode to
    use web endpoint if the data is not available locally

* Remove fallback to cache bucket for CBOE/FRED/EIA/VIXCentralContango

* Address review: force VIX as ticker for all VIXCentralContango data
2021-06-09 20:44:53 -03:00
Colton Sellers 0f0eeaff82 DataProvider Updates & Refactoring (#5641)
* Use DataProvider for fetching Shortable and FutureMargins files

* Use data provider for AllShortableSymbols

* Refactor of NeedToDownload logic

* Adjustments

* nit - cleanup

* Expand test cases

* Create the directory if it doesn't exist
2021-06-08 22:11:21 -03:00
Colton Sellers ab6027723f Refactor Api (#5251)
Build & Test Lean / build (push) Has been cancelled
* Use lean data key as param for request

* key -> filePath rename and some cleanup

* Refactor

* Add Organizations Endpoints

* Add some organization api wrapper objects

* Address namespace issue

* Reorganize Api Test into seperate files using one ApiTestBase

* Add Organization tests

* Use capitalized "API" test namespace to reduce amount of file changes

* Add License to test base

* Update /data endpoint functions and response objects

* Update ApiDataProvider Logic

* Handle deserialization of organization products

* Simplify converter

* Only throw for equity requests when not subscribed to map/factor files

* Add missing header

* Make arguement exception

* Api adjustments

* Add Zip factor and map file providers

- Common project will now reference Compression project and not the other way
  round.
- Adding Zip FactorFile and MapFile providers

* Refactor FactorFileProvider to use DataProvider to fetch files

* Use resulting MinimumDate in construction of FactorFile

* Nit FactorFile comments and arrangement

* Refactor MapFileProviders to use DataProvider for fetching files

* Refactor ZipFileProvider

* Clean up

* Refactor Backtesting Future/Option chain providers to use dataprovider

* Fixes for data/ endpoints and test adjustments

* Response objects adjustments/cleanups

* ApiDateProvider fixes and testing

* Add LocalZipFactorFileTests

* Update ApiDataProvider download test to verify stream is not null

* Implement posting of agreement summary and signed time

* Mark all Api related tests as explicit and document details on running

* Clarify default token on ApiTestBase

* Adjust summary

* Update Api responses for QCC, except org products which are sold in USD

* Implement cache expiration for zip MapFile and FactorFiles. Adding unit tests

* Fix multiple markets for ZipFactorFile provider

* Use Symbol as cache key

* Api.cs review

* Dispose of factorFileStream after reading

* Use zip.EntryFileNames

* Address a few reviews

* Few more fixes

* Address Api Review

* Add Job Org id to config

* Minor tweaks

* Compare with invariant culture

* Fixes Option Universe selection

* ZipEntryNameSubscriptionDataSourceReader will use IDataProvider

* Fix research

* Fix null reference exception

* Make duplicate log debug

Co-authored-by: Martin-Molinero <martin@quantconnect.com>
2021-06-07 11:37:44 -03:00
Colton Sellers 41f9d9da94 Remove SecurityInitializer from Universe Selection (#5418)
Build & Test Lean / build (push) Has been cancelled
* Attempt to remove SecurityInitializer from UniverseSelection stack entirely

* Fix broken Python Universe Selection model

* Remove mistake change (meant for another branch)
2021-05-28 14:53:13 -03:00
Gerardo Salazar 94fb9293dc Adds filtering of negative ticks in CoinAPI converter consolidation (#5597)
Build & Test Lean / build (push) Has been cancelled
* Adds filtering of negative ticks in CoinAPI converter

  * Tick data will remain the same, only aggregated data
    will have changes reflected in its output as a result

* Adds suspicious tick filtering to ticks

* Address review: Add Crypto Trade Tick suspicious test case
2021-05-25 19:38:53 -03:00
Colton Sellers 8ee9432bda Wicked Renko Consolidator Fix (#5589)
Build & Test Lean / build (push) Has been cancelled
* Round first value to Renko BarSize

* Add unit tests
2021-05-24 21:39:21 -03:00
Martin-Molinero 978da37cda Minor tweaks (#5580)
Build & Test Lean / build (push) Has been cancelled
* Minor tweaks

- Add AccountCurrency and symbol for live result packet
- Improve performance of Tick clone and reading

* Reduce unit test from 9s to 500ms
2021-05-20 12:00:42 -03:00
Martin-Molinero 9887a05d77 Fix FutureOptionMarginModel Margin Requirements (#5570)
Build & Test Lean / build (push) Has been cancelled
* Fix FutureOptionMarginModel Margin Requirements

- Fixes FuturesOptionsMarginModel margin requirements being zero due to
  underlying being null on model creation. Adding unit test

* Address reviews

- SecurityService will set underlying when creating a new security when
  provided
2021-05-18 14:04:30 -03:00
Martin-Molinero f3c386663b Feature .net 5 (#5505)
* Update projects to use .NET 5.0, the successor to .NET Core

* Fix ambiguous errors. Add IBAutomator net5

* Remove FXCM

* Upgrade IBAutomater to v1.0.51

ignored, and an empty message aborts the commit.

* Fix rebase

- Fix ambiguous Index
- Remove StrategyCapacity.cs
- Update System.Threading.Tasks.Extensionsy

* Remove unrequired references

* Fixes

- Travis will use dotnet, not nunit nor mono
- Remove mono from foundation image
- Fix python setup in research
- Fix unit tests

* Don't call ReadKey when input is redirected

* Fix ConsoleLeanOptimizer

* Research fixes

* Update comment

* Add vsdbg to Dockerfile

* Fixes

- Revert dockerfile FROM custom changes
- Adjust and fix regression algorithms
   - Option assignment will be deterministic in the order
   - 'Rolling Averaged Population' is calculated using doubles, updating
     expected values.
- Update readme, removing references to mono
- Add missing Py.Gil lock

* Replace ICSharp with .NET Interactive

* Fixes after rebase

* CSharp research fixes

- Adding new Initialize.csx that pre loads all assemblies
- Adjusting template research file
- Moving steps in dockerfilejupyter
- Fix unit tests and regression tests after rebase

Co-authored-by: Gerardo Salazar <gsalaz9800@gmail.com>
Co-authored-by: Stefano Raggi <stefano.raggi67@gmail.com>
Co-authored-by: Jasper van Merle <jaspervmerle@gmail.com>
2021-05-06 17:23:51 -03:00
Colton Sellers 865da14540 Scheduled Universe Timed Triggers Bug (#5521)
Build & Test Lean / build (push) Has been cancelled
* Add unit test

* Make test cover issue case

* Filter out times before startTimeUtc

* Refactor solution, fixes missing first date

* Fix case where none is expected

* Cleanup tests, add TriggerTimesNone

* Drop unused imports

* Add missing dipose call

Co-authored-by: Martin-Molinero <martin@quantconnect.com>
2021-05-05 11:23:38 -03:00
Colton Sellers 3579fecc58 Live Consolidator Bug Fix (#5429)
* Set _lastEmit before emitting, otherwise _workingBar is always null

* Aggregate bars if the data endTime is past lastEmit

* Add unit test

* Address Review

* Clean up unit tests

* Refactor solution to set consistent _lastEmit behaviour

* Add another unit test

* Make fixture non-parallelizable

* Undo last change, and adjust breaking test directly
2021-04-05 17:05:53 -07:00
Colton Sellers 1c3d849ad5 Fix Warnings V2 (#5436)
* Reconcile duplicated code

* Add License header

* CS0219 Fixes: Value assigned, but never used

* CA1507: Use nameof in place of string literals

* CS0108 : Hides Inherited Member; Use new keyword to overwrite formally

* CS0114: Hides inherited member; use override keyword

* CS0168: Variable is declared but never used

* Tests CS1062; using obsolete implicit Symbol -> String; fix via .ToString()

* CS0472: Non Nullable Obj getting Null Checked

* CS0067 Member not used; ignore all cases for future use

* CS00162 : Unreachable code; either removed or ignored for debugging and test cases

* CS0169 Remove non-used fields; ignore those that may be used in future

* CS0414; Field is assigned but never used.

* CS0618; Obsolete properties and members; Only fixes simple ones, rest will have to broken up

* CS0649; Field never assigned too

* CS0659 & CS0661 ; Overwrite operators and equals but not hashcode; I don't really override it but just call base

* Small comment fix

* Cleanup pragma statement
2021-04-02 11:20:01 -07:00
Colton Sellers e2a0873b7c Fix Lean Warnings V1 (#5408)
Cleanup all non-breaking warnings
2021-03-22 11:08:48 -07:00
Gerardo Salazar 4c085ff853 Adds Indexes and Index Options asset types (Backtesting/Live, IB only) (#5379)
* Add support for Index SecurityType  🚀 (#5364)

* Add Index SecurityType  🚀

* Extend SecurityIdentifier & Lean Data classes with Index support

* Add Index SecurityType  🚀

* Extend SecurityIdentifier & Lean Data classes with Index support

* Fixes

* Added index cross basic template demonstration

* WIP: Prototype index security type for LEAN as non tradable asset

* Re-adds Index entries to MHDB after rebase

* First steps to getting Index Options running

  * Looks at any instance where we pattern match for an option type
    and replaces it with a generic call to `.IsOption()` for easier
    extensibility in the future for additional option security types

  * Adds IndexOption security and misc. classes

  * Misc. changes, mainly related to any sort of special casing of
    equity options and made index options take the same path

* Enables index options data for backtesting

  * Adds new index options market hours to MHDB
  * Misc. bug fixes for index options
  * WIP: add live support for index options and indexes
  * Use OptionMarginModel for Index Options because they both use the
    same calculation for margin requirements

* Fixes contract not found errors on SPX index options and SPX index in IB

  * Turns out index options' last trading day is the day before expiry,
    which IB was expecting the last trading day.

* Add index option test cases (temp)

* LiveOptionChainProvider fix, use Symbol vs. ticker

  * Description updates to regression algorithms

* Fixes bug in live trading for indexes and index options

  * Adds overridable minimum price variation symbol property
  * Adds variable sized minimum price variation for index options
  * Adjusts symbol properties for index options
  * Misc. bug fixes

* Fixes option assignment simulation for European options

  * Updates index options regression algorithms (WIP)

* Fixes bug where index option exercise would trade index underlying

  * Fixes bugs where SecurityType.Index was getting flagged as tradable

* Regression algorithms updates and addresses review

  * Misc. style fixes and refactoring + a few bug fixes
  * Updates regression algorithms to run without runtime errors
  * Adds data for regression algos

* Sets DefaultOptionStyle on Canonical and support index options

* Update regression algos statistics

* Removes bad line in regression algorithm causing build to fail

* Minor tweaks

* Address review add comment about quoteBar parse scale

Co-authored-by: Balamurali Pandranki <balamurali@live.com>
Co-authored-by: Jared Broad <jaredbroad@gmail.com>
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
2021-03-12 20:46:23 -03:00
Colton Sellers 43a540cbb1 OpenInterest Bug Fixes (#5207)
* Filter values that are before subscription start time; also adjust starttime for OpenInterest

* Use data EndTime for comparison

* Allow Auxiliary data through

* Fix OpenInterest DataReader Logic

* Add regression

* Address review

* Ignore open interest for time slice

- TimeSliceFactory will directly ignore open interest for determining if
  the slice has data or not. Open interest will still be available
  through the Tick collection. Reverting some of the previous commits
  changes since they are no longer required.
- HistoryRequests and SubscriptionRequest will use AlwaysOpen exchange
  for open interest requests. Adding unit test reproducing issue
- Adding `BaseDataRequest` to avoid duplication logic.

* Make OpenInterest an internal feed and ignored by default in history

- Adding unit tests

* Revert SubscriptionFilterEnumerator Start time addition

Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
2021-02-18 19:04:29 -03:00
Juan José D'Ambrosio 2d99afc8c3 Refactor IPrimaryExchangeProvider.GetPrimaryExchange (#5302)
* Refactor IPrimaryExchangeProvider.GetPrimaryExchange 

now it returns a PrimaryExchange instead of a string

* Update MapFilePrimaryExchangeProvider.cs

Co-authored-by: Martin-Molinero <martin@quantconnect.com>
2021-02-17 12:43:41 -03:00
Juan José D'Ambrosio df4abd3c3a Bug algoseek taq writes tick exchanges correctly (#5301)
* Implements wrapper to get the exchange as a single letter representation

Also replace lower 'l' with capital 'L' in condition flags enums. it is a new warning on the compiler and it looks cool :)

* There are cases where the exchange is null or empty, in those cases return null

* Add tests for single character exchange representation

* Update GlobalTest.cs

Co-authored-by: Martin-Molinero <martin@quantconnect.com>
2021-02-16 12:51:08 -03:00
Gerardo Salazar bc4dbf5707 Removes extra day added to CBOE Time causing data to shift one day forward (#5266)
* Added test
2021-02-05 19:42:01 -03:00
Gerardo Salazar 5117609ea5 Fixes CBOE custom data consolidation (#5252)
* Adds regression algorithm testing consolidation of data.
2021-02-03 10:56:58 -03:00
Juan José D'Ambrosio 3a483fc4ec Migrate TradeConditionFlags and QuoteConditionFlags to Lean (#5227)
* Migrate TradeConditionFlags and QuoteConditionFlags to Lean

* Address revirew

Implement flag systems using long, this allows us to declare up to 64 flags
2021-02-01 13:11:30 -03:00
Martin-Molinero 5df29f16dd Fix thread safety of Mapfile exchange provider (#5208) 2021-01-21 19:02:50 -08:00
Juan José D'Ambrosio 765f2cab3b Add primary exchage to map files (#5194)
* Add main exchange to map file

minimum minimorum test

* Add MainExchange to MapFilRow IEquatable implementation
Also write CSV line correctly

* Normalize exchanges

Also add small test form MapFileRow

* Exchange name normalization

- Exchange name normalization.
- Tick will protobuf `ExchangeCode` which will set existing `Exchange`

* Include single character exchange code

* Encode exchange as byte, Include exchange name as Enum

* Update MapFileRow.cs to accept a byte instead a string for primary exchange

* Fix issues in encoding exchanges map

* Fix broken MapFile tests

* Rename Main Exchange property in MapFileRow

* Use BATS instead of BATSZ

* Add primary exchange from map file

* Move MapFilePrimaryExchangeProvider to its own file

* Implement Char for primary exchange in map files

* Fix broken tests after implementing char exchange into map file

* Add a new constructor for MapFileRow

it  accepts a PrimaryExxchange object as parameter

* Fix broken test 

WIP

* Address review 

WIP

* Handles unexpected encoded exchanges

* Keep addressing review

* Address review

* Fix broken tests

Make PrimaryExchange as default even if the IPrimaryExchangeProvider is not instantiated

* Revert breaking change.

* Add a check that resolved map file actually have content

The issue that leads to this change was caused by a map file resolved to APC in the CoarseFundamentalTop3Algorithm. 
The resolver returns a not-null MapFile, but it is empty.
To my best understand, the MapFile Resolver should return only existing map files in the map_files folder. But the LocalMapFileProvider returned a map file with a permtick that doesn't exist in the file system as CSV.

* Set PrimaryExchange.UNKNOWN as default

* Address review couple minor bugs

- Use SID market instead of USA
- Default ticke exchange is UNKNOWN

* Update map files with latest format and latest info

This include:
 - VXX, it was delisted, then VXXB appears and later it chnages from VXXB -> VXX
 - AAA delisting and a new listing with ticker AAA
 - FOXA ticker changue to TFCFA and later it was delisted

* Update test after updating latest map files

* Fix bug with factor files after updating map files

* Fix bug with factor files after updating map files

and revert statistics in regression tests

* Update OrderHash in regression test

Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
2021-01-21 18:46:12 -03:00
Aaron Janeiro Stone 5dcbed535d 5052 - Adds missing constructor routing (#5201)
* Adds missing constructor routing

* Minor tweak remove unrequired lines and format

Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
2021-01-21 10:49:27 -03:00
Aaron Janeiro Stone 7aefe07274 4553 - Refactor RenkoConsolidator (#5052)
* Separated Update calls and other non-common RenkoConsolidator.cs artifacts.

* Added type safe init for wicked

* Refactor; moved wicked-specific fields to WickedRenkoConsolidator.cs

* Addresses requested changes (and compacts constructors)

* Separated Update calls and other non-common RenkoConsolidator.cs artifacts.

* Added type safe init for wicked

* Refactor; moved wicked-specific fields to WickedRenkoConsolidator.cs

* Addresses requested changes (and compacts constructors)

* Removes base class for brevity

* Delete BaseRenkoConsolidator.cs

* Fixes naming for classic Renko consolidator

* Addresses review concerns

* Fixes missing null checks and missing typesafe constructor

* Adds warning

* WickedRenkoConsolidator -> RenkoConsolidator calls

* Separated Update calls and other non-common RenkoConsolidator.cs artifacts.

* Added type safe init for wicked

* Refactor; moved wicked-specific fields to WickedRenkoConsolidator.cs

* Addresses requested changes (and compacts constructors)

* Removes base class for brevity

* Fixes naming for classic Renko consolidator

* Separated Update calls and other non-common RenkoConsolidator.cs artifacts.

* Added type safe init for wicked

* Refactor; moved wicked-specific fields to WickedRenkoConsolidator.cs

* Addresses requested changes (and compacts constructors)

* Delete BaseRenkoConsolidator.cs

* Addresses review concerns

* Fixes missing null checks and missing typesafe constructor

* Adds warning

* WickedRenkoConsolidator -> RenkoConsolidator calls

* Addresses review

* unneeded tag removed

* Minor tweaks address reviews

Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
2021-01-18 15:54:15 -03:00
Gerardo Salazar 4d4ad92fa7 Easy To Borrow Hotfix (#5167) 2021-01-14 15:07:54 -08:00
Gerardo Salazar 3e7af17e02 Implements Easy To Borrow Reality Modeling (WIP) (#5159)
* Creates IShortableProvider interface + interface impls

  * This is the foundational work for the addition of the shortable stocks
    feature for backtesting and live trading. Note that the QCAlgorithm
    API and the backend transaction handling will be implemented
    separately.

* temp; work on preorder checks

* improve checks

* tmep

* Enforces ETB checks at BrokerageTransactionHandler

  * Adds ETB Shortable Provider to QCAlgorithm and IAlgorithm
  * Removes ETB check from PreOrderChecksImpl
  * Removes outdated test, new test to come soon for relevant class

* Work in progress commit, pushing for review.

  * Adds new regression algorithm (WIP)
  * Enhances ShortableProviderOrdersRejectedRegressionAlgorithm
  * Adds new methods to QCAlgorithm
  * Move IShortableProvider to have BrokerageModel own it
  * Comments updates
  * Adds new properties to Equity Security for shortable

  * Fixes bug where retrieving open order quantities would aggregate the
    submitted order quantity rather than the remaining order quantity for
    open orders.

  * Fixes bug where quantity of zero would result in a
    false positive in QCAlgorithm.Shortable(...)

  * Code refactoring and comments updates

Co-authored-by: Jared <jaredbroad@gmail.com>
2021-01-14 13:33:44 -08:00
Adalyat Nazirov 8d48996948 fix covesion to string: decimal are represented using comma decimal separator for non-US English cultures, but expected dot (#5138) 2021-01-07 17:35:51 -03:00
Martin-Molinero d3b9f8bc7a Protobuf OpenInterest (#5107)
- Add OpenInterest protobuf support. Adding unit test
- Use composer for future and option chain provider
2021-01-04 10:40:35 -03:00
Martin-Molinero f39acceadc Load fine fundamental from data cache folder (#4995)
* Load fine fundamental from data cache folder

* Add daily backwards lookup for live fine
2020-12-04 18:28:24 -08:00
Colton Sellers 041f1d9db5 Limit Futures Resolutions in BaseData (#4979)
* Limit futures support resolutions

* Fix broken tests

* Address review
2020-12-03 21:50:17 -03:00
Gerardo Salazar eb1181f5f7 Adds Futures Options Asset Class w/ IB Support (#4928)
* Adds preliminary universe selection for Future Options

* Fixes scaling issues with Future Options

* Fixes scaling multiplying by 10000x instead of using _scaleFactor

* Fixes scaling for Tick

* Revert changes to Tick since it divides the scaling factor

* Changes stale method name to new method name after rebase

* Fixes selection bugs, adds new methods, and adds unit tests

  * Fixes bug where Equity Symbol was created for an underlying
    non-equity Symbol, resulting in equity data trying to be loaded

  * Adds unit tests covering changes to Tick, QuoteBar, TradeBar and
    LeanData

  * Adds regression test for AddUniverseOption filter contract selection
    for Future Options

* Addresses review - modifies the AddFutureOption signature

  * Adds new AddUniverseOptions method overload
  * Removes and adds a new unit test
  * Misc. modifications to account for new changes

* Fixes bug where futures were loaded using default SID Date

  * Refactors and removes unnecessary work
  * Fixes regression algorithm, which previously made no trades

* Adds future option data

  * Adds the corresponding underlying data, in this case, futures data
    to enable usage of future options data

* Replaces data with new data (ES18Z20)

  * Improves Future chain filtering and updates regression stats

* Add AddFutureOptionContract API

* Expands regression and unit tests to test in finer detail

* Adds Python regression algorithms for AddFutureOption[Contract] methods

* Adds new unit test for BacktestingOptionChainProvider

  * Fixes bug with BacktesingOptionChainProvider where we
    attempted to load the Trades option chain first, resulting
    in breakage of backwards compatibility and limitation of the
    option chain.

  * Adds new regression algorithms (Py) to Algorithm.Python project

* Adds FutureOptionMarginBuyingPowerModel

  * Modifies code paths used to select margin model
  * Adds related unit tests for margin model

* Fixes issue with unit test and MHDB/SPDB lookup for Future Options

* Preliminary regression algorithm testing ITM call/put option buying

  * Fixes bug where fee model used did not find non-US market
    options fee model. We now use the futures fee model for future
    options because IB charges the same commissions per contract
    between futures and futures options

* Adds proper regression algorithm for ITM future options expiration

* Pushing broken algorithm for review

  * Currently, algorithm does not fill forward, causing
    a single future option to not get exercised when it is delisted.

* Adds FutureOptionPutITMExpiryRegressionAlgorithm

  * Improves existing regression algorithm for call side
  * Fixes bug in existing regression algorithm
  * Adds AAPL daily data to advance enumerator for ^^^ fix

* Adds additional future option regression algorithms

  * Adds Buy OTM expiration regression algorithms
  * Adds Sell ITM/OTM expiration regression algorithms
  * Adds missing Python regression algorithms

* Adds remaining Python regression algorithms and fixes issues

  * Fixes naming issues and statistics
  * Adds short option OTM regression algorithms (Py)

* Add license header and class comments to python algorithms

  * Cleans up comments and docstrings
  * Create Buy/Sell call intraday regression algo

* Redirects future options symbol properties to futures symbol properties

  * Asserts exercise/assignment price and updates stats in regression algos
  * Adds new unit test covering changes to SecurityService

* Adds comments and fixes failing test

* Partially fixes future option mis-calculated profit/loss

* Adjusts portfolio model to calculate FOP as a no upfront pay asset class

  * Updates regression algorithm statistics

* Begin IB FOP support

* Initial support for FOP IB data streaming, live í¾‰

  * Adds additional functionality to LiveOptionChainProvider
    - Allows querying CME API to retrieve option chains for CME products
    - Ultimately, it's also the groundwork for the CME
      LiveFutureChainProvider

  * Edits IDataQueueUniverseProvider interface to provide greater
    control to implementors of it

  * Misc. bug fixes required to get FOP data streaming through IB

* Adds comments, adds missing rategate call, and cleans up code

* Force exchange for FOP and Futures when no exchange is provided

* Fixes bug with Portfolio modeling across all asset classes

* Adds LiveOptionChainProvider tests for Future Options

* IB brokerage option symbol bug fixes and improvements

* Fixes contract multiplier lookup bug

  * Fixes issue where we attempted to subscribe to IB data feed with canonical security
  * Adds ES MHDB entry

* Reverts portfolio modeling changes for Futures Options

  * Since IB eats into our account's cash balance when
    a new FOP contract is purchased, we must model by applying funds
    to our cash whenever a new purchase/sell occurs.
    If we choose to model FOPs exactly as we do with futures, we
    will end up with an invalid TotalPortfolioValue on algorithm
    restart. By all means and purposes, FOPs are modeled exactly
    the same as equity options with respect to the portfolio.

  * Adds comments clarifying portfolio modeling and clarifies
    existing portfolio modeling comments with additional context.

* Fixes IB symbol lookup for future options

  * Fixes LiveOptionChainProvider looping 5 times per option chain
    request, even on success

  * Sets OptionChainedUniverseSelectionModel to produce a canonical
    future/future option/option Symbol to avoid creating two Symbols

  * Adds GLOBEX future option symbol mapping from future -> fop

* Fixes LiveOptionChainProvider loading wrong contract option chains

  * Fixes loading of futures options ZIP files when backtesting
  * Adds a string -> decimal JSON converter
  * Additional fixes/refactoring to the LiveOptionChainProvider

* Adds tests for changes to Symbol and LeanData

  * Reverts changes to IB-symbol-map

* Fixes Value for mapped future options tickers

  * Fixes Symbol test

* Changes path of future options to future's expiry date

  * Extra changes made to remove scaling from writing CSV
  * Added method to map from FOP Globex -> FUT Globex

* Fixes MOO and MOC orders for future options

  * Note: this order type might not be supported by IB or CME.

* Bug fixes and updates unit tests

* Update regression tests and data format

* Rebase changes

* 1. Multiple bug fixes for LiveOptionChainProvider, reverts IQFeed changes
2. Address review (partial): Code reuse and cleanup

1.
  * Modifies check in
    `AddFutureOptionShort(Call|Put)ITMExpiryRegressionAlgorithm`
    to ensure no buys have negative quantity

  * Code reuse changes in IB brokerage

  * Bug fix in IB brokerage where we assigned the FOP expiry
    as the futures expiry (requires verification)

  * Doc changes and adds missing summaries/license banners
  * Disposes of HTTP client resources in LiveOptionChainProvider
  * Renames classes and adds FutureOption folder in Common/Securities

2.
  * We revert back to the quotes API for the option chain,
    since the settlement API sometimes had missing strikes.

  * Fixes future option expiry being set as future's expiry
    in LiveOptionChainProvider

  * Fixes bug where wrong option chain was selected because of bad
    expiry lookup in the futures expiries returned from CME

  * Fixes multiple looping bug in LiveOptionChainProvider
  * Adds strike price scaling for LiveOptionChainProvider

  * Reverts IQFeed changes and simplifies interface upgrade changes

  Some additional challenges we'll have to solve as part of FOPs:

    - The `OptionSymbol.IsStandard` method makes the assumption that
      weeklies contracts follow the pattern equities follows, which
      does not apply to Futures Options

    - The Subscription created in:
        `OptionChainUniverseSubscriptionEnumeratorFactory`

      ...adds a Trade config. For illiquid contracts, this
      will delay universe selection for the option symbol
      until we get a trade. However, if we add a quote config,
      the data would instead be loaded based on the first quote
      we received from the brokerage.

      But since we're currently using a trade config, illiquid
      contracts won't start streaming data until it receives a trade.

NOTE: this commit is a WIP to addressing the reviews received in the PR,
but has been committed early for efficiency in the review process

* Fixes regression algorithms and misc. bugs

  * Fixes map file lookup for non-equity options
  * Adds extra assertion at end of algorithm to ensure no holdings are
    left when the algorithm ends.

  * Adds FutureOptionSymbol, allowing all contracts through as standard
  * Changes SPDB to allow defaulting to underlying future symbol
    properties if no entry is found for the given FOP

  * Fixes calls to SPDB in SecurityService, IBBrokerage
  * Reverts AAPL daily ZIP file to fix majority of regression algorithms
  * Adds FOPs symbol properties
  * Fixes existing symbol properties for a few futures
  * Adds tests for changes to Symbol Properties Database

* Removes string SPDB lookup method

  * Updates tests and misc callees of previous method

* Updates all regression tests to use data of already expired contracts

  * Adds Futures Options Expiry Functions tests
  * Adds required futures data for 2020-01-05

* Address review (partial): Expands test coverage and fixes tests

* Set option chain tests parallelism to fixture only

* Fixes broken test for contract month delta for FuturesOptionsExpiryFunctions

* Changes delisting date logic for Futures Options

* Address review: removes duplicate code, misc code fixes

  * Bug fix in MarketHoursDatabase.GetDatabaseSymbolKey() where
    we would use the underlying's Symbol for lookup in the MHDB

  * Adds missing license banner
  * Removes Futures Options entries from MHDB
  * Adds new tests

* Adds SecurityType.FutureOption

  * Converts any underlying comparisons and uses SecurityType directly
    instead for FOP specific behavior

  * Extra code modifications to acommodate new SecurityType

* Addresses review: fixes order fee bug on exercise

  * Additional bug fixes and adding of SecurityType.FutureOption
  * Updates regression algorithms OrderListHash

* Fixes various bugs in IB live implementation

  * Fixes bug setting the right contract expiration date for FOP
    generated by LiveOptionChainProvider

  * Adds new function to FuturesOptionsExpiryFunctions

  * Clarifies parameter names better in some functions/methods

  * Fixes bugs in IB brokerage for FOPs

* Address review - code cleanup and refactor

  * Remove MappingEventProvider, SplitEventProvider, and
    DividendEventProvider for Futures Options in
    CorporateEventEnumeratorFactory

* Address review: Use MHDB key resolver in SPDB

* Makes regression tests pass and adds comment for expiry issue

* Fixes MHDB lookup on string symbol method

* Adds Futures Options greeks regression algorithm (C# only)

* Adds explanitory comment on MHDB FOP lookup

* Remove python from FutureOptionCallITMGreeksExpiryRegressionAlgorithm
2020-12-02 21:49:59 -03:00
Gerardo Salazar c67845bd45 Fixes issue parsing SI data w/ unknown enum value (#4961)
* Extends tests to cover new additions
2020-12-01 20:17:25 -03:00
Colton Sellers 7174fcb9d7 RoundDown fix for PeriodCountConsolidatorBase (#4940)
* Limit rounding interval to 1 day, add SubtractRoundDown and AddRoundUp functions

* fix error message

* Only subtract round down if period is greater than a day

* Tests

* Comment clarification

* Pivot solution to simple fix

* fix error message

* Remove RoundDown/Up limitations; add remarks

* address review
2020-11-17 21:40:10 -03:00
James Kardatzke 46ef9a9dbb Quiver Quantitative Integration (#4869)
* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Delete QuiverHouseDataDownloader.cs

* Delete QuiverSenateDataDownloader.cs

* Delete QuiverPoliticalBetaDataDownloader.cs

* Add files via upload

* Delete QuiverHouse.cs

* Delete QuiverSenate.cs

* Delete QuiverPoliticalBeta.cs

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Delete QuiverDataAlgorithm.cs

* Add files via upload

* Add files via upload

* Add files via upload

* Addresses self review: Cleans up code and adds new unit tests

  * Adds Quiver* C# files to project
  * Adds new unit test for QuiverCongress
  * Adds Python algorithm example

* Address self reviews

- Adding some missing xml docs
- Removing unrequired imports.
- Minor rename from Date to ReportDate
- Live trading will throw InvalidOperationException

* Fixes for example algorithms

Co-authored-by: Gerardo Salazar <gsalaz9800@gmail.com>
Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
2020-11-13 20:07:17 -03:00
Martin-Molinero 91e8393aac DividedEventProvider distribution computation (#4828)
* DividedEventProvider distribution computation

- Update regression algorithm which was using a different reference
  price when calculating the dividend
- Adjust divided event provider to compute distribution using factor
  file reference price, if not 0. Adding unit tests
- For equities, only emit auxiliary data points for
  TradeBar configurations, not for QuoteBars, nor internal.

* Address reviews

- Split and Dividend event provider will throw an exception when there
  is no reference price available. Updating `wm` factor file which was
  missing references price and regression algorithms using WM.
- Updating unit tests asserting new exception
2020-11-11 15:47:51 -03:00
Martin-Molinero 61fda8b62c Protobuf will use recyclable memory stream (#4921)
* Protobuf will use recyclable memory stream

- Serialization will reuse recyclable memory streams
- Remove serialization of exchange and sale condition for ticks.
  Updating unit tests.
- There is no need to serialize BaseData.EndTime, covered by unit tests.

* Tick will keep a parsed sale condition property

* Readd tick exchange. Json ignore ParsedSaleCondition
2020-11-06 20:59:25 -03:00
Juan José D'Ambrosio 8e525c63fc Fix issue in TiingoNews converter if tickers contains space (#4908)
Also, if ticker contains pipe ( "|" )  it'll be ignored.
2020-11-02 13:58:39 -03:00
Juan José D'Ambrosio a26414d273 Add decimal places as parameters to get dividends with arbitrary precision (#4883)
* Add decimal places as parameters to get dividends with arbitrary precision

 
Also, increase precision when generating strings from factor files rows.

* Add xml documentation entries for new optional arguments.
2020-10-27 20:30:17 -03:00
Martin-Molinero bfd319c91e OptionChain and OptionContract improvements (#4804)
* OptionChain and OptionContract improvements

- QCAlgorithm.AddUniverse will return the added Universe instance.
- Adding new OptionChainedUniverseSelectionModel will monitor a Universe changes
  and will spwan new OptionChainUniverse from it's selections. Adding
  regression test Py/C#.
- Adding new OptionContractUniverse that will own option contracts and
  their underlying symbol. Adding regression test
- Fix double notification for security changes, bug seen in updated
  UniverseSelectionRegressionAlgorithm
- Remove UniverseSelection special handling for Option and Future chains
- Fix DataManager not removing SubscriptionDataConfigs for Subscriptions
  which finished before being removed from the universe
- Refactor detection of user added Universe so that they do not get
  removed after calling the UniverseSelectionModel

* Add check for option underlying price is set

* Address reviews

- Adding python regression algorithm for
  `AddOptionContractFromUniverseRegressionAlgorithm`
  and `AddOptionContractExpiresRegressionAlgorithm`
- Rename QCAlgorithm new api method to `AddChainedOptionUniverse`

* Fix universe refresh bug

- Fix bug where a universe selection refresh would cause option or
  future chain universes from being removed. Adding regression algorithm
  reproducing the issue.

* Rename new option universe Algorithm API method

- Rename new option universe Algorith API method from
  AddChainedOptionUniverse to AddUniverseOptions
- Rebase and update regression test order hash because of
  option expiration message changed
2020-10-09 10:52:50 -03:00
Gerardo Salazar 28160e1301 Fixes generation of explicit "null" value in Smart Insider Transactions/Intentions (#4817)
* Adds new unit tests covering changes and testing for old case

  * JsonConvert.SerializeObject would convert a `null` value into a
    literal string of "null" when writing to a file via the ToLine
    abstract method. We opt for an empty string whenever the underlying
    value is null so that the parsing works correctly later in the
    data loading cycle.
2020-10-06 10:20:59 -03:00
Gerardo Salazar e5b5f80d9d Fixes "0" group code value for FixturesAndAppliances (#4820)
* Fixes "0" group code value for FixturesAndAppliances

* Address review: fixes wrong variable name
2020-10-05 20:12:27 -03:00
Martin-Molinero ff5fc5db5d HistoryRequestFactory Hour ExtendedMarketHours (#4786)
- `HistoryRequestFactory` will not sure extended market hours for hour
  resolution when determining the start time using quantity of bars.
  Adding regression test
2020-09-29 18:16:27 -03:00
Stefano Raggi 934128cfa0 Binance Brokerage implementation (#4688)
* Binance Brokerage skeleton

* Market hours

* Implement Symbol Mapper

- known symbols available on /api/v1/exchangeInfo
- fiat currencies are pegged

* Implement GetCashBalance

* Implement GetAccountHoldings

- there are no pre-existing currency swaps
- cash balances are pulled and stored in the cashbook

* Implement GetOpenOrders

* Manage orders: PlaceOrder

* Manage orders: UpdateOrder

Update operation is not supported

* Manage orders: CancelOrder

* Messaging: order book

* Messaging: trades

* Messaging: combine streams

- connect to fake /ws/open channel on init
- case by channel name, but not event type

* Messaging: order depth updates

- ticker symbol is not enough as it pushes updates only once a second, this would be a very incomplete data stream
- fetch ticker snapshot if lastUpdateId == 0
- follow Binance instructions for keeping local orderbook fresh

* Messaging: user data streaming

- Request userDataStream endpoint to get listenKey
- keep listenkey alive
- handle order close event
- handle order fill event

* DataDownloader: get history

- we can aggregate minute candles for higher resolutions

* fix data stream

* Tests: FeeModel tests

* Tests: base brokerage tests

* Tests: download history

* Tests: symbol mapper

* Support StopLimit andd StopMarket orders

* StopMarket orders disabled

Take profit and Stop loss orders are not supported for any symbols (tested with BTCUSDT, ETHUSDT)

* Tests: StopLimit order

* Tests: crypto parsing

* Reissue user data listen key

* comment custom currency limitation

* rework websocket connections

* implement delayed subscription

* adapt ignore message

* add license banner

* use better suited exception type

* avoid message double parsing

* support custom fee values

* extract BinanceApiClient to manage the request/response between lean and binance

* use api events to invoke brokerage events

* do not allow to terminate session if it wasn't allocated.

* update binance exchange info

* tool to add or update binance exchange info

* ExchangeInfo basic test

* Rebase + Resharp

* Binance brokerage updates

- Fix sign bug in sell order fills
- Fix bug in GetHistory
- Remove duplicate symbol from symbol properties db

* Remove unused code

* Revert removal of account currency check

* Update symbols properties database

* Address review

* Address review

- Upgrade API endpoints from v1 to v3
- Updated sub/unsub for new subscription manager
- Subscribe best bid/ask quotes instead of full order book
- Added handling of websocket error messages
- Cleanup + refactor

* Update symbol properties database

* Remove list from symbol mapper

* Fix symbol mapper tests

* Address review

- Fix resubscribe after reconnect
- Fix quote tick edge case

* Fix EnsureCurrencyDataFeed for non-tradeable currencies

* Fix check in EnsureCurrencyDataFeed

* Reuse base class subscribe on reconnect

Co-authored-by: Adalyat Nazirov <aenazirov@gmail.com>
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
2020-09-28 15:57:10 -03:00
Martin-Molinero 776caea1d3 Convert a CoarseFundamental data point into a row (#4761)
* Convert a CoarseFundamental data point into a row

- Convert a CoarseFundamental data point into a row. Adding unit tests

* Remove fixed PriceFactor format
2020-09-25 20:31:46 -03:00
Adalyat Nazirov 1e3a1e3c43 Historical data requests start & time fix (#4733)
* regression tests

* fix: apply the same time convertion to history request time as for data time

* ver2

* fixup

* unit tests

* do not need this conversion because RoundDownInTimeZone returns in proper TZ

* comment

* requested changes

* refactoring

* more refactoring

* fix existing test: should return Sunday if open

* more symbols

* fix existing tests: submit new btcusd data

* fix

* add Cfd symbol
2020-09-24 14:51:35 -03:00
Adalyat Nazirov 75eac27795 Bug 4600 brokerage unsubscribe symbol granular (#4640)
* bitfinex unsubscribe impl

* IB implementation

* Alpaca Brokerage

* Fxcm Brokerage

* Bitfinex remake

* GDAX Brokerage

* move & rename DataQueueHandler subscription manager to better place for using in ToolBox

* performance tuning

use unified Channel class everywhere
BaseWebsocketBrokerage.GetSubscribed method relies on DQHSubscriptionManager

* TradierBrokerage implementation

* OandaBrokerage implementation

* FakeDataQueue subcription manager

* Fake SubscriptionManager for testing

* CoinApi implementation

* IEXDataQueueHandler implementation

* IQFeed Implementation

* unit tests

* fix SubscribeSymbols params

* thread safe subscribed symbols

* don't need to lock Keys prop, because it's thread safe

* accurate PolygonDataQueueHandler subscribe/unsubscribe methods

* Alpaca isnt DataQueueHAndler anymore

* remove unused variable

* remove redundant hashset

* fix coin api subscribe method

* disclaimer

* prettify code

* race condition?

* fix symbol conversion

* log if unsubscribed; change Channel Id type

* requested changes

* pass GetChannelName func as required parameter

* centralized logs

* use single name for all idqh with no difference in tick type

* change Oanda resubscribe method

* CanSubscribe doesn't change instance state - can be marked as static.

* change Oanda Subscription tracking

* style changes

* bitfinex fix

* fix spelling
2020-09-14 20:11:44 -03:00
Martin-Molinero 4a0fb30df5 Fixes for OpenInterest storing (#4712)
- Fix Slice.Get OpenInterest type. Adding unit test
- Fix for SecurityCache that wasn't storing OpenInterest types
- Updateing regression tests to covere these usages
2020-09-11 15:28:27 -07:00