* Adds preliminary universe selection for Future Options
* Fixes scaling issues with Future Options
* Fixes scaling multiplying by 10000x instead of using _scaleFactor
* Fixes scaling for Tick
* Revert changes to Tick since it divides the scaling factor
* Changes stale method name to new method name after rebase
* Fixes selection bugs, adds new methods, and adds unit tests
* Fixes bug where Equity Symbol was created for an underlying
non-equity Symbol, resulting in equity data trying to be loaded
* Adds unit tests covering changes to Tick, QuoteBar, TradeBar and
LeanData
* Adds regression test for AddUniverseOption filter contract selection
for Future Options
* Addresses review - modifies the AddFutureOption signature
* Adds new AddUniverseOptions method overload
* Removes and adds a new unit test
* Misc. modifications to account for new changes
* Fixes bug where futures were loaded using default SID Date
* Refactors and removes unnecessary work
* Fixes regression algorithm, which previously made no trades
* Adds future option data
* Adds the corresponding underlying data, in this case, futures data
to enable usage of future options data
* Replaces data with new data (ES18Z20)
* Improves Future chain filtering and updates regression stats
* Add AddFutureOptionContract API
* Expands regression and unit tests to test in finer detail
* Adds Python regression algorithms for AddFutureOption[Contract] methods
* Adds new unit test for BacktestingOptionChainProvider
* Fixes bug with BacktesingOptionChainProvider where we
attempted to load the Trades option chain first, resulting
in breakage of backwards compatibility and limitation of the
option chain.
* Adds new regression algorithms (Py) to Algorithm.Python project
* Adds FutureOptionMarginBuyingPowerModel
* Modifies code paths used to select margin model
* Adds related unit tests for margin model
* Fixes issue with unit test and MHDB/SPDB lookup for Future Options
* Preliminary regression algorithm testing ITM call/put option buying
* Fixes bug where fee model used did not find non-US market
options fee model. We now use the futures fee model for future
options because IB charges the same commissions per contract
between futures and futures options
* Adds proper regression algorithm for ITM future options expiration
* Pushing broken algorithm for review
* Currently, algorithm does not fill forward, causing
a single future option to not get exercised when it is delisted.
* Adds FutureOptionPutITMExpiryRegressionAlgorithm
* Improves existing regression algorithm for call side
* Fixes bug in existing regression algorithm
* Adds AAPL daily data to advance enumerator for ^^^ fix
* Adds additional future option regression algorithms
* Adds Buy OTM expiration regression algorithms
* Adds Sell ITM/OTM expiration regression algorithms
* Adds missing Python regression algorithms
* Adds remaining Python regression algorithms and fixes issues
* Fixes naming issues and statistics
* Adds short option OTM regression algorithms (Py)
* Add license header and class comments to python algorithms
* Cleans up comments and docstrings
* Create Buy/Sell call intraday regression algo
* Redirects future options symbol properties to futures symbol properties
* Asserts exercise/assignment price and updates stats in regression algos
* Adds new unit test covering changes to SecurityService
* Adds comments and fixes failing test
* Partially fixes future option mis-calculated profit/loss
* Adjusts portfolio model to calculate FOP as a no upfront pay asset class
* Updates regression algorithm statistics
* Begin IB FOP support
* Initial support for FOP IB data streaming, live í¾
* Adds additional functionality to LiveOptionChainProvider
- Allows querying CME API to retrieve option chains for CME products
- Ultimately, it's also the groundwork for the CME
LiveFutureChainProvider
* Edits IDataQueueUniverseProvider interface to provide greater
control to implementors of it
* Misc. bug fixes required to get FOP data streaming through IB
* Adds comments, adds missing rategate call, and cleans up code
* Force exchange for FOP and Futures when no exchange is provided
* Fixes bug with Portfolio modeling across all asset classes
* Adds LiveOptionChainProvider tests for Future Options
* IB brokerage option symbol bug fixes and improvements
* Fixes contract multiplier lookup bug
* Fixes issue where we attempted to subscribe to IB data feed with canonical security
* Adds ES MHDB entry
* Reverts portfolio modeling changes for Futures Options
* Since IB eats into our account's cash balance when
a new FOP contract is purchased, we must model by applying funds
to our cash whenever a new purchase/sell occurs.
If we choose to model FOPs exactly as we do with futures, we
will end up with an invalid TotalPortfolioValue on algorithm
restart. By all means and purposes, FOPs are modeled exactly
the same as equity options with respect to the portfolio.
* Adds comments clarifying portfolio modeling and clarifies
existing portfolio modeling comments with additional context.
* Fixes IB symbol lookup for future options
* Fixes LiveOptionChainProvider looping 5 times per option chain
request, even on success
* Sets OptionChainedUniverseSelectionModel to produce a canonical
future/future option/option Symbol to avoid creating two Symbols
* Adds GLOBEX future option symbol mapping from future -> fop
* Fixes LiveOptionChainProvider loading wrong contract option chains
* Fixes loading of futures options ZIP files when backtesting
* Adds a string -> decimal JSON converter
* Additional fixes/refactoring to the LiveOptionChainProvider
* Adds tests for changes to Symbol and LeanData
* Reverts changes to IB-symbol-map
* Fixes Value for mapped future options tickers
* Fixes Symbol test
* Changes path of future options to future's expiry date
* Extra changes made to remove scaling from writing CSV
* Added method to map from FOP Globex -> FUT Globex
* Fixes MOO and MOC orders for future options
* Note: this order type might not be supported by IB or CME.
* Bug fixes and updates unit tests
* Update regression tests and data format
* Rebase changes
* 1. Multiple bug fixes for LiveOptionChainProvider, reverts IQFeed changes
2. Address review (partial): Code reuse and cleanup
1.
* Modifies check in
`AddFutureOptionShort(Call|Put)ITMExpiryRegressionAlgorithm`
to ensure no buys have negative quantity
* Code reuse changes in IB brokerage
* Bug fix in IB brokerage where we assigned the FOP expiry
as the futures expiry (requires verification)
* Doc changes and adds missing summaries/license banners
* Disposes of HTTP client resources in LiveOptionChainProvider
* Renames classes and adds FutureOption folder in Common/Securities
2.
* We revert back to the quotes API for the option chain,
since the settlement API sometimes had missing strikes.
* Fixes future option expiry being set as future's expiry
in LiveOptionChainProvider
* Fixes bug where wrong option chain was selected because of bad
expiry lookup in the futures expiries returned from CME
* Fixes multiple looping bug in LiveOptionChainProvider
* Adds strike price scaling for LiveOptionChainProvider
* Reverts IQFeed changes and simplifies interface upgrade changes
Some additional challenges we'll have to solve as part of FOPs:
- The `OptionSymbol.IsStandard` method makes the assumption that
weeklies contracts follow the pattern equities follows, which
does not apply to Futures Options
- The Subscription created in:
`OptionChainUniverseSubscriptionEnumeratorFactory`
...adds a Trade config. For illiquid contracts, this
will delay universe selection for the option symbol
until we get a trade. However, if we add a quote config,
the data would instead be loaded based on the first quote
we received from the brokerage.
But since we're currently using a trade config, illiquid
contracts won't start streaming data until it receives a trade.
NOTE: this commit is a WIP to addressing the reviews received in the PR,
but has been committed early for efficiency in the review process
* Fixes regression algorithms and misc. bugs
* Fixes map file lookup for non-equity options
* Adds extra assertion at end of algorithm to ensure no holdings are
left when the algorithm ends.
* Adds FutureOptionSymbol, allowing all contracts through as standard
* Changes SPDB to allow defaulting to underlying future symbol
properties if no entry is found for the given FOP
* Fixes calls to SPDB in SecurityService, IBBrokerage
* Reverts AAPL daily ZIP file to fix majority of regression algorithms
* Adds FOPs symbol properties
* Fixes existing symbol properties for a few futures
* Adds tests for changes to Symbol Properties Database
* Removes string SPDB lookup method
* Updates tests and misc callees of previous method
* Updates all regression tests to use data of already expired contracts
* Adds Futures Options Expiry Functions tests
* Adds required futures data for 2020-01-05
* Address review (partial): Expands test coverage and fixes tests
* Set option chain tests parallelism to fixture only
* Fixes broken test for contract month delta for FuturesOptionsExpiryFunctions
* Changes delisting date logic for Futures Options
* Address review: removes duplicate code, misc code fixes
* Bug fix in MarketHoursDatabase.GetDatabaseSymbolKey() where
we would use the underlying's Symbol for lookup in the MHDB
* Adds missing license banner
* Removes Futures Options entries from MHDB
* Adds new tests
* Adds SecurityType.FutureOption
* Converts any underlying comparisons and uses SecurityType directly
instead for FOP specific behavior
* Extra code modifications to acommodate new SecurityType
* Addresses review: fixes order fee bug on exercise
* Additional bug fixes and adding of SecurityType.FutureOption
* Updates regression algorithms OrderListHash
* Fixes various bugs in IB live implementation
* Fixes bug setting the right contract expiration date for FOP
generated by LiveOptionChainProvider
* Adds new function to FuturesOptionsExpiryFunctions
* Clarifies parameter names better in some functions/methods
* Fixes bugs in IB brokerage for FOPs
* Address review - code cleanup and refactor
* Remove MappingEventProvider, SplitEventProvider, and
DividendEventProvider for Futures Options in
CorporateEventEnumeratorFactory
* Address review: Use MHDB key resolver in SPDB
* Makes regression tests pass and adds comment for expiry issue
* Fixes MHDB lookup on string symbol method
* Adds Futures Options greeks regression algorithm (C# only)
* Adds explanitory comment on MHDB FOP lookup
* Remove python from FutureOptionCallITMGreeksExpiryRegressionAlgorithm
* OptionChain and OptionContract improvements
- QCAlgorithm.AddUniverse will return the added Universe instance.
- Adding new OptionChainedUniverseSelectionModel will monitor a Universe changes
and will spwan new OptionChainUniverse from it's selections. Adding
regression test Py/C#.
- Adding new OptionContractUniverse that will own option contracts and
their underlying symbol. Adding regression test
- Fix double notification for security changes, bug seen in updated
UniverseSelectionRegressionAlgorithm
- Remove UniverseSelection special handling for Option and Future chains
- Fix DataManager not removing SubscriptionDataConfigs for Subscriptions
which finished before being removed from the universe
- Refactor detection of user added Universe so that they do not get
removed after calling the UniverseSelectionModel
* Add check for option underlying price is set
* Address reviews
- Adding python regression algorithm for
`AddOptionContractFromUniverseRegressionAlgorithm`
and `AddOptionContractExpiresRegressionAlgorithm`
- Rename QCAlgorithm new api method to `AddChainedOptionUniverse`
* Fix universe refresh bug
- Fix bug where a universe selection refresh would cause option or
future chain universes from being removed. Adding regression algorithm
reproducing the issue.
* Rename new option universe Algorithm API method
- Rename new option universe Algorith API method from
AddChainedOptionUniverse to AddUniverseOptions
- Rebase and update regression test order hash because of
option expiration message changed
- Adding new DataPermissionManager that will own the existing
datachannelProvider. Will assert configurations before added to the
data feed. Adding unit tests
- AlgorithmPythonWrapper will directly call base OnFrameworkData()
implementation skipping going through python and it's overhead
- Small performance improvement for adding Tick data points into a Ticks
collection
- For python always wrap slice with PythonSlice, so that slice.Get()
works even when no custom data is present, adding test.
Update test to the new crypto and equity subscriptions rules:
- Only consolidates trades
- Low resolution data are only trades.
Update and fix tests
Add missing minute sample files
Update Regression algorithms statistics
Crypto and Equities will consolidate trades by default
Daily and hourly resolution will return only trades for crypto and equities.
Update equity TAQ regression test
Add minute sample files
Updating regression algorithms
Update SpotMarket test cases
Add regression test for equity trades and quotes
- History request.
- Trades and quotes pumped into OnData.
- Subscriptions are added correctly.
Add sample data
Checks low resolution only subscribes to trade bars
- `UserDefinedUniverse` will no longer be removed as a data subscription.
- When `algorithm.AddData()` is called a universe selection data point
will be added to the `UserDefinedUniverse` subscription to trigger
selection and add the requested data.
- `DataManager` will make sure an active subscriptions
`SubscriptionDataConfig` will be present in the configuration collection
- Adding unit and regression tests
- Replacing `BaseData.AdjustResolution` for `DefaultResolution` and
`SupportedResolutions`
- Making `Resolution` nullable for `Algorithm.AddData` methods
- The `ISubscriptionDataConfigService` will set the default resolution
if none was provided and assert it is supported
- Fix bug with `PythonData` `IsSparseData` and `RequiresMapping`
resolution
- Adding `BaseData.AdjustResolution()` that should return a valid
resolution for the given data and security type.
This allows us to set a limitation which is useful to avoid invalid data
requests or unnecessary fill forward situations. The user will be
notified through a console message.
- Adding unit and regression test
- Updating example algorithms custom data resolution
- Some performance improvements. Wont change console color if
`SelectedOptimization` is defined
- Adding `SecurityCacheProvider` this class allows for two different
`Security` to share the same data type cache through different instance
of `SecurityCache`. This is used to directly access custom data types
through their underlying in a peformant maner
- Some small improvements
Adds IRegisteredSecurityDataTypesProvider to track all the data types
registered in the algorithm. Using this data, we can detect if it's
possible that we'll eventually have a property of a certain type name.
For example, consider I wish to use security.Data.TradeBar but we haven't
received any trade bars yet. Before this change a KeyNotFoundException
would be raised, but since we can determine that we expect to have trade
bars, we can detect this and return an empty list when we haven't received
any data yet. This also removes the need to constantly do a HasData<T>()
check before accessing the dynamic members.
Closes#3620
- Setting SPY as the default security benchmark
- The security benchmark subscription will be added at `UniverseSelection`
as an internal subscription. Using its own dedicated Security instance
which doesn't live in the algorithms.Securities collection.
- Reducing algorithms exposure to internal subscriptions
- `TimeSliceFactory` will prioritize higher resolution bars, when same
symbol is present twice (for non-internal subscriptionst)
- Adding regression test `CustomUniverseWithBenchmarkRegressionAlgorithm`
- Moving `MapFile` and `FactorFile` resolution to the data feed stack so
that they do not add a performance overhead to the algorithm thread.
- Create logging string messaged only if required.
- Calculate `FactorFileRow.PriceScaleFactor` the least amount of times
- `FactorFile` will keep an ordered reversed list with the dates.
Calling `Reverse()` on the `SortedList` is expensive.
- `MapFiles` will keep first and last date, so we don't need to call
`First()` and `Last()` multiple times.
- `Liquidate` will go through all the algorithms securities only if
necessary
- `TradeBar` parsing will not call `new T` for pure `TradeBar` which is
expensive
- Removing `Lazy` hash code and security type for the
`SecurityIdentifier`, replacing for direct initialization. Accessing the
`Lazy` value adds an overhead.
- Replacing `Enum` to string for hardcoded switch statement. `Enum.ToString` is expensive.
- `DataManager` will be lazy for counting the subscriptions for
determining if its above the limit
- Adding `AlgorithmSecurityValuesProvider.GetAllValues()`, removes the
need to fetch all the security keys twice.
- During universe selection, will not try to re add already added symbol
Implements `DataNormalizationMode` field in `UniverseSettings` to enable the settings of a desired `DataNormalizationMode` to the securities that are chosen in Universe Selection.
- Adding new `IAlgorithm.AccountCurrency { get; }` that will point to the
`Portfolio.CashBook.AccountCurrency`. Setter will be added in a
following PR.
- Base `Brokerage` class will now have a `AccountCurrency { get }`
pointing to the `IAccountCurrencyProvider`. Will be used by the different
brokerages implementations.
- Modifying `IFillModel` interface removing old methods and adding new
method `Fill Fill(FillModelParameters)`. This is a breaking change.
- Adding new `PythonWrapper` property for the `FillModel` base class.
This is required due to a limitation in PythonNet:
- Given C# class T has `virtual` methods A and B. Where method A
calls method B. And given custom python class L inherits class T.
And overrides method B. When class L calls
base method A (of class T). And when method A internally calls method B.
It will call C# implementation, not the python override. This issue
is solved going back to the `PythonWrapper`. Adding unit tests.
- Adding new `Parameters` property for the `FillModel` base class that will
be set by the call to `Fill()`. The `Parameters` property will be used by
the modified `XxxxFill()` implementations
- Adding new `Fill` result object for the `Fill(FillModelParameters)`
method
- Adding new check before removing a `SubscriptionDataConfig` due to the FillModels consuming the configuration collection when determining which Price to use. WIll now only remove the `SDC` if the symbol was removed from the selecting `universe`, this will avoid the case where the symbol is never deselected and the subscription ends, which happens at the end of all executions.
- Adding unit tests showcasing retro compatibility.
- Enabling C# `CustomModelsAlgorithm` as a regression test. Python
version returns a different result due to random number generation.
- Previously when a user requested to remove a universe, they would call
`RemoveSecurity()`->`UniverseManager.Remove()`-> `Universe.Dispose()` ->
collection change notification would call `DataManager()` that would
call `RemoveSubscription()` that *would not perform any operation* since
it would return `false` through
`if(subscription.IsUniverseSelectionSubscription &&
subscription.Universe.DisposeRequested)`. Previous comment stated that
the `Subscription` would be removed by the `SubscriptionSynchronizer`
via `SubscriptionFinished()` but this called would go through the same
if statemente as above, returning false and no performing any operation.
So the `Universe` `Subscription` was never removed.
PR https://github.com/QuantConnect/Lean/pull/2039
- In this PR I'm removing the mentioned `if` statement and the call
relation between `UniverseManager.Remove()` and
`DataManager.RemoveSubscription()` since this will be covered by the
`SubscriptionSynchronizer` in the next loop.
- Modified regression test (originally added with these changes at
https://github.com/QuantConnect/Lean/pull/2039) to verify this issue.
- After removing a `Subscription` entirely, we will now also remove the
matching `SubscriptionDataConfig` from the `SubscriptionManager`
- Adding a new regression test, which fails in `master`
- Fixing previous PR change that removed `Trace` logging when adding
and removing a `Subscription`
- Fixing a bug in the `UserDefinedUniverse` where calling `RemoveMember`
would cause the `SubscriptionDataConfig` to be re added to the
`SubscriptionManager` (not the `DF`). Found this through added
regression test.
- The class `Subscription` will internally track each `Universe`
`SubscriptionRequest` added or removed
- Adding regression test in which two different `Universe` request the
same `SubscriptionDataConfig` and one of them removes/adds it in a
toggle fashion (fails on current master)
- `UniverseSelection` pending removals will also be tracked by
`Universe`
- `UniverseDecorator` will overwrite the `Universe` member of
`SubscriptionsRequests` at `GetSubscriptionRequests()`. This is due to
`this != this,Universe`
- Adding `Subscription` unit tests covering expected behavior
- Extracting pending removals logic from `UniverseSelection` class into
a new helper class `PendingRemovalsManager`. This new class will keep
track of the `universes` requesting to remove a security. Adding unit tests
- Completly move `DataManager` in front of `DataFeed`. Specifically
`AddSubscription()` and `RemoveSubscription()` implementations. Also
removing IDataFeed.Subscriptions
- Creating new `Synchronizer` which will consolidate and combine `TimeSlice`
streaming for both live and backtests modes. Will live in front of the
`DataManager`
- `SubscriptionSynchronizer` will be the `FrontierTimeProvider` exposed,
and owned, by the new `Synchronizer`
- Updating existing tests
- Adding new ISecurityService and its implementation SecurityService.
Expose by SecurityManager.
This class will expose a method for creating new securities. The
SecurityManager is exposing this new interface, calling _securityService
internally, so Future/OptionUniverseSelectionModel.cs can use it
- Replacing all usages of SecurityManager.CreateSecurity for new
ISecurityService
- Modifying `Cash.cs` and `CashBook.cs` `EnsureCurrencyDataFeeds()` to
return newly added `SubscriptionDataConfig` instead of `Security`. This
will avoid using `Security.Subscriptions` at call site.
- Moving old SecurityManager.CreateSecurity into new
SecurityServiceTests.cs
- Removed usages of algorithm.Securities.key as a parameter for the
`ManualUniverseSelectionModel()` since those securities, added through
`AddXXXX` calls will be managed by the `UserDefinedUniverse`. This was
causing for Universes to try to add the same subscription requests
- Adding new empty constructor for ManualUniverseSelectionModel,
required for Python
- ManualUniverse will return any existing SDC for the
symbol. This is for maintaining existing behavior and
preventing breaking changes: Specifically motivated by usages of
Algorithm.Securities.Keys as constructor parameter of the
ManualUniverseSelectionModel, since those Symbols added by Addxxx()
calls will already be managed by the UserDefinedUniverse
- Making some format modifications to aling with used Lean formatting
- Adding new ISubscriptionDataConfigBuilder implemented by DataManager, exposed by
SubscriptionManager
- SecurityManager and SubscriptionManager will use new interface.
In a next PR, its intended for it to also be consumed by Universe.GetSubscriptionRequests().
- Moving LookupSubscriptionConfigDataTypes implementation into
DataManager
Refactoring previous commit so DataManager only keeps and receives a
reference to IAlgorithmSettigs, with the objective of reducing tight
coupling
Note: Investigate if IAlgorithmSettings.DataSubscriptionLimit should limit subscriptions or unique securities.
Today its using SubscriptionManagerSubscriptions.Where(x => !x.Symbol.IsCanonical()).DistinctBy(x => x.Symbol.Value).Count() @DataManager
Gerardo Salazar