1000 Commits

Author SHA1 Message Date
Gerardo Salazar eb1181f5f7 Adds Futures Options Asset Class w/ IB Support (#4928)
* Adds preliminary universe selection for Future Options

* Fixes scaling issues with Future Options

* Fixes scaling multiplying by 10000x instead of using _scaleFactor

* Fixes scaling for Tick

* Revert changes to Tick since it divides the scaling factor

* Changes stale method name to new method name after rebase

* Fixes selection bugs, adds new methods, and adds unit tests

  * Fixes bug where Equity Symbol was created for an underlying
    non-equity Symbol, resulting in equity data trying to be loaded

  * Adds unit tests covering changes to Tick, QuoteBar, TradeBar and
    LeanData

  * Adds regression test for AddUniverseOption filter contract selection
    for Future Options

* Addresses review - modifies the AddFutureOption signature

  * Adds new AddUniverseOptions method overload
  * Removes and adds a new unit test
  * Misc. modifications to account for new changes

* Fixes bug where futures were loaded using default SID Date

  * Refactors and removes unnecessary work
  * Fixes regression algorithm, which previously made no trades

* Adds future option data

  * Adds the corresponding underlying data, in this case, futures data
    to enable usage of future options data

* Replaces data with new data (ES18Z20)

  * Improves Future chain filtering and updates regression stats

* Add AddFutureOptionContract API

* Expands regression and unit tests to test in finer detail

* Adds Python regression algorithms for AddFutureOption[Contract] methods

* Adds new unit test for BacktestingOptionChainProvider

  * Fixes bug with BacktesingOptionChainProvider where we
    attempted to load the Trades option chain first, resulting
    in breakage of backwards compatibility and limitation of the
    option chain.

  * Adds new regression algorithms (Py) to Algorithm.Python project

* Adds FutureOptionMarginBuyingPowerModel

  * Modifies code paths used to select margin model
  * Adds related unit tests for margin model

* Fixes issue with unit test and MHDB/SPDB lookup for Future Options

* Preliminary regression algorithm testing ITM call/put option buying

  * Fixes bug where fee model used did not find non-US market
    options fee model. We now use the futures fee model for future
    options because IB charges the same commissions per contract
    between futures and futures options

* Adds proper regression algorithm for ITM future options expiration

* Pushing broken algorithm for review

  * Currently, algorithm does not fill forward, causing
    a single future option to not get exercised when it is delisted.

* Adds FutureOptionPutITMExpiryRegressionAlgorithm

  * Improves existing regression algorithm for call side
  * Fixes bug in existing regression algorithm
  * Adds AAPL daily data to advance enumerator for ^^^ fix

* Adds additional future option regression algorithms

  * Adds Buy OTM expiration regression algorithms
  * Adds Sell ITM/OTM expiration regression algorithms
  * Adds missing Python regression algorithms

* Adds remaining Python regression algorithms and fixes issues

  * Fixes naming issues and statistics
  * Adds short option OTM regression algorithms (Py)

* Add license header and class comments to python algorithms

  * Cleans up comments and docstrings
  * Create Buy/Sell call intraday regression algo

* Redirects future options symbol properties to futures symbol properties

  * Asserts exercise/assignment price and updates stats in regression algos
  * Adds new unit test covering changes to SecurityService

* Adds comments and fixes failing test

* Partially fixes future option mis-calculated profit/loss

* Adjusts portfolio model to calculate FOP as a no upfront pay asset class

  * Updates regression algorithm statistics

* Begin IB FOP support

* Initial support for FOP IB data streaming, live í¾‰

  * Adds additional functionality to LiveOptionChainProvider
    - Allows querying CME API to retrieve option chains for CME products
    - Ultimately, it's also the groundwork for the CME
      LiveFutureChainProvider

  * Edits IDataQueueUniverseProvider interface to provide greater
    control to implementors of it

  * Misc. bug fixes required to get FOP data streaming through IB

* Adds comments, adds missing rategate call, and cleans up code

* Force exchange for FOP and Futures when no exchange is provided

* Fixes bug with Portfolio modeling across all asset classes

* Adds LiveOptionChainProvider tests for Future Options

* IB brokerage option symbol bug fixes and improvements

* Fixes contract multiplier lookup bug

  * Fixes issue where we attempted to subscribe to IB data feed with canonical security
  * Adds ES MHDB entry

* Reverts portfolio modeling changes for Futures Options

  * Since IB eats into our account's cash balance when
    a new FOP contract is purchased, we must model by applying funds
    to our cash whenever a new purchase/sell occurs.
    If we choose to model FOPs exactly as we do with futures, we
    will end up with an invalid TotalPortfolioValue on algorithm
    restart. By all means and purposes, FOPs are modeled exactly
    the same as equity options with respect to the portfolio.

  * Adds comments clarifying portfolio modeling and clarifies
    existing portfolio modeling comments with additional context.

* Fixes IB symbol lookup for future options

  * Fixes LiveOptionChainProvider looping 5 times per option chain
    request, even on success

  * Sets OptionChainedUniverseSelectionModel to produce a canonical
    future/future option/option Symbol to avoid creating two Symbols

  * Adds GLOBEX future option symbol mapping from future -> fop

* Fixes LiveOptionChainProvider loading wrong contract option chains

  * Fixes loading of futures options ZIP files when backtesting
  * Adds a string -> decimal JSON converter
  * Additional fixes/refactoring to the LiveOptionChainProvider

* Adds tests for changes to Symbol and LeanData

  * Reverts changes to IB-symbol-map

* Fixes Value for mapped future options tickers

  * Fixes Symbol test

* Changes path of future options to future's expiry date

  * Extra changes made to remove scaling from writing CSV
  * Added method to map from FOP Globex -> FUT Globex

* Fixes MOO and MOC orders for future options

  * Note: this order type might not be supported by IB or CME.

* Bug fixes and updates unit tests

* Update regression tests and data format

* Rebase changes

* 1. Multiple bug fixes for LiveOptionChainProvider, reverts IQFeed changes
2. Address review (partial): Code reuse and cleanup

1.
  * Modifies check in
    `AddFutureOptionShort(Call|Put)ITMExpiryRegressionAlgorithm`
    to ensure no buys have negative quantity

  * Code reuse changes in IB brokerage

  * Bug fix in IB brokerage where we assigned the FOP expiry
    as the futures expiry (requires verification)

  * Doc changes and adds missing summaries/license banners
  * Disposes of HTTP client resources in LiveOptionChainProvider
  * Renames classes and adds FutureOption folder in Common/Securities

2.
  * We revert back to the quotes API for the option chain,
    since the settlement API sometimes had missing strikes.

  * Fixes future option expiry being set as future's expiry
    in LiveOptionChainProvider

  * Fixes bug where wrong option chain was selected because of bad
    expiry lookup in the futures expiries returned from CME

  * Fixes multiple looping bug in LiveOptionChainProvider
  * Adds strike price scaling for LiveOptionChainProvider

  * Reverts IQFeed changes and simplifies interface upgrade changes

  Some additional challenges we'll have to solve as part of FOPs:

    - The `OptionSymbol.IsStandard` method makes the assumption that
      weeklies contracts follow the pattern equities follows, which
      does not apply to Futures Options

    - The Subscription created in:
        `OptionChainUniverseSubscriptionEnumeratorFactory`

      ...adds a Trade config. For illiquid contracts, this
      will delay universe selection for the option symbol
      until we get a trade. However, if we add a quote config,
      the data would instead be loaded based on the first quote
      we received from the brokerage.

      But since we're currently using a trade config, illiquid
      contracts won't start streaming data until it receives a trade.

NOTE: this commit is a WIP to addressing the reviews received in the PR,
but has been committed early for efficiency in the review process

* Fixes regression algorithms and misc. bugs

  * Fixes map file lookup for non-equity options
  * Adds extra assertion at end of algorithm to ensure no holdings are
    left when the algorithm ends.

  * Adds FutureOptionSymbol, allowing all contracts through as standard
  * Changes SPDB to allow defaulting to underlying future symbol
    properties if no entry is found for the given FOP

  * Fixes calls to SPDB in SecurityService, IBBrokerage
  * Reverts AAPL daily ZIP file to fix majority of regression algorithms
  * Adds FOPs symbol properties
  * Fixes existing symbol properties for a few futures
  * Adds tests for changes to Symbol Properties Database

* Removes string SPDB lookup method

  * Updates tests and misc callees of previous method

* Updates all regression tests to use data of already expired contracts

  * Adds Futures Options Expiry Functions tests
  * Adds required futures data for 2020-01-05

* Address review (partial): Expands test coverage and fixes tests

* Set option chain tests parallelism to fixture only

* Fixes broken test for contract month delta for FuturesOptionsExpiryFunctions

* Changes delisting date logic for Futures Options

* Address review: removes duplicate code, misc code fixes

  * Bug fix in MarketHoursDatabase.GetDatabaseSymbolKey() where
    we would use the underlying's Symbol for lookup in the MHDB

  * Adds missing license banner
  * Removes Futures Options entries from MHDB
  * Adds new tests

* Adds SecurityType.FutureOption

  * Converts any underlying comparisons and uses SecurityType directly
    instead for FOP specific behavior

  * Extra code modifications to acommodate new SecurityType

* Addresses review: fixes order fee bug on exercise

  * Additional bug fixes and adding of SecurityType.FutureOption
  * Updates regression algorithms OrderListHash

* Fixes various bugs in IB live implementation

  * Fixes bug setting the right contract expiration date for FOP
    generated by LiveOptionChainProvider

  * Adds new function to FuturesOptionsExpiryFunctions

  * Clarifies parameter names better in some functions/methods

  * Fixes bugs in IB brokerage for FOPs

* Address review - code cleanup and refactor

  * Remove MappingEventProvider, SplitEventProvider, and
    DividendEventProvider for Futures Options in
    CorporateEventEnumeratorFactory

* Address review: Use MHDB key resolver in SPDB

* Makes regression tests pass and adds comment for expiry issue

* Fixes MHDB lookup on string symbol method

* Adds Futures Options greeks regression algorithm (C# only)

* Adds explanitory comment on MHDB FOP lookup

* Remove python from FutureOptionCallITMGreeksExpiryRegressionAlgorithm
2020-12-02 21:49:59 -03:00
Michael Handschuh b9974e6f54 Add OptionStrategyMatcher (#4924)
* Reformat/cleanup OptionStrategies

This file was breaking pretty much every style convention in LEAN.
There are other things that should be addressed in here that weren't,
such as passing non-argument names as argument names for ArgumentException,
as well as preferring constructors over property initializer syntax, but
such changes aren't being made to keep this commit strictly reformatting
instead of refactoring.

Added braces and reformatted long lines to make code more legible.

* Add abstract base class for OptionStrategy Option/UnderlyingLegData

This allows us to create either or and later use the Invoke method to push it
into the appropriate list on OptionStrategy.

* Replace O(n) option contract search with 2 O(1) TryGetValue calls

A better improvement would be resolving the correct symbol in the strategy, but
this immediate change is instead just focused on removing the O(n) search inside
a loop.

* Add BinaryComparison and supporting methods in ExpressionBuilder

We're going to use these binary comparisons to make it possible to create
ad-hoc queries against a collection of symbols. Using these expressions,
along with type supporting composition of these expression, we'll be able
to define predicates that can declaratively define how to match an option
strategy with an algorithms current holdings.

* Make GetValueOrDefault defaultValue optional

Was receiving ambiguous invocations leading to neading to invoke this
method explicitly (LinqExtensions.GetValueOrDefault) instead of being
able to use it as an extension method. Making the default value optional
seems to have resolved this ambiguity, leading to cleaner code in the
OptionPositionCollection (forthcoming)

* Add OptionPosition and OptionPositionCollection

OptionPositionCollection aims to provide a single coherent interface
for querying an algorithm's option contract positions and the underlying
equity's position in a performant, immutable way. The immutability of
the type is necessary for how the options matcher will operate. We need
to recursively evaluate potential matches, each step down the stack removing
positions from the collection consumed by each leg matched. This will enable
parallelism of the solution as well as simplifying the mental model for
understanding due to not needing to track mutations to the collection
instance.

* Add Option test class for easily creating option symbol objects

* Add OptionStrategyLegPredicate and OptionStrategyLegDefinition

The definition is a composition of predicates, and each predicate supports
matching against a set of pre-existing legs and a current position being
checked for the next leg (this leg). In addition to the matching functionality,
it also supports filtering the OptionPositionCollection, which is where much
of the work for resolving potential option strategies is done. By successively
filtering the OptionPositionCollection through successive application of predicates,
we wil end up with a small set of remaining positions that can be individually
evaluated for best margin impacts.

All of this effectively unrolls into a giant evaluation tree. Because of this
inherent structure, common in combinatorial optimization, the OptionPositionCollection
is an immutable type to support concurrent evaluations of different branches of
the tree. For large position collections this will dramatically improve strategy
resolution times. Finally, the interface between the predicate and the positions
collection is purposefully thin and provides a target for future optimizations.

* Add OptionStrategyDefinition and OptionStrategyDefinitions pre-defined definitions

The OptionStrategyDefinition is a definitional object provided a template and functions
used to match algorithm holdings (via OptionPositionCollection) to this definition. The
definition defines a particular way in which option positions can be combined in order to
achieve a more favorable margin requirement, thereby allowing the algorithm to hold more
positions than otherwise possible. This ties into the existing OptionStrategy classes and
the end result of the matching process will be OptionStrategy instances definiing all
strategies matched according to the provided definitions.

* Add OptionStrategyMatcher and Options class, w/ supporting types

OptionStrategyMatcherOptions aims to provide some knobs and dials to control how
the matcher behaves, and more importantly, which positions get prioritized when
matching. Prioritization is controlled via two different enumerators, one controller
which definitions are matched first and the other controller which positions are
matched first. Still unimplemented, is computing multiple solutions and running the
provided objective function to determine the best match. When this gets implemented,
we'll also want to implement the timer. For anyone looking to implement these features,
please talk with Michael Handschuh as there's a particular way of representing these
types of combinatorial solutions (a 3D tree) that can be used as a variation of the
linear simplex method for optimizing combinatorial problems.

* OptionStrategyMatcher: Address PR review comments

* Ensure created OptionStrategy legs all have the same multiplier

Each leg definition match gets it's own multiplier which indicates the
maximum number of times we matched that particular leg. When we finish
matching all legs, we pick the smallest multiplier from all the legs in
the definition and use that as the definition's multiplier. When we go
to create the OptionStrategy object we MUST make sure we're using the
multiplier from the definition and not from the individual legs.

This change fixes this issue and also provides a guard clause to ensure
that we're not trying to use a multiplier larger than what was matched.

* Add XML docs for OptionStrategyDefinitions from OptionStrategies
2020-12-02 18:42:24 -03:00
Colton Sellers fbf5300bb6 Block SetWarmUp after Algorithm has compeleted initialization (#4975)
* Fix for #4939

* Address review

* Logic fix
2020-12-02 13:44:04 -03:00
Colton Sellers 724e52c0b3 Make StartDate relative to Algorithm TimeZone in Live mode (#4871) 2020-10-19 15:44:08 -03:00
Aaron Janeiro Stone 888c443264 Moves cash brokerage/IExecutionModel test to post-init (#4826) 2020-10-12 11:14:20 -03:00
adam-may 03efc1b735 Remove internal usages of implicit operator in Indicator code (#4844)
* Convert usages of implicit operator in IndicatorBase and IndicatorDataPoint

* Reverting changes to example code
2020-10-12 10:31:05 -03:00
Martin-Molinero bfd319c91e OptionChain and OptionContract improvements (#4804)
* OptionChain and OptionContract improvements

- QCAlgorithm.AddUniverse will return the added Universe instance.
- Adding new OptionChainedUniverseSelectionModel will monitor a Universe changes
  and will spwan new OptionChainUniverse from it's selections. Adding
  regression test Py/C#.
- Adding new OptionContractUniverse that will own option contracts and
  their underlying symbol. Adding regression test
- Fix double notification for security changes, bug seen in updated
  UniverseSelectionRegressionAlgorithm
- Remove UniverseSelection special handling for Option and Future chains
- Fix DataManager not removing SubscriptionDataConfigs for Subscriptions
  which finished before being removed from the universe
- Refactor detection of user added Universe so that they do not get
  removed after calling the UniverseSelectionModel

* Add check for option underlying price is set

* Address reviews

- Adding python regression algorithm for
  `AddOptionContractFromUniverseRegressionAlgorithm`
  and `AddOptionContractExpiresRegressionAlgorithm`
- Rename QCAlgorithm new api method to `AddChainedOptionUniverse`

* Fix universe refresh bug

- Fix bug where a universe selection refresh would cause option or
  future chain universes from being removed. Adding regression algorithm
  reproducing the issue.

* Rename new option universe Algorithm API method

- Rename new option universe Algorith API method from
  AddChainedOptionUniverse to AddUniverseOptions
- Rebase and update regression test order hash because of
  option expiration message changed
2020-10-09 10:52:50 -03:00
Stefano Raggi 5f61456df8 Set Account Base Currency from Brokerage in Live Mode (#4806)
* Add property IBrokerage.AccountBaseCurrency

* Set AccountCurrency to brokerage AccountBaseCurrency

* Remove USD AccountCurrency check

* Fix Oanda account base currency

* Fix currency symbol in CashBook.ToString()

* Fix unit tests

* Address review

* Add DebugMessage when changing account currency

* Add debug message for brokerage account base currency

* Fix currency symbol in equity chart and runtime statistics

* Update unit tests
2020-10-09 09:58:01 -03:00
Michael Handschuh cf9b547e2e Refactor OptionExerciseOrder.Quantity to be consistent with other Order types (#4796)
* Improve information tracked in regression's {algorithm}.{lang}.details.log

The details.log file aims at providing a diff-able document that quickly and
easily provides actionable information. Since many regression algorithms use
the algorithm's debug/error messaging facilities to log various pieces of algo
state. This document also support a configuration option: regression-high-fidelity-logging'
that logs EVERY piece of data, again, with the aim of providing an easily diff-able
documenbt to quickly highlight actionable information. I may have missed omse key
pieces of information here, but now that the entire QC knows about this regression
tool, if additional information is required then hopefully it's easy enough at this
point to extend the RegressionResultHandler to suit our needs.

The RegressionResultHandler was initially implemented to provide a concise log of
all orders. This was achieved by simply using the Order.ToString method. While
testing/investigating OptionExerciseOrder behavior, it became evident that more
information was required to properly identify the source of potential failures or
differences between previous regression test runs. This change adds logging for
almost every IResultHandler method and additionally attempts to capture the
actual portfolio impact of every OrderEvent. This is accomplished by logging
the portfolio's TotalPortfolioValue, Cash properties and the security's
SecurityHolding.Quantity property.

This change also standardizes the timestamps used to folloow the ISO-8601 format.

When using the RegressionResultHandler, it is highly recommeded to also disable
'forward-console-message' configuration option to ensure algorithm Debug/Error
message logging is done synchronously to ensure correct ordering with respect to
log messages via Log.Debug/Trace/Error.

* Fix typo in options OrderTests test case name

* Update SymbolRepresentation.GenerationOptionTickerOSI to extension method

Far more convenient as an extension method

* Improve R# default code formatting rules

Many of these rule changes focus on improving the readability of code,
with a particular emphasis on multi-line constructs, chained method calls
and multi-line method invocations/declarations.

* Add braces, use string interpolation and limit long lines

* Refactor OptionExerciseOrder.Quantity to indicate change in #contracts

For all other order types, the Order.Quantity indicates the change in the algorithm's
holdings upon order execution for the order's symbol. For OptionExerciseOrder, this
convention was broken. It appears as though only exercise was initially implemented,
in which case only long positions were supported and a code comment indicated that
only positive values of quantity were acceptable, indicating the number of contracts
to exercise. At a later date, assignment simulation was added and utilized a negative
order quantity. This caused some major inconsistencies in how models view exercise
orders compared to all other order types. This change brings OptionExerciseOrder.Quantity
into alignment with the other order types by making it represent the change in holdings
quantity upon order execution.

This change was originally going to be much larger, but in order to minimize risks and to
make for an easier review experience, the additional changes will be committed separately
and pushed in their own PR. Some of the issues identified include:
* Manual Exercise (especially for OTM) is not covered
* Margin Calculations (in particular taking into account opposing contracts held)
* IBrokerage.OptionPositionAssigned is raised for exercise (later filtered by tx handler)

Fixes OptionPortfolioModelTests to use exercise model to properly model exercise of
non-account quote currency option contract.
2020-10-08 20:05:20 -03:00
Adalyat Nazirov a450cea8d0 keep actual exchange hours even it different from hours database (#4781) 2020-09-28 20:22:08 -03:00
Adalyat Nazirov 1e3a1e3c43 Historical data requests start & time fix (#4733)
* regression tests

* fix: apply the same time convertion to history request time as for data time

* ver2

* fixup

* unit tests

* do not need this conversion because RoundDownInTimeZone returns in proper TZ

* comment

* requested changes

* refactoring

* more refactoring

* fix existing test: should return Sunday if open

* more symbols

* fix existing tests: submit new btcusd data

* fix

* add Cfd symbol
2020-09-24 14:51:35 -03:00
Alexandre Catarino dd8dc473b3 Improve Error Message For Arithmetic Overflow In Decimal Cast of Double (#4728)
* SafeDecimalCast Throws Exception For Non-Finite Numbers

* Fixes Arithmetic Overflow Exception in QCAlgorithm.Trading Methods

Replace decimal cast for `SafeDecimalCast()`.

If the algorithm uses a non-finite number in QCAlgorithm trading methods, it will throw with an user-frieldly exception message.

* Fixes KellyCriterionProbabilityValue Calculation
2020-09-18 09:17:09 -03:00
Colton Sellers 34676f0c46 Bug 4668 python register indicator (#4670)
* Convert to timespan if possible

* Fix to support both timespans and custom consolidators

* Adjust tests for new wrapper requirements

* Cleanup
2020-08-29 02:53:32 -03:00
Colton Sellers b8674731a5 Feature 2456 custom Python consolidator support (#4637)
* DataConsolidator Wrapper for Python Consolidators

* Regression Unit Test

* Refactor Regression test

* Bad test fix

* pre review

* self review

* Add RegisterIndicator for Python Consolidator

* Python base class for consolidators

* Modify regression algo to register indicator

* unit test - attach event

* Test fix

* Fix test python imports

* Add license header file and null check

Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
2020-08-25 17:26:55 -03:00
Jovad Uribe fc6ccdbc11 Feature emv indicator addition (#4591)
* Create EaseOfMovementValue.cs

Added ease of movement file

* Update EaseOfMovementValue.cs

Added calculation for EMV and return its value

* Update EaseOfMovementValue.cs

* Update EaseOfMovementValue.cs

* Update EaseOfMovementValue.cs

* Update EaseOfMovementValue.cs

Rearranged code and removed all IndicatorBases

* Update EaseOfMovementValue.cs

* Update EaseOfMovementValue.cs

Added Min and Max Indicator

* Added Tests and Compile

* Fixed Bugs and Removed Reset

* Added Current Value and revereted to Bar data

* Fixed test file and refined indicator file

* TradeBar to IBaseDataBar

* Bug fixes

* bug fix

* Switching to TradeBar and attempting to fix Volume bug

There are two bugs that I have been having trouble fixing. 1. Cannot implicitly convert decimal to int (simple fix but cannot find where bug is taking place)
2. IBaseDataBar does not contain a definition for Volume

* Update EaseOfMovementValueTests.cs

* bug fix

* added data

* updated assertion

* added reset

* Update EaseOfMovementValueTests.cs

* Update EaseOfMovementValue.cs

* Update spy_emv.txt

I had the wrong test data in. Was throwing failed test for many pull requests.

* Update EaseOfMovementValue.cs

* Update EaseOfMovementValue.cs

* Update EaseOfMovementValue.cs

* Update EaseOfMovementValue.cs

* Cleaned Data

* Bug fixes

Fixed zero division error. Used better Test Data.

* removed readonly from _previous...price

* Update EaseOfMovementValue.cs

* Update EaseOfMovementValue.cs

* Update EaseOfMovementValue.cs

* bug fix

* Test Bug Fix

* EMV data from online

* Cosmetics

* Out of bounds fix

* Update EaseOfMovementValueTests.cs

* Update spy_emv.txt

* Update spy_emv.txt

* Added changes requested

Placed constructor first, fixed nullable type if statement, set 10,000 to default argument, added SMA.

* Update EaseOfMovementValue.cs

added variables

* Update EaseOfMovementValue.cs

* Update EaseOfMovementValue.cs

* Update EaseOfMovementValue.cs

* Fixed bugs

* Changed Delta, Added Assert

Create Indicator -> Update EMV -> Assert Status. Also changed delta from 1 to 0.000001 to improve test accuracy.

* Added unit test testing the SMA
2020-08-24 09:54:58 -03:00
Martin-Molinero 9cdb4a91c5 Refactor live data feed (#4636)
* Live Coarse universe refactor

- Live trading will source Coarse and Fine fundamental data directly
  from disk. Updating unit tests.

* Adds ILiveDataProvider interface

  * Adds wrapper for IDataQueueHandler implementations

  * Replaces IDataQueueHandler with ILiveDataProvider in
    LiveTradingDataFeed

  * Edits IDataQueueHandler documentation

* Maintains aggregation for current IDQH impls and skips for ILDF impls

  * Note: No unit test was created for this method, go back and TODO

* Protobuf Market data

- Adding protobuf support for Ticks, TradeBars and QuoteBars. Adding
  unit tests.

* Adds unit tests for LiveDataAggregator changes

  * Fixes bug where custom data was not handled as it was before
  * Fixes race condition bug because of variable reuse in class

* Add protobuf extension serialization

* Fixes for protobuf serialization

* Refactor

* Fix OptionChainUniverse

* replace BaseDataExchange pumping ticks with consolidators

* AlpacaBrokerage

* BitfinexBrokerage

* GDAXBrokerage

* OandaBrokerage

* InteractiveBrokers

* TradierBrokerage

* FxcmBrokerage

* PaperBrokerage

* etc

* WIP fixes for existing LTDF unit tests

* Fixes more LTDF unit tests

* make IDataAggregator.Update recieving Generic BaseData rather than Tick

* Change IDataQueueHandler.Subscribe method

* Some fixes after adding new commits

* Adds protobuf (de)serialization support for Dividend and Split

* Serialize protobuf with length prefix

* Fix missing LTDF unit tests

* Adds TiingoNews protobuf definitions

* fix comments

* more fixes on IQFeedDataQueueHandler

* disallow putting ticks into enumerator directly

* ScannableEnumerator tests

* fix OandaBrokerage

* AggregationManager unit tests

* fix AlpacaBrokerage tests

* fix InteractiveBrokers

* fix FxcmBrokerage tests

* call AggregationManager.Remove method on unsubscribe

* fix GDAX existing tests

* Fixes, refactor adding more tests for AggregatorManager

* Adds BenzingaNews protobuf definitions and round trip unit test

* Adds missing TiingoNews unit test to Protobuf round trip tests

* Improve sleep sequence of LiveSynchronizer

* need start aggregating first, and then can subscribe

* More test fixes and refactor

- Refactoring AggregationManager and ScannableEnumerator so the last is
  the one that owns the consolidator
- Adding pulse on the main LiveSynchronizer

* Improve performance of LEquityDataSynchronizingEnu

* Add missing Set job packet method

* Minor performance improvements

* Improvements add test timeout

- Improvements adding test timeout to find blocking test in travis

* Improve aggregationManager performance

* Testing improvements for travis

* Remove test timeouts

* More test fixes

- Adding more missing dispose calls and improving determinism

* fix IEXDataQueueHandler and tests

* Final tweaks to LTDF tests

* more AggregationManager tests

* consume and log ticks

* fix test: couldn't subscribe to Forex tickers

* change Resolution for all bar configs

* Improve RealTimeScheduleEventServiceAccuracy

* refactoring: move common code to base class

* fixed bug; unsubscribe SubscriptionDataConfig

* Small performance improvement

* Minor fixes

* Avoid Symbol serialization

* Fixes coarse selection in live mode

* Fix for live coarse

* Adds protobuf (de)serialization support for Robintrack

  * Adds round-trip unit test

* Minor performance improvements

* More minor performance improvements

* pass LiveNodePacket through to OandaBrokerage

* Fixes empty list becoming null value when deserializing with protobuf

* Reverts BZ live trading exception removal and fixes tests

* Refactor WorkQueue making it abstract

* Add try catch for composer

* Adds optional data batching period to LiveFillForwardEnumerator

* Override data-queue-handler with config

* Improve PeriodCountConsolidator.Scan performance

* Move batching delay to main Synchornizer thread

* Reverts addition of Robintrack protobuf definitions

* Give priority to config history provider if set

* Add Estimize protobuffing

- Add Estimize protobuffing support. Adding unit tests

* Always dispose of data queue handler

Co-authored-by: Gerardo Salazar <gsalaz9800@gmail.com>
Co-authored-by: Adalyat Nazirov <aenazirov@gmail.com>
2020-08-18 20:21:10 -03:00
Colton Sellers 64a99d1194 Feature shorcut helper for STC (#4623)
* Shorcut helper for STC

* Use name variable
2020-08-12 14:41:20 -03:00
Adalyat Nazirov e37c9dbb1a fix CreateIndicatorName; add unit tests 2020-07-31 22:50:31 +03:00
Adalyat Nazirov c2cac7fbee remove unused using 2020-07-31 22:03:57 +03:00
Adalyat Nazirov 22155bf3ce add extension methods for new indicators 2020-07-31 21:13:56 +03:00
Martin Molinero 39e56ea7c0 Address reviews
- Removing the need to call MHDB.SetEntry, this will be handled by the
  data manager
2020-06-15 20:05:12 -03:00
Martin Molinero b575880739 Coarse tiingo new universe selection algorithm
- Adding an example algorithm of a custom universe selection using
  coarse data and adding tiingo news. If conditions are met will add the
  underlying and trade it
- Adding required UniversePythonWrapper
2020-06-15 20:05:08 -03:00
Jared 94678d5810 Merge pull request #4504 from QuantConnect/feature-4366-add-security-by-symbol
Adds Overload to AddSecurity with Symbol Parameter
2020-06-11 13:23:17 -07:00
AlexCatarino dfc55e9380 Adds Overload to AddSecurity with Symbol Parameter 2020-06-11 00:20:38 +01:00
Martin Molinero 2d0a046c1e Reviews: Add unit test and improve exception message 2020-06-10 19:20:21 -03:00
Martin Molinero 170d12859f Address reviews
- Reduce code duplication. Handle some missing History<TradeBar> for Tick
  resolution.
2020-06-09 19:49:36 -03:00
Martin Molinero a3c5fd9871 Fix Resolution.Tick TradeBar history request
- Throw exception if requesting Resolution.Tick using the TradeBar
  history request API
- Adding unit tests, asserting current behavior
2020-06-08 21:46:13 -03:00
AlexCatarino fffd6a8493 Adds Optional Parameters to KaufmanAdaptiveMovingAverage
Our reference, http://stockcharts.com/school/doku.php?id=chart_school:technical_indicators:kaufman_s_adaptive_moving_average, points to three parameters, but only one is defined. The missing parameters were added to reflect the reference.
2020-05-31 14:44:53 +01:00
AlexCatarino 961f2292c2 Renames rebalancingParam to rebalance in PCMs 2020-05-27 20:48:29 +01:00
Martin Molinero 9631cd6749 Fix nuget QC dependency version 2020-05-21 16:21:59 -03:00
Martin Molinero 09c17b71ed Add assembly description to nuspec
- Due to bug in nuget failing to replace description and copyright token, setting
  .nuspec description to match assembly description
2020-05-21 14:42:12 -03:00
Martin-Molinero 1d0d4f4d7b Add assembly descriptions (#4442)
Add assembly descriptions
2020-05-21 09:35:12 -07:00
Martin Molinero 891cf4d69b Update to .Net 4.6.2 2020-05-15 17:32:46 -03:00
Martin Molinero 49abcd08b3 Normalize and improve Zero security price message 2020-05-13 21:19:37 -03:00
Martin Molinero b1d6d21d07 Refactor Future symbol Market
- Replacing usage of Market.USA as market for Future Symbols. Will now
  check SymbolPropertiesDatabase
2020-04-28 16:29:14 -03:00
Martin Molinero 8e71e48c07 PythonSlice
- AlgorithmPythonWrapper will directly call base OnFrameworkData()
  implementation skipping going through python and it's overhead
- Small performance improvement for adding Tick data points into a Ticks
  collection
- For python always wrap slice with PythonSlice, so that slice.Get()
  works even when no custom data is present, adding test.
2020-04-27 19:03:03 -03:00
Martin Molinero b3970bdc3b Fixes and rebase
- FineFundamentalSubscriptionEnumeratorFactory will not emit empty fine
  points
2020-04-24 17:21:28 -03:00
Martin Molinero 89a73a5671 Fix and regression tests
- Fix FineFundamentalFilteredUniverse for a custom universe
- Adding regression test
2020-04-24 17:21:28 -03:00
Huai Chun Shih fa79fc8a41 Let Python can AddUniverse(universe, fineFunc) 2020-04-24 17:21:28 -03:00
Martin Molinero 058c9e4eac Revert Crypto Hour/Daily QuoteBars
- Reverting removal of Crypto Hour/Daily QuoteBars
- Updating tests
- Adding unit test where Equities ignore QuoteBars
2020-04-06 10:33:02 -03:00
Martin Molinero 499248fe12 Revert "Revert "Adding Support for Equity L1 Quote Data""
This reverts commit 8cd8d206ca.
2020-04-06 10:32:59 -03:00
Martin Molinero 982ab3099f Remove Order based insights
- Removing order based insight generator.
- Removing regression and unit tests.
2020-04-01 20:21:19 -03:00
AlexCatarino 72f0f44f60 Updates Pythonnet Version
From 1.0.5.29 to 1.0.5.30
Ref.: https://github.com/QuantConnect/pythonnet/pull/42
2020-04-01 16:15:05 +01:00
Martin Molinero 20909a9569 Reduce some code duplication
- Small refactor to reduce some code duplication in
QCAlgorithm.Indicators Consolidate and ResolveConsolidator
2020-03-26 11:06:47 -03:00
Martin-Molinero e0f595cfd2 Merge pull request #4238 from QuantConnect/bug-4205-warmup-indicator-type
IndicatorWarmup data type fixes
2020-03-25 21:25:57 -03:00
Martin Molinero 2c950bc740 Improve comments
- Improve comments with an example
- Remove reference to regression algorithm
2020-03-25 21:21:13 -03:00
Martin Molinero f4e9b0cc3d Address reviews
- Add more and improve existing comments
2020-03-25 20:38:22 -03:00
Martin Molinero 2a2711d056 IndicatorWarmup fixes
- WarmupIndicator will be able to determine the correct type to use
- Fix bug in `History.GetMatchingSubscriptions()` which would use the
same TZ for exchange and data. Covered by regression algorithm.
- Consolidate will only infer `TickType` from `T` is not abstract
- Adding regression algorithm
- Slice will expose `Get(Type)` to get data by type, adding unit tests
2020-03-25 18:52:48 -03:00
Martin Molinero ccaa643bc7 Address self review
- Remove unrequired symbol check
- Dispose of `ZipDataCacheProvider` used in unit tests.
To avoid issues with other tests since zip files could remain open.
2020-03-25 17:41:11 -03:00
Martin Molinero 15cbe4afa2 Refactor QCAlgorithm.Indicators
- Make `GetSubscription` private and remove unrequired overload
- Add warning message when trying to warm up an indicator which does not
have a warm up period
2020-03-25 16:51:26 -03:00