* Add underlying holdings to regression result handler details log
When debugging option exercise/assignment issues it's useful to see the
underlying holdings at the time the option contract fill event is processed.
Also adds the full symbol string to the top of the order event section.
The Symbol.Value was being logged via OrderEvent.ToString(), but it wasn't
the full SecurityIdentifier - by including the full SID string it makes it
easier to correlate fills over symbol rename boundaries.
* Fix automatic option assignment from market simulation
During the recent OptionExerciseOrder.Quantity refactor, this case was missed.
Additionally, it was realized that there were no regression tests covering the
automatic assignment via the market conditions simulation. This change introduces
a regression algorithm that covers the automatic assignment of put/call options.
* Update BasicOptionAssignmentSimulation._rand to be non-static
If this value is static then we reuse the same Random instance for ALL regression
tests, thereby defeating the purpose of using a well known seed number. This means
we get different results based on the order execution of preceding algorithms.
By making this an instance variable each algorithm will start with the same seed
value, ensuring consistent runs between regression tests, either run as a suite or
running a single algorithm in isolation.
* Do not update price scale for fillforward data
- Do no update price scale for fill forward data. FillForward data
should keep using the prev scale for which it was created. Adding unit tests
- When cloning do not lose IsFillForward flag state, affects
QuoteBars/Ticks, does not affect TradeBars since they perform a memberwise clone.
Adding unit tests
* Auxiliaries shouldn't really affect on applied price factor scale.
Despite we can receeive FillForward'ed data points, corresponding
Auxiliaries for them are not FillForward so we do meet the condition
and then refresh price factor. As a result all futher FF data points are scaled too.
* Regression algorithm to check that FillForward'ed data points arrived with last real price factor
* Add trade for regression algorithm
- Minot tweaks and adding trade for new regression algorithm.
- Updating AddOptionContractExpiresRegressionAlgorithm because it is
using the symbol for which new data was added.
Co-authored-by: Adalyat Nazirov <aenazirov@gmail.com>
* OptionChain and OptionContract improvements
- QCAlgorithm.AddUniverse will return the added Universe instance.
- Adding new OptionChainedUniverseSelectionModel will monitor a Universe changes
and will spwan new OptionChainUniverse from it's selections. Adding
regression test Py/C#.
- Adding new OptionContractUniverse that will own option contracts and
their underlying symbol. Adding regression test
- Fix double notification for security changes, bug seen in updated
UniverseSelectionRegressionAlgorithm
- Remove UniverseSelection special handling for Option and Future chains
- Fix DataManager not removing SubscriptionDataConfigs for Subscriptions
which finished before being removed from the universe
- Refactor detection of user added Universe so that they do not get
removed after calling the UniverseSelectionModel
* Add check for option underlying price is set
* Address reviews
- Adding python regression algorithm for
`AddOptionContractFromUniverseRegressionAlgorithm`
and `AddOptionContractExpiresRegressionAlgorithm`
- Rename QCAlgorithm new api method to `AddChainedOptionUniverse`
* Fix universe refresh bug
- Fix bug where a universe selection refresh would cause option or
future chain universes from being removed. Adding regression algorithm
reproducing the issue.
* Rename new option universe Algorithm API method
- Rename new option universe Algorith API method from
AddChainedOptionUniverse to AddUniverseOptions
- Rebase and update regression test order hash because of
option expiration message changed
* Add property IBrokerage.AccountBaseCurrency
* Set AccountCurrency to brokerage AccountBaseCurrency
* Remove USD AccountCurrency check
* Fix Oanda account base currency
* Fix currency symbol in CashBook.ToString()
* Fix unit tests
* Address review
* Add DebugMessage when changing account currency
* Add debug message for brokerage account base currency
* Fix currency symbol in equity chart and runtime statistics
* Update unit tests
* Improve information tracked in regression's {algorithm}.{lang}.details.log
The details.log file aims at providing a diff-able document that quickly and
easily provides actionable information. Since many regression algorithms use
the algorithm's debug/error messaging facilities to log various pieces of algo
state. This document also support a configuration option: regression-high-fidelity-logging'
that logs EVERY piece of data, again, with the aim of providing an easily diff-able
documenbt to quickly highlight actionable information. I may have missed omse key
pieces of information here, but now that the entire QC knows about this regression
tool, if additional information is required then hopefully it's easy enough at this
point to extend the RegressionResultHandler to suit our needs.
The RegressionResultHandler was initially implemented to provide a concise log of
all orders. This was achieved by simply using the Order.ToString method. While
testing/investigating OptionExerciseOrder behavior, it became evident that more
information was required to properly identify the source of potential failures or
differences between previous regression test runs. This change adds logging for
almost every IResultHandler method and additionally attempts to capture the
actual portfolio impact of every OrderEvent. This is accomplished by logging
the portfolio's TotalPortfolioValue, Cash properties and the security's
SecurityHolding.Quantity property.
This change also standardizes the timestamps used to folloow the ISO-8601 format.
When using the RegressionResultHandler, it is highly recommeded to also disable
'forward-console-message' configuration option to ensure algorithm Debug/Error
message logging is done synchronously to ensure correct ordering with respect to
log messages via Log.Debug/Trace/Error.
* Fix typo in options OrderTests test case name
* Update SymbolRepresentation.GenerationOptionTickerOSI to extension method
Far more convenient as an extension method
* Improve R# default code formatting rules
Many of these rule changes focus on improving the readability of code,
with a particular emphasis on multi-line constructs, chained method calls
and multi-line method invocations/declarations.
* Add braces, use string interpolation and limit long lines
* Refactor OptionExerciseOrder.Quantity to indicate change in #contracts
For all other order types, the Order.Quantity indicates the change in the algorithm's
holdings upon order execution for the order's symbol. For OptionExerciseOrder, this
convention was broken. It appears as though only exercise was initially implemented,
in which case only long positions were supported and a code comment indicated that
only positive values of quantity were acceptable, indicating the number of contracts
to exercise. At a later date, assignment simulation was added and utilized a negative
order quantity. This caused some major inconsistencies in how models view exercise
orders compared to all other order types. This change brings OptionExerciseOrder.Quantity
into alignment with the other order types by making it represent the change in holdings
quantity upon order execution.
This change was originally going to be much larger, but in order to minimize risks and to
make for an easier review experience, the additional changes will be committed separately
and pushed in their own PR. Some of the issues identified include:
* Manual Exercise (especially for OTM) is not covered
* Margin Calculations (in particular taking into account opposing contracts held)
* IBrokerage.OptionPositionAssigned is raised for exercise (later filtered by tx handler)
Fixes OptionPortfolioModelTests to use exercise model to properly model exercise of
non-account quote currency option contract.
* Include Order.Tag/OrderEvent.Message in their ToString, Fix default tag values
There was inconsistencies in what we were checking for. The order constructors
default the tag parameter to an empty string but Order.CreateOrder checks for
a null string. Additionally, the order constructors (limit,stopmarket,stoplimit)
would check for an empty string and if so, apply a default order tag.
This change cleans these checks up using string.IsNullOrEmpty and also removes the
check from Order.CreateOrder since we're passing the tag into the various order
constructors.
* Improve information tracked in regression's {algorithm}.{lang}.details.log
The details.log file aims at providing a diff-able document that quickly and
easily provides actionable information. Since many regression algorithms use
the algorithm's debug/error messaging facilities to log various pieces of algo
state. This document also support a configuration option: regression-high-fidelity-logging'
that logs EVERY piece of data, again, with the aim of providing an easily diff-able
documenbt to quickly highlight actionable information. I may have missed omse key
pieces of information here, but now that the entire QC knows about this regression
tool, if additional information is required then hopefully it's easy enough at this
point to extend the RegressionResultHandler to suit our needs.
The RegressionResultHandler was initially implemented to provide a concise log of
all orders. This was achieved by simply using the Order.ToString method. While
testing/investigating OptionExerciseOrder behavior, it became evident that more
information was required to properly identify the source of potential failures or
differences between previous regression test runs. This change adds logging for
almost every IResultHandler method and additionally attempts to capture the
actual portfolio impact of every OrderEvent. This is accomplished by logging
the portfolio's TotalPortfolioValue, Cash properties and the security's
SecurityHolding.Quantity property.
This change also standardizes the timestamps used to folloow the ISO-8601 format.
When using the RegressionResultHandler, it is highly recommeded to also disable
'forward-console-message' configuration option to ensure algorithm Debug/Error
message logging is done synchronously to ensure correct ordering with respect to
log messages via Log.Debug/Trace/Error.
* Fix typo in options OrderTests test case name
* Update SymbolRepresentation.GenerationOptionTickerOSI to extension method
Far more convenient as an extension method
* Improve R# default code formatting rules
Many of these rule changes focus on improving the readability of code,
with a particular emphasis on multi-line constructs, chained method calls
and multi-line method invocations/declarations.
* Add braces, use string interpolation and limit long lines
* Refactor OptionExerciseOrder.Quantity to indicate change in #contracts
For all other order types, the Order.Quantity indicates the change in the algorithm's
holdings upon order execution for the order's symbol. For OptionExerciseOrder, this
convention was broken. It appears as though only exercise was initially implemented,
in which case only long positions were supported and a code comment indicated that
only positive values of quantity were acceptable, indicating the number of contracts
to exercise. At a later date, assignment simulation was added and utilized a negative
order quantity. This caused some major inconsistencies in how models view exercise
orders compared to all other order types. This change brings OptionExerciseOrder.Quantity
into alignment with the other order types by making it represent the change in holdings
quantity upon order execution.
This change was originally going to be much larger, but in order to minimize risks and to
make for an easier review experience, the additional changes will be committed separately
and pushed in their own PR. Some of the issues identified include:
* Manual Exercise (especially for OTM) is not covered
* Margin Calculations (in particular taking into account opposing contracts held)
* IBrokerage.OptionPositionAssigned is raised for exercise (later filtered by tx handler)
Fixes OptionPortfolioModelTests to use exercise model to properly model exercise of
non-account quote currency option contract.
* Adds new unit tests covering changes and testing for old case
* JsonConvert.SerializeObject would convert a `null` value into a
literal string of "null" when writing to a file via the ToLine
abstract method. We opt for an empty string whenever the underlying
value is null so that the parsing works correctly later in the
data loading cycle.
* avoid preliminary typing command lines arguments
* unit tests: Config.Get can parse and cast values
* unit tests: parse command line args and return string values
* unit test: parameter attribute converter
* merge&parse unit test
- LocalObjectStore.Delete() will also delete file from the local object
store path if present, this will avoid the issue where restarting the
object store will re load the same deleted file. Adding unit test.
Issue https://github.com/QuantConnect/Lean/issues/4811
* Fixes issue where BidPrice/AskPrice were not adjusted for Quote Ticks
* Previously, ticks would have their prices (Tick.Value) adjusted whenever
TickType == TickType.Quote, but would not have their
BidPrice/AskPrice fields adjusted, thus potentially being orders
of magnitude such as 4x from the actual Bid/Ask prices.
This commit applies the pricing scaling factor in a critical
path where Ticks are adjusted to their scaled price. This issue
only applied to Resolution.Tick && SecurityType.Equity data.
* Refactors Extensions Tick Scale extension method
* Adjusts unit test to dispose of resources and assert history count
* Replaces use of FileSystemDataFeed for NullDataFeed in Adjustment test
* Adds regression algorithm testing BidPrice & AskPrice adjustment
* Address review: remove SecurityType check on TickType.Trade adjustments
* Append the full stacktrace to the algorithm loading exception message.
* Remove exception message loader duplication
Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
- `HistoryRequestFactory` will not sure extended market hours for hour
resolution when determining the start time using quantity of bars.
Adding regression test
* Improves stability and fixes various bugs
* Adds unit tests covering changes
* Adds COVID-19 crisis plots
* Adjusts styling of crisis plots for more pleasant viewing
* Fixes bug where null BacktestResult caused application to crash
* Order JSON bug fixes and stability improvements
* MaxDrawdownReportElement now produces results for Live
* Replaced Estimated Capacity w/ Days Live
* Added Live marker to sharpe ratio
* Added support for MOO and MOC orders in PortfolioLooper
* Address review: adds new unit tests and cleans up code
* Bug fix: use LastFillTime instead of Order.Time for MOO and MOC
* Address review: Fixes tests and cleans up code
* Binance Brokerage skeleton
* Market hours
* Implement Symbol Mapper
- known symbols available on /api/v1/exchangeInfo
- fiat currencies are pegged
* Implement GetCashBalance
* Implement GetAccountHoldings
- there are no pre-existing currency swaps
- cash balances are pulled and stored in the cashbook
* Implement GetOpenOrders
* Manage orders: PlaceOrder
* Manage orders: UpdateOrder
Update operation is not supported
* Manage orders: CancelOrder
* Messaging: order book
* Messaging: trades
* Messaging: combine streams
- connect to fake /ws/open channel on init
- case by channel name, but not event type
* Messaging: order depth updates
- ticker symbol is not enough as it pushes updates only once a second, this would be a very incomplete data stream
- fetch ticker snapshot if lastUpdateId == 0
- follow Binance instructions for keeping local orderbook fresh
* Messaging: user data streaming
- Request userDataStream endpoint to get listenKey
- keep listenkey alive
- handle order close event
- handle order fill event
* DataDownloader: get history
- we can aggregate minute candles for higher resolutions
* fix data stream
* Tests: FeeModel tests
* Tests: base brokerage tests
* Tests: download history
* Tests: symbol mapper
* Support StopLimit andd StopMarket orders
* StopMarket orders disabled
Take profit and Stop loss orders are not supported for any symbols (tested with BTCUSDT, ETHUSDT)
* Tests: StopLimit order
* Tests: crypto parsing
* Reissue user data listen key
* comment custom currency limitation
* rework websocket connections
* implement delayed subscription
* adapt ignore message
* add license banner
* use better suited exception type
* avoid message double parsing
* support custom fee values
* extract BinanceApiClient to manage the request/response between lean and binance
* use api events to invoke brokerage events
* do not allow to terminate session if it wasn't allocated.
* update binance exchange info
* tool to add or update binance exchange info
* ExchangeInfo basic test
* Rebase + Resharp
* Binance brokerage updates
- Fix sign bug in sell order fills
- Fix bug in GetHistory
- Remove duplicate symbol from symbol properties db
* Remove unused code
* Revert removal of account currency check
* Update symbols properties database
* Address review
* Address review
- Upgrade API endpoints from v1 to v3
- Updated sub/unsub for new subscription manager
- Subscribe best bid/ask quotes instead of full order book
- Added handling of websocket error messages
- Cleanup + refactor
* Update symbol properties database
* Remove list from symbol mapper
* Fix symbol mapper tests
* Address review
- Fix resubscribe after reconnect
- Fix quote tick edge case
* Fix EnsureCurrencyDataFeed for non-tradeable currencies
* Fix check in EnsureCurrencyDataFeed
* Reuse base class subscribe on reconnect
Co-authored-by: Adalyat Nazirov <aenazirov@gmail.com>
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Add OrderRight.GetExerciseDirection(isShort) extension
Returns the OrderDirection resulting from exercise/assignment of a particular
option right
See: BUG #4731
* Fix option exercise/assignment order tags and order event messages
The algorithm manager was doing work to determine whether or not the option ended
in exercise or assignment at expiration. This decision should be left for the exercise
model to decide -- from the algorithm manager's perspective, all that matters is that
the option was expired. The DefaultExerciseModel was updated to properly track whether
the option expired with automatic assignment or exercise, dependending on whether or
not we wrote or bought the option (held liability or right, respectively). Updated unit
tests to check for order event counts and order event messages for option exercise cases.
Fixes: #4731
* Fix typo in algorithm documentation
* Update regression tests order hash
Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
In OptionChainUniverseDataCollectionEnumerator.IsValid we check the underlying
foir null and ensure that there are data points but we forget to check the actual
collection object for null, which it is initialized to. The collection remains
null in the case where no data is found.
* Convert a CoarseFundamental data point into a row
- Convert a CoarseFundamental data point into a row. Adding unit tests
* Remove fixed PriceFactor format
* Simplify BaseWebSocketsBrokerage reconnection
- Gdax does not emit time pulse so when no subscription was present the
DefaultConnectionHandler would trigger a reconnect. Replacing for
directly resubscribing on reconnect.
- TPV == 0 will send warning message instead of being an exception
* Fix unit test
* regression tests
* fix: apply the same time convertion to history request time as for data time
* ver2
* fixup
* unit tests
* do not need this conversion because RoundDownInTimeZone returns in proper TZ
* comment
* requested changes
* refactoring
* more refactoring
* fix existing test: should return Sunday if open
* more symbols
* fix existing tests: submit new btcusd data
* fix
* add Cfd symbol
* Fix regression *.{lang}.details.logs
Regression tests produce syslogs and a details.logs file. The details log file
provides a mechanism for logging data that passes through the result handler,
such as order event data. A change in the initialization ordeer of components in
the engine cause the algorithm id to not be set yet. The base result handler's
AlgorithmId property is populated via the job and is available, so we use that
instance.
* Update RegressionResultHandler.cs
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Add DataProviderEventArgs base class for IDataProviderEvents event args
This base class includes a Symbol property. This will empower event listeners
to make decisions based on which security (symbol) raised the event. The immediate
use case is preventing multiple numerical precision messages for the same security.
This pattern can equally be applied to other error messages that are raised each
time a security is added to a universe.
See: #BUG-4722
* Update ConcurrentSet.Add to use ISet<T>.Add returning bool
It's a very common pattern to use if (set.Add(item)) which is enabled
via bool ISet<T>.Add(item) but not enabled via void ICollectiont<T>.Add(item).
This change simples changes the default Add implementation to use the ISet<T>
overload and relegates the ICollection<T>.Add implementation to be explicit.
See: #BUG-4722
* Prevent multiple numerical precision messages for same symbol
If a security is continually added/removed from a universe, then the user will
see this message each time the security is added. This results in some spam.
This change simply remembers for which symbols we've notified the user about the
numerical precision issue.
Fixes: #BUG-4722
* Fix for BaseWebSocketsBrokerage reconnection
- `BaseWebsocketsBrokerage` will handle reconnection using the
existing `DefaultConnectionHandler` to avoid code duplication.
* Fix for Bitfinex failed Subscription calls
* Fix for bitfinex orderbook
* Fix unit test
* SafeDecimalCast Throws Exception For Non-Finite Numbers
* Fixes Arithmetic Overflow Exception in QCAlgorithm.Trading Methods
Replace decimal cast for `SafeDecimalCast()`.
If the algorithm uses a non-finite number in QCAlgorithm trading methods, it will throw with an user-frieldly exception message.
* Fixes KellyCriterionProbabilityValue Calculation
* Add unit tests
* Refactor Py and create C# function
* Update readme to include local
* Refactor solution; fix python cases
* Update tests
* Don't accept null selector for python; Create SelectedData class
* Fix Testing
* Pre review
* Fix tests for Travis
* Test fix V2
* Test fix V3
* Refactor quantbook and fix tests
* Sort list by date
* Move ConvertToSymbols to Python Util
* Address review
* Order dataframe columns by Security ID
* Address review V2
* header for PythonUtilTests
* Oanda default forex Market
- Use Oanda as default forex Market since it has more pairs.
- Remove FXCM data add Oanda equivalente data.
- Update unit and regression tests
* Address reviews
- Revert FXCM data removal
- Remove unrequired commented code
* Fix rebase
Michael Handschuh