* Add RequestId to request information logging
* Use unique request id across all request types (orders, subscriptions, data queries)
- Previously we had three separate counters for request types and this was causing request information messages to be overwritten (different request types with same ids)
* Update GetContractDetails to log all contracts found
* Store temp files in subdirectory
* Fix Dispose case for new temp dir
* Adjust tests for new temp dir
* Dispose unit tests
* Unit test for issue 4811
* Refactor for not using temp files
* Fix storage checks for saving data, plus tests
* Use Base64 for storing keys and decoding them; handles odd key strings
* Don't allow "?" in a key
* Address review
* Deleted test cases
* PersistData handle deletion of files
* Refactor GetFilePath to use Persist()
* Make PathForKey protected
* Add decimal places as parameters to get dividends with arbitrary precision
Also, increase precision when generating strings from factor files rows.
* Add xml documentation entries for new optional arguments.
* Adds CustomBuyingPowerModelAlgorithm
This algorithms is an example on how to implement a custom buying power model.
In this particular case, it shows how to override `HasSufficientBuyingPowerForOrder` in order to place orders without sufficient buying power according to the default model.
* Upgrades CustomModelsAlgorithm to Include CustomBuyingPowerModel
The custom buying power model overrides `HasSufficientBuyingPowerForOrderResult` but it doesn't change the trades and, consequently, the regression statistics.
* Allow account currency to be overridden by the algorithm
* Fix failing unit test
* Address review
- Revert removal of call check in SecurityPortfolioManager.SetAccountCurrency
- Revert changes to unit tests
- IBrokerage.AccountBaseCurrency now defaults to null
- BrokerageSetupHandler will not change the algorithm's account currency if the brokerage returns null, allowing the algorithm to call SetAccountCurrency in Initialize
* Fail on restart investing after liquidation
I added a line so that the trailing high value could be rebalanced and the investment process won't be stop by high value always more than current value by drawdown percent.
* Update MaximumDrawdownPercentPortfolio.py
* Fix for MaximumDrawdownPercentPortfolio
- Fix C# MaximumDrawdownPercentPortfolio to reset portfolio value after
liquidation. Only reset once we have actually adjusted some targets.
Updating regression algorithms.
Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
* Remove F# Project, Splits, and Dividends Tests
* Separate tests that require external accounts; read from config
* Removal of non supported "prices" endpoint test
* Removal of unsupported API functions
* Address review
* NOP GetLastPrice for removal of Prices endpoint
* Post rebase fix
* Rebase fix 2
* remove /r/n from eof for api tests
* Reflect similar refactors to NodeTests
* Fix for live algorithm API testing
* Address Review
* Add underlying holdings to regression result handler details log
When debugging option exercise/assignment issues it's useful to see the
underlying holdings at the time the option contract fill event is processed.
Also adds the full symbol string to the top of the order event section.
The Symbol.Value was being logged via OrderEvent.ToString(), but it wasn't
the full SecurityIdentifier - by including the full SID string it makes it
easier to correlate fills over symbol rename boundaries.
* Fix automatic option assignment from market simulation
During the recent OptionExerciseOrder.Quantity refactor, this case was missed.
Additionally, it was realized that there were no regression tests covering the
automatic assignment via the market conditions simulation. This change introduces
a regression algorithm that covers the automatic assignment of put/call options.
* Update BasicOptionAssignmentSimulation._rand to be non-static
If this value is static then we reuse the same Random instance for ALL regression
tests, thereby defeating the purpose of using a well known seed number. This means
we get different results based on the order execution of preceding algorithms.
By making this an instance variable each algorithm will start with the same seed
value, ensuring consistent runs between regression tests, either run as a suite or
running a single algorithm in isolation.
* Do not update price scale for fillforward data
- Do no update price scale for fill forward data. FillForward data
should keep using the prev scale for which it was created. Adding unit tests
- When cloning do not lose IsFillForward flag state, affects
QuoteBars/Ticks, does not affect TradeBars since they perform a memberwise clone.
Adding unit tests
* Auxiliaries shouldn't really affect on applied price factor scale.
Despite we can receeive FillForward'ed data points, corresponding
Auxiliaries for them are not FillForward so we do meet the condition
and then refresh price factor. As a result all futher FF data points are scaled too.
* Regression algorithm to check that FillForward'ed data points arrived with last real price factor
* Add trade for regression algorithm
- Minot tweaks and adding trade for new regression algorithm.
- Updating AddOptionContractExpiresRegressionAlgorithm because it is
using the symbol for which new data was added.
Co-authored-by: Adalyat Nazirov <aenazirov@gmail.com>
* OptionChain and OptionContract improvements
- QCAlgorithm.AddUniverse will return the added Universe instance.
- Adding new OptionChainedUniverseSelectionModel will monitor a Universe changes
and will spwan new OptionChainUniverse from it's selections. Adding
regression test Py/C#.
- Adding new OptionContractUniverse that will own option contracts and
their underlying symbol. Adding regression test
- Fix double notification for security changes, bug seen in updated
UniverseSelectionRegressionAlgorithm
- Remove UniverseSelection special handling for Option and Future chains
- Fix DataManager not removing SubscriptionDataConfigs for Subscriptions
which finished before being removed from the universe
- Refactor detection of user added Universe so that they do not get
removed after calling the UniverseSelectionModel
* Add check for option underlying price is set
* Address reviews
- Adding python regression algorithm for
`AddOptionContractFromUniverseRegressionAlgorithm`
and `AddOptionContractExpiresRegressionAlgorithm`
- Rename QCAlgorithm new api method to `AddChainedOptionUniverse`
* Fix universe refresh bug
- Fix bug where a universe selection refresh would cause option or
future chain universes from being removed. Adding regression algorithm
reproducing the issue.
* Rename new option universe Algorithm API method
- Rename new option universe Algorith API method from
AddChainedOptionUniverse to AddUniverseOptions
- Rebase and update regression test order hash because of
option expiration message changed
* Add property IBrokerage.AccountBaseCurrency
* Set AccountCurrency to brokerage AccountBaseCurrency
* Remove USD AccountCurrency check
* Fix Oanda account base currency
* Fix currency symbol in CashBook.ToString()
* Fix unit tests
* Address review
* Add DebugMessage when changing account currency
* Add debug message for brokerage account base currency
* Fix currency symbol in equity chart and runtime statistics
* Update unit tests
* Improve information tracked in regression's {algorithm}.{lang}.details.log
The details.log file aims at providing a diff-able document that quickly and
easily provides actionable information. Since many regression algorithms use
the algorithm's debug/error messaging facilities to log various pieces of algo
state. This document also support a configuration option: regression-high-fidelity-logging'
that logs EVERY piece of data, again, with the aim of providing an easily diff-able
documenbt to quickly highlight actionable information. I may have missed omse key
pieces of information here, but now that the entire QC knows about this regression
tool, if additional information is required then hopefully it's easy enough at this
point to extend the RegressionResultHandler to suit our needs.
The RegressionResultHandler was initially implemented to provide a concise log of
all orders. This was achieved by simply using the Order.ToString method. While
testing/investigating OptionExerciseOrder behavior, it became evident that more
information was required to properly identify the source of potential failures or
differences between previous regression test runs. This change adds logging for
almost every IResultHandler method and additionally attempts to capture the
actual portfolio impact of every OrderEvent. This is accomplished by logging
the portfolio's TotalPortfolioValue, Cash properties and the security's
SecurityHolding.Quantity property.
This change also standardizes the timestamps used to folloow the ISO-8601 format.
When using the RegressionResultHandler, it is highly recommeded to also disable
'forward-console-message' configuration option to ensure algorithm Debug/Error
message logging is done synchronously to ensure correct ordering with respect to
log messages via Log.Debug/Trace/Error.
* Fix typo in options OrderTests test case name
* Update SymbolRepresentation.GenerationOptionTickerOSI to extension method
Far more convenient as an extension method
* Improve R# default code formatting rules
Many of these rule changes focus on improving the readability of code,
with a particular emphasis on multi-line constructs, chained method calls
and multi-line method invocations/declarations.
* Add braces, use string interpolation and limit long lines
* Refactor OptionExerciseOrder.Quantity to indicate change in #contracts
For all other order types, the Order.Quantity indicates the change in the algorithm's
holdings upon order execution for the order's symbol. For OptionExerciseOrder, this
convention was broken. It appears as though only exercise was initially implemented,
in which case only long positions were supported and a code comment indicated that
only positive values of quantity were acceptable, indicating the number of contracts
to exercise. At a later date, assignment simulation was added and utilized a negative
order quantity. This caused some major inconsistencies in how models view exercise
orders compared to all other order types. This change brings OptionExerciseOrder.Quantity
into alignment with the other order types by making it represent the change in holdings
quantity upon order execution.
This change was originally going to be much larger, but in order to minimize risks and to
make for an easier review experience, the additional changes will be committed separately
and pushed in their own PR. Some of the issues identified include:
* Manual Exercise (especially for OTM) is not covered
* Margin Calculations (in particular taking into account opposing contracts held)
* IBrokerage.OptionPositionAssigned is raised for exercise (later filtered by tx handler)
Fixes OptionPortfolioModelTests to use exercise model to properly model exercise of
non-account quote currency option contract.
* Include Order.Tag/OrderEvent.Message in their ToString, Fix default tag values
There was inconsistencies in what we were checking for. The order constructors
default the tag parameter to an empty string but Order.CreateOrder checks for
a null string. Additionally, the order constructors (limit,stopmarket,stoplimit)
would check for an empty string and if so, apply a default order tag.
This change cleans these checks up using string.IsNullOrEmpty and also removes the
check from Order.CreateOrder since we're passing the tag into the various order
constructors.
* Improve information tracked in regression's {algorithm}.{lang}.details.log
The details.log file aims at providing a diff-able document that quickly and
easily provides actionable information. Since many regression algorithms use
the algorithm's debug/error messaging facilities to log various pieces of algo
state. This document also support a configuration option: regression-high-fidelity-logging'
that logs EVERY piece of data, again, with the aim of providing an easily diff-able
documenbt to quickly highlight actionable information. I may have missed omse key
pieces of information here, but now that the entire QC knows about this regression
tool, if additional information is required then hopefully it's easy enough at this
point to extend the RegressionResultHandler to suit our needs.
The RegressionResultHandler was initially implemented to provide a concise log of
all orders. This was achieved by simply using the Order.ToString method. While
testing/investigating OptionExerciseOrder behavior, it became evident that more
information was required to properly identify the source of potential failures or
differences between previous regression test runs. This change adds logging for
almost every IResultHandler method and additionally attempts to capture the
actual portfolio impact of every OrderEvent. This is accomplished by logging
the portfolio's TotalPortfolioValue, Cash properties and the security's
SecurityHolding.Quantity property.
This change also standardizes the timestamps used to folloow the ISO-8601 format.
When using the RegressionResultHandler, it is highly recommeded to also disable
'forward-console-message' configuration option to ensure algorithm Debug/Error
message logging is done synchronously to ensure correct ordering with respect to
log messages via Log.Debug/Trace/Error.
* Fix typo in options OrderTests test case name
* Update SymbolRepresentation.GenerationOptionTickerOSI to extension method
Far more convenient as an extension method
* Improve R# default code formatting rules
Many of these rule changes focus on improving the readability of code,
with a particular emphasis on multi-line constructs, chained method calls
and multi-line method invocations/declarations.
* Add braces, use string interpolation and limit long lines
* Refactor OptionExerciseOrder.Quantity to indicate change in #contracts
For all other order types, the Order.Quantity indicates the change in the algorithm's
holdings upon order execution for the order's symbol. For OptionExerciseOrder, this
convention was broken. It appears as though only exercise was initially implemented,
in which case only long positions were supported and a code comment indicated that
only positive values of quantity were acceptable, indicating the number of contracts
to exercise. At a later date, assignment simulation was added and utilized a negative
order quantity. This caused some major inconsistencies in how models view exercise
orders compared to all other order types. This change brings OptionExerciseOrder.Quantity
into alignment with the other order types by making it represent the change in holdings
quantity upon order execution.
This change was originally going to be much larger, but in order to minimize risks and to
make for an easier review experience, the additional changes will be committed separately
and pushed in their own PR. Some of the issues identified include:
* Manual Exercise (especially for OTM) is not covered
* Margin Calculations (in particular taking into account opposing contracts held)
* IBrokerage.OptionPositionAssigned is raised for exercise (later filtered by tx handler)
Fixes OptionPortfolioModelTests to use exercise model to properly model exercise of
non-account quote currency option contract.
* Adds new unit tests covering changes and testing for old case
* JsonConvert.SerializeObject would convert a `null` value into a
literal string of "null" when writing to a file via the ToLine
abstract method. We opt for an empty string whenever the underlying
value is null so that the parsing works correctly later in the
data loading cycle.
* avoid preliminary typing command lines arguments
* unit tests: Config.Get can parse and cast values
* unit tests: parse command line args and return string values
* unit test: parameter attribute converter
* merge&parse unit test
- LocalObjectStore.Delete() will also delete file from the local object
store path if present, this will avoid the issue where restarting the
object store will re load the same deleted file. Adding unit test.
Issue https://github.com/QuantConnect/Lean/issues/4811
* Fixes issue where BidPrice/AskPrice were not adjusted for Quote Ticks
* Previously, ticks would have their prices (Tick.Value) adjusted whenever
TickType == TickType.Quote, but would not have their
BidPrice/AskPrice fields adjusted, thus potentially being orders
of magnitude such as 4x from the actual Bid/Ask prices.
This commit applies the pricing scaling factor in a critical
path where Ticks are adjusted to their scaled price. This issue
only applied to Resolution.Tick && SecurityType.Equity data.
* Refactors Extensions Tick Scale extension method
* Adjusts unit test to dispose of resources and assert history count
* Replaces use of FileSystemDataFeed for NullDataFeed in Adjustment test
* Adds regression algorithm testing BidPrice & AskPrice adjustment
* Address review: remove SecurityType check on TickType.Trade adjustments
* Append the full stacktrace to the algorithm loading exception message.
* Remove exception message loader duplication
Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
- `HistoryRequestFactory` will not sure extended market hours for hour
resolution when determining the start time using quantity of bars.
Adding regression test
* Improves stability and fixes various bugs
* Adds unit tests covering changes
* Adds COVID-19 crisis plots
* Adjusts styling of crisis plots for more pleasant viewing
* Fixes bug where null BacktestResult caused application to crash
* Order JSON bug fixes and stability improvements
* MaxDrawdownReportElement now produces results for Live
* Replaced Estimated Capacity w/ Days Live
* Added Live marker to sharpe ratio
* Added support for MOO and MOC orders in PortfolioLooper
* Address review: adds new unit tests and cleans up code
* Bug fix: use LastFillTime instead of Order.Time for MOO and MOC
* Address review: Fixes tests and cleans up code
* Binance Brokerage skeleton
* Market hours
* Implement Symbol Mapper
- known symbols available on /api/v1/exchangeInfo
- fiat currencies are pegged
* Implement GetCashBalance
* Implement GetAccountHoldings
- there are no pre-existing currency swaps
- cash balances are pulled and stored in the cashbook
* Implement GetOpenOrders
* Manage orders: PlaceOrder
* Manage orders: UpdateOrder
Update operation is not supported
* Manage orders: CancelOrder
* Messaging: order book
* Messaging: trades
* Messaging: combine streams
- connect to fake /ws/open channel on init
- case by channel name, but not event type
* Messaging: order depth updates
- ticker symbol is not enough as it pushes updates only once a second, this would be a very incomplete data stream
- fetch ticker snapshot if lastUpdateId == 0
- follow Binance instructions for keeping local orderbook fresh
* Messaging: user data streaming
- Request userDataStream endpoint to get listenKey
- keep listenkey alive
- handle order close event
- handle order fill event
* DataDownloader: get history
- we can aggregate minute candles for higher resolutions
* fix data stream
* Tests: FeeModel tests
* Tests: base brokerage tests
* Tests: download history
* Tests: symbol mapper
* Support StopLimit andd StopMarket orders
* StopMarket orders disabled
Take profit and Stop loss orders are not supported for any symbols (tested with BTCUSDT, ETHUSDT)
* Tests: StopLimit order
* Tests: crypto parsing
* Reissue user data listen key
* comment custom currency limitation
* rework websocket connections
* implement delayed subscription
* adapt ignore message
* add license banner
* use better suited exception type
* avoid message double parsing
* support custom fee values
* extract BinanceApiClient to manage the request/response between lean and binance
* use api events to invoke brokerage events
* do not allow to terminate session if it wasn't allocated.
* update binance exchange info
* tool to add or update binance exchange info
* ExchangeInfo basic test
* Rebase + Resharp
* Binance brokerage updates
- Fix sign bug in sell order fills
- Fix bug in GetHistory
- Remove duplicate symbol from symbol properties db
* Remove unused code
* Revert removal of account currency check
* Update symbols properties database
* Address review
* Address review
- Upgrade API endpoints from v1 to v3
- Updated sub/unsub for new subscription manager
- Subscribe best bid/ask quotes instead of full order book
- Added handling of websocket error messages
- Cleanup + refactor
* Update symbol properties database
* Remove list from symbol mapper
* Fix symbol mapper tests
* Address review
- Fix resubscribe after reconnect
- Fix quote tick edge case
* Fix EnsureCurrencyDataFeed for non-tradeable currencies
* Fix check in EnsureCurrencyDataFeed
* Reuse base class subscribe on reconnect
Co-authored-by: Adalyat Nazirov <aenazirov@gmail.com>
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
Juan José D'Ambrosio