* Add property IBrokerage.AccountBaseCurrency
* Set AccountCurrency to brokerage AccountBaseCurrency
* Remove USD AccountCurrency check
* Fix Oanda account base currency
* Fix currency symbol in CashBook.ToString()
* Fix unit tests
* Address review
* Add DebugMessage when changing account currency
* Add debug message for brokerage account base currency
* Fix currency symbol in equity chart and runtime statistics
* Update unit tests
* Improve information tracked in regression's {algorithm}.{lang}.details.log
The details.log file aims at providing a diff-able document that quickly and
easily provides actionable information. Since many regression algorithms use
the algorithm's debug/error messaging facilities to log various pieces of algo
state. This document also support a configuration option: regression-high-fidelity-logging'
that logs EVERY piece of data, again, with the aim of providing an easily diff-able
documenbt to quickly highlight actionable information. I may have missed omse key
pieces of information here, but now that the entire QC knows about this regression
tool, if additional information is required then hopefully it's easy enough at this
point to extend the RegressionResultHandler to suit our needs.
The RegressionResultHandler was initially implemented to provide a concise log of
all orders. This was achieved by simply using the Order.ToString method. While
testing/investigating OptionExerciseOrder behavior, it became evident that more
information was required to properly identify the source of potential failures or
differences between previous regression test runs. This change adds logging for
almost every IResultHandler method and additionally attempts to capture the
actual portfolio impact of every OrderEvent. This is accomplished by logging
the portfolio's TotalPortfolioValue, Cash properties and the security's
SecurityHolding.Quantity property.
This change also standardizes the timestamps used to folloow the ISO-8601 format.
When using the RegressionResultHandler, it is highly recommeded to also disable
'forward-console-message' configuration option to ensure algorithm Debug/Error
message logging is done synchronously to ensure correct ordering with respect to
log messages via Log.Debug/Trace/Error.
* Fix typo in options OrderTests test case name
* Update SymbolRepresentation.GenerationOptionTickerOSI to extension method
Far more convenient as an extension method
* Improve R# default code formatting rules
Many of these rule changes focus on improving the readability of code,
with a particular emphasis on multi-line constructs, chained method calls
and multi-line method invocations/declarations.
* Add braces, use string interpolation and limit long lines
* Refactor OptionExerciseOrder.Quantity to indicate change in #contracts
For all other order types, the Order.Quantity indicates the change in the algorithm's
holdings upon order execution for the order's symbol. For OptionExerciseOrder, this
convention was broken. It appears as though only exercise was initially implemented,
in which case only long positions were supported and a code comment indicated that
only positive values of quantity were acceptable, indicating the number of contracts
to exercise. At a later date, assignment simulation was added and utilized a negative
order quantity. This caused some major inconsistencies in how models view exercise
orders compared to all other order types. This change brings OptionExerciseOrder.Quantity
into alignment with the other order types by making it represent the change in holdings
quantity upon order execution.
This change was originally going to be much larger, but in order to minimize risks and to
make for an easier review experience, the additional changes will be committed separately
and pushed in their own PR. Some of the issues identified include:
* Manual Exercise (especially for OTM) is not covered
* Margin Calculations (in particular taking into account opposing contracts held)
* IBrokerage.OptionPositionAssigned is raised for exercise (later filtered by tx handler)
Fixes OptionPortfolioModelTests to use exercise model to properly model exercise of
non-account quote currency option contract.
* Include Order.Tag/OrderEvent.Message in their ToString, Fix default tag values
There was inconsistencies in what we were checking for. The order constructors
default the tag parameter to an empty string but Order.CreateOrder checks for
a null string. Additionally, the order constructors (limit,stopmarket,stoplimit)
would check for an empty string and if so, apply a default order tag.
This change cleans these checks up using string.IsNullOrEmpty and also removes the
check from Order.CreateOrder since we're passing the tag into the various order
constructors.
* Improve information tracked in regression's {algorithm}.{lang}.details.log
The details.log file aims at providing a diff-able document that quickly and
easily provides actionable information. Since many regression algorithms use
the algorithm's debug/error messaging facilities to log various pieces of algo
state. This document also support a configuration option: regression-high-fidelity-logging'
that logs EVERY piece of data, again, with the aim of providing an easily diff-able
documenbt to quickly highlight actionable information. I may have missed omse key
pieces of information here, but now that the entire QC knows about this regression
tool, if additional information is required then hopefully it's easy enough at this
point to extend the RegressionResultHandler to suit our needs.
The RegressionResultHandler was initially implemented to provide a concise log of
all orders. This was achieved by simply using the Order.ToString method. While
testing/investigating OptionExerciseOrder behavior, it became evident that more
information was required to properly identify the source of potential failures or
differences between previous regression test runs. This change adds logging for
almost every IResultHandler method and additionally attempts to capture the
actual portfolio impact of every OrderEvent. This is accomplished by logging
the portfolio's TotalPortfolioValue, Cash properties and the security's
SecurityHolding.Quantity property.
This change also standardizes the timestamps used to folloow the ISO-8601 format.
When using the RegressionResultHandler, it is highly recommeded to also disable
'forward-console-message' configuration option to ensure algorithm Debug/Error
message logging is done synchronously to ensure correct ordering with respect to
log messages via Log.Debug/Trace/Error.
* Fix typo in options OrderTests test case name
* Update SymbolRepresentation.GenerationOptionTickerOSI to extension method
Far more convenient as an extension method
* Improve R# default code formatting rules
Many of these rule changes focus on improving the readability of code,
with a particular emphasis on multi-line constructs, chained method calls
and multi-line method invocations/declarations.
* Add braces, use string interpolation and limit long lines
* Refactor OptionExerciseOrder.Quantity to indicate change in #contracts
For all other order types, the Order.Quantity indicates the change in the algorithm's
holdings upon order execution for the order's symbol. For OptionExerciseOrder, this
convention was broken. It appears as though only exercise was initially implemented,
in which case only long positions were supported and a code comment indicated that
only positive values of quantity were acceptable, indicating the number of contracts
to exercise. At a later date, assignment simulation was added and utilized a negative
order quantity. This caused some major inconsistencies in how models view exercise
orders compared to all other order types. This change brings OptionExerciseOrder.Quantity
into alignment with the other order types by making it represent the change in holdings
quantity upon order execution.
This change was originally going to be much larger, but in order to minimize risks and to
make for an easier review experience, the additional changes will be committed separately
and pushed in their own PR. Some of the issues identified include:
* Manual Exercise (especially for OTM) is not covered
* Margin Calculations (in particular taking into account opposing contracts held)
* IBrokerage.OptionPositionAssigned is raised for exercise (later filtered by tx handler)
Fixes OptionPortfolioModelTests to use exercise model to properly model exercise of
non-account quote currency option contract.
* Adds new unit tests covering changes and testing for old case
* JsonConvert.SerializeObject would convert a `null` value into a
literal string of "null" when writing to a file via the ToLine
abstract method. We opt for an empty string whenever the underlying
value is null so that the parsing works correctly later in the
data loading cycle.
* Fixes issue where BidPrice/AskPrice were not adjusted for Quote Ticks
* Previously, ticks would have their prices (Tick.Value) adjusted whenever
TickType == TickType.Quote, but would not have their
BidPrice/AskPrice fields adjusted, thus potentially being orders
of magnitude such as 4x from the actual Bid/Ask prices.
This commit applies the pricing scaling factor in a critical
path where Ticks are adjusted to their scaled price. This issue
only applied to Resolution.Tick && SecurityType.Equity data.
* Refactors Extensions Tick Scale extension method
* Adjusts unit test to dispose of resources and assert history count
* Replaces use of FileSystemDataFeed for NullDataFeed in Adjustment test
* Adds regression algorithm testing BidPrice & AskPrice adjustment
* Address review: remove SecurityType check on TickType.Trade adjustments
- `HistoryRequestFactory` will not sure extended market hours for hour
resolution when determining the start time using quantity of bars.
Adding regression test
* Improves stability and fixes various bugs
* Adds unit tests covering changes
* Adds COVID-19 crisis plots
* Adjusts styling of crisis plots for more pleasant viewing
* Fixes bug where null BacktestResult caused application to crash
* Order JSON bug fixes and stability improvements
* MaxDrawdownReportElement now produces results for Live
* Replaced Estimated Capacity w/ Days Live
* Added Live marker to sharpe ratio
* Added support for MOO and MOC orders in PortfolioLooper
* Address review: adds new unit tests and cleans up code
* Bug fix: use LastFillTime instead of Order.Time for MOO and MOC
* Address review: Fixes tests and cleans up code
* Binance Brokerage skeleton
* Market hours
* Implement Symbol Mapper
- known symbols available on /api/v1/exchangeInfo
- fiat currencies are pegged
* Implement GetCashBalance
* Implement GetAccountHoldings
- there are no pre-existing currency swaps
- cash balances are pulled and stored in the cashbook
* Implement GetOpenOrders
* Manage orders: PlaceOrder
* Manage orders: UpdateOrder
Update operation is not supported
* Manage orders: CancelOrder
* Messaging: order book
* Messaging: trades
* Messaging: combine streams
- connect to fake /ws/open channel on init
- case by channel name, but not event type
* Messaging: order depth updates
- ticker symbol is not enough as it pushes updates only once a second, this would be a very incomplete data stream
- fetch ticker snapshot if lastUpdateId == 0
- follow Binance instructions for keeping local orderbook fresh
* Messaging: user data streaming
- Request userDataStream endpoint to get listenKey
- keep listenkey alive
- handle order close event
- handle order fill event
* DataDownloader: get history
- we can aggregate minute candles for higher resolutions
* fix data stream
* Tests: FeeModel tests
* Tests: base brokerage tests
* Tests: download history
* Tests: symbol mapper
* Support StopLimit andd StopMarket orders
* StopMarket orders disabled
Take profit and Stop loss orders are not supported for any symbols (tested with BTCUSDT, ETHUSDT)
* Tests: StopLimit order
* Tests: crypto parsing
* Reissue user data listen key
* comment custom currency limitation
* rework websocket connections
* implement delayed subscription
* adapt ignore message
* add license banner
* use better suited exception type
* avoid message double parsing
* support custom fee values
* extract BinanceApiClient to manage the request/response between lean and binance
* use api events to invoke brokerage events
* do not allow to terminate session if it wasn't allocated.
* update binance exchange info
* tool to add or update binance exchange info
* ExchangeInfo basic test
* Rebase + Resharp
* Binance brokerage updates
- Fix sign bug in sell order fills
- Fix bug in GetHistory
- Remove duplicate symbol from symbol properties db
* Remove unused code
* Revert removal of account currency check
* Update symbols properties database
* Address review
* Address review
- Upgrade API endpoints from v1 to v3
- Updated sub/unsub for new subscription manager
- Subscribe best bid/ask quotes instead of full order book
- Added handling of websocket error messages
- Cleanup + refactor
* Update symbol properties database
* Remove list from symbol mapper
* Fix symbol mapper tests
* Address review
- Fix resubscribe after reconnect
- Fix quote tick edge case
* Fix EnsureCurrencyDataFeed for non-tradeable currencies
* Fix check in EnsureCurrencyDataFeed
* Reuse base class subscribe on reconnect
Co-authored-by: Adalyat Nazirov <aenazirov@gmail.com>
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Add OrderRight.GetExerciseDirection(isShort) extension
Returns the OrderDirection resulting from exercise/assignment of a particular
option right
See: BUG #4731
* Fix option exercise/assignment order tags and order event messages
The algorithm manager was doing work to determine whether or not the option ended
in exercise or assignment at expiration. This decision should be left for the exercise
model to decide -- from the algorithm manager's perspective, all that matters is that
the option was expired. The DefaultExerciseModel was updated to properly track whether
the option expired with automatic assignment or exercise, dependending on whether or
not we wrote or bought the option (held liability or right, respectively). Updated unit
tests to check for order event counts and order event messages for option exercise cases.
Fixes: #4731
* Fix typo in algorithm documentation
* Update regression tests order hash
Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
* Convert a CoarseFundamental data point into a row
- Convert a CoarseFundamental data point into a row. Adding unit tests
* Remove fixed PriceFactor format
* regression tests
* fix: apply the same time convertion to history request time as for data time
* ver2
* fixup
* unit tests
* do not need this conversion because RoundDownInTimeZone returns in proper TZ
* comment
* requested changes
* refactoring
* more refactoring
* fix existing test: should return Sunday if open
* more symbols
* fix existing tests: submit new btcusd data
* fix
* add Cfd symbol
* Add DataProviderEventArgs base class for IDataProviderEvents event args
This base class includes a Symbol property. This will empower event listeners
to make decisions based on which security (symbol) raised the event. The immediate
use case is preventing multiple numerical precision messages for the same security.
This pattern can equally be applied to other error messages that are raised each
time a security is added to a universe.
See: #BUG-4722
* Update ConcurrentSet.Add to use ISet<T>.Add returning bool
It's a very common pattern to use if (set.Add(item)) which is enabled
via bool ISet<T>.Add(item) but not enabled via void ICollectiont<T>.Add(item).
This change simples changes the default Add implementation to use the ISet<T>
overload and relegates the ICollection<T>.Add implementation to be explicit.
See: #BUG-4722
* Prevent multiple numerical precision messages for same symbol
If a security is continually added/removed from a universe, then the user will
see this message each time the security is added. This results in some spam.
This change simply remembers for which symbols we've notified the user about the
numerical precision issue.
Fixes: #BUG-4722
* SafeDecimalCast Throws Exception For Non-Finite Numbers
* Fixes Arithmetic Overflow Exception in QCAlgorithm.Trading Methods
Replace decimal cast for `SafeDecimalCast()`.
If the algorithm uses a non-finite number in QCAlgorithm trading methods, it will throw with an user-frieldly exception message.
* Fixes KellyCriterionProbabilityValue Calculation
* Add unit tests
* Refactor Py and create C# function
* Update readme to include local
* Refactor solution; fix python cases
* Update tests
* Don't accept null selector for python; Create SelectedData class
* Fix Testing
* Pre review
* Fix tests for Travis
* Test fix V2
* Test fix V3
* Refactor quantbook and fix tests
* Sort list by date
* Move ConvertToSymbols to Python Util
* Address review
* Order dataframe columns by Security ID
* Address review V2
* header for PythonUtilTests
* Oanda default forex Market
- Use Oanda as default forex Market since it has more pairs.
- Remove FXCM data add Oanda equivalente data.
- Update unit and regression tests
* Address reviews
- Revert FXCM data removal
- Remove unrequired commented code
* Fix rebase
* Remove invalid symbols from symbol properties db
* Add SymbolPropertiesDatabase.GetSymbolPropertiesList
* Remove symbol list in BitfinexSymbolMapper
* In EnsureCurrencyDataFeed fetch symbols from symbol properties database
* Remove symbol list in OandaSymbolMapper
* Remove unused code
* Address review
- Remove StringComparer.OrdinalIgnoreCase usage
- Rename KnownSymbolStrings to KnownTickers
- Fix Slice.Get OpenInterest type. Adding unit test
- Fix for SecurityCache that wasn't storing OpenInterest types
- Updateing regression tests to covere these usages
* Calculate both raw and adjuasted prices for backtesting
* disable second price factoring
* move and reuse method
* test coverage for new methods
* reuse scaling method
* reuse subscriptionData.Create method
* removed unused code
* regression test
* switch to aapl
* fix regression test output
* more asserts
* fix comments - reduce shortcuts and abbrevation
* more comments
* merge parameters
* reduce number of getting price factors
* fix tests
* fix tests
* fix regression tests
* calculate TotalReturn on demand
* include TotalReturn calculations
* perf tuning
* more unit tests for SubscriptionData.Create
* simplify things - store and return only raw and precalculated data
* fix regression tests; change it back
* factor equals 1 for Raw data
* small changes
* follow code style
* implement backward compatibility
* Support List and OptionFilterUniverse for Py filter
* Regression algorithm for testing
* Unit Tests
* Fix for process
* Tighten filters to reduce load on automated testing
* Address review v2
* Add internal subscription manager
- Add InternalSubscriptionManager that will handle internal
Subscription. Replaces the realtime updates
- Fix thread race condition in the TimeTriggeredUniverseSubscription, we
have one thread injecting data points, the main algorithm thread, and
the base exchange is pulling from it
- Fixes for FakeDataQueue
- Adding unit tests
* Address reviews and fixes
- Internal subscription will use extended market hours
- Only sample charts accordingly
- Get api-url once
* Add PolygonDataQueueHandler
* Add history provider and downloader for Polygon
* Add aggregator to PolygonDataQueueHandler
* Address review
- Removed duplication in message classes
- Added public Subscribe/Unsubscribe methods in PolygonWebSocketClientWrapper
- Added history requests for Forex and Crypto
* Address review
- Add security type and market arguments to downloader
- Fix time zone bug in downloader
* Remove unnecessary locks
* Add Polygon history for all resolutions
- Equity: trades and quotes
- Forex: quotes only
- Crypto: trades only
* DataConsolidator Wrapper for Python Consolidators
* Regression Unit Test
* Refactor Regression test
* Bad test fix
* pre review
* self review
* Add RegisterIndicator for Python Consolidator
* Python base class for consolidators
* Modify regression algo to register indicator
* unit test - attach event
* Test fix
* Fix test python imports
* Add license header file and null check
Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
* Refactors SecurityCache to Support Access to Tick of Different Type
Adds two read-only lists of BaseData to save the last list of Tick of TickType.Trade and TickType.Quote. Changes methods accordingly to get and set these lists.
* Addresses Peer Review
The assignment of `_lastData` is moved to keep the current behavior of not beeing set with fill forward data.
`Reset` and `ShareTypeCacheInstance` implements the new class objects.
Adds unit tests for `Reset` and `ShareTypeCacheInstance`.
* Only Cache Non Fill-Forward Data
Undo changes that made AddData cache non fill-forward data.
* Live Coarse universe refactor
- Live trading will source Coarse and Fine fundamental data directly
from disk. Updating unit tests.
* Adds ILiveDataProvider interface
* Adds wrapper for IDataQueueHandler implementations
* Replaces IDataQueueHandler with ILiveDataProvider in
LiveTradingDataFeed
* Edits IDataQueueHandler documentation
* Maintains aggregation for current IDQH impls and skips for ILDF impls
* Note: No unit test was created for this method, go back and TODO
* Protobuf Market data
- Adding protobuf support for Ticks, TradeBars and QuoteBars. Adding
unit tests.
* Adds unit tests for LiveDataAggregator changes
* Fixes bug where custom data was not handled as it was before
* Fixes race condition bug because of variable reuse in class
* Add protobuf extension serialization
* Fixes for protobuf serialization
* Refactor
* Fix OptionChainUniverse
* replace BaseDataExchange pumping ticks with consolidators
* AlpacaBrokerage
* BitfinexBrokerage
* GDAXBrokerage
* OandaBrokerage
* InteractiveBrokers
* TradierBrokerage
* FxcmBrokerage
* PaperBrokerage
* etc
* WIP fixes for existing LTDF unit tests
* Fixes more LTDF unit tests
* make IDataAggregator.Update recieving Generic BaseData rather than Tick
* Change IDataQueueHandler.Subscribe method
* Some fixes after adding new commits
* Adds protobuf (de)serialization support for Dividend and Split
* Serialize protobuf with length prefix
* Fix missing LTDF unit tests
* Adds TiingoNews protobuf definitions
* fix comments
* more fixes on IQFeedDataQueueHandler
* disallow putting ticks into enumerator directly
* ScannableEnumerator tests
* fix OandaBrokerage
* AggregationManager unit tests
* fix AlpacaBrokerage tests
* fix InteractiveBrokers
* fix FxcmBrokerage tests
* call AggregationManager.Remove method on unsubscribe
* fix GDAX existing tests
* Fixes, refactor adding more tests for AggregatorManager
* Adds BenzingaNews protobuf definitions and round trip unit test
* Adds missing TiingoNews unit test to Protobuf round trip tests
* Improve sleep sequence of LiveSynchronizer
* need start aggregating first, and then can subscribe
* More test fixes and refactor
- Refactoring AggregationManager and ScannableEnumerator so the last is
the one that owns the consolidator
- Adding pulse on the main LiveSynchronizer
* Improve performance of LEquityDataSynchronizingEnu
* Add missing Set job packet method
* Minor performance improvements
* Improvements add test timeout
- Improvements adding test timeout to find blocking test in travis
* Improve aggregationManager performance
* Testing improvements for travis
* Remove test timeouts
* More test fixes
- Adding more missing dispose calls and improving determinism
* fix IEXDataQueueHandler and tests
* Final tweaks to LTDF tests
* more AggregationManager tests
* consume and log ticks
* fix test: couldn't subscribe to Forex tickers
* change Resolution for all bar configs
* Improve RealTimeScheduleEventServiceAccuracy
* refactoring: move common code to base class
* fixed bug; unsubscribe SubscriptionDataConfig
* Small performance improvement
* Minor fixes
* Avoid Symbol serialization
* Fixes coarse selection in live mode
* Fix for live coarse
* Adds protobuf (de)serialization support for Robintrack
* Adds round-trip unit test
* Minor performance improvements
* More minor performance improvements
* pass LiveNodePacket through to OandaBrokerage
* Fixes empty list becoming null value when deserializing with protobuf
* Reverts BZ live trading exception removal and fixes tests
* Refactor WorkQueue making it abstract
* Add try catch for composer
* Adds optional data batching period to LiveFillForwardEnumerator
* Override data-queue-handler with config
* Improve PeriodCountConsolidator.Scan performance
* Move batching delay to main Synchornizer thread
* Reverts addition of Robintrack protobuf definitions
* Give priority to config history provider if set
* Add Estimize protobuffing
- Add Estimize protobuffing support. Adding unit tests
* Always dispose of data queue handler
Co-authored-by: Gerardo Salazar <gsalaz9800@gmail.com>
Co-authored-by: Adalyat Nazirov <aenazirov@gmail.com>
Stefano Raggi