* Fixes 4815 by loading the requested assembly from different folder.
# Conflicts:
# ToolBox/Program.cs
* Upgrades System.Collections.Immutable to Version=1.2.5.0
* Creates a prototype for SSE streaming in IEXDataQueueHandler.
* Revert the changes in Tick.cs
* Implements a logic in IEXDataQueueHandler that updates the data-feed subscription after Subscribe/Unsubscribe
* Implements IEXCouldSubscribeMoreThan100Symbols - which fails and other small fixes.
* Implements DoForEach LinqExtensions
* Implements IEXEventSourceCollection that wraps all logic that is SSE-subscriptions and symbol-limits-per-connection concerned.
* Changes:
1) Fixes to address review.
2) Makes IexMarketPercent in QuoteSSE nullable as null values are assigned to in this field in data object received before the traing session start.
3) Deprecates helper Subscribe/Unsubscribe in IEXDataQueueHandler and IEXCouldSubscribe test.
* Fixes:
1) _refreshEvent.Reset() order was not correct - should be called before UpdateSubscription
2) ProcessJsonObject- leaves only the functionality to emit ticks.
3) IEXEventSourceCollection - replaces int counter with CountdownEvent to improve the logic - in particular, need a mechanism that would not allow the repeated call to continue until the first one is completed
* Refines the logic with parsing a data snapshot.
* Fixes few more bugs:
1) Logic in ProcessJsonObject
2) Logic in UpdateSubscription - need to introduce additional ManualResetEvent to implement the intended logic - otherwise the logic is not suitable for general case
* Introduce rate-gate limit in IEXEventSourceCollection:
because when subscribing to a bunch of shares (more than 200 for example)
the violation of rate gate policy may occur, which described in API docs asRequest Limits
IEX Cloud only applies request limits per IP address to ensure system stability.
We limit requests to 100 per second per IP measured in milliseconds, so no more than 1 request per 10 milliseconds.
SSE endpoints are limited to 50 symbols per connection. You can make multiple connections if you need to consume more than 50 symbols.:
* Few additional fixes done after real time testing
* Adds xml-docs in stream response object + renaming a file.
* Fixes:
1) Additional StreamResponseStocksUS parsing issues, that can happen outside of regular exchange hours.
2) Cancel clientUpdateThread by means of CancellationTokenSource
3) Replace BuildSymbolsQuery by string.Join
* Fixes:
1) Changing Log Trace -> Debug
2) Adds ConfigureAwait(false) to async method call
3) Removes direct reference to System.Net.Http
* Removes a task and manual reset event in IEXEventSourceCollection
* Additions:
1) IEXEventSourceCollectionSubscribes test
2) GetSnpStocksArray() helper method
3) Installs packages in QC.tests : HtmlAgilityPack & LaunchDarkly.EventSource
* IEX history provider fixes :
1) Tiny bug in ProcessJsonObject - use continue instead of return as execution is inside the for-each block)
2) Adds period variable for the historical data retrieved
3) Fixing from if (date.Date < start.Date || date.Date > end.Date) conditional check --> if (date < start || date > end) for more precise sorting.
* Changes:
1) Removes HtmlAgilityPack and SNP scraper
2) Uses hard coded symbols instead
* Bug fix:
- at certain hours (example: before pre-market open or on holidays) IEX may send no data on subscription - when trying to connect during those hours Message handler may not be fired - need to place the counter signal to client.Opened to be informed of successful connect.
* Implements:
1) IEXEventSourceCollectionSubscriptionThoroughTest and MockedIEXEventSourceCollection
2) Makes changes to IEXEventSourceCollection accordingly to allow the thorough testing.
* Fixes formatting issue in StreamResponseStocksUS
* Small fix for a new tests:
- Change RemovedClientSymbols to keep not clients itself, but symbols array, because clients are being disposed right further
* Enables extended logging in Toolbox.
* Fixing IEX historical data fetcher bugs:
1) Bug in IEXDataDownloader.cs - HistoryRequest not precisely correct.
2) Enables day-by-day daily bar downloading in IEXDataQueueHandler.
Motivation: Suppose we need data for some interval in the past - from-date=20170915-00:00:00 --to-date=20171103-00:00:00.
With current behavior IEX would have to download all the historical data from =20170915-00:00:00 up to this day.
3) Extends SynchronizingHistoryProvider
* Enables async fashion historical data download
* More fixes to IEXCouldGetHistory test.
* Reverts day-by-day daily bar downloading and other fixes.
* Removes needless packages & references
* Fix package reference
* To address review
* Sort out zero price ticks:
after testing on real-time algo 30 min before the market open now - IEX may send updates for many securities with zero lastPrice, lastSize - fix to sort such entries out
* Workaround for missing QuoteTicks timestamps:
Since we don't have a stamp for quote tick updates (only for trades) we calculate the average delay between trade tick's time stamp and local time, and
assuming that delay in average is the same for quote updates - just assign the local machine time adjusted for this average
* Simplifies the things.
* Changes:
1) Deprecates quote updates for IEX stocks.
2) Reduce the stream updates to reduce costs to ->
# Stock Quotes every 1 second (per symbol? )
# Can be up to 54,000 messages per symbol per day
https://iexcloud.io/docs/api/#how-messages-work
* Fixes:
1) IEXDataQueueHandler: give an error message on extended market hours or tick resolution subscription request. As they are not really well supported by IEX.
2) Few small fixes in IEXEventSourceCollection, including additional condition for when the subscription remains irrevocable.
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Delete QuiverHouseDataDownloader.cs
* Delete QuiverSenateDataDownloader.cs
* Delete QuiverPoliticalBetaDataDownloader.cs
* Add files via upload
* Delete QuiverHouse.cs
* Delete QuiverSenate.cs
* Delete QuiverPoliticalBeta.cs
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Delete QuiverDataAlgorithm.cs
* Add files via upload
* Add files via upload
* Add files via upload
* Addresses self review: Cleans up code and adds new unit tests
* Adds Quiver* C# files to project
* Adds new unit test for QuiverCongress
* Adds Python algorithm example
* Address self reviews
- Adding some missing xml docs
- Removing unrequired imports.
- Minor rename from Date to ReportDate
- Live trading will throw InvalidOperationException
* Fixes for example algorithms
Co-authored-by: Gerardo Salazar <gsalaz9800@gmail.com>
Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
* Add offset capabilities and related tests
* Add tests for Weekend Offsets on Symbol DateRules
* Unify Iterator Behavior
* Refactor and consolidate functions to reduce duplicate code
* Positive offset values only
* Address review
* Expand Month tests to include Forex and Crypto cases
* Ensure order of days in schedule
* Refactor and unify behavior
* More edge cases and tuning
* Address review + more tests
* Refactor and hotfix run script
Removed dependency on `realpath`, notably absent from macOS and some debian distros. Replaced with an equivalent bash function.
Cleaned up prompt response handling, replaced var tests with param expressions.
Removed potentially uneeded calls to sudo (again absent on some systems and most users are part of the docker group anyway). Instead, test if it's needed and get preauth with sudo -v.
Poll for running and stopped Lean containers, and prompt before replacing them. Would fail prior.
Uppercased variabled. Sorry.
* Revert docker output redirection to stderr
* Adjustments to maintain all functionality
* Mimic behavior across run scripts
* Update readme to reflect Docker script changes
* Ignore results storage
* Correct print statement
* Mirror changes on research docker scripts
* Make executable
* Handle already running container and sudo privs
* Fix for issues found in testing
* Address Review
* Small adjustments found in linux testing
* Doc improvement
* Add auto update option for Docker images
* Fix image var reference
Co-authored-by: Peter Kazazes <peter@peterk.co>
* DividedEventProvider distribution computation
- Update regression algorithm which was using a different reference
price when calculating the dividend
- Adjust divided event provider to compute distribution using factor
file reference price, if not 0. Adding unit tests
- For equities, only emit auxiliary data points for
TradeBar configurations, not for QuoteBars, nor internal.
* Address reviews
- Split and Dividend event provider will throw an exception when there
is no reference price available. Updating `wm` factor file which was
missing references price and regression algorithms using WM.
- Updating unit tests asserting new exception
* Protobuf will use recyclable memory stream
- Serialization will reuse recyclable memory streams
- Remove serialization of exchange and sale condition for ticks.
Updating unit tests.
- There is no need to serialize BaseData.EndTime, covered by unit tests.
* Tick will keep a parsed sale condition property
* Readd tick exchange. Json ignore ParsedSaleCondition
* Implement future type filter
* Filter weeklys test
* Fix for test, contracts were being filtered out by new type filter
* Share core contract filtering logic in new base class
* Catch Future symbols we don't have Expiry functions for
* Expand tests for new filtering
* Address review
* Small doc change
* Compare Date component for expiry
* Clarifying comment
* Implements Equity Fill Model
This commit sets the base to create a new equity fill model and the `EquityFillModel` is just a copy of `FillModel`.
* Adds Summary to FillModelPythonWrapper.GetPricesInternal
Adds summary to FillModelPythonWrapper.GetPricesInternal with remarks that it's a temporarily method to help the refactoring of fill models.
* Change GetSubscriptionDataConfigs to return IEnumerable<SubscriptionDataConfig>
* Move localTime outside loop
* Remove legacy code
* Remove unnecessary OrderBy
* Reorder conditions to reduce number of times Contains() is called
* Revert "Remove unnecessary OrderBy"
This reverts commit 85383b062e5ac958ec09231df21e0fa3efd3622d.
* Revert "Change GetSubscriptionDataConfigs to return IEnumerable<SubscriptionDataConfig>"
This reverts commit cbd97c9f36e7ed349f1dea740b417e73837bd42b.
* Add RequestId to request information logging
* Use unique request id across all request types (orders, subscriptions, data queries)
- Previously we had three separate counters for request types and this was causing request information messages to be overwritten (different request types with same ids)
* Update GetContractDetails to log all contracts found
* Store temp files in subdirectory
* Fix Dispose case for new temp dir
* Adjust tests for new temp dir
* Dispose unit tests
* Unit test for issue 4811
* Refactor for not using temp files
* Fix storage checks for saving data, plus tests
* Use Base64 for storing keys and decoding them; handles odd key strings
* Don't allow "?" in a key
* Address review
* Deleted test cases
* PersistData handle deletion of files
* Refactor GetFilePath to use Persist()
* Make PathForKey protected
* Add decimal places as parameters to get dividends with arbitrary precision
Also, increase precision when generating strings from factor files rows.
* Add xml documentation entries for new optional arguments.
* Adds CustomBuyingPowerModelAlgorithm
This algorithms is an example on how to implement a custom buying power model.
In this particular case, it shows how to override `HasSufficientBuyingPowerForOrder` in order to place orders without sufficient buying power according to the default model.
* Upgrades CustomModelsAlgorithm to Include CustomBuyingPowerModel
The custom buying power model overrides `HasSufficientBuyingPowerForOrderResult` but it doesn't change the trades and, consequently, the regression statistics.
* Allow account currency to be overridden by the algorithm
* Fix failing unit test
* Address review
- Revert removal of call check in SecurityPortfolioManager.SetAccountCurrency
- Revert changes to unit tests
- IBrokerage.AccountBaseCurrency now defaults to null
- BrokerageSetupHandler will not change the algorithm's account currency if the brokerage returns null, allowing the algorithm to call SetAccountCurrency in Initialize
* Fail on restart investing after liquidation
I added a line so that the trailing high value could be rebalanced and the investment process won't be stop by high value always more than current value by drawdown percent.
* Update MaximumDrawdownPercentPortfolio.py
* Fix for MaximumDrawdownPercentPortfolio
- Fix C# MaximumDrawdownPercentPortfolio to reset portfolio value after
liquidation. Only reset once we have actually adjusted some targets.
Updating regression algorithms.
Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
* Remove F# Project, Splits, and Dividends Tests
* Separate tests that require external accounts; read from config
* Removal of non supported "prices" endpoint test
* Removal of unsupported API functions
* Address review
* NOP GetLastPrice for removal of Prices endpoint
* Post rebase fix
* Rebase fix 2
* remove /r/n from eof for api tests
* Reflect similar refactors to NodeTests
* Fix for live algorithm API testing
* Address Review
* Add underlying holdings to regression result handler details log
When debugging option exercise/assignment issues it's useful to see the
underlying holdings at the time the option contract fill event is processed.
Also adds the full symbol string to the top of the order event section.
The Symbol.Value was being logged via OrderEvent.ToString(), but it wasn't
the full SecurityIdentifier - by including the full SID string it makes it
easier to correlate fills over symbol rename boundaries.
* Fix automatic option assignment from market simulation
During the recent OptionExerciseOrder.Quantity refactor, this case was missed.
Additionally, it was realized that there were no regression tests covering the
automatic assignment via the market conditions simulation. This change introduces
a regression algorithm that covers the automatic assignment of put/call options.
* Update BasicOptionAssignmentSimulation._rand to be non-static
If this value is static then we reuse the same Random instance for ALL regression
tests, thereby defeating the purpose of using a well known seed number. This means
we get different results based on the order execution of preceding algorithms.
By making this an instance variable each algorithm will start with the same seed
value, ensuring consistent runs between regression tests, either run as a suite or
running a single algorithm in isolation.
* Do not update price scale for fillforward data
- Do no update price scale for fill forward data. FillForward data
should keep using the prev scale for which it was created. Adding unit tests
- When cloning do not lose IsFillForward flag state, affects
QuoteBars/Ticks, does not affect TradeBars since they perform a memberwise clone.
Adding unit tests
* Auxiliaries shouldn't really affect on applied price factor scale.
Despite we can receeive FillForward'ed data points, corresponding
Auxiliaries for them are not FillForward so we do meet the condition
and then refresh price factor. As a result all futher FF data points are scaled too.
* Regression algorithm to check that FillForward'ed data points arrived with last real price factor
* Add trade for regression algorithm
- Minot tweaks and adding trade for new regression algorithm.
- Updating AddOptionContractExpiresRegressionAlgorithm because it is
using the symbol for which new data was added.
Co-authored-by: Adalyat Nazirov <aenazirov@gmail.com>
* OptionChain and OptionContract improvements
- QCAlgorithm.AddUniverse will return the added Universe instance.
- Adding new OptionChainedUniverseSelectionModel will monitor a Universe changes
and will spwan new OptionChainUniverse from it's selections. Adding
regression test Py/C#.
- Adding new OptionContractUniverse that will own option contracts and
their underlying symbol. Adding regression test
- Fix double notification for security changes, bug seen in updated
UniverseSelectionRegressionAlgorithm
- Remove UniverseSelection special handling for Option and Future chains
- Fix DataManager not removing SubscriptionDataConfigs for Subscriptions
which finished before being removed from the universe
- Refactor detection of user added Universe so that they do not get
removed after calling the UniverseSelectionModel
* Add check for option underlying price is set
* Address reviews
- Adding python regression algorithm for
`AddOptionContractFromUniverseRegressionAlgorithm`
and `AddOptionContractExpiresRegressionAlgorithm`
- Rename QCAlgorithm new api method to `AddChainedOptionUniverse`
* Fix universe refresh bug
- Fix bug where a universe selection refresh would cause option or
future chain universes from being removed. Adding regression algorithm
reproducing the issue.
* Rename new option universe Algorithm API method
- Rename new option universe Algorith API method from
AddChainedOptionUniverse to AddUniverseOptions
- Rebase and update regression test order hash because of
option expiration message changed
* Add property IBrokerage.AccountBaseCurrency
* Set AccountCurrency to brokerage AccountBaseCurrency
* Remove USD AccountCurrency check
* Fix Oanda account base currency
* Fix currency symbol in CashBook.ToString()
* Fix unit tests
* Address review
* Add DebugMessage when changing account currency
* Add debug message for brokerage account base currency
* Fix currency symbol in equity chart and runtime statistics
* Update unit tests
* Improve information tracked in regression's {algorithm}.{lang}.details.log
The details.log file aims at providing a diff-able document that quickly and
easily provides actionable information. Since many regression algorithms use
the algorithm's debug/error messaging facilities to log various pieces of algo
state. This document also support a configuration option: regression-high-fidelity-logging'
that logs EVERY piece of data, again, with the aim of providing an easily diff-able
documenbt to quickly highlight actionable information. I may have missed omse key
pieces of information here, but now that the entire QC knows about this regression
tool, if additional information is required then hopefully it's easy enough at this
point to extend the RegressionResultHandler to suit our needs.
The RegressionResultHandler was initially implemented to provide a concise log of
all orders. This was achieved by simply using the Order.ToString method. While
testing/investigating OptionExerciseOrder behavior, it became evident that more
information was required to properly identify the source of potential failures or
differences between previous regression test runs. This change adds logging for
almost every IResultHandler method and additionally attempts to capture the
actual portfolio impact of every OrderEvent. This is accomplished by logging
the portfolio's TotalPortfolioValue, Cash properties and the security's
SecurityHolding.Quantity property.
This change also standardizes the timestamps used to folloow the ISO-8601 format.
When using the RegressionResultHandler, it is highly recommeded to also disable
'forward-console-message' configuration option to ensure algorithm Debug/Error
message logging is done synchronously to ensure correct ordering with respect to
log messages via Log.Debug/Trace/Error.
* Fix typo in options OrderTests test case name
* Update SymbolRepresentation.GenerationOptionTickerOSI to extension method
Far more convenient as an extension method
* Improve R# default code formatting rules
Many of these rule changes focus on improving the readability of code,
with a particular emphasis on multi-line constructs, chained method calls
and multi-line method invocations/declarations.
* Add braces, use string interpolation and limit long lines
* Refactor OptionExerciseOrder.Quantity to indicate change in #contracts
For all other order types, the Order.Quantity indicates the change in the algorithm's
holdings upon order execution for the order's symbol. For OptionExerciseOrder, this
convention was broken. It appears as though only exercise was initially implemented,
in which case only long positions were supported and a code comment indicated that
only positive values of quantity were acceptable, indicating the number of contracts
to exercise. At a later date, assignment simulation was added and utilized a negative
order quantity. This caused some major inconsistencies in how models view exercise
orders compared to all other order types. This change brings OptionExerciseOrder.Quantity
into alignment with the other order types by making it represent the change in holdings
quantity upon order execution.
This change was originally going to be much larger, but in order to minimize risks and to
make for an easier review experience, the additional changes will be committed separately
and pushed in their own PR. Some of the issues identified include:
* Manual Exercise (especially for OTM) is not covered
* Margin Calculations (in particular taking into account opposing contracts held)
* IBrokerage.OptionPositionAssigned is raised for exercise (later filtered by tx handler)
Fixes OptionPortfolioModelTests to use exercise model to properly model exercise of
non-account quote currency option contract.
* Include Order.Tag/OrderEvent.Message in their ToString, Fix default tag values
There was inconsistencies in what we were checking for. The order constructors
default the tag parameter to an empty string but Order.CreateOrder checks for
a null string. Additionally, the order constructors (limit,stopmarket,stoplimit)
would check for an empty string and if so, apply a default order tag.
This change cleans these checks up using string.IsNullOrEmpty and also removes the
check from Order.CreateOrder since we're passing the tag into the various order
constructors.
* Improve information tracked in regression's {algorithm}.{lang}.details.log
The details.log file aims at providing a diff-able document that quickly and
easily provides actionable information. Since many regression algorithms use
the algorithm's debug/error messaging facilities to log various pieces of algo
state. This document also support a configuration option: regression-high-fidelity-logging'
that logs EVERY piece of data, again, with the aim of providing an easily diff-able
documenbt to quickly highlight actionable information. I may have missed omse key
pieces of information here, but now that the entire QC knows about this regression
tool, if additional information is required then hopefully it's easy enough at this
point to extend the RegressionResultHandler to suit our needs.
The RegressionResultHandler was initially implemented to provide a concise log of
all orders. This was achieved by simply using the Order.ToString method. While
testing/investigating OptionExerciseOrder behavior, it became evident that more
information was required to properly identify the source of potential failures or
differences between previous regression test runs. This change adds logging for
almost every IResultHandler method and additionally attempts to capture the
actual portfolio impact of every OrderEvent. This is accomplished by logging
the portfolio's TotalPortfolioValue, Cash properties and the security's
SecurityHolding.Quantity property.
This change also standardizes the timestamps used to folloow the ISO-8601 format.
When using the RegressionResultHandler, it is highly recommeded to also disable
'forward-console-message' configuration option to ensure algorithm Debug/Error
message logging is done synchronously to ensure correct ordering with respect to
log messages via Log.Debug/Trace/Error.
* Fix typo in options OrderTests test case name
* Update SymbolRepresentation.GenerationOptionTickerOSI to extension method
Far more convenient as an extension method
* Improve R# default code formatting rules
Many of these rule changes focus on improving the readability of code,
with a particular emphasis on multi-line constructs, chained method calls
and multi-line method invocations/declarations.
* Add braces, use string interpolation and limit long lines
* Refactor OptionExerciseOrder.Quantity to indicate change in #contracts
For all other order types, the Order.Quantity indicates the change in the algorithm's
holdings upon order execution for the order's symbol. For OptionExerciseOrder, this
convention was broken. It appears as though only exercise was initially implemented,
in which case only long positions were supported and a code comment indicated that
only positive values of quantity were acceptable, indicating the number of contracts
to exercise. At a later date, assignment simulation was added and utilized a negative
order quantity. This caused some major inconsistencies in how models view exercise
orders compared to all other order types. This change brings OptionExerciseOrder.Quantity
into alignment with the other order types by making it represent the change in holdings
quantity upon order execution.
This change was originally going to be much larger, but in order to minimize risks and to
make for an easier review experience, the additional changes will be committed separately
and pushed in their own PR. Some of the issues identified include:
* Manual Exercise (especially for OTM) is not covered
* Margin Calculations (in particular taking into account opposing contracts held)
* IBrokerage.OptionPositionAssigned is raised for exercise (later filtered by tx handler)
Fixes OptionPortfolioModelTests to use exercise model to properly model exercise of
non-account quote currency option contract.
IlshatGaripov