* Fixes 4815 by loading the requested assembly from different folder.
# Conflicts:
# ToolBox/Program.cs
* Upgrades System.Collections.Immutable to Version=1.2.5.0
* Creates a prototype for SSE streaming in IEXDataQueueHandler.
* Revert the changes in Tick.cs
* Implements a logic in IEXDataQueueHandler that updates the data-feed subscription after Subscribe/Unsubscribe
* Implements IEXCouldSubscribeMoreThan100Symbols - which fails and other small fixes.
* Implements DoForEach LinqExtensions
* Implements IEXEventSourceCollection that wraps all logic that is SSE-subscriptions and symbol-limits-per-connection concerned.
* Changes:
1) Fixes to address review.
2) Makes IexMarketPercent in QuoteSSE nullable as null values are assigned to in this field in data object received before the traing session start.
3) Deprecates helper Subscribe/Unsubscribe in IEXDataQueueHandler and IEXCouldSubscribe test.
* Fixes:
1) _refreshEvent.Reset() order was not correct - should be called before UpdateSubscription
2) ProcessJsonObject- leaves only the functionality to emit ticks.
3) IEXEventSourceCollection - replaces int counter with CountdownEvent to improve the logic - in particular, need a mechanism that would not allow the repeated call to continue until the first one is completed
* Refines the logic with parsing a data snapshot.
* Fixes few more bugs:
1) Logic in ProcessJsonObject
2) Logic in UpdateSubscription - need to introduce additional ManualResetEvent to implement the intended logic - otherwise the logic is not suitable for general case
* Introduce rate-gate limit in IEXEventSourceCollection:
because when subscribing to a bunch of shares (more than 200 for example)
the violation of rate gate policy may occur, which described in API docs asRequest Limits
IEX Cloud only applies request limits per IP address to ensure system stability.
We limit requests to 100 per second per IP measured in milliseconds, so no more than 1 request per 10 milliseconds.
SSE endpoints are limited to 50 symbols per connection. You can make multiple connections if you need to consume more than 50 symbols.:
* Few additional fixes done after real time testing
* Adds xml-docs in stream response object + renaming a file.
* Fixes:
1) Additional StreamResponseStocksUS parsing issues, that can happen outside of regular exchange hours.
2) Cancel clientUpdateThread by means of CancellationTokenSource
3) Replace BuildSymbolsQuery by string.Join
* Fixes:
1) Changing Log Trace -> Debug
2) Adds ConfigureAwait(false) to async method call
3) Removes direct reference to System.Net.Http
* Removes a task and manual reset event in IEXEventSourceCollection
* Additions:
1) IEXEventSourceCollectionSubscribes test
2) GetSnpStocksArray() helper method
3) Installs packages in QC.tests : HtmlAgilityPack & LaunchDarkly.EventSource
* IEX history provider fixes :
1) Tiny bug in ProcessJsonObject - use continue instead of return as execution is inside the for-each block)
2) Adds period variable for the historical data retrieved
3) Fixing from if (date.Date < start.Date || date.Date > end.Date) conditional check --> if (date < start || date > end) for more precise sorting.
* Changes:
1) Removes HtmlAgilityPack and SNP scraper
2) Uses hard coded symbols instead
* Bug fix:
- at certain hours (example: before pre-market open or on holidays) IEX may send no data on subscription - when trying to connect during those hours Message handler may not be fired - need to place the counter signal to client.Opened to be informed of successful connect.
* Implements:
1) IEXEventSourceCollectionSubscriptionThoroughTest and MockedIEXEventSourceCollection
2) Makes changes to IEXEventSourceCollection accordingly to allow the thorough testing.
* Fixes formatting issue in StreamResponseStocksUS
* Small fix for a new tests:
- Change RemovedClientSymbols to keep not clients itself, but symbols array, because clients are being disposed right further
* Enables extended logging in Toolbox.
* Fixing IEX historical data fetcher bugs:
1) Bug in IEXDataDownloader.cs - HistoryRequest not precisely correct.
2) Enables day-by-day daily bar downloading in IEXDataQueueHandler.
Motivation: Suppose we need data for some interval in the past - from-date=20170915-00:00:00 --to-date=20171103-00:00:00.
With current behavior IEX would have to download all the historical data from =20170915-00:00:00 up to this day.
3) Extends SynchronizingHistoryProvider
* Enables async fashion historical data download
* More fixes to IEXCouldGetHistory test.
* Reverts day-by-day daily bar downloading and other fixes.
* Removes needless packages & references
* Fix package reference
* To address review
* Sort out zero price ticks:
after testing on real-time algo 30 min before the market open now - IEX may send updates for many securities with zero lastPrice, lastSize - fix to sort such entries out
* Workaround for missing QuoteTicks timestamps:
Since we don't have a stamp for quote tick updates (only for trades) we calculate the average delay between trade tick's time stamp and local time, and
assuming that delay in average is the same for quote updates - just assign the local machine time adjusted for this average
* Simplifies the things.
* Changes:
1) Deprecates quote updates for IEX stocks.
2) Reduce the stream updates to reduce costs to ->
# Stock Quotes every 1 second (per symbol? )
# Can be up to 54,000 messages per symbol per day
https://iexcloud.io/docs/api/#how-messages-work
* Fixes:
1) IEXDataQueueHandler: give an error message on extended market hours or tick resolution subscription request. As they are not really well supported by IEX.
2) Few small fixes in IEXEventSourceCollection, including additional condition for when the subscription remains irrevocable.
* Live Coarse universe refactor
- Live trading will source Coarse and Fine fundamental data directly
from disk. Updating unit tests.
* Adds ILiveDataProvider interface
* Adds wrapper for IDataQueueHandler implementations
* Replaces IDataQueueHandler with ILiveDataProvider in
LiveTradingDataFeed
* Edits IDataQueueHandler documentation
* Maintains aggregation for current IDQH impls and skips for ILDF impls
* Note: No unit test was created for this method, go back and TODO
* Protobuf Market data
- Adding protobuf support for Ticks, TradeBars and QuoteBars. Adding
unit tests.
* Adds unit tests for LiveDataAggregator changes
* Fixes bug where custom data was not handled as it was before
* Fixes race condition bug because of variable reuse in class
* Add protobuf extension serialization
* Fixes for protobuf serialization
* Refactor
* Fix OptionChainUniverse
* replace BaseDataExchange pumping ticks with consolidators
* AlpacaBrokerage
* BitfinexBrokerage
* GDAXBrokerage
* OandaBrokerage
* InteractiveBrokers
* TradierBrokerage
* FxcmBrokerage
* PaperBrokerage
* etc
* WIP fixes for existing LTDF unit tests
* Fixes more LTDF unit tests
* make IDataAggregator.Update recieving Generic BaseData rather than Tick
* Change IDataQueueHandler.Subscribe method
* Some fixes after adding new commits
* Adds protobuf (de)serialization support for Dividend and Split
* Serialize protobuf with length prefix
* Fix missing LTDF unit tests
* Adds TiingoNews protobuf definitions
* fix comments
* more fixes on IQFeedDataQueueHandler
* disallow putting ticks into enumerator directly
* ScannableEnumerator tests
* fix OandaBrokerage
* AggregationManager unit tests
* fix AlpacaBrokerage tests
* fix InteractiveBrokers
* fix FxcmBrokerage tests
* call AggregationManager.Remove method on unsubscribe
* fix GDAX existing tests
* Fixes, refactor adding more tests for AggregatorManager
* Adds BenzingaNews protobuf definitions and round trip unit test
* Adds missing TiingoNews unit test to Protobuf round trip tests
* Improve sleep sequence of LiveSynchronizer
* need start aggregating first, and then can subscribe
* More test fixes and refactor
- Refactoring AggregationManager and ScannableEnumerator so the last is
the one that owns the consolidator
- Adding pulse on the main LiveSynchronizer
* Improve performance of LEquityDataSynchronizingEnu
* Add missing Set job packet method
* Minor performance improvements
* Improvements add test timeout
- Improvements adding test timeout to find blocking test in travis
* Improve aggregationManager performance
* Testing improvements for travis
* Remove test timeouts
* More test fixes
- Adding more missing dispose calls and improving determinism
* fix IEXDataQueueHandler and tests
* Final tweaks to LTDF tests
* more AggregationManager tests
* consume and log ticks
* fix test: couldn't subscribe to Forex tickers
* change Resolution for all bar configs
* Improve RealTimeScheduleEventServiceAccuracy
* refactoring: move common code to base class
* fixed bug; unsubscribe SubscriptionDataConfig
* Small performance improvement
* Minor fixes
* Avoid Symbol serialization
* Fixes coarse selection in live mode
* Fix for live coarse
* Adds protobuf (de)serialization support for Robintrack
* Adds round-trip unit test
* Minor performance improvements
* More minor performance improvements
* pass LiveNodePacket through to OandaBrokerage
* Fixes empty list becoming null value when deserializing with protobuf
* Reverts BZ live trading exception removal and fixes tests
* Refactor WorkQueue making it abstract
* Add try catch for composer
* Adds optional data batching period to LiveFillForwardEnumerator
* Override data-queue-handler with config
* Improve PeriodCountConsolidator.Scan performance
* Move batching delay to main Synchornizer thread
* Reverts addition of Robintrack protobuf definitions
* Give priority to config history provider if set
* Add Estimize protobuffing
- Add Estimize protobuffing support. Adding unit tests
* Always dispose of data queue handler
Co-authored-by: Gerardo Salazar <gsalaz9800@gmail.com>
Co-authored-by: Adalyat Nazirov <aenazirov@gmail.com>
The SubscriptionDataReaderHistoryProvider was using StubResultHandler, so no error messages were being shown or logged. By adding events to IHistoryProvider and SubscriptionDataReader, the dependency on IResultHandler could be removed completely and error messages are now pushed up the stack.
This is only a mechanical refactor for the updated IHistoryProvider.Initialize method in all IHistoryProvider implementations, call sites and unit tests.
When selecting IEXDataQueueHandler as a data source for live trading,
it would not connect. The following fixed the issue:
- Updated Socket IO NuGet package (and its dependencies) to latest;
- In GzipStreamProvider, removed obsolete "using" statement refering
to deprecated Socket IO API (using statement was not used anyways);
- Increased the reconnection delay value to its recommended
value (1000ms), this was necessary for the feed to connect;
- Minor change in IEXDataQueueHandler to parse Json message.
"(JObject)message" would compile, but threw a runtime error, as this
formulation could not implicitly convert string to JObject. Used
the cleaner "JObject.Parse()" method instead.
Tick.Time was not set with the information from IEX.
Tick.TickTime was defined as TickType.Quote even when it was a TickType.Trade tick. This distinction is helpful when an algorithm filters out quote ticks.
IlshatGaripov