Commit Graph

155 Commits

Author SHA1 Message Date
IlshatGaripov 29e9d678f2 Bug 4815 iex web socket library (#4914)
* Fixes 4815 by loading the requested assembly from different folder.

# Conflicts:
#	ToolBox/Program.cs

* Upgrades System.Collections.Immutable to Version=1.2.5.0

* Creates a prototype for SSE streaming in IEXDataQueueHandler.

* Revert the changes in Tick.cs

* Implements a logic in IEXDataQueueHandler that updates the data-feed subscription after Subscribe/Unsubscribe

* Implements IEXCouldSubscribeMoreThan100Symbols - which fails and other small fixes.

* Implements DoForEach LinqExtensions

* Implements IEXEventSourceCollection that wraps all logic that is SSE-subscriptions and symbol-limits-per-connection concerned.

* Changes:

1) Fixes to address review.
2) Makes IexMarketPercent in QuoteSSE nullable as null values are assigned to in this field in data object received before the traing session start.
3) Deprecates helper Subscribe/Unsubscribe in IEXDataQueueHandler and IEXCouldSubscribe test.

* Fixes:

1) _refreshEvent.Reset() order was not correct - should be called before UpdateSubscription
2) ProcessJsonObject- leaves only the functionality to emit ticks.
3) IEXEventSourceCollection - replaces int counter with CountdownEvent to improve the logic - in particular, need a mechanism that would not allow the repeated call to continue until the first one is completed

* Refines the logic with parsing a data snapshot.

* Fixes few more bugs:

1) Logic in ProcessJsonObject
2) Logic in UpdateSubscription - need to introduce additional ManualResetEvent to implement the intended logic - otherwise the logic is not suitable for general case

* Introduce rate-gate limit in IEXEventSourceCollection:

because when subscribing to a bunch of shares (more than 200 for example)
the violation of rate gate policy may occur, which described in API docs asRequest Limits

IEX Cloud only applies request limits per IP address to ensure system stability.
We limit requests to 100 per second per IP measured in milliseconds, so no more than 1 request per 10 milliseconds.
SSE endpoints are limited to 50 symbols per connection. You can make multiple connections if you need to consume more than 50 symbols.:

* Few additional fixes done after real time testing

* Adds xml-docs in stream response object + renaming a file.

* Fixes:

1) Additional StreamResponseStocksUS parsing issues, that can happen outside of regular exchange hours.
2) Cancel clientUpdateThread by means of CancellationTokenSource
3) Replace BuildSymbolsQuery by string.Join

* Fixes:

1) Changing Log Trace -> Debug
2) Adds ConfigureAwait(false) to async method call
3) Removes direct reference to System.Net.Http

* Removes a task and manual reset event in IEXEventSourceCollection

* Additions:

1) IEXEventSourceCollectionSubscribes test
2) GetSnpStocksArray() helper method
3) Installs packages in QC.tests : HtmlAgilityPack & LaunchDarkly.EventSource

* IEX history provider fixes :

1) Tiny bug in ProcessJsonObject - use continue instead of return as execution is inside the for-each block)
2) Adds period variable for the historical data retrieved
3) Fixing from  if (date.Date < start.Date || date.Date > end.Date)  conditional check -->  if (date < start || date > end)  for more precise sorting.

* Changes:

1) Removes HtmlAgilityPack and SNP scraper
2) Uses hard coded symbols instead

* Bug fix:

 - at certain hours (example: before pre-market open or on holidays) IEX may send no data on subscription - when trying to connect during those hours Message handler may not be fired - need to place the counter signal to client.Opened to be informed of successful connect.

* Implements:

1) IEXEventSourceCollectionSubscriptionThoroughTest and MockedIEXEventSourceCollection
2) Makes changes to IEXEventSourceCollection accordingly to allow the thorough testing.

* Fixes formatting issue in StreamResponseStocksUS

* Small fix for a new tests:

- Change RemovedClientSymbols to keep not clients itself, but symbols array, because clients are being disposed right further

* Enables extended logging in Toolbox.

* Fixing IEX historical data fetcher bugs:

1) Bug in IEXDataDownloader.cs - HistoryRequest not precisely correct.
2) Enables day-by-day daily bar downloading in IEXDataQueueHandler.
Motivation: Suppose we need data for some interval in the past - from-date=20170915-00:00:00 --to-date=20171103-00:00:00.
With current behavior IEX would have to download all the historical data from =20170915-00:00:00  up to this day.
3) Extends SynchronizingHistoryProvider

* Enables async fashion historical data download

* More fixes to IEXCouldGetHistory test.

* Reverts day-by-day daily bar downloading and other fixes.

* Removes needless packages & references

* Fix package reference

* To address review

* Sort out zero price ticks:

after testing on real-time algo 30 min before the market open now - IEX may send updates for many securities with zero lastPrice, lastSize - fix to sort such entries out

* Workaround for missing QuoteTicks timestamps:

Since we don't have a stamp for quote tick updates (only for trades) we calculate the average delay between trade tick's time stamp and local time, and
assuming that delay in average is the same for quote updates - just assign the local machine time adjusted for this average

* Simplifies the things.

* Changes:

1) Deprecates quote updates for IEX stocks.
2) Reduce the stream updates to reduce costs to ->
# Stock Quotes every 1 second (per symbol? )
# Can be up to 54,000 messages per symbol per day
https://iexcloud.io/docs/api/#how-messages-work

* Fixes:

1) IEXDataQueueHandler: give an error message on extended market hours or tick resolution subscription request. As they are not really well supported by IEX.
2) Few small fixes in IEXEventSourceCollection, including additional condition for when the subscription remains irrevocable.
2020-11-16 21:10:14 -03:00
James Kardatzke 46ef9a9dbb Quiver Quantitative Integration (#4869)
* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Delete QuiverHouseDataDownloader.cs

* Delete QuiverSenateDataDownloader.cs

* Delete QuiverPoliticalBetaDataDownloader.cs

* Add files via upload

* Delete QuiverHouse.cs

* Delete QuiverSenate.cs

* Delete QuiverPoliticalBeta.cs

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Delete QuiverDataAlgorithm.cs

* Add files via upload

* Add files via upload

* Add files via upload

* Addresses self review: Cleans up code and adds new unit tests

  * Adds Quiver* C# files to project
  * Adds new unit test for QuiverCongress
  * Adds Python algorithm example

* Address self reviews

- Adding some missing xml docs
- Removing unrequired imports.
- Minor rename from Date to ReportDate
- Live trading will throw InvalidOperationException

* Fixes for example algorithms

Co-authored-by: Gerardo Salazar <gsalaz9800@gmail.com>
Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
2020-11-13 20:07:17 -03:00
Stefano Raggi 934128cfa0 Binance Brokerage implementation (#4688)
* Binance Brokerage skeleton

* Market hours

* Implement Symbol Mapper

- known symbols available on /api/v1/exchangeInfo
- fiat currencies are pegged

* Implement GetCashBalance

* Implement GetAccountHoldings

- there are no pre-existing currency swaps
- cash balances are pulled and stored in the cashbook

* Implement GetOpenOrders

* Manage orders: PlaceOrder

* Manage orders: UpdateOrder

Update operation is not supported

* Manage orders: CancelOrder

* Messaging: order book

* Messaging: trades

* Messaging: combine streams

- connect to fake /ws/open channel on init
- case by channel name, but not event type

* Messaging: order depth updates

- ticker symbol is not enough as it pushes updates only once a second, this would be a very incomplete data stream
- fetch ticker snapshot if lastUpdateId == 0
- follow Binance instructions for keeping local orderbook fresh

* Messaging: user data streaming

- Request userDataStream endpoint to get listenKey
- keep listenkey alive
- handle order close event
- handle order fill event

* DataDownloader: get history

- we can aggregate minute candles for higher resolutions

* fix data stream

* Tests: FeeModel tests

* Tests: base brokerage tests

* Tests: download history

* Tests: symbol mapper

* Support StopLimit andd StopMarket orders

* StopMarket orders disabled

Take profit and Stop loss orders are not supported for any symbols (tested with BTCUSDT, ETHUSDT)

* Tests: StopLimit order

* Tests: crypto parsing

* Reissue user data listen key

* comment custom currency limitation

* rework websocket connections

* implement delayed subscription

* adapt ignore message

* add license banner

* use better suited exception type

* avoid message double parsing

* support custom fee values

* extract BinanceApiClient to manage the request/response between lean and binance

* use api events to invoke brokerage events

* do not allow to terminate session if it wasn't allocated.

* update binance exchange info

* tool to add or update binance exchange info

* ExchangeInfo basic test

* Rebase + Resharp

* Binance brokerage updates

- Fix sign bug in sell order fills
- Fix bug in GetHistory
- Remove duplicate symbol from symbol properties db

* Remove unused code

* Revert removal of account currency check

* Update symbols properties database

* Address review

* Address review

- Upgrade API endpoints from v1 to v3
- Updated sub/unsub for new subscription manager
- Subscribe best bid/ask quotes instead of full order book
- Added handling of websocket error messages
- Cleanup + refactor

* Update symbol properties database

* Remove list from symbol mapper

* Fix symbol mapper tests

* Address review

- Fix resubscribe after reconnect
- Fix quote tick edge case

* Fix EnsureCurrencyDataFeed for non-tradeable currencies

* Fix check in EnsureCurrencyDataFeed

* Reuse base class subscribe on reconnect

Co-authored-by: Adalyat Nazirov <aenazirov@gmail.com>
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
2020-09-28 15:57:10 -03:00
Stefano Raggi 375f192f07 Add PolygonDataQueueHandler (#4604)
* Add PolygonDataQueueHandler

* Add history provider and downloader for Polygon

* Add aggregator to PolygonDataQueueHandler

* Address review

- Removed duplication in message classes
- Added public Subscribe/Unsubscribe methods in PolygonWebSocketClientWrapper
- Added history requests for Forex and Crypto

* Address review

- Add security type and market arguments to downloader
- Fix time zone bug in downloader

* Remove unnecessary locks

* Add Polygon history for all resolutions

- Equity: trades and quotes
- Forex: quotes only
- Crypto: trades only
2020-08-27 12:09:20 -03:00
Mathieu Paquette 2d46674fb7 feat(Toolbox\IQFeed): optimized download history for IQFeed (#4302)
* feat(Toolbox\IQFeed): optimized download history for IQFeed

Fixes #4301

* feat(IQFeed/ToolBox): code review comments

* feat(IQFeed/ToolBox): add support for quotes and trades

* feat(IQFeed/ToolBox): trade constructor change

* feat(IQFeed/ToolBox): always apply timezone conversion

* feat(IQFeed/ToolBox): fix interval start
2020-08-21 11:26:46 -03:00
Stefano Raggi 0a046699fc Consolidate SharpZipLib library usage 2020-05-28 11:18:44 +02:00
Stefano Raggi cfb3329f33 Update System.Net.Http to latest version
- also removed unused references
2020-05-22 22:55:18 +02:00
Martin Molinero 891cf4d69b Update to .Net 4.6.2 2020-05-15 17:32:46 -03:00
Martin Molinero 269b8ba204 Remove PsychSignal data
- Remove PsychSignal data since it's no longer available
2020-05-11 18:48:13 -03:00
Michael Sena fce5d9d2e6 Remove references to IKVM.Reflection in Brokerages and ToolBox 2020-04-22 01:31:53 +10:00
AlexCatarino 72f0f44f60 Updates Pythonnet Version
From 1.0.5.29 to 1.0.5.30
Ref.: https://github.com/QuantConnect/pythonnet/pull/42
2020-04-01 16:15:05 +01:00
Martin Molinero bed8817482 Fix toolbox building always 2020-03-26 15:04:18 -03:00
Juan José D'Ambrosio ba2e64f9f4 Add blacklisted ticker list
set file as content
2020-03-11 14:45:29 +00:00
Juan José D'Ambrosio ae2531cf05 Fix issues add xml docs
Concurrent collections concerns 

Reading of fine fundamental folder improvement

Create output folder 

Implement changes in ToolBox and RDG rprograms
2020-03-10 18:26:32 +00:00
Jared ec4f714799 Merge pull request #3862 from gsalaz98/feature-3775-add-benzinga-custom-data
Implements Benzinga Custom Data
2019-11-27 15:39:22 -08:00
Gerardo Salazar e4789e0b23 Implement Benzinga News Downloader
* Make Benzinga News Converter more efficient (and cleaner!)
* Implement JSON converter for serialization/deserialization process
* Fix bug where duplicate symbols would be added to news.Symbols
* Modify algorithm strategies
2019-11-19 10:28:48 -08:00
Gerardo Salazar 629de49b10 Implement Benzinga custom data as ToolBox application and BaseData class 2019-11-13 13:38:36 -08:00
Martin Molinero ec7a8fa906 Bump PythonNet to 1.0.5.29 2019-11-13 17:33:32 -03:00
Martin Molinero c6ab54f6d2 Bump PythonNet to 1.0.5.28 2019-11-12 16:06:55 -03:00
Martin Molinero b35bc31f51 Version bump 1.0.5.26 2019-10-15 18:43:33 -03:00
Stefano Raggi 961157ed59 Update CoinApiDataQueueHandler to use CoinApi's NuGet package (#3645) 2019-10-10 17:05:01 -07:00
Martin Molinero b7f5abc43e Address reviews and improvements
- Adding new `TemporaryPathProvider`. Helper class that will provide and
clean temporary paths
- Some code clean up
2019-10-01 00:26:36 -03:00
Martin Molinero 65737297c7 Adding Tiingo new data converter 2019-10-01 00:26:36 -03:00
Martin Molinero e7ca829a0d PythonNet bump 1.0.5.25 2019-09-23 19:45:46 -03:00
Martin Molinero f96cb2a0cf Bump PythonNet to 1.0.5.24 2019-09-10 12:11:42 -03:00
Martin Molinero e396c3ef10 Bump PythonNet to 1.0.5.23 2019-09-05 23:55:30 -03:00
Martin Molinero f00049f709 Bump PythonNet to 1.0.5.22 2019-09-03 16:05:12 -03:00
Michael Handschuh c7df4e9fb3 Add StringExtensions and Parse utilizing CultureInfo.InvariantCulture
Adds static methods of the form Parse.<TypeName>(string str) that use
CultureInfo.InvariantCulture. These are to be used when parsing strings.
It's still safe (from the CA1304/CA1305 perspective) to use the ToDecimal
extension method for decimals.

Adds string extension methods for common operations that will now require
CultureInfo.InvariantCulture. These are to be used when converting values
to strings, such as ToStringInvariant()/ToStringInvariant(format), but also
useful for searching within strings, StartsWithInvariant, EndsWithInvariant
 and IndexOfInvariant.
2019-08-16 18:08:28 -03:00
Michael Handschuh 703f915182 Configure QuantConnect.ruleset in all projects
The initial ruleset is specifically aimed at addressing #3045. When we'd like
to start addressing other issues available via the FxCopAnalyzers, we can update
the action in QuantConnect.ruleset. A nice way to do it is to first set it to
warning, then go through the solution, project-by-project, fixing each warning.
At the end you can flip it to error and ensure it still builds. Moving forward,
any changes will fail if they violate the rule.

In support of #3045, we'll be configuring the following rules:
> CA1304: https://docs.microsoft.com/en-us/visualstudio/code-quality/ca1304-specify-cultureinfo?view=vs-2017
> CA1305: https://docs.microsoft.com/en-us/visualstudio/code-quality/ca1305-specify-iformatprovider?view=vs-2017
2019-08-16 18:08:18 -03:00
Michael Handschuh c42057809d Add Microsoft.CodeAnalysis.FxCopAnalyzers to all projects
FxCop has various rulesets for enforcing things within our codebase.
For this particular issue, we'll be enforcing CA1304 and CA1305 to
ensure we're always using an IFormatProvider or a CultureInfo where
applicable.

Linked Issue: #3045
2019-08-16 18:08:05 -03:00
Gerardo Salazar 436a72f435 Implements SmartInsider custom data source
Implements Smart Insider data converter ToolBox application

Includes example algorithms demonstrating use of data
2019-08-05 13:42:08 -07:00
Gerardo Salazar d394251ac9 Implement U.S. Treasury yield curve alternative data source 2019-07-24 16:48:55 -07:00
Gerardo Salazar d5e148b05c Adds Psychsignal data converter and custom data implementation 2019-07-09 17:45:31 -07:00
Jared 4a9f9448c5 Merge pull request #3264 from aytacasan/master
IQFeed Downloader Program
2019-07-03 16:53:04 -07:00
Stefano Raggi 062b27e0b0 Bitfinex DataQueueHandler June 2019 Update
- Moved IConnectionHandler and DefaultConnectionHandler from ToolBox project to Brokerages project
- Added connectionId to IConnectionHandler.Initialize to support multiple connections
- Added new BitfinexSubscriptionManager to BitfinexBrokerage to handle max 30 channels per websocket
2019-07-02 15:17:06 +02:00
AlexCatarino 7b81f1a64d Implements Estimize Equity Endpoints Alternative Data
- Release: Financial releases for the specified company
- Estimate: Financial estimates for the specified company
- Consensus: Consensus of the specified release

For each equity endpoint, we create a `BaseData`, a Downloader and add unit tests.
2019-06-21 13:14:47 +01:00
Jared 0675d5be90 Merge pull request #3329 from StefanoRaggi/coinapi-dqh-keepalive-fix
CoinApiDataQueueHandler connection handling updates
2019-06-20 13:35:18 -07:00
Stefano Raggi ed3d65dce1 Add extra logging for reconnect 2019-06-20 13:36:25 +02:00
AlexCatarino 6ce48315a6 Implements Trading Economics Data Downloader
Implements an interface for all Trading Economics Data
- Adds Trading Economics Calendar Downloader and program to be used in the Toolbox.
- Adds Trading Economics Indicators Downloader and program to be used in the Toolbox.
- Adds Trading Economics Earnings Downloader and program to be used in the Toolbox.
2019-06-20 10:34:01 +01:00
Jared bf29323df3 Merge pull request #3318 from QuantConnect/fxcm-update-jun-2019-2
Update FxcmBrokerage for TLS 1.2 support (resubmit)
2019-06-19 11:13:42 -07:00
Stefano Raggi ec66c8e415 Update FxcmBrokerage for TLS 1.2 support
These changes are required to connect and receive data properly since the TLS 1.2 server upgrade on 6/16/2019. This also required upgrading the IKVM libraries for C#/Java interop.
2019-06-18 18:35:11 +02:00
Stefano Raggi 74cdc4a767 Revert "Update FxcmBrokerage for TLS 1.2 support" 2019-06-18 18:03:12 +02:00
Stefano Raggi ca60e1c03b Update FxcmBrokerage for TLS 1.2 support
These changes are required to connect and receive data properly since the TLS 1.2 server upgrade on 6/16/2019. This also required upgrading the IKVM libraries for C#/Java interop.
2019-06-18 15:40:49 +02:00
Gerardo Salazar 2ad1f19f85 Implements Sec Custom Data, ToolBox Application, and Sample Algorithm 2019-06-17 16:45:53 -07:00
Gerardo Salazar 340cc3a330 Upgrade dependency DotNetZip to version 1.13.3 to fix security issue 2019-06-11 11:03:37 -07:00
Stefano Raggi fa2bc58564 Add IDataQueueHandler implementation for CoinAPI 2019-06-03 15:24:12 +02:00
Aytaç Aşan d6c5bc5635 IQFeed Downloader
IQFeed data downloader program that download history data for all resolutions
2019-05-31 11:36:35 -04:00
Stefano Raggi 5d6bf2ad66 Address reviews 2019-05-30 15:34:02 +02:00
Juan José D'Ambrosio 516149b793 Implement CoinApiBitfinexSymbolMapper 2019-05-29 21:49:00 +00:00
Juan José D'Ambrosio 50b3326bf7 CoinAPI refactoring includes new parameters in ToolBox 2019-05-23 23:23:52 +00:00