Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* WIP removal of custom data references & tests updates
* Regression algos updated and python algorithms moved to DataSource repos
* Fixes failing unit tests
* Add "LiveDataTypes" field to LiveNodePacket
* Adds Initialize() to IDataChannelProvider
* Adds new extension method to convert
HistoryRequest -> SubscriptionDataConfig
* Address review: Add protobuf definitions for Iconic data types
* Address review: re-adds DynamicSecurityDataAlgorithm as regression algo
* Small adjustments to variable naming and documentation
* Move test files to respective DataSource repos
Build & Test Lean / build (push) Has been cancelled
* Implemented AlphaVantageDataDownloader
* Style and documentation changes requested
* Throw exception for unexpected response content
* Added rate limiting to AlphaVantageDataDownloader
* Minor tweaks and rebase
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
Build & Test Lean / build (push) Has been cancelled
* Adds processed data directory to read price data from
* Make coarse universe generator look at data directory before failing to find daily data
* Set coarse generator output of missing daily file to debug log
* Add CoarseUniverseGenerator logs
* Fixes 100 nanosecond increment lookahead bias when parsing large numbers
* Whenever we parse a number that is has precision greater than
DateTime ticks (sub-100 nanoseconds), if we have nanoseconds
between [0, 1000), excluding numbers divisible by 100,
we will have leftover nanoseconds between [0, 100) nanoseconds, but
they won't be factored in to the DateTime calculation, since casting
to `long` only takes the integer component of the number, so we lose
the extra nanoseconds that came with the decimal, and time is set to
the "floored" value without those nanoseconds.
Since .NET `DateTime` type has a limitation of only being able
to represent time in increments of 100 nanoseconds, by not
considering the sub-100 nanoseconds, we introduce a look-ahead
bias of at most 100 nanoseconds/1 tick
* Misc adjustment to make method use `decimal` instead of `double`
for increased precision when parsing large numbers
* Changes CoinAPI data converter to support processing raw files in original directory structure and file name
* Removes Market requirement from CoinAPI data converter
* Remove timeout on decompression of raw AlgoSeek futures data
* Updates SEC downloader to use HttpClient where requests were failing
* For some unknown reason, valid requests to a valid URL were
failing when using WebClient. Changing our requester to
HttpClient fixes the issue, and enables us to leverage
async capabilities where applicable.
* Added fault tolerance to index file downloads, including a
rate limit in case we've been rate limited
* Further refactoring; catches 429 errors, adds missing rategate calls
* Replace all usage of WebClient, force retry for all failures
* Adds optional config value for Benzinga News API key in downloader
* Modifies Estimize Downloader api config name and fixes directory not found bug
* Refactor Estimize to speed up processing time
* Adds ticker limits if desired
* Misc. bug fixes, performance improvements, code cleanup
* Remove debug log statements leftover from previous commit
* Add support for non-tick Index resolutions in LeanDataWriter
* Empty commit
* Empty commit
* Empty commit
* Empty commit
* Empty commit
* Empty commit
* Lower requests/second for SEC downloader, add missing rategate call
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Update projects to use .NET 5.0, the successor to .NET Core
* Fix ambiguous errors. Add IBAutomator net5
* Remove FXCM
* Upgrade IBAutomater to v1.0.51
ignored, and an empty message aborts the commit.
* Fix rebase
- Fix ambiguous Index
- Remove StrategyCapacity.cs
- Update System.Threading.Tasks.Extensionsy
* Remove unrequired references
* Fixes
- Travis will use dotnet, not nunit nor mono
- Remove mono from foundation image
- Fix python setup in research
- Fix unit tests
* Don't call ReadKey when input is redirected
* Fix ConsoleLeanOptimizer
* Research fixes
* Update comment
* Add vsdbg to Dockerfile
* Fixes
- Revert dockerfile FROM custom changes
- Adjust and fix regression algorithms
- Option assignment will be deterministic in the order
- 'Rolling Averaged Population' is calculated using doubles, updating
expected values.
- Update readme, removing references to mono
- Add missing Py.Gil lock
* Replace ICSharp with .NET Interactive
* Fixes after rebase
* CSharp research fixes
- Adding new Initialize.csx that pre loads all assemblies
- Adjusting template research file
- Moving steps in dockerfilejupyter
- Fix unit tests and regression tests after rebase
Co-authored-by: Gerardo Salazar <gsalaz9800@gmail.com>
Co-authored-by: Stefano Raggi <stefano.raggi67@gmail.com>
Co-authored-by: Jasper van Merle <jaspervmerle@gmail.com>
* Fixes 4815 by loading the requested assembly from different folder.
# Conflicts:
# ToolBox/Program.cs
* Upgrades System.Collections.Immutable to Version=1.2.5.0
* Creates a prototype for SSE streaming in IEXDataQueueHandler.
* Revert the changes in Tick.cs
* Implements a logic in IEXDataQueueHandler that updates the data-feed subscription after Subscribe/Unsubscribe
* Implements IEXCouldSubscribeMoreThan100Symbols - which fails and other small fixes.
* Implements DoForEach LinqExtensions
* Implements IEXEventSourceCollection that wraps all logic that is SSE-subscriptions and symbol-limits-per-connection concerned.
* Changes:
1) Fixes to address review.
2) Makes IexMarketPercent in QuoteSSE nullable as null values are assigned to in this field in data object received before the traing session start.
3) Deprecates helper Subscribe/Unsubscribe in IEXDataQueueHandler and IEXCouldSubscribe test.
* Fixes:
1) _refreshEvent.Reset() order was not correct - should be called before UpdateSubscription
2) ProcessJsonObject- leaves only the functionality to emit ticks.
3) IEXEventSourceCollection - replaces int counter with CountdownEvent to improve the logic - in particular, need a mechanism that would not allow the repeated call to continue until the first one is completed
* Refines the logic with parsing a data snapshot.
* Fixes few more bugs:
1) Logic in ProcessJsonObject
2) Logic in UpdateSubscription - need to introduce additional ManualResetEvent to implement the intended logic - otherwise the logic is not suitable for general case
* Introduce rate-gate limit in IEXEventSourceCollection:
because when subscribing to a bunch of shares (more than 200 for example)
the violation of rate gate policy may occur, which described in API docs asRequest Limits
IEX Cloud only applies request limits per IP address to ensure system stability.
We limit requests to 100 per second per IP measured in milliseconds, so no more than 1 request per 10 milliseconds.
SSE endpoints are limited to 50 symbols per connection. You can make multiple connections if you need to consume more than 50 symbols.:
* Few additional fixes done after real time testing
* Adds xml-docs in stream response object + renaming a file.
* Fixes:
1) Additional StreamResponseStocksUS parsing issues, that can happen outside of regular exchange hours.
2) Cancel clientUpdateThread by means of CancellationTokenSource
3) Replace BuildSymbolsQuery by string.Join
* Fixes:
1) Changing Log Trace -> Debug
2) Adds ConfigureAwait(false) to async method call
3) Removes direct reference to System.Net.Http
* Removes a task and manual reset event in IEXEventSourceCollection
* Additions:
1) IEXEventSourceCollectionSubscribes test
2) GetSnpStocksArray() helper method
3) Installs packages in QC.tests : HtmlAgilityPack & LaunchDarkly.EventSource
* IEX history provider fixes :
1) Tiny bug in ProcessJsonObject - use continue instead of return as execution is inside the for-each block)
2) Adds period variable for the historical data retrieved
3) Fixing from if (date.Date < start.Date || date.Date > end.Date) conditional check --> if (date < start || date > end) for more precise sorting.
* Changes:
1) Removes HtmlAgilityPack and SNP scraper
2) Uses hard coded symbols instead
* Bug fix:
- at certain hours (example: before pre-market open or on holidays) IEX may send no data on subscription - when trying to connect during those hours Message handler may not be fired - need to place the counter signal to client.Opened to be informed of successful connect.
* Implements:
1) IEXEventSourceCollectionSubscriptionThoroughTest and MockedIEXEventSourceCollection
2) Makes changes to IEXEventSourceCollection accordingly to allow the thorough testing.
* Fixes formatting issue in StreamResponseStocksUS
* Small fix for a new tests:
- Change RemovedClientSymbols to keep not clients itself, but symbols array, because clients are being disposed right further
* Enables extended logging in Toolbox.
* Fixing IEX historical data fetcher bugs:
1) Bug in IEXDataDownloader.cs - HistoryRequest not precisely correct.
2) Enables day-by-day daily bar downloading in IEXDataQueueHandler.
Motivation: Suppose we need data for some interval in the past - from-date=20170915-00:00:00 --to-date=20171103-00:00:00.
With current behavior IEX would have to download all the historical data from =20170915-00:00:00 up to this day.
3) Extends SynchronizingHistoryProvider
* Enables async fashion historical data download
* More fixes to IEXCouldGetHistory test.
* Reverts day-by-day daily bar downloading and other fixes.
* Removes needless packages & references
* Fix package reference
* To address review
* Sort out zero price ticks:
after testing on real-time algo 30 min before the market open now - IEX may send updates for many securities with zero lastPrice, lastSize - fix to sort such entries out
* Workaround for missing QuoteTicks timestamps:
Since we don't have a stamp for quote tick updates (only for trades) we calculate the average delay between trade tick's time stamp and local time, and
assuming that delay in average is the same for quote updates - just assign the local machine time adjusted for this average
* Simplifies the things.
* Changes:
1) Deprecates quote updates for IEX stocks.
2) Reduce the stream updates to reduce costs to ->
# Stock Quotes every 1 second (per symbol? )
# Can be up to 54,000 messages per symbol per day
https://iexcloud.io/docs/api/#how-messages-work
* Fixes:
1) IEXDataQueueHandler: give an error message on extended market hours or tick resolution subscription request. As they are not really well supported by IEX.
2) Few small fixes in IEXEventSourceCollection, including additional condition for when the subscription remains irrevocable.
* Binance Brokerage skeleton
* Market hours
* Implement Symbol Mapper
- known symbols available on /api/v1/exchangeInfo
- fiat currencies are pegged
* Implement GetCashBalance
* Implement GetAccountHoldings
- there are no pre-existing currency swaps
- cash balances are pulled and stored in the cashbook
* Implement GetOpenOrders
* Manage orders: PlaceOrder
* Manage orders: UpdateOrder
Update operation is not supported
* Manage orders: CancelOrder
* Messaging: order book
* Messaging: trades
* Messaging: combine streams
- connect to fake /ws/open channel on init
- case by channel name, but not event type
* Messaging: order depth updates
- ticker symbol is not enough as it pushes updates only once a second, this would be a very incomplete data stream
- fetch ticker snapshot if lastUpdateId == 0
- follow Binance instructions for keeping local orderbook fresh
* Messaging: user data streaming
- Request userDataStream endpoint to get listenKey
- keep listenkey alive
- handle order close event
- handle order fill event
* DataDownloader: get history
- we can aggregate minute candles for higher resolutions
* fix data stream
* Tests: FeeModel tests
* Tests: base brokerage tests
* Tests: download history
* Tests: symbol mapper
* Support StopLimit andd StopMarket orders
* StopMarket orders disabled
Take profit and Stop loss orders are not supported for any symbols (tested with BTCUSDT, ETHUSDT)
* Tests: StopLimit order
* Tests: crypto parsing
* Reissue user data listen key
* comment custom currency limitation
* rework websocket connections
* implement delayed subscription
* adapt ignore message
* add license banner
* use better suited exception type
* avoid message double parsing
* support custom fee values
* extract BinanceApiClient to manage the request/response between lean and binance
* use api events to invoke brokerage events
* do not allow to terminate session if it wasn't allocated.
* update binance exchange info
* tool to add or update binance exchange info
* ExchangeInfo basic test
* Rebase + Resharp
* Binance brokerage updates
- Fix sign bug in sell order fills
- Fix bug in GetHistory
- Remove duplicate symbol from symbol properties db
* Remove unused code
* Revert removal of account currency check
* Update symbols properties database
* Address review
* Address review
- Upgrade API endpoints from v1 to v3
- Updated sub/unsub for new subscription manager
- Subscribe best bid/ask quotes instead of full order book
- Added handling of websocket error messages
- Cleanup + refactor
* Update symbol properties database
* Remove list from symbol mapper
* Fix symbol mapper tests
* Address review
- Fix resubscribe after reconnect
- Fix quote tick edge case
* Fix EnsureCurrencyDataFeed for non-tradeable currencies
* Fix check in EnsureCurrencyDataFeed
* Reuse base class subscribe on reconnect
Co-authored-by: Adalyat Nazirov <aenazirov@gmail.com>
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Add PolygonDataQueueHandler
* Add history provider and downloader for Polygon
* Add aggregator to PolygonDataQueueHandler
* Address review
- Removed duplication in message classes
- Added public Subscribe/Unsubscribe methods in PolygonWebSocketClientWrapper
- Added history requests for Forex and Crypto
* Address review
- Add security type and market arguments to downloader
- Fix time zone bug in downloader
* Remove unnecessary locks
* Add Polygon history for all resolutions
- Equity: trades and quotes
- Forex: quotes only
- Crypto: trades only
* Adds support for live algorithms using TE calendar events
* Modifies TradingEconomics tickers to include country
* Fixes bug where `TECal.Clone(...)` could result in non-deterministic
results
* Fixes bugs in TradingEconomicsEventFilter
* Adds new CSV parsing methods in StreamReaderExtensions
* Moves ToolBox/TradingEconomicsCalendarTests to Common tests folder
* Updates TradingEconomicsCalendar tests to test CSV parsing
* TradingEconomicsCalendarDownloader now writes files to disk as JSON
* Make Benzinga News Converter more efficient (and cleaner!)
* Implement JSON converter for serialization/deserialization process
* Fix bug where duplicate symbols would be added to news.Symbols
* Modify algorithm strategies
- Release: Financial releases for the specified company
- Estimate: Financial estimates for the specified company
- Consensus: Consensus of the specified release
For each equity endpoint, we create a `BaseData`, a Downloader and add unit tests.
Implements an interface for all Trading Economics Data
- Adds Trading Economics Calendar Downloader and program to be used in the Toolbox.
- Adds Trading Economics Indicators Downloader and program to be used in the Toolbox.
- Adds Trading Economics Earnings Downloader and program to be used in the Toolbox.
Delete converter because an existing utility already existed
Moved code from factory to converter class as statics
Add rankandfile as a CIK to ticker mapping source
Created random-seed argument for rdg in order to let the user get deterministic output
Update documentation in 'FactorFileRow.cs' to accurately reflect factor
file structure
Update CSV generation for FactorFile so that it uses FactorFileRow's CSV
generator
Add FinancialCalendar to make it easier to implement logic regarding
financial quarters
Add mapping events to RandomDataGenerator
Update MapFileRow ToCsv method to correctly emit the ticker as lowercase
Fix bug in FactorFile where we would get the same initial data point twice
when we converted it to CSV
Create new method to convert a MapFile to CSV
Create new method to write MapFile to disk as CSV
Add unit test to test for successful CSV generation in MapFile
Add new files to project
Add FinancialCalendar unit tests
Create new class to handle generation of dividends, splits, and maps
This side steps the issue reported in #2840 by removing the desire to pass
zero for the maximum deviation. The previous issue was that we were leaning
on the trade ticks to produce the series variation and then trying to fit
quote ticks around a previously generated trade tick. This solution permits
both quote and trade ticks to produce variations and prevents the generator
from emitting both at the same time step.
A new parameter, --quote-trade-ratio, determines the relative density of each.
For example, a quote trade ratio of 1 means equal trade and quote ticks, whereas
a value of 2 means twice as many quote ticks as trade ticks.
This change also removes special treatment regarding the downsizing of the
requested deviation for quote ticks. If the consumer wants to limit the
deviation of quote ticks than the consumer can make that decision, but the
random value generator should simply follow instructions like a good little
boy.
Fixes#2840
Adalyat Nazirov