Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
* Handles Data Subscriptions with One Security Type
Polygon sells data subscriptions for security types separately, so we should not test whether the users have all three subscriptions, Equity, Forex, and Crypto. Users just need the type they will use.
Adds new crypto markets, and unit tests for them. Adapt unit test.
* Fixes ToolBox Downloader
Adds `--api-key` argument.
Closes#6687
* Address reviews
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
Python Virtual Environments / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
* WIP
* Add base currency cash
* Symbol properties and data processing
* Add basic template algorithm
* Add hourly crypto future algorithm
* Minor fixes after live trading testing
* CoinApiDataQueueHandler CryptoFuture support
* Address reviews
* Fix regression algorithms after update
* use research-object-store-name key from config while creating object store for research notebooks
* initial removal
* remove reference
* move test to private repo containing downloaders
* use research-object-store-name key from config while creating object store for research notebooks
* remove bitfinex files
* remove redundant ref
* move test to private repo
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* make public safe to expose test properties
* extract ib files
* remove dependencies
* refactor to use ib reference as string
* remove un-used dependencies
* remove redundant refrences
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* add gdax exchange info downloader
* add downloader method to program]
* fetch currency description
* change definition to include headers
* use extension method to make request
* remove log from test
* replace WebRequest
* cleanup
* use relevant name
* implement IExchangeInfoDownloader for bitfinex, initial commit
* add default values
* use default market value
* use correct attribute for lotsize
* don't skip missing values
* handle multiple downloaders
* add gdax and bitfinex exchange downloader
* follow LEAN data directory structure
* update SPDB
* order tickers
* order tickers
* add exchange info downloader test template
* delete files
* update SPDB
* use currency mapping
* update bitfinex symbols
* update currency mapping
* sort result after old currency symbols are used
* use market of the respective brokerage
* no more unknown symbol
* change minimum order size value
* direct conversion possible
* update bitfinex symbols
* change user-agent
* add test for indirect conversion
* update stats
* extract binance brokerage
* remove binance tools from main toolbox app
* modify OrderTestParameters constructor
* fix BinanceBrokerage ref config.json
* update symbol properties
* move BinanceFeeModel tests to the right place
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* replace to local functions as they are more performant
* fix random generator upper bound
Next() includes minValue, but not maxValue, so we increment it +1
* introduce abstract layers
* refactoring
* fix tets
* adapt tests
* fixup
* implement blackschole price model for options
* use risk free rate
* use ql price model
* wip
* change interface
* fix
* tidy up the code
* wip
* iterate groupped symbols
* wip
* wip
* fix
* allow symbol of different types
* improve settings
* wip
* iterate full range
* fix issue with negative option
* fix
* fixup
* use StandardDeviationOfReturnsVolatilityModel
* re-use existing tick types per security type
* parametrize underlying security type
* use default option style
* dynamic option price model
* fix enumeration
* test
* fix unit tests
* refactor code
* remove unused file
* minor tweaks and refactoring
* rename symbol generator class
* fix interface
* add comments
* more comments and unit tests
* more tests
* add disclaimer
* more tests
* more comments and tests
* split tests into different files
* tidy up the code
* tidy up the code; more tests
* refactor TickGenerator => use security price directly on each iteration
* remove dupe; reuse main constructor
* use SecurityManager, refactor code
* bugfix: save ticks in history array
* check volatility warm up & tests
* more unit tests
* describe volatility period span in settings
* rename command line option
* Minor adjusments. Address review
- Use Lean log handler instead of writting directly to console
- Rename BlackShcolesPriceGenerator to generically OptionPriceModelPriceGenerator
- Minor format clean up & standarization
- Add support for specifying the option chain size
* Rename TickGenerator private fields
* Fix unit tests
* fix tests class name
* Support tickers being specified
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* WIP removal of custom data references & tests updates
* Regression algos updated and python algorithms moved to DataSource repos
* Fixes failing unit tests
* Add "LiveDataTypes" field to LiveNodePacket
* Adds Initialize() to IDataChannelProvider
* Adds new extension method to convert
HistoryRequest -> SubscriptionDataConfig
* Address review: Add protobuf definitions for Iconic data types
* Address review: re-adds DynamicSecurityDataAlgorithm as regression algo
* Small adjustments to variable naming and documentation
* Move test files to respective DataSource repos
Build & Test Lean / build (push) Has been cancelled
* Implemented AlphaVantageDataDownloader
* Style and documentation changes requested
* Throw exception for unexpected response content
* Added rate limiting to AlphaVantageDataDownloader
* Minor tweaks and rebase
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
Build & Test Lean / build (push) Has been cancelled
* Adds processed data directory to read price data from
* Make coarse universe generator look at data directory before failing to find daily data
* Set coarse generator output of missing daily file to debug log
* Add CoarseUniverseGenerator logs
* Fixes 100 nanosecond increment lookahead bias when parsing large numbers
* Whenever we parse a number that is has precision greater than
DateTime ticks (sub-100 nanoseconds), if we have nanoseconds
between [0, 1000), excluding numbers divisible by 100,
we will have leftover nanoseconds between [0, 100) nanoseconds, but
they won't be factored in to the DateTime calculation, since casting
to `long` only takes the integer component of the number, so we lose
the extra nanoseconds that came with the decimal, and time is set to
the "floored" value without those nanoseconds.
Since .NET `DateTime` type has a limitation of only being able
to represent time in increments of 100 nanoseconds, by not
considering the sub-100 nanoseconds, we introduce a look-ahead
bias of at most 100 nanoseconds/1 tick
* Misc adjustment to make method use `decimal` instead of `double`
for increased precision when parsing large numbers
* Changes CoinAPI data converter to support processing raw files in original directory structure and file name
* Removes Market requirement from CoinAPI data converter
* Remove timeout on decompression of raw AlgoSeek futures data
* Updates SEC downloader to use HttpClient where requests were failing
* For some unknown reason, valid requests to a valid URL were
failing when using WebClient. Changing our requester to
HttpClient fixes the issue, and enables us to leverage
async capabilities where applicable.
* Added fault tolerance to index file downloads, including a
rate limit in case we've been rate limited
* Further refactoring; catches 429 errors, adds missing rategate calls
* Replace all usage of WebClient, force retry for all failures
* Adds optional config value for Benzinga News API key in downloader
* Modifies Estimize Downloader api config name and fixes directory not found bug
* Refactor Estimize to speed up processing time
* Adds ticker limits if desired
* Misc. bug fixes, performance improvements, code cleanup
* Remove debug log statements leftover from previous commit
* Add support for non-tick Index resolutions in LeanDataWriter
* Empty commit
* Empty commit
* Empty commit
* Empty commit
* Empty commit
* Empty commit
* Lower requests/second for SEC downloader, add missing rategate call
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Update projects to use .NET 5.0, the successor to .NET Core
* Fix ambiguous errors. Add IBAutomator net5
* Remove FXCM
* Upgrade IBAutomater to v1.0.51
ignored, and an empty message aborts the commit.
* Fix rebase
- Fix ambiguous Index
- Remove StrategyCapacity.cs
- Update System.Threading.Tasks.Extensionsy
* Remove unrequired references
* Fixes
- Travis will use dotnet, not nunit nor mono
- Remove mono from foundation image
- Fix python setup in research
- Fix unit tests
* Don't call ReadKey when input is redirected
* Fix ConsoleLeanOptimizer
* Research fixes
* Update comment
* Add vsdbg to Dockerfile
* Fixes
- Revert dockerfile FROM custom changes
- Adjust and fix regression algorithms
- Option assignment will be deterministic in the order
- 'Rolling Averaged Population' is calculated using doubles, updating
expected values.
- Update readme, removing references to mono
- Add missing Py.Gil lock
* Replace ICSharp with .NET Interactive
* Fixes after rebase
* CSharp research fixes
- Adding new Initialize.csx that pre loads all assemblies
- Adjusting template research file
- Moving steps in dockerfilejupyter
- Fix unit tests and regression tests after rebase
Co-authored-by: Gerardo Salazar <gsalaz9800@gmail.com>
Co-authored-by: Stefano Raggi <stefano.raggi67@gmail.com>
Co-authored-by: Jasper van Merle <jaspervmerle@gmail.com>
* Fixes 4815 by loading the requested assembly from different folder.
# Conflicts:
# ToolBox/Program.cs
* Upgrades System.Collections.Immutable to Version=1.2.5.0
* Creates a prototype for SSE streaming in IEXDataQueueHandler.
* Revert the changes in Tick.cs
* Implements a logic in IEXDataQueueHandler that updates the data-feed subscription after Subscribe/Unsubscribe
* Implements IEXCouldSubscribeMoreThan100Symbols - which fails and other small fixes.
* Implements DoForEach LinqExtensions
* Implements IEXEventSourceCollection that wraps all logic that is SSE-subscriptions and symbol-limits-per-connection concerned.
* Changes:
1) Fixes to address review.
2) Makes IexMarketPercent in QuoteSSE nullable as null values are assigned to in this field in data object received before the traing session start.
3) Deprecates helper Subscribe/Unsubscribe in IEXDataQueueHandler and IEXCouldSubscribe test.
* Fixes:
1) _refreshEvent.Reset() order was not correct - should be called before UpdateSubscription
2) ProcessJsonObject- leaves only the functionality to emit ticks.
3) IEXEventSourceCollection - replaces int counter with CountdownEvent to improve the logic - in particular, need a mechanism that would not allow the repeated call to continue until the first one is completed
* Refines the logic with parsing a data snapshot.
* Fixes few more bugs:
1) Logic in ProcessJsonObject
2) Logic in UpdateSubscription - need to introduce additional ManualResetEvent to implement the intended logic - otherwise the logic is not suitable for general case
* Introduce rate-gate limit in IEXEventSourceCollection:
because when subscribing to a bunch of shares (more than 200 for example)
the violation of rate gate policy may occur, which described in API docs asRequest Limits
IEX Cloud only applies request limits per IP address to ensure system stability.
We limit requests to 100 per second per IP measured in milliseconds, so no more than 1 request per 10 milliseconds.
SSE endpoints are limited to 50 symbols per connection. You can make multiple connections if you need to consume more than 50 symbols.:
* Few additional fixes done after real time testing
* Adds xml-docs in stream response object + renaming a file.
* Fixes:
1) Additional StreamResponseStocksUS parsing issues, that can happen outside of regular exchange hours.
2) Cancel clientUpdateThread by means of CancellationTokenSource
3) Replace BuildSymbolsQuery by string.Join
* Fixes:
1) Changing Log Trace -> Debug
2) Adds ConfigureAwait(false) to async method call
3) Removes direct reference to System.Net.Http
* Removes a task and manual reset event in IEXEventSourceCollection
* Additions:
1) IEXEventSourceCollectionSubscribes test
2) GetSnpStocksArray() helper method
3) Installs packages in QC.tests : HtmlAgilityPack & LaunchDarkly.EventSource
* IEX history provider fixes :
1) Tiny bug in ProcessJsonObject - use continue instead of return as execution is inside the for-each block)
2) Adds period variable for the historical data retrieved
3) Fixing from if (date.Date < start.Date || date.Date > end.Date) conditional check --> if (date < start || date > end) for more precise sorting.
* Changes:
1) Removes HtmlAgilityPack and SNP scraper
2) Uses hard coded symbols instead
* Bug fix:
- at certain hours (example: before pre-market open or on holidays) IEX may send no data on subscription - when trying to connect during those hours Message handler may not be fired - need to place the counter signal to client.Opened to be informed of successful connect.
* Implements:
1) IEXEventSourceCollectionSubscriptionThoroughTest and MockedIEXEventSourceCollection
2) Makes changes to IEXEventSourceCollection accordingly to allow the thorough testing.
* Fixes formatting issue in StreamResponseStocksUS
* Small fix for a new tests:
- Change RemovedClientSymbols to keep not clients itself, but symbols array, because clients are being disposed right further
* Enables extended logging in Toolbox.
* Fixing IEX historical data fetcher bugs:
1) Bug in IEXDataDownloader.cs - HistoryRequest not precisely correct.
2) Enables day-by-day daily bar downloading in IEXDataQueueHandler.
Motivation: Suppose we need data for some interval in the past - from-date=20170915-00:00:00 --to-date=20171103-00:00:00.
With current behavior IEX would have to download all the historical data from =20170915-00:00:00 up to this day.
3) Extends SynchronizingHistoryProvider
* Enables async fashion historical data download
* More fixes to IEXCouldGetHistory test.
* Reverts day-by-day daily bar downloading and other fixes.
* Removes needless packages & references
* Fix package reference
* To address review
* Sort out zero price ticks:
after testing on real-time algo 30 min before the market open now - IEX may send updates for many securities with zero lastPrice, lastSize - fix to sort such entries out
* Workaround for missing QuoteTicks timestamps:
Since we don't have a stamp for quote tick updates (only for trades) we calculate the average delay between trade tick's time stamp and local time, and
assuming that delay in average is the same for quote updates - just assign the local machine time adjusted for this average
* Simplifies the things.
* Changes:
1) Deprecates quote updates for IEX stocks.
2) Reduce the stream updates to reduce costs to ->
# Stock Quotes every 1 second (per symbol? )
# Can be up to 54,000 messages per symbol per day
https://iexcloud.io/docs/api/#how-messages-work
* Fixes:
1) IEXDataQueueHandler: give an error message on extended market hours or tick resolution subscription request. As they are not really well supported by IEX.
2) Few small fixes in IEXEventSourceCollection, including additional condition for when the subscription remains irrevocable.
* Binance Brokerage skeleton
* Market hours
* Implement Symbol Mapper
- known symbols available on /api/v1/exchangeInfo
- fiat currencies are pegged
* Implement GetCashBalance
* Implement GetAccountHoldings
- there are no pre-existing currency swaps
- cash balances are pulled and stored in the cashbook
* Implement GetOpenOrders
* Manage orders: PlaceOrder
* Manage orders: UpdateOrder
Update operation is not supported
* Manage orders: CancelOrder
* Messaging: order book
* Messaging: trades
* Messaging: combine streams
- connect to fake /ws/open channel on init
- case by channel name, but not event type
* Messaging: order depth updates
- ticker symbol is not enough as it pushes updates only once a second, this would be a very incomplete data stream
- fetch ticker snapshot if lastUpdateId == 0
- follow Binance instructions for keeping local orderbook fresh
* Messaging: user data streaming
- Request userDataStream endpoint to get listenKey
- keep listenkey alive
- handle order close event
- handle order fill event
* DataDownloader: get history
- we can aggregate minute candles for higher resolutions
* fix data stream
* Tests: FeeModel tests
* Tests: base brokerage tests
* Tests: download history
* Tests: symbol mapper
* Support StopLimit andd StopMarket orders
* StopMarket orders disabled
Take profit and Stop loss orders are not supported for any symbols (tested with BTCUSDT, ETHUSDT)
* Tests: StopLimit order
* Tests: crypto parsing
* Reissue user data listen key
* comment custom currency limitation
* rework websocket connections
* implement delayed subscription
* adapt ignore message
* add license banner
* use better suited exception type
* avoid message double parsing
* support custom fee values
* extract BinanceApiClient to manage the request/response between lean and binance
* use api events to invoke brokerage events
* do not allow to terminate session if it wasn't allocated.
* update binance exchange info
* tool to add or update binance exchange info
* ExchangeInfo basic test
* Rebase + Resharp
* Binance brokerage updates
- Fix sign bug in sell order fills
- Fix bug in GetHistory
- Remove duplicate symbol from symbol properties db
* Remove unused code
* Revert removal of account currency check
* Update symbols properties database
* Address review
* Address review
- Upgrade API endpoints from v1 to v3
- Updated sub/unsub for new subscription manager
- Subscribe best bid/ask quotes instead of full order book
- Added handling of websocket error messages
- Cleanup + refactor
* Update symbol properties database
* Remove list from symbol mapper
* Fix symbol mapper tests
* Address review
- Fix resubscribe after reconnect
- Fix quote tick edge case
* Fix EnsureCurrencyDataFeed for non-tradeable currencies
* Fix check in EnsureCurrencyDataFeed
* Reuse base class subscribe on reconnect
Co-authored-by: Adalyat Nazirov <aenazirov@gmail.com>
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Add PolygonDataQueueHandler
* Add history provider and downloader for Polygon
* Add aggregator to PolygonDataQueueHandler
* Address review
- Removed duplication in message classes
- Added public Subscribe/Unsubscribe methods in PolygonWebSocketClientWrapper
- Added history requests for Forex and Crypto
* Address review
- Add security type and market arguments to downloader
- Fix time zone bug in downloader
* Remove unnecessary locks
* Add Polygon history for all resolutions
- Equity: trades and quotes
- Forex: quotes only
- Crypto: trades only
* Adds support for live algorithms using TE calendar events
* Modifies TradingEconomics tickers to include country
* Fixes bug where `TECal.Clone(...)` could result in non-deterministic
results
* Fixes bugs in TradingEconomicsEventFilter
* Adds new CSV parsing methods in StreamReaderExtensions
* Moves ToolBox/TradingEconomicsCalendarTests to Common tests folder
* Updates TradingEconomicsCalendar tests to test CSV parsing
* TradingEconomicsCalendarDownloader now writes files to disk as JSON
* Make Benzinga News Converter more efficient (and cleaner!)
* Implement JSON converter for serialization/deserialization process
* Fix bug where duplicate symbols would be added to news.Symbols
* Modify algorithm strategies
Roman Yavnikov