355 Commits

Author SHA1 Message Date
chriscdev adb6afc4f1 Feature 6208 add nyseliffe exchange to interactive brokers (#6209)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Added NYSELIFFE exchange support to Interactive Brokers brokerage.

* Fixed unit test AddSecurityWithSymbol by adding market hours for Equity-nyseliffe-[*], Index-nyseliffe-[*], Option-nyseliffe-[*].

* Added NYSELIFFE exchange support to Interactive Brokers brokerage.

* Fixed unit test AddSecurityWithSymbol by adding market hours for Equity-nyseliffe-[*], Index-nyseliffe-[*], Option-nyseliffe-[*].

* Added NYSELIFFE exchange support to Interactive Brokers brokerage.

* Fixed unit test AddSecurityWithSymbol by adding market hours for Equity-nyseliffe-[*], Index-nyseliffe-[*], Option-nyseliffe-[*].

* - Added NYSELIFFE futures to Futures.cs
- Added method GetGoodFriday() to calculate Good Friday to FuturesExpiryUtilityFunctions.cs.
- Added unit tests for GetGoodFriday().
- Added NYSELIFFE futures expiries to FuturesExpiryFunctions.cs which uses GetGoodFriday() instead of hardcoding the date of Good Friday in the market-hours-database.json.
 - Added unit tests for NYSELIFFE futures expiry.
- Added NYSELIFFE to Exchange.cs
- Removed Equity-nyseliffe-[*] and Index-nyseliffe-[*] from market-hours-database.json
- Updated Option-nyseliffe-[*] to FutureOption-nyseliffe-[*] on market-hours-database.json
- Added the supported NYSELIFFE futureoptions to symbol-properties-database.csv.

* Removed blanks after India tuple

* Added the dates for 2022 and 2023 to the market-hours-database.json.
Removed "FutureOption-nyseliffe-[]" from market-hours-database.json.
Added to FuturesOptionsSymbolMappings.cs:
{ "YG", "OYG" },
{ "ZG", "OZG" },
{ "ZI", "OZI" }

* Minor tweaks for nyseliffe FOPs

Co-authored-by: Chris Coetzee <chris@polidata.ai>
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
2022-03-04 15:24:17 -03:00
Adalyat Nazirov 18b99338fc Implement Binance US exchange (#6222)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Binance.US base changes

* biniance exchange info update

* Binance US uses same fee rates as main Binance

* binance us brokerage model tests

* use base implementation of GetBuyingPowerModel method
2022-02-22 13:32:59 -03:00
Ronit Jain f129ab1a09 Feature implement ExchangeInfoDownloader (#6213)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* add gdax exchange info downloader

* add downloader method to program]

* fetch currency description

* change definition to include headers

* use extension method to make request

* remove log from test

* replace WebRequest

* cleanup

* use relevant name

* implement IExchangeInfoDownloader for bitfinex, initial commit

* add default values

* use default market value

* use correct attribute for lotsize

* don't skip missing values

* handle multiple downloaders

* add gdax and bitfinex exchange downloader

* follow LEAN data directory structure

* update SPDB

* order tickers

* order tickers

* add exchange info downloader test template

* delete files

* update SPDB

* use currency mapping

* update bitfinex symbols

* update currency mapping

* sort result after old currency symbols are used

* use market of the respective brokerage

* no more unknown symbol

* change minimum order size value

* direct conversion possible

* update bitfinex symbols

* change user-agent

* add test for indirect conversion

* update stats
2022-02-22 13:00:54 -03:00
Martin-Molinero e316f12394 Refactor MarginRequirementEntry. Update margins (#6210)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* Fix master

* Refactor MarginRequirementEntry to it's own file

* Update margin files

* Remove duplicate dates cases

* Update margins after fix
2022-02-15 20:24:28 -03:00
Adalyat Nazirov e8160f33d5 Binance extraction (#6193)
* extract binance brokerage

* remove binance tools from main toolbox app

* modify OrderTestParameters constructor

* fix BinanceBrokerage ref config.json

* update symbol properties

* move BinanceFeeModel tests to the right place
2022-02-15 12:05:47 -03:00
Ricardo Andrés Marino Rojas eb55311052 Feature 5157 micro futures update (#6190)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Update MHDB and SPDB

Update MHDB and SPDB with micro futures

* Fix bugs

* Add more FutureExpiryFunctions

* Requested changes

* Nit changes

* Nit changes

- Fix some future values in SPDB
- Test LastFriday() method
- Apparently Micro CHF/USD Futures(MSF) refers to the micro of Swiss Franc Futures (6S). The same happens with Micro JPY/USD (MJY) Futures and Japanese Yen Futures (6J)

* Fix bugs

* Micro futures MultipleFactor

* Fix some futures MultipleFactor
2022-02-10 17:45:07 -03:00
Ricardo Andrés Marino Rojas 74321d1727 Update CME future holidays and early closes in MHDB (#6181)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* CME Futures update in MHDB

* Fix indent spaces nit errors

* Revert "Fix indent spaces nit errors"

This reverts commit 6ebb8614c0c050258253bd5dbd5c34eaaa30b30b.

* Fix tabs errors

* Update MHDB with 2022 holidays

* Fix bugs

* Remove earlyOpens
2022-02-02 19:28:21 -03:00
Martin-Molinero c4a2d6eef4 Crypto base currency fees handled correctly (#6166)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Binance fees deducted from fill quantity accordengly

- For Binance cash accounts while buying, if fees are from the base
  currency of leans virtual position, we need to deduct the fee from the
  fill quantity, else we can end with a position bigger that it actually
  is and not be able to liquidate

* Refactor solution

- Refactor solution into a more generic approach solving fees in base
  currency at the BrokerageTransactionHandler level, covering all
  brokerages that require it. Adding regression algorithm reproducing
  issue.
- Update Bitfinex and Binance fee models to correctly reflact reality

* Log fill quantity adjusment once
2022-01-28 13:02:58 -03:00
Ronit Jain f34be8e3ff Feature add India index algorithms and data (#6145)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* add data

(cherry picked from commit 814011d89e5316d150f88ffca5f48d8d5f0ea7d9)

* update market hours for index

(cherry picked from commit edac40732c120eb84d27de00594b59eebb4983f5)

* add index algorithms

(cherry picked from commit b22d27b4fa98172c435f7c26de4a3a297c49a6b7)

* update statistics

* add cash

* Add india market

* add leverage for index

* can subscribe to index

* update format

* fix wrong cash

* fix ticker names

* update data

* update ticker and stats

* update docs
2022-01-19 12:14:48 -03:00
Adalyat Nazirov 2529ba124d FTX API endpoint is cinfigurable (#6026)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* specify enpoint url of ftx api (can be FTX pr FTX.US)

* more metadata for ftxus

* tidy up code

* tests

* more tests

* fix tests

* fix us fee rates

* add account tier

* update symbol props

* typo

* typo-2

* update symbol properties

* use FTXUS fee model

* minor tweaks
2022-01-12 13:51:18 -03:00
Ronit Jain 3837c32b36 Add India market local data and regression algorithm (#6088)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* Add india market data

* use local data for algo

* Add India market regression algo

* update india market data

* Update readme

* add python algo for BasicTemplateIndiaAlgorithm

* Add India data regression python algo

* data india data files

* update tickers

* Fix algo template

* remove data

* fix stats

* update stats

* remove unused data
2022-01-06 18:24:32 -03:00
Colton Sellers c11a09e08a Feature Index Option Low Resolution Support (#6075)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Activate Index and IndexOptions low res

* Add Index and IndexOption low res data

* Add IndexOption regressions

* Fix bug with IndexOption QuoteBar data not being scaled

* nit - update comment for _scaleFactor

* Update data and regressions to reflect fix in data

* Tweak regressions
2021-11-26 13:11:35 -03:00
Colton Sellers 1358bd8115 Future and FutureOption low res support (#6069)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Fixes for FutureOptions support in LeanData

* Add CreateCanonicalOption() utility function for Symbol.cs

* Add aggregated Futures/FuturesOptions data to Lean

* Add FutureOptions regressions for daily/hourly data

* Allow Futures to be added with low resolution

* Add Future regressions using hour/daily data

* Nit - Python Class names

* Add reviews

* Add alias into CreateCanonicalOption
2021-11-24 09:53:21 -03:00
Martin-Molinero 5436275901 Remove comments from MHDB json file (#6065)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
2021-11-18 20:41:02 -03:00
Martin-Molinero b8b0d18993 Continuous futures (#6034)
* Continuous Future Contracts

* Mapping approach

* Tweaks WIP

* Live mapping

* Live mapping

- Add support for live mapping, refreshing mapfiles
- Fix future expiration functions
- Adding unit tests

* Update moq test package

* Continuous futures price scaling

* Refactor price factors scaling

* Factor file related renames

* Address reviews
2021-11-15 14:44:30 -03:00
Colton Sellers dd4da7ba95 Feature Daily/Hourly Options Support (#6017)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* Create generic writing for LeanDataWriter, + notes on todos

* Make Options Daily/Hourly data store by year

* Refactor Generic Write

* Permit hour and daily resolutions for options

* Refactor writer to merge when needed with other files

* Cleanup redundancies, run write tasks in parallel

* Make needed classes/vars available

* Update tests to reflect new naming convention for daily hourly options data

* Add Byte[] overloads for ZipData functions in compression

* Implemented Store() for ZipDataCacheProvider

* Have LeanDataWriter use a DataCacheProvider

* ZipDataCacheProvider cleanup

* ZipDataCacheProvider tweaks, doesn't support storing non-zips

* Test adjustments

* Update LeanDataWriter to use Write instead of SaveDailyHourly/SaveMinuteSecond

* Implement tests to verify DownloadAndSave behavior

* Nit cleanup on DownloadAndSave tests

* Fix for options daily/hourly underlying equity subscription read

* Add daily/hourly options data and regressions

* Add missing open interest for hourly

* Fix writing of OpenInterest Daily/Hourly data

* Update data

* Fix Date typo in regression

* Use daily algorithm to test delisting

* Revisions part 1

* Expand test for DataCacheProviders; refactor DiskDataCacheProvider

* nit - test adjustments

* ZipDataCacheProvider test setup refactor

* Adjust multithreaded read/write test; fixes for ZipDataCacheProvider

* Move DiskDataCacheProvider to its own file and add write test

* Remove _appendToZips; always overwrite entry or create zip

* Add mapping regression for daily options

* nit - add license to regression

* Fix Tick write case where more than one data point for a DateTime

* Fix data issue

* Address review

* Tweaks for tests

* Stop Store() early if no entry name is given
2021-11-11 20:05:31 -03:00
Martin-Molinero b8768ae274 Add FOPs price magnifier (#5979)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* Add FOPs price magnifier

* Improve unit tests
2021-10-14 11:57:48 -03:00
Ricardo Andrés Marino Rojas db04b5e110 Feature 5930 price magnifier (#5977)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* Update SPDB with price magnifier parameter
- Add a new column for new parameter in symbol-properties-database.csv
- Change InteractiveBrokersBrokerage.cs implementation of GetContractPriceMagnifier() method in order to get the parameter directly from the SPDB
- Add test in SymbolPropertiesDatabaseTests.cs to check if the new parameter is loading properly

* Change GetContractPriceMagnifier() method name and implementation
- Change NormalizePriceToLean() and NormalizePriceToBrokerage() methods

* Nit change in GetSymbolPriceMagnifier() method

* Remove GetSymbolPriceMagnifier() method
- Remove redundant and unnecessary parameters in NormalizePriceToLean() and NormalizePriceToBrokerage()

* Requested changes and SPDB update
- Update SPDB with minimum order size parameter for futures in cents. The minimum order size parameter was extracted from the CME group API

* Address reviews. Adding unit test for failing case

Co-authored-by: Martin-Molinero <martin@quantconnect.com>
2021-10-14 11:16:23 -03:00
Adalyat Nazirov 3a591c3a72 Fix Binance ExchangeInfo downloader (#5965)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* fix exchangeinfo downloader

* normalize data, update symbol properties

* min ordersize should be in base currency

* check min order size using quote currency price

* check order size in quote currency

* improve checks

* improve code; more unit tests

* more description

* fix failing test
2021-10-11 12:36:41 -03:00
Adalyat Nazirov 7cfdd1243f FTX brokerage essentials (#5963)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* FTX config boilerplate

fixup

* required

* order props

* make orger props setter public

* add market props

* brokerage model

* deny modify order directly

* update markets

* unit tests

* fix market properties

* add comments

* return security benchmark

* fix fee calculation

* dont change access modifiers

* check brokerage CanSubmitOrder using min order quantity

* fix ticker values - use lean notation

* fix fee model

* fix error message
2021-10-08 19:16:38 -03:00
Martin-Molinero 659735946a Bitfinex fees cash accounts (#5957)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Bitfinex Fee adjustment

- Fix for bitfinex fee adjustment for cash account type only happen if
  the currencies are the expected ones, else log message.

* Adjust error handling to terminate algorithm

* Add bitfinex TESTBTCTESTUSD

* Minor improvements

- Add bitfinex test symbols to the SPDB
- Make SecurityDefinitionSymbolResolver log using trace
- Minor tweak for Bitfinex DQH lock contention
2021-10-04 18:15:32 -03:00
Ricardo Andrés Marino Rojas af958f0ce4 Bug 5904 minimum order size rejected order (#5949)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* - Update symbol-properties-database.csv, SymbolPropertiesDatabase.cs and SymbolProperties.cs with a new column for MinimumOrderSize parameter for Crypto type
- Change CanSubmitOrder() method implementation in GDAXBrokerageModel.cs to get the MinimumOrderSize from the security SymbolProperties directly
- Add CanSubmitOrder() method in Binance, Bitfinex and Kraken Brokerages models
- Add tests to check if CanSubmitOrder() method is working as expected in Binance, Bitfinex, Kraken and GDAX brokerages models
- Add tests to check if MinimumOrderSize parameter is loading correctly from symbol-properties-database.csv

* Revert "- Update symbol-properties-database.csv, SymbolPropertiesDatabase.cs and SymbolProperties.cs with a new column for MinimumOrderSize parameter for Crypto type"

This reverts commit e0fd66a5fa5215a6fccb98b64d4e3bf709007fdf.

* Revert "Revert "- Update symbol-properties-database.csv, SymbolPropertiesDatabase.cs and SymbolProperties.cs with a new column for MinimumOrderSize parameter for Crypto type""

This reverts commit b0f37935fc334ba8ff3b8ab09b7d979e4e731526.

* Requested changes in Crypto brokerage models
- Remove unnecessary commas in symbols-properties-database.csv
- Add TestHelpers class to reuse GetSecurity() method
- Add IsValidOrderSize() method in DefaultBrokerageModel class
- Remove unnecessary classes for brokerage models test helpers

* Nit change
- Remove unnecessary attribute from DefaultBrokerageModel.cs

* Nit change

* Nit changes and Regression test
- Add Bitfinex BTCUSD hourly data for 2 days
- Add regression test BuyBTCWithLessThanOneDollarAlgorithm.cs to test that the order size is taking into account when placing or updating an order
- Nit changes

* Change CanUpdateOrder() implementation
- CanUpdateOrder() now checks if the requested quantity value is valid

* Requested Changes
- Change regression algorithm BuyBTCWithLessThanOneDollar.cs name and implementation

* Nit changes
2021-09-30 18:30:04 -03:00
Martin-Molinero d3280c5e60 Move VX margin files (#5948)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
2021-09-28 14:21:07 -03:00
Stas Kotykhin 8026785116 Feature kraken brokerage setup (#5896)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Kraken Brokerage initial setup

* Fix naming in config

* Add license messages

* Add fees for stablecoins

* add kraken databases

* Review fixes, tests added

* Remove redundant field and using

* add CanUpdateOrder & summary

* Remove null

* Change to equal

* complete xml comments
2021-09-08 12:49:13 -03:00
Martin-Molinero 8eb023e854 Remove duplicate market hours segments (#5897)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
2021-09-02 19:09:51 -03:00
Stefano Raggi 1d904e716c Add new Market.CFE (#5391)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
As most brokers use separate exchange codes for CBOE and CFE, we added the new Market.CFE (previously Market.CBOE was used for both exchanges). This change also avoids hacky code in symbol mappers.
2021-08-31 19:34:14 -03:00
Martin-Molinero 3410832ccf Alpha holdings state (#5770)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Alpha holdings state

- Alpha result packet will optionally provide the algorithms portfolio
  state

* Rename

* Convert AlphaStreamsPortfolio to data source

* Improvements on AlphaStreams algorithm

* Fix regression tests

* Add unit tests for EW AS PCM and fixing bugs

* Protobuf AlphaStreamsPortfoliot staState

- Protobuf AlphaStreamsPortfolioState. Adding unit tests
- Add variable TPV tests for EW ASPCM

* Add alpha license to Organization response

* Improvements EW AS PCM respects free portfolio value

* Fixes

- Update tests expected statistics results affected by MHDB custom data timezone fix
- Fix for Extensions.IsCustomDataType

* Fixed and adding more regression tests

- Adding support and regression test with alpha consumer with different account currency
- Adding support and regression test of a universe adding custom data
  types
- Add support and regression test for algorithm alpha consumer with existing holdings

* Add AlphaStreamsOrderEvent data type
2021-08-19 13:56:18 -03:00
Gerardo Salazar cd9e451ead Adds Support for ETF Constituent Universes (#5862)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* Adds support for ETF constituent universes

  * Adds filtering for universe data if it doesn't match the
    universe subscription type

  * Update mapping for ALL underlying Symbols if
    `Symbol.UpdateMappedSymbol(...)` is called. Required to support
    constituent ETF universes that might have mapping events

  * Delistings of composite constituent universe Symbol will result in
    removal of universe securities.

  * Added regression algorithms for ETF constituent mappings (C#/Python),
    along with data required to run locally

  * Refactor universe delistings in SubscriptionSynchronizer -
    big thank you to @Martin-Molinero :)

* Address review: update regression algorithms and add explanatory comments

* Address review: add additional checks to delisting regression algorithms

  * Adds new regression algorithm testing the addition of a universe
    without calling AddEquity() and asserts same behavior
2021-08-17 19:54:29 -03:00
Martin-Molinero e92230a7fe Fix SubscriptionDataReader auxiliary data out of order (#5843)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* Fix SubscriptionDataReader auxiliary data out of order

- Fix for SubscriptionDataReader emitting auxiliary data out of order
  due to sparse that and enumerator refresh logic. Adding regression
  algorithm
- Minor tweaks for unit tests failing on and off

* Fix for SDR internal subscriptions

- Fix for SubscriptionDataReader enumerator refresh for internal
  subscriptions which were ignored. Adding regression test
- Updating custom data regression algorithms affected by the issue
	- DropboxBaseDataUniverse was emitting a custom data point being end time
	- UnlinkedTraderBarIconicType was emitting a single data point
	  of the underlying SPY minute data when if should of emitted
	  all data points
2021-08-10 18:57:23 -03:00
IlshatGaripov 190fe012f1 Perform update (#5841)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
2021-08-10 11:18:23 -03:00
Stefano Raggi 0c4e577885 BitfinexBrokerage updates (#5787)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Update BinanceBrokerage to handle more than 512 symbols

* Address review

- fetch symbol weights only if required
- remove code duplication

* Add rate limiting for new connections

* Update WebSocketMessage to include the websocket instance

* Handle resubscriptions on reconnect

* Address review

* Address review

* Remove unnecessary locking

* WebSocketClientWrapper updates

- remove allocation of receive buffer on each message
- add missing lock in Close method
- log message data when message type is Close
- fix race condition after unexpected websocket close

* Set WebSocketClientWrapper task to LongRunning

* Add missing check in GetHistory

* Fix exceptions with Binance downloader

- closes #5794

* Update Bitfinex symbols in symbol properties database

* Update BitfinexBrokerage to use BrokerageMultiWebSocketSubscriptionManager

* Address review

* Remove unnecessary locking

* Remove old channels on resubscription
2021-07-28 11:10:12 -03:00
Ishant Jain 808ba17bff Indian Market Holidays & Early Open/Late Close Modified (#5792)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
2021-07-26 11:14:22 -03:00
Stefano Raggi e453635458 Update Binance symbols in symbol properties database (#5759)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* Update Binance symbols in symbol properties database

* Fix failing unit tests
2021-07-12 13:28:28 -03:00
Ronit Jain c8d13080a8 Indian Market support for backtesting and live deploys through Zerodha (#5710)
* added symbolData class to store exchange and token values of tickers

* support for India Market

(cherry picked from commit 5f629dcb9e72426dc7b560e782cb1fae72861574)

* removed local vscode workspace file

* support for india market instead of nse and other exchanges

* changed hardcoded india values to Market.India

* changed zerodha getcash fromm available to net

* addded edge case handoing with expections

* support for market info (nse, bse) for india market

* exchange info for EmitQuoteTick

* basic testing with c#

* c# algo

* c# testing

* live orders during market open success

* support orderproperties for exchange value for order placement

* added exceptional handling cases

* fix xml descriptions for the files

* fix zerodha-product-type in DefaultOrderProperties

* add basic template for india market

* fix orderProperites for default values

* fix for local data, python implementation not available

* fix access modifiers of SymbolData

* fix variable name case

* fix single check for exchange, call to base

* fix styling, redundent code and access modifers

* fix non static not callable from static

* fix static
2021-07-02 11:06:47 -03:00
Gerardo Salazar c211d04798 Adjust AlgoSeek Futures price multipliers to quote/process prices in USD (#5725)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Adjust AlgoSeek Futures price multipliers to quote prices in USD

* Revert changes to AUP scaling

  * After a second review, seems that this contract isn't scaled.
    The contract multiplier differs from the NYMEX Aluminum Index's
    contract multiplier, and so does the minimum price variation.
    The minimum price fluctuation seems consistent with CME's
    quoted prices, which also claims the contract is quoted in USD.

* Fix futures contract multipliers for some scaled contracts
2021-07-01 18:46:36 -03:00
Alexandre Catarino 5361f87dd1 Updates Equity Market Data (#5576)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Updates Equity Market Data

* Updates Unit Tests

* Updates Regression Tests

In this commit we include regression tests with small changes (slightly different CAGR, Alpha, etc, but same number of trades) due to the data update.

* Updates Regression Tests 2

The following regression tests were adapt because of verification of hard-coded market data values:
- `AdjustedVolumeRegressionAlgorithm`
- `HistoryWithSymbolChangesRegressionAlgorithm`
- `OptionRenameRegressionAlgorithm`
- `RawDataRegressionAlgorithm`
- `SwitchDataModeRegressionAlgorithm`

The following regression tests have more trades since adjusted prices allowed more 1-2 shares trades that were rounded down to zero before:
- `AddUniverseSelectionModelCoarseAlgorithm` 23 -> 35
- `MeanVarianceOptimizationFrameworkAlgorithm` 12 -> 14
- `PortfolioRebalanceOnDateRulesRegressionAlgorithm` 298 -> 324
- `PortfolioRebalanceOnInsightChangesRegressionAlgorithm` 83 -> 86
- `ScheduledUniverseSelectionModelRegressionAlgorithm` 86 -> 90
- `SectorExposureRiskFrameworkAlgorithm` 17 -> 22
- `SetHoldingsMultipleTargetsRegressionAlgorithm` 8 -> 9
- `StandardDeviationExecutionModelRegressionAlgorithm` 196 -> 199
- `UniverseUnchangedRegressionAlgorithm` 11 -> 17
- `VolumeWeightedAveragePriceExecutionModelRegressionAlgorithm` 237 -> 238

Especial cases:
- `BlackLittermanPortfolioOptimizationFrameworkAlgorithm` 18 -> 17
 - BLM model sensibility
- `OptionChainedAndUniverseSelectionRegressionAlgorithm`

The following regression tests have different Capacity because of different volume from lowest capacity asset, except:
- `OptionEquityCoveredCallRegressionAlgorithm` New lowest capacity asset is underlying
- `OptionEquityCoveredPutRegressionAlgorithm` New lowest capacity asset is underlying

* Revert File Update for SPWR and SPWRA

* Fix Regression Tests

Temporarily removes python regression test for `MeanVarianceOptimizationFrameworkAlgorithm` as the `MeanVarianceOptimizationPortfolioConstructionModel` for each version are yeilding different results. If we use C# version in `MeanVarianceOptimizationPortfolioConstructionModel.py`, the results match.

* Changes Optimization Method in MinimumVariancePortfolioOptimizer [Py]

Uses `trust-constr`  method.
See https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html
2021-06-17 14:04:51 -03:00
Colton Sellers dcdfc7fdb3 Update Oanda CFDs Lot Sizes and Market Hours (#5581)
Build & Test Lean / build (push) Has been cancelled
* Update lot sizes on symbols

* Update some incorrect marketHours

* Update all Oanda CFD market hours

* unify rollover to next day endtime
2021-05-21 16:16:26 -03:00
Colton Sellers a83eb7bf88 Update GDAX Brokerage MinimumOrderSizes (#5586)
Build & Test Lean / build (push) Has been cancelled
* Update GDAX Brokerage MinimumOrderSizes

* Re-add XRP for historical purposes
2021-05-21 15:44:16 -03:00
Łukasz Szymański 9d93dc1570 Fix wrong market hours for Oanda DE30EUR CFD (#5564)
Build & Test Lean / build (push) Has been cancelled
* Fix wrong market hours for Oanda DE30EUR CFD

* Update unit tests expected result

Co-authored-by: Martin-Molinero <martin@quantconnect.com>
2021-05-17 18:32:58 -03:00
Alexandre Catarino 96f42eb591 Updates Fundamental Data (#5502)
Build & Test Lean / build (push) Has been cancelled
* Updates Fundamental Data

* Updates Regression Algorithms

- `CoarseFundamentalTop3Algorithm`
  - New coarse has higher `DollarVolume` for `FB`
- `CoarseNoLookAheadBiasAlgorithm`
  - Change from `SPY` update not included in #5493
- `SectorExposureRiskFrameworkAlgorithm`
  - `HasFundamentalData` was false for `GOOG` on 20140401 and 20140402.

* Updates Unit Tests

Minor changes in expected values

* Updates Regression Tests

Should have been included in #5493:
- `OptionsExpiredContractRegression`
- `FuturesExpiredContractRegression`

* Adjusted Quantity By Lot Size in OrderTargetsByMarginImpact

`OrderTargetsByMarginImpact` didn't calculate the order value with the quantity adjusted by order size which less to values that did not reflect the actual order value.
It has a particular affect in `SectorExposureRiskFrameworkAlgorithm` where the Python and C# versions have the same orders but placed in a different sequence because of decimal/double precision.

* Reduce dictionary access x3 on OrderTargetsByMarginImpact

Co-authored-by: Martin-Molinero <martin@quantconnect.com>
2021-05-11 17:54:45 -03:00
Colton Sellers c082f0dda3 Address Missing StableCoin Pairs in Crypto Exchanges (#5488)
* Allow USDC in cashbook without USDC-USD pair

* Cover more stablecoin cases unique to our crypto brokers

* Add unit test

* Cleanup and expand test cases

* Add USDCEUR and USDCGBP to GDAX Symbols

* Add missing tickers to SPDB

* Cleanup test
2021-04-28 14:20:15 -03:00
Alexandre Catarino 2b0fd2e607 Updates SPY Market Data (#5493)
* Fixes Double to Decimal Cast in GetAnnualPerformance

`GetAnnualPerformance` raises an exception if the `AnnualPerformance` calculation returns a double that cannot be cast to decimal (smaller than `decimal.MinValue` or bigger than `decimal.MaxValue`).
See `ProbabilisticSharpeRatio` where the same solution was applied.

* Updates SPY Market Data

SPY is a key asset since it is the default benchmark, and any change can lead to different `Alpha` and `Beta`

* Updates Unit Tests to Reflect Data Update

* Updates Regression Tests to Reflect Data Update I

Most of the regression tests change because of updated data (market and factors) of SPY (default benchmark) while the total trade remain the same.

* Updates Regression Tests to Reflect Data Update II

The following regression tests were changed to adapt to adjusted prices and keep the total trades:
- `BacktestingBrokerageRegressionAlgorithm`
- `LimitIfTouchedRegressionAlgorithm`
- `PortfolioRebalanceOnCustomFuncRegressionAlgorithm`
- `SetAccountCurrencySecurityMarginModelRegressionAlgorithm`
- `StopLossOnOrderEventRegressionAlgorithm`
- `TimeInForceAlgorithm`

The following regression tests have more trades since adjusted prices allowed more 1-2 shares trades that were rounded down to zero before:
- `FreePortfolioValueRegressionAlgorithm` 2 -> 3
- `PortfolioRebalanceOnDateRulesRegressionAlgorithm` 291 -> 298
- `TrailingStopRiskFrameworkAlgorithm` 5 -> 7

Especial cases:
- `AutoRegressiveIntegratedMovingAverageRegressionAlgorithm` 65 -> 52
 - ARIMA model sensibility
- `BlackLittermanPortfolioOptimizationFrameworkAlgorithm` 18 -> 19
 - BLM model sensibility
- `ExtendedMarketHoursHistoryRegressionAlgorithm` 20 -> 18
 - Less minute bars before market opens

* Addresses Peer-Review

Fix `BacktestingBrokerageRegressionAlgorithm` to use `CalculateOrderQuantity` and round down `quantity` to an even number to pass a value assertion and update the expected value from 50 to 52.
The quantity calculated by `CalculateOrderQuantity` has changed from 50 to 53 because of factor file update.
2021-04-19 13:31:01 -03:00
Jared 7625e232f4 Update readme.md 2021-03-19 14:53:46 -07:00
Gerardo Salazar 4c085ff853 Adds Indexes and Index Options asset types (Backtesting/Live, IB only) (#5379)
* Add support for Index SecurityType  🚀 (#5364)

* Add Index SecurityType  🚀

* Extend SecurityIdentifier & Lean Data classes with Index support

* Add Index SecurityType  🚀

* Extend SecurityIdentifier & Lean Data classes with Index support

* Fixes

* Added index cross basic template demonstration

* WIP: Prototype index security type for LEAN as non tradable asset

* Re-adds Index entries to MHDB after rebase

* First steps to getting Index Options running

  * Looks at any instance where we pattern match for an option type
    and replaces it with a generic call to `.IsOption()` for easier
    extensibility in the future for additional option security types

  * Adds IndexOption security and misc. classes

  * Misc. changes, mainly related to any sort of special casing of
    equity options and made index options take the same path

* Enables index options data for backtesting

  * Adds new index options market hours to MHDB
  * Misc. bug fixes for index options
  * WIP: add live support for index options and indexes
  * Use OptionMarginModel for Index Options because they both use the
    same calculation for margin requirements

* Fixes contract not found errors on SPX index options and SPX index in IB

  * Turns out index options' last trading day is the day before expiry,
    which IB was expecting the last trading day.

* Add index option test cases (temp)

* LiveOptionChainProvider fix, use Symbol vs. ticker

  * Description updates to regression algorithms

* Fixes bug in live trading for indexes and index options

  * Adds overridable minimum price variation symbol property
  * Adds variable sized minimum price variation for index options
  * Adjusts symbol properties for index options
  * Misc. bug fixes

* Fixes option assignment simulation for European options

  * Updates index options regression algorithms (WIP)

* Fixes bug where index option exercise would trade index underlying

  * Fixes bugs where SecurityType.Index was getting flagged as tradable

* Regression algorithms updates and addresses review

  * Misc. style fixes and refactoring + a few bug fixes
  * Updates regression algorithms to run without runtime errors
  * Adds data for regression algos

* Sets DefaultOptionStyle on Canonical and support index options

* Update regression algos statistics

* Removes bad line in regression algorithm causing build to fail

* Minor tweaks

* Address review add comment about quoteBar parse scale

Co-authored-by: Balamurali Pandranki <balamurali@live.com>
Co-authored-by: Jared Broad <jaredbroad@gmail.com>
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
2021-03-12 20:46:23 -03:00
Balamurali Pandranki c22a538bad Indian stock markets support & Zerodha brokerage implementation (#4873)
* Indian stock markets & Samco and Zerodha brokerage implementations

* Build fixes & Implement multi leg orders (BracketOrder & CoverOrder)

* Build fixes

* Clean & refactor

* Clean up & remove samco brokerage

* Clean up & remove samco brokerage

* Fix Nifty, BankNifty & Sensex Index futures expiry time functions

* Fix Nifty, BankNifty & Sensex Index futures expiry time functions

* Fix Futures Expiry Testcases

* Fix Futures Expiry Testcases

* Refactor Zerodha Symbol Mapper

* Refactor Zerodha Symbol Mapper

* Add Future symbols to symbol prop db csv

* Fix Symbol Mapper context

* Fix Market Hours Database

* Fix OrderJsonConverter

* Add Zerodha AccountBaseCurrency

* Add QuantConnect License headers to new source files

* cleanup config.json formatting & tick aggregator implementation

* cleanup config.json formatting & tick aggregator implementation

* Refactor ZerodhaBrokerageModel

* Fix

* Build Fixes

* Refactor

* Refactor Brokerage class & remove TextFieldParser

* Refactor Brokerage FeeModel

* Add ZerodhaOrderProperties

* Add Refactor HistoryProvider

* Refactor CanExecuteOrder

* Refactor SymbolMapper

* Refactor market names

* Refactor & remove Zerodha subscription manager

* Refactor & remove ZerodhaWebSocketChannels

* Refactor & remove ZerodhaWebSocketChannels

* Refactor ZerodhaBrokerage

* Refactor symbol prop db

* zerodha update downloader ticker data using symbol name

* 1 Zerodha mapper class update to get instruments based on Market. 2 Zerodha Getholdings data fetch fix.

* Update market-hours-database.json

* Implement basic testcases for ZerodhaBrokerage

* Update market-hours-database.json

* Update Market.cs

* 1 Fix for Zerodha subscriber and unsubscribe 2 User of CSVHelper to read instrument list

* Rename Zerodha DataQueueHandler class implementation

* Changes related to TradeBar and emit tick

* Fix DataQueueUniverseProvider & Handle Timezone conversion in WS EmitQuotes

* Emit Order Fixes

* DataQueueHandler refactoring and build fix

* Update config.json

* Removal of IHistoryProvider impplementation

* Reverting timezone logic as already taken care by BrokerImplementation using Unix time

* Place, modify and cancel order implementation

* fix zerodha test cases

* Updating get quotes and restricting getHoldings to MIS

* Testcase Fixes

* Testcase Fixes

* Testcase Fixes

* Updating emitFillOrder

* Add ProductType to ZerodhaOrderProperties

* Addition of ZerodhaProduct Type property and test update

* Fix for unit tests and minor changes for place and update order

* Unit test fix for Zerodha

* Addition of the product type and trading segment configs

* PR review fixes

* Addition of trading segment and product type configs

* Nuget Fixes

* Fix ZerodhaBrokerage DataQueueHandler

* Cleanup OrderTypes & rm global.json

* Update UpdateOrderRequest.cs

* Removal of custom csvreader

* Implementing additional method CanPerformSelection

* Add LiveOptionChainProvider

* Clean up and add QC license headers

* Quick cleanup

* Update ZerodhaBrokerage.cs

* Fix OrderEvent timezone conversions

* use item.Unrealised for Intraday position holdings

* Refactor Option chain get instruments calls

* Refactor Option chain get instruments calls

* Review fixes

* Remove whitespaces

* Updating options strike price

* Undo time stamp change

* Optional gethistory

* Addition of comments and minor changes

* Build Fixes

* Add comments for LogType enums

* Options fix

* Fix json name in CsvInstrument

* Update tick generation in DataQueueHandler

- Use Timestamp field for both trades and quotes
- Fix incorrect bid/ask sizes
- Avoid reading depth on each tick to find top quote
- Use new Tick constructors
- Remove unnecessary locking

* Add missing null check in Utils.StringToDate

* Order fixes

- Include TriggerPending orders in GetOpenOrders
- Remove unnecessary invalid check in PlaceOrder
- Fix GetOrderPrice and GetOrderTriggerPrice

* Zerodha account balance fetch

* Removal of Futures and Options

* Remove Futures & Options Support

* Update ZerodhaBrokerage.cs

* Update ZerodhaBrokerage.cs

* Add License headers

* Add License Headers

* Refactor

* historical TradeBar & security fee calculation fixes

* fix brokerage test case

* Update ZerodhaBrokerage.cs

* Update ZerodhaBrokerageHistoryProviderTests.cs

Co-authored-by: Aman Ray <ray.aman9876@gmail.com>
Co-authored-by: Stefano Raggi <stefano.raggi67@gmail.com>
2021-03-08 18:51:54 -03:00
Martin-Molinero 8cbdfcaf28 Fix hour resolution mapping data handling (#5233)
- In some cases hour resolution data subscription would end early
  because of remapping not being correctly handled. Adding regression
  test
2021-01-28 20:41:04 -03:00
Gerardo Salazar 5f73a05e04 Set default market hours for futures exchanges to 16:00 Central Time (#5226) 2021-01-27 20:00:19 -03:00
Derek Melchin 97deb4bbc6 Fix typo (#5216) 2021-01-25 10:24:15 -03:00
Juan José D'Ambrosio 765f2cab3b Add primary exchage to map files (#5194)
* Add main exchange to map file

minimum minimorum test

* Add MainExchange to MapFilRow IEquatable implementation
Also write CSV line correctly

* Normalize exchanges

Also add small test form MapFileRow

* Exchange name normalization

- Exchange name normalization.
- Tick will protobuf `ExchangeCode` which will set existing `Exchange`

* Include single character exchange code

* Encode exchange as byte, Include exchange name as Enum

* Update MapFileRow.cs to accept a byte instead a string for primary exchange

* Fix issues in encoding exchanges map

* Fix broken MapFile tests

* Rename Main Exchange property in MapFileRow

* Use BATS instead of BATSZ

* Add primary exchange from map file

* Move MapFilePrimaryExchangeProvider to its own file

* Implement Char for primary exchange in map files

* Fix broken tests after implementing char exchange into map file

* Add a new constructor for MapFileRow

it  accepts a PrimaryExxchange object as parameter

* Fix broken test 

WIP

* Address review 

WIP

* Handles unexpected encoded exchanges

* Keep addressing review

* Address review

* Fix broken tests

Make PrimaryExchange as default even if the IPrimaryExchangeProvider is not instantiated

* Revert breaking change.

* Add a check that resolved map file actually have content

The issue that leads to this change was caused by a map file resolved to APC in the CoarseFundamentalTop3Algorithm. 
The resolver returns a not-null MapFile, but it is empty.
To my best understand, the MapFile Resolver should return only existing map files in the map_files folder. But the LocalMapFileProvider returned a map file with a permtick that doesn't exist in the file system as CSV.

* Set PrimaryExchange.UNKNOWN as default

* Address review couple minor bugs

- Use SID market instead of USA
- Default ticke exchange is UNKNOWN

* Update map files with latest format and latest info

This include:
 - VXX, it was delisted, then VXXB appears and later it chnages from VXXB -> VXX
 - AAA delisting and a new listing with ticker AAA
 - FOXA ticker changue to TFCFA and later it was delisted

* Update test after updating latest map files

* Fix bug with factor files after updating map files

* Fix bug with factor files after updating map files

and revert statistics in regression tests

* Update OrderHash in regression test

Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
2021-01-21 18:46:12 -03:00
Karthik Kailash ce42c888ce Add 2001-12-24 to US equities early closes in market-hours-database.json (#5184) 2021-01-19 13:21:00 -08:00