10696 Commits

Author SHA1 Message Date
Martin-Molinero 5df29f16dd Fix thread safety of Mapfile exchange provider (#5208) 10429 2021-01-21 19:02:50 -08:00
Martin-Molinero 56a0d52212 Allow limiting streaming security Types per job packet (#5206) 10426 2021-01-21 21:23:32 -03:00
Juan José D'Ambrosio 765f2cab3b Add primary exchage to map files (#5194)
* Add main exchange to map file

minimum minimorum test

* Add MainExchange to MapFilRow IEquatable implementation
Also write CSV line correctly

* Normalize exchanges

Also add small test form MapFileRow

* Exchange name normalization

- Exchange name normalization.
- Tick will protobuf `ExchangeCode` which will set existing `Exchange`

* Include single character exchange code

* Encode exchange as byte, Include exchange name as Enum

* Update MapFileRow.cs to accept a byte instead a string for primary exchange

* Fix issues in encoding exchanges map

* Fix broken MapFile tests

* Rename Main Exchange property in MapFileRow

* Use BATS instead of BATSZ

* Add primary exchange from map file

* Move MapFilePrimaryExchangeProvider to its own file

* Implement Char for primary exchange in map files

* Fix broken tests after implementing char exchange into map file

* Add a new constructor for MapFileRow

it  accepts a PrimaryExxchange object as parameter

* Fix broken test 

WIP

* Address review 

WIP

* Handles unexpected encoded exchanges

* Keep addressing review

* Address review

* Fix broken tests

Make PrimaryExchange as default even if the IPrimaryExchangeProvider is not instantiated

* Revert breaking change.

* Add a check that resolved map file actually have content

The issue that leads to this change was caused by a map file resolved to APC in the CoarseFundamentalTop3Algorithm. 
The resolver returns a not-null MapFile, but it is empty.
To my best understand, the MapFile Resolver should return only existing map files in the map_files folder. But the LocalMapFileProvider returned a map file with a permtick that doesn't exist in the file system as CSV.

* Set PrimaryExchange.UNKNOWN as default

* Address review couple minor bugs

- Use SID market instead of USA
- Default ticke exchange is UNKNOWN

* Update map files with latest format and latest info

This include:
 - VXX, it was delisted, then VXXB appears and later it chnages from VXXB -> VXX
 - AAA delisting and a new listing with ticker AAA
 - FOXA ticker changue to TFCFA and later it was delisted

* Update test after updating latest map files

* Fix bug with factor files after updating map files

* Fix bug with factor files after updating map files

and revert statistics in regression tests

* Update OrderHash in regression test

Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
10420
2021-01-21 18:46:12 -03:00
Martin-Molinero f175ac6c6b Fix universe selection symbol cache removal (#5193)
- Only remove symbol from cache during universe selection if there is no
  other universe using it. Adding regression test reproducing issue
10414
2021-01-21 15:48:33 -03:00
Gerardo Salazar 79c544d3a2 Reduces output log size and warnings emitted when building with mono's msbuild (#5200) 10409 2021-01-21 11:09:16 -03:00
Aaron Janeiro Stone 5dcbed535d 5052 - Adds missing constructor routing (#5201)
* Adds missing constructor routing

* Minor tweak remove unrequired lines and format

Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
10408
2021-01-21 10:49:27 -03:00
Gerardo Salazar 35393b66a3 Updates report generator template.crisis.html (#5202) 10407 2021-01-20 18:01:44 -08:00
Karthik Kailash ce42c888ce Add 2001-12-24 to US equities early closes in market-hours-database.json (#5184) 10383 2021-01-19 13:21:00 -08:00
Colton Sellers f2fd10d9d2 Build Configuration Hotfix (#5186)
* Reflect properties prior to migration onto projects

* Do not prefer 32bit

* Reinstate SelectedOptimization var
10372
2021-01-19 09:38:16 -03:00
Colton Sellers 94d766ff89 Refactor Default Benchmark (#5158)
* Have BrokerageModel determine default benchmark

* Add DefaultBenchmark to Python wrapper

* Handle Null benchmark case

* Add NullBenchmarkRegressionAlgorithm

* Refactor solution to have BrokerageModel return IBenchmark; also refactor QCAlgorithm benchmark handling

* Always create a new security for benchmark

* Drop security overload, Always create a new security for benchmark

* Check our securities for a symbol matching the ticker before creating a new one

* No Python version of this regression

* Address review

* Create shared SecurityBenchmark creator function

* Add Python regression and needed FuncBenchmark constructor
10371
2021-01-19 08:40:41 -03:00
Martin-Molinero 592d037085 Fix delisted liquidation orders being cancelled (#5169)
* Fix delisted liquidation orders being cancelled

- Place delisted liquidation orders 10 min before market closes of 10
  min before the end of the delisting warning date. Adding regression
  test and unit tests. Updating existing tests.

* Fix failing python option unit tests

* Fix bug where positions would be open delisting liquidation

* Fix universe selection and delisting

- Delisting will happen ASAP for all types. Giving priority to close
  positions on derivates first
- Fix bug in universe selection where OptionChain would remove
  underlying even if holding a position in derivate.
- Updating regression tests statistics

* Add unit test, fix unit test expected stats
10370
2021-01-18 22:02:40 -03:00
Gerardo Salazar 17ab10531a Fixes ETB/shortable regression algorithms that were previously failing (#5174)
* Fixes ETB/shortable regression algorithms that were previously failing

* Addresses review and fixes bug

  * Fixes bug where orders would be denied regardless of direction
    whenever they exceeded the absolute value of the shortable quantity

  * Updates regression algorithm + statistics + simplifies test

  * Fixes python regression algorithm

* Addresses review: removes comments and unneeded imports
10369
2021-01-18 21:19:42 -03:00
Juan José D'Ambrosio 9d576a4121 Update regression tests statistics (#5187)
Those statistic were affected for the new early market close entries in MHBD.
10368
2021-01-18 21:06:26 -03:00
Aaron Janeiro Stone 7aefe07274 4553 - Refactor RenkoConsolidator (#5052)
* Separated Update calls and other non-common RenkoConsolidator.cs artifacts.

* Added type safe init for wicked

* Refactor; moved wicked-specific fields to WickedRenkoConsolidator.cs

* Addresses requested changes (and compacts constructors)

* Separated Update calls and other non-common RenkoConsolidator.cs artifacts.

* Added type safe init for wicked

* Refactor; moved wicked-specific fields to WickedRenkoConsolidator.cs

* Addresses requested changes (and compacts constructors)

* Removes base class for brevity

* Delete BaseRenkoConsolidator.cs

* Fixes naming for classic Renko consolidator

* Addresses review concerns

* Fixes missing null checks and missing typesafe constructor

* Adds warning

* WickedRenkoConsolidator -> RenkoConsolidator calls

* Separated Update calls and other non-common RenkoConsolidator.cs artifacts.

* Added type safe init for wicked

* Refactor; moved wicked-specific fields to WickedRenkoConsolidator.cs

* Addresses requested changes (and compacts constructors)

* Removes base class for brevity

* Fixes naming for classic Renko consolidator

* Separated Update calls and other non-common RenkoConsolidator.cs artifacts.

* Added type safe init for wicked

* Refactor; moved wicked-specific fields to WickedRenkoConsolidator.cs

* Addresses requested changes (and compacts constructors)

* Delete BaseRenkoConsolidator.cs

* Addresses review concerns

* Fixes missing null checks and missing typesafe constructor

* Adds warning

* WickedRenkoConsolidator -> RenkoConsolidator calls

* Addresses review

* unneeded tag removed

* Minor tweaks address reviews

Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
10364
2021-01-18 15:54:15 -03:00
Karthik Kailash 6de9a146a9 Issue 5149: Added missing early closes for 1999-2008 (#5163)
* Added missing early closes for 1999-2008

See GitHub issue for details

* Fix expected Treynor ratio for backtest in test suite

This backtest runs from 2008-10-10 to 2010-10-10 so it is affected by the early close date now recorded on 2008-12-24.
10348
2021-01-15 11:22:44 -03:00
Gerardo Salazar 4d4ad92fa7 Easy To Borrow Hotfix (#5167) 10344 2021-01-14 15:07:54 -08:00
Jared 345fe7357e Update readme.md 2021-01-14 13:58:26 -08:00
Gerardo Salazar 3e7af17e02 Implements Easy To Borrow Reality Modeling (WIP) (#5159)
* Creates IShortableProvider interface + interface impls

  * This is the foundational work for the addition of the shortable stocks
    feature for backtesting and live trading. Note that the QCAlgorithm
    API and the backend transaction handling will be implemented
    separately.

* temp; work on preorder checks

* improve checks

* tmep

* Enforces ETB checks at BrokerageTransactionHandler

  * Adds ETB Shortable Provider to QCAlgorithm and IAlgorithm
  * Removes ETB check from PreOrderChecksImpl
  * Removes outdated test, new test to come soon for relevant class

* Work in progress commit, pushing for review.

  * Adds new regression algorithm (WIP)
  * Enhances ShortableProviderOrdersRejectedRegressionAlgorithm
  * Adds new methods to QCAlgorithm
  * Move IShortableProvider to have BrokerageModel own it
  * Comments updates
  * Adds new properties to Equity Security for shortable

  * Fixes bug where retrieving open order quantities would aggregate the
    submitted order quantity rather than the remaining order quantity for
    open orders.

  * Fixes bug where quantity of zero would result in a
    false positive in QCAlgorithm.Shortable(...)

  * Code refactoring and comments updates

Co-authored-by: Jared <jaredbroad@gmail.com>
2021-01-14 13:33:44 -08:00
Colton Sellers a2a874e333 CSProj Update (#5145)
* Migrate to newer CSProj format; build succesful

Rebase onto master

* Maintain binaries directory

* Reinstate FSharp default condition

* Remove reference to packages.config

* Test project fixes

* Remove nuget.config and "Solution Items"

* Remove repetitive assembly info

* Reinstate default compiles and cleanup redundant settings

* Reinstate any specific settings that defaults are unclear for

* Reinstate Framework.nuspec

* Make test internal classes public for composer use; undo assembly change

Rebase onto master part 2

* Address review; restore some settings

* Restore RootNamespace and AssemblyName attributes

* Use NugetPackageRoot instead of NugetPackageFolders

* Reinstate "RestorePackages" setting

* Transfer compilied Python files to launcher and tests bin

* Remove conflicting settings

* Fix WebSocketSharpFork references

* Fix FSharp Core reference

* Upgrade TestPlatform.ObjectModel to 16.8.3, and redirect V11 binding

* Add Mono.Cecil package

* Address review
2021-01-14 12:55:09 -08:00
Juan José D'Ambrosio 96592b3387 Update RTY price multiplier (#5166) 10339 2021-01-14 16:45:24 -03:00
Colton Sellers 7e3741d983 Allow Custom Security Properties and MarketHours (#5100)
* Add interface to allow custom security entries for MHDB and SPD

* Simplify adding custom Properties and MarketHours via AddData overload

* Refactor

* Remove AddData and GetDatabaseSymbolKey overloads

* Add unit tests

* Remove AddData overload, for real.

* Nit changes

* Set CustomDataBitcoinAlgorithm back to original

* Add Python and C# Regression

* nit typo

* nit typo actual

* Reset symbol property to default

* Reflect last change to py regression

* Revert "Remove AddData overload, for real."

This reverts commit dc877495c079f17dda1695494ffe36886b1c117e.

* Implement AddData solution fully

* Function nit fix

* Address review

* nit - remove param comment

* Address review
10335
2021-01-14 11:34:02 -03:00
Martin-Molinero e6d16882f7 Remove raw Sleeps for Sleeps+Event (#5153)
- Remove raw Thread.Sleep for Event driven wait/sleeps. Improves startup
  performance significantly
10321
2021-01-12 19:12:07 -03:00
Gerardo Salazar d5aa9e65f3 Adds FuturesOptionsUnderlyingMapper and FuturesListing (#5142)
* Adds FuturesOptionsUnderlyingMapper

  This adds a mapper for Futures options that have a different
  contract month between the FOP and the underlying future, as is
  observed in the contracts ZB, ZC, ZS, ZT, ZW, HG, SI, GC.
  An example of this is GC (e.g. OGH21 -> GCJ21). Other contracts follow
  different rules, which have been included as part of this commit.

  * Adds unit tests for FuturesOptionsUnderlyingMapper

  * Adds unit and regresssion tests for underlying Futures mapping for FOPs

* Addresses review and adds FuturesListings

  * Removes dependency on IFuturesChainProvider in
    FuturesOptionsUnderlyingMapper

  * Fixes bugs related to FuturesOptionsUnderlyingMapper and incorrect
    results

  * Modifies misc. code to handle new results correctly from FOPs
    underlying mapper

  * Makes FOPs underlying mapper static, and makes other methods private

  * Adds new tests for FuturesListings

* Addresses review: code cleanup

* Address review: Makes arrays to List in FuturesListing private methods
10317
2021-01-12 06:12:00 -08:00
Mathieu Paquette ec1c4b7142 update IQFeed.CSharpApiClient (#5148)
* update IQFeed.CSharpApiClient

fixes #5147

* Update IQFeed client csproj referece

Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
10307
2021-01-11 18:11:16 -03:00
Adalyat Nazirov 9a0d943f1d Fill forward data points until delisted date (#4973)
* fill forward data points until expiry date

* wip: test non Delisting

* handle Aux not-Delisting dp; keep order while ff'ing

* simplify test checks

* wip

* fix liquidation expected date

* change statistics

* check EndTime, because it's used for time synchronization

* clean up code; remove duplicated test
10306
2021-01-11 16:45:09 -03:00
Ari Cooperman 771244aaa0 Add order related mappings to resolve type & status IDs in pure python (#5075)
* Add order related mappings to resolve tye & status IDs in pure python

* Update order.py

Co-authored-by: Martin-Molinero <martin@quantconnect.com>
10304
2021-01-11 15:45:51 -03:00
Alexandre Catarino 659a57884c Improves Error Message for Api.Download (#5038)
* Improves Error Message for Api.Download

Catches the exceptios, logs it and returns null.

* Replace Logging for Re-Throwing the Exception
10303
2021-01-11 15:40:46 -03:00
Martin-Molinero 3f2a558f91 Remove pythonNet from stacktrace (#5146)
* Avoid pythonNet  stackTrace in algorithm status

* Handle algorithm initialization exception
10298
2021-01-11 10:28:32 -03:00
Martin-Molinero 9bb7c77fb7 Fix UniverseSelectionAddAndRemove race condition (#5144)
* Fix UniverseSelectionAddAndRemove race condition

- Fix unit test UniverseSelectionAddAndRemove thread race condition

* Avoid boolean flag thread race condition

* Exclude test from travis build
10291
2021-01-08 19:18:58 -03:00
Derek Melchin c8eb9e0099 Fix MACD WarmUpPeriod and Updating (#5110)
* Fix MACD WarmUpPeriod and Updating

* Add System to use Math library

* Fix WarmUpPeriod math and add tests

* Fix SchaffTrendCycle Indicator WarmUpPeriod

* Ensure fastPeriod < slowPeriod

* Minor tweaks

- Remove unrequired Math.Max operation
- Remove unrequired changes in solution file

Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
10285
2021-01-08 16:58:53 -03:00
Jovad Uribe b85a32a391 Bug #5030 Generation of CFD data not working (#5112)
* Update RandomDataGeneratorProgram.cs

Replaced AddMonths(6) with a Datetime value half way between settings.Start and settings.End.

* Update RandomDataGeneratorProgram.cs

* Update to bug-5030-CFDdatanotoworking

* Added midpoint unit test

* Minor test assert improvement

Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
10283
2021-01-08 11:38:36 -03:00
Martin-Molinero 31d101253c LeanDataReader future/option data (#5117)
* LeanDataReader future option data

- The LeanDataReader will be able to read an entire zip of future and
  option data, currently it's just returning the first entry in the zip
  file. Adding unit tests

* Fix added unit test data path
10282
2021-01-08 11:13:05 -03:00
Aaron Janeiro Stone 7531046f14 Feature 4999 -- Adds RVI (#5041)
* adds RVI

* RVI refactor and tests
TODO: fix ComparesWithExternalDataRviSignal

* Fixed to allow all tests to pass.

* Addresses requested changes

* Addresses requested changes

* Addresses reviewer recommendations
10281
2021-01-08 11:11:00 -03:00
Martin-Molinero 6b99266669 Update IBAutomator increased init timeout (#5139)
- Updating IBAutomator to 1.0.40 which has an increased initialization
  timeout
10278
2021-01-07 19:13:07 -03:00
Adalyat Nazirov 8d48996948 fix covesion to string: decimal are represented using comma decimal separator for non-US English cultures, but expected dot (#5138) 10273 2021-01-07 17:35:51 -03:00
Martin-Molinero 7f381cb135 Universe dispose and removal (#5134)
* Universe dispose and removal

- Fix bug where in some cases the disposed universe would not get removed
from UniverseManager not allowing new universes to get added correctly,
using the same symbol. Adding regression test
- Fix bug where in some cases selected symbols would not get removed
  when the parent universe would get disposed of.
- Minor normalization on UniverseSelection call

* Address review

- Add status plots for the added regression test

* Adding regression algorithm reproducing issue 3914
10266
2021-01-07 12:03:17 -03:00
Riley Shea 2b7372783a Fix column headings in Data/equity/readme.md (#5125)
* Swapped incorrect column heading postions
10263
2021-01-07 11:01:51 -03:00
Adalyat Nazirov 54fce3f666 FillForward enumerator does not loop infinitely if fillforward resolution is different from data resolution (#5118)
* change filename template. add fillforward resolution suffix

* replicate GH issue 5116 on master

it's easy to reproduce on FXCM market by using data resolution different from fillforward resolution.
in this cases daily vs hour/minute were added

* change priorities of Time & EndTime values.

it's necessary to calculate EndTime  properly, and then we can align Time to it.
potential end time should be also calculated using ptoper TZ. Because of we store open hours without TZ movement TZ in market-hours it's necessary to reapply TZ

fix tests

* Miss 2AM bar on Sunday of DST; FF 2AM bar on Sunday ST

* use UTC TimeZone as baseline during comparison
10259
2021-01-06 20:18:08 -03:00
Colton Sellers 12db2261dc Allow Saving of Octet Stream in API (#5130)
* Allow downloading of octet stream

* Refactor solution

* Match lowercase ContentType
10258
2021-01-06 19:46:09 -03:00
Alexandre Catarino c6d4888e70 Refactors Market Fill of Equity Fill Model (#5114)
* Refactors Market Fill Model

Create `GetBidPrice` and `GetAskPrice` methods to request the most suitable price to fill market orders.

Fix unit tests to show that new implementation prevents using non-market data to fill the order.

* Addresses Peer Review …

Modifies EquityTickQuoteAdjustedModeRegressionAlgorithm to reflect changes in the market fill method. The previous implementation only looks for the last tick when it should look for the last tick of quote type, so current implementation is an improvement.
10249
2021-01-06 12:35:33 -03:00
Martin-Molinero 33b58b0dbc Fixes for IB malformed symbols (#5127)
* Fixes for IB malformed symbols

- Add handling for IB future malformed symbols. Adding unit test
- Fix IB option malformed symbols which had more whitespaces than
  expected. Adding unit test

* Address review
10248
2021-01-06 11:59:22 -03:00
Martin-Molinero 56cb9fbec5 IB will wait for connection response (#5124)
- IB.Connect() will wait for connection response before starting reader
10246
2021-01-05 19:44:04 -03:00
Jared 194e6f6b58 Update benchmark algorithm to include disabling (#5115)
* Update benchmark algorithm to include disabling

* Update C# benchmark example algo

Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
10236
2021-01-04 18:11:18 -03:00
Martin-Molinero 612c4f8b66 Fix BTC future price multiplier (#5113) 10234 2021-01-04 15:44:38 -03:00
Martin-Molinero d3b9f8bc7a Protobuf OpenInterest (#5107)
- Add OpenInterest protobuf support. Adding unit test
- Use composer for future and option chain provider
10230
2021-01-04 10:40:35 -03:00
Gerardo Salazar c4a1c245a7 Adds data parsing for malformed option contracts for InteractiveBrokers (#5101)
* Adds data parsing for malformed option contracts in IBBrokerage

  * Some contracts come in malformed from IB when downloading our
    account holdings. We attempt to detect whenever this happens with
    our options Symbols and try to recover the contract.

* Addresses review and fixes bug

  * Uses contract currency for the new contract created from the
    malformed contract.

  * Bug fix: Sets ibSymbol to new Symbol of the contract once the
    malformed contract has been parsed.

  * Adds support for Futures and FOPs

  * Refactoring code and cleanup

* Addresses review: code cleanup and refactoring

* Addresses review: remove redundant logging of contract in log statement
10222
2020-12-30 16:28:33 -08:00
Martin-Molinero 050dadc21f Increase low resolution cache size (#5105)
- Increasing low resolution cache size so it can hold QC500
10220
2020-12-30 20:59:36 -03:00
Colton Sellers 7b0ea0982e Update ExpectedStatistics (#5094) 10206 2020-12-29 12:04:36 -03:00
Adalyat Nazirov f8cf923d7b upload SHY quote bars (#5097) 10205 2020-12-29 11:58:13 -03:00
Colton Sellers 14b44bbe17 Reduce Travis Logs v2 (#5049)
* Silence DotNet in building stubs

* Reapply silence to stub publishing

* Have QuantBook store and restore initial LogHandler

* Add assembly action to maintain LogHandler

* Allow AlgorithmRunner to use ConsoleLogHandler

* Use MaintainLogHandlerAttribute to provide current test logger

* Refactor LogHandler creation
10201
2020-12-28 21:10:21 -03:00