2915 Commits

Author SHA1 Message Date
Gerardo Salazar e2964dd4b1 Make OrderListHash deterministic by using MD5 as its underlying hash function (#5276)
* Update OrderListHash to use MD5 as hash instead of hash code

* Update regression algorithm OrderListHash statistic

* Use full MD5 hash as OrderListHash, update regression statistic

* Fixes failing regression tests
2021-02-09 12:19:25 -03:00
Martin-Molinero d077651f32 Package updates (#5273)
- Update `System.Collections.Immutable`
2021-02-08 15:49:15 -03:00
Gerardo Salazar bc4dbf5707 Removes extra day added to CBOE Time causing data to shift one day forward (#5266)
* Added test
2021-02-05 19:42:01 -03:00
Gerardo Salazar fe1948d0ae Removes/updates System.ComponentModel.Composition package (#5257) 2021-02-05 12:03:49 -03:00
Gerardo Salazar 254d0896f1 Replaces Microsoft.Extensions.CommandLineUtils with McMaster CLI utils (#5255)
* Replaces Microsoft.Extensions.CommandLineUtils with McMaster CLI utils

  * Updates ValueTuple to 5.0.0 for compatibility with package.

* Retrigger build

* Retrigger build
2021-02-03 19:56:14 -03:00
Gerardo Salazar 115dcab789 Removes app.config and sets automatic binding redirects (#5248)
* Some app.config files were kept to either enable features
    that are only enable-able via the config files, or binding redirects
    for DLLs that are not packaged with NuGet (i.e. VisualStudio
    package, required to run tests successfully @ 15.0.0.0)
2021-02-03 11:45:29 -03:00
Martin-Molinero eb25dce462 Remove Alpaca brokerage implementation (#5249)
- Removing Alpaca brokerage implementation. See https://www.quantconnect.com/forum/discussion/10079/alpaca-paper-disabled
2021-02-02 20:16:34 -03:00
IlshatGaripov 0d0389ef6a BrokerageSetupHandler min resolution variable default value bug fix (#5243)
* fix

* Minor tweaks adding unit tests

* Address reviews

- BrokerageSetupHandler will use UniverseSettings.Resolution as default
  resolution

Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
2021-02-02 12:10:18 -03:00
IlshatGaripov 3e962b2e76 Fixes Bitfinex Brokerage history call bug (#5242)
* Fixes a bug #5173

* This is not timestamp but total ms.

* BitfinexBrokerageHistoryProviderTests fix
2021-02-01 21:58:57 -03:00
Aaron Janeiro Stone 7d70698c9a Feature #5098 - Time Series Indicators (#5099)
* Cleans history for ArimaIndicator/TimeSeriesIndicator.

-- removes commits from a tracked, already merged branch
-- removes artifacts from debugging sessions

* Removes AR/MA method as a user-specifiable method.
-- Prevents need to reference dll for MathNet in Tests (and potentially elsewhere).
-- Wrapper can be implemented around this functionality.

* Removes AR/MA method as a user-specifiable method.
-- Prevents need to reference dll for MathNet in Tests (and potentially elsewhere).
-- Wrapper can be implemented around this functionality.

* Better adherence to established code style

* Makes _intercept = true by default in constructor where it is not parameter

* WIP -- addressing reviews

* Passing tests following prior refactor

* Rearranged code, access modifiers adjusted

* Fixed indexing of _mafits, adds example algorithm

* Adds regression algo in python + addresses some refactors

* Addresses review

* Adds regression stats

* Fixes missing value signs

* Removes redundant code

* style changes

* style changes

* style: "err" -> "error"

* Minor tweaks

* Fixes python arima regression test

* Refactors AutoregressiveIntegratedMovingAverageTests.cs

Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
2021-02-01 20:17:49 -03:00
Martin-Molinero 1606b12475 Fix IB future chain being started at reset hours (#5246)
* Fix IB future chain being started at reset hours

- Fix IB future chain being started at reset hours. Adding unit test

* Refactor CanAdvanceTime into CanPerformSelection
2021-02-01 18:53:33 -03:00
Gerardo Salazar fe74d037c6 Updates NodaTime to latest version (3.0.5) (#5237)
* Updates NodaTime to version 3.0.5

  * Updates code to ensure compatibility with new NodaTime version
  * Updates System.Runtime.CompilerServices.Unsafe to 5.0.0
    - Required by NodaTime

* Adds comments explaining changes in NodaTime
2021-02-01 13:06:56 -03:00
Colton Sellers 472896c2d0 Update JSON Library (#5218)
* Update NewtonSoft.Json from 10.0.3 to 12.0.3

* Remove JsonSerializer.Populate(), values are already populated
2021-01-28 19:24:49 -03:00
Stefano Raggi 15cb846570 Add missing connection check in IB brokerage GetHistory (#5230) 2021-01-28 15:43:04 -03:00
Colton Sellers a5eaa43e35 Remove F# and VB Projects (#5223)
* Remove F# and VB projects

* Remove FSharp.Core package

* Readd FSharp to engine

Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
2021-01-27 11:31:49 -03:00
Martin-Molinero e91af0814f Add notification targets and events to live packet (#5210)
* Add notification targets and events to live packet

- Add notification targets and events to live packet. Adding unit tests

* Add project name to job packet

* Add ShortToString and WebNotification Header

- Adding ShortToString implemnetation of OrderEvent and Insight
- Add Web notification headers

* Add more unit tests

* Revert json lower case fields
2021-01-27 10:26:35 -03:00
Colton Sellers a2850fb20c Address Non-Compiled Files (#5225)
* Include MortgageRateVolatilityAlpha and fixes

* Include PlaceHolder PythonAlgorithm.cs

* Delete BinanceUtil

* Remove unused Regression Algorithms

* Minor tweaks

Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
2021-01-27 09:56:03 -03:00
Colton Sellers c86fceac97 Fix regression typo (#5224) 2021-01-26 15:31:57 -03:00
Adalyat Nazirov c6d29c8be5 add FIX Protocol UTC timestamp format (#5212) 2021-01-25 11:24:54 -03:00
Martin-Molinero 3ccf428498 Adjust delisting liquidation time (#5203)
* Adjust delisting liquidation time

- Adjust delisting liquidation time to 15 min before market closes.
  Adding unit tests. Updating existing.
- Handle `Statistics.CompoundingAnnualPerformance` invalid calculation
  to avoid exception.
- AlgorithmManager will not handle delisting events in live trading
- Fix bug where due to a split driven liquidation matching delisting
  date a position in the option would remain open. Reproduced by
  `BasicTemplateOptionsFrameworkAlgorithm`

* Address review

- Address review add documentation on delisting offset span
2021-01-22 14:41:29 -03:00
Colton Sellers 5d8a62c2e1 Symbol Alias Fix (#5205)
* Fix Symbol Alias

- Fix Symbol Alias being wrong in some cases for futures and options

* Add test cases

* Address review

Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
2021-01-22 13:00:14 -03:00
Juan José D'Ambrosio 765f2cab3b Add primary exchage to map files (#5194)
* Add main exchange to map file

minimum minimorum test

* Add MainExchange to MapFilRow IEquatable implementation
Also write CSV line correctly

* Normalize exchanges

Also add small test form MapFileRow

* Exchange name normalization

- Exchange name normalization.
- Tick will protobuf `ExchangeCode` which will set existing `Exchange`

* Include single character exchange code

* Encode exchange as byte, Include exchange name as Enum

* Update MapFileRow.cs to accept a byte instead a string for primary exchange

* Fix issues in encoding exchanges map

* Fix broken MapFile tests

* Rename Main Exchange property in MapFileRow

* Use BATS instead of BATSZ

* Add primary exchange from map file

* Move MapFilePrimaryExchangeProvider to its own file

* Implement Char for primary exchange in map files

* Fix broken tests after implementing char exchange into map file

* Add a new constructor for MapFileRow

it  accepts a PrimaryExxchange object as parameter

* Fix broken test 

WIP

* Address review 

WIP

* Handles unexpected encoded exchanges

* Keep addressing review

* Address review

* Fix broken tests

Make PrimaryExchange as default even if the IPrimaryExchangeProvider is not instantiated

* Revert breaking change.

* Add a check that resolved map file actually have content

The issue that leads to this change was caused by a map file resolved to APC in the CoarseFundamentalTop3Algorithm. 
The resolver returns a not-null MapFile, but it is empty.
To my best understand, the MapFile Resolver should return only existing map files in the map_files folder. But the LocalMapFileProvider returned a map file with a permtick that doesn't exist in the file system as CSV.

* Set PrimaryExchange.UNKNOWN as default

* Address review couple minor bugs

- Use SID market instead of USA
- Default ticke exchange is UNKNOWN

* Update map files with latest format and latest info

This include:
 - VXX, it was delisted, then VXXB appears and later it chnages from VXXB -> VXX
 - AAA delisting and a new listing with ticker AAA
 - FOXA ticker changue to TFCFA and later it was delisted

* Update test after updating latest map files

* Fix bug with factor files after updating map files

* Fix bug with factor files after updating map files

and revert statistics in regression tests

* Update OrderHash in regression test

Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
2021-01-21 18:46:12 -03:00
Gerardo Salazar 79c544d3a2 Reduces output log size and warnings emitted when building with mono's msbuild (#5200) 2021-01-21 11:09:16 -03:00
Colton Sellers f2fd10d9d2 Build Configuration Hotfix (#5186)
* Reflect properties prior to migration onto projects

* Do not prefer 32bit

* Reinstate SelectedOptimization var
2021-01-19 09:38:16 -03:00
Martin-Molinero 592d037085 Fix delisted liquidation orders being cancelled (#5169)
* Fix delisted liquidation orders being cancelled

- Place delisted liquidation orders 10 min before market closes of 10
  min before the end of the delisting warning date. Adding regression
  test and unit tests. Updating existing tests.

* Fix failing python option unit tests

* Fix bug where positions would be open delisting liquidation

* Fix universe selection and delisting

- Delisting will happen ASAP for all types. Giving priority to close
  positions on derivates first
- Fix bug in universe selection where OptionChain would remove
  underlying even if holding a position in derivate.
- Updating regression tests statistics

* Add unit test, fix unit test expected stats
2021-01-18 22:02:40 -03:00
Aaron Janeiro Stone 7aefe07274 4553 - Refactor RenkoConsolidator (#5052)
* Separated Update calls and other non-common RenkoConsolidator.cs artifacts.

* Added type safe init for wicked

* Refactor; moved wicked-specific fields to WickedRenkoConsolidator.cs

* Addresses requested changes (and compacts constructors)

* Separated Update calls and other non-common RenkoConsolidator.cs artifacts.

* Added type safe init for wicked

* Refactor; moved wicked-specific fields to WickedRenkoConsolidator.cs

* Addresses requested changes (and compacts constructors)

* Removes base class for brevity

* Delete BaseRenkoConsolidator.cs

* Fixes naming for classic Renko consolidator

* Addresses review concerns

* Fixes missing null checks and missing typesafe constructor

* Adds warning

* WickedRenkoConsolidator -> RenkoConsolidator calls

* Separated Update calls and other non-common RenkoConsolidator.cs artifacts.

* Added type safe init for wicked

* Refactor; moved wicked-specific fields to WickedRenkoConsolidator.cs

* Addresses requested changes (and compacts constructors)

* Removes base class for brevity

* Fixes naming for classic Renko consolidator

* Separated Update calls and other non-common RenkoConsolidator.cs artifacts.

* Added type safe init for wicked

* Refactor; moved wicked-specific fields to WickedRenkoConsolidator.cs

* Addresses requested changes (and compacts constructors)

* Delete BaseRenkoConsolidator.cs

* Addresses review concerns

* Fixes missing null checks and missing typesafe constructor

* Adds warning

* WickedRenkoConsolidator -> RenkoConsolidator calls

* Addresses review

* unneeded tag removed

* Minor tweaks address reviews

Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
2021-01-18 15:54:15 -03:00
Karthik Kailash 6de9a146a9 Issue 5149: Added missing early closes for 1999-2008 (#5163)
* Added missing early closes for 1999-2008

See GitHub issue for details

* Fix expected Treynor ratio for backtest in test suite

This backtest runs from 2008-10-10 to 2010-10-10 so it is affected by the early close date now recorded on 2008-12-24.
2021-01-15 11:22:44 -03:00
Gerardo Salazar 4d4ad92fa7 Easy To Borrow Hotfix (#5167) 2021-01-14 15:07:54 -08:00
Gerardo Salazar 3e7af17e02 Implements Easy To Borrow Reality Modeling (WIP) (#5159)
* Creates IShortableProvider interface + interface impls

  * This is the foundational work for the addition of the shortable stocks
    feature for backtesting and live trading. Note that the QCAlgorithm
    API and the backend transaction handling will be implemented
    separately.

* temp; work on preorder checks

* improve checks

* tmep

* Enforces ETB checks at BrokerageTransactionHandler

  * Adds ETB Shortable Provider to QCAlgorithm and IAlgorithm
  * Removes ETB check from PreOrderChecksImpl
  * Removes outdated test, new test to come soon for relevant class

* Work in progress commit, pushing for review.

  * Adds new regression algorithm (WIP)
  * Enhances ShortableProviderOrdersRejectedRegressionAlgorithm
  * Adds new methods to QCAlgorithm
  * Move IShortableProvider to have BrokerageModel own it
  * Comments updates
  * Adds new properties to Equity Security for shortable

  * Fixes bug where retrieving open order quantities would aggregate the
    submitted order quantity rather than the remaining order quantity for
    open orders.

  * Fixes bug where quantity of zero would result in a
    false positive in QCAlgorithm.Shortable(...)

  * Code refactoring and comments updates

Co-authored-by: Jared <jaredbroad@gmail.com>
2021-01-14 13:33:44 -08:00
Colton Sellers a2a874e333 CSProj Update (#5145)
* Migrate to newer CSProj format; build succesful

Rebase onto master

* Maintain binaries directory

* Reinstate FSharp default condition

* Remove reference to packages.config

* Test project fixes

* Remove nuget.config and "Solution Items"

* Remove repetitive assembly info

* Reinstate default compiles and cleanup redundant settings

* Reinstate any specific settings that defaults are unclear for

* Reinstate Framework.nuspec

* Make test internal classes public for composer use; undo assembly change

Rebase onto master part 2

* Address review; restore some settings

* Restore RootNamespace and AssemblyName attributes

* Use NugetPackageRoot instead of NugetPackageFolders

* Reinstate "RestorePackages" setting

* Transfer compilied Python files to launcher and tests bin

* Remove conflicting settings

* Fix WebSocketSharpFork references

* Fix FSharp Core reference

* Upgrade TestPlatform.ObjectModel to 16.8.3, and redirect V11 binding

* Add Mono.Cecil package

* Address review
2021-01-14 12:55:09 -08:00
Colton Sellers 7e3741d983 Allow Custom Security Properties and MarketHours (#5100)
* Add interface to allow custom security entries for MHDB and SPD

* Simplify adding custom Properties and MarketHours via AddData overload

* Refactor

* Remove AddData and GetDatabaseSymbolKey overloads

* Add unit tests

* Remove AddData overload, for real.

* Nit changes

* Set CustomDataBitcoinAlgorithm back to original

* Add Python and C# Regression

* nit typo

* nit typo actual

* Reset symbol property to default

* Reflect last change to py regression

* Revert "Remove AddData overload, for real."

This reverts commit dc877495c079f17dda1695494ffe36886b1c117e.

* Implement AddData solution fully

* Function nit fix

* Address review

* nit - remove param comment

* Address review
2021-01-14 11:34:02 -03:00
Martin-Molinero e6d16882f7 Remove raw Sleeps for Sleeps+Event (#5153)
- Remove raw Thread.Sleep for Event driven wait/sleeps. Improves startup
  performance significantly
2021-01-12 19:12:07 -03:00
Gerardo Salazar d5aa9e65f3 Adds FuturesOptionsUnderlyingMapper and FuturesListing (#5142)
* Adds FuturesOptionsUnderlyingMapper

  This adds a mapper for Futures options that have a different
  contract month between the FOP and the underlying future, as is
  observed in the contracts ZB, ZC, ZS, ZT, ZW, HG, SI, GC.
  An example of this is GC (e.g. OGH21 -> GCJ21). Other contracts follow
  different rules, which have been included as part of this commit.

  * Adds unit tests for FuturesOptionsUnderlyingMapper

  * Adds unit and regresssion tests for underlying Futures mapping for FOPs

* Addresses review and adds FuturesListings

  * Removes dependency on IFuturesChainProvider in
    FuturesOptionsUnderlyingMapper

  * Fixes bugs related to FuturesOptionsUnderlyingMapper and incorrect
    results

  * Modifies misc. code to handle new results correctly from FOPs
    underlying mapper

  * Makes FOPs underlying mapper static, and makes other methods private

  * Adds new tests for FuturesListings

* Addresses review: code cleanup

* Address review: Makes arrays to List in FuturesListing private methods
2021-01-12 06:12:00 -08:00
Adalyat Nazirov 9a0d943f1d Fill forward data points until delisted date (#4973)
* fill forward data points until expiry date

* wip: test non Delisting

* handle Aux not-Delisting dp; keep order while ff'ing

* simplify test checks

* wip

* fix liquidation expected date

* change statistics

* check EndTime, because it's used for time synchronization

* clean up code; remove duplicated test
2021-01-11 16:45:09 -03:00
Alexandre Catarino 659a57884c Improves Error Message for Api.Download (#5038)
* Improves Error Message for Api.Download

Catches the exceptios, logs it and returns null.

* Replace Logging for Re-Throwing the Exception
2021-01-11 15:40:46 -03:00
Martin-Molinero 9bb7c77fb7 Fix UniverseSelectionAddAndRemove race condition (#5144)
* Fix UniverseSelectionAddAndRemove race condition

- Fix unit test UniverseSelectionAddAndRemove thread race condition

* Avoid boolean flag thread race condition

* Exclude test from travis build
2021-01-08 19:18:58 -03:00
Derek Melchin c8eb9e0099 Fix MACD WarmUpPeriod and Updating (#5110)
* Fix MACD WarmUpPeriod and Updating

* Add System to use Math library

* Fix WarmUpPeriod math and add tests

* Fix SchaffTrendCycle Indicator WarmUpPeriod

* Ensure fastPeriod < slowPeriod

* Minor tweaks

- Remove unrequired Math.Max operation
- Remove unrequired changes in solution file

Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
2021-01-08 16:58:53 -03:00
Jovad Uribe b85a32a391 Bug #5030 Generation of CFD data not working (#5112)
* Update RandomDataGeneratorProgram.cs

Replaced AddMonths(6) with a Datetime value half way between settings.Start and settings.End.

* Update RandomDataGeneratorProgram.cs

* Update to bug-5030-CFDdatanotoworking

* Added midpoint unit test

* Minor test assert improvement

Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
2021-01-08 11:38:36 -03:00
Martin-Molinero 31d101253c LeanDataReader future/option data (#5117)
* LeanDataReader future option data

- The LeanDataReader will be able to read an entire zip of future and
  option data, currently it's just returning the first entry in the zip
  file. Adding unit tests

* Fix added unit test data path
2021-01-08 11:13:05 -03:00
Aaron Janeiro Stone 7531046f14 Feature 4999 -- Adds RVI (#5041)
* adds RVI

* RVI refactor and tests
TODO: fix ComparesWithExternalDataRviSignal

* Fixed to allow all tests to pass.

* Addresses requested changes

* Addresses requested changes

* Addresses reviewer recommendations
2021-01-08 11:11:00 -03:00
Adalyat Nazirov 54fce3f666 FillForward enumerator does not loop infinitely if fillforward resolution is different from data resolution (#5118)
* change filename template. add fillforward resolution suffix

* replicate GH issue 5116 on master

it's easy to reproduce on FXCM market by using data resolution different from fillforward resolution.
in this cases daily vs hour/minute were added

* change priorities of Time & EndTime values.

it's necessary to calculate EndTime  properly, and then we can align Time to it.
potential end time should be also calculated using ptoper TZ. Because of we store open hours without TZ movement TZ in market-hours it's necessary to reapply TZ

fix tests

* Miss 2AM bar on Sunday of DST; FF 2AM bar on Sunday ST

* use UTC TimeZone as baseline during comparison
2021-01-06 20:18:08 -03:00
Alexandre Catarino c6d4888e70 Refactors Market Fill of Equity Fill Model (#5114)
* Refactors Market Fill Model

Create `GetBidPrice` and `GetAskPrice` methods to request the most suitable price to fill market orders.

Fix unit tests to show that new implementation prevents using non-market data to fill the order.

* Addresses Peer Review …

Modifies EquityTickQuoteAdjustedModeRegressionAlgorithm to reflect changes in the market fill method. The previous implementation only looks for the last tick when it should look for the last tick of quote type, so current implementation is an improvement.
2021-01-06 12:35:33 -03:00
Martin-Molinero 33b58b0dbc Fixes for IB malformed symbols (#5127)
* Fixes for IB malformed symbols

- Add handling for IB future malformed symbols. Adding unit test
- Fix IB option malformed symbols which had more whitespaces than
  expected. Adding unit test

* Address review
2021-01-06 11:59:22 -03:00
Martin-Molinero d3b9f8bc7a Protobuf OpenInterest (#5107)
- Add OpenInterest protobuf support. Adding unit test
- Use composer for future and option chain provider
2021-01-04 10:40:35 -03:00
Gerardo Salazar c4a1c245a7 Adds data parsing for malformed option contracts for InteractiveBrokers (#5101)
* Adds data parsing for malformed option contracts in IBBrokerage

  * Some contracts come in malformed from IB when downloading our
    account holdings. We attempt to detect whenever this happens with
    our options Symbols and try to recover the contract.

* Addresses review and fixes bug

  * Uses contract currency for the new contract created from the
    malformed contract.

  * Bug fix: Sets ibSymbol to new Symbol of the contract once the
    malformed contract has been parsed.

  * Adds support for Futures and FOPs

  * Refactoring code and cleanup

* Addresses review: code cleanup and refactoring

* Addresses review: remove redundant logging of contract in log statement
2020-12-30 16:28:33 -08:00
Colton Sellers 14b44bbe17 Reduce Travis Logs v2 (#5049)
* Silence DotNet in building stubs

* Reapply silence to stub publishing

* Have QuantBook store and restore initial LogHandler

* Add assembly action to maintain LogHandler

* Allow AlgorithmRunner to use ConsoleLogHandler

* Use MaintainLogHandlerAttribute to provide current test logger

* Refactor LogHandler creation
2020-12-28 21:10:21 -03:00
Adalyat Nazirov 017b464e21 Map legacy symbols using map file (#5017)
* map IB brokerage symbol

* map Alpaca symbols

* improve empty symbol checking
2020-12-28 17:54:00 -03:00
Gerardo Salazar 4dec21ccc0 Fixes failing option chain provider test for futures options (#5088)
* The test didn't take into account that the contract expiring on
    December would mean that there would temporarily be no Dec. FOP
    contract. We fix this by looking for the Mar. contract instead
    if the december contract has expired.
2020-12-28 16:28:41 -03:00
Martin-Molinero 529d0a3634 Fix low resolution cache (#5080)
* Replace cache size for MB

* Remove setting capacity

* Custom cache implementation

* Reduce locks in custom cache

* Add cache performance unit test
2020-12-28 16:27:11 -03:00
Martin-Molinero 9c89e8d403 Fix backtest result packet deserializing (#5084)
* added test covering minValue / maxValue issue with JsonRoundingConverter

* change namespaces

* JsonRoundingConverter fix decimal.Min and MaxValues (cannot deserialize)

* remove dependency on 3rd party library

* c# 6 compatible code (remove pattern matching)

* Fixes BacktestResultPacket deserializing

- Serializing decimals as strings to avoid precision loss, since json
  convert will use floating point precision. Updating unit tests.
- Fix logging unit test failing to delete file being used.

Co-authored-by: Mark Virchenko <mark.virchenko@calienteam.com>
2020-12-28 16:26:40 -03:00