331 Commits

Author SHA1 Message Date
Martin-Molinero e22565c849 C# stacktrace exception normalization (#7328)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
- Following the Py pattern of exception handling we will normalize stack
  traces for C# too, adding unit test
2023-06-21 10:51:17 -03:00
Martin-Molinero ef01980b07 Python history improvements (#7283)
Benchmarks / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Pandas frame include all ticks

- Pandas data framde history response will include all ticks. Updating
  existing and adding new tests

* Python pandas converter performance improvement
2023-05-31 10:22:21 -03:00
Martin-Molinero c33ce2f8c3 MHDB Merge Common Holidays, LateOpens & EarlyCloses (#7278)
* MHDB will merge common entry

- The MHDB will merge the market and security common entry holidays,
  early closes and late opens

* Normalize & reuse future US holidays

- Normalize & reuse future US holidays

* Update existing unit tests expected stats
2023-05-30 11:54:28 -03:00
Martin-Molinero 645e1ca88c Downloader Data Provider Improvements (#7159)
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
* Downloader data provider fixes

- Fix stored data time zone, which is expected in data time zone
- Minor logging improvements
- LeanData Parse handles etf universes path. Adding unit test
- LeanDataWrite will only map if there's something to map to

* Fix download data provider synchronization

* Synchronization improvements after more testing

- KeyStringSynchronizer will handle reentrancy. Adding unit tests
- Define a unique Synchronizare for downloading data, since not related
  to disk reading & writting
2023-03-31 18:29:55 -03:00
Martin-Molinero c836c62945 LeanDataWriter can handle mapping (#7150)
* LeanDataWriter can handle mapping

- Adjust LeanDataWrite so that it can handle symbol mapping if required.
  Adding unit tests

* Fix Lean Data path parsing
2023-03-29 16:04:23 -03:00
Martin-Molinero 1877d59daa Improve aggregation helper methods (#7100)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
- Improve LeanData aggregation helper methods to use shared consolidator based implementation
- Adding Tick to tradeBar aggregation
- Adding more unit tests
2023-03-20 16:08:25 -03:00
Martin-Molinero 56270464cc IsOption Use Underlying Holidays and Early Closes if Empty (#7007)
Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* IsOption uses underlyings holidays

- Option assets will use underlyings holidays, early closes and late
  opens if their own are not defined

* Update regression algorithms
2023-02-24 12:49:51 -03:00
Martin-Molinero a52822853d Expand Index option support (#6849)
Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Expand Index option support

- Adjust APIs so that the same underlying can be used
  for different options, adding support for SPX weekly options. Adding
  regression tests

* Fix IndexOption.IsStandard

* Add IndexOption test
2023-01-12 16:39:02 -03:00
Martin-Molinero ee683933a3 Feature Perpetual Crypto Futures (#6807)
Python Virtual Environments / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
* WIP

* Add base currency cash

* Symbol properties and data processing

* Add basic template algorithm

* Add hourly crypto future algorithm

* Minor fixes after live trading testing

* CoinApiDataQueueHandler CryptoFuture support

* Address reviews

* Fix regression algorithms after update
2022-12-30 13:58:11 -03:00
Martin-Molinero 45e13bb35b Memory Related Performance improvements (#6700)
* Memory Related Performance improvements

- Make sure we cleanup & dipose of python related objects during pandas
  data generation.
- Disable memoizing enumerable use while creating pandas data frames,
  since we do not require it
- Reduce unrequired object creations
- Replace concurrentCollections for ordinary locks

* Decimal parsing typo fix
2022-10-21 15:11:18 -03:00
Martin-Molinero 37dca9d458 Remove log packet aggregation (#6538)
Regression Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
- Remove log packet aggregation
- Clean up Queue api usage
2022-08-11 19:06:11 -03:00
Jhonathan Abreu 460ef10ff4 Add more info to python runtime error message (#6526) 2022-08-09 17:56:23 -03:00
Martin-Molinero 3f2bcc1784 Add missing CancellationTokenSource dispose (#6512)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
- Add missing CancellationTokenSource disposals to avoid memory leaks
2022-08-01 11:12:01 -03:00
Martin-Molinero 31fc7ac236 Do not generate Equity Quote Daily data (#6483)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
- Fix for so the random data generator does not try to create daily
  equity quote data
2022-07-14 10:20:34 -03:00
Martin-Molinero 630503bf9c Remove deprecated inspect.getargspec (#6443)
Research Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
- Remove deprecated python `inspect.getargspec` usage, replace for new
  `getfullargspec`. Adding unit tests reproducing issue
2022-06-27 18:49:46 -03:00
Ricardo Andrés Marino Rojas 8f3ced7639 Fix bug with Coinbase Pro Stable Pairs (#6362)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* First Commit

* Change Unit tests
2022-06-01 17:19:41 -03:00
Martin-Molinero b7a21ce4be Update to net6 (#6311)
* Update to net6

* Bump pythonNet to 2.0.13
2022-05-03 11:45:55 -03:00
Martin-Molinero b9d3d99917 Bump pythonNet version 2.0.12 (#6310)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Updates after pythonNet rebase

* Bump pythonNet version 2.0.12

* Fix exception types being thrown
2022-05-02 14:38:50 -03:00
Martin-Molinero 06228a8de8 Improve python stack trace parsing (#6316)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
- Improve python stack trace parsing removing base Lean path directory.
  Adding unit test
2022-05-02 11:42:36 -03:00
Ricardo Andrés Marino Rojas e0d29e1da7 Charge correct fees for Coinbase Pro stable pairs (#6312)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* First commit

* Improve implementation

* Improve implementation
2022-04-29 16:29:21 -03:00
Ricardo Andrés Marino Rojas f617d25af9 Add missing StableCoins pairs in Crypto Exchanges (#6259)
Research Regression Tests / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Add missing stablecoins

* Fix bugs

* Remove MIMUSD stablecoin pair

That pair is already in the SPDB

* Add more stablecoins pairs

- Add SUSD and IDRT

* Fix bug

* Requested changes

The algorithm posted in the GH was failing because when finding a conversion between currencies BTC to USD, it wasn't taking into account BTCUSDT and that USD = USDT because is a stablecoin

* Solve bugs

* Nit changes and more tests

* Requested changes

* Nit changes

* Requested changes

* Nit change
2022-04-12 18:10:37 -03:00
Martin-Molinero a8e104f696 Allow LeanDataWriter to merge existing ticks if desired (#6284)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Allow DataWriter to merge existing ticks if desired

- Optionally allow lean data writer to merge new ticks with existing
- Adjust ZipStreamWriter
- Fix bug in compression lib that was leaving files open. Reproduced by
  added tests.

* Address reviews

- Define new WritePolicy to configure the behavior of the LeanDataWriter
  regarding how it should handle writing to a file: merge, overwrite,
  append
2022-04-08 18:20:23 -03:00
Martin-Molinero c4a2d6eef4 Crypto base currency fees handled correctly (#6166)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Binance fees deducted from fill quantity accordengly

- For Binance cash accounts while buying, if fees are from the base
  currency of leans virtual position, we need to deduct the fee from the
  fill quantity, else we can end with a position bigger that it actually
  is and not be able to liquidate

* Refactor solution

- Refactor solution into a more generic approach solving fees in base
  currency at the BrokerageTransactionHandler level, covering all
  brokerages that require it. Adding regression algorithm reproducing
  issue.
- Update Bitfinex and Binance fee models to correctly reflact reality

* Log fill quantity adjusment once
2022-01-28 13:02:58 -03:00
Martin-Molinero b698641c90 Minor tweak for ApiDataProvider to support India (#6169)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
- Minor tweaks for the ApiDataProvider to better support India market
2022-01-24 21:27:22 -03:00
Martin-Molinero 30d7fb042b Always reuse aggregator instance if any (#6161)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Always reuse aggregator instance if any

- When fetching a IDataAggregator instance from the composer, do not
  enfore type name on existing instances

* Fix unit tests
2022-01-20 18:52:22 -03:00
Martin-Molinero e1d1e28bb8 Fix for Tick subscription history requests (#6156)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
- Fix for history requests != Tick for existing subscriptions with Tick
  resolution. Adding unit test reproducing issue
2022-01-18 17:36:27 -03:00
Andreas Sundebo e2de241c2b Feature 5090 add api optimization methods (#6108)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* Move Optimizer-related DTOs and JSON converters into Common/Optimizer

* Add REST methods for Optimization

* Move OptimizationStatus into Common

* Change optimizationId parameter type to string

* Update Optimization and add lightweight optimization object

* Rename lightweight optimization to BaseOptimization and remove unneccessary properties

* Remove snapshotId from Optimization, add ParameterSet to Backtest

* Add missing IApi.cs method signatures

* Move ParameterSet into Common

* Replace Backtest with OptimizationBacktest

* Update UpdateOptimization to not include null or empty name and layout params in the request

* Change Objective targetTemplate regex pattern from ['(.+)'] to (.+) to prevent escaping target strings without whitespace

* Return Estimate object when calling EstimateOptimization

* Use DefaultNamingStrategy when serializing constraint operators

* Revert "Change Objective targetTemplate regex pattern from ['(.+)'] to (.+) to prevent escaping target strings without whitespace"

This reverts commit fbe7de0fd77dccc62e8c52c42a4ec9c18347acb2.

* Update Api method signatures

* Add unit tests

* Fix XML comment referring to the old class name

* Fix XML summary for OptimizationResponseWrapper

* Address review feedback
- Remove unused testOrganizationId
- Change NodeType from string to NodeType enum
- Clarify unit types for Estimate time and balance
- Simplify JsonConverter classes

* Add accessors to Common/Api classes

* Define performance metrics names in PerformanceMetrics class

* Remove unnecessary branching logic from GetSeriesValues method

* Add crefs and examples to XML comments in the Api class

* Revert "Change NodeType from string to NodeType enum"

* Remove layout param from UpdateOptimization method

* Backtest property ParameterSet should be of type ParameterSet

* Add asserts for deserialization in OptimizationBacktestJsonConverterTests

* Replace the three target-related properties with Criterion

* Add serialization and deserialization tests for Optimization

* Remove Optimization Serialization test

* Add EstimateDeserialization test

* Add asserts for integration tests

* Address self review

* Revert test case

* Update Nodes.cs

* Update Nodes.cs

* Set Aborted status when Optimization fails to start

* Add ParameterSetJsonConverter and ParameterSetJsonConverterTests

Co-authored-by: Martin-Molinero <martin@quantconnect.com>
2021-12-22 13:14:56 -03:00
Martin-Molinero abbb50e209 Minor improvements for data reading and caching (#6113)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* Minor improvements for data reading and caching

* Address reviews

* Adjust zip cache error check
2021-12-16 19:58:37 -03:00
Ronit Jain 264c3c8374 Composite IDQH - Support multiple live data feeds (#6047)
* initial commit

* Follow IDQH implementation

* Expect a list of data handlers from LiveNodePacket

* Return null if can not subscribe

* refctor to add check for subscription

* initialze null

* Add tests

* Check subscribe retuns null/not-null

* cleanup

* Read all required IDQH credentails to job

* Use CDQH to handle all IDQH instances

* constructor abstraction to call from setjob

* use flag

* remove redundant because derived will call initialize on it

* abstract and initialize from setjob

* handle null enumerators

* get creds from data handlers

* handle single data handler value from data-queue-handler

* Fix to support a json array

* Fix missed constructor call

* change access modifier to access from Tests files

* Add test to get brokerageFactory from dataQueueHandler

* Fix init flag to handle all conditions

* Add docs

* initialize from setjob

* Check if websocket open before using

* change defination of initialzie to include tradier

* clean up

* change defination

* fix wrong api key name

* return empty enumerator

* check websocket open before sending request

* check connection before subscribing

* fix to include more cases

* check websocket open before sending request

* Minor refactoring

* reafctor and use IsConnected

* remove unused

* clean up

* Fix test cases

* reverse change

* include config changes

* connect to websocket from setjob

* check websocket connection from setjob

* clean up

* include condition for IDQH that are not brokerage

* Address review

* Add market check condition before subscribe

* Remove deprecated

* Minor fix for deserializing data queue handler

Co-authored-by: Martin-Molinero <martin@quantconnect.com>
2021-12-03 16:18:41 -03:00
Colton Sellers 1358bd8115 Future and FutureOption low res support (#6069)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Fixes for FutureOptions support in LeanData

* Add CreateCanonicalOption() utility function for Symbol.cs

* Add aggregated Futures/FuturesOptions data to Lean

* Add FutureOptions regressions for daily/hourly data

* Allow Futures to be added with low resolution

* Add Future regressions using hour/daily data

* Nit - Python Class names

* Add reviews

* Add alias into CreateCanonicalOption
2021-11-24 09:53:21 -03:00
Martin-Molinero b8b0d18993 Continuous futures (#6034)
* Continuous Future Contracts

* Mapping approach

* Tweaks WIP

* Live mapping

* Live mapping

- Add support for live mapping, refreshing mapfiles
- Fix future expiration functions
- Adding unit tests

* Update moq test package

* Continuous futures price scaling

* Refactor price factors scaling

* Factor file related renames

* Address reviews
2021-11-15 14:44:30 -03:00
Colton Sellers dd4da7ba95 Feature Daily/Hourly Options Support (#6017)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* Create generic writing for LeanDataWriter, + notes on todos

* Make Options Daily/Hourly data store by year

* Refactor Generic Write

* Permit hour and daily resolutions for options

* Refactor writer to merge when needed with other files

* Cleanup redundancies, run write tasks in parallel

* Make needed classes/vars available

* Update tests to reflect new naming convention for daily hourly options data

* Add Byte[] overloads for ZipData functions in compression

* Implemented Store() for ZipDataCacheProvider

* Have LeanDataWriter use a DataCacheProvider

* ZipDataCacheProvider cleanup

* ZipDataCacheProvider tweaks, doesn't support storing non-zips

* Test adjustments

* Update LeanDataWriter to use Write instead of SaveDailyHourly/SaveMinuteSecond

* Implement tests to verify DownloadAndSave behavior

* Nit cleanup on DownloadAndSave tests

* Fix for options daily/hourly underlying equity subscription read

* Add daily/hourly options data and regressions

* Add missing open interest for hourly

* Fix writing of OpenInterest Daily/Hourly data

* Update data

* Fix Date typo in regression

* Use daily algorithm to test delisting

* Revisions part 1

* Expand test for DataCacheProviders; refactor DiskDataCacheProvider

* nit - test adjustments

* ZipDataCacheProvider test setup refactor

* Adjust multithreaded read/write test; fixes for ZipDataCacheProvider

* Move DiskDataCacheProvider to its own file and add write test

* Remove _appendToZips; always overwrite entry or create zip

* Add mapping regression for daily options

* nit - add license to regression

* Fix Tick write case where more than one data point for a DateTime

* Fix data issue

* Address review

* Tweaks for tests

* Stop Store() early if no entry name is given
2021-11-11 20:05:31 -03:00
Martin-Molinero 9f29e3bf4e Some performance improvements for IndexOptions (#6025)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
- Some performance improvements for IndexOptions specially affects
  debugging C# algorithms.
2021-10-27 20:26:51 -03:00
Stas Kotykhin 3fc042af33 Move bar aggregators to utils, add time overloads & add check of orderType for Kraken (#5969)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* Move Aggregate Quote and Trade Bars to utils

* Add decimal and long overload to UnixTimeStampToDateTime

* Add Kraken OrderType check in CanSubmitOrder

* PR !5969 review fixes
2021-10-08 17:59:32 -03:00
Colton Sellers d2d99b1f10 Algorithm Sampling and Statistics Fixes (#5936)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Implement scheduled event sampling solution

* Use UTC time, only update daily portfolio value once a day

* For daily resolutions sample chart always

* Cleanup

* Drop resample daily all together

* Force final sample

* Regression updates

* FIx LiveResultHandler to update portfolio and benchmark values outside of sampling event

* Name the daily sampling event

* Address review pt 1

* Drop force and use reference wrapper

* Adjust tests

* Fix warning for Benchmark Timezone Misalignment and also add test

* Fix for daily resolution orders and test adjustments

* Also warn on universe settings with daily resolution

* Update missed regression

* Fix reference wrapper use

* Update regression after rebase

* Add values back in for Daylight Algo

* Have statistics builder skip day 1 performance

* Regression adjustments

* Test adjustments

* Update regression unit test

* Adjust some regressions starts to show performance values

* Add hourly algorithm for beta comparison

* Address missing Python regression changes

* Remove null comment
2021-10-05 19:31:25 -03:00
Martin-Molinero 4e6e0c85c6 Lean exchange improvements (#5932)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Lean exchanges improvements

- Adding new Exchange class to avoid exchange code clash.
  Adding and updating unit test

* Add Market for MapFile API

* Self review
2021-09-20 13:50:15 -03:00
Martin-Molinero b77f0122b2 Improve python exception parsing (#5831)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Improve python exception parsing

- Improve python exception parsing adding support for line shift. Adding
  unit tests

* PythonException revert change

* Centralized and normalize algorithm runtime handling

* Adding support for C# line and file exception report
2021-08-06 20:11:54 -03:00
Martin-Molinero e5a50109b4 DownloaderDataProvider improvements (#5730)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
- Ignore margin files requests. Only provided by ApiDataProvider
- Reduce noise by LeanData.TryParse() call on failure
2021-07-02 14:03:16 -03:00
Martin-Molinero 66552b49bd Add new DownloaderDataProvider (#5709)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Add new DownloaderDataProvider

- Moving LeanDataWrite and IDataDownloader to common project
- Add new BaseDataDownloaderDataProvider with concurrency helper method
- Add new DownloaderDataProvider which will use a IDataDownloader or
  IBrokerage implementation as data source

* Add support for downloading symbol chains data
2021-06-28 15:56:10 -03:00
Colton Sellers a882aadeef Research Fixes and Doc Updates (#5649)
Build & Test Lean / build (push) Has been cancelled
* Updates to default notebooks

* Update readme

* Notebook adjustments

* Drop launch script support

* Use our cwd if no QC dlls are in AppDomain directory

* Modify Intialize.csx to use parent directory only if QC files are detected there

* Clean up some details after testing

* Adjustments

* nit, NullOrWhiteSpace
2021-06-11 16:06:47 -03:00
Colton Sellers 0f0eeaff82 DataProvider Updates & Refactoring (#5641)
* Use DataProvider for fetching Shortable and FutureMargins files

* Use data provider for AllShortableSymbols

* Refactor of NeedToDownload logic

* Adjustments

* nit - cleanup

* Expand test cases

* Create the directory if it doesn't exist
2021-06-08 22:11:21 -03:00
Martin-Molinero 42c58d6e53 Ensure data is downloaded only once (#5635)
Build & Test Lean / build (push) Has been cancelled
- Ensure data is downloaded only once with the ApiDataProvider. Adding
  unit test
- Log invalid securityType ApiDataProvider requests once
- Assert Factor file and map file providers are compatible with the
  ApiDataProvider
2021-06-07 14:26:24 -03:00
Gerardo Salazar 94fb9293dc Adds filtering of negative ticks in CoinAPI converter consolidation (#5597)
Build & Test Lean / build (push) Has been cancelled
* Adds filtering of negative ticks in CoinAPI converter

  * Tick data will remain the same, only aggregated data
    will have changes reflected in its output as a result

* Adds suspicious tick filtering to ticks

* Address review: Add Crypto Trade Tick suspicious test case
2021-05-25 19:38:53 -03:00
Jasper van Merle bb8bd27fe5 Support 2-leg currency conversions (#5552)
Build & Test Lean / build (push) Has been cancelled
* Apply changes from #2146 on current master

Credit for most of these changes goes to https://github.com/viliwonka.
Rebasing three years of changes didn't go so well, so I manually
applied the changes in #2146 on the current master branch.

This commit attempts to represent the changes in #2146 as closely as
possible, no cleanup has been performed yet. Despite this, some changes
were inevitable because part of #2146 has been implemented another way
in the past three years.

* Clean up

* Process review comments

* Add tests

* symbol -> potentialConversionRateSymbol

* Ensure regression tests run/get data

* Fix broken test

* Process review comments

* Process review comments

* Simplify currency conversion logic

* Performance improvement, doc updates and test rename

* Rebase fixes

* Remove obsolete properties

Co-authored-by: Martin-Molinero <martin@quantconnect.com>
2021-05-20 13:19:39 -03:00
Gerardo Salazar 82c9b6ccb7 Updates and improvements to ToolBox projects (#5537)
Build & Test Lean / build (push) Has been cancelled
* Adds processed data directory to read price data from

* Make coarse universe generator look at data directory before failing to find daily data

* Set coarse generator output of missing daily file to debug log

* Add CoarseUniverseGenerator logs

* Fixes 100 nanosecond increment lookahead bias when parsing large numbers

  * Whenever we parse a number that is has precision greater than
    DateTime ticks (sub-100 nanoseconds), if we have nanoseconds
    between [0, 1000), excluding numbers divisible by 100,
    we will have leftover nanoseconds between [0, 100) nanoseconds, but
    they won't be factored in to the DateTime calculation, since casting
    to `long` only takes the integer component of the number, so we lose
    the extra nanoseconds that came with the decimal, and time is set to
    the "floored" value without those nanoseconds.

    Since .NET `DateTime` type has a limitation of only being able
    to represent time in increments of 100 nanoseconds, by not
    considering the sub-100 nanoseconds, we introduce a look-ahead
    bias of at most 100 nanoseconds/1 tick

  * Misc adjustment to make method use `decimal` instead of `double`
    for increased precision when parsing large numbers

* Changes CoinAPI data converter to support processing raw files in original directory structure and file name

  * Removes Market requirement from CoinAPI data converter

* Remove timeout on decompression of raw AlgoSeek futures data

* Updates SEC downloader to use HttpClient where requests were failing

  * For some unknown reason, valid requests to a valid URL were
    failing when using WebClient. Changing our requester to
    HttpClient fixes the issue, and enables us to leverage
    async capabilities where applicable.

  * Added fault tolerance to index file downloads, including a
    rate limit in case we've been rate limited

* Further refactoring; catches 429 errors, adds missing rategate calls

* Replace all usage of WebClient, force retry for all failures

* Adds optional config value for Benzinga News API key in downloader

* Modifies Estimize Downloader api config name and fixes directory not found bug

* Refactor Estimize to speed up processing time

  * Adds ticker limits if desired
  * Misc. bug fixes, performance improvements, code cleanup

* Remove debug log statements leftover from previous commit

* Add support for non-tick Index resolutions in LeanDataWriter

* Empty commit

* Empty commit

* Empty commit

* Empty commit

* Empty commit

* Empty commit

* Lower requests/second for SEC downloader, add missing rategate call

Co-authored-by: Martin-Molinero <martin@quantconnect.com>
2021-05-14 17:06:47 -03:00
Martin-Molinero f3c386663b Feature .net 5 (#5505)
* Update projects to use .NET 5.0, the successor to .NET Core

* Fix ambiguous errors. Add IBAutomator net5

* Remove FXCM

* Upgrade IBAutomater to v1.0.51

ignored, and an empty message aborts the commit.

* Fix rebase

- Fix ambiguous Index
- Remove StrategyCapacity.cs
- Update System.Threading.Tasks.Extensionsy

* Remove unrequired references

* Fixes

- Travis will use dotnet, not nunit nor mono
- Remove mono from foundation image
- Fix python setup in research
- Fix unit tests

* Don't call ReadKey when input is redirected

* Fix ConsoleLeanOptimizer

* Research fixes

* Update comment

* Add vsdbg to Dockerfile

* Fixes

- Revert dockerfile FROM custom changes
- Adjust and fix regression algorithms
   - Option assignment will be deterministic in the order
   - 'Rolling Averaged Population' is calculated using doubles, updating
     expected values.
- Update readme, removing references to mono
- Add missing Py.Gil lock

* Replace ICSharp with .NET Interactive

* Fixes after rebase

* CSharp research fixes

- Adding new Initialize.csx that pre loads all assemblies
- Adjusting template research file
- Moving steps in dockerfilejupyter
- Fix unit tests and regression tests after rebase

Co-authored-by: Gerardo Salazar <gsalaz9800@gmail.com>
Co-authored-by: Stefano Raggi <stefano.raggi67@gmail.com>
Co-authored-by: Jasper van Merle <jaspervmerle@gmail.com>
2021-05-06 17:23:51 -03:00
Martin-Molinero 83f9499b4a Option Margin Strategies (#5511)
* Refactor HasSufficientBuyingPowerForOrder implementations

Adds Sufficient and Insufficient helper methods to HashSufficientbuyingPowerForOrderParameters
enabling syntax like:

return paraeeters.Sufficient()
returnparameters.Insufficient(reason)

The next change will add the initial margin required which will simply require
updating both of these helper methods to accept the value.

* IBuyingPowerModel: Add margin functions Maintenance/Initial/ForOrder

These were originally hidden in an effort to only expose what's necessary
for the engine to perform its work. Additionally, we encapsulated all of
the method arguments into parameters classes to prevent having to break
anyone in the future. Not including these foundational methods turns out to
be an oversight. These methods are not required by the engine, but rather by
other models. Another possible solution here is to add an additional abstraction
and include these methods on this new abstraction. BuyingPowerModel would then
explicitly implement these methods and models that depend on them would require
two code paths, one for when the buying power model implements this interface
and another for when it doesn't.

Tests were additionally updated to remove test model implementations created for
the sole purpose of exposing these private methods.

* Add ConstantBuyingPowerModel

Provides an implementation of IBuyingPowerModel that returns the same
constant value

* Update BuyingPowerModelPythonWrapper to use reflection for method names

Having a bunch of hard-coded strings is a sure fire way for someone to
overlook when changing methods. This change ensures that noone needs to
remember that this code exists :)

Cleans up the syntax around verifying a python object implements a particular
C# interface via the ValidateImplementationOf<T> method by having it return a
value since the only use cases are in constructors when setting the models.

I was initially going to update ALL python wrappers to validate the passed
in models, but such a change could break many things that are 'working' right
now. Such an effort should be saved for its own dedicated PR.

* Add Parameters/Result types for new buying power model methods

* Support computing maintenance margin for arbitrary quantities

The existing GetMaintenanceMargin function assumes that we're only interested
in the maintenance margin for the entirety of the provided security's holdings.
This makes it impossible to perform what-if analysis or to even ask how much
maintenance margin is devoted to a particular subset of the security's holdings.
This change adds the quantity to the MaintenanceMarginParameters class. Futures
and Options models also depend on holdings cost and holdings value, so they have
also been added to the parameters type. Finally, static factory methods were
added to improve discernment of intent: ForCurrentHoldings provides the existing
behavior and then ForQuantityAtCurrentPrice to support what-if scenarios where
we're looking for the change in maintenance margin if we were to execute an order
for the securiy at the current time step. Obviously a constructor is provided to
set all of the values explicitly, using any price metric the caller desires.

* Address review

- Fix BPM xml documentation
- Fix python unit tests and PythonWrapper validate method

* Add SecurityHolding.QuantityChanged event

Adding event handlers will allow us to orchestrate complex
events from distant parts of the codebase through wiring
them up. If we continue down this path, it will move us away
from the current, very 'mechanical' data flows expressed in
LEAN and towards a more modern, event processing based system.
This is but a baby step in that direction and the initial use
case is using this QuantityChanged event to trigger resolution
of the algoritm's positions groups. This is part of an effort
to improve the fidelity of options margin modeling where we'll
model an OptionStrategy as an IPositionGroup. This will allow
us to compute the margin requirements of an OptionStrategy as
a unit instead of computing margin of each security individually
in isolation.

See #4065

* PortfolioManager: Group fields and remove unused field

This codebase generally places fields as the first members, but
this class had some fields at the top, then some properties, and
then some more fields. This change brings all the fields together
at the top of the file and also removes pointless comments placed
directly above some of the fields. Additionally, an unused field
was removed.

* Remove unused _currencyConverter from Security

Looks like at some point the only code using this member variable was removed
and the necessary clean up was overlooked.

* Add Parse.Enum functions

* Support disabling regression algorithms by language via config.json

Adds 'regression-test-languages' to config.json and filters regerssion algorithms to
run based on this value. When cycling on a particular feature, it's nice to be able
to run the entire regression set while ignoring the python algorithms. Once the C#
algorithms are all passing, one can then go back and run C# and Python in a final run,
since 99% of feature work doesn't impact python specifically.

* Implement IComparable in SecurityIdentitfier

This can be used to deterministically sort securities and symbols

* Add .editorconfig to enforce common formatting for json/sh files

* Fix typo in IBuyingPowerModel.GetBuyingPower xml docs

* Add ListEquals/GetListHashCode and OrderDirection.Closes(PositionSide)

ListEquals and GetListHashCode are designed to be used together as they
complement each other according to C#'s requirements for Equals and
GetHashCode functions.

PositionSide.ToOrderDirection() extension simply converts a PositionSide
to its logical equivalent OrderDirection. Long->Buy, Short->Sell, None->Hold

OrderDirection.Closes(PositionSide) determines if a particular OrderDirection
would have the effect of reducing a position's absolute size. This function
greatly improves the readability of buying power functions that must provide
adjustments when an order/contemplated trade reduces/closes an existing position.
OrderDirection.Buy.Closes(PositionSide.Short)
OrderDirection.Sell.Closes(PositionSide.Long)
All other combinations return false

Adds ToArray/ToImmutableArray convenience functions that combine a call
to Select followed by To(Immutable)Array all in one function call.

* Add decimal.DiscretelyRoundBy extension method

Supports rounding a decimal value by an arbitrarily chosen maximum precision,
or 'quanta'

* Update FutureMarginBuyingPowerModelTests to respect the security's lot size

* Add core position group classes and abstractions

* Add initial/maintenance margin support, buying power model consistency tests

* Add SufficientBuyingPower and GetReservedBuyingPower to position group model

Includes update to BrokerageTransactionHandler to use position group BPM for
sufficient buying power checks.

* Resolve position groups on each fill

We need to update the state of our position groups on each fill so that
we can properly handle multiple orders within the same time step. We
also limit the number of positions sent into the resolver by removing
securities without any holdings.

* fixup! Add SufficientBuyingPower and GetReservedBuyingPower to position group model

* Add GetMaximumLotsFor{Target|Delta}BuyingPower

Instead of computing order quantity, these functions compute the
maximum number of position group lots, which is the position group
quantity, and is guaranteed to be a whole number, for the provided
target/delta buying power parameters.

The SecurityPositionGroupBuyingPowerModel delegates to the security's
IBuyingPowerModel by applying a scaling factor equal to the security's
lot size.

This change also updates references to IBuyingPowerModel.GetMaximum...
to use the new position group model methods.

* Convert remaining IBuyingPowerModel call sites to position groups

* Rename PositionManasger.CreateDefaultGroup -> GetOrCreateDefaultGroup

Better describes its behavior

* Add Position Groups readme.md

* Add Option Strategy BuyingPowerModel

- Adding CompositePrositionGroupResolver and
  OptionStrategyPositionGroupResolver
- Adding OptionStrategyPositionGroupBuyingPowerModel handling option
  strategies based on IBs margin table. Adding regression algorithms
- Few changes so that option strategies executed by multiple orders are
  detected
- Adjust OptionStrategyDefinitionMatch to include equity legs in the
  matching result
- Minor tweaks fixing previous rebase
- Minor fixes for existing option strategies definitions, adding new
  missing strategies.
- Fixing minor bugs in option strategy matcher. Adding more unit tests

* Address self reviews

- Fixing bug in 'PositionGroupCollection'
- Few minor simplificaitons
- Adding BasicTemplateOptionEquityStrategyAlgorithm

* Address reviews

- Improve regression algorithms margin remaining and used assert logic to be exact. Taking into account spread and fees

Co-authored-by: Michael Handschuh <mhandschuh@gmail.com>
2021-04-30 18:45:27 -03:00
Colton Sellers 1c3d849ad5 Fix Warnings V2 (#5436)
* Reconcile duplicated code

* Add License header

* CS0219 Fixes: Value assigned, but never used

* CA1507: Use nameof in place of string literals

* CS0108 : Hides Inherited Member; Use new keyword to overwrite formally

* CS0114: Hides inherited member; use override keyword

* CS0168: Variable is declared but never used

* Tests CS1062; using obsolete implicit Symbol -> String; fix via .ToString()

* CS0472: Non Nullable Obj getting Null Checked

* CS0067 Member not used; ignore all cases for future use

* CS00162 : Unreachable code; either removed or ignored for debugging and test cases

* CS0169 Remove non-used fields; ignore those that may be used in future

* CS0414; Field is assigned but never used.

* CS0618; Obsolete properties and members; Only fixes simple ones, rest will have to broken up

* CS0649; Field never assigned too

* CS0659 & CS0661 ; Overwrite operators and equals but not hashcode; I don't really override it but just call base

* Small comment fix

* Cleanup pragma statement
2021-04-02 11:20:01 -07:00
Colton Sellers e2a0873b7c Fix Lean Warnings V1 (#5408)
Cleanup all non-breaking warnings
2021-03-22 11:08:48 -07:00
Gerardo Salazar 4c085ff853 Adds Indexes and Index Options asset types (Backtesting/Live, IB only) (#5379)
* Add support for Index SecurityType  🚀 (#5364)

* Add Index SecurityType  🚀

* Extend SecurityIdentifier & Lean Data classes with Index support

* Add Index SecurityType  🚀

* Extend SecurityIdentifier & Lean Data classes with Index support

* Fixes

* Added index cross basic template demonstration

* WIP: Prototype index security type for LEAN as non tradable asset

* Re-adds Index entries to MHDB after rebase

* First steps to getting Index Options running

  * Looks at any instance where we pattern match for an option type
    and replaces it with a generic call to `.IsOption()` for easier
    extensibility in the future for additional option security types

  * Adds IndexOption security and misc. classes

  * Misc. changes, mainly related to any sort of special casing of
    equity options and made index options take the same path

* Enables index options data for backtesting

  * Adds new index options market hours to MHDB
  * Misc. bug fixes for index options
  * WIP: add live support for index options and indexes
  * Use OptionMarginModel for Index Options because they both use the
    same calculation for margin requirements

* Fixes contract not found errors on SPX index options and SPX index in IB

  * Turns out index options' last trading day is the day before expiry,
    which IB was expecting the last trading day.

* Add index option test cases (temp)

* LiveOptionChainProvider fix, use Symbol vs. ticker

  * Description updates to regression algorithms

* Fixes bug in live trading for indexes and index options

  * Adds overridable minimum price variation symbol property
  * Adds variable sized minimum price variation for index options
  * Adjusts symbol properties for index options
  * Misc. bug fixes

* Fixes option assignment simulation for European options

  * Updates index options regression algorithms (WIP)

* Fixes bug where index option exercise would trade index underlying

  * Fixes bugs where SecurityType.Index was getting flagged as tradable

* Regression algorithms updates and addresses review

  * Misc. style fixes and refactoring + a few bug fixes
  * Updates regression algorithms to run without runtime errors
  * Adds data for regression algos

* Sets DefaultOptionStyle on Canonical and support index options

* Update regression algos statistics

* Removes bad line in regression algorithm causing build to fail

* Minor tweaks

* Address review add comment about quoteBar parse scale

Co-authored-by: Balamurali Pandranki <balamurali@live.com>
Co-authored-by: Jared Broad <jaredbroad@gmail.com>
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
2021-03-12 20:46:23 -03:00