130 Commits

Author SHA1 Message Date
Martin-Molinero b7a21ce4be Update to net6 (#6311)
* Update to net6

* Bump pythonNet to 2.0.13
2022-05-03 11:45:55 -03:00
Adalyat Nazirov 0c26d42561 Feature 2839 black scholes data generator (#6135)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* replace to local functions as they are more performant

* fix random generator upper bound

Next() includes minValue, but not maxValue, so we increment it +1

* introduce abstract layers

* refactoring

* fix tets

* adapt tests

* fixup

* implement blackschole price model for options

* use risk free rate

* use ql price model

* wip

* change interface

* fix

* tidy up the code

* wip

* iterate groupped symbols

* wip

* wip

* fix

* allow symbol of different types

* improve settings

* wip

* iterate full range

* fix issue with negative option

* fix

* fixup

* use StandardDeviationOfReturnsVolatilityModel

* re-use existing tick types per security type

* parametrize underlying security type

* use default option style

* dynamic option price model

* fix enumeration

* test

* fix unit tests

* refactor code

* remove unused file

* minor tweaks and refactoring

* rename symbol generator class

* fix interface

* add comments

* more comments and unit tests

* more tests

* add disclaimer

* more tests

* more comments and tests

* split tests into different files

* tidy up the code

* tidy up the code; more tests

* refactor TickGenerator => use security price directly on each iteration

* remove dupe; reuse main constructor

* use SecurityManager, refactor code

* bugfix: save ticks in history array

* check volatility warm up & tests

* more unit tests

* describe volatility period span in settings

* rename command line option

* Minor adjusments. Address review

- Use Lean log handler instead of writting directly to console
- Rename BlackShcolesPriceGenerator to generically OptionPriceModelPriceGenerator
- Minor format clean up & standarization
- Add support for specifying the option chain size

* Rename TickGenerator private fields

* Fix unit tests

* fix tests class name

* Support tickers being specified

Co-authored-by: Martin-Molinero <martin@quantconnect.com>
2022-01-10 17:21:03 -03:00
Omid K. Rad 2557a36feb Bug: Config.TryGetValue returns true if key is not found (#6128)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Fix typo

* Fix TryGetValue to return false if key is not found

* Revert "Fix TryGetValue to return false if key is not found"

This reverts commit b85b7b579aa722e1fec2599afbeee61163334184.

* Update documentation for TryGetValue
2021-12-28 12:19:46 -03:00
Adalyat Nazirov 46ce138fa1 move methods for further usage across other brokerages (#5981)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
2021-10-14 17:08:11 -03:00
Martin-Molinero 46829f0f01 Dependency clean up (#5834)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
2021-08-11 12:39:00 -03:00
Gerardo Salazar 650ce2e2b5 Removes References To Built-in Custom Data Implementations (#5793)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* WIP removal of custom data references & tests updates

* Regression algos updated and python algorithms moved to DataSource repos

* Fixes failing unit tests

* Add "LiveDataTypes" field to LiveNodePacket

* Adds Initialize() to IDataChannelProvider

  * Adds new extension method to convert
    HistoryRequest -> SubscriptionDataConfig

* Address review: Add protobuf definitions for Iconic data types

* Address review: re-adds DynamicSecurityDataAlgorithm as regression algo

  * Small adjustments to variable naming and documentation

* Move test files to respective DataSource repos
2021-08-02 19:24:45 -03:00
Colton Sellers 41f9d9da94 Remove SecurityInitializer from Universe Selection (#5418)
Build & Test Lean / build (push) Has been cancelled
* Attempt to remove SecurityInitializer from UniverseSelection stack entirely

* Fix broken Python Universe Selection model

* Remove mistake change (meant for another branch)
2021-05-28 14:53:13 -03:00
Martin-Molinero 9e96aeb440 Language update. Csproj clean up (#5590)
Build & Test Lean / build (push) Has been cancelled
- Update CSharp language version to latest 9
- Remove old csproj build configurations
2021-05-24 11:10:03 -03:00
Martin-Molinero f3c386663b Feature .net 5 (#5505)
* Update projects to use .NET 5.0, the successor to .NET Core

* Fix ambiguous errors. Add IBAutomator net5

* Remove FXCM

* Upgrade IBAutomater to v1.0.51

ignored, and an empty message aborts the commit.

* Fix rebase

- Fix ambiguous Index
- Remove StrategyCapacity.cs
- Update System.Threading.Tasks.Extensionsy

* Remove unrequired references

* Fixes

- Travis will use dotnet, not nunit nor mono
- Remove mono from foundation image
- Fix python setup in research
- Fix unit tests

* Don't call ReadKey when input is redirected

* Fix ConsoleLeanOptimizer

* Research fixes

* Update comment

* Add vsdbg to Dockerfile

* Fixes

- Revert dockerfile FROM custom changes
- Adjust and fix regression algorithms
   - Option assignment will be deterministic in the order
   - 'Rolling Averaged Population' is calculated using doubles, updating
     expected values.
- Update readme, removing references to mono
- Add missing Py.Gil lock

* Replace ICSharp with .NET Interactive

* Fixes after rebase

* CSharp research fixes

- Adding new Initialize.csx that pre loads all assemblies
- Adjusting template research file
- Moving steps in dockerfilejupyter
- Fix unit tests and regression tests after rebase

Co-authored-by: Gerardo Salazar <gsalaz9800@gmail.com>
Co-authored-by: Stefano Raggi <stefano.raggi67@gmail.com>
Co-authored-by: Jasper van Merle <jaspervmerle@gmail.com>
2021-05-06 17:23:51 -03:00
Colton Sellers 1c3d849ad5 Fix Warnings V2 (#5436)
* Reconcile duplicated code

* Add License header

* CS0219 Fixes: Value assigned, but never used

* CA1507: Use nameof in place of string literals

* CS0108 : Hides Inherited Member; Use new keyword to overwrite formally

* CS0114: Hides inherited member; use override keyword

* CS0168: Variable is declared but never used

* Tests CS1062; using obsolete implicit Symbol -> String; fix via .ToString()

* CS0472: Non Nullable Obj getting Null Checked

* CS0067 Member not used; ignore all cases for future use

* CS00162 : Unreachable code; either removed or ignored for debugging and test cases

* CS0169 Remove non-used fields; ignore those that may be used in future

* CS0414; Field is assigned but never used.

* CS0618; Obsolete properties and members; Only fixes simple ones, rest will have to broken up

* CS0649; Field never assigned too

* CS0659 & CS0661 ; Overwrite operators and equals but not hashcode; I don't really override it but just call base

* Small comment fix

* Cleanup pragma statement
2021-04-02 11:20:01 -07:00
Martin-Molinero 4632840da9 Add license file for project packages (#5314) 2021-02-18 18:57:36 -03:00
Martin-Molinero 9232427663 Remove all nuspec files (#5311) 2021-02-18 17:13:11 -03:00
Martin-Molinero df6e133833 Fixes for assembly information (#5306)
- Update version to 2.5
- Normalize version and assembly information
- Remove commented and unrequired code
2021-02-17 16:42:23 -03:00
IlshatGaripov d17fe34c74 Fixing up Coin API streaming (#5295)
* fixing up Coin API streaming

* Fixing file name

* Adding log showing CoinApi product plan

Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
2021-02-15 09:55:00 -03:00
Martin-Molinero a0f39ec6a4 Remove Microsoft.Net.Compilers (#5282)
* Remove Microsoft.Net.Compilers

* Remove deprecated analyzer packages

- Replace deprecated analyzer packages

* Travis will use dotnet for building
2021-02-11 09:31:37 -03:00
Gerardo Salazar 254d0896f1 Replaces Microsoft.Extensions.CommandLineUtils with McMaster CLI utils (#5255)
* Replaces Microsoft.Extensions.CommandLineUtils with McMaster CLI utils

  * Updates ValueTuple to 5.0.0 for compatibility with package.

* Retrigger build

* Retrigger build
2021-02-03 19:56:14 -03:00
Colton Sellers 472896c2d0 Update JSON Library (#5218)
* Update NewtonSoft.Json from 10.0.3 to 12.0.3

* Remove JsonSerializer.Populate(), values are already populated
2021-01-28 19:24:49 -03:00
Gerardo Salazar 79c544d3a2 Reduces output log size and warnings emitted when building with mono's msbuild (#5200) 2021-01-21 11:09:16 -03:00
Colton Sellers f2fd10d9d2 Build Configuration Hotfix (#5186)
* Reflect properties prior to migration onto projects

* Do not prefer 32bit

* Reinstate SelectedOptimization var
2021-01-19 09:38:16 -03:00
Colton Sellers a2a874e333 CSProj Update (#5145)
* Migrate to newer CSProj format; build succesful

Rebase onto master

* Maintain binaries directory

* Reinstate FSharp default condition

* Remove reference to packages.config

* Test project fixes

* Remove nuget.config and "Solution Items"

* Remove repetitive assembly info

* Reinstate default compiles and cleanup redundant settings

* Reinstate any specific settings that defaults are unclear for

* Reinstate Framework.nuspec

* Make test internal classes public for composer use; undo assembly change

Rebase onto master part 2

* Address review; restore some settings

* Restore RootNamespace and AssemblyName attributes

* Use NugetPackageRoot instead of NugetPackageFolders

* Reinstate "RestorePackages" setting

* Transfer compilied Python files to launcher and tests bin

* Remove conflicting settings

* Fix WebSocketSharpFork references

* Fix FSharp Core reference

* Upgrade TestPlatform.ObjectModel to 16.8.3, and redirect V11 binding

* Add Mono.Cecil package

* Address review
2021-01-14 12:55:09 -08:00
Adalyat Nazirov a4f66628fd Lean Optimization interface in QCAlgorithm (#4923)
* initial commit

* run parametrized algorithm with command line parameters

* skeleton: top level structure

* OptimizationNodePacket scheme

* pass parameters as HashSet

* run Lean and read results

* call method on optimization completion

* refactor public interfaces

- close ParameterSet collection; allow only get operations
- explicit method to start LeanOptimizer

* synchronize RunLean method; the result could come in before the backtest id is set in the collections

* another portion of refactoring and interface changes

* comments

* comments & tests for Extremum, Minimization and Maximization classes

* unify optimization paramater values (min, max, step) & mode GridSearch tests

- swap min&max if necessary
- iterate left => right (negate step value if necessary) & provide default step value if step == 0
- no StackOverflow Exception
- parameterSet Id should be global for current generator and retain between steps
- test signle point boundary (min == max)

* BruteForceStrategy tests

* more comments

* Update Optimizer assembly information

- Update Optimizer projects assembly information to match behavior of
  the other projects

* Tweaks

- Adding comments
- Replace OnComplete for Ended event
- Replace Abort for Dispose
- ConsoleLeanOptimizer will keep track of running processes
- Each backtest will store results in a separated directory, so they
  don't fight for the log.txt file.
- Adding cmdline option for lean to close automatically
- Adding concurrent execution backtest limit
- Console optimizer will start Lean minimized
- Escape spaces in Json path

* remove parameter set generator abstraction layer

we don't need this flexibility now.

* refactor public methods; Step shouldn't be public

* constraints: wip

* define contract

* comparison operators and tests

* specify JsonProperty values

* Move SafeMultiply100 to extensions

* Throw exception on failed Optimizer.Start

* constraints: wip

* change finish & dispose process

* minor fixes

- handle force lean abort
- notify consumer if target has been reached

* target & constraints; adapt unit tests

* Minor Tweaks and fixes

- Some logging improvements
- Remove Public since not required

* Ignore empty ParameterValue

* simplify condition

* avoid reinitialization

* reduce type; force immutable

* unit tests for constraints  and target value

* parse & normalize percent values, i.e. 20% => 0.2

* fixup

* Target & Constraint & OptimizationNodePacket unit tests

* Add more json unit tests

- Adding more json conversion unit tests. Fix bug for Extremum which
  wasn't using the converter.

* LeanOptimizer tests

* Estimation results

* User thread safe counters

* LeanOptimizer unit tests; push OptimizationResult on Ended event

* more unit tests

* Minor tweaks

-Estimate ToString in a single line.
-Typos and missing header file

* Add base SendUpdate method

- Add base SendUpdate method for LeanOptimizer

* fix LeanOptimizer test; rely on internal Update rather than timer

* Add OptimizationStatus

- Add missing commments and OptimizationStatus

* EulerSearch implementation: wip

* OptimizationParameter custom converter

* change the type

* make step optional

* change folder structure

* enumerate optimization parameter using IEnumerable & IEnumerator

* unit tests: parameters & objectives

* unit tests: strategies

* remove redundant TODO

* change Euler search boundaries

* more Euler tests

* prevent race condition

* Add account/read endpoint

- Adding account/read endpoint. Adding unit test

* Add status check before running lean

* Minor self review

- Adding missing comments, minor changes

* remove array parameters

* minor changes

- tidy up config file, rename variable
- accept min less or equal than max

* move OptimizationParameter methods to strategies

* Minor improvements for BaseResultHandler derivates

* minor changes

- strict requirements for Step and MinStep values
- strategy specific settigs

* Add TotalRuntime to estimate

Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
2020-12-02 20:10:40 -03:00
Martin-Molinero e8c316cbcf Fix Toolbox tickers parsing (#4876)
- Fix ToolBox tickers parsing, adding unit test.
2020-10-20 11:27:41 -03:00
Adalyat Nazirov f4679785a5 Pass command line parameters values as string (#4819)
* avoid preliminary typing command lines arguments

* unit tests: Config.Get can parse and cast values

* unit tests: parse command line args and return string values

* unit test: parameter attribute converter

* merge&parse unit test
2020-10-06 10:19:56 -03:00
Martin-Molinero 9cdb4a91c5 Refactor live data feed (#4636)
* Live Coarse universe refactor

- Live trading will source Coarse and Fine fundamental data directly
  from disk. Updating unit tests.

* Adds ILiveDataProvider interface

  * Adds wrapper for IDataQueueHandler implementations

  * Replaces IDataQueueHandler with ILiveDataProvider in
    LiveTradingDataFeed

  * Edits IDataQueueHandler documentation

* Maintains aggregation for current IDQH impls and skips for ILDF impls

  * Note: No unit test was created for this method, go back and TODO

* Protobuf Market data

- Adding protobuf support for Ticks, TradeBars and QuoteBars. Adding
  unit tests.

* Adds unit tests for LiveDataAggregator changes

  * Fixes bug where custom data was not handled as it was before
  * Fixes race condition bug because of variable reuse in class

* Add protobuf extension serialization

* Fixes for protobuf serialization

* Refactor

* Fix OptionChainUniverse

* replace BaseDataExchange pumping ticks with consolidators

* AlpacaBrokerage

* BitfinexBrokerage

* GDAXBrokerage

* OandaBrokerage

* InteractiveBrokers

* TradierBrokerage

* FxcmBrokerage

* PaperBrokerage

* etc

* WIP fixes for existing LTDF unit tests

* Fixes more LTDF unit tests

* make IDataAggregator.Update recieving Generic BaseData rather than Tick

* Change IDataQueueHandler.Subscribe method

* Some fixes after adding new commits

* Adds protobuf (de)serialization support for Dividend and Split

* Serialize protobuf with length prefix

* Fix missing LTDF unit tests

* Adds TiingoNews protobuf definitions

* fix comments

* more fixes on IQFeedDataQueueHandler

* disallow putting ticks into enumerator directly

* ScannableEnumerator tests

* fix OandaBrokerage

* AggregationManager unit tests

* fix AlpacaBrokerage tests

* fix InteractiveBrokers

* fix FxcmBrokerage tests

* call AggregationManager.Remove method on unsubscribe

* fix GDAX existing tests

* Fixes, refactor adding more tests for AggregatorManager

* Adds BenzingaNews protobuf definitions and round trip unit test

* Adds missing TiingoNews unit test to Protobuf round trip tests

* Improve sleep sequence of LiveSynchronizer

* need start aggregating first, and then can subscribe

* More test fixes and refactor

- Refactoring AggregationManager and ScannableEnumerator so the last is
  the one that owns the consolidator
- Adding pulse on the main LiveSynchronizer

* Improve performance of LEquityDataSynchronizingEnu

* Add missing Set job packet method

* Minor performance improvements

* Improvements add test timeout

- Improvements adding test timeout to find blocking test in travis

* Improve aggregationManager performance

* Testing improvements for travis

* Remove test timeouts

* More test fixes

- Adding more missing dispose calls and improving determinism

* fix IEXDataQueueHandler and tests

* Final tweaks to LTDF tests

* more AggregationManager tests

* consume and log ticks

* fix test: couldn't subscribe to Forex tickers

* change Resolution for all bar configs

* Improve RealTimeScheduleEventServiceAccuracy

* refactoring: move common code to base class

* fixed bug; unsubscribe SubscriptionDataConfig

* Small performance improvement

* Minor fixes

* Avoid Symbol serialization

* Fixes coarse selection in live mode

* Fix for live coarse

* Adds protobuf (de)serialization support for Robintrack

  * Adds round-trip unit test

* Minor performance improvements

* More minor performance improvements

* pass LiveNodePacket through to OandaBrokerage

* Fixes empty list becoming null value when deserializing with protobuf

* Reverts BZ live trading exception removal and fixes tests

* Refactor WorkQueue making it abstract

* Add try catch for composer

* Adds optional data batching period to LiveFillForwardEnumerator

* Override data-queue-handler with config

* Improve PeriodCountConsolidator.Scan performance

* Move batching delay to main Synchornizer thread

* Reverts addition of Robintrack protobuf definitions

* Give priority to config history provider if set

* Add Estimize protobuffing

- Add Estimize protobuffing support. Adding unit tests

* Always dispose of data queue handler

Co-authored-by: Gerardo Salazar <gsalaz9800@gmail.com>
Co-authored-by: Adalyat Nazirov <aenazirov@gmail.com>
2020-08-18 20:21:10 -03:00
Martin Molinero 8a63a1ac32 Rename Jupyter to Research add nuget
- Rename Jupyter to Research and add it's nuget package
2020-05-22 12:42:59 -03:00
Martin Molinero 9631cd6749 Fix nuget QC dependency version 2020-05-21 16:21:59 -03:00
Martin Molinero 09c17b71ed Add assembly description to nuspec
- Due to bug in nuget failing to replace description and copyright token, setting
  .nuspec description to match assembly description
2020-05-21 14:42:12 -03:00
Martin-Molinero 1d0d4f4d7b Add assembly descriptions (#4442)
Add assembly descriptions
2020-05-21 09:35:12 -07:00
Martin Molinero 891cf4d69b Update to .Net 4.6.2 2020-05-15 17:32:46 -03:00
Martin Molinero 269b8ba204 Remove PsychSignal data
- Remove PsychSignal data since it's no longer available
2020-05-11 18:48:13 -03:00
Martin Molinero ed6e2fd0a2 Replace ResultPreFix config for custom algorithm id 2020-03-09 11:59:23 -03:00
Adam May c0c5bda6e9 Remove insights-destination-folder 2020-03-08 17:15:32 +11:00
Adam May f56d2a9db3 Adding results-destination-folder and results-destination-file-prefix config options 2020-03-04 20:15:33 +11:00
Adam May 98412fcbef Add insights-destination-folder to allow for insights to be stored in a custom directory 2020-03-04 17:01:49 +11:00
Adam May 381105b1df Have Config.Write() refer to config being used. 2020-03-04 15:47:36 +11:00
Gerardo Salazar 1c4935fc60 Address review - increase performance and refactor
* Deletes NullAlphaHandler, NullLeanManager, NullSynchronizer
* Calculate the backtest and live PointInTimePortfolios only once now
* Refactor Metrics calculations
* Add missing license headers to some files
* Reverts accessibility of AddToUserDefinedAlgorithm to private
* Other misc. fixes and cleanup
2019-12-23 14:13:28 -08:00
Jared Broad f66a682f63 Debug commit 2019-12-18 16:08:40 -08:00
Stefano Raggi e0a08cbb8d Add XML documentation generation for release builds 2019-12-14 13:50:10 +01:00
Gerardo Salazar 37fd362c8b Add processedFilesDirectory argument
* Changed to read existing indexes from `processedFilesDirectory`
2019-11-19 17:27:52 -08:00
Gerardo Salazar e4789e0b23 Implement Benzinga News Downloader
* Make Benzinga News Converter more efficient (and cleaner!)
* Implement JSON converter for serialization/deserialization process
* Fix bug where duplicate symbols would be added to news.Symbols
* Modify algorithm strategies
2019-11-19 10:28:48 -08:00
Gerardo Salazar 629de49b10 Implement Benzinga custom data as ToolBox application and BaseData class 2019-11-13 13:38:36 -08:00
Martin Molinero 65737297c7 Adding Tiingo new data converter 2019-10-01 00:26:36 -03:00
Gerardo Salazar 1e10cf7f7e Fixes issue in SmartInsider where we could not choose where the final processed directory is. 2019-09-25 15:06:25 -07:00
Michael Handschuh f9ddb316e6 Update Configuration to respect CA1304 and CA1305
Updates all occurrences of parsing/ToString-ing to use CultureInfo if
possible. This project can not reference QuantConnect.Common, so it's
unable to use the new StringExtensions.

See #3045
2019-09-12 15:30:44 -04:00
Michael Handschuh c7df4e9fb3 Add StringExtensions and Parse utilizing CultureInfo.InvariantCulture
Adds static methods of the form Parse.<TypeName>(string str) that use
CultureInfo.InvariantCulture. These are to be used when parsing strings.
It's still safe (from the CA1304/CA1305 perspective) to use the ToDecimal
extension method for decimals.

Adds string extension methods for common operations that will now require
CultureInfo.InvariantCulture. These are to be used when converting values
to strings, such as ToStringInvariant()/ToStringInvariant(format), but also
useful for searching within strings, StartsWithInvariant, EndsWithInvariant
 and IndexOfInvariant.
2019-08-16 18:08:28 -03:00
Michael Handschuh 703f915182 Configure QuantConnect.ruleset in all projects
The initial ruleset is specifically aimed at addressing #3045. When we'd like
to start addressing other issues available via the FxCopAnalyzers, we can update
the action in QuantConnect.ruleset. A nice way to do it is to first set it to
warning, then go through the solution, project-by-project, fixing each warning.
At the end you can flip it to error and ensure it still builds. Moving forward,
any changes will fail if they violate the rule.

In support of #3045, we'll be configuring the following rules:
> CA1304: https://docs.microsoft.com/en-us/visualstudio/code-quality/ca1304-specify-cultureinfo?view=vs-2017
> CA1305: https://docs.microsoft.com/en-us/visualstudio/code-quality/ca1305-specify-iformatprovider?view=vs-2017
2019-08-16 18:08:18 -03:00
Michael Handschuh c42057809d Add Microsoft.CodeAnalysis.FxCopAnalyzers to all projects
FxCop has various rulesets for enforcing things within our codebase.
For this particular issue, we'll be enforcing CA1304 and CA1305 to
ensure we're always using an IFormatProvider or a CultureInfo where
applicable.

Linked Issue: #3045
2019-08-16 18:08:05 -03:00
Gerardo Salazar 436a72f435 Implements SmartInsider custom data source
Implements Smart Insider data converter ToolBox application

Includes example algorithms demonstrating use of data
2019-08-05 13:42:08 -07:00
Gerardo Salazar d394251ac9 Implement U.S. Treasury yield curve alternative data source 2019-07-24 16:48:55 -07:00
Gerardo Salazar 4a83793067 Address review 2019-07-09 17:46:15 -07:00