Commit Graph

99 Commits

Author SHA1 Message Date
Adalyat Nazirov a4f66628fd Lean Optimization interface in QCAlgorithm (#4923)
* initial commit

* run parametrized algorithm with command line parameters

* skeleton: top level structure

* OptimizationNodePacket scheme

* pass parameters as HashSet

* run Lean and read results

* call method on optimization completion

* refactor public interfaces

- close ParameterSet collection; allow only get operations
- explicit method to start LeanOptimizer

* synchronize RunLean method; the result could come in before the backtest id is set in the collections

* another portion of refactoring and interface changes

* comments

* comments & tests for Extremum, Minimization and Maximization classes

* unify optimization paramater values (min, max, step) & mode GridSearch tests

- swap min&max if necessary
- iterate left => right (negate step value if necessary) & provide default step value if step == 0
- no StackOverflow Exception
- parameterSet Id should be global for current generator and retain between steps
- test signle point boundary (min == max)

* BruteForceStrategy tests

* more comments

* Update Optimizer assembly information

- Update Optimizer projects assembly information to match behavior of
  the other projects

* Tweaks

- Adding comments
- Replace OnComplete for Ended event
- Replace Abort for Dispose
- ConsoleLeanOptimizer will keep track of running processes
- Each backtest will store results in a separated directory, so they
  don't fight for the log.txt file.
- Adding cmdline option for lean to close automatically
- Adding concurrent execution backtest limit
- Console optimizer will start Lean minimized
- Escape spaces in Json path

* remove parameter set generator abstraction layer

we don't need this flexibility now.

* refactor public methods; Step shouldn't be public

* constraints: wip

* define contract

* comparison operators and tests

* specify JsonProperty values

* Move SafeMultiply100 to extensions

* Throw exception on failed Optimizer.Start

* constraints: wip

* change finish & dispose process

* minor fixes

- handle force lean abort
- notify consumer if target has been reached

* target & constraints; adapt unit tests

* Minor Tweaks and fixes

- Some logging improvements
- Remove Public since not required

* Ignore empty ParameterValue

* simplify condition

* avoid reinitialization

* reduce type; force immutable

* unit tests for constraints  and target value

* parse & normalize percent values, i.e. 20% => 0.2

* fixup

* Target & Constraint & OptimizationNodePacket unit tests

* Add more json unit tests

- Adding more json conversion unit tests. Fix bug for Extremum which
  wasn't using the converter.

* LeanOptimizer tests

* Estimation results

* User thread safe counters

* LeanOptimizer unit tests; push OptimizationResult on Ended event

* more unit tests

* Minor tweaks

-Estimate ToString in a single line.
-Typos and missing header file

* Add base SendUpdate method

- Add base SendUpdate method for LeanOptimizer

* fix LeanOptimizer test; rely on internal Update rather than timer

* Add OptimizationStatus

- Add missing commments and OptimizationStatus

* EulerSearch implementation: wip

* OptimizationParameter custom converter

* change the type

* make step optional

* change folder structure

* enumerate optimization parameter using IEnumerable & IEnumerator

* unit tests: parameters & objectives

* unit tests: strategies

* remove redundant TODO

* change Euler search boundaries

* more Euler tests

* prevent race condition

* Add account/read endpoint

- Adding account/read endpoint. Adding unit test

* Add status check before running lean

* Minor self review

- Adding missing comments, minor changes

* remove array parameters

* minor changes

- tidy up config file, rename variable
- accept min less or equal than max

* move OptimizationParameter methods to strategies

* Minor improvements for BaseResultHandler derivates

* minor changes

- strict requirements for Step and MinStep values
- strategy specific settigs

* Add TotalRuntime to estimate

Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
2020-12-02 20:10:40 -03:00
Martin Molinero 9631cd6749 Fix nuget QC dependency version 2020-05-21 16:21:59 -03:00
Martin Molinero 09c17b71ed Add assembly description to nuspec
- Due to bug in nuget failing to replace description and copyright token, setting
  .nuspec description to match assembly description
2020-05-21 14:42:12 -03:00
Martin-Molinero 1d0d4f4d7b Add assembly descriptions (#4442)
Add assembly descriptions
2020-05-21 09:35:12 -07:00
Martin Molinero 891cf4d69b Update to .Net 4.6.2 2020-05-15 17:32:46 -03:00
Martin Molinero ed6e2fd0a2 Replace ResultPreFix config for custom algorithm id 2020-03-09 11:59:23 -03:00
Gerardo Salazar d3a34dcd70 Fixes issue where report creator could not be ran in Mac OSX
* Adds `PythonInitializer.SetPythonPathEnvironmentVariable(...?)` method
2020-02-11 15:01:47 -08:00
Martin Molinero 87bc03d71e Address reviews
- When BacktestNodePacket has the inital `CashAmount` set we will clear
all existing cash amounts and set the account currency
- Adding more unit tests
2019-12-31 13:25:58 -03:00
Stefano Raggi e0a08cbb8d Add XML documentation generation for release builds 2019-12-14 13:50:10 +01:00
Martin Molinero 118c1e49cc Add data channel provider
- Adding `IDataChannelProvider` that will be used to determine if
streaming should be used
- Adding `data-channel-provider` for `config.json`
- Adding unit tests
2019-11-01 14:53:56 -03:00
Martin Molinero 9ec1a41235 Performance improvements
- `PortfolioTargetCollection` avoid calling `Count` on
ConcurrentDictionary directly -> has to take all locks
- `SecurityChanges` change Union for Concat since constructor will call
HashSet
- Make `DynamicSecurityData` hold lazy data objects
- `RegisteredSecurityDataTypesProvider` avoid looping over all
registered types, adding `TryGetType`
- `Security.Update()` will no call group by on data since this data is
already grouped by type. Adding `ContainsFillForwardData` will allows to
be lazy and not re loop through the data unless necessary
- `DefaultAlphaHandler` will use the `static`
`Enumerable.Empty<Insight>` instance when possible
- `SubscriptionSynchronizer` will be lazy to construct the
`universeData` dictionary which is not used in most of the times. Will
use `Count` vs `Any` -> `Count` is known by the dictionary
- For python algorithms `JobQueue` will respect `AlgorithmLocation`, was
using unexisting `"algorithm-path-python"`
2019-10-17 10:26:37 -03:00
Michael Handschuh 30c8772a7e Update Compression/Jupyter/Launcher/Queues/UserInterface to respect CA1304 and CA1305
Updates all occurrences of parsing/ToString-ing to go through the new
StringExtensions methods that use CultureInfo.InvariantCulture

See #3045
2019-09-12 15:30:44 -04:00
Michael Handschuh c7df4e9fb3 Add StringExtensions and Parse utilizing CultureInfo.InvariantCulture
Adds static methods of the form Parse.<TypeName>(string str) that use
CultureInfo.InvariantCulture. These are to be used when parsing strings.
It's still safe (from the CA1304/CA1305 perspective) to use the ToDecimal
extension method for decimals.

Adds string extension methods for common operations that will now require
CultureInfo.InvariantCulture. These are to be used when converting values
to strings, such as ToStringInvariant()/ToStringInvariant(format), but also
useful for searching within strings, StartsWithInvariant, EndsWithInvariant
 and IndexOfInvariant.
2019-08-16 18:08:28 -03:00
Michael Handschuh 703f915182 Configure QuantConnect.ruleset in all projects
The initial ruleset is specifically aimed at addressing #3045. When we'd like
to start addressing other issues available via the FxCopAnalyzers, we can update
the action in QuantConnect.ruleset. A nice way to do it is to first set it to
warning, then go through the solution, project-by-project, fixing each warning.
At the end you can flip it to error and ensure it still builds. Moving forward,
any changes will fail if they violate the rule.

In support of #3045, we'll be configuring the following rules:
> CA1304: https://docs.microsoft.com/en-us/visualstudio/code-quality/ca1304-specify-cultureinfo?view=vs-2017
> CA1305: https://docs.microsoft.com/en-us/visualstudio/code-quality/ca1305-specify-iformatprovider?view=vs-2017
2019-08-16 18:08:18 -03:00
Michael Handschuh c42057809d Add Microsoft.CodeAnalysis.FxCopAnalyzers to all projects
FxCop has various rulesets for enforcing things within our codebase.
For this particular issue, we'll be enforcing CA1304 and CA1305 to
ensure we're always using an IFormatProvider or a CultureInfo where
applicable.

Linked Issue: #3045
2019-08-16 18:08:05 -03:00
Martin Molinero f887c42638 Enable python and CSharp debugging
- Adding `DebuggerHelper` class, handles debugging initialization
- Setting the "PYTHONPATH" will be handled by the `JobQueue`
2019-08-12 22:03:13 -03:00
Gerardo Salazar 2033c7873b Update outdated DotNetZip dependency in nuspec files 2019-07-02 18:02:23 -07:00
Martin Molinero f6b5e92a96 Add conditional optimization flag 2019-06-19 16:11:15 -03:00
Martin Molinero 17995f4c86 Adding a configuration for maximum data points per chart series. Limit will only apply for backtesting result handler 2018-08-17 18:14:33 -03:00
Stefano Raggi 2fe41495d1 Set QC API UserToken for GDAX 2018-08-13 23:03:55 +02:00
Stefano Raggi 483f20c78a Change vars from static to instance in JobQueue
When running consecutive algorithms, this was causing the algorithm runner to look for the wrong file name.
2018-07-03 23:15:34 +02:00
Juan José D'Ambrosio bda5b11cc2 Nuspec files added for all projects.
Launcher project is packed as `QuantConnect.Lean`but as a mean ot having all Lean features just calling one package. 

In the same sense, `QuantConnect.Algorithm.CSharp` is included as package and added as dependecy in the `QuantConnect.Lean` package just to have a working example aailable out-of-the-box.
2018-04-13 07:28:34 -03:00
Andrew e116709b24 Specify <LangVersion> of 6 in all csproj files 2017-08-07 15:27:26 -04:00
Andrew 6d67dbd2a2 Added ILeanManagement.Update() method 2017-07-14 12:40:53 -04:00
Andrew f7bd4f59b7 Remove commands from Lean 2017-07-14 12:40:53 -04:00
Stefano Raggi d88d82af12 Upgrade Json.NET to version 10.0.3 2017-07-07 23:21:47 +02:00
Stefano Raggi e463ccf088 Upgrade all projects to target .NET Framework 4.5.2
In the advanced build settings, Language version was also upgraded from C# 5.0 to C# 6.0. The minimum Visual Studio version required will now be VS2015.
2017-07-07 22:49:17 +02:00
Stefano Raggi b9ea48e1b4 Add IApi parameter to IJobQueueHandler.Initialize 2017-06-14 17:11:28 +02:00
jaredbroad cd56b5323d Tidy up the local loading of pyc files 2017-04-30 13:10:27 -04:00
Victor Baybekov de21307a32 Update Json.Net to 9.0.1 2017-03-14 01:35:59 +03:00
jaredbroad 9de8d3a68b Merge branch 'seed-security-from-history' of https://github.com/andrewhart098/Lean into seeding 2016-12-02 00:00:03 -05:00
quant1729 f859c548f0 Fixed typos, bugs 2016-11-21 16:58:22 +01:00
quant1729 9dd5b8f4ef Updated engine: it gets history provider, data queue handler from the job now. JobQueue loads data from config. 2016-11-21 16:58:12 +01:00
jaredbroad 41dab29bc1 Shuffle ram allocation to controls; update job queues and engine usage 2016-11-21 04:43:09 -05:00
Ray Bohac e74b14227e moves Control symbol limits config to read into vairable, then assigns in either LiveNodePacket of BacktestNodePacket 2016-10-17 20:43:13 -04:00
Ray Bohac ab127aaf04 increases symbol limit on Lean local and also adds a config option to alter the limits 2016-10-17 12:31:18 -04:00
jaredbroad dd8a6442b6 Tidy unneccessary TravisCI build configuration 2016-04-19 19:51:51 -04:00
jaredbroad e93006b6a5 Added Travis CI release configuration and disabled views from building in linux 2016-04-15 12:44:37 -04:00
jaredbroad 2670b41be6 Bug fixes and event onload for when embed url is loaded 2016-04-11 20:06:25 -04:00
jaredbroad 632c21c99c New UX for LEAN, launched from Launcher using embedded HTML browser 2016-04-11 18:29:50 -04:00
Michael Handschuh 3610d3fc8c Jobqueue.AcknowledgeJob: Ensure final message makes it to console
If you're running without a ConsoleLogHandler then the cmd window just looks like it is dead, but really it's already finished and is waiting for Console.Read to return, but there was no message since the Log.Trace was only being piped to a file
2016-04-05 15:52:27 -04:00
Juan Manuel Alvarez b3494afc47 Constants renamed to Globals.
Globals values can be reloaded from configuration.
2016-03-12 01:07:31 -03:00
AlexCatarino 871f03cbe9 JobQueue.Language is no longer a static field
Consecutive calls of JobQueue set up with different languages did not acknowlegde the changes when Language was a static field
2016-02-22 20:08:18 -03:00
Michael Handschuh 40d47d5523 Set c# language version to 5.0 2016-01-29 17:41:20 -05:00
Michael Handschuh 8acb8504be Cleans up JobQueue job init parameters 2016-01-19 18:12:28 -05:00
snugs 5d6d48f7c8 Make ICommandQueueHandler : IDisposable 2015-12-18 11:57:28 -05:00
snugs e31fd9d990 Read parameters from config in JobQueue 2015-12-10 13:45:38 -05:00
snugs be950e756a Log full errors/stack traces
Many places in the code used Log.Error(err.Message) or equivalent which
strips out all the really useful information, such as the stack trace
and inner exceptions. Using Log.Error(exception) is the correct way to
log an error as it will correctly write all the message details, also,
by passing the full Exception object we can improve the logging in this
one place and all call sites will automatically benefit from the improvements
2015-12-02 12:35:33 -05:00
snugs 7c2261f172 Removed warnings as errors for obsoletes
This is causing issues for mono users, seems to be something different
in the compilers (mono vs msft) with method resolution in ambiguos situations.
2015-11-24 18:56:21 -05:00
snugs 0ae568d980 Remove warnings as errors from Release
This is a temporary fix until we figure out why travis complains about implicits but visual studio doesn't
2015-11-23 15:45:24 -05:00