* Updates CPU performance metrics gathering for project modernization
* PerformanceCounters are not supported cross-platform in
.NET Core/5.0, which requires the use of an alternative method
in the gathering of performance metrics. Since no exposed .NET API
exists to gather these metrics without blocking, a new thread is
created to block the time necessary for the CPU performance to be
calculated.
* Address review: remove ResetEvent and make CpuPercent atomic
* Dispose of CpuPerformance instance before exiting Lean
* Minor tweak for Task and comment
Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
* Some app.config files were kept to either enable features
that are only enable-able via the config files, or binding redirects
for DLLs that are not packaged with NuGet (i.e. VisualStudio
package, required to run tests successfully @ 15.0.0.0)
* Atreyu Brokerage initial setup, and changes required for integration with main Atreyu project
* Address reviews
Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
* Migrate to newer CSProj format; build succesful
Rebase onto master
* Maintain binaries directory
* Reinstate FSharp default condition
* Remove reference to packages.config
* Test project fixes
* Remove nuget.config and "Solution Items"
* Remove repetitive assembly info
* Reinstate default compiles and cleanup redundant settings
* Reinstate any specific settings that defaults are unclear for
* Reinstate Framework.nuspec
* Make test internal classes public for composer use; undo assembly change
Rebase onto master part 2
* Address review; restore some settings
* Restore RootNamespace and AssemblyName attributes
* Use NugetPackageRoot instead of NugetPackageFolders
* Reinstate "RestorePackages" setting
* Transfer compilied Python files to launcher and tests bin
* Remove conflicting settings
* Fix WebSocketSharpFork references
* Fix FSharp Core reference
* Upgrade TestPlatform.ObjectModel to 16.8.3, and redirect V11 binding
* Add Mono.Cecil package
* Address review
* Fixed Python runtime issue that was occurring when trying to generate reports locally on OSX/mono, but assume the issue impacts all configurations.
* Move Python.Runtime config to common
* Remove duplicate files
* Update readme
* Typo
* Change destination in build directory
Co-authored-by: Charles Naccio <cnaccio@gmail.com>
* initial commit
* run parametrized algorithm with command line parameters
* skeleton: top level structure
* OptimizationNodePacket scheme
* pass parameters as HashSet
* run Lean and read results
* call method on optimization completion
* refactor public interfaces
- close ParameterSet collection; allow only get operations
- explicit method to start LeanOptimizer
* synchronize RunLean method; the result could come in before the backtest id is set in the collections
* another portion of refactoring and interface changes
* comments
* comments & tests for Extremum, Minimization and Maximization classes
* unify optimization paramater values (min, max, step) & mode GridSearch tests
- swap min&max if necessary
- iterate left => right (negate step value if necessary) & provide default step value if step == 0
- no StackOverflow Exception
- parameterSet Id should be global for current generator and retain between steps
- test signle point boundary (min == max)
* BruteForceStrategy tests
* more comments
* Update Optimizer assembly information
- Update Optimizer projects assembly information to match behavior of
the other projects
* Tweaks
- Adding comments
- Replace OnComplete for Ended event
- Replace Abort for Dispose
- ConsoleLeanOptimizer will keep track of running processes
- Each backtest will store results in a separated directory, so they
don't fight for the log.txt file.
- Adding cmdline option for lean to close automatically
- Adding concurrent execution backtest limit
- Console optimizer will start Lean minimized
- Escape spaces in Json path
* remove parameter set generator abstraction layer
we don't need this flexibility now.
* refactor public methods; Step shouldn't be public
* constraints: wip
* define contract
* comparison operators and tests
* specify JsonProperty values
* Move SafeMultiply100 to extensions
* Throw exception on failed Optimizer.Start
* constraints: wip
* change finish & dispose process
* minor fixes
- handle force lean abort
- notify consumer if target has been reached
* target & constraints; adapt unit tests
* Minor Tweaks and fixes
- Some logging improvements
- Remove Public since not required
* Ignore empty ParameterValue
* simplify condition
* avoid reinitialization
* reduce type; force immutable
* unit tests for constraints and target value
* parse & normalize percent values, i.e. 20% => 0.2
* fixup
* Target & Constraint & OptimizationNodePacket unit tests
* Add more json unit tests
- Adding more json conversion unit tests. Fix bug for Extremum which
wasn't using the converter.
* LeanOptimizer tests
* Estimation results
* User thread safe counters
* LeanOptimizer unit tests; push OptimizationResult on Ended event
* more unit tests
* Minor tweaks
-Estimate ToString in a single line.
-Typos and missing header file
* Add base SendUpdate method
- Add base SendUpdate method for LeanOptimizer
* fix LeanOptimizer test; rely on internal Update rather than timer
* Add OptimizationStatus
- Add missing commments and OptimizationStatus
* EulerSearch implementation: wip
* OptimizationParameter custom converter
* change the type
* make step optional
* change folder structure
* enumerate optimization parameter using IEnumerable & IEnumerator
* unit tests: parameters & objectives
* unit tests: strategies
* remove redundant TODO
* change Euler search boundaries
* more Euler tests
* prevent race condition
* Add account/read endpoint
- Adding account/read endpoint. Adding unit test
* Add status check before running lean
* Minor self review
- Adding missing comments, minor changes
* remove array parameters
* minor changes
- tidy up config file, rename variable
- accept min less or equal than max
* move OptimizationParameter methods to strategies
* Minor improvements for BaseResultHandler derivates
* minor changes
- strict requirements for Step and MinStep values
- strategy specific settigs
* Add TotalRuntime to estimate
Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
* Add property IBrokerage.AccountBaseCurrency
* Set AccountCurrency to brokerage AccountBaseCurrency
* Remove USD AccountCurrency check
* Fix Oanda account base currency
* Fix currency symbol in CashBook.ToString()
* Fix unit tests
* Address review
* Add DebugMessage when changing account currency
* Add debug message for brokerage account base currency
* Fix currency symbol in equity chart and runtime statistics
* Update unit tests
* Binance Brokerage skeleton
* Market hours
* Implement Symbol Mapper
- known symbols available on /api/v1/exchangeInfo
- fiat currencies are pegged
* Implement GetCashBalance
* Implement GetAccountHoldings
- there are no pre-existing currency swaps
- cash balances are pulled and stored in the cashbook
* Implement GetOpenOrders
* Manage orders: PlaceOrder
* Manage orders: UpdateOrder
Update operation is not supported
* Manage orders: CancelOrder
* Messaging: order book
* Messaging: trades
* Messaging: combine streams
- connect to fake /ws/open channel on init
- case by channel name, but not event type
* Messaging: order depth updates
- ticker symbol is not enough as it pushes updates only once a second, this would be a very incomplete data stream
- fetch ticker snapshot if lastUpdateId == 0
- follow Binance instructions for keeping local orderbook fresh
* Messaging: user data streaming
- Request userDataStream endpoint to get listenKey
- keep listenkey alive
- handle order close event
- handle order fill event
* DataDownloader: get history
- we can aggregate minute candles for higher resolutions
* fix data stream
* Tests: FeeModel tests
* Tests: base brokerage tests
* Tests: download history
* Tests: symbol mapper
* Support StopLimit andd StopMarket orders
* StopMarket orders disabled
Take profit and Stop loss orders are not supported for any symbols (tested with BTCUSDT, ETHUSDT)
* Tests: StopLimit order
* Tests: crypto parsing
* Reissue user data listen key
* comment custom currency limitation
* rework websocket connections
* implement delayed subscription
* adapt ignore message
* add license banner
* use better suited exception type
* avoid message double parsing
* support custom fee values
* extract BinanceApiClient to manage the request/response between lean and binance
* use api events to invoke brokerage events
* do not allow to terminate session if it wasn't allocated.
* update binance exchange info
* tool to add or update binance exchange info
* ExchangeInfo basic test
* Rebase + Resharp
* Binance brokerage updates
- Fix sign bug in sell order fills
- Fix bug in GetHistory
- Remove duplicate symbol from symbol properties db
* Remove unused code
* Revert removal of account currency check
* Update symbols properties database
* Address review
* Address review
- Upgrade API endpoints from v1 to v3
- Updated sub/unsub for new subscription manager
- Subscribe best bid/ask quotes instead of full order book
- Added handling of websocket error messages
- Cleanup + refactor
* Update symbol properties database
* Remove list from symbol mapper
* Fix symbol mapper tests
* Address review
- Fix resubscribe after reconnect
- Fix quote tick edge case
* Fix EnsureCurrencyDataFeed for non-tradeable currencies
* Fix check in EnsureCurrencyDataFeed
* Reuse base class subscribe on reconnect
Co-authored-by: Adalyat Nazirov <aenazirov@gmail.com>
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Add PolygonDataQueueHandler
* Add history provider and downloader for Polygon
* Add aggregator to PolygonDataQueueHandler
* Address review
- Removed duplication in message classes
- Added public Subscribe/Unsubscribe methods in PolygonWebSocketClientWrapper
- Added history requests for Forex and Crypto
* Address review
- Add security type and market arguments to downloader
- Fix time zone bug in downloader
* Remove unnecessary locks
* Add Polygon history for all resolutions
- Equity: trades and quotes
- Forex: quotes only
- Crypto: trades only
* Live Coarse universe refactor
- Live trading will source Coarse and Fine fundamental data directly
from disk. Updating unit tests.
* Adds ILiveDataProvider interface
* Adds wrapper for IDataQueueHandler implementations
* Replaces IDataQueueHandler with ILiveDataProvider in
LiveTradingDataFeed
* Edits IDataQueueHandler documentation
* Maintains aggregation for current IDQH impls and skips for ILDF impls
* Note: No unit test was created for this method, go back and TODO
* Protobuf Market data
- Adding protobuf support for Ticks, TradeBars and QuoteBars. Adding
unit tests.
* Adds unit tests for LiveDataAggregator changes
* Fixes bug where custom data was not handled as it was before
* Fixes race condition bug because of variable reuse in class
* Add protobuf extension serialization
* Fixes for protobuf serialization
* Refactor
* Fix OptionChainUniverse
* replace BaseDataExchange pumping ticks with consolidators
* AlpacaBrokerage
* BitfinexBrokerage
* GDAXBrokerage
* OandaBrokerage
* InteractiveBrokers
* TradierBrokerage
* FxcmBrokerage
* PaperBrokerage
* etc
* WIP fixes for existing LTDF unit tests
* Fixes more LTDF unit tests
* make IDataAggregator.Update recieving Generic BaseData rather than Tick
* Change IDataQueueHandler.Subscribe method
* Some fixes after adding new commits
* Adds protobuf (de)serialization support for Dividend and Split
* Serialize protobuf with length prefix
* Fix missing LTDF unit tests
* Adds TiingoNews protobuf definitions
* fix comments
* more fixes on IQFeedDataQueueHandler
* disallow putting ticks into enumerator directly
* ScannableEnumerator tests
* fix OandaBrokerage
* AggregationManager unit tests
* fix AlpacaBrokerage tests
* fix InteractiveBrokers
* fix FxcmBrokerage tests
* call AggregationManager.Remove method on unsubscribe
* fix GDAX existing tests
* Fixes, refactor adding more tests for AggregatorManager
* Adds BenzingaNews protobuf definitions and round trip unit test
* Adds missing TiingoNews unit test to Protobuf round trip tests
* Improve sleep sequence of LiveSynchronizer
* need start aggregating first, and then can subscribe
* More test fixes and refactor
- Refactoring AggregationManager and ScannableEnumerator so the last is
the one that owns the consolidator
- Adding pulse on the main LiveSynchronizer
* Improve performance of LEquityDataSynchronizingEnu
* Add missing Set job packet method
* Minor performance improvements
* Improvements add test timeout
- Improvements adding test timeout to find blocking test in travis
* Improve aggregationManager performance
* Testing improvements for travis
* Remove test timeouts
* More test fixes
- Adding more missing dispose calls and improving determinism
* fix IEXDataQueueHandler and tests
* Final tweaks to LTDF tests
* more AggregationManager tests
* consume and log ticks
* fix test: couldn't subscribe to Forex tickers
* change Resolution for all bar configs
* Improve RealTimeScheduleEventServiceAccuracy
* refactoring: move common code to base class
* fixed bug; unsubscribe SubscriptionDataConfig
* Small performance improvement
* Minor fixes
* Avoid Symbol serialization
* Fixes coarse selection in live mode
* Fix for live coarse
* Adds protobuf (de)serialization support for Robintrack
* Adds round-trip unit test
* Minor performance improvements
* More minor performance improvements
* pass LiveNodePacket through to OandaBrokerage
* Fixes empty list becoming null value when deserializing with protobuf
* Reverts BZ live trading exception removal and fixes tests
* Refactor WorkQueue making it abstract
* Add try catch for composer
* Adds optional data batching period to LiveFillForwardEnumerator
* Override data-queue-handler with config
* Improve PeriodCountConsolidator.Scan performance
* Move batching delay to main Synchornizer thread
* Reverts addition of Robintrack protobuf definitions
* Give priority to config history provider if set
* Add Estimize protobuffing
- Add Estimize protobuffing support. Adding unit tests
* Always dispose of data queue handler
Co-authored-by: Gerardo Salazar <gsalaz9800@gmail.com>
Co-authored-by: Adalyat Nazirov <aenazirov@gmail.com>
There's no reason for algorithms to have direct access to this implementation.
Moving this into the engine prevents algoriths from directly accessing LocalObjectStore
and instead can only reference it through the IObjectStore abstraction
persistenceIntervalSeconds defines the number of seconds between
each save operation. For the local object store, this dictates
how often the contents of the object store is packaged and written
to disk. The PersistData virtual method is provided for subclasses
to provide a different implementation of how/where to persist the
data. The change to be in-memory aims at keeping the object store
performant with reasonable persistence guarantees.
- Adding `IDataChannelProvider` that will be used to determine if
streaming should be used
- Adding `data-channel-provider` for `config.json`
- Adding unit tests
We restrict each algorithm time loop to a pre-determined amount of time.
Exceeding this limit will cause the algorithm to immediately terminate.
This quickly becomes an issue when considering users running trainable
models that have a long initialization period that exceeds the time loop
maximum.
This change provides a mechanism through which a long-running scheduled
event is permitted to keep running and is permitted to avoid the time loop
permitted by requesting additional time. Requests for additional time are
limited according to a leaky bucket implementation whose parameters are
set via the job's controls structure. The fundamental time unit for the
algorithm is a single minute.
Here's how it works. If a scheduled event takes longer than one full wall
clock second then a request is made to the leaky bucket for one more minute.
If the scheduled event continues to take more time, it will continue to
request additional minutes. Each requested minute will prevent the algorithm's
time loop check from terminating the algorithm. When the bucket is empty and
no more minutes are available to be requested, a TimeoutException is thrown
causing a cascade that ends in the algorithm's termination and status being
flipped to RuntimeError.
Additionally, this applies equally to ALL scheduled events. While some helpers
were added with the naming of Train and TrainNow to the ScheduleManager, these
methods don't do anything special and the infrastructure doesn't otherwise
flag them as different, so this feature becomes part of the core Scheduled
Event feature set.
Further, the live scheduled events were not touched and are still pending
further discussion regarding the value added by enforcing a time restriction
when simulation time and wall clock time are equivalent.
Fixes#3319
Adalyat Nazirov