* Initialize securities on re-addition
* Minor unit tests fixes
* Reset Index.ManualSetIsTradable when security is removed
* More changes to clean up and simplify
* Address security re-utilization only
* Cleanup regression algorithms
* Add algorithm unit tests
* Add Brokerage unit tests
* Add Common unit tests
* Fix unit test bugs
* Add remaining unit tests
* Address requested changes
* Address requested changes
* Fix failing unit test
* Address requested changes
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
* Portfolio state chart
- Cleanup and renames
- Add series.IndexName. Uodating unit tests
- Add Chart.LegendDisabled. Adding new unit tests
- Add ChartPoint.Tooltip. Updating unit tests
- Minor compression tweak. Adding unit test
- Add ChartJsonConverter. Adding unit tests
* Minor chart serialization order tweak
* Refactor portfolio state sampling and storing
* Move PortfolioMargin into a lean side chart
Python Virtual Environments / build (push) Has been cancelled
API Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Implement backtest self-naming and self-tagging
* Minor unit test fix
* Minor api changes
* Minor changes
* Send packet messages when algorithm's name or tags are updated
* Minor fixes
* Address peer review
* Backtest name and tags API calls
Plus, first version of CI for API tests
* Add endpoint to update a backtest set of tags
* Minor GH actions change
* Address peer review
* Add algorithm naming and tagging unit tests
* Minor backtest API changes
* Make statistics available at runtime to algorithms
* Re-calculate statistics on every call
* Housekeeping
* Add regression algorithms
* Address peer review
* Support for custom summary statistics at runtime
* Minor changes
* Address peer review
* Address peer review
* Minor changes
* Minor changes
Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Refactor alpha statistics
- Refactor alpha statistics, cleaning up and simplifying no longer required calculations and scoring
- Adding new InsightEvaluator abstraction, adding C# & PY regression
algorithms
* Optimization backtest result json converter update
* Address reviews
- Remove IAlphaHandler, move insight storage responsability to IResultHandler
and centralizing insight collection on the QCAlgorithm.Insights to be
reused by the framework models
- Fix portfolio turnover single day backtests and duplicate time
sampling handling. Updating regression algorithms
* Add InsightCollection tests and minor fixes
* Adding more & improved tests
Python Virtual Environments / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* CryptoFutures adjustments
- Address reviews of CryptoFutures:
- Add new Slice MarginInterestRates collection
- Add new regression tests asserting funding rate application behaves
the same no matter the resolution
- Add auxiliary data by type into the security cache
- Revert BuyingPowerModel changes
* Make interest rate application time deterministic
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* remove un-used
* use utc time for slice sync
* use utc time
* refactor
* add regression test
* use utc time
* use utc time
* make utctime required parameter
* add utcTime in slice creation
* assert warm up complete
* check if algorithm is still warmingup
* use exchange tz
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Implement scheduled event sampling solution
* Use UTC time, only update daily portfolio value once a day
* For daily resolutions sample chart always
* Cleanup
* Drop resample daily all together
* Force final sample
* Regression updates
* FIx LiveResultHandler to update portfolio and benchmark values outside of sampling event
* Name the daily sampling event
* Address review pt 1
* Drop force and use reference wrapper
* Adjust tests
* Fix warning for Benchmark Timezone Misalignment and also add test
* Fix for daily resolution orders and test adjustments
* Also warn on universe settings with daily resolution
* Update missed regression
* Fix reference wrapper use
* Update regression after rebase
* Add values back in for Daylight Algo
* Have statistics builder skip day 1 performance
* Regression adjustments
* Test adjustments
* Update regression unit test
* Adjust some regressions starts to show performance values
* Add hourly algorithm for beta comparison
* Address missing Python regression changes
* Remove null comment
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Alpha holdings state
- Alpha result packet will optionally provide the algorithms portfolio
state
* Rename
* Convert AlphaStreamsPortfolio to data source
* Improvements on AlphaStreams algorithm
* Fix regression tests
* Add unit tests for EW AS PCM and fixing bugs
* Protobuf AlphaStreamsPortfoliot staState
- Protobuf AlphaStreamsPortfolioState. Adding unit tests
- Add variable TPV tests for EW ASPCM
* Add alpha license to Organization response
* Improvements EW AS PCM respects free portfolio value
* Fixes
- Update tests expected statistics results affected by MHDB custom data timezone fix
- Fix for Extensions.IsCustomDataType
* Fixed and adding more regression tests
- Adding support and regression test with alpha consumer with different account currency
- Adding support and regression test of a universe adding custom data
types
- Add support and regression test for algorithm alpha consumer with existing holdings
* Add AlphaStreamsOrderEvent data type
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Add IResultHandler.BrokerageMessage
- Result handlers can receive notifications for brokerage message events
* Make LiveTradingResultHandler.RuntimeError virtual
Build & Test Lean / build (push) Has been cancelled
* Use lean data key as param for request
* key -> filePath rename and some cleanup
* Refactor
* Add Organizations Endpoints
* Add some organization api wrapper objects
* Address namespace issue
* Reorganize Api Test into seperate files using one ApiTestBase
* Add Organization tests
* Use capitalized "API" test namespace to reduce amount of file changes
* Add License to test base
* Update /data endpoint functions and response objects
* Update ApiDataProvider Logic
* Handle deserialization of organization products
* Simplify converter
* Only throw for equity requests when not subscribed to map/factor files
* Add missing header
* Make arguement exception
* Api adjustments
* Add Zip factor and map file providers
- Common project will now reference Compression project and not the other way
round.
- Adding Zip FactorFile and MapFile providers
* Refactor FactorFileProvider to use DataProvider to fetch files
* Use resulting MinimumDate in construction of FactorFile
* Nit FactorFile comments and arrangement
* Refactor MapFileProviders to use DataProvider for fetching files
* Refactor ZipFileProvider
* Clean up
* Refactor Backtesting Future/Option chain providers to use dataprovider
* Fixes for data/ endpoints and test adjustments
* Response objects adjustments/cleanups
* ApiDateProvider fixes and testing
* Add LocalZipFactorFileTests
* Update ApiDataProvider download test to verify stream is not null
* Implement posting of agreement summary and signed time
* Mark all Api related tests as explicit and document details on running
* Clarify default token on ApiTestBase
* Adjust summary
* Update Api responses for QCC, except org products which are sold in USD
* Implement cache expiration for zip MapFile and FactorFiles. Adding unit tests
* Fix multiple markets for ZipFactorFile provider
* Use Symbol as cache key
* Api.cs review
* Dispose of factorFileStream after reading
* Use zip.EntryFileNames
* Address a few reviews
* Few more fixes
* Address Api Review
* Add Job Org id to config
* Minor tweaks
* Compare with invariant culture
* Fixes Option Universe selection
* ZipEntryNameSubscriptionDataSourceReader will use IDataProvider
* Fix research
* Fix null reference exception
* Make duplicate log debug
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Add internal subscription manager
- Add InternalSubscriptionManager that will handle internal
Subscription. Replaces the realtime updates
- Fix thread race condition in the TimeTriggeredUniverseSubscription, we
have one thread injecting data points, the main algorithm thread, and
the base exchange is pulling from it
- Fixes for FakeDataQueue
- Adding unit tests
* Address reviews and fixes
- Internal subscription will use extended market hours
- Only sample charts accordingly
- Get api-url once
- `UniverseSelection` and `Research` will always use
`AlgorithmHandler.DataProvider` instance instead of
`DefaultDataProvider`
- Remove Compression at test/app.config since the dll isn't required
- Add missing license header
- Adding new DataPermissionManager that will own the existing
datachannelProvider. Will assert configurations before added to the
data feed. Adding unit tests
* Removed Sample[a-zA-Z]+ methods from IResultHandler definition
* Converted Sample[a-zA-Z]+ methods from public to protected
* Updated inheritors of BaseResultHandler to use new accessibility
modifiers
* Removes useless code in ResolutionSwitchingAlgorithm
* Refactors AlgorithmManager loop
* Refactors StatisticsBuilder methods and strategy for series alignment
* Move sampling logic to the corresponding IResultHandler
* Changes benchmark resolution to Resolution.Hour
* Modifies IResultHandler to enable external sampling
* Adds BacktestResultHandler unit tests
* Adds ResolutionSwitchingAlgorithm to test misalignment
* Adds support to AlgorithmRunner to store algorithm IResultHandler
Warning: this commit breaks accurate calculations for algorithms that
only make use of `Daily` resolution data. Previously, because
the benchmark was added in Daily resolution in backtesting, any
algorithm that only made use of daily data would have an accurate
calculation for beta and various other statistics.
These changes serve to fix the statistics calculations of non-daily
resolution algorithms, with daily resolution to be revisited at a later
time.
- Adding `SecurityCacheProvider` this class allows for two different
`Security` to share the same data type cache through different instance
of `SecurityCache`. This is used to directly access custom data types
through their underlying in a peformant maner
- Some small improvements
We restrict each algorithm time loop to a pre-determined amount of time.
Exceeding this limit will cause the algorithm to immediately terminate.
This quickly becomes an issue when considering users running trainable
models that have a long initialization period that exceeds the time loop
maximum.
This change provides a mechanism through which a long-running scheduled
event is permitted to keep running and is permitted to avoid the time loop
permitted by requesting additional time. Requests for additional time are
limited according to a leaky bucket implementation whose parameters are
set via the job's controls structure. The fundamental time unit for the
algorithm is a single minute.
Here's how it works. If a scheduled event takes longer than one full wall
clock second then a request is made to the leaky bucket for one more minute.
If the scheduled event continues to take more time, it will continue to
request additional minutes. Each requested minute will prevent the algorithm's
time loop check from terminating the algorithm. When the bucket is empty and
no more minutes are available to be requested, a TimeoutException is thrown
causing a cascade that ends in the algorithm's termination and status being
flipped to RuntimeError.
Additionally, this applies equally to ALL scheduled events. While some helpers
were added with the naming of Train and TrainNow to the ScheduleManager, these
methods don't do anything special and the infrastructure doesn't otherwise
flag them as different, so this feature becomes part of the core Scheduled
Event feature set.
Further, the live scheduled events were not touched and are still pending
further discussion regarding the value added by enforcing a time restriction
when simulation time and wall clock time are equivalent.
Fixes#3319
Adds IRegisteredSecurityDataTypesProvider to track all the data types
registered in the algorithm. Using this data, we can detect if it's
possible that we'll eventually have a property of a certain type name.
For example, consider I wish to use security.Data.TradeBar but we haven't
received any trade bars yet. Before this change a KeyNotFoundException
would be raised, but since we can determine that we expect to have trade
bars, we can detect this and return an empty list when we haven't received
any data yet. This also removes the need to constantly do a HasData<T>()
check before accessing the dynamic members.
Closes#3620
- Setting SPY as the default security benchmark
- The security benchmark subscription will be added at `UniverseSelection`
as an internal subscription. Using its own dedicated Security instance
which doesn't live in the algorithms.Securities collection.
- Reducing algorithms exposure to internal subscriptions
- `TimeSliceFactory` will prioritize higher resolution bars, when same
symbol is present twice (for non-internal subscriptionst)
- Adding regression test `CustomUniverseWithBenchmarkRegressionAlgorithm`
- Adding `IRealTimeHandler.OnSecurityChanged()` will be used to update
the `OnEndOfDay` security related scheduled events
- Adding `BaseRealTimeHandler.cs` to reduce code duplication in the
`Backtesting` and `LiveTrading` `RealTimeHandlers`
- Adding CSharp and Python regression tests
- Deprecating `OnEndOfDay()` callback because of two reasons, mainly
because Python does not support two methods with the same name, but also
because different assets have different market close times.
- `ScheduledEvents` set at the same time will now be deterministic
- Removing `using QCAlgorithmFramework = QuantConnect.Algorithm.QCAlgorithm`
- Removing `QCAlgorithmFrameworkBridge`
- Removing `IsFrameworkAlgorithm`
- Making `EmitInsightBasedOnFill` private. Adding new
`IOrderEventProvider` exposing an `event` to which `QCAlgorithm` will
subscribe.
- `AccountType.Cash` algorithms will be allowed to manually trade and
emight insights manually or with alpha model.
- Moving `UniverseSelection.EnsureCurrencyDataFeeds` call into the
`IResultHandler` implementation through usage of the new `SetupHandlerHelper`
class, that will also set an initial conversion rate if none present.
- Adding regression test, that reproduces original issue
- Removing unneeded `CashBook` instance to create a new `TimeSlice`
- Adding new `TimeSliceFactory`, an instance base class that will
provide methods to create a new `TimeSlice`. Will own the `DateTimeZone`
property.
- Adding new `ISecurityPrice` and `IOptionPrice` that will provide a
reduced interface for accessing price properties and methods used when
creating a new `TimeSlice`
- Completly move `DataManager` in front of `DataFeed`. Specifically
`AddSubscription()` and `RemoveSubscription()` implementations. Also
removing IDataFeed.Subscriptions
- Creating new `Synchronizer` which will consolidate and combine `TimeSlice`
streaming for both live and backtests modes. Will live in front of the
`DataManager`
- `SubscriptionSynchronizer` will be the `FrontierTimeProvider` exposed,
and owned, by the new `Synchronizer`
- Updating existing tests
- Adding new ISecurityService and its implementation SecurityService.
Expose by SecurityManager.
This class will expose a method for creating new securities. The
SecurityManager is exposing this new interface, calling _securityService
internally, so Future/OptionUniverseSelectionModel.cs can use it
- Replacing all usages of SecurityManager.CreateSecurity for new
ISecurityService
- Modifying `Cash.cs` and `CashBook.cs` `EnsureCurrencyDataFeeds()` to
return newly added `SubscriptionDataConfig` instead of `Security`. This
will avoid using `Security.Subscriptions` at call site.
- Moving old SecurityManager.CreateSecurity into new
SecurityServiceTests.cs
This bug was only recently introduced via PR #2592. The issue is that the
PaperBrokerage's Scan method is invoked twice per time loop. This change
adds tracking to ensure we only invoke the dividend detection/application
on the first Scan invocation of a time loop. Tests added to confirm behavior.
Refactoring previous commit so DataManager only keeps and receives a
reference to IAlgorithmSettigs, with the objective of reducing tight
coupling
Note: Investigate if IAlgorithmSettings.DataSubscriptionLimit should limit subscriptions or unique securities.
Today its using SubscriptionManagerSubscriptions.Where(x => !x.Symbol.IsCanonical()).DistinctBy(x => x.Symbol.Value).Count() @DataManager
JosueNina