* Calculate both raw and adjuasted prices for backtesting
* disable second price factoring
* move and reuse method
* test coverage for new methods
* reuse scaling method
* reuse subscriptionData.Create method
* removed unused code
* regression test
* switch to aapl
* fix regression test output
* more asserts
* fix comments - reduce shortcuts and abbrevation
* more comments
* merge parameters
* reduce number of getting price factors
* fix tests
* fix tests
* fix regression tests
* calculate TotalReturn on demand
* include TotalReturn calculations
* perf tuning
* more unit tests for SubscriptionData.Create
* simplify things - store and return only raw and precalculated data
* fix regression tests; change it back
* factor equals 1 for Raw data
* small changes
* follow code style
* implement backward compatibility
* Add set market price during extended market hours
- Set market prices during extended market hours for live trading.
Adding unit test
* Add assert on internal data count
- Do not reuse parameter variables `start` and `end`. Create new variables with meaningful names and rename them to `startUtc` and `endUtc`.
- Use `EachTradeableDayInTimeZone` to calculate `tradableDates`.
- Adds another test/assertion in the regression algorithms to ensure tests in the scheduled event were performed.
The tradable days of the history request should respect the data time zone since the data source files also do.
Upgrade `BasicTemplateFuturesHistoryAlgorithm` to a regression algorithm and add a schedule event to test history requests every hour.
- Adding new ConcatEnumerator that will join enumerators together
secuentially. Adding unit tests
- SubscriptionDataReaderHistoryProvider will consume an optional
intraday enumerator that will be concatenated with the existing
SubscriptionDataReader
- Enable parallel workers for history requests, improving performance.
- Adding SubscriptionUtils to be used by backtesting through the FileSystemDataFeed and
SubscriptionHistoryProvider and live deployments which use the
SubscriptionHistoryProvider in the core
- Fix some timezone issues when using local
start/end time instead of UTC
- Adding missing `Enumerator.Dispose()` to be called by the worker
- EventProvider will receive start date during initialization, this will
be used by the `MappingEventProvider` to correctly set current mapped
symbol
- Adding unit test, updating existing regression test
The CreateSubscription method has been moved from the BrokerageHistoryProvider to the SynchronizingHistoryProvider base class, to enable reuse and avoid code duplication.
- `SubscriptionDataReader` will check map file first data and adjust
start date based on it
- Adding unit test
- Reducing code duplication
- Setting up `HistoryProvider` event handling
- Adding `SecurityCacheProvider` this class allows for two different
`Security` to share the same data type cache through different instance
of `SecurityCache`. This is used to directly access custom data types
through their underlying in a peformant maner
- Some small improvements
Adds IRegisteredSecurityDataTypesProvider to track all the data types
registered in the algorithm. Using this data, we can detect if it's
possible that we'll eventually have a property of a certain type name.
For example, consider I wish to use security.Data.TradeBar but we haven't
received any trade bars yet. Before this change a KeyNotFoundException
would be raised, but since we can determine that we expect to have trade
bars, we can detect this and return an empty list when we haven't received
any data yet. This also removes the need to constantly do a HasData<T>()
check before accessing the dynamic members.
Closes#3620
- Adding new `SubscriptionDataEnumerator` to fix broken enumerator Dipose chain
- Adding `Dispose` calls on enumerators in unit tests
- Fix unit test HandlesCoarseFundamentalData
- Custom data types will know whether or not Lean should use map files
- Updating regression test with sample custom data using map files,
which can run locally
- Adding unit tests for the `SubscriptionDataReaderHistoryProvider`,
checking it mappes equities and options correctly
- Adding new `PriceScaleFactorEnumerator` that will scale raw prices
based on a provided `SubscriptionDataConfig` and update the
`SubscriptionDataConfig.PriceScaleFactor`. Adding unit tests.
- `BaseData` factories (`TradeBar.cs`, `QuoteBar.cs`, `Tick.cs`) will
no longer scale factor prices, they will generate data points in raw
mode.
- `SubscriptionDataReader` will no longer update the
`SuscriptionDataConfig.PriceScaleFactor`
- Fix `BrokerageTransactionHandlerTests` unit test that was having a
race condition.
- Removing `AccountCurrency` from `Cash` and `Brokerage` classes.
`ICurrencyConverter` will now provide the `AccountCurrency`
- Adding new static `OrderFee.Zero` which will return a 0 order fee in
`NullCurrency`
- Adding static `Currencies.USD` value, replacing all "USD".
- Addin new static `Currencies.NullCurrency`
- Updating Bitfinex `FeeModel` so it return fees in quote currency.
Adding unit tests
- Adding new `IAlgorithm.AccountCurrency { get; }` that will point to the
`Portfolio.CashBook.AccountCurrency`. Setter will be added in a
following PR.
- Base `Brokerage` class will now have a `AccountCurrency { get }`
pointing to the `IAccountCurrencyProvider`. Will be used by the different
brokerages implementations.
- Will now use the `SynchronizingEnumerator` and avoid the duplicated
synchronization logic.
- Slightly modified the `SynchronizingEnumerator` implementation to
avoid removing enumerators with current `null` returning `true`. Adding unit tests
- Adding unit tests for `DelistingEnumerator`
- Fixing issue where price was not correctly set. Adding new check for
`HourSplitRegressionAlgorithm`
- Moving `Splits`, `Dividends`, `Mappings`, `Delistings`, from the
`SubscriptionDataReader` into there own enumerator, inheriting from
new `CorporateEventBaseEnumerator`
- Adding new helper `static class CorporateEventEnumeratorFactory` to
create the new enumerators
- Adding unit tests for new `CorporateEventBaseEnumerator`
- Adding some extra checks to `HourSplitRegressionAlgorithm` and
`DelistingEventsAlgorithm`
- Removing unneeded `CashBook` instance to create a new `TimeSlice`
- Adding new `TimeSliceFactory`, an instance base class that will
provide methods to create a new `TimeSlice`. Will own the `DateTimeZone`
property.
- Adding new `ISecurityPrice` and `IOptionPrice` that will provide a
reduced interface for accessing price properties and methods used when
creating a new `TimeSlice`
- The class `Subscription` will internally track each `Universe`
`SubscriptionRequest` added or removed
- Adding regression test in which two different `Universe` request the
same `SubscriptionDataConfig` and one of them removes/adds it in a
toggle fashion (fails on current master)
- `UniverseSelection` pending removals will also be tracked by
`Universe`
- `UniverseDecorator` will overwrite the `Universe` member of
`SubscriptionsRequests` at `GetSubscriptionRequests()`. This is due to
`this != this,Universe`
- Adding `Subscription` unit tests covering expected behavior
- Extracting pending removals logic from `UniverseSelection` class into
a new helper class `PendingRemovalsManager`. This new class will keep
track of the `universes` requesting to remove a security. Adding unit tests
The SubscriptionDataReaderHistoryProvider was using StubResultHandler, so no error messages were being shown or logged. By adding events to IHistoryProvider and SubscriptionDataReader, the dependency on IResultHandler could be removed completely and error messages are now pushed up the stack.
This is only a mechanical refactor for the updated IHistoryProvider.Initialize method in all IHistoryProvider implementations, call sites and unit tests.
- Modifying `SineHistoryProvider` so it does not use `Security`
configurations properties, including `Security.Subscriptions`.
- `SineHistoryProvider` will create SDC based on the `HistoryRequests`,
as done in other HistoryProviders
Security instances will require private access to this value in order to
compute close profit.
NOTE: The extent of these changes for simply adding a constructor argument
insinuates that we're missing an abstraction to manage the construction of
these objects, such as a factor object for Security. This will need some
careful TLC in the near future.
Pipes universe data from the data feed, through TimeSlice.Create and adds
TimeSlice.UniverseData dictionary property for read access from the algo
manager, where the data will be placed onto the correct security object.
A large percentage of synchronization time is spent performing time zone conversions.
Previously, it was possible that we may compute the same tz conversion multiple times
on the same piece of data until it's emitted. This change moves the time zone conversion
logic onto the data feed thread right before it puts the data into enqueueable enumerator.
In addition, we're not performing the clone operation at the same moment. This removes
unnecessary computation while moving the necessary computation to the data feed thread to
lighten the load on the synchronization/algorithm thread.
Adalyat Nazirov