- Reduce the amount of `Path.Combine()` usages -> it has a peformance
overhead
- Improving `FineFundamentalSubscriptionFactory` GetSource algorithm,
now it will not check if each file exists while finding the appropriate,
since we already iterated the directory before
- `DefaultDataProvider` will not check if file exists since `new
FileStream` performance the same operation internally
Previously the enumerator would get stuck and stop emitting data if one of the underlying enumerators returned a data point with the time greater than the current time. The existing unit test would only emit the first data point for the two underlying streams.
The enumerator has been updated to support data points in the future and the unit test has been extended to assert both data point counts and values.
- Moving `MapFile` and `FactorFile` resolution to the data feed stack so
that they do not add a performance overhead to the algorithm thread.
- Create logging string messaged only if required.
- Calculate `FactorFileRow.PriceScaleFactor` the least amount of times
- Adding new `PriceScaleFactorEnumerator` that will scale raw prices
based on a provided `SubscriptionDataConfig` and update the
`SubscriptionDataConfig.PriceScaleFactor`. Adding unit tests.
- `BaseData` factories (`TradeBar.cs`, `QuoteBar.cs`, `Tick.cs`) will
no longer scale factor prices, they will generate data points in raw
mode.
- `SubscriptionDataReader` will no longer update the
`SuscriptionDataConfig.PriceScaleFactor`
- Fix `BrokerageTransactionHandlerTests` unit test that was having a
race condition.
- Removing `AccountCurrency` from `Cash` and `Brokerage` classes.
`ICurrencyConverter` will now provide the `AccountCurrency`
- Adding new static `OrderFee.Zero` which will return a 0 order fee in
`NullCurrency`
- Adding static `Currencies.USD` value, replacing all "USD".
- Addin new static `Currencies.NullCurrency`
- Updating Bitfinex `FeeModel` so it return fees in quote currency.
Adding unit tests
- Adding new `IAlgorithm.AccountCurrency { get; }` that will point to the
`Portfolio.CashBook.AccountCurrency`. Setter will be added in a
following PR.
- Base `Brokerage` class will now have a `AccountCurrency { get }`
pointing to the `IAccountCurrencyProvider`. Will be used by the different
brokerages implementations.
- Will now use the `SynchronizingEnumerator` and avoid the duplicated
synchronization logic.
- Slightly modified the `SynchronizingEnumerator` implementation to
avoid removing enumerators with current `null` returning `true`. Adding unit tests
- Adding unit tests for `DelistingEnumerator`
- Fixing issue where price was not correctly set. Adding new check for
`HourSplitRegressionAlgorithm`
- Moving `Splits`, `Dividends`, `Mappings`, `Delistings`, from the
`SubscriptionDataReader` into there own enumerator, inheriting from
new `CorporateEventBaseEnumerator`
- Adding new helper `static class CorporateEventEnumeratorFactory` to
create the new enumerators
- Adding unit tests for new `CorporateEventBaseEnumerator`
- Adding some extra checks to `HourSplitRegressionAlgorithm` and
`DelistingEventsAlgorithm`
Security instances will require private access to this value in order to
compute close profit.
NOTE: The extent of these changes for simply adding a constructor argument
insinuates that we're missing an abstraction to manage the construction of
these objects, such as a factor object for Security. This will need some
careful TLC in the near future.
The frontier in LiveCustomDataSubscriptionEnumeratorFactory.CreateEnumerator is no longer initialized with DateTime.MinValue but is now set to request.StartTimeLocal, so these tests were updated to reflect this change.
These two classes were improperly reusing the same instance of SingleEntryDataCacheProvider in CreateEnumerator. Depending on the caller usage, they could be causing memory leaks.
Two memory usage tests were also added.
The FineFundamentalSubscriptionEnumeratorFactory used in combination with SingleEntryDataCacheProvider and DefaultDataProvider was never disposing of the underlying file stream, leaving files open.
Defines an enumerator factory for live custom data. This is a refactoring
and bugfix over the previous implementation. This implementation now checks
the transport medium of each refresh of the stack to see if it's a file or
a rest server and provides the appropriate fast forward/rate limiting behavior.
This enumerator was breaking the enumerator contract and returning
true when the underlying enumerator had finished. Some enumerators
in the live stack must always return true because they're never
really finished but the fast forward is a wrapper around another
enumerator and should respect when the underlying has completed
The refresh enumerator is intended to refresh an enumerator stack
after the stack has been exhausted. The existing implementation
refreshed the enumerato stack on EVERY MoveNext invocation leading
to data being dropped.
Refactored IDataProvider interface to return stream. The IDataProvider
Fetch method now only takes a key. The IDataProvider
has been reshuffled to be at the bottom of the LeanDataStack. It provides
data to the rest of the Lean stack. The default implementation of IDataProvider reads
data from disc.
All IDataCacheProviders now have constructors which take IDataProviders
and use them to find data on disc.
Renamed DataCacheProvider to ZipDataCacheProvider
Added comments to IDataProvider and it's implementations
Added comments to IDataCacheProvider and it's implementations
The DefaultFileProvider is now configured as by default as the file-provider in config.json and passed in the the engine through the leanAlgorithmHandlers object.
The FileProvider is now passed into the TextSubscriptionDataSourceReader and ZipEntryNameSubscriptionDataSourceReader. Most of the changes in this commit are just getting the FileProvider to those two methods.
When running a remote file in live mode with a high resolution (~second), it is possible
for us to download the file, fast forward to the end, and then emit duplicate data.
This change adds an extra filter in the FastForwardEnumerator to prevent it from emitting
data in the past, that is, data the is earlier than the previous piece of data emitted
-- forward only behavior --
Martin Molinero