- Reduce the amount of `Path.Combine()` usages -> it has a peformance
overhead
- Improving `FineFundamentalSubscriptionFactory` GetSource algorithm,
now it will not check if each file exists while finding the appropriate,
since we already iterated the directory before
- `DefaultDataProvider` will not check if file exists since `new
FileStream` performance the same operation internally
Previously the enumerator would get stuck and stop emitting data if one of the underlying enumerators returned a data point with the time greater than the current time. The existing unit test would only emit the first data point for the two underlying streams.
The enumerator has been updated to support data points in the future and the unit test has been extended to assert both data point counts and values.
- C# `Timer` can sometimes be triggered before the expected due time.
Implementing `RealTimeScheduleEventService` that will guarantee a
scheduled event due time is respected and the event never happens before
time.
- Moving `MapFile` and `FactorFile` resolution to the data feed stack so
that they do not add a performance overhead to the algorithm thread.
- Create logging string messaged only if required.
- Calculate `FactorFileRow.PriceScaleFactor` the least amount of times
- Adding `IStreamReader.RateLimit { get; }` specifying if a stream
reader `ReadLine()` calls should be rate limited by the source readers.
- Adding two failling unit tests in `master`
- Deleting `ParallelRunnerController` and `ParallelRunnerWorker`.
Replacing them for `Tasks`. The consumer, `EnqueueableEnumerator` will
directly spin up a new producer Task when he is running low on items.
- Adding new `PriceScaleFactorEnumerator` that will scale raw prices
based on a provided `SubscriptionDataConfig` and update the
`SubscriptionDataConfig.PriceScaleFactor`. Adding unit tests.
- `BaseData` factories (`TradeBar.cs`, `QuoteBar.cs`, `Tick.cs`) will
no longer scale factor prices, they will generate data points in raw
mode.
- `SubscriptionDataReader` will no longer update the
`SuscriptionDataConfig.PriceScaleFactor`
- Fix `BrokerageTransactionHandlerTests` unit test that was having a
race condition.
- Will now use the `SynchronizingEnumerator` and avoid the duplicated
synchronization logic.
- Slightly modified the `SynchronizingEnumerator` implementation to
avoid removing enumerators with current `null` returning `true`. Adding unit tests
- Adding unit tests for `DelistingEnumerator`
- Fixing issue where price was not correctly set. Adding new check for
`HourSplitRegressionAlgorithm`
- Moving `Splits`, `Dividends`, `Mappings`, `Delistings`, from the
`SubscriptionDataReader` into there own enumerator, inheriting from
new `CorporateEventBaseEnumerator`
- Adding new helper `static class CorporateEventEnumeratorFactory` to
create the new enumerators
- Adding unit tests for new `CorporateEventBaseEnumerator`
- Adding some extra checks to `HourSplitRegressionAlgorithm` and
`DelistingEventsAlgorithm`
The SubscriptionDataReaderHistoryProvider was using StubResultHandler, so no error messages were being shown or logged. By adding events to IHistoryProvider and SubscriptionDataReader, the dependency on IResultHandler could be removed completely and error messages are now pushed up the stack.
With the enumerator in blocking mode (backtesting), MoveNext was returning true while Current was being set to null.
This was causing a NullReferenceException in SubscriptionSynchronizer.Sync.
Quandl does not refresh data during weekends, so Friday data points will be included in the Sunday night update.
The previous data age check was preventing these data points from being emitted after a weekend.
Provides a common abstraction for UserDefinedUniverse and the
yet-to-be-impemented ScheduledUniverse. Categorized by the fact
that they don't operate on data, but instead a fixed schedule and
any data required is fetched through external means, such as a
web request.
These two classes were improperly reusing the same instance of SingleEntryDataCacheProvider in CreateEnumerator. Depending on the caller usage, they could be causing memory leaks.
Two memory usage tests were also added.
The FineFundamentalSubscriptionEnumeratorFactory used in combination with SingleEntryDataCacheProvider and DefaultDataProvider was never disposing of the underlying file stream, leaving files open.
Defines an enumerator factory for live custom data. This is a refactoring
and bugfix over the previous implementation. This implementation now checks
the transport medium of each refresh of the stack to see if it's a file or
a rest server and provides the appropriate fast forward/rate limiting behavior.
This enumerator was breaking the enumerator contract and returning
true when the underlying enumerator had finished. Some enumerators
in the live stack must always return true because they're never
really finished but the fast forward is a wrapper around another
enumerator and should respect when the underlying has completed
This duplicate data check does NOT belong here. If we need to implement
a filter to screen out duplicate data then it should be done somewhere
more specific to live custom data instead of the generic frontier aware
enumerator.
The concern that this was attempting to address was repeating data for
remote files. This is handled by preventing duplicate time stamps from
being emitted following a refresh of the remote file
The refresh enumerator is intended to refresh an enumerator stack
after the stack has been exhausted. The existing implementation
refreshed the enumerato stack on EVERY MoveNext invocation leading
to data being dropped.
This new logic harmonizes the implementation with the intent.
In short, there are 4 distinct cases that must be evaluated,
two for each interval, fill forward and data resolution.
I suspect there are some performance losses here but will save
optimization of this new algorithm as another exercise.
Fixes#1073
Martin Molinero