- For backtests download RemoteFile just once
- Live subscriptions and history requests will continue to download the
`RemoteFile` on each request.
- Adding unit tests
- Adding `IStreamReader.RateLimit { get; }` specifying if a stream
reader `ReadLine()` calls should be rate limited by the source readers.
- Adding two failling unit tests in `master`
Through `SubscriptionDataSource`.
Exceptions, cocumenting, cleaning and testing adjust
make isLiveMode mandatory in the SubscriptionDataSource constructor
Calls to SubscriptionDataSource implements new bool argument
csproj file mistake fixed
Clean up nullable from prevoius implementation.
Revert "Exceptions, cocumenting, cleaning and testing adjust"
This reverts commit 7da7a9fca195b327a1322c0c7c4a7a088b2f1099.
SubscriptionDataSource back to default.
Uisng LiveMode flag from the calling contex
_delivered variabl setting.
Comments fixed
xml doc fix
It is not uncommon that a web service would require some form of
authentication headers to be provided. This change aims to enable
this by adding the header information to the source. This design
allows a user's custom data type to return the source properly
hydrated with the required header values before being sent back
through the rest of the LEAN infrastructure.
The streamReader expects a stream to be provided by the DataCacheProvider. When the file is not available, most dataCacheProviders will look for the file on disc. This works for most Lean data types, but not for remote data. After downloading a remote file, this PR sends the data to the DataCacheProvider via the Store() method. This way, when the streamReader asks for the data from the DataCacheProvider - the remote file will already have been cached.
* Fixes 5 warnings with code CD1572
* Adds the Xml Comment parameter and implements the missing selector parameter
* Fixes rest of the 13 warnings with code CS1572
Refactored IDataProvider interface to return stream. The IDataProvider
Fetch method now only takes a key. The IDataProvider
has been reshuffled to be at the bottom of the LeanDataStack. It provides
data to the rest of the Lean stack. The default implementation of IDataProvider reads
data from disc.
All IDataCacheProviders now have constructors which take IDataProviders
and use them to find data on disc.
Renamed DataCacheProvider to ZipDataCacheProvider
Added comments to IDataProvider and it's implementations
Added comments to IDataCacheProvider and it's implementations
Fixed bug where DataCacheProvider was returning a stream without closing
the stream. Also, ZipFiles returned from ZipFileCacheProvider are
effectively disposed.
Removed date field from IDataCacheProvider interface
This interface was built around the existing implementation of the
DataCacheProvider. It's purpose is to define a method for caching data
files. Currently, most data files are either read from disc or retrieved
remotely. The IDataChceProvider returns returns a Stream and can handle
many types of data. The IDataFileCacheProvider is needed because keeping data in memory will improve performance and overcome certain File read/write limitations (such as sharing violations) currently experienced by Lean.
There are two implementations of this interface. The first is the
SingleZipEntryDataCache. This is the default implementation. It does not
cache and returns the first entry found within a ZipFile. The second is
the ZipEntryCacheProvider. This is a rework of the existing
DataCachePrivider. It caches ZipFiles for up to 10 seconds. It can
return specific entries for option and future ZipFiles. Otherwise, it
returns the first entry in a ZipFile.
Added IDataFileCacheProvider as parameter to IHistoryProvider.Initialize()
The IDataFileCacheProvider is very useful for the IHistoryProvider in that it can eliminate the need for history request to touch disc. This can greatly improve peerformance and eliminate disc bugs associated with disc read/writes. To minimize changes throughtout Lean, the default value for the IDataFileCacheProvider is null in the IHistoryProvider.Initialize method.
IDataCacheProviders are passed down the Lean stack. Each instance of a Subscription Enumerator factory decides what cache it
will use. In other words, the IDataCacheProvider is not configurable from
config.json. The IDataCacheProvider is passwed down the stack form the
Enumerator factory to the IStreamReader where it is used to retrieve data.
Added useful helper method to Unizp files into a Stream.
- Added IQFeed options support to toolbox: option chain universe, option symbol list, subscriptions, live prices
- Added LiveTradingDataFeed to support options live screaming
- Added IB live options trading support: trading orders, holdings, live option exercising
- Modified ISymbolMapper interface to support derivatives (options, futures)
- Fixed minor bugs with (introduced ealier) symbol changes for options
Tested with IQFeed version 5.2.4.2. IB TWS offline version 957.
Many places in the code used Log.Error(err.Message) or equivalent which
strips out all the really useful information, such as the stack trace
and inner exceptions. Using Log.Error(exception) is the correct way to
log an error as it will correctly write all the message details, also,
by passing the full Exception object we can improve the logging in this
one place and all call sites will automatically benefit from the improvements
Includes default impl of BaseDataSubscriptionFactory
This change is to prepare for binary data sources and defines a factory to
handle the reading of a SubscriptionDataSource and the conversion of that data
into BaseData instances
DataFeeds now expose a single bridge (BlockingCollection<TimeSlice>)
- IDataFeed now responsible for all time syncing, DataStream was removed
- BlockingCollection allows for easy throttling of data feed thread without sleeps (internally using semaphore slim)
- Slimmed down IDataFeed interface to not include members unused externally
DataFeeds now use raw IEnumerator<BaseData> instead of subscription data readers directly
- Simplifies fill forward logic (see FillForwardEnumerator.cs and FillForwardEnumeratorTest.cs)
- Allows for composition (raw reader -> fill forward, user/market filters)
- SubscriptionDataReader now handles refreshing source (continuous enumeration)
LiveTradingDataFeed cleaned up to work with time slice bridge
- RealTimeSynchronizedTimer handles pushing time slices into bridge every second, with or without data
- Subscriptions with tick resolution get their data pushed directly into the bridge immediately
Fill forward changes
- Implemented as an IEnumerator<BaseData>
- Fill forward lower res on higher res (daily gets filled forward on second/minute/hour bars)
- Second/minute/hour will not be filled forward on daily bars (at midnight) in line with current behavior
MISC:
- Performance boost in backtesting brokerage via not checking _pending.Count, instead use local bool to determine if there's pending orders
- Added a FakeDataQueue to aid in testing LiveTradingDataFeed
- FileSystemDataFeed now uses CancellationTokenSource to handle thread exits
- Added logging
Updates SubscriptionDataReader to use IStreamReader
Adds FactorFile and MapFile helpers
Adds support for Raw, Adjusted, and TotalReturn data normalization and portfolio application
Martin Molinero