* Add DataProviderEventArgs base class for IDataProviderEvents event args
This base class includes a Symbol property. This will empower event listeners
to make decisions based on which security (symbol) raised the event. The immediate
use case is preventing multiple numerical precision messages for the same security.
This pattern can equally be applied to other error messages that are raised each
time a security is added to a universe.
See: #BUG-4722
* Update ConcurrentSet.Add to use ISet<T>.Add returning bool
It's a very common pattern to use if (set.Add(item)) which is enabled
via bool ISet<T>.Add(item) but not enabled via void ICollectiont<T>.Add(item).
This change simples changes the default Add implementation to use the ISet<T>
overload and relegates the ICollection<T>.Add implementation to be explicit.
See: #BUG-4722
* Prevent multiple numerical precision messages for same symbol
If a security is continually added/removed from a universe, then the user will
see this message each time the security is added. This results in some spam.
This change simply remembers for which symbols we've notified the user about the
numerical precision issue.
Fixes: #BUG-4722
- Only add OnEndOfDay ScheduledEvent if the algorithm implements the
method. Adding unit tests
- Avoid creating a new baseData instance at
`SubscriptionDataSourceReader`
- Adding static `FineFundamental` instance since creating new ones is
expensive
- `SubscriptionDataReader` will check map file first data and adjust
start date based on it
- Adding unit test
- Reducing code duplication
- Setting up `HistoryProvider` event handling
- Add `TiingoNews.HistoricalCrawlOffset`, timespan to add for
backtesting
- Rename: remove `Data` from `TiingoNewsData` and rename `TiingoDailyData` to `TiingoPrice`
Moves the FNF logging out of the DefaultDataProvider and into the
CreateStreamReaderError event handler. This move was required since
we don't have the required data in scope to perform this conditional
logging.
See #3618
Updates usages of GetBaseDataInstance that have a config in scope to use this
new method. This method properly assigns the symbol to the newly created base
data instance. This enables some of the new flag methods to work as expected.
Updated other poor usages where we weren't setting the symbol property.
See #3618
- Add Pandas backwards compatibility shim
- Adding `MappingExtensions` which will remove data type from the
`Symbol.ID.Symbol` value to resolve the `MapFile`
- `SecurityIdentifier.TryParse()` will throw when given an invalid
`SecurityType`
- Custom data types will know whether or not Lean should use map files
- Updating regression test with sample custom data using map files,
which can run locally
- Adding unit tests for the `SubscriptionDataReaderHistoryProvider`,
checking it mappes equities and options correctly
- Moving `MapFile` and `FactorFile` resolution to the data feed stack so
that they do not add a performance overhead to the algorithm thread.
- Create logging string messaged only if required.
- Calculate `FactorFileRow.PriceScaleFactor` the least amount of times
New custom data class USEnergyInformation with new demonstration algorithms, the updated config file for users to set their EIA token.
Adds `CloseTime` to represent the time that the data period end. `EndTime` represents, in turn, the time the data is emitted. There is an offset between `CloseTime` and `EndTime` that is defined by the difference between the last bar as emitted and its time.
In live mode, if the `USEnergyInformation.Reader` returns null, the `CollectionSubscriptionDataSourceReader.Read` method will pull for new data constantly. Therefore it should return an empty `BaseDataCollection` object.
In a few places SubscriptionDataReader was incorrectly assuming the TradeableDates were in ExchangeTimeZone instead of DataTimeZone.
This issue was found specifically with CFD assets which have the exchange time zone ahead of the data time zone.
- `TextSubscriptionDataSourceReader` will now cache raw price mode data
points for each requested local source file, per data type.
Caching will be handled by the `MemoryCache` class.
- `GetLastKnownPrice()` will not `RoundDown` end time. This was causing
it to fail to fetch a data point in some cases.
- Adding new `PriceScaleFactorEnumerator` that will scale raw prices
based on a provided `SubscriptionDataConfig` and update the
`SubscriptionDataConfig.PriceScaleFactor`. Adding unit tests.
- `BaseData` factories (`TradeBar.cs`, `QuoteBar.cs`, `Tick.cs`) will
no longer scale factor prices, they will generate data points in raw
mode.
- `SubscriptionDataReader` will no longer update the
`SuscriptionDataConfig.PriceScaleFactor`
- Fix `BrokerageTransactionHandlerTests` unit test that was having a
race condition.
- Will now use the `SynchronizingEnumerator` and avoid the duplicated
synchronization logic.
- Slightly modified the `SynchronizingEnumerator` implementation to
avoid removing enumerators with current `null` returning `true`. Adding unit tests
- Adding unit tests for `DelistingEnumerator`
- Fixing issue where price was not correctly set. Adding new check for
`HourSplitRegressionAlgorithm`
- Moving `Splits`, `Dividends`, `Mappings`, `Delistings`, from the
`SubscriptionDataReader` into there own enumerator, inheriting from
new `CorporateEventBaseEnumerator`
- Adding new helper `static class CorporateEventEnumeratorFactory` to
create the new enumerators
- Adding unit tests for new `CorporateEventBaseEnumerator`
- Adding some extra checks to `HourSplitRegressionAlgorithm` and
`DelistingEventsAlgorithm`
- In some cases, for Resolution.Daily, application of splits were
up to one day behind the data. Causing the 'early' data to use a
wrong factor value
- Updating regression test affected by issue
- Adding new regression test
The SubscriptionDataReaderHistoryProvider was using StubResultHandler, so no error messages were being shown or logged. By adding events to IHistoryProvider and SubscriptionDataReader, the dependency on IResultHandler could be removed completely and error messages are now pushed up the stack.
Adds reference price to dividend/split
Adds GetSplitsAndDividends to FactorFile
Adds Apply methods to FactorFile/FactorFileRow
Updates factor files to include reference prices
Move delisting date estimation to the constructor
Using `MarketHoursDatabase` for option delisting date estimation instead USHolidays
Adding file header.
Regression algorithm added.
Use of `ExchangeHours.IsDateOpen` instead `ExchangeHours.IsDateOpen.Holidays`.
Extract values from `symbol.ID` into a local variable.
Improved Regression test
This follows the pattern used by delisting events, where we send a warning event
before the start of trading on the trading day before the split will happen. This
change also adds a SplitType enum having a Warning and SplitOccurred values.
Before this PR, backtesting some symbols fails with an error when parsing the factor file.
Some factor files have INF split values, indicating that the stock has so many splits that prices can't be calculated with correct numerical precision.
To allow backtesting these symbols, we need to move the starting date forward when reading the data.
Known symbols: GBSN, JUNI, NEWL
Refactored IDataProvider interface to return stream. The IDataProvider
Fetch method now only takes a key. The IDataProvider
has been reshuffled to be at the bottom of the LeanDataStack. It provides
data to the rest of the Lean stack. The default implementation of IDataProvider reads
data from disc.
All IDataCacheProviders now have constructors which take IDataProviders
and use them to find data on disc.
Renamed DataCacheProvider to ZipDataCacheProvider
Added comments to IDataProvider and it's implementations
Added comments to IDataCacheProvider and it's implementations
Fixed bug where DataCacheProvider was returning a stream without closing
the stream. Also, ZipFiles returned from ZipFileCacheProvider are
effectively disposed.
Removed date field from IDataCacheProvider interface
This interface was built around the existing implementation of the
DataCacheProvider. It's purpose is to define a method for caching data
files. Currently, most data files are either read from disc or retrieved
remotely. The IDataChceProvider returns returns a Stream and can handle
many types of data. The IDataFileCacheProvider is needed because keeping data in memory will improve performance and overcome certain File read/write limitations (such as sharing violations) currently experienced by Lean.
There are two implementations of this interface. The first is the
SingleZipEntryDataCache. This is the default implementation. It does not
cache and returns the first entry found within a ZipFile. The second is
the ZipEntryCacheProvider. This is a rework of the existing
DataCachePrivider. It caches ZipFiles for up to 10 seconds. It can
return specific entries for option and future ZipFiles. Otherwise, it
returns the first entry in a ZipFile.
Added IDataFileCacheProvider as parameter to IHistoryProvider.Initialize()
The IDataFileCacheProvider is very useful for the IHistoryProvider in that it can eliminate the need for history request to touch disc. This can greatly improve peerformance and eliminate disc bugs associated with disc read/writes. To minimize changes throughtout Lean, the default value for the IDataFileCacheProvider is null in the IHistoryProvider.Initialize method.
IDataCacheProviders are passed down the Lean stack. Each instance of a Subscription Enumerator factory decides what cache it
will use. In other words, the IDataCacheProvider is not configurable from
config.json. The IDataCacheProvider is passwed down the stack form the
Enumerator factory to the IStreamReader where it is used to retrieve data.
Added useful helper method to Unizp files into a Stream.
1. Made sure we treat properly option expiration dates before Feb 2015 and after. Added tests.
2. Refactored expiration delisting, assignments/option exercise to happen in the end of the date, not MOC orders in the beginning of the day. Regression test.
3. Refactored option exercise model to generate proper fills on assignments/option. Those fills are reflected correctly in margins, in stats and correspond to IB model. Still need to run IB real-life live tests.
4. Refactored option symbol related functions into separate module (OptionSymbol.cs)
5. Made sure OnAssignmentEvent arrived to the user algo in regression test. Do we need OnExercise event? Not sure.
6. Tested end-to-end Ray's current code (covered call strat) to see short option legs expire worthless, and stats updated.
Michael Handschuh