- Only add OnEndOfDay ScheduledEvent if the algorithm implements the
method. Adding unit tests
- Avoid creating a new baseData instance at
`SubscriptionDataSourceReader`
- Adding static `FineFundamental` instance since creating new ones is
expensive
We restrict each algorithm time loop to a pre-determined amount of time.
Exceeding this limit will cause the algorithm to immediately terminate.
This quickly becomes an issue when considering users running trainable
models that have a long initialization period that exceeds the time loop
maximum.
This change provides a mechanism through which a long-running scheduled
event is permitted to keep running and is permitted to avoid the time loop
permitted by requesting additional time. Requests for additional time are
limited according to a leaky bucket implementation whose parameters are
set via the job's controls structure. The fundamental time unit for the
algorithm is a single minute.
Here's how it works. If a scheduled event takes longer than one full wall
clock second then a request is made to the leaky bucket for one more minute.
If the scheduled event continues to take more time, it will continue to
request additional minutes. Each requested minute will prevent the algorithm's
time loop check from terminating the algorithm. When the bucket is empty and
no more minutes are available to be requested, a TimeoutException is thrown
causing a cascade that ends in the algorithm's termination and status being
flipped to RuntimeError.
Additionally, this applies equally to ALL scheduled events. While some helpers
were added with the naming of Train and TrainNow to the ScheduleManager, these
methods don't do anything special and the infrastructure doesn't otherwise
flag them as different, so this feature becomes part of the core Scheduled
Event feature set.
Further, the live scheduled events were not touched and are still pending
further discussion regarding the value added by enforcing a time restriction
when simulation time and wall clock time are equivalent.
Fixes#3319
- Adding `IRealTimeHandler.OnSecurityChanged()` will be used to update
the `OnEndOfDay` security related scheduled events
- Adding `BaseRealTimeHandler.cs` to reduce code duplication in the
`Backtesting` and `LiveTrading` `RealTimeHandlers`
- Adding CSharp and Python regression tests
- Deprecating `OnEndOfDay()` callback because of two reasons, mainly
because Python does not support two methods with the same name, but also
because different assets have different market close times.
- `ScheduledEvents` set at the same time will now be deterministic
Without the inner exception, information on the user exception is lost. In python algorithm, the stack trace is completely lost.
Removes ScheduledEventExceptionMessage property as it is redundant to Message property inherited from parent class.
This PR is an attempt to reduce contention in concurrent dictionaries, replacing method calls using full locks with lock-free equivalents:
- dictionary.Count -> dictionary.Skip(0).Count()
- dictionary.Keys -> dictionary.Select(x => x.Key)
- dictionary.Values -> dictionary.Select(x => x.Value)
The most frequent usages of these methods are: CashBook, SecurityManager, UniverseManager and indirectly, SecurityPortfolioManager.
The reasons for this update are explained very clearly in this article:
https://arbel.net/2013/02/03/best-practices-for-using-concurrentdictionary/
Catching the exception in BacktestingRealTimeHandler prevents unhandled errors in backtesting and provides more specific logging to make debugging easier for users
These changes fix a bug so that when an exception is thrown with the LiveTradingResultHandler, the error is handled and the next scheduled event can happen as scheduled.
If backtesting, we need to check if there are realtime events in the past
which didn't fire because at the scheduled times there was no data (i.e. markets closed) and fire them with the correct date/time.
In live mode, no changes are needed.
Adds the ScheduleManager which allows an algorithm to add/remove scheduled events
Check out the ScheduledEventsAlgorithm for syntax
ScheduledEvents are at their core an IEnumerator<DateTime> that defines the event times coupled with a callback
IDateRule defines dates for events
ITimeRule defines time(s) on a given date for events
In some cases we need to exit the applications, in others we just want to log.
For instance, if the transaction handler throws an unexpected exception then we
can't be confident in the state of our orders but if the live result handler fails,
the algo is still OK and consistent state, we just may not receive a packet for that time step
Time sync:
* Data feeds are required to time sync in UTC time
* TimeSlice.Time is now in UTC
IAlgorithm
* Time is now exclusively the algorithm's local time zone
* Added UtcTime
* SetDateTime( DateTime ) accepts a UTC time and is internally converted
SubscriptionDataConfig
* Adds market and time zone as required ctor parameters
SecurityExchange
* Now passes most calls directly through to SecurityExchangeHours class
SecurityExchangeHours
* Holds market hours for each day of week (LocalMarketHours)
* Talks in terms of local times in the SecurityExchangeHours.TimeZone time zone
Data/market-hours/
* New data folder to hold market hour information
* Includes market-hours-database.csv to hold market hours per market/symbol/security (see doc in file)
* Includes holidays-usa.csv to hold holidays for 'usa' market
+ The holiday files follow the pattern 'holidays-*.csv' where * is the market
TimeKeeper
* Receives updates in UTC time
* Passes that to LocalTimeKeeper's who lazily evaluate the time in their respective time zones
* Eventually this can grow to be the sole source of time in the algorithm's scope
MISC:
* Fixes exception thrown when exiting LiveTradingDataFeed
* Fixes exception thrown when exiting FileSystemDataFeed
* Fixes exception thrown when exiting StatusPing
* Simplify FillForwardEnumerator logic with GetNextMarketOpen
* Adds many time zones, see TimeZones.cs
Michael Sena