Errors raised during persistence aren't able to be handled by user code,
and in fact, are swallowed by the implementation after being logged. By
exposing these errors as events we allow the algorithm to be notified of
such an error and take any step necessary to handle the persistence error.
persistenceIntervalSeconds defines the number of seconds between
each save operation. For the local object store, this dictates
how often the contents of the object store is packaged and written
to disk. The PersistData virtual method is provided for subclasses
to provide a different implementation of how/where to persist the
data. The change to be in-memory aims at keeping the object store
performant with reasonable persistence guarantees.
This commit is squashed from iterative development:
- More consistent method naming
- Storage root path updated to be absolute and include algorithm name
- Storage root path created only if object store is actually used
- Implemented XML save/load
- Added missing unit tests
- Replaced Log.Trace with Log.Error calls
- Added the object store name logging in Engine.Main
- Read storage root from config
- Create algorithm storage root folder in Initialize
- Remove empty folder in Dispose
- Added null checks in all methods
- Added missing XML parameter docs
- make Initialize and Dispose virtual
- make AlgorithmStorageRoot protected
The IObjectStore abstraction provides algorithms with a persistent
storage mechanism. While the algorithm is running, data is maintained
in memory as a dictionary of raw bytes (string -> byte[]). This ensures
we avoid any reference type shenanigans. Periodically, the data in the
object store is persisted and additionally, when the algorithm shuts
down, the object store's data will again be persisted. This ensures that
when the algorithm starts up again, it will have access to any state
that has been saved into the object store.
A great use case for IObjectStore is saving a compute heavy model.
For example, computing the weights of a deep neural network is very
CPU intensive, but after the weights are computed, evaluation is fairly
quick. An initial backtest can be used to solved for the network's weights
and then subsequent backtests or even in live mode, the weights will be
available to the algorithm provided they were saved into the object store.
Also, some libraries require a file path to load model data. The object
store provides a `GetFilePath(key)` method which will copy the data for
the provided key to the disk and return that path so the library can load
the model data.
Michael Handschuh