* Store temp files in subdirectory
* Fix Dispose case for new temp dir
* Adjust tests for new temp dir
* Dispose unit tests
* Unit test for issue 4811
* Refactor for not using temp files
* Fix storage checks for saving data, plus tests
* Use Base64 for storing keys and decoding them; handles odd key strings
* Don't allow "?" in a key
* Address review
* Deleted test cases
* PersistData handle deletion of files
* Refactor GetFilePath to use Persist()
* Make PathForKey protected
- LocalObjectStore.Delete() will also delete file from the local object
store path if present, this will avoid the issue where restarting the
object store will re load the same deleted file. Adding unit test.
Issue https://github.com/QuantConnect/Lean/issues/4811
* DataConsolidator Wrapper for Python Consolidators
* Regression Unit Test
* Refactor Regression test
* Bad test fix
* pre review
* self review
* Add RegisterIndicator for Python Consolidator
* Python base class for consolidators
* Modify regression algo to register indicator
* unit test - attach event
* Test fix
* Fix test python imports
* Add license header file and null check
Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
Since extension methods don't play well with pythonnet, this change converts
the extensions class into a decorator class. Additionally, this ObjectStore
type is the type that gets exposed via QCAlgorithm so users can access these
methods directly without requiring the use of extension methods.
This approach has many good properties. For one, it doesn't force implementors
of IObjectStore to use a base class. Second, it maintains healthy separation of
API level concerns (such as convenient methods) from the abstraction level conerns
of IObjectStore. Setting it up in this way ensures ANY implementation of IObjectStore
will still get access to these additional methods. Another thing to note is this
prevents using a base class on QCAlgorithm's public interface. Instead, we have a
specific type that is dedicated to fulfilling API level requirements, which also
provides us flexibility in the event the API needs to be updated. If it were a subclass,
you run the risk of breaking the implementors of the subclass.
There's no reason for algorithms to have direct access to this implementation.
Moving this into the engine prevents algoriths from directly accessing LocalObjectStore
and instead can only reference it through the IObjectStore abstraction
persistenceIntervalSeconds defines the number of seconds between
each save operation. For the local object store, this dictates
how often the contents of the object store is packaged and written
to disk. The PersistData virtual method is provided for subclasses
to provide a different implementation of how/where to persist the
data. The change to be in-memory aims at keeping the object store
performant with reasonable persistence guarantees.
Invoking FileInfo.Length when the file does not exist on disk throws an
error. Additionally, it was clunky to use the object store when always
required to perform ContainsKey first.
This commit is squashed from iterative development:
- More consistent method naming
- Storage root path updated to be absolute and include algorithm name
- Storage root path created only if object store is actually used
- Implemented XML save/load
- Added missing unit tests
- Replaced Log.Trace with Log.Error calls
- Added the object store name logging in Engine.Main
- Read storage root from config
- Create algorithm storage root folder in Initialize
- Remove empty folder in Dispose
- Added null checks in all methods
- Added missing XML parameter docs
- make Initialize and Dispose virtual
- make AlgorithmStorageRoot protected
The IObjectStore abstraction provides algorithms with a persistent
storage mechanism. While the algorithm is running, data is maintained
in memory as a dictionary of raw bytes (string -> byte[]). This ensures
we avoid any reference type shenanigans. Periodically, the data in the
object store is persisted and additionally, when the algorithm shuts
down, the object store's data will again be persisted. This ensures that
when the algorithm starts up again, it will have access to any state
that has been saved into the object store.
A great use case for IObjectStore is saving a compute heavy model.
For example, computing the weights of a deep neural network is very
CPU intensive, but after the weights are computed, evaluation is fairly
quick. An initial backtest can be used to solved for the network's weights
and then subsequent backtests or even in live mode, the weights will be
available to the algorithm provided they were saved into the object store.
Also, some libraries require a file path to load model data. The object
store provides a `GetFilePath(key)` method which will copy the data for
the provided key to the disk and return that path so the library can load
the model data.
Colton Sellers