* Add DataProviderEventArgs base class for IDataProviderEvents event args
This base class includes a Symbol property. This will empower event listeners
to make decisions based on which security (symbol) raised the event. The immediate
use case is preventing multiple numerical precision messages for the same security.
This pattern can equally be applied to other error messages that are raised each
time a security is added to a universe.
See: #BUG-4722
* Update ConcurrentSet.Add to use ISet<T>.Add returning bool
It's a very common pattern to use if (set.Add(item)) which is enabled
via bool ISet<T>.Add(item) but not enabled via void ICollectiont<T>.Add(item).
This change simples changes the default Add implementation to use the ISet<T>
overload and relegates the ICollection<T>.Add implementation to be explicit.
See: #BUG-4722
* Prevent multiple numerical precision messages for same symbol
If a security is continually added/removed from a universe, then the user will
see this message each time the security is added. This results in some spam.
This change simply remembers for which symbols we've notified the user about the
numerical precision issue.
Fixes: #BUG-4722
* Add unit tests
* Refactor Py and create C# function
* Update readme to include local
* Refactor solution; fix python cases
* Update tests
* Don't accept null selector for python; Create SelectedData class
* Fix Testing
* Pre review
* Fix tests for Travis
* Test fix V2
* Test fix V3
* Refactor quantbook and fix tests
* Sort list by date
* Move ConvertToSymbols to Python Util
* Address review
* Order dataframe columns by Security ID
* Address review V2
* header for PythonUtilTests
* Add internal subscription manager
- Add InternalSubscriptionManager that will handle internal
Subscription. Replaces the realtime updates
- Fix thread race condition in the TimeTriggeredUniverseSubscription, we
have one thread injecting data points, the main algorithm thread, and
the base exchange is pulling from it
- Fixes for FakeDataQueue
- Adding unit tests
* Address reviews and fixes
- Internal subscription will use extended market hours
- Only sample charts accordingly
- Get api-url once
* Live Coarse universe refactor
- Live trading will source Coarse and Fine fundamental data directly
from disk. Updating unit tests.
* Adds ILiveDataProvider interface
* Adds wrapper for IDataQueueHandler implementations
* Replaces IDataQueueHandler with ILiveDataProvider in
LiveTradingDataFeed
* Edits IDataQueueHandler documentation
* Maintains aggregation for current IDQH impls and skips for ILDF impls
* Note: No unit test was created for this method, go back and TODO
* Protobuf Market data
- Adding protobuf support for Ticks, TradeBars and QuoteBars. Adding
unit tests.
* Adds unit tests for LiveDataAggregator changes
* Fixes bug where custom data was not handled as it was before
* Fixes race condition bug because of variable reuse in class
* Add protobuf extension serialization
* Fixes for protobuf serialization
* Refactor
* Fix OptionChainUniverse
* replace BaseDataExchange pumping ticks with consolidators
* AlpacaBrokerage
* BitfinexBrokerage
* GDAXBrokerage
* OandaBrokerage
* InteractiveBrokers
* TradierBrokerage
* FxcmBrokerage
* PaperBrokerage
* etc
* WIP fixes for existing LTDF unit tests
* Fixes more LTDF unit tests
* make IDataAggregator.Update recieving Generic BaseData rather than Tick
* Change IDataQueueHandler.Subscribe method
* Some fixes after adding new commits
* Adds protobuf (de)serialization support for Dividend and Split
* Serialize protobuf with length prefix
* Fix missing LTDF unit tests
* Adds TiingoNews protobuf definitions
* fix comments
* more fixes on IQFeedDataQueueHandler
* disallow putting ticks into enumerator directly
* ScannableEnumerator tests
* fix OandaBrokerage
* AggregationManager unit tests
* fix AlpacaBrokerage tests
* fix InteractiveBrokers
* fix FxcmBrokerage tests
* call AggregationManager.Remove method on unsubscribe
* fix GDAX existing tests
* Fixes, refactor adding more tests for AggregatorManager
* Adds BenzingaNews protobuf definitions and round trip unit test
* Adds missing TiingoNews unit test to Protobuf round trip tests
* Improve sleep sequence of LiveSynchronizer
* need start aggregating first, and then can subscribe
* More test fixes and refactor
- Refactoring AggregationManager and ScannableEnumerator so the last is
the one that owns the consolidator
- Adding pulse on the main LiveSynchronizer
* Improve performance of LEquityDataSynchronizingEnu
* Add missing Set job packet method
* Minor performance improvements
* Improvements add test timeout
- Improvements adding test timeout to find blocking test in travis
* Improve aggregationManager performance
* Testing improvements for travis
* Remove test timeouts
* More test fixes
- Adding more missing dispose calls and improving determinism
* fix IEXDataQueueHandler and tests
* Final tweaks to LTDF tests
* more AggregationManager tests
* consume and log ticks
* fix test: couldn't subscribe to Forex tickers
* change Resolution for all bar configs
* Improve RealTimeScheduleEventServiceAccuracy
* refactoring: move common code to base class
* fixed bug; unsubscribe SubscriptionDataConfig
* Small performance improvement
* Minor fixes
* Avoid Symbol serialization
* Fixes coarse selection in live mode
* Fix for live coarse
* Adds protobuf (de)serialization support for Robintrack
* Adds round-trip unit test
* Minor performance improvements
* More minor performance improvements
* pass LiveNodePacket through to OandaBrokerage
* Fixes empty list becoming null value when deserializing with protobuf
* Reverts BZ live trading exception removal and fixes tests
* Refactor WorkQueue making it abstract
* Add try catch for composer
* Adds optional data batching period to LiveFillForwardEnumerator
* Override data-queue-handler with config
* Improve PeriodCountConsolidator.Scan performance
* Move batching delay to main Synchornizer thread
* Reverts addition of Robintrack protobuf definitions
* Give priority to config history provider if set
* Add Estimize protobuffing
- Add Estimize protobuffing support. Adding unit tests
* Always dispose of data queue handler
Co-authored-by: Gerardo Salazar <gsalaz9800@gmail.com>
Co-authored-by: Adalyat Nazirov <aenazirov@gmail.com>
- Adding new ReferenceWrapper for structs, value types, to avoid thread
race conditions while reading and writting. In C# reference type
assignments are atomic, so it allows us avoid using locks.
- Add StreamReader Reader for Tick data type.
- Adding stream reader GetString extension, adding tests.
- BacktestingBrokerage will not create unnecessary order events
- Adding new SerializedOrderEvent and SerializedOrder with new
JsonConverters
- Specifying OrderEvent json converter when storing, streaming data
- Adding unit tests
Crypto and Equities will consolidate trades by default
Daily and hourly resolution will return only trades for crypto and equities.
Update equity TAQ regression test
Add minute sample files
Updating regression algorithms
Update SpotMarket test cases
Add regression test for equity trades and quotes
- History request.
- Trades and quotes pumped into OnData.
- Subscriptions are added correctly.
Add sample data
Checks low resolution only subscribes to trade bars
- The new scheduler will create a dedicated thread pool
- Work will be prioritized based on their weight and new items will have
highest priority. The weight will be determined by the size of the
subscription enqueueable.
- The work queue will be sorted and the weights updated by a dedicated
thread
- There will be a maximum weight value that will 'disable' a work item
until their weight comes down
- Consumer will not know about workers or anything alike anymore.
- Applying a general max work queue size of 400 items, this will reduce
CPU and RAM usage. Can be set by config.
- Add `BaseData.Reader` implementation which consumes the `StreamReader`
directly, avoiding in between substrings and parsing improving
performance and reducing resource consumption
- Adding unit tests, including performance unit test showing a >50%
improvement
- `ConcurrentSet` will use a `OrderedDictionary` internally so that items
are ordered deterministically, respecting insertion order.
- Adding unit tests
We restrict each algorithm time loop to a pre-determined amount of time.
Exceeding this limit will cause the algorithm to immediately terminate.
This quickly becomes an issue when considering users running trainable
models that have a long initialization period that exceeds the time loop
maximum.
This change provides a mechanism through which a long-running scheduled
event is permitted to keep running and is permitted to avoid the time loop
permitted by requesting additional time. Requests for additional time are
limited according to a leaky bucket implementation whose parameters are
set via the job's controls structure. The fundamental time unit for the
algorithm is a single minute.
Here's how it works. If a scheduled event takes longer than one full wall
clock second then a request is made to the leaky bucket for one more minute.
If the scheduled event continues to take more time, it will continue to
request additional minutes. Each requested minute will prevent the algorithm's
time loop check from terminating the algorithm. When the bucket is empty and
no more minutes are available to be requested, a TimeoutException is thrown
causing a cascade that ends in the algorithm's termination and status being
flipped to RuntimeError.
Additionally, this applies equally to ALL scheduled events. While some helpers
were added with the naming of Train and TrainNow to the ScheduleManager, these
methods don't do anything special and the infrastructure doesn't otherwise
flag them as different, so this feature becomes part of the core Scheduled
Event feature set.
Further, the live scheduled events were not touched and are still pending
further discussion regarding the value added by enforcing a time restriction
when simulation time and wall clock time are equivalent.
Fixes#3319
The inspiration for this implementation design comes directly from:
https://github.com/mxplusb/TokenBucket
Minor modifications were made, including a prominent behavior change
that prevents the Refill method from executing the initial refill
before any time has passed.
We'll use the leaky bucket algorithm to track consumption of CPU
resources by the soon-to-be-implemented training feature.
The `NullReferenceException` type is intended to only be thrown by the CLR.
In most cases, it should be converted to an `ArgumentException` or an
`InvalidOperationException`, depending on if the null value is a parameter
to the current method or not.
The `Exception` type should never really be thrown as it doesn't provide any
additional information or hints as to the issue. It also forces users that
would like to handle expected exceptions to catch all exceptions. These are
converted to an exception type that more accurately describes the reason for
raising the exception: `KeyNotFoundException`, `InvalidOperationException`
* Adds Validate class in util with accompanying regular expressions
* Provides tests for NotificationEmail and Validate.EmailAddress
* Provides default values for NotificationEmail Subject and Message
These default values resolve the underlying issue of #3021 w/out
needing to modify external sources (such as impl of IMessagingHandler)
- For backtesting the algorithm thread will sequentially process pending
order requests
- Adding new non blocking `BusyCollection`
- Updating `UpdateOrderRegressionAlgorithm` that suffered of a race
condition: it supposed `OnOrderEvent` would be called after the call to
`Transactions.AddOrder()` was finished
- Adding `WorkerThread` class, wrapper for a worker thread that will
execute given `Actions`.
- Algorithm related code (`Construction`, `Initialization`,
`Execution` will be executed by the same `WorkerThread` instance,
this is required for `Python` debugging.
- `Alpha Assets` chart will only store last data point
- Adding new `JsonRoundingConverter` that will round to 4 (number of
digits currently used for comparing alpha statistics) fractional
digits.
- Will be used for `Insights` and `ChartPoint`
Energy futures (CL, HO, RB, NG) expire in the month before the contract month.
To handle these properly, the following changes have been implemented:
- the AlgoSeekFuturesReader has been updated to create the future symbol with the correct expiry date
- the zip entry names in futures data files now contain the full expiry date (in addition to the contract month)
- the sample data files have been updated to use the new zip entry names
- new unit tests have been added and existing ones updated
- `FactorFile` will keep an ordered reversed list with the dates.
Calling `Reverse()` on the `SortedList` is expensive.
- `MapFiles` will keep first and last date, so we don't need to call
`First()` and `Last()` multiple times.
- `Liquidate` will go through all the algorithms securities only if
necessary
- `TradeBar` parsing will not call `new T` for pure `TradeBar` which is
expensive
- Removing `Lazy` hash code and security type for the
`SecurityIdentifier`, replacing for direct initialization. Accessing the
`Lazy` value adds an overhead.
- Replacing `Enum` to string for hardcoded switch statement. `Enum.ToString` is expensive.
- `DataManager` will be lazy for counting the subscriptions for
determining if its above the limit
- Adding `AlgorithmSecurityValuesProvider.GetAllValues()`, removes the
need to fetch all the security keys twice.
- During universe selection, will not try to re add already added symbol
- `Composer` will load `QuantConnect` assemblies and keep the `Types`
which is much faster than using previous wway of getting types that uses
reflexion.
- Adding static `MapFileProvider` improving calls to
`QuantConnect.commo.Symbol.Create` or `Addxxx` which created a new
instance every time
- `Engine.StackExceptionInterpreter` will be lazy initialized, since it
takes some time because it goes through all assemblies
- Moving `MarketHoursDataBase` initialization earlier and in a `Task`
since it takes some time.
Michael Handschuh