27 Commits

Author SHA1 Message Date
Jhonathan Abreu 26584c2fd0 Stop and delete unused threads in tests (#9219)
API Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Report Generator Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Syntax Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
2026-01-13 13:13:37 -03:00
Martin-Molinero 8ee7a190c6 Minor fix for negative rolling window index (#9026)
* Minor fix for negative rolling window index

- Minor fix for negative rolling window index when partially full,
  adding unit test

* Reduce RefreshesSymbolProperties test runtime
2025-10-10 17:47:58 -03:00
Martin-Molinero 890fcd114e Fix Symbol EOD internal SecurityChanges handling (#8824)
- Fix Symbol EOD internal SecurityChanges handling. Adding unit test and
  regression test
2025-06-12 19:18:15 -03:00
Jhonathan Abreu 8149c291fc Dynamic market hours for time and date rules (#8541)
API Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
Report Generator Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Add unit test reproducing the bug

* Improve unit test

* Apply market hours database updates to instance

Make sure entries are updated instead of overridden so that consumers holding the mhdb can pick up the changes

* Improved unit tests

* Update data folder MHDB entries instead of overriding instance

* Optionally reset custom mhdb entries on reset

* Simplify mhdb reset in live trading

Cleanup and minor fixes

* Ensure symbol properties database updates are picked up by consumers

Update SPDB entries instead of overriding them, just like for the MHDB so that consumers pick up updates after refresh

* Fix for symbol properties update thread safety

* Minor improvements and cleanup

* Mhdb and spdb update logic simplification

* Move "force exchange always open" logic to mhdb

* Simplify symbol properties holder and updates

* Refactor security databases to use a common base class

Introduce BaseSecurityDatabase<T, TEntry> to encapsulate common functionality for MarketHoursDatabase and SymbolPropertiesDatabase.

* Cleanup
2025-01-31 15:00:14 -04:00
Ricardo Andrés Marino Rojas 56d36b97d5 Fix warnings part 5 (#8105)
* Add algorithm unit tests

* Add Brokerage unit tests

* Add Common unit tests

* Fix unit test bugs

* Add remaining unit tests

* Address requested changes

* Address requested changes

* Fix failing unit test

* Address requested changes
2024-06-26 10:11:36 -03:00
Jhonathan Abreu 5432c7c6d8 Fix dynamic SPDB unit tests taking too long (#8094)
* Fix dynamic SPDB unit tests taking too long

* Minor improvements in unit tests

* Minor improvements in unit tests

* Cleanup
2024-06-24 14:36:04 -03:00
Jhonathan Abreu a159950117 Fix: refresh symbol properties using right type for security. (#8087)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Report Generator Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
API Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Fix for live trading SPDB refresh
2024-06-11 13:50:33 -04:00
Jhonathan Abreu 6b963c735c Make SPDB dynamic with periodic updates (#8083)
* Dynamic spdb

* Minor changes and unit test

* Get MHDB and SPDB refresh period from config

* Add databases refresh period to algorithm settings

* Address peer review and add unit test
2024-06-07 15:48:27 -03:00
Jhonathan Abreu b89d7d65b2 Fix live trading MHDB update (#7848)
API Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Report Generator Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
* Add schedule manager unit tests

* Add schedule manager unit tests

* Fix overriding security exchange hours

* Minor comments changes

* Minor change

* Add SecurityExchangeHours.Update method to set without changing the reference

* Minor change
2024-03-13 12:09:15 -04:00
Ricardo Andrés Marino Rojas 518ab562ec Fix bug in LiveTradingRealTimeHandler (#7696)
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
* Initial draft of the solution

* Add improvments

* nit changes

* Improve implementation

* Enhance unit tests

* Add one more unit test

* Fix failing tests and update MHDB

* Add requested changes

* Nit change

* Add tweaks to MarketHoursDatabase class

* Solve bug and add unit tests

* Update unit tests

* Address requested changes

* Handle potential bug

* Nit change

* Minor review

---------

Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
2024-01-24 11:59:09 -03:00
Martin-Molinero e0b9a2735f Warmup fixes (#6293)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Add regression test reproducing issue

* WIP

* Update regression algorithms

* Improvements

* Cleanup and more fixes

- Clean up BaseDataExchange

* RealTimeHandler fixes

* Address reviews

* More comments, tests and minor tweaks

* Avoid false test failures

* Fixes

* Minor improvements

* Scheduled Event start time fix

* Add warmup option regression algorithm and fix

* Add WarmupFutureRegressionAlgorithm

* Normalize backtesting and live future selection
2022-05-31 10:03:56 -03:00
Martin-Molinero 68e2a9170a Do not send internal SecurityChanges to Algorithm (#6118)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Do not send internal SecurityChanges to Algorithm

- Will not send internal security changes to the Algorithm by default.
  Following custom security changes filter pattern. Updating regression
  algorithms to assert behavior.
- The universe member will know wether it was added with internal
  configurations or not

* Address reviews use a separate collection for internals

* Refactor solution. Adding security changes constructor class
2021-12-21 20:24:32 -03:00
Martin-Molinero 3b826535c7 Fix for ConcurrentDictionary Thread Safety (#6105)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
- ConcurrentDictionary with OrderBy is not thread safe, adding thread
  safe extension method. Adding unit tests
2021-12-13 10:45:58 -03:00
Colton Sellers 395c1123da Remove Obsolete QCAlgorithm.OnEndOfDay() (#5441)
* Remove and replace OnEndOfDay() ref

* Restore functionality of obsolete EOD, waiting for deprecation in August 2021

* Cleanup

* Adjustments to only post message when using obsolete EOD

* nit, extra space

* Address review

* Adjust test to reflect new behaviour

* Move GetPythonArgCount to an extension method

* Add unit test

* nit accidental import

* Refactor broken test

* Use Py.GIL() state for extension
2021-04-07 13:39:49 -03:00
Martin-Molinero 9cdb4a91c5 Refactor live data feed (#4636)
* Live Coarse universe refactor

- Live trading will source Coarse and Fine fundamental data directly
  from disk. Updating unit tests.

* Adds ILiveDataProvider interface

  * Adds wrapper for IDataQueueHandler implementations

  * Replaces IDataQueueHandler with ILiveDataProvider in
    LiveTradingDataFeed

  * Edits IDataQueueHandler documentation

* Maintains aggregation for current IDQH impls and skips for ILDF impls

  * Note: No unit test was created for this method, go back and TODO

* Protobuf Market data

- Adding protobuf support for Ticks, TradeBars and QuoteBars. Adding
  unit tests.

* Adds unit tests for LiveDataAggregator changes

  * Fixes bug where custom data was not handled as it was before
  * Fixes race condition bug because of variable reuse in class

* Add protobuf extension serialization

* Fixes for protobuf serialization

* Refactor

* Fix OptionChainUniverse

* replace BaseDataExchange pumping ticks with consolidators

* AlpacaBrokerage

* BitfinexBrokerage

* GDAXBrokerage

* OandaBrokerage

* InteractiveBrokers

* TradierBrokerage

* FxcmBrokerage

* PaperBrokerage

* etc

* WIP fixes for existing LTDF unit tests

* Fixes more LTDF unit tests

* make IDataAggregator.Update recieving Generic BaseData rather than Tick

* Change IDataQueueHandler.Subscribe method

* Some fixes after adding new commits

* Adds protobuf (de)serialization support for Dividend and Split

* Serialize protobuf with length prefix

* Fix missing LTDF unit tests

* Adds TiingoNews protobuf definitions

* fix comments

* more fixes on IQFeedDataQueueHandler

* disallow putting ticks into enumerator directly

* ScannableEnumerator tests

* fix OandaBrokerage

* AggregationManager unit tests

* fix AlpacaBrokerage tests

* fix InteractiveBrokers

* fix FxcmBrokerage tests

* call AggregationManager.Remove method on unsubscribe

* fix GDAX existing tests

* Fixes, refactor adding more tests for AggregatorManager

* Adds BenzingaNews protobuf definitions and round trip unit test

* Adds missing TiingoNews unit test to Protobuf round trip tests

* Improve sleep sequence of LiveSynchronizer

* need start aggregating first, and then can subscribe

* More test fixes and refactor

- Refactoring AggregationManager and ScannableEnumerator so the last is
  the one that owns the consolidator
- Adding pulse on the main LiveSynchronizer

* Improve performance of LEquityDataSynchronizingEnu

* Add missing Set job packet method

* Minor performance improvements

* Improvements add test timeout

- Improvements adding test timeout to find blocking test in travis

* Improve aggregationManager performance

* Testing improvements for travis

* Remove test timeouts

* More test fixes

- Adding more missing dispose calls and improving determinism

* fix IEXDataQueueHandler and tests

* Final tweaks to LTDF tests

* more AggregationManager tests

* consume and log ticks

* fix test: couldn't subscribe to Forex tickers

* change Resolution for all bar configs

* Improve RealTimeScheduleEventServiceAccuracy

* refactoring: move common code to base class

* fixed bug; unsubscribe SubscriptionDataConfig

* Small performance improvement

* Minor fixes

* Avoid Symbol serialization

* Fixes coarse selection in live mode

* Fix for live coarse

* Adds protobuf (de)serialization support for Robintrack

  * Adds round-trip unit test

* Minor performance improvements

* More minor performance improvements

* pass LiveNodePacket through to OandaBrokerage

* Fixes empty list becoming null value when deserializing with protobuf

* Reverts BZ live trading exception removal and fixes tests

* Refactor WorkQueue making it abstract

* Add try catch for composer

* Adds optional data batching period to LiveFillForwardEnumerator

* Override data-queue-handler with config

* Improve PeriodCountConsolidator.Scan performance

* Move batching delay to main Synchornizer thread

* Reverts addition of Robintrack protobuf definitions

* Give priority to config history provider if set

* Add Estimize protobuffing

- Add Estimize protobuffing support. Adding unit tests

* Always dispose of data queue handler

Co-authored-by: Gerardo Salazar <gsalaz9800@gmail.com>
Co-authored-by: Adalyat Nazirov <aenazirov@gmail.com>
2020-08-18 20:21:10 -03:00
Martin Molinero 1fcd9588d6 Dispose Timers and threads in tests 2020-04-27 17:46:16 -03:00
Martin Molinero 7ea241b735 Fix backtesting event bug
- After skipping scheduled events it will re order them
- Fix bug where when only a single scheduled event was present it
wouldn't fire
2020-03-03 12:56:50 -03:00
Martin Molinero b45b7e427f Address reviews
- Address review improve regression and unit test
- Fix same time scheduled event order
- Adding comments in unit test
- Increasing unit test sleep time
2020-03-03 10:45:37 -03:00
Martin Molinero b3feea86a1 Refactor IsolatorLimitResultProvider
- Replace `new Task` at `IsolatorLimitResultProvider.Consume()` for
single instance base timer.
- Adding unit test
2020-03-02 18:23:20 -03:00
Martin Molinero 2f4f7dc2cb ScheduledEvents fire in order
- Scheduled events will fire in order even after market hours
- Improving scheduled events performance
- Adding regression and unit test
2020-03-02 11:05:54 -03:00
Martin Molinero 6cd12abff4 Remove duplicate EndOfDayEvent for Symbols
- Remove duplicate EndOfDayEvent for Symbols added with `AddData` calls
- Adding regression test
2020-01-14 18:06:52 -03:00
Martin Molinero 3a5a2c582f Lazy OnEndOfDay ScheduledEvent
- Only add OnEndOfDay ScheduledEvent if the algorithm implements the
method. Adding unit tests
- Avoid creating a new baseData instance at
`SubscriptionDataSourceReader`
- Adding static `FineFundamental` instance since creating new ones is
expensive
2019-12-19 19:07:16 -03:00
Martin Molinero a92bee5bf1 Fix OnEndOfDay not fired
- Adding `IRealTimeHandler.OnSecurityChanged()` will be used to update
the `OnEndOfDay` security related scheduled events
- Adding `BaseRealTimeHandler.cs` to reduce code duplication in the
`Backtesting` and `LiveTrading` `RealTimeHandlers`
- Adding CSharp and Python regression tests
- Deprecating `OnEndOfDay()` callback because of two reasons, mainly
because Python does not support two methods with the same name, but also
because different assets have different market close times.
- `ScheduledEvents` set at the same time will now be deterministic
2019-06-20 17:35:28 -03:00
Stefano Raggi a8fc134c57 Support for scheduled events with same names or date/time rules
Fixes #1061
2018-02-05 11:53:09 +01:00
snugs e1938baafe Clean up using directives 2015-09-01 22:17:35 -04:00
snugs 730430b1bb Adds scheduling feature
Adds the ScheduleManager which allows an algorithm to add/remove scheduled events
Check out the ScheduledEventsAlgorithm for syntax
ScheduledEvents are at their core an IEnumerator<DateTime> that defines the event times coupled with a callback
IDateRule defines dates for events
ITimeRule defines time(s) on a given date for events
2015-08-10 10:56:14 -04:00
snugs 7fa9a4b1be Implements self re-scheduling events
A ScheduledEvent can be defined using an enumerator of event times and a callback
2015-08-05 13:13:08 -04:00