54 Commits

Author SHA1 Message Date
Jhonathan Abreu 5845f3cd81 Use StubsIgnore attribute in some methods (#8708)
IBaseData.Reader
IBaseData.GetSource
IndicatorBase.Update
IAlgorithm.OnEndOfDay
2025-04-21 16:27:37 -04:00
Martin-Molinero 8ad1016710 Remove index resolution limitations (#8519)
- Remove index resolution limitations being imposed
2025-01-10 14:59:19 -03:00
Jhonathan Abreu bc5d51806d Universe data frames normalization (#8385)
API Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Report Generator Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
* Normalize universe data frames

Universe and (generically BaseDataCollection) data frames are not normalize and unpacked into a data frame, instead of just creating data frames with the universe lists within it

* Fix unit tests and algorithms to expecte new universe dataframe format

* Fixes

* Add PandasConverter.DataFrameGenerator class

* Pandas data frame generator class fixes

* Add comments

* Housekeeping

* Add attributes to mark classes and properties for pandas processing

* Improve pandas properties expanding

Allow and handle duplicate names

* Use PandasData generalization for Lean common data types

* Add points time as column when converting base data collections to data frames

* Cleanup and minor changes

* Minor change

* Pandas data to get type members on demand

* Move Pandas helper classes to their own files

* Minor changes

* Add flatten argument to python history api

This allows users to decide whether they want fully expanded dataframes for universe and other collection data types. Else, master behavior is kept

* Adding missing changes to last commit

* Update Pythonnet version to 2.0.40

* Add flattent argument to algorithm's OptionChain api

* Minor changes

* Housekeeping

* Minor changes

* Bug fix skipping data collection data points

* Add comment

* Set correct exchange time to OptionUniverse instances

* Address peer review and cleanup

* Cleanup

* Minor changes
2024-11-26 16:16:34 -04:00
Ricardo Andrés Marino Rojas 16a737ee26 Fix most of CA1725 warnings (#8154)
* Fix CA1725 warning

* Address requested changes
2024-07-05 16:19:56 -03:00
Martin-Molinero 7705895cc6 Portfolio margin chart (#7719)
API Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
* Portfolio state chart

- Cleanup and renames
- Add series.IndexName. Uodating unit tests
- Add Chart.LegendDisabled. Adding new unit tests
- Add ChartPoint.Tooltip. Updating unit tests
- Minor compression tweak. Adding unit test
- Add ChartJsonConverter. Adding unit tests

* Minor chart serialization order tweak

* Refactor portfolio state sampling and storing

* Move PortfolioMargin into a lean side chart
2024-02-01 12:43:08 -03:00
Jhonathan Abreu a9d26084f9 Options algorithm performance enhancement (#7555)
* Ignore non-market universe data for security cache storage

This is a performance enhancement

* Minor refactor
2023-11-02 17:20:54 -04:00
Louis Szeto 413a084254 Add Price in CoarseFundamental (#6464)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Update CoarseFundamental.cs

* Fix typo

* Add virtual/override
2022-07-06 13:28:57 -03:00
Colton Sellers c11a09e08a Feature Index Option Low Resolution Support (#6075)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Activate Index and IndexOptions low res

* Add Index and IndexOption low res data

* Add IndexOption regressions

* Fix bug with IndexOption QuoteBar data not being scaled

* nit - update comment for _scaleFactor

* Update data and regressions to reflect fix in data

* Tweak regressions
2021-11-26 13:11:35 -03:00
Colton Sellers 1358bd8115 Future and FutureOption low res support (#6069)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Fixes for FutureOptions support in LeanData

* Add CreateCanonicalOption() utility function for Symbol.cs

* Add aggregated Futures/FuturesOptions data to Lean

* Add FutureOptions regressions for daily/hourly data

* Allow Futures to be added with low resolution

* Add Future regressions using hour/daily data

* Nit - Python Class names

* Add reviews

* Add alias into CreateCanonicalOption
2021-11-24 09:53:21 -03:00
Martin-Molinero b8b0d18993 Continuous futures (#6034)
* Continuous Future Contracts

* Mapping approach

* Tweaks WIP

* Live mapping

* Live mapping

- Add support for live mapping, refreshing mapfiles
- Fix future expiration functions
- Adding unit tests

* Update moq test package

* Continuous futures price scaling

* Refactor price factors scaling

* Factor file related renames

* Address reviews
2021-11-15 14:44:30 -03:00
Colton Sellers dd4da7ba95 Feature Daily/Hourly Options Support (#6017)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* Create generic writing for LeanDataWriter, + notes on todos

* Make Options Daily/Hourly data store by year

* Refactor Generic Write

* Permit hour and daily resolutions for options

* Refactor writer to merge when needed with other files

* Cleanup redundancies, run write tasks in parallel

* Make needed classes/vars available

* Update tests to reflect new naming convention for daily hourly options data

* Add Byte[] overloads for ZipData functions in compression

* Implemented Store() for ZipDataCacheProvider

* Have LeanDataWriter use a DataCacheProvider

* ZipDataCacheProvider cleanup

* ZipDataCacheProvider tweaks, doesn't support storing non-zips

* Test adjustments

* Update LeanDataWriter to use Write instead of SaveDailyHourly/SaveMinuteSecond

* Implement tests to verify DownloadAndSave behavior

* Nit cleanup on DownloadAndSave tests

* Fix for options daily/hourly underlying equity subscription read

* Add daily/hourly options data and regressions

* Add missing open interest for hourly

* Fix writing of OpenInterest Daily/Hourly data

* Update data

* Fix Date typo in regression

* Use daily algorithm to test delisting

* Revisions part 1

* Expand test for DataCacheProviders; refactor DiskDataCacheProvider

* nit - test adjustments

* ZipDataCacheProvider test setup refactor

* Adjust multithreaded read/write test; fixes for ZipDataCacheProvider

* Move DiskDataCacheProvider to its own file and add write test

* Remove _appendToZips; always overwrite entry or create zip

* Add mapping regression for daily options

* nit - add license to regression

* Fix Tick write case where more than one data point for a DateTime

* Fix data issue

* Address review

* Tweaks for tests

* Stop Store() early if no entry name is given
2021-11-11 20:05:31 -03:00
Martin-Molinero 3410832ccf Alpha holdings state (#5770)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Alpha holdings state

- Alpha result packet will optionally provide the algorithms portfolio
  state

* Rename

* Convert AlphaStreamsPortfolio to data source

* Improvements on AlphaStreams algorithm

* Fix regression tests

* Add unit tests for EW AS PCM and fixing bugs

* Protobuf AlphaStreamsPortfoliot staState

- Protobuf AlphaStreamsPortfolioState. Adding unit tests
- Add variable TPV tests for EW ASPCM

* Add alpha license to Organization response

* Improvements EW AS PCM respects free portfolio value

* Fixes

- Update tests expected statistics results affected by MHDB custom data timezone fix
- Fix for Extensions.IsCustomDataType

* Fixed and adding more regression tests

- Adding support and regression test with alpha consumer with different account currency
- Adding support and regression test of a universe adding custom data
  types
- Add support and regression test for algorithm alpha consumer with existing holdings

* Add AlphaStreamsOrderEvent data type
2021-08-19 13:56:18 -03:00
Gerardo Salazar 650ce2e2b5 Removes References To Built-in Custom Data Implementations (#5793)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* WIP removal of custom data references & tests updates

* Regression algos updated and python algorithms moved to DataSource repos

* Fixes failing unit tests

* Add "LiveDataTypes" field to LiveNodePacket

* Adds Initialize() to IDataChannelProvider

  * Adds new extension method to convert
    HistoryRequest -> SubscriptionDataConfig

* Address review: Add protobuf definitions for Iconic data types

* Address review: re-adds DynamicSecurityDataAlgorithm as regression algo

  * Small adjustments to variable naming and documentation

* Move test files to respective DataSource repos
2021-08-02 19:24:45 -03:00
Colton Sellers 74f0c89499 Option Chain Provider Symbol Map Resolve (#5664)
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Use mapFileProvider to resolve symbol mapping

* Add reproducing unit test

* Only resolve mapping for types that require it

* address review
2021-06-16 19:42:38 -03:00
Gerardo Salazar 4c085ff853 Adds Indexes and Index Options asset types (Backtesting/Live, IB only) (#5379)
* Add support for Index SecurityType  🚀 (#5364)

* Add Index SecurityType  🚀

* Extend SecurityIdentifier & Lean Data classes with Index support

* Add Index SecurityType  🚀

* Extend SecurityIdentifier & Lean Data classes with Index support

* Fixes

* Added index cross basic template demonstration

* WIP: Prototype index security type for LEAN as non tradable asset

* Re-adds Index entries to MHDB after rebase

* First steps to getting Index Options running

  * Looks at any instance where we pattern match for an option type
    and replaces it with a generic call to `.IsOption()` for easier
    extensibility in the future for additional option security types

  * Adds IndexOption security and misc. classes

  * Misc. changes, mainly related to any sort of special casing of
    equity options and made index options take the same path

* Enables index options data for backtesting

  * Adds new index options market hours to MHDB
  * Misc. bug fixes for index options
  * WIP: add live support for index options and indexes
  * Use OptionMarginModel for Index Options because they both use the
    same calculation for margin requirements

* Fixes contract not found errors on SPX index options and SPX index in IB

  * Turns out index options' last trading day is the day before expiry,
    which IB was expecting the last trading day.

* Add index option test cases (temp)

* LiveOptionChainProvider fix, use Symbol vs. ticker

  * Description updates to regression algorithms

* Fixes bug in live trading for indexes and index options

  * Adds overridable minimum price variation symbol property
  * Adds variable sized minimum price variation for index options
  * Adjusts symbol properties for index options
  * Misc. bug fixes

* Fixes option assignment simulation for European options

  * Updates index options regression algorithms (WIP)

* Fixes bug where index option exercise would trade index underlying

  * Fixes bugs where SecurityType.Index was getting flagged as tradable

* Regression algorithms updates and addresses review

  * Misc. style fixes and refactoring + a few bug fixes
  * Updates regression algorithms to run without runtime errors
  * Adds data for regression algos

* Sets DefaultOptionStyle on Canonical and support index options

* Update regression algos statistics

* Removes bad line in regression algorithm causing build to fail

* Minor tweaks

* Address review add comment about quoteBar parse scale

Co-authored-by: Balamurali Pandranki <balamurali@live.com>
Co-authored-by: Jared Broad <jaredbroad@gmail.com>
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
2021-03-12 20:46:23 -03:00
Colton Sellers 041f1d9db5 Limit Futures Resolutions in BaseData (#4979)
* Limit futures support resolutions

* Fix broken tests

* Address review
2020-12-03 21:50:17 -03:00
Gerardo Salazar eb1181f5f7 Adds Futures Options Asset Class w/ IB Support (#4928)
* Adds preliminary universe selection for Future Options

* Fixes scaling issues with Future Options

* Fixes scaling multiplying by 10000x instead of using _scaleFactor

* Fixes scaling for Tick

* Revert changes to Tick since it divides the scaling factor

* Changes stale method name to new method name after rebase

* Fixes selection bugs, adds new methods, and adds unit tests

  * Fixes bug where Equity Symbol was created for an underlying
    non-equity Symbol, resulting in equity data trying to be loaded

  * Adds unit tests covering changes to Tick, QuoteBar, TradeBar and
    LeanData

  * Adds regression test for AddUniverseOption filter contract selection
    for Future Options

* Addresses review - modifies the AddFutureOption signature

  * Adds new AddUniverseOptions method overload
  * Removes and adds a new unit test
  * Misc. modifications to account for new changes

* Fixes bug where futures were loaded using default SID Date

  * Refactors and removes unnecessary work
  * Fixes regression algorithm, which previously made no trades

* Adds future option data

  * Adds the corresponding underlying data, in this case, futures data
    to enable usage of future options data

* Replaces data with new data (ES18Z20)

  * Improves Future chain filtering and updates regression stats

* Add AddFutureOptionContract API

* Expands regression and unit tests to test in finer detail

* Adds Python regression algorithms for AddFutureOption[Contract] methods

* Adds new unit test for BacktestingOptionChainProvider

  * Fixes bug with BacktesingOptionChainProvider where we
    attempted to load the Trades option chain first, resulting
    in breakage of backwards compatibility and limitation of the
    option chain.

  * Adds new regression algorithms (Py) to Algorithm.Python project

* Adds FutureOptionMarginBuyingPowerModel

  * Modifies code paths used to select margin model
  * Adds related unit tests for margin model

* Fixes issue with unit test and MHDB/SPDB lookup for Future Options

* Preliminary regression algorithm testing ITM call/put option buying

  * Fixes bug where fee model used did not find non-US market
    options fee model. We now use the futures fee model for future
    options because IB charges the same commissions per contract
    between futures and futures options

* Adds proper regression algorithm for ITM future options expiration

* Pushing broken algorithm for review

  * Currently, algorithm does not fill forward, causing
    a single future option to not get exercised when it is delisted.

* Adds FutureOptionPutITMExpiryRegressionAlgorithm

  * Improves existing regression algorithm for call side
  * Fixes bug in existing regression algorithm
  * Adds AAPL daily data to advance enumerator for ^^^ fix

* Adds additional future option regression algorithms

  * Adds Buy OTM expiration regression algorithms
  * Adds Sell ITM/OTM expiration regression algorithms
  * Adds missing Python regression algorithms

* Adds remaining Python regression algorithms and fixes issues

  * Fixes naming issues and statistics
  * Adds short option OTM regression algorithms (Py)

* Add license header and class comments to python algorithms

  * Cleans up comments and docstrings
  * Create Buy/Sell call intraday regression algo

* Redirects future options symbol properties to futures symbol properties

  * Asserts exercise/assignment price and updates stats in regression algos
  * Adds new unit test covering changes to SecurityService

* Adds comments and fixes failing test

* Partially fixes future option mis-calculated profit/loss

* Adjusts portfolio model to calculate FOP as a no upfront pay asset class

  * Updates regression algorithm statistics

* Begin IB FOP support

* Initial support for FOP IB data streaming, live í¾‰

  * Adds additional functionality to LiveOptionChainProvider
    - Allows querying CME API to retrieve option chains for CME products
    - Ultimately, it's also the groundwork for the CME
      LiveFutureChainProvider

  * Edits IDataQueueUniverseProvider interface to provide greater
    control to implementors of it

  * Misc. bug fixes required to get FOP data streaming through IB

* Adds comments, adds missing rategate call, and cleans up code

* Force exchange for FOP and Futures when no exchange is provided

* Fixes bug with Portfolio modeling across all asset classes

* Adds LiveOptionChainProvider tests for Future Options

* IB brokerage option symbol bug fixes and improvements

* Fixes contract multiplier lookup bug

  * Fixes issue where we attempted to subscribe to IB data feed with canonical security
  * Adds ES MHDB entry

* Reverts portfolio modeling changes for Futures Options

  * Since IB eats into our account's cash balance when
    a new FOP contract is purchased, we must model by applying funds
    to our cash whenever a new purchase/sell occurs.
    If we choose to model FOPs exactly as we do with futures, we
    will end up with an invalid TotalPortfolioValue on algorithm
    restart. By all means and purposes, FOPs are modeled exactly
    the same as equity options with respect to the portfolio.

  * Adds comments clarifying portfolio modeling and clarifies
    existing portfolio modeling comments with additional context.

* Fixes IB symbol lookup for future options

  * Fixes LiveOptionChainProvider looping 5 times per option chain
    request, even on success

  * Sets OptionChainedUniverseSelectionModel to produce a canonical
    future/future option/option Symbol to avoid creating two Symbols

  * Adds GLOBEX future option symbol mapping from future -> fop

* Fixes LiveOptionChainProvider loading wrong contract option chains

  * Fixes loading of futures options ZIP files when backtesting
  * Adds a string -> decimal JSON converter
  * Additional fixes/refactoring to the LiveOptionChainProvider

* Adds tests for changes to Symbol and LeanData

  * Reverts changes to IB-symbol-map

* Fixes Value for mapped future options tickers

  * Fixes Symbol test

* Changes path of future options to future's expiry date

  * Extra changes made to remove scaling from writing CSV
  * Added method to map from FOP Globex -> FUT Globex

* Fixes MOO and MOC orders for future options

  * Note: this order type might not be supported by IB or CME.

* Bug fixes and updates unit tests

* Update regression tests and data format

* Rebase changes

* 1. Multiple bug fixes for LiveOptionChainProvider, reverts IQFeed changes
2. Address review (partial): Code reuse and cleanup

1.
  * Modifies check in
    `AddFutureOptionShort(Call|Put)ITMExpiryRegressionAlgorithm`
    to ensure no buys have negative quantity

  * Code reuse changes in IB brokerage

  * Bug fix in IB brokerage where we assigned the FOP expiry
    as the futures expiry (requires verification)

  * Doc changes and adds missing summaries/license banners
  * Disposes of HTTP client resources in LiveOptionChainProvider
  * Renames classes and adds FutureOption folder in Common/Securities

2.
  * We revert back to the quotes API for the option chain,
    since the settlement API sometimes had missing strikes.

  * Fixes future option expiry being set as future's expiry
    in LiveOptionChainProvider

  * Fixes bug where wrong option chain was selected because of bad
    expiry lookup in the futures expiries returned from CME

  * Fixes multiple looping bug in LiveOptionChainProvider
  * Adds strike price scaling for LiveOptionChainProvider

  * Reverts IQFeed changes and simplifies interface upgrade changes

  Some additional challenges we'll have to solve as part of FOPs:

    - The `OptionSymbol.IsStandard` method makes the assumption that
      weeklies contracts follow the pattern equities follows, which
      does not apply to Futures Options

    - The Subscription created in:
        `OptionChainUniverseSubscriptionEnumeratorFactory`

      ...adds a Trade config. For illiquid contracts, this
      will delay universe selection for the option symbol
      until we get a trade. However, if we add a quote config,
      the data would instead be loaded based on the first quote
      we received from the brokerage.

      But since we're currently using a trade config, illiquid
      contracts won't start streaming data until it receives a trade.

NOTE: this commit is a WIP to addressing the reviews received in the PR,
but has been committed early for efficiency in the review process

* Fixes regression algorithms and misc. bugs

  * Fixes map file lookup for non-equity options
  * Adds extra assertion at end of algorithm to ensure no holdings are
    left when the algorithm ends.

  * Adds FutureOptionSymbol, allowing all contracts through as standard
  * Changes SPDB to allow defaulting to underlying future symbol
    properties if no entry is found for the given FOP

  * Fixes calls to SPDB in SecurityService, IBBrokerage
  * Reverts AAPL daily ZIP file to fix majority of regression algorithms
  * Adds FOPs symbol properties
  * Fixes existing symbol properties for a few futures
  * Adds tests for changes to Symbol Properties Database

* Removes string SPDB lookup method

  * Updates tests and misc callees of previous method

* Updates all regression tests to use data of already expired contracts

  * Adds Futures Options Expiry Functions tests
  * Adds required futures data for 2020-01-05

* Address review (partial): Expands test coverage and fixes tests

* Set option chain tests parallelism to fixture only

* Fixes broken test for contract month delta for FuturesOptionsExpiryFunctions

* Changes delisting date logic for Futures Options

* Address review: removes duplicate code, misc code fixes

  * Bug fix in MarketHoursDatabase.GetDatabaseSymbolKey() where
    we would use the underlying's Symbol for lookup in the MHDB

  * Adds missing license banner
  * Removes Futures Options entries from MHDB
  * Adds new tests

* Adds SecurityType.FutureOption

  * Converts any underlying comparisons and uses SecurityType directly
    instead for FOP specific behavior

  * Extra code modifications to acommodate new SecurityType

* Addresses review: fixes order fee bug on exercise

  * Additional bug fixes and adding of SecurityType.FutureOption
  * Updates regression algorithms OrderListHash

* Fixes various bugs in IB live implementation

  * Fixes bug setting the right contract expiration date for FOP
    generated by LiveOptionChainProvider

  * Adds new function to FuturesOptionsExpiryFunctions

  * Clarifies parameter names better in some functions/methods

  * Fixes bugs in IB brokerage for FOPs

* Address review - code cleanup and refactor

  * Remove MappingEventProvider, SplitEventProvider, and
    DividendEventProvider for Futures Options in
    CorporateEventEnumeratorFactory

* Address review: Use MHDB key resolver in SPDB

* Makes regression tests pass and adds comment for expiry issue

* Fixes MHDB lookup on string symbol method

* Adds Futures Options greeks regression algorithm (C# only)

* Adds explanitory comment on MHDB FOP lookup

* Remove python from FutureOptionCallITMGreeksExpiryRegressionAlgorithm
2020-12-02 21:49:59 -03:00
Martin-Molinero 61fda8b62c Protobuf will use recyclable memory stream (#4921)
* Protobuf will use recyclable memory stream

- Serialization will reuse recyclable memory streams
- Remove serialization of exchange and sale condition for ticks.
  Updating unit tests.
- There is no need to serialize BaseData.EndTime, covered by unit tests.

* Tick will keep a parsed sale condition property

* Readd tick exchange. Json ignore ParsedSaleCondition
2020-11-06 20:59:25 -03:00
Martin-Molinero 9cdb4a91c5 Refactor live data feed (#4636)
* Live Coarse universe refactor

- Live trading will source Coarse and Fine fundamental data directly
  from disk. Updating unit tests.

* Adds ILiveDataProvider interface

  * Adds wrapper for IDataQueueHandler implementations

  * Replaces IDataQueueHandler with ILiveDataProvider in
    LiveTradingDataFeed

  * Edits IDataQueueHandler documentation

* Maintains aggregation for current IDQH impls and skips for ILDF impls

  * Note: No unit test was created for this method, go back and TODO

* Protobuf Market data

- Adding protobuf support for Ticks, TradeBars and QuoteBars. Adding
  unit tests.

* Adds unit tests for LiveDataAggregator changes

  * Fixes bug where custom data was not handled as it was before
  * Fixes race condition bug because of variable reuse in class

* Add protobuf extension serialization

* Fixes for protobuf serialization

* Refactor

* Fix OptionChainUniverse

* replace BaseDataExchange pumping ticks with consolidators

* AlpacaBrokerage

* BitfinexBrokerage

* GDAXBrokerage

* OandaBrokerage

* InteractiveBrokers

* TradierBrokerage

* FxcmBrokerage

* PaperBrokerage

* etc

* WIP fixes for existing LTDF unit tests

* Fixes more LTDF unit tests

* make IDataAggregator.Update recieving Generic BaseData rather than Tick

* Change IDataQueueHandler.Subscribe method

* Some fixes after adding new commits

* Adds protobuf (de)serialization support for Dividend and Split

* Serialize protobuf with length prefix

* Fix missing LTDF unit tests

* Adds TiingoNews protobuf definitions

* fix comments

* more fixes on IQFeedDataQueueHandler

* disallow putting ticks into enumerator directly

* ScannableEnumerator tests

* fix OandaBrokerage

* AggregationManager unit tests

* fix AlpacaBrokerage tests

* fix InteractiveBrokers

* fix FxcmBrokerage tests

* call AggregationManager.Remove method on unsubscribe

* fix GDAX existing tests

* Fixes, refactor adding more tests for AggregatorManager

* Adds BenzingaNews protobuf definitions and round trip unit test

* Adds missing TiingoNews unit test to Protobuf round trip tests

* Improve sleep sequence of LiveSynchronizer

* need start aggregating first, and then can subscribe

* More test fixes and refactor

- Refactoring AggregationManager and ScannableEnumerator so the last is
  the one that owns the consolidator
- Adding pulse on the main LiveSynchronizer

* Improve performance of LEquityDataSynchronizingEnu

* Add missing Set job packet method

* Minor performance improvements

* Improvements add test timeout

- Improvements adding test timeout to find blocking test in travis

* Improve aggregationManager performance

* Testing improvements for travis

* Remove test timeouts

* More test fixes

- Adding more missing dispose calls and improving determinism

* fix IEXDataQueueHandler and tests

* Final tweaks to LTDF tests

* more AggregationManager tests

* consume and log ticks

* fix test: couldn't subscribe to Forex tickers

* change Resolution for all bar configs

* Improve RealTimeScheduleEventServiceAccuracy

* refactoring: move common code to base class

* fixed bug; unsubscribe SubscriptionDataConfig

* Small performance improvement

* Minor fixes

* Avoid Symbol serialization

* Fixes coarse selection in live mode

* Fix for live coarse

* Adds protobuf (de)serialization support for Robintrack

  * Adds round-trip unit test

* Minor performance improvements

* More minor performance improvements

* pass LiveNodePacket through to OandaBrokerage

* Fixes empty list becoming null value when deserializing with protobuf

* Reverts BZ live trading exception removal and fixes tests

* Refactor WorkQueue making it abstract

* Add try catch for composer

* Adds optional data batching period to LiveFillForwardEnumerator

* Override data-queue-handler with config

* Improve PeriodCountConsolidator.Scan performance

* Move batching delay to main Synchornizer thread

* Reverts addition of Robintrack protobuf definitions

* Give priority to config history provider if set

* Add Estimize protobuffing

- Add Estimize protobuffing support. Adding unit tests

* Always dispose of data queue handler

Co-authored-by: Gerardo Salazar <gsalaz9800@gmail.com>
Co-authored-by: Adalyat Nazirov <aenazirov@gmail.com>
2020-08-18 20:21:10 -03:00
Martin Molinero 3aa745ee42 Add StreamReader BaseData.Reader
- Add `BaseData.Reader` implementation which consumes the `StreamReader`
directly, avoiding in between substrings and parsing improving
performance and reducing resource consumption
- Adding unit tests, including performance unit test showing a >50%
improvement
2019-11-18 17:03:27 -03:00
Martin Molinero 68ad2f51b6 Address reviews
- Replacing `BaseData.AdjustResolution` for `DefaultResolution` and
`SupportedResolutions`
- Making `Resolution` nullable for `Algorithm.AddData` methods
- The `ISubscriptionDataConfigService` will set the default resolution
if none was provided and assert it is supported
- Fix bug with `PythonData` `IsSparseData` and `RequiresMapping`
resolution
2019-11-04 20:42:30 -03:00
Martin Molinero 1d43dcd601 Add BaseData.AdjustResolution
- Adding `BaseData.AdjustResolution()` that should return a valid
resolution for the given data and security type.
This allows us to set a limitation which is useful to avoid invalid data
requests or unnecessary fill forward situations. The user will be
notified through a console message.
- Adding unit and regression test
- Updating example algorithms custom data resolution
- Some performance improvements. Wont change console color if
`SelectedOptimization` is defined
2019-11-04 20:38:26 -03:00
Martin Molinero af6cc80b48 Adding BaseData.DataTimeZone()
- Adding `BaseData.DataTimeZone()` with the objective of allowing custom
data types to determine their data time zone.
2019-09-30 23:59:36 -03:00
Michael Handschuh 7807ae7af6 Add BaseData.IsSparseData
Flags a particular data type as having a sparse data set, meaning,
we don't expect it to have data for every single day. We'll use
this to prevent spamming log messages
2019-09-27 14:24:08 -04:00
Michael Handschuh c1ee66e26d Use auto-properties in BaseData
See #3618
2019-09-27 14:24:08 -04:00
Michael Handschuh d709d1c4e0 Update Common to respect CA1304 and CA1305
Updates all occurrences of parsing/ToString-ing to go through the new
StringExtensions methods that use CultureInfo.InvariantCulture

See #3045
2019-09-12 14:13:24 -04:00
Martin Molinero 24527e0cc2 Address reviews - Renaming 2019-08-21 16:30:03 -03:00
Martin Molinero 73d780daae Moving UsesMapFiles to DataType
- Custom data types will know whether or not Lean should use map files
- Updating regression test with sample custom data using map files,
which can run locally
- Adding unit tests for the `SubscriptionDataReaderHistoryProvider`,
checking it mappes equities and options correctly
2019-08-20 17:33:41 -03:00
Stefano Raggi db942e0d2e Zero volume out in fill forward TradeBars 2018-09-25 22:29:53 +02:00
Michael Handschuh 57affafdc2 Implement 'live' factor files
Adds reference price to dividend/split
Adds GetSplitsAndDividends to FactorFile
Adds Apply methods to FactorFile/FactorFileRow
Updates factor files to include reference prices
2018-08-14 17:58:44 -04:00
Stefano Raggi 508e63af4d Update Bar types Volume and Bid/AskSize from long to decimal
This PR is another required step towards crypto-currency support.
2017-06-26 21:14:55 +02:00
Andrew Hart 4121856e80 Handle subscription errors from server in ApiWebSocketConnection 2016-11-30 08:27:35 -05:00
Andrew Hart 505ebb947a ApiWebSocketConnection does not use seperate thread for websocket 2016-11-07 18:37:15 -05:00
snugs d854769e9b Convert raw data time to exchange time in Reader methods 2015-12-01 12:20:06 -05:00
snugs ef4e3f5ab5 Clean up warnings from bad pragma comments 2015-11-19 19:46:36 -05:00
snugs d9dcd9b9aa Clean up Obsolete compiler warnings 2015-11-18 13:31:29 -05:00
snugs b2d96bdfa0 Fixes bug in TradeBar.Update and QuoteBar.Update
The issue here was if we initialized with a default instance (all zeros) then
the Low would be forever zero.

In it's new form we use an initialized flag to determine whether or not the bar
has been seeded, and if it hasn't been seeded yet, seed OHLC with the current
data
2015-11-05 13:30:51 -05:00
snugs f4c3e5746f BaseData.Update now includes bidSize and askSize
Removed copied functions from TradeBar.cs - replaced with NotImplementedException
    - These functions relied on QuoteBar.OHLC being writeable, now they're read only

QuoteBar.Clone - fix object reference bug
2015-10-31 18:48:07 -04:00
snugs e1938baafe Clean up using directives 2015-09-01 22:17:35 -04:00
snugs df79ce30c2 Adds the Symbol type
This includes updating all usages of symbol as a security identifier to use the new type.
The type includes a unique field, SID, as well as the current ticker's value. This allows
for consistent addressability while also allowing the ticker to evolve over time with the
mapping changes.

Effort was made to maintain compile and runtime backwards compatibility.
2015-09-01 22:17:10 -04:00
snugs 4bc2092d8c Adds OnData(SymbolChangedEvents) handler 2015-08-25 21:22:43 -04:00
snugs bbacf2b674 Remove remaining /** banners 2015-07-09 16:18:17 -04:00
QuantConnect fde0cbed3a Strip out block quote section header comments 2015-06-13 16:04:27 -04:00
snugs 44a38f3e1d Updates LiveTradeDataFeed to allow custom data with remote files 2015-06-11 17:34:54 -04:00
snugs 96396c676a OBSOLETES:: BaseData.Reader and BaseData.GetSource
The existing method signatures have been made obsolete in favor of something slightly simpler and more deterministic.
The new methods accept a bool isLiveMode flag instead of the DataFeedEndpoint.
The GetSource method returns a new type: SubscriptionDataSource, which combines the source string and the SubscriptionTransportMedium.

Existing sub-types of BaseData have been updated with the new signatures.  IBaseData has not been touch as it is mostly unused by LEAN infrastructure.
2015-06-11 17:34:53 -04:00
snugs 89b6e98af8 Fixes warnings mostly relating to xml comments
Also cleans up some block comments

Via PR#72 - thanks @ammachado!
2015-05-13 10:56:47 -04:00
bizcad bed1941b77 issue-79/Assure-Sell-quantity-is-negative 2015-05-09 19:18:30 -07:00
snugs 49b398f8ff DataStream sync by end of bar
Adds EndTime to BaseData
Adds Period to TradeBar
Now filling data forward on days missing files/data
Some performance enhancements
Removed sorted dictonary in DataStream/AlgorithmManager

We round down fill forward data in the data stream since if we were to perform this in the data feed we would continually produce data at the same time and it would all go into the algorithm at once.
2015-05-01 19:23:57 -04:00
snugs 6fd357b710 Adds support for dividends and splits
Updates SubscriptionDataReader to use IStreamReader
Adds FactorFile and MapFile helpers
Adds support for Raw, Adjusted, and TotalReturn data normalization and portfolio application
2015-04-23 20:39:59 -04:00
snugs 3af7d5ed43 Adds firing of dividend events 2015-04-23 14:31:15 -04:00