Commit Graph

200 Commits

Author SHA1 Message Date
Gerardo Salazar c67845bd45 Fixes issue parsing SI data w/ unknown enum value (#4961)
* Extends tests to cover new additions
2020-12-01 20:17:25 -03:00
James Kardatzke 46ef9a9dbb Quiver Quantitative Integration (#4869)
* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Delete QuiverHouseDataDownloader.cs

* Delete QuiverSenateDataDownloader.cs

* Delete QuiverPoliticalBetaDataDownloader.cs

* Add files via upload

* Delete QuiverHouse.cs

* Delete QuiverSenate.cs

* Delete QuiverPoliticalBeta.cs

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Delete QuiverDataAlgorithm.cs

* Add files via upload

* Add files via upload

* Add files via upload

* Addresses self review: Cleans up code and adds new unit tests

  * Adds Quiver* C# files to project
  * Adds new unit test for QuiverCongress
  * Adds Python algorithm example

* Address self reviews

- Adding some missing xml docs
- Removing unrequired imports.
- Minor rename from Date to ReportDate
- Live trading will throw InvalidOperationException

* Fixes for example algorithms

Co-authored-by: Gerardo Salazar <gsalaz9800@gmail.com>
Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
2020-11-13 20:07:17 -03:00
Juan José D'Ambrosio 8e525c63fc Fix issue in TiingoNews converter if tickers contains space (#4908)
Also, if ticker contains pipe ( "|" )  it'll be ignored.
2020-11-02 13:58:39 -03:00
Gerardo Salazar 28160e1301 Fixes generation of explicit "null" value in Smart Insider Transactions/Intentions (#4817)
* Adds new unit tests covering changes and testing for old case

  * JsonConvert.SerializeObject would convert a `null` value into a
    literal string of "null" when writing to a file via the ToLine
    abstract method. We opt for an empty string whenever the underlying
    value is null so that the parsing works correctly later in the
    data loading cycle.
2020-10-06 10:20:59 -03:00
Martin-Molinero 9cdb4a91c5 Refactor live data feed (#4636)
* Live Coarse universe refactor

- Live trading will source Coarse and Fine fundamental data directly
  from disk. Updating unit tests.

* Adds ILiveDataProvider interface

  * Adds wrapper for IDataQueueHandler implementations

  * Replaces IDataQueueHandler with ILiveDataProvider in
    LiveTradingDataFeed

  * Edits IDataQueueHandler documentation

* Maintains aggregation for current IDQH impls and skips for ILDF impls

  * Note: No unit test was created for this method, go back and TODO

* Protobuf Market data

- Adding protobuf support for Ticks, TradeBars and QuoteBars. Adding
  unit tests.

* Adds unit tests for LiveDataAggregator changes

  * Fixes bug where custom data was not handled as it was before
  * Fixes race condition bug because of variable reuse in class

* Add protobuf extension serialization

* Fixes for protobuf serialization

* Refactor

* Fix OptionChainUniverse

* replace BaseDataExchange pumping ticks with consolidators

* AlpacaBrokerage

* BitfinexBrokerage

* GDAXBrokerage

* OandaBrokerage

* InteractiveBrokers

* TradierBrokerage

* FxcmBrokerage

* PaperBrokerage

* etc

* WIP fixes for existing LTDF unit tests

* Fixes more LTDF unit tests

* make IDataAggregator.Update recieving Generic BaseData rather than Tick

* Change IDataQueueHandler.Subscribe method

* Some fixes after adding new commits

* Adds protobuf (de)serialization support for Dividend and Split

* Serialize protobuf with length prefix

* Fix missing LTDF unit tests

* Adds TiingoNews protobuf definitions

* fix comments

* more fixes on IQFeedDataQueueHandler

* disallow putting ticks into enumerator directly

* ScannableEnumerator tests

* fix OandaBrokerage

* AggregationManager unit tests

* fix AlpacaBrokerage tests

* fix InteractiveBrokers

* fix FxcmBrokerage tests

* call AggregationManager.Remove method on unsubscribe

* fix GDAX existing tests

* Fixes, refactor adding more tests for AggregatorManager

* Adds BenzingaNews protobuf definitions and round trip unit test

* Adds missing TiingoNews unit test to Protobuf round trip tests

* Improve sleep sequence of LiveSynchronizer

* need start aggregating first, and then can subscribe

* More test fixes and refactor

- Refactoring AggregationManager and ScannableEnumerator so the last is
  the one that owns the consolidator
- Adding pulse on the main LiveSynchronizer

* Improve performance of LEquityDataSynchronizingEnu

* Add missing Set job packet method

* Minor performance improvements

* Improvements add test timeout

- Improvements adding test timeout to find blocking test in travis

* Improve aggregationManager performance

* Testing improvements for travis

* Remove test timeouts

* More test fixes

- Adding more missing dispose calls and improving determinism

* fix IEXDataQueueHandler and tests

* Final tweaks to LTDF tests

* more AggregationManager tests

* consume and log ticks

* fix test: couldn't subscribe to Forex tickers

* change Resolution for all bar configs

* Improve RealTimeScheduleEventServiceAccuracy

* refactoring: move common code to base class

* fixed bug; unsubscribe SubscriptionDataConfig

* Small performance improvement

* Minor fixes

* Avoid Symbol serialization

* Fixes coarse selection in live mode

* Fix for live coarse

* Adds protobuf (de)serialization support for Robintrack

  * Adds round-trip unit test

* Minor performance improvements

* More minor performance improvements

* pass LiveNodePacket through to OandaBrokerage

* Fixes empty list becoming null value when deserializing with protobuf

* Reverts BZ live trading exception removal and fixes tests

* Refactor WorkQueue making it abstract

* Add try catch for composer

* Adds optional data batching period to LiveFillForwardEnumerator

* Override data-queue-handler with config

* Improve PeriodCountConsolidator.Scan performance

* Move batching delay to main Synchornizer thread

* Reverts addition of Robintrack protobuf definitions

* Give priority to config history provider if set

* Add Estimize protobuffing

- Add Estimize protobuffing support. Adding unit tests

* Always dispose of data queue handler

Co-authored-by: Gerardo Salazar <gsalaz9800@gmail.com>
Co-authored-by: Adalyat Nazirov <aenazirov@gmail.com>
2020-08-18 20:21:10 -03:00
Gerardo Salazar b866c8b893 Removes Robintrack data source (#4628) 2020-08-14 19:46:30 -03:00
Juan José D'Ambrosio 211f1144b5 Implement NotSpecified SmartInsiderEventType 2020-07-29 21:08:52 +00:00
JJD e7682cbd37 Add Plan Re-started to SmartInsiderEventType 2020-07-29 11:24:44 -03:00
Gerardo Salazar 9087b0f782 Address review - designate Smart Insider timezone as UTC 2020-05-21 10:01:32 -07:00
Gerardo Salazar f610b3a4fb Updates Smart Insider transactions and intentions to use different time index
* We now use the time at which Smart Insider added a given event to
   their database. Because time indexes for Smart Insider are now
   non-duplicated per symbol, the History request will properly work.
2020-05-20 14:55:28 -07:00
Jared 88cb12ae8c Update SmartInsiderEventType.cs 2020-05-19 08:09:47 -07:00
Jared c82ad8638a Merge pull request #4399 from gsalaz98/bug-4382-smartinsider-new-eventtypes
Adds missing fields from intentions dataset to SmartInsiderEventType enum
2020-05-15 14:38:25 -07:00
Jared 1403b2370d Update SmartInsiderEventType.cs 2020-05-15 14:38:12 -07:00
Martin Molinero 6009e3ffeb Fix TiingoNews Time
- Fix TiingoNews time for backtesting, converter will store and organize
  data based on the 'QC time' which is crawldate or publish data plus an
  offset
2020-05-12 16:15:26 -03:00
Martin Molinero 269b8ba204 Remove PsychSignal data
- Remove PsychSignal data since it's no longer available
2020-05-11 18:48:13 -03:00
Gerardo Salazar 51cbbf404b Adds missing fields from intentions dataset to SmartInsiderEventType enum 2020-05-11 11:50:59 -07:00
Gerardo Salazar d2eb0482fa Address reviews - refactors portions of code and adds tests 2020-05-08 18:22:16 -07:00
Gerardo Salazar d3a5f52fe9 Implements Robintrack BaseData and example algorithms 2020-05-07 16:26:15 -07:00
Stefano Raggi 9034da2716 Fix EstimizeConsensus.Symbol serialization 2020-03-31 16:36:21 +02:00
Jared f0187cebf3 Merge pull request #4233 from QuantConnect/bug-4231-trading-economics-calendar-serialization-fix
Fix TradingEconomicsCalendar.Symbol serialization
2020-03-20 16:38:57 -07:00
Stefano Raggi acf6f8d6dc Address review - remove Symbol override 2020-03-21 00:23:24 +01:00
Stefano Raggi 4ffcb7cad9 Fix TradingEconomicsCalendar.Symbol serialization 2020-03-20 21:54:25 +01:00
Gerardo Salazar ad3236e0e8 Skips serialization of null values in BenzingaNewsJsonConverter
* Adds new unit tests for BenzingaNewsJsonConverter

* Fixes issue in BenzingaNewsDataConverter where we would
  write data in local time, resulting in non-UTC times from
  the deserialization in BenzingaNews.Reader(...). Unit tests
  were added for this issue as well
2020-03-19 17:56:34 -07:00
Stefano Raggi 48c184344d Fix serialization of BenzingaNews Time and EndTime properties 2020-03-19 12:28:52 +01:00
Stefano Raggi 319df53535 Fix deserialization of TradingEconomicsCalendar Time property 2020-03-13 17:40:53 +01:00
Stefano Raggi 7e652cba47 Fix deserialization of Estimize Time property 2020-03-12 20:53:22 +01:00
AlexCatarino 1abe6274db Replace <inheritdoc/> by Hard-Coded Documentation 2020-02-07 23:49:28 +00:00
AlexCatarino 1d9e719a0b Implements NullData Custom Data
Represents a custom data type that works as a heartbeat of data in live mode
2020-02-07 21:39:24 +00:00
Gerardo Salazar 55361cda98 Change TradingEconomicsCalendar to use SetAuthCode(...) pattern 2020-02-05 14:09:09 -08:00
Gerardo Salazar 01f7629c70 Address review - remove GetNextCsv() in favor of ToCsvData()
* Fixes bug in ToCsvData() where an empty final value would not be
parsed
* Removes GetNextCsv()
* Reworked `Reader` logic in TradingEconomicsCalenda
* Use delimiter var as separator in TradingEconomics.Calendar
* Convert country names to uppercase in TradingEconomics.Calendar
2020-02-05 11:44:57 -08:00
Gerardo Salazar bbe1416632 Address review - Refactor and add additional tests
* Refactors CountryToCurrencyCode(...) method
* Adds new test cases to GetNextCsv() method
* Refactors GetNextCsv() to prevent code duplication
2020-02-04 13:48:05 -08:00
Gerardo Salazar 5db43205ef Fixes issues for TradingEconomicsCalendar live trading
* Adds new tests for successful continuation of the stream in case of an
error
* Deletes unused `ReadToEndOfLine()` method
2020-02-04 13:48:04 -08:00
Gerardo Salazar 050089a3ae Address review and add TradingEconomics.Event definitions
* Updates TradingEconomics algorithms to use new event definition
* Separated Calendar and Indicator definitions into partial class
* Refactors portions of TradingEconomicsCalendar
* Makes TradingEconomicsCalendar.GetSource return RemoteFile for live
* Fixes bugs in TradingEconomicsEventFilter
* Fixes bugs in StreamReaderExtensions (thanks Martin :))
* Adds new unit tests to cover changes
2020-02-04 13:48:04 -08:00
Gerardo Salazar f85e59383a Reworks storage of Trading Economics calendar events
* Adds support for live algorithms using TE calendar events
* Modifies TradingEconomics tickers to include country
* Fixes bug where `TECal.Clone(...)` could result in non-deterministic
results
* Fixes bugs in TradingEconomicsEventFilter
* Adds new CSV parsing methods in StreamReaderExtensions
* Moves ToolBox/TradingEconomicsCalendarTests to Common tests folder
* Updates TradingEconomicsCalendar tests to test CSV parsing
* TradingEconomicsCalendarDownloader now writes files to disk as JSON
2020-02-04 13:48:04 -08:00
Gerardo Salazar 0a87b78ae8 Address review - update documentation and exception messages 2020-01-28 09:55:47 -08:00
Gerardo Salazar ec9de51ca3 Address review - Applies new datetime parsing to intentions data
* Moves SmartInsider Transactions DateTime parse method to base class
* Replace "CSV" mentions with "TSV" in documentation
* Update tests
2020-01-28 09:40:10 -08:00
Gerardo Salazar 58cf6d57ec Fixes parsing of new date format in Smart Insider transactions data
* Makes converter throw hard error if it fails to parse from raw data
* Adds new unit tests
* Fixes typos where we called data `CSV` instead of `TSV`
2020-01-27 18:20:16 -08:00
Gerardo Salazar e7a9d06b16 Implement filtering and normalization of Trading Economics calendar
event names

* Adds unit test for new filter method

NOTE: This is a breaking change for Trading Economics calendar events
property. Any previous string matching might result in a mismatch if
results were not previously being normalized.
2020-01-27 11:32:26 -08:00
Gerardo Salazar 55b8447012 Fixes failing EstimizeConsensus parsing in Reader 2019-12-09 11:17:29 -08:00
Jared ec4f714799 Merge pull request #3862 from gsalaz98/feature-3775-add-benzinga-custom-data
Implements Benzinga Custom Data
2019-11-27 15:39:22 -08:00
Gerardo Salazar e1ae9807ab Address review - Disable live mode for BenzingaNews 2019-11-26 11:53:27 -08:00
Gerardo Salazar 4bd441bead Address review - Write indexes based on UpdatedAt time instead of
reference

* Code cleanup as per review
2019-11-26 10:51:32 -08:00
Gerardo Salazar 8c16ffde8d Address review - Fixes bug where tags were not being deserialized
* Write files by `UpdatedAt` date instead of the `CreatedAt` date
* Added ability to append to compressed files if file does not exist
* Cleaned up code as per review
* misc. documentation changes
2019-11-25 14:32:45 -08:00
Gerardo Salazar a3c96059f7 Address review - Remove RSS converter and only parse API responses
* RSS converter now lives at: https://gist.github.com/gsalaz98/b87992cd5a5a214d01c63dbbefbbf12c
* Removed BenzingaNewsFactory as it is no longer needed
* Cleaned up code as per review to make this more maintainable
* Included new documentation
* Filter articles with no Symbols from being written
* General code improvements (duplication reduction, etc.)
2019-11-20 14:47:32 -08:00
Gerardo Salazar d79d720919 Address review - Makes FixedLiveOffset private readonly in PsychSignalSentiment
* Fix styling to follow project guidelines
2019-11-19 14:29:49 -08:00
Gerardo Salazar 791f75389f Fixes bug where time would be output as 15 seconds past the original EndTime
* Avoids boxing of time values (performance)

This fix addresses Martin's review and concerns that the EndTime would
be rounded down before being emitted. Local testing shows that EndTime
and _algorithm.Time are aligned when the data is included in the Slice
at HH:mm:15s.

Regarding the bug fix, previously when Clone was called, we would set
EndTime equal to EndTime, advancing the time by one minute and fifteen
seconds. This would make the time show up as HH:mm:30s and a minute into
the future.

Additional note: the only place where EndTime is being rounded down is in
AlgorithmManager in the method EndTimeIsNativeResolution which is only
used for Consolidators.
2019-11-19 14:29:49 -08:00
Gerardo Salazar e440efd72d Address review - Increase FixedLiveOffset to 15 seconds
This is done to accurately model live trading. Because PsychSignal
takes approximately 10-15s to aggregate their data for the past minute,
we need to take into account the live delay. Otherwise, we'll have data
emitted before it was actually available.
2019-11-19 14:29:49 -08:00
Gerardo Salazar 0862e0e549 Fixes EndTime of PsychSignalSentiment
* Adds FixedLiveOffset property
* Implements `Period` property for calculating `Time` and `EndTime`
2019-11-19 14:29:49 -08:00
Gerardo Salazar 435735a738 Add support for tickers with share classes
* Documentation changes
2019-11-19 13:56:02 -08:00
Gerardo Salazar e4789e0b23 Implement Benzinga News Downloader
* Make Benzinga News Converter more efficient (and cleaner!)
* Implement JSON converter for serialization/deserialization process
* Fix bug where duplicate symbols would be added to news.Symbols
* Modify algorithm strategies
2019-11-19 10:28:48 -08:00