17 Commits

Author SHA1 Message Date
Jhonathan Abreu 496f210116 Fix start time calculation for daily requests (#8975)
Python Virtual Environments / build (push) Has been cancelled
API Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Syntax Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Report Generator Tests / build (push) Has been cancelled
* Fix start time calculation for daily request

* Minor regression algorithm fix
2025-09-15 18:35:25 -03:00
Ricardo Andrés Marino Rojas dcb5f8ee4e Merge branches 7501 and 7506 (#7605)
Research Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* Update Future-cme-[*] and Future-cme-ES

Acoording to `pandas_market_calendars` there were some dates in
Future-cme-[*] who wasn't early_closes, so they needed to be removed
from there. On the other hand, the early closes list of Future-cme-ES were
shifted by 1 hour according to CME webpage. Besides, there were some
missing dates.

* Update CME Future entries in MHDB

* Rebase

* nit change

* Fix unit tests

* Resume after early close/halts

* Add missing dates in MHDB and fix bugs in it

* Fix bug, add more unit tests and add docs

* fix regression algos

* address required changes

* Update failing regression test stats

After debugging the tests it was found they were failing due to the last
change on SecurityExchangeHours.IsOpen(). That method wasn't taking into
account that even if there is a late open after an early close if the
timespan is after the early close but before the late open, the market
is still close.

* enhance solution

* Update and fix bugs in MHDB

* Address required changes and update stats

* Update stats after rebase

* Nit change

* Missing update to regression test

* Use MHDB instead of USHoliday for Expiration Dates

VIX expiry function now relies completely on MHDB. However, it had to be
created an entry in MHDB for VIX since there wasn't one for it. CBOE
webpage only provided 2023 holidays so only those dates were considered
in the Holidays entry in MHDB. Therefore, some unit tests failed so it
was necessary to change also the VIX entry in FuturesExpiryFunctionsTestData.xml.

* Remove Global.cs/USHolidays class

* Use a lazy implementation

* First draft of the solution

* Use MHDB in FuturesExpiryFunctions.cs

* Remove unused class and fix indentation errors

* Fix indentation errors

* Nit changes

* Merge branches 7501 and 7506

* Merge changes in 7501 and 7506

In order to check compatibility between those branches, a new branch
was created out of branch 7501 and then it was merged with branch 7506.
2 regression tests and 8 unit tests failed, the regression tests failed on
the DataPoint stats. On the other hand, the unit tests failed since the
default parameter UseEquityHoliday was removed from
FuturesExpirtyUtilityFunctions.AddBusinessDays() and from other methods in
the same class too.

* Add missing changes

* Remove repeated good fridays

* Address minor review

---------

Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
2023-12-04 18:09:57 -03:00
Martin-Molinero ee3cea805e FactorFiles bases live auxiliary events (#7293)
- Adjusting live stack to emit live auxiliary events based on factor files.
    Updating existing tests
  - Minor improvements for map & factor file provider daily refresh time.
    Adding unit tests
2023-06-08 13:27:55 -03:00
Stas Kotykhin 3fc042af33 Move bar aggregators to utils, add time overloads & add check of orderType for Kraken (#5969)
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* Move Aggregate Quote and Trade Bars to utils

* Add decimal and long overload to UnixTimeStampToDateTime

* Add Kraken OrderType check in CanSubmitOrder

* PR !5969 review fixes
2021-10-08 17:59:32 -03:00
Gerardo Salazar 82c9b6ccb7 Updates and improvements to ToolBox projects (#5537)
Build & Test Lean / build (push) Has been cancelled
* Adds processed data directory to read price data from

* Make coarse universe generator look at data directory before failing to find daily data

* Set coarse generator output of missing daily file to debug log

* Add CoarseUniverseGenerator logs

* Fixes 100 nanosecond increment lookahead bias when parsing large numbers

  * Whenever we parse a number that is has precision greater than
    DateTime ticks (sub-100 nanoseconds), if we have nanoseconds
    between [0, 1000), excluding numbers divisible by 100,
    we will have leftover nanoseconds between [0, 100) nanoseconds, but
    they won't be factored in to the DateTime calculation, since casting
    to `long` only takes the integer component of the number, so we lose
    the extra nanoseconds that came with the decimal, and time is set to
    the "floored" value without those nanoseconds.

    Since .NET `DateTime` type has a limitation of only being able
    to represent time in increments of 100 nanoseconds, by not
    considering the sub-100 nanoseconds, we introduce a look-ahead
    bias of at most 100 nanoseconds/1 tick

  * Misc adjustment to make method use `decimal` instead of `double`
    for increased precision when parsing large numbers

* Changes CoinAPI data converter to support processing raw files in original directory structure and file name

  * Removes Market requirement from CoinAPI data converter

* Remove timeout on decompression of raw AlgoSeek futures data

* Updates SEC downloader to use HttpClient where requests were failing

  * For some unknown reason, valid requests to a valid URL were
    failing when using WebClient. Changing our requester to
    HttpClient fixes the issue, and enables us to leverage
    async capabilities where applicable.

  * Added fault tolerance to index file downloads, including a
    rate limit in case we've been rate limited

* Further refactoring; catches 429 errors, adds missing rategate calls

* Replace all usage of WebClient, force retry for all failures

* Adds optional config value for Benzinga News API key in downloader

* Modifies Estimize Downloader api config name and fixes directory not found bug

* Refactor Estimize to speed up processing time

  * Adds ticker limits if desired
  * Misc. bug fixes, performance improvements, code cleanup

* Remove debug log statements leftover from previous commit

* Add support for non-tick Index resolutions in LeanDataWriter

* Empty commit

* Empty commit

* Empty commit

* Empty commit

* Empty commit

* Empty commit

* Lower requests/second for SEC downloader, add missing rategate call

Co-authored-by: Martin-Molinero <martin@quantconnect.com>
2021-05-14 17:06:47 -03:00
Adalyat Nazirov c6d29c8be5 add FIX Protocol UTC timestamp format (#5212) 2021-01-25 11:24:54 -03:00
Adalyat Nazirov 1e3a1e3c43 Historical data requests start & time fix (#4733)
* regression tests

* fix: apply the same time convertion to history request time as for data time

* ver2

* fixup

* unit tests

* do not need this conversion because RoundDownInTimeZone returns in proper TZ

* comment

* requested changes

* refactoring

* more refactoring

* fix existing test: should return Sunday if open

* more symbols

* fix existing tests: submit new btcusd data

* fix

* add Cfd symbol
2020-09-24 14:51:35 -03:00
Adalyat Nazirov ea9007601d fix date parsing: TwelveCharacter dateformat can't be parsed properly;
more String Extensions: extract substring safely
add more US formats
2020-07-31 17:53:55 +03:00
Stefano Raggi 4d3923e47c Add support for Late Opens in market hours database 2019-09-27 20:43:40 +02:00
Michael Handschuh 86157cd5e7 Manually create US equity security exchange hours
Travis was having issues loading the market-hours-database.json and ended
up constantly using the always open exchange hours.
2018-08-07 15:02:44 -04:00
Michael Handschuh 4fd16f6daf Fix resolution of insight close times, allow user defined close times
Fixes a bug where we were using the security's data resolution to compute
the insight's close time. This led a case such as insight.Period == 20days
to step 20days worth of tradable minutes (assuming minute data resolution),
yielding a close time that was very far in the future.

We also add different means of specifying an insight's period/close time:
1. Specify insight period as a TimeSpan and we compute close time
2. Specify insight period and a resolution and bar count and we compute close time
3. Specify insight close time local directly and we compute the insight period

The key here is maintaining consistency between the three different approaches
which is heavily validated with the corresponding unit tests.

Edits also made to trust the insight's close time as the analysis end time in
the case where the analysis period == insight period (extra analysis period = 0).
Given the current setup (extra analysis period == 0), this guarantees that close
and analysis end times are equivalent.

Regression statistics were updated and expectedly we get many more insights that
have completed analysis, and as such, average scores have also changed.
2018-08-07 11:21:11 -04:00
Michael Handschuh d52ea82573 Adds TimeSpan.Multiply extension method
The alternative is fairly verbose and often obfuscates the intent.
2018-04-03 12:31:01 -04:00
Michael Handschuh 2450f40d37 Improve date time from unix stamp precision
AddSeconds and AddMilliseconds methods are only accurate to the millisecond.
This fix uses the AddTicks methods to get the highest possible precision
2018-03-10 16:38:12 -05:00
Michael Handschuh 5e491744b2 Adds Time.EachTradeableDayInTimeZone 2016-02-12 13:42:55 -05:00
snugs 284b83bfde Remove empty half test
Accidentally commited previously
2015-09-20 22:31:20 -04:00
snugs f30259a08c Properly set algorithm.UtcTime for usage in Initialize 2015-09-20 21:21:50 -04:00
snugs 65f75708bb Initial implementation of IHistoryProvider 2015-09-15 15:40:09 -04:00