Report Generator Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
API Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* Make FOPs selection universe file-based for backtesting
* Make FOPs option chains universe file based
* Make Future universe selection file-based like option universe
* Make Future universe selection file-based like option universe
* Abstraction cleanup
* Add FuturesChains API to QC algorithm
Also refactor future chain provider to use the new FutureUniverse instead of zip file names
* Update regression algorithms stats
* Refactor QuantBook option and future history to use new universes
* Fix failing tests
* Fix failing tests
* Fix failing tests
* Minor future chains unit test improvement
* Add futures chains DataFrame property
Also, remove IDerivativeSecurity interface from Future
* Add DataFrame property to FuturesChains class
* Add regression algorithms
* Add regression algorithms
* Replace QCAlgorithm.FutureChainProvider usages with new FuturesChain api
* Minor fixes
* Reduce number of universe files in repo
* Minor data fixes
* Regression algorithms updates
* Add implicit conversion from FuturesContract to Symbol
Modified algorithms to use futures contract objects directly instead of accessing their Symbol property.
Removed unnecessary import statements and redundant lines in various files.
* Improve resolution handling for history requests
* Changed _auxiliaryData field to lazily-initialized AuxiliaryData property
* Refactor data handling in BaseChain and TimeSliceFactory
- Added `AddData` method to `BaseChain` for adding market data
- Refactored `TimeSliceFactory` to use `BaseChain.AddData` method
* Remove specific constructors and indexers from Chain classes
Removed public indexers in `BaseChains` for getting or setting `BaseChain` instances by `ticker` or `Symbol`, which were used for Pythonnet compatibility.
* Remove chain cache logic from FuturesChainUniverse
* Refactor class and interface names for clarity
Renamed `FileBasedUniverse` to `BaseChainUniverseData` and
`IFileBasedUniverse` to `IChainUniverseData`.
* Add base class for options and futures contracts
- Introduced `BaseContract` as an abstract base class for contracts, consolidating common properties and methods.
- Removed ISymbolInterface
* Add minor fix for future options tickers parsing
Added tests
* Trigger Build
* Trigger Build
* Trigger Build
* Trigger Build
* Clean chain provider classes up
* Remove ZipEntryName other classes and unused code
Removed ZipEntryName class and references across various files.
Removed DataQueueFuturesChainUniverseDataCollectionEnumerator and DataQueueOptionChainUniverseDataCollectionEnumerator classes.
Removed OptionChainUniverseSubscriptionEnumeratorFactory class.
Removed unused code for handling OptionChainUniverse and FuturesChainUniverse in FileSystemDataFeed.cs and LiveTradingDataFeed.cs.
Removed several test files related to enumerator factories and universe data collection.
* Minor changes and cleanup
* Trigger Build
* Trigger Build
* Refactor FuturesContract data handling
Forward price data from bars and ticks stored in private fields for improved memory usage
* Fix: use universe data for market data in FuturesContract
* Update regression algorithms stats after rebase
Added HSI futures universe files
* Sort configs by internal flag
Internals go first
* Throw from option universe data filters for future options
Future options IV, Open interest and greeks are not supported for future options
* Minor changes
* Improve some regression algorithms
* Minor fix for failing unit tests
* Update FOPs universe file header
Removed greeks and IV columns.
Updated FOPs universe files: removed outdated columns.
* Minor unit test fix
* Trigger Build
* Trigger Build
* Trigger Build
* Trigger Build
* Trigger Build
* Trigger Build
* Trigger Build
* Trigger Build
* Trigger Build
* Trigger Build
* Trigger Build
* Trigger Build
* Trigger Build
* Minor fix
* Add history provider as constructor argument for chain providers
* Update new regression algorithms data points count
* Minor fix for FakeDataQueue
* Add initialize method to chain providers classes
* Minor changes
* Trigger Build
* Trigger Build
* Trigger Build
* Minor fix
* Minor fix
* Trigger Build
* Trigger Build
* Trigger Build
* Trigger Build
* Add logs to ProcessedDataProvider
* Removed test logs
* Minor fix
* Support downloading options and futures universe files from api data provider
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Report Generator Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
API Tests / build (push) Has been cancelled
* Fix half of the CA1051 warnings
This warning is about not declaring visible instance fields. There are
something about 500 warnings in the solution, mostly in the QuantConnect and QuantConnect.Algorithm.CSharp projects. I aim to fix one of them in this PR and the other half of them in a second one. To fix it, I'm changing the visible instancce fields for properties.
* fix bugs
* Addressing minor reviews
* More minor fixes
---------
Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
* Daily data Time & EndTime Improvement
- Adjust daily data Time & EndTime to actually reflect the time of the
data used, for example US Equity from 9.30 to 4PM. Adding new unit and
regression tests
* Refactor solution to use enumerator
- Refactor daily strict end times solution to be through enumerator
usage, so it applies for history providers too
* Minor fixes
* Revert fill forward enumerator change
- Revert FillForward enumerator causing stats changing, enhancing unit
tests
* Some cleanup
* Improve handling of live trading FF enumerator
- Improve handling of live trading FF enumerator, by adding support for
bars to arrive with a delay so we can handle auction close/option
prices or data providers which might have some delay making the data
available. Adding new unit tests asserting the behavior
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Report Generator Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
* refactor: map files in lean/data folder
* feat: Indexers in MapFile
* feat: extension for mapFile and downloader param
* refactor: getAllTickers from MapFiles
* feat: addition out variable in TryParsePath
fix: tests with new variable
* refactor: DownloadDataProvider with getting all ticker with different DateTime Ranges
feat: test
* revert: "refactor: map files in lean/data folder"
This reverts commit 41fa26eb8627446028d4f2d4c536511ea82a9a48.
* revert: "feat: addition out variable in TryParsePath"
refactor: some actual changing
This reverts commit 9de482bd6490eecd1eeee83e035dfc1ece8d6415.
* revert: "feat: Indexers in MapFile"
fix: actual changes
This reverts commit 393181c6316a354a8cdcbfb093707cb545383a8a.
* remove: high performance Any validation
feat: validation on ticker start with specific sign
fix: option test
* feat: validation on mapping
refactor: when resolution less hour
test: new test cases + validation of SecurityType
* refactor: add additional day when return date range from MapFile
fix: FirstTicker instead of Permtick when generate new Equity
feat: validation of request Date in DownloaderDataProvider
refactor: by style proj DownloaderDataProvider
test:fix: starting Date in TestCases
* remove: duplicate code
* fix: check of income data.Time with requested Start/End-Date
* test:fix: download test cuz we can not get/write future data
* remove: extra Symbol Create mthd
* test:feat: validate of TryParse returns correct symbol
* remove: excess if condition
* test:remove: extra chaning
connected # 3a62adf54c42b01d28ff410e93e1ce277ee6d534
* refactor: split big method to small ones (add readability)
* fix: typo in xml description
* refactor: get the correct symbol value
* refactor: use local time instead of Utc in download Param extension
refactor: use struct instead of class
* test:fix: convert time to Utc from local time
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Report Generator Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
* feat: helper to return old name of symbols
feat: test
refactor: test a little bit neat
* feat: SplitHistoryRequest() with different mapped symbol
feat: test
refactor: RetrieveSymbolHistoricalDefinitions
* refactor: split historyRequest by historical symbol name
refactor: change location of extension
refactor: change location of test extension
* revert: extra changes
* fix: validation on RequiresMapping in SplitHistoryRequest
feat: add Future Symbol tests
* move: extension method to another file
* fix: wrong extension call param position
* fix: wrong extension call param position in test
* refactor: IEnumerable foreach to more simple form
feat: add TickerDateRange struct instead of Tuple
fix: test
* refactor: map extension alway returns empty result
* refactor: SplitHistory extension returns original HistoryRequest
* feature: new extension for RequiresMapping except Futures
* fix: validation of symbol map file to contains something
* refactor: extension test cases
* feat: assert of right ticker in tests
* fix: split historical request with using Local Time instead of UTC
fix: test description
* feat: test helper IEnumerable of TimeZones
* test: validate right EndDate with different TimeZones
* remove: validation of end of date
test: several new test cases
* feat: additional test with specific dateTime
* remove: extra Extension method validation on future and mapping
- Adjusting live stack to emit live auxiliary events based on factor files.
Updating existing tests
- Minor improvements for map & factor file provider daily refresh time.
Adding unit tests
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
- Validate cache folder exists before using it
- Avoid duplication fetching factor & map files path
- Adding helper method to check if directory is empty. Adding unit tests
Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Apply splits and dividends to volatility models
* Apply splits and dividends to volatility models using history requests
* Add new ScaleRaw data normalization mode
Handling the new mode in the price scale enumerator.
* DataNormalizationMode.ScaledRaw history requests
* Minor changes
* Minor changes
* Disable new normalization mode in AddSecurity methods and other minor changes
* Peer review
* Minor changes
* Peer review
* Minor changes
* Peer review
* Peer review
* Peer review
* Add scaled raw history regression algorithm
* Add more regression algorithms
* Add more regression algorithms
* Add Slice.TryGet unit tests
* Peer review
* Peer review
* Peer review
* Peer review
* Peer review
* Update algorithms stats
* Peer review
* Peer review
* Fix default future chain universe selection
- Fix default future chain universe selection that should always return
empty, no selection. In some cases it would allos some contracts
though. Adding regression algorithm reproducing issue.
* Address review
Regression Tests / build (push) Has been cancelled
Python Virtual Environments / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
Benchmarks / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
* Minor debugging improvements
- Improve exception message if user provides empty/null ticker
- Log margin information on algorithn initialization
- Add unit test asserting option strategy position groups resolve
correctly for existing holdings
* Symbol Lazy SecurityDefinitionSymbolResolver
- Symbol will have a lazy SecurityDefinitionSymbolResolver instance
resolution
* Memory Related Performance improvements
- Make sure we cleanup & dipose of python related objects during pandas
data generation.
- Disable memoizing enumerable use while creating pandas data frames,
since we do not require it
- Reduce unrequired object creations
- Replace concurrentCollections for ordinary locks
* Decimal parsing typo fix
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
Research Regression Tests / build (push) Has been cancelled
- Adding OpenInterestAnnual where the contract maps when any of the back month
contracts of the next year have a higher volume that the current front month.
- Add support for backwards compatible data mapping mode additions
- Updating ES map and factor files. Adding a little daily data too
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Lean exchanges improvements
- Adding new Exchange class to avoid exchange code clash.
Adding and updating unit test
* Add Market for MapFile API
* Self review
* Adds support for getting a Lean symbol based on FIGI, CUSIP, ISIN, SEDOL
* Address review: bug fixes for mapping, improve docs, cleanup & refactor
* Make LocalZipMapFileProvider and LocalZipFactorFileProvider only
initialize if they haven't been initialized yet.
* Address review: makes ISIN, SEDOL, CUSIP case-insensitive
* Cleans up map file provider in SecurityDefinitionSymbolResolver
* Modifies some test cases to test for case-insensitivity
* Add null check for SecurityDefinitions
* Fix unit tests
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
Build & Test Lean / build (push) Has been cancelled
Regression Tests / build (push) Has been cancelled
* Few improvements
- Alpha Streams algorithm will add target securities right away
- If algorithm is warming up PortfolioTargetCollection will emit not
insight
- Fix local disk factor file provider to use local cache instead of data
folder
- EWAS PCM will not emit targets for securities which have not been
added by the algorithm yet, to avoid runtime exception
* Fix GetLastKnownPrice default order adding more unit tests
Regression Tests / build (push) Has been cancelled
Build & Test Lean / build (push) Has been cancelled
* Add basic template Atreyu algorithm
- Add C# and Python basic template atreyu algorithm. Show casing how to
specify exchange to execute in different ways.
- Adjust trading API to allow specifying order properties to use
* Lean Exchange improvements
- Rename PrimaryExchange to Exchange
- OrderPropeties will use Exchange enum instead of string
- Adding BSE exchange value
* Regression tests fixes
Build & Test Lean / build (push) Has been cancelled
* Use lean data key as param for request
* key -> filePath rename and some cleanup
* Refactor
* Add Organizations Endpoints
* Add some organization api wrapper objects
* Address namespace issue
* Reorganize Api Test into seperate files using one ApiTestBase
* Add Organization tests
* Use capitalized "API" test namespace to reduce amount of file changes
* Add License to test base
* Update /data endpoint functions and response objects
* Update ApiDataProvider Logic
* Handle deserialization of organization products
* Simplify converter
* Only throw for equity requests when not subscribed to map/factor files
* Add missing header
* Make arguement exception
* Api adjustments
* Add Zip factor and map file providers
- Common project will now reference Compression project and not the other way
round.
- Adding Zip FactorFile and MapFile providers
* Refactor FactorFileProvider to use DataProvider to fetch files
* Use resulting MinimumDate in construction of FactorFile
* Nit FactorFile comments and arrangement
* Refactor MapFileProviders to use DataProvider for fetching files
* Refactor ZipFileProvider
* Clean up
* Refactor Backtesting Future/Option chain providers to use dataprovider
* Fixes for data/ endpoints and test adjustments
* Response objects adjustments/cleanups
* ApiDateProvider fixes and testing
* Add LocalZipFactorFileTests
* Update ApiDataProvider download test to verify stream is not null
* Implement posting of agreement summary and signed time
* Mark all Api related tests as explicit and document details on running
* Clarify default token on ApiTestBase
* Adjust summary
* Update Api responses for QCC, except org products which are sold in USD
* Implement cache expiration for zip MapFile and FactorFiles. Adding unit tests
* Fix multiple markets for ZipFactorFile provider
* Use Symbol as cache key
* Api.cs review
* Dispose of factorFileStream after reading
* Use zip.EntryFileNames
* Address a few reviews
* Few more fixes
* Address Api Review
* Add Job Org id to config
* Minor tweaks
* Compare with invariant culture
* Fixes Option Universe selection
* ZipEntryNameSubscriptionDataSourceReader will use IDataProvider
* Fix research
* Fix null reference exception
* Make duplicate log debug
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Reconcile duplicated code
* Add License header
* CS0219 Fixes: Value assigned, but never used
* CA1507: Use nameof in place of string literals
* CS0108 : Hides Inherited Member; Use new keyword to overwrite formally
* CS0114: Hides inherited member; use override keyword
* CS0168: Variable is declared but never used
* Tests CS1062; using obsolete implicit Symbol -> String; fix via .ToString()
* CS0472: Non Nullable Obj getting Null Checked
* CS0067 Member not used; ignore all cases for future use
* CS00162 : Unreachable code; either removed or ignored for debugging and test cases
* CS0169 Remove non-used fields; ignore those that may be used in future
* CS0414; Field is assigned but never used.
* CS0618; Obsolete properties and members; Only fixes simple ones, rest will have to broken up
* CS0649; Field never assigned too
* CS0659 & CS0661 ; Overwrite operators and equals but not hashcode; I don't really override it but just call base
* Small comment fix
* Cleanup pragma statement
* Refactor IPrimaryExchangeProvider.GetPrimaryExchange
now it returns a PrimaryExchange instead of a string
* Update MapFilePrimaryExchangeProvider.cs
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Implements wrapper to get the exchange as a single letter representation
Also replace lower 'l' with capital 'L' in condition flags enums. it is a new warning on the compiler and it looks cool :)
* There are cases where the exchange is null or empty, in those cases return null
* Add tests for single character exchange representation
* Update GlobalTest.cs
Co-authored-by: Martin-Molinero <martin@quantconnect.com>
* Migrate TradeConditionFlags and QuoteConditionFlags to Lean
* Address revirew
Implement flag systems using long, this allows us to declare up to 64 flags
* Add main exchange to map file
minimum minimorum test
* Add MainExchange to MapFilRow IEquatable implementation
Also write CSV line correctly
* Normalize exchanges
Also add small test form MapFileRow
* Exchange name normalization
- Exchange name normalization.
- Tick will protobuf `ExchangeCode` which will set existing `Exchange`
* Include single character exchange code
* Encode exchange as byte, Include exchange name as Enum
* Update MapFileRow.cs to accept a byte instead a string for primary exchange
* Fix issues in encoding exchanges map
* Fix broken MapFile tests
* Rename Main Exchange property in MapFileRow
* Use BATS instead of BATSZ
* Add primary exchange from map file
* Move MapFilePrimaryExchangeProvider to its own file
* Implement Char for primary exchange in map files
* Fix broken tests after implementing char exchange into map file
* Add a new constructor for MapFileRow
it accepts a PrimaryExxchange object as parameter
* Fix broken test
WIP
* Address review
WIP
* Handles unexpected encoded exchanges
* Keep addressing review
* Address review
* Fix broken tests
Make PrimaryExchange as default even if the IPrimaryExchangeProvider is not instantiated
* Revert breaking change.
* Add a check that resolved map file actually have content
The issue that leads to this change was caused by a map file resolved to APC in the CoarseFundamentalTop3Algorithm.
The resolver returns a not-null MapFile, but it is empty.
To my best understand, the MapFile Resolver should return only existing map files in the map_files folder. But the LocalMapFileProvider returned a map file with a permtick that doesn't exist in the file system as CSV.
* Set PrimaryExchange.UNKNOWN as default
* Address review couple minor bugs
- Use SID market instead of USA
- Default ticke exchange is UNKNOWN
* Update map files with latest format and latest info
This include:
- VXX, it was delisted, then VXXB appears and later it chnages from VXXB -> VXX
- AAA delisting and a new listing with ticker AAA
- FOXA ticker changue to TFCFA and later it was delisted
* Update test after updating latest map files
* Fix bug with factor files after updating map files
* Fix bug with factor files after updating map files
and revert statistics in regression tests
* Update OrderHash in regression test
Co-authored-by: Martin Molinero <martin.molinero1@gmail.com>
* DividedEventProvider distribution computation
- Update regression algorithm which was using a different reference
price when calculating the dividend
- Adjust divided event provider to compute distribution using factor
file reference price, if not 0. Adding unit tests
- For equities, only emit auxiliary data points for
TradeBar configurations, not for QuoteBars, nor internal.
* Address reviews
- Split and Dividend event provider will throw an exception when there
is no reference price available. Updating `wm` factor file which was
missing references price and regression algorithms using WM.
- Updating unit tests asserting new exception
* Add decimal places as parameters to get dividends with arbitrary precision
Also, increase precision when generating strings from factor files rows.
* Add xml documentation entries for new optional arguments.
in the previous solution, I basically tried to pass the first row as a split of 1.
That's is not a correct solution, the correct one is treat the first row as a different _kind_ of row. In fact, the the main usage of the first row is a date reference for the factors defined in the second row
Previously, if there is an `inf` or exponential notation in a factor file row, the FactorFile.Row method will add a new line with a date equal to the original row date less one day. This is a bug because the dates in factor files row should always be a trading day, the previous logic didn't assure that.
Even more, that new line is useless, without that line addition, the parser still parses the latest row with valid factors and use it as the first row.
This commit removes the addition of a new line. Also, updates the test cases, now the FactorFile minimum date will be the latest row without inf or exponential notation less one day.
- EventProvider will receive start date during initialization, this will
be used by the `MappingEventProvider` to correctly set current mapped
symbol
- Adding unit test, updating existing regression test
- Add Pandas backwards compatibility shim
- Adding `MappingExtensions` which will remove data type from the
`Symbol.ID.Symbol` value to resolve the `MapFile`
- `SecurityIdentifier.TryParse()` will throw when given an invalid
`SecurityType`
The `NullReferenceException` type is intended to only be thrown by the CLR.
In most cases, it should be converted to an `ArgumentException` or an
`InvalidOperationException`, depending on if the null value is a parameter
to the current method or not.
The `Exception` type should never really be thrown as it doesn't provide any
additional information or hints as to the issue. It also forces users that
would like to handle expected exceptions to catch all exceptions. These are
converted to an exception type that more accurately describes the reason for
raising the exception: `KeyNotFoundException`, `InvalidOperationException`
- Moving `MapFile` and `FactorFile` resolution to the data feed stack so
that they do not add a performance overhead to the algorithm thread.
- Create logging string messaged only if required.
- Calculate `FactorFileRow.PriceScaleFactor` the least amount of times
Alexandre Catarino