- Reduce the amount of `Path.Combine()` usages -> it has a peformance
overhead
- Improving `FineFundamentalSubscriptionFactory` GetSource algorithm,
now it will not check if each file exists while finding the appropriate,
since we already iterated the directory before
- `DefaultDataProvider` will not check if file exists since `new
FileStream` performance the same operation internally
Improve logging across all TradingEconomics Downloaders
Add retries to HTTP requests due to occassional failure
Added skipping of duplicates, skipping of
fields with no actual value, temp file writing, temp file moving
Write to proper output directory
Update data directory structure
Update GetSource in BaseData implementation for TradingEconomics
Parallelize data processing and writing to disk
- Release: Financial releases for the specified company
- Estimate: Financial estimates for the specified company
- Consensus: Consensus of the specified release
For each equity endpoint, we create a `BaseData`, a Downloader and add unit tests.
- Changes the destination folder
- Remove Symbol converter
- Refactors Get method to return `Task<string>`
- Since `HttpRequester` returns `Task<string>`, it makes more sense to pass this result than returning a `IEnumerable<string>`.
Implements an interface for all Trading Economics Data
- Adds Trading Economics Calendar Downloader and program to be used in the Toolbox.
- Adds Trading Economics Indicators Downloader and program to be used in the Toolbox.
- Adds Trading Economics Earnings Downloader and program to be used in the Toolbox.
- Adding new `StepTimeProvider` that will advance time based on a
desired custom evaluator.
- Live trading `Options`, `Futures` and `Coarse` data will use a
`FrontierAwareEnumerator` + `StepTimeProvider`. This will allow to hold
the selection data until its the desired time
- `Universe.CanRemoveMember` will round the members time in the universe
based on the `UniverseSettings.MinimumTimeInUniverse`
Delete converter because an existing utility already existed
Moved code from factory to converter class as statics
Add rankandfile as a CIK to ticker mapping source
- Moving `MapFile` and `FactorFile` resolution to the data feed stack so
that they do not add a performance overhead to the algorithm thread.
- Create logging string messaged only if required.
- Calculate `FactorFileRow.PriceScaleFactor` the least amount of times
- `SecurityIdentifier` will calculate `StrikePrice` just once. Replacing
`!=` for a direct call to `Equals()` preventing unnecessary checks.
- Slightly improving Linq queries at `OptionFilterUniverse`
- Replace `OpenReader().CopyTo` for `Extract(stream)` avoids copying the
data twice.
- Replace `DateTime.Now` for `DateTime.UtcNow`
- Adding `LazyToUpper()` implementation, that will avoid the call to
`ToUpper` if the string is already upper.
- Reduce the timezone conversions at `Time.EachTradeableDayInTimeZone`
- `TotalPortfolioValue` will iterate over all securities once
- Adding new `SecurityIdentifier` cache, significant impact for
algorithms using coarse/fine data
New custom data class USEnergyInformation with new demonstration algorithms, the updated config file for users to set their EIA token.
Adds `CloseTime` to represent the time that the data period end. `EndTime` represents, in turn, the time the data is emitted. There is an offset between `CloseTime` and `EndTime` that is defined by the difference between the last bar as emitted and its time.
In live mode, if the `USEnergyInformation.Reader` returns null, the `CollectionSubscriptionDataSourceReader.Read` method will pull for new data constantly. Therefore it should return an empty `BaseDataCollection` object.
- `FactorFile` will keep an ordered reversed list with the dates.
Calling `Reverse()` on the `SortedList` is expensive.
- `MapFiles` will keep first and last date, so we don't need to call
`First()` and `Last()` multiple times.
- `Liquidate` will go through all the algorithms securities only if
necessary
- `TradeBar` parsing will not call `new T` for pure `TradeBar` which is
expensive
- Removing `Lazy` hash code and security type for the
`SecurityIdentifier`, replacing for direct initialization. Accessing the
`Lazy` value adds an overhead.
- Replacing `Enum` to string for hardcoded switch statement. `Enum.ToString` is expensive.
- `DataManager` will be lazy for counting the subscriptions for
determining if its above the limit
- Adding `AlgorithmSecurityValuesProvider.GetAllValues()`, removes the
need to fetch all the security keys twice.
- During universe selection, will not try to re add already added symbol
- When subscribing to `Coarse` data the `LiveTradingDataFeed` will use
the normalized `CoarseFundamental Universe Symbol` for that market ->
not using the random GUID
- Adding unit tests which reproduce issue.
Implements `DataNormalizationMode` field in `UniverseSettings` to enable the settings of a desired `DataNormalizationMode` to the securities that are chosen in Universe Selection.
- Using `Aggregate(lambda)` vs `Sum(lambda)` since the later is slower
due to performing an extra `Select`
- For `QCAlgorithm.Framework.OnFrameworkData()` will avoid calling
`ToArray()` on empty `Enumerables` due to its cost * the number of
calls. If the `Enumerable` is the empty instance, which is static,
will create a new empty array and return it instead.
- Replacing `SecurityIdentifier` `SecurityType` and `GetHashCode`
implementations for `Lazy` versions, that are performed just once, since
these values do not change and are used multiple times.
- For the different `DataDictionary<T>` implementations adding `this[
Symbol] get; set` since existing overload `this [string]` produces an
extra round operations `Symbol->string->Symbol` with a significant
impact.
- Adding `PortfolioTargetCollection.AddRange()` overload using an array
to avoid unnecessary convertions.
Adds constructor overload that accepts a PyObject representing a `Func<DateTime, Symbol[]>` object to `ScheduledUniverse`. With this overload, we are able to create a `ScheduledUniverse` with a selector defined in a python script.
Replaces Enum `CalendarType` for static class with the same name. This class defines two properties (`Weekly` and `Monthly`) that can be used to define the previous calendar date (Monday or 1st of current month) which will correspont to the `Time` of a `IBaseData` object.
Refactor `PeriodCountConsolidatorBase` to define use `GetRoundedBarTime` based on a period specification that depends on the constructor overload: `integer`, `TimeSpan` or `Func<DateTime, CalendarInfo>`. The last one can be set with the `CalendarType` properties.
- Implementing `QCAlgorithm.AddUniverseSelectionModel`
- Adding C#/Py regression algorithms using the new API
- Modifying `ManualUniverSelectionModels` symbol, adding hash of
the selected `Symbol.Values`
- Modifying `Coarse` and `Fine` symbol, adding random GUID
- Adding `NullUniverseSelectionModel`
Fixed bug where factor file reference price output had very high
precision
Fixed bug where map file had a useless entry before the final line if
the asset was not delisted
Fixed bug where program would crash because of NextPrice reaching its
maximum attempts. Fixed by removing call to NextPrice and using another
generation method.
Martin Molinero