Michael Handschuh
09cc78599a
We restrict each algorithm time loop to a pre-determined amount of time. Exceeding this limit will cause the algorithm to immediately terminate. This quickly becomes an issue when considering users running trainable models that have a long initialization period that exceeds the time loop maximum. This change provides a mechanism through which a long-running scheduled event is permitted to keep running and is permitted to avoid the time loop permitted by requesting additional time. Requests for additional time are limited according to a leaky bucket implementation whose parameters are set via the job's controls structure. The fundamental time unit for the algorithm is a single minute. Here's how it works. If a scheduled event takes longer than one full wall clock second then a request is made to the leaky bucket for one more minute. If the scheduled event continues to take more time, it will continue to request additional minutes. Each requested minute will prevent the algorithm's time loop check from terminating the algorithm. When the bucket is empty and no more minutes are available to be requested, a TimeoutException is thrown causing a cascade that ends in the algorithm's termination and status being flipped to RuntimeError. Additionally, this applies equally to ALL scheduled events. While some helpers were added with the naming of Train and TrainNow to the ScheduleManager, these methods don't do anything special and the infrastructure doesn't otherwise flag them as different, so this feature becomes part of the core Scheduled Event feature set. Further, the live scheduled events were not touched and are still pending further discussion regarding the value added by enforcing a time restriction when simulation time and wall clock time are equivalent. Fixes #3319
91 lines
4.7 KiB
C#
91 lines
4.7 KiB
C#
/*
|
|
* QUANTCONNECT.COM - Democratizing Finance, Empowering Individuals.
|
|
* Lean Algorithmic Trading Engine v2.0. Copyright 2014 QuantConnect Corporation.
|
|
*
|
|
* Licensed under the Apache License, Version 2.0 (the "License");
|
|
* you may not use this file except in compliance with the License.
|
|
* You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
|
|
*
|
|
* Unless required by applicable law or agreed to in writing, software
|
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
* See the License for the specific language governing permissions and
|
|
* limitations under the License.
|
|
*
|
|
*/
|
|
|
|
using System;
|
|
using Newtonsoft.Json;
|
|
using QuantConnect.Configuration;
|
|
|
|
namespace QuantConnect.Packets
|
|
{
|
|
/// <summary>
|
|
/// Provides parameters that control the behavior of a leaky bucket rate limiting algorithm. The
|
|
/// parameter names below are phrased in the positive, such that the bucket is filled up over time
|
|
/// vs leaking out over time.
|
|
/// </summary>
|
|
public class LeakyBucketControlParameters
|
|
{
|
|
// defaults represent 2 hour max capacity refilling at one seventh the capacity (~17.2 => 18) each day.
|
|
// rounded up to 18 to prevent a very small decrease in refilling, IOW, if it's defaulting to 17, then
|
|
// after 7 days have passed, we'll end up being at 119 and not completely refilled, but at 18, on the 6th
|
|
// day we'll reach 108 and on the seventh day it will top off at 120 since it's not permitted to exceed the max
|
|
public static int DefaultCapacity => Config.GetInt("scheduled-event-leaky-bucket-capacity", 2 * 60);
|
|
public static int DefaultTimeInterval => Config.GetInt("scheduled-event-leaky-bucket-time-interval-minutes", 1440);
|
|
public static int DefaultRefillAmount => Config.GetInt("scheduled-event-leaky-bucket-refill-amount", (int)Math.Ceiling(DefaultCapacity/7.0));
|
|
|
|
/// <summary>
|
|
/// Sets the total capacity of the bucket in a leaky bucket algorithm. This is the maximum
|
|
/// number of 'units' the bucket can hold and also defines the maximum burst rate, assuming
|
|
/// instantaneous usage of 'units'. In reality, the usage of 'units' takes times, and so it
|
|
/// is possible for the bucket to incrementally refill while consuming from the bucket.
|
|
/// </summary>
|
|
[JsonProperty("iCapacity")]
|
|
public int Capacity;
|
|
|
|
/// <summary>
|
|
/// Sets the refill amount of the bucket. This defines the quantity of 'units' that become available
|
|
/// to a consuming entity after the time interval has elapsed. For example, if the refill amount is
|
|
/// equal to one, then each time interval one new 'unit' will be made available for a consumer that is
|
|
/// throttled by the leaky bucket.
|
|
/// </summary>
|
|
[JsonProperty("iRefillAmount")]
|
|
public int RefillAmount;
|
|
|
|
/// <summary>
|
|
/// Sets the time interval for the refill amount of the bucket, in minutes. After this amount of wall-clock
|
|
/// time has passed, the bucket will refill the refill amount, thereby making more 'units' available
|
|
/// for a consumer. For example, if the refill amount equals 10 and the time interval is 30 minutes, then
|
|
/// every 30 minutes, 10 more 'units' become available for a consumer. The available 'units' will
|
|
/// continue to increase until the bucket capacity is reached.
|
|
/// </summary>
|
|
[JsonProperty("iTimeIntervalMinutes")]
|
|
public int TimeIntervalMinutes;
|
|
|
|
/// <summary>
|
|
/// Initializes a new instance of the <see cref="LeakyBucketControlParameters"/> using default values
|
|
/// </summary>
|
|
public LeakyBucketControlParameters()
|
|
{
|
|
Capacity = DefaultCapacity;
|
|
RefillAmount = DefaultRefillAmount;
|
|
TimeIntervalMinutes = DefaultTimeInterval;
|
|
}
|
|
|
|
/// <summary>
|
|
/// Initializes a new instance of the <see cref="LeakyBucketControlParameters"/> with the specified value
|
|
/// </summary>
|
|
/// <param name="capacity">The total capacity of the bucket in minutes</param>
|
|
/// <param name="refillAmount">The number of additional minutes to add to the bucket
|
|
/// after <paramref name="timeIntervalMinutes"/> has elapsed</param>
|
|
/// <param name="timeIntervalMinutes">The interval, in minutes, that must pass before the <paramref name="refillAmount"/>
|
|
/// is added back to the bucket for reuse</param>
|
|
public LeakyBucketControlParameters(int capacity, int refillAmount, int timeIntervalMinutes)
|
|
{
|
|
Capacity = capacity;
|
|
RefillAmount = refillAmount;
|
|
TimeIntervalMinutes = timeIntervalMinutes;
|
|
}
|
|
}
|
|
} |