Add stale gauge / idle queue handling via last-known-value carry-forward

When a queue is idle (no operations), no rows are produced in the 5s
table but the queue may still have items. The presenter handles this
by querying the most recent row before the window start and carrying
forward gauge values into empty buckets. Counters are zero-filled.

https://claude.ai/code/session_01PkVYeZ8tQzgcT4YQsctGo1
This commit is contained in:
Claude
2026-02-07 09:14:45 +00:00
parent a9d741ab7b
commit 7c4019523c
+40 -1
View File
@@ -323,7 +323,46 @@ FROM trigger_dev.queue_metrics_by_minute_v1
GROUP BY organization_id, project_id, environment_id, queue_name, bucket_start;
```
### 3.5 Query routing by time range (ABR-inspired)
### 3.5 Handling idle queues (the "stale gauge" problem)
Since we only emit metrics on queue operations, an idle queue with 500 items sitting in it produces **zero rows** in any 5s window where no enqueue/dequeue/ack occurs. But the queue isn't empty — the user's dashboard should still show depth = 500.
This only affects **gauge metrics** (queue_length, concurrency_current, oldest_message_age_ms). Counter metrics are fine — zero rows correctly means zero activity.
**Solution: "last known value" carry-forward at query time**
When the presenter queries a time window, it also fetches the most recent row *before* the window start for each queue to seed the initial gauge values:
```sql
-- Get the last known gauge values before the requested window
SELECT queue_name,
max_queue_length,
max_concurrency_current,
max_oldest_message_age_ms
FROM queue_metrics_5s_v1
WHERE environment_id = {envId}
AND queue_name = {queueName}
AND bucket_start < {windowStart}
ORDER BY bucket_start DESC
LIMIT 1
```
The presenter then fills gaps in the timeseries:
- For any 5s bucket with no row, carry forward the gauge values from the most recent preceding bucket (or the seed query above)
- Counter metrics are zero-filled (no row = no activity, which is correct)
This is the standard approach for gauge metrics in time-series systems (Prometheus/Grafana use identical "last value" semantics for `gauge` types). The worst-case staleness is bounded by the 5s resolution.
**Why not periodic heartbeats?**
An alternative is emitting a "heartbeat" snapshot for all non-empty queues every 5s from Node.js, guaranteeing every active queue has at least one row per window. This would work but:
- Adds Redis polling overhead (ZCARD per queue per 5s) that scales with total queues, not active queues — exactly the scaling property we want to avoid
- Requires maintaining a "known queues" registry
- Carry-forward at query time achieves the same UX with zero additional infrastructure
Heartbeats could be added later if carry-forward proves insufficient (e.g., if alert evaluation needs gap-free data). But alert evaluation can also use the same seed query pattern.
### 3.6 Query routing by time range (ABR-inspired)
The query routing here borrows the core idea from [Cloudflare's ABR (Adaptive Bit Rate) analytics](https://blog.cloudflare.com/explaining-cloudflares-abr-analytics/): automatically select the best resolution table for each query based on the requested time range, so dashboards stay fast regardless of how far back the user looks.