-
[OPIK-7274] [BE] fix: make Ollie report stale timeout configurable and add pipeline metrics (#7576)
frostbyte_neo 发布于
2026-07-24 11:40:30 +00:00 - [OPIK-7274] [BE] fix: extend Ollie report stale timeout to 30m and add pipeline metrics
Raise the stale-report sweep threshold from 10 to 30 minutes: reports
queued behind orchestrator pod provisioning routinely take longer than
10 minutes, so the old threshold marked still-in-flight reports as
failed.Also instrument the daily-report flow with OTel metrics so failures are
observable from opik-backend instead of only through logs:- triggered / trigger_error (OllieDailyReportJob), counting only reports
actually created (skips no longer inflate the triggered count) - finished{result=completed|failed|trigger_failed} + end_to_end and
scheduled_to_completion duration histograms (ReportService); the
async trigger-failure callback now records trigger_failed - stale_swept (StaleReportCleanupJob) as the safety-net signal
Completion metrics are recorded after the write transaction commits and
never fail the callback.- fix(reports): dimension daily-report metrics and count manual triggers
Address PR review on the daily-report metrics (metrics-instrumentation
contract):- Dimension triggered, trigger_error, finished, both duration
histograms, and stale_swept by workspace_id + workspace_name (paired,
name->id fallback); trigger_error also carries error_type. - Emit the triggered counter from createAndTriggerReport so manual
/generate triggers are counted alongside scheduled ones, keeping the
funnel consistent with finished/stale_swept (which already count both). - Per-workspace stale_swept: failStaleReports returns per-workspace
counts; the 30-minute cutoff is a single STALE_THRESHOLD_MINUTES
constant bound into both stale queries. - Hoist result/workspace AttributeKeys to constants; add
workspaceId/projectId to markReportFailed logs; drop the duplicate
job-side stale-sweep INFO log.
- fix(reports): configurable stale timeout, drop scheduled_to_completion
Address review (thiagohora):
- Make the stale-report sweep timeout configurable via
reportGeneration.staleReportTimeoutMinutes (default 10 = unchanged
behavior; tunable per-env without redeploy). - Drop the scheduled_to_completion_duration histogram. It only added the
bounded cron-pickup delay (<=10m) over end_to_end, required a blocking
preference read + schedule parse on the completion callback, and was
polluted by manual triggers on daily-enabled projects. Removing it makes
recordCompletionMetrics a cheap non-blocking side-effect (finished +
end_to_end only), so updateReport uses doOnNext again and there's no
callback DB read / parse-failure path.
下载附件