-
[BE] fix: discount audio output tokens in cost for OpenAI audio models (#7253)
frostbyte_neo 发布于
2026-07-06 08:55:06 +00:00 - [BE] fix: discount audio output tokens in cost for OpenAI audio models
26 OpenAI models in model_prices_and_context_window.json publish
output_cost_per_audio_token at a rate distinct from output_cost_per_token
(gpt-4o-audio-preview is 8x: $8e-5 vs $1e-5 per token). The Java backend had
no reference to the field, so audio completion tokens were silently billed at
the standard output rate.Wires output_cost_per_audio_token through ModelCostData -> ModelPrice ->
CostService, and extends SpanCostCalculator.textGenerationCost to read the
audio output count from the OTel key logged by the Python SDK
(original_usage.completion_tokens_details.audio_tokens with a bare-key
fallback). When the configured audio rate is zero behaviour matches the
previous path; otherwise (completion_tokens - audio_tokens) is billed at the
standard output rate and audio tokens at the configured audio rate.This stacks on the input-audio fix in #7137 and mirrors its plumbing
symmetrically. Mutation-tested: stashing the SpanCostCalculator change makes
the new test fail with the previously buggy 0.0055 instead of 0.0195.- [BE] fix: split audio tokens in the OpenAI cache calculator too
Addressing the Baz review on #7253: textGenerationWithCacheCostOpenAI was
billing all completion_tokens at outputPrice and all (non-cached)
prompt_tokens at inputPrice, so OpenAI realtime models that publish BOTH
cache_read_input_token_cost and input_cost_per_audio_token /
output_cost_per_audio_token were silently underbilling audio tokens at the
standard text rate.Affected real models loaded by Opik today (litellm_provider: openai, cache
rates configured):- gpt-4o-realtime-preview / -2024-12-17 / -2025-06-03 (out audio 8e-5 vs
text 2e-5; 4x underbill) - gpt-4o-mini-realtime-preview / -2024-12-17 (out audio 2e-5 vs text 2.4e-6
/ 2.64e-6; 8x underbill) - gpt-realtime, gpt-realtime-1.5, gpt-realtime-2, -2025-08-28,
gpt-realtime-mini-2025-10-06 / -2025-12-15 (out audio 6.4e-5 vs text
1.6e-5; 4x underbill)
Mirror the same audio-token split that textGenerationCost already does for
non-cache OpenAI audio models, using the same usage keys
(original_usage.prompt_tokens_details.audio_tokens with the bare OTel key
as a fallback, symmetric for completion_tokens_details). When the model has
no audio rates configured, audio token keys in usage are ignored — the
existing non-realtime behavior is preserved.Added two parameterized unit-test cases in SpanCostCalculatorTest covering
both branches of the gate.- [BE] fix: prefer prompt_tokens_details.text_tokens in OpenAI cache calc
Two follow-up review notes on #7253:
-
baz-reviewer flagged a theoretical double-charge on OpenAI Realtime models
where cached_tokens and audio_tokens buckets may overlap: cached-audio
tokens would be subtracted twice from the text bucket and then billed at
the audio rate instead of the cheaper cache-read rate. In practice this
only matters if the API reports an overlap, which the OpenAI Realtime
usage payload does via prompt_tokens_details.text_tokens (LiteLLM's
_calculate_input_cost consumes this field directly, so no subtraction is
needed).Prefer prompt_tokens_details.text_tokens (with the original_usage.*
prefix taking precedence) when the payload carries it, falling back to
the existing cached/audio subtraction path when it doesn't. Same
treatment on the completion side via completion_tokens_details.text_tokens.
Behavior is unchanged for every fixture that doesn't include the new
key, so existing OpenAI cache-cost tests remain green. -
baz-reviewer flagged that the parameterized test only covered the bare
audio-token keys, not the SDK 1.6.0+ original_usage.prompt_tokens_details.*
/ completion_tokens_details.* variants. Added a parameterized case that
uses the prefixed keys throughout so both branches of the audio-token
extraction are exercised in regressions.Also added a parameterized case exercising the new text_tokens preference
path — with a 20-token cached-audio overlap reflected, the expected cost
drops from 37.00 (subtraction path) to 36.80 (text_tokens path),
demonstrating the fix.
Co-authored-by: Andres Cruz andresc@comet.com
下载附件