-
[BE] fix: apply above_128k and above_272k tier pricing in cost calculation (#7347)
frostbyte_neo 发布于
2026-07-08 14:40:35 +00:00 Follow-up to #7023, which introduced the whole-prompt above_200k_tokens
tier for Gemini 2.5 Pro / Claude Sonnet 4.5. LiteLLM's model prices JSON
also publishes above_128k_tokens rates (Gemini 1.5 Flash — 2x at 128k) and
above_272k_tokens rates (OpenAI GPT-5.4/5.5 families and their Azure-hosted
twins registered by #7262 — 2x input, 1.5x output at 272k). Opik silently
ignored both tiers, undercharging every above-threshold request.Add plumbing symmetric to the 200k tier: parse each *_above_NNNk_tokens
field into ModelPrice, and have effectiveInputPrice / effectiveOutputPrice
walk the applicable thresholds descending (272k -> 200k -> 128k -> base)
so the highest matching tier wins. Cache-side tier rates for 128k / 272k
are not published by LiteLLM at those thresholds, so cache handling stays
200k-only.Two additional call sites also had to switch from inputPrice() / outputPrice()
to the effective helpers: textGenerationCost and textGenerationWithCacheCostOpenAI
both bypassed the tier logic before, which is why the 272k GPT-5.4/5.5
rates never took effect even though ModelPrice knew about them. Cache-side
tier for OpenAI now also routes through effectiveCacheReadInputTokenPrice.Reachable models today: gemini/gemini-1.5-flash at 128k; every OpenAI and
Azure gpt-5.4 / gpt-5.5 (and *-pro) variant at 272k — 17 flagship models
whose above-threshold bill just got closer to reality.Parameterized tests added mirroring the existing 200k Gemini test: at
threshold uses base, above threshold uses tier, for both new thresholds.Co-authored-by: Andres Cruz andresc@comet.com
下载附件