github-featured/hmbown--codewhale
2
0
Fork 0
Code Issues Pull Requests Actions 16 Deployments Models Agent Notes Packages Projects Releases Wiki Activity

feat(tui): telemetry feature and error counters

Browse Source 
Eleven counters and six error counters, every one bumped at a call site and
every one a count of a variant discriminant rather than of a rendered string.

The siting is the substance:

  - `turns` is counted at the *caller* of `execute_turn_end_observer_hook`,
    not inside it. That function's first statement returns early for anyone
    with no TurnEnd hooks, and the natural future optimization hoists the
    check up to the caller — which would silently zero the counter for
    exactly the users who do not use hooks.
  - `tool_calls` and `memory_search` sit at the surface-agnostic dispatch
    choke point, so they fire for exec and the CLI and not only the TUI.
  - `fleet_dispatch` sits at the single creation funnel, after validation:
    `create_run` and `create_queued_run` both land there, so counting at
    either would double-count a plain `fleet run`, and a rejected spec is
    not a dispatch.
  - `workflow_run` keys off the parsed `WorkflowAction` discriminant, never
    off `input["action"]`. The JSON Schema published to the model is a
    declaration, not a guard: the real parse also accepts spawn, wait, list,
    inspect, stop, and abort, and its reject arm embeds the model's string
    verbatim.

What is deliberately *not* recorded is the other half. The approval counters
are counts with no matched rule, no reason, no command, no argv — auto-allow
patterns are user-authored command strings. The MCP counter is a count of
connected servers with no name, command, URL, or error; server names routinely
name internal infrastructure. The tool error counters take the match arm, not
the error: `ToolError::PathEscape`'s `Display` *is* an absolute path.

Provider HTTP status is captured from the response at all three request sites,
before any `LlmError` is built, because every variant of that error carries the
raw provider body verbatim and a 400 from a content filter routinely echoes the
prompt. The same call records the provider as a `ProviderKind` by value. That
is the single most likely leak in this feature: the persistence identity, the
exec stream meta, and the planned route's effective label all return the
customer's own `[providers.<name>]` table key when the route is custom, and
`/status` already prints it. `ProviderKind::Custom` yields the literal
`"custom"`, and a test asserts every recorded provider is a member of the
closed set.

Gate: cargo test -p codewhale-tui --bin codewhale-tui (9797 passed)
This commit is contained in:
Hmbown
2026-08-03 20:57:07 -07:00
parent cc0b803a1f
commit 68d8d94b19
12 changed files with 719 additions and 79 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/tui/src/client.rs
+121 -75
View File
@@ -2563,11 +2563,40 @@ impl DeepSeekClient {
}
}
/// Record that a request was routed to `provider` and came back with `status`.
///
/// Called at every provider response site, **before** the error is built: an
/// `LlmError` carries the raw provider body verbatim, so the status class has
/// to be taken from the response itself.
///
/// The provider is recorded as a `ProviderKind` by value. Every accessor that
/// looks like the natural seam here — the persistence identity, the stream
/// meta's `provider_id`, the planned route's effective label — returns the
/// customer's own `[providers.<name>]` table key when the route is custom.
/// `ProviderKind::Custom` yields the literal `"custom"` and nothing else, and
/// no model id is sent for any provider.
pub(crate) fn record_provider_response(provider: crate::config::ApiProvider, status: u16) {
let counters = codewhale_telemetry::session_counters();
if let Some(kind) = provider.kind() {
counters.record_provider(kind);
}
if let Some(counter) = codewhale_telemetry::counters::http_status_counter(status) {
counters.bump_error(counter);
}
}
/// Translate the structured `LlmError` into both a categorical label
/// (for structured logs / metrics) and a short human reason string
/// (for the retry banner). Returning both from one match avoids the
/// double-classification we had before.
fn retry_reason_label_and_human(err: &LlmError) -> (&'static str, String) {
// The variant, never the payload. Every `LlmError` variant carries the raw
// provider HTTP body verbatim, and a 400 from a content filter routinely
// echoes the prompt.
if matches!(err, LlmError::NetworkError(_) | LlmError::Timeout(_)) {
codewhale_telemetry::session_counters()
.bump_error(codewhale_telemetry::ErrorCounter::NetworkError);
}
match err {
LlmError::RateLimited { retry_after, .. } => {
let human = if let Some(after) = retry_after {
@@ -3081,21 +3110,17 @@ pub(super) fn apply_reasoning_effort(
// (qwen-max, deepseek-chat, gpt-4o, claude, etc.) accepts the same
// reasoning dialect (#4188 review: verify against actual behavior).
ApiProvider::Telecomjs => {}
// Model Studio (DashScope): the OpenAI-compatible Chat Completions
// endpoints (Token Plan `compatible-mode/v1`, Coding Plan
// `coding-intl.dashscope.aliyuncs.com/v1`) accept the non-standard
// top-level `enable_thinking` switch for their hybrid-thinking
// catalog (qwen3.x, deepseek-v4, glm-5.x) and stream reasoning as
// `delta.reasoning_content`. The Anthropic-dialect variants never
// reach this Chat path — the Messages adapter shapes their
// `thinking` block instead. Source:
// <https://www.alibabacloud.com/help/en/model-studio/deep-thinking>
// Model Studio (DashScope): its top-level controls are route- AND
// model-specific, so the provider enum alone cannot decide them —
// a custom `base_url` on the same identity is an arbitrary
// gateway. `apply_modelstudio_route_reasoning_controls` in
// client::chat is the sole writer; it strips these fields for all
// four variants and re-adds them only on a verified Alibaba host.
// Source: <https://www.alibabacloud.com/help/en/model-studio/deep-thinking>
ApiProvider::ModelstudioTokenPlan
| ApiProvider::ModelstudioTokenPlanAnthropic
| ApiProvider::ModelstudioCodingPlan
| ApiProvider::ModelstudioCodingPlanAnthropic => {
body["enable_thinking"] = json!(false);
}
| ApiProvider::ModelstudioCodingPlanAnthropic => {}
ApiProvider::OpenaiCodex => {
// OpenAI Codex uses Responses API — thinking handled differently
}
@@ -3184,15 +3209,12 @@ pub(super) fn apply_reasoning_effort(
// TelecomJS: see comment in the "off" branch above — the gateway's
// Chat Completions API does not support reasoning_effort or thinking.
ApiProvider::Telecomjs => {}
// Model Studio: DashScope's hybrid-thinking switch (see the "off"
// branch). The OpenAI dialect has no effort ladder — any non-off
// level simply turns thinking on.
// Model Studio: see the "off" branch — the route- and model-aware
// shaper in client::chat is the sole writer of these fields.
ApiProvider::ModelstudioTokenPlan
| ApiProvider::ModelstudioTokenPlanAnthropic
| ApiProvider::ModelstudioCodingPlan
| ApiProvider::ModelstudioCodingPlanAnthropic => {
body["enable_thinking"] = json!(true);
}
| ApiProvider::ModelstudioCodingPlanAnthropic => {}
// OpenRouter/Novita/Together: pass through the actual user-chosen value.
// OpenRouter's unified scale is none/minimal/low/medium/high/xhigh;
// DeepSeek models hosted there accept those directly.
@@ -3288,15 +3310,12 @@ pub(super) fn apply_reasoning_effort(
// TelecomJS: see comment in the "off" branch above — the gateway's
// Chat Completions API does not support reasoning_effort or thinking.
ApiProvider::Telecomjs => {}
// Model Studio: DashScope's hybrid-thinking switch (see the "off"
// branch). The OpenAI dialect has no effort ladder — any non-off
// level simply turns thinking on.
// Model Studio: see the "off" branch — the route- and model-aware
// shaper in client::chat is the sole writer of these fields.
ApiProvider::ModelstudioTokenPlan
| ApiProvider::ModelstudioTokenPlanAnthropic
| ApiProvider::ModelstudioCodingPlan
| ApiProvider::ModelstudioCodingPlanAnthropic => {
body["enable_thinking"] = json!(true);
}
| ApiProvider::ModelstudioCodingPlanAnthropic => {}
ApiProvider::Openrouter | ApiProvider::Novita | ApiProvider::Together => {
body["reasoning_effort"] = json!("xhigh");
body["thinking"] = json!({ "type": "enabled" });
@@ -4290,61 +4309,98 @@ mod tests {
}
async fn assert_modelstudio_request_truth(streaming: bool) {
// Token Plan and Coding Plan share the DashScope `enable_thinking`
// switch on their OpenAI-compatible Chat Completions endpoints.
// Token Plan and Coding Plan share DashScope's reasoning controls on
// their OpenAI-compatible Chat Completions endpoints — but the fields
// are model-specific, not provider-wide.
for base_url in [
crate::config::DEFAULT_MODELSTUDIO_TOKEN_PLAN_BASE_URL,
crate::config::DEFAULT_MODELSTUDIO_CODING_PLAN_BASE_URL,
] {
let (off_path, off) = capture_modelstudio_chat_request(
base_url,
crate::config::DEFAULT_MODELSTUDIO_TOKEN_PLAN_MODEL,
Some("off"),
streaming,
)
.await;
assert_eq!(off_path, "/v1/chat/completions");
assert_eq!(off["enable_thinking"], json!(false), "{base_url}: {off}");
assert!(off.get("thinking").is_none(), "{base_url}: {off}");
assert!(off.get("reasoning_effort").is_none(), "{base_url}: {off}");
for effort in ["high", "max"] {
let (_, body) = capture_modelstudio_chat_request(
// The default model, qwen3.8-max, is thinking-only: the bundled
// catalog records it as `thinking: always_on`, and
// qwen3.8-max-preview has effort/budget options with no toggle.
// Neither accepts an enable/disable switch, so CodeWhale must not
// send one — not even `false` for an explicit `off`. This assertion
// used to pin the opposite; PR #5233 caught it.
for effort in [None, Some("off"), Some("high"), Some("max")] {
let (path, body) = capture_modelstudio_chat_request(
base_url,
crate::config::DEFAULT_MODELSTUDIO_TOKEN_PLAN_MODEL,
Some(effort),
effort,
streaming,
)
.await;
assert_eq!(
body["enable_thinking"],
json!(true),
"{base_url} {effort}: {body}"
assert_eq!(path, "/v1/chat/completions");
assert!(
body.get("enable_thinking").is_none(),
"{base_url} {effort:?}: {body}"
);
assert!(
body.get("thinking").is_none(),
"{base_url} {effort}: {body}"
"{base_url} {effort:?}: {body}"
);
assert!(
body.get("reasoning_effort").is_none(),
"{base_url} {effort}: {body}"
"{base_url} {effort:?}: {body}"
);
}
// Unset effort sends no switch — the server default (thinking-ON
// for the qwen3.x families) is left alone.
let (_, unset) = capture_modelstudio_chat_request(
// A hybrid model does get the documented switch, plus
// `preserve_thinking` so the next turn keeps its trace.
for (effort, enabled) in [(None, true), (Some("high"), true), (Some("off"), false)] {
let (_, body) =
capture_modelstudio_chat_request(base_url, "qwen3.7-plus", effort, streaming)
.await;
assert_eq!(
body["enable_thinking"],
json!(enabled),
"{base_url} {effort:?}: {body}"
);
assert_eq!(
body["preserve_thinking"],
json!(enabled),
"{base_url} {effort:?}: {body}"
);
// The hybrid Qwen families have no effort ladder on the wire.
assert!(
body.get("reasoning_effort").is_none(),
"{base_url} {effort:?}: {body}"
);
}
// DeepSeek-V4 is one of the two families with a documented effort
// ladder (`high` / `max`).
let (_, deepseek) = capture_modelstudio_chat_request(
base_url,
crate::config::DEFAULT_MODELSTUDIO_TOKEN_PLAN_MODEL,
None,
"deepseek-v4-pro",
Some("xhigh"),
streaming,
)
.await;
assert!(
unset.get("enable_thinking").is_none(),
"{base_url}: {unset}"
assert_eq!(
deepseek["enable_thinking"],
json!(true),
"{base_url}: {deepseek}"
);
assert_eq!(
deepseek["reasoning_effort"],
json!("max"),
"{base_url}: {deepseek}"
);
}
// Fail closed: the same provider identity pointed at a custom gateway
// must not be handed Alibaba's dialect.
let (_, proxied) = capture_modelstudio_chat_request(
"https://proxy.example/v1",
"qwen3.7-plus",
Some("high"),
streaming,
)
.await;
assert!(proxied.get("enable_thinking").is_none(), "{proxied}");
assert!(proxied.get("preserve_thinking").is_none(), "{proxied}");
assert!(proxied.get("reasoning_effort").is_none(), "{proxied}");
}
async fn assert_zai_request_truth(streaming: bool) {
@@ -7404,34 +7460,24 @@ mod tests {
}
#[test]
fn reasoning_effort_modelstudio_speaks_dashscope_enable_thinking() {
// All four plan/dialect variants share the DashScope hybrid-thinking
// switch on the OpenAI-compatible Chat Completions path.
fn reasoning_effort_modelstudio_writes_nothing_without_a_verified_route() {
// The provider enum cannot decide DashScope's controls: `enable_thinking`
// is wrong for the thinking-only models, `reasoning_effort` is only
// valid for DeepSeek-V4/GLM, and a custom `base_url` on any of these
// identities is an arbitrary gateway. All four variants must therefore
// leave the body untouched here — the route shaper in client::chat is
// the sole writer.
for provider in [
ApiProvider::ModelstudioTokenPlan,
ApiProvider::ModelstudioTokenPlanAnthropic,
ApiProvider::ModelstudioCodingPlan,
ApiProvider::ModelstudioCodingPlanAnthropic,
] {
let mut off = json!({});
apply_reasoning_effort(&mut off, Some("off"), provider);
assert_eq!(off, json!({ "enable_thinking": false }), "{provider:?}");
for effort in ["low", "high", "max"] {
for effort in [None, Some("off"), Some("low"), Some("high"), Some("max")] {
let mut body = json!({});
apply_reasoning_effort(&mut body, Some(effort), provider);
assert_eq!(
body,
json!({ "enable_thinking": true }),
"{provider:?} {effort}"
);
apply_reasoning_effort(&mut body, effort, provider);
assert_eq!(body, json!({}), "{provider:?} {effort:?}");
}
// Unset effort stays silent: the qwen3.x families default to
// thinking-ON server-side and the client must not flip that.
let mut unset = json!({});
apply_reasoning_effort(&mut unset, None, provider);
assert_eq!(unset, json!({}), "{provider:?}");
}
}
crates/tui/src/client/chat.rs
+461 -3
View File
@@ -291,6 +291,170 @@ fn apply_minimax_route_reasoning_controls(
}
}
/// Model Studio's OpenAI-compatible API uses its own top-level reasoning
/// controls. Keep them on verified Alibaba Chat Completions routes: a custom
/// `base_url` points the same provider identity at an arbitrary gateway, and
/// that gateway must not be handed Alibaba's dialect.
///
/// This is the *sole* writer of Model Studio reasoning fields —
/// `apply_reasoning_effort` deliberately writes nothing for the `Modelstudio*`
/// identities — so the strip below runs for all four variants, including the
/// two Anthropic-dialect ones. Those normally reach the Messages adapter
/// instead, but `wire = "openai"` can route them here, and an unmatched
/// `enable_thinking` left in the body would then go out unguarded.
fn apply_modelstudio_route_reasoning_controls(
body: &mut Value,
provider: ApiProvider,
base_url: &str,
model: &str,
effort: Option<&str>,
) {
if !matches!(
provider,
ApiProvider::ModelstudioTokenPlan
| ApiProvider::ModelstudioTokenPlanAnthropic
| ApiProvider::ModelstudioCodingPlan
| ApiProvider::ModelstudioCodingPlanAnthropic
) {
return;
}
if let Some(object) = body.as_object_mut() {
object.remove("thinking");
object.remove("enable_thinking");
object.remove("preserve_thinking");
object.remove("reasoning_effort");
}
if !is_exact_modelstudio_chat_route(provider, base_url) {
return;
}
let thinking_only = modelstudio_model_is_thinking_only(model);
if !thinking_only && !modelstudio_model_is_hybrid(model) {
return;
}
let thinking_enabled = !modelstudio_effort_disables_thinking(effort);
// Thinking-only models emit `reasoning_content` but reject an
// enable/disable control. Hybrid models use `enable_thinking`.
if !thinking_only {
body["enable_thinking"] = json!(thinking_enabled);
}
if modelstudio_model_supports_preserve_thinking(model) {
// Model Studio otherwise drops assistant `reasoning_content` from the
// next turn's context. This applies even when the provider default
// leaves thinking enabled and no explicit UI effort was selected.
body["preserve_thinking"] = json!(thinking_only || thinking_enabled);
}
if !thinking_only
&& thinking_enabled
&& let Some(effort) = effort.and_then(modelstudio_reasoning_effort_for_model)
&& modelstudio_model_supports_reasoning_effort(model)
{
body["reasoning_effort"] = json!(effort);
}
}
/// Fail-closed host guard: only Alibaba's own OpenAI-compatible Chat
/// Completions URL shapes count. Anything else (a proxy, a self-hosted
/// gateway, a typo) gets the Model Studio fields stripped and nothing added.
fn is_exact_modelstudio_chat_route(provider: ApiProvider, base_url: &str) -> bool {
let trimmed = base_url.trim().trim_end_matches('/').to_ascii_lowercase();
let Some((host, path)) = trimmed
.strip_prefix("https://")
.and_then(|rest| rest.split_once('/'))
else {
return false;
};
// Includes Token Plan's default and workspace-scoped
// `{workspace}.<region>.maas.aliyuncs.com/compatible-mode/v1` hosts.
let token_plan_chat = host.ends_with(".maas.aliyuncs.com") && path == "compatible-mode/v1";
let coding_plan_chat = host == "coding-intl.dashscope.aliyuncs.com" && path == "v1";
match provider {
// The primary Model Studio provider selects Coding Plan through
// `mode = "coding-plan"`, which resolves this base URL without
// changing the provider enum. Legacy Coding Plan identities remain
// supported as well, so recognize either official Chat route for the
// complete Model Studio OpenAI family. The `*Anthropic` identities
// speak the Messages dialect and are never verified here.
ApiProvider::ModelstudioTokenPlan | ApiProvider::ModelstudioCodingPlan => {
token_plan_chat || coding_plan_chat
}
_ => false,
}
}
fn is_exact_modelstudio_thinking_only_route(
provider: ApiProvider,
base_url: &str,
model: &str,
) -> bool {
is_exact_modelstudio_chat_route(provider, base_url) && modelstudio_model_is_thinking_only(model)
}
fn modelstudio_effort_disables_thinking(effort: Option<&str>) -> bool {
effort.is_some_and(|value| {
matches!(
value.trim().to_ascii_lowercase().as_str(),
"off" | "disabled" | "none" | "false"
)
})
}
/// Models with no enable/disable control at all. `models_dev.bundled.json`
/// lists `qwen3.8-max` as `thinking: always_on` and gives `qwen3.8-max-preview`
/// effort/budget options with no `toggle`, so sending `enable_thinking` to
/// either is at best ignored and at worst a 400.
fn modelstudio_model_is_thinking_only(model: &str) -> bool {
let model = model.trim().to_ascii_lowercase();
matches!(model.as_str(), "qwen3.8-max" | "qwen3.8-max-preview")
// Kimi K2.7 Code on Model Studio is reported always-thinking and
// supporting preserve_thinking. Keep it separate from hybrid Kimi
// variants so we do not send the unsupported enable_thinking switch.
|| model.starts_with("kimi-k2.7-code")
}
fn modelstudio_model_is_hybrid(model: &str) -> bool {
let model = model.trim().to_ascii_lowercase();
model.starts_with("qwen3.7-")
|| model.starts_with("qwen3.6-")
|| model.starts_with("qwen3.5-")
|| model.starts_with("qwen3-")
|| model.starts_with("deepseek-v4")
|| model.starts_with("deepseek-v3.2")
|| model.starts_with("deepseek-v3.1")
|| model.starts_with("kimi-k2.6")
|| model.starts_with("kimi-k2.5")
|| model.starts_with("glm-")
}
fn modelstudio_model_supports_preserve_thinking(model: &str) -> bool {
let model = model.trim().to_ascii_lowercase();
model.starts_with("qwen3.7-max")
|| model.starts_with("qwen3.7-plus")
|| model.starts_with("qwen3.6-max-preview")
|| model.starts_with("qwen3.6-plus")
|| model.starts_with("qwen3.6-flash")
|| model.starts_with("kimi-k2.6")
|| model.starts_with("kimi-k2.7-code")
}
fn modelstudio_model_supports_reasoning_effort(model: &str) -> bool {
let model = model.trim().to_ascii_lowercase();
model.starts_with("deepseek-v4") || matches!(model.as_str(), "glm-5.2" | "glm-5.1" | "glm-5")
}
fn modelstudio_reasoning_effort_for_model(effort: &str) -> Option<&'static str> {
match effort.trim().to_ascii_lowercase().as_str() {
// Model Studio documents low and medium as aliases for high.
"minimal" | "low" | "medium" | "mid" | "high" | "" => Some("high"),
"xhigh" | "max" | "highest" | "ultracode" => Some("max"),
_ => None,
}
}
/// Final reasoning-control pass shared by streaming and non-streaming Chat
/// Completions requests. Route-specific shapers run after the generic provider
/// layer so they can remove fields that are invalid for their exact endpoint.
@@ -302,6 +466,7 @@ pub(super) fn apply_route_reasoning_controls(
effort: Option<&str>,
) {
apply_reasoning_effort(body, effort, provider);
apply_modelstudio_route_reasoning_controls(body, provider, base_url, model, effort);
apply_minimax_route_reasoning_controls(body, provider, base_url, model, effort);
apply_inkling_reasoning_effort(body, provider, model, effort);
apply_openai_reasoning_effort(body, provider, model, effort);
@@ -594,6 +759,7 @@ impl DeepSeekClient {
let response = self.send_json_with_retry(url, body).await?;
let status = response.status();
crate::client::record_provider_response(self.api_provider, status.as_u16());
if !status.is_success() {
let raw_error_text = bounded_error_text(response, ERROR_BODY_MAX_BYTES).await;
let error_text = sanitize_http_error_body(
@@ -676,6 +842,7 @@ impl DeepSeekClient {
let (response, stream_idle_timeout) = self.open_chat_stream_response(&url, &body).await?;
let status = response.status();
crate::client::record_provider_response(self.api_provider, status.as_u16());
if !status.is_success() {
let raw_error_text = bounded_error_text(response, ERROR_BODY_MAX_BYTES).await;
let error_text = sanitize_http_error_body(
@@ -2480,11 +2647,13 @@ fn should_replay_reasoning_content_for_provider_on_route(
model: &str,
effort: Option<&str>,
) -> bool {
// Both exact K3 routes are always-thinking. A stale caller may still carry
// `off` before route normalization; retaining the assistant reasoning
// trace is required for multi-turn/tool-call continuity regardless.
// Both exact K3 routes and Model Studio's thinking-only models are
// always-thinking. A stale caller may still carry `off` before route
// normalization; retaining the assistant reasoning trace is required for
// multi-turn/tool-call continuity regardless.
if is_exact_direct_moonshot_k3_route(provider, base_url, model)
|| is_exact_kimi_code_k3_route(provider, base_url, model)
|| is_exact_modelstudio_thinking_only_route(provider, base_url, model)
{
return true;
}
@@ -2504,6 +2673,19 @@ fn should_replay_reasoning_content_for_provider_on_route(
return true;
}
// Model Studio replay is deliberately narrower than PR #5233 proposed:
// only models Alibaba documents as accepting `preserve_thinking` get their
// `reasoning_content` sent back. `deepseek-v3.1`, `deepseek-v3.2` and the
// `glm-*` ids stay stripped until someone with a Model Studio key confirms
// DashScope does not 400 on `reasoning_content` in input messages.
// `deepseek-v4*` already replays via `requires_reasoning_content` above, so
// this narrowing removes nothing that exists.
if is_exact_modelstudio_chat_route(provider, base_url)
&& modelstudio_model_supports_preserve_thinking(model)
{
return true;
}
if !provider_accepts_reasoning_content(provider) {
// Generic non-DeepSeek model on a provider that rejects the field:
// keep stripping it (preserves the #1542 fix). But a known DeepSeek
@@ -5021,6 +5203,282 @@ mod alias_thinking_detection_tests {
assert!(provider_accepts_reasoning_content(ApiProvider::Moonshot));
}
#[test]
fn modelstudio_hybrid_routes_send_documented_thinking_controls() {
let base_url = crate::config::DEFAULT_MODELSTUDIO_TOKEN_PLAN_BASE_URL;
for (effort, enabled) in [
(None, true),
(Some("low"), true),
(Some("high"), true),
(Some("xhigh"), true),
(Some("off"), false),
] {
let mut body = json!({});
apply_route_reasoning_controls(
&mut body,
ApiProvider::ModelstudioTokenPlan,
base_url,
"qwen3.7-plus",
effort,
);
assert_eq!(body["enable_thinking"], json!(enabled), "{effort:?}");
assert_eq!(body["preserve_thinking"], json!(enabled), "{effort:?}");
assert!(body.get("thinking").is_none(), "{effort:?}: {body}");
assert!(body.get("reasoning_effort").is_none(), "{effort:?}: {body}");
}
}
#[test]
fn modelstudio_deepseek_v4_maps_effort_to_documented_values() {
let base_url = crate::config::DEFAULT_MODELSTUDIO_TOKEN_PLAN_BASE_URL;
for (requested, expected) in [("low", "high"), ("high", "high"), ("xhigh", "max")] {
let mut body = json!({});
apply_route_reasoning_controls(
&mut body,
ApiProvider::ModelstudioTokenPlan,
base_url,
"deepseek-v4-pro",
Some(requested),
);
assert_eq!(body["enable_thinking"], json!(true), "{requested}");
assert_eq!(body["reasoning_effort"], json!(expected), "{requested}");
}
}
#[test]
fn modelstudio_reasoning_controls_fail_closed_on_custom_gateways() {
let mut body = json!({
"enable_thinking": true,
"preserve_thinking": true,
"reasoning_effort": "high",
});
apply_route_reasoning_controls(
&mut body,
ApiProvider::ModelstudioTokenPlan,
"https://proxy.example/v1",
"qwen3.7-plus",
Some("high"),
);
assert!(body.get("enable_thinking").is_none());
assert!(body.get("preserve_thinking").is_none());
assert!(body.get("reasoning_effort").is_none());
}
#[test]
fn modelstudio_anthropic_identities_write_nothing_on_the_chat_path() {
// The Messages adapter owns these two. If `wire = "openai"` ever routes
// them through Chat Completions, the shaper must strip rather than
// inherit the OpenAI-dialect fields — there is no provider-enum writer
// left to re-add them.
for provider in [
ApiProvider::ModelstudioTokenPlanAnthropic,
ApiProvider::ModelstudioCodingPlanAnthropic,
] {
for base_url in [
crate::config::DEFAULT_MODELSTUDIO_TOKEN_PLAN_BASE_URL,
crate::config::MODELSTUDIO_TOKEN_PLAN_ANTHROPIC_BASE_URL,
] {
let mut body = json!({ "enable_thinking": true });
apply_route_reasoning_controls(
&mut body,
provider,
base_url,
"qwen3.7-plus",
Some("high"),
);
assert_eq!(body, json!({}), "{provider:?} {base_url}");
}
}
}
#[test]
fn modelstudio_qwen38_route_classifies_reasoning_and_replays_history() {
let base_url = crate::config::DEFAULT_MODELSTUDIO_TOKEN_PLAN_BASE_URL;
for model in ["qwen3.8-max", "qwen3.8-max-preview"] {
// qwen3.8 is thinking-only. Effort selection must never hide its
// separate reasoning stream, including the stale `off` state
// that can arrive before route normalization.
assert_eq!(
reasoning_stream_style_for_route(
ApiProvider::ModelstudioTokenPlan,
base_url,
model,
None,
),
ReasoningStreamStyle::SeparateField,
"{model}"
);
for effort in [None, Some("off"), Some("high"), Some("xhigh")] {
assert!(
should_replay_reasoning_content_for_provider_on_route(
ApiProvider::ModelstudioTokenPlan,
base_url,
model,
effort,
),
"{model} {effort:?}"
);
}
// ...and no enable/disable switch is ever sent for them.
let mut body = json!({});
apply_route_reasoning_controls(
&mut body,
ApiProvider::ModelstudioTokenPlan,
base_url,
model,
Some("off"),
);
assert!(body.get("enable_thinking").is_none(), "{model}: {body}");
}
}
#[test]
fn modelstudio_hybrid_route_classifies_reasoning_and_replays_history() {
let base_url = crate::config::DEFAULT_MODELSTUDIO_TOKEN_PLAN_BASE_URL;
assert_eq!(
reasoning_stream_style_for_route(
ApiProvider::ModelstudioTokenPlan,
base_url,
"qwen3.7-plus",
None,
),
ReasoningStreamStyle::SeparateField,
);
assert!(should_replay_reasoning_content_for_provider_on_route(
ApiProvider::ModelstudioTokenPlan,
base_url,
"qwen3.7-plus",
None,
));
assert!(!should_replay_reasoning_content_for_provider_on_route(
ApiProvider::ModelstudioTokenPlan,
base_url,
"qwen3.7-plus",
Some("off"),
));
}
#[test]
fn modelstudio_replay_stays_narrow_until_a_live_key_confirms_it() {
// Deliberately narrower than PR #5233: only `preserve_thinking` models
// replay. GLM and DeepSeek-V3.x on Model Studio stay stripped until
// someone with a key confirms DashScope accepts `reasoning_content` in
// input messages. deepseek-v4* is unaffected — it replays through
// `requires_reasoning_content` on every provider.
let base_url = crate::config::DEFAULT_MODELSTUDIO_TOKEN_PLAN_BASE_URL;
for model in ["glm-5.2", "deepseek-v3.2", "deepseek-v3.1"] {
assert!(
!should_replay_reasoning_content_for_provider_on_route(
ApiProvider::ModelstudioTokenPlan,
base_url,
model,
None,
),
"{model}"
);
}
assert!(should_replay_reasoning_content_for_provider_on_route(
ApiProvider::ModelstudioTokenPlan,
base_url,
"deepseek-v4-pro",
None,
));
}
#[test]
fn modelstudio_coding_plan_chat_route_is_classified_for_all_supported_identities() {
// The picker represents Coding Plan as mode = "coding-plan" under
// the primary provider id, so the chat client receives
// ModelstudioTokenPlan with the Coding Plan URL. Direct configuration
// also retains the legacy ModelstudioCodingPlan identity.
let base_url = crate::config::DEFAULT_MODELSTUDIO_CODING_PLAN_BASE_URL;
for provider in [
ApiProvider::ModelstudioTokenPlan,
ApiProvider::ModelstudioCodingPlan,
] {
let mut body = json!({});
apply_route_reasoning_controls(
&mut body,
provider,
base_url,
"qwen3.7-plus",
Some("high"),
);
assert_eq!(body["enable_thinking"], json!(true), "{provider:?}");
assert_eq!(body["preserve_thinking"], json!(true), "{provider:?}");
assert_eq!(
reasoning_stream_style_for_route(provider, base_url, "qwen3.7-plus", None),
ReasoningStreamStyle::SeparateField,
"{provider:?}",
);
assert!(should_replay_reasoning_content_for_provider_on_route(
provider,
base_url,
"qwen3.7-plus",
None,
));
}
}
#[test]
fn modelstudio_workspace_scoped_token_plan_route_is_recognized() {
let workspace_url =
"https://workspace-123.ap-southeast-1.maas.aliyuncs.com/compatible-mode/v1";
assert_eq!(
reasoning_stream_style_for_route(
ApiProvider::ModelstudioTokenPlan,
workspace_url,
"qwen3.8-max",
None,
),
ReasoningStreamStyle::SeparateField,
);
assert!(should_replay_reasoning_content_for_provider_on_route(
ApiProvider::ModelstudioTokenPlan,
workspace_url,
"qwen3.8-max",
None,
));
}
#[test]
fn modelstudio_kimi_k27_code_is_thinking_only_and_preserves_trace() {
// NOTE: unlike the qwen3.8 pair, this classification is asserted by
// PR #5233 rather than corroborated by models_dev.bundled.json, which
// lists kimi-k2.7-code with `reasoning: true` and no `always_on`.
let base_url = crate::config::DEFAULT_MODELSTUDIO_TOKEN_PLAN_BASE_URL;
let mut body = json!({});
apply_route_reasoning_controls(
&mut body,
ApiProvider::ModelstudioTokenPlan,
base_url,
"kimi-k2.7-code",
Some("off"),
);
assert!(body.get("enable_thinking").is_none());
assert_eq!(body["preserve_thinking"], json!(true));
assert_eq!(
reasoning_stream_style_for_route(
ApiProvider::ModelstudioTokenPlan,
base_url,
"kimi-k2.7-code",
None,
),
ReasoningStreamStyle::SeparateField,
);
assert!(should_replay_reasoning_content_for_provider_on_route(
ApiProvider::ModelstudioTokenPlan,
base_url,
"kimi-k2.7-code",
Some("off"),
));
}
#[test]
fn stream_classifies_moonshot_kimi_as_reasoning() {
// #3016: without this, Kimi thinking leaked into answer text.
crates/tui/src/client/responses.rs
+1
View File
@@ -183,6 +183,7 @@ impl DeepSeekClient {
.await?;
let status = response.status();
crate::client::record_provider_response(self.api_provider, status.as_u16());
if !status.is_success() {
let raw = bounded_error_text(response, ERROR_BODY_MAX_BYTES).await;
anyhow::bail!("Responses API error (HTTP {status}): {raw}");
crates/tui/src/core/engine.rs
+5
View File
@@ -5569,6 +5569,11 @@ fn tool_ask_rule_decision_for_context(
} else if decision.requires_approval {
Some(ToolAskRuleDecision::Prompt(decision.reason().to_string()))
} else if decision.matched_action == Some(codewhale_execpolicy::PermissionAction::Allow) {
// Count only. Never `matched_rule`, never `reason()`, never the
// command or its argv: `auto_allow` patterns are user-authored command
// strings.
codewhale_telemetry::session_counters()
.bump(codewhale_telemetry::Counter::ApprovalAutoAllowed);
Some(ToolAskRuleDecision::Allow)
} else {
None
crates/tui/src/core/engine/tool_execution.rs
+20
View File
@@ -472,6 +472,14 @@ impl Engine {
};
let duration_ms = started_at.elapsed().as_millis() as u64;
// The surface-agnostic choke point for every tool call, so this one
// bump covers exec and the CLI as well as the TUI. `memory_search` is
// counted here for the same reason — one site, not one per tool.
let telemetry = codewhale_telemetry::session_counters();
telemetry.bump(codewhale_telemetry::Counter::ToolCalls);
if tool_name == "memory_search" {
telemetry.bump(codewhale_telemetry::Counter::MemorySearch);
}
match &outcome {
Ok(result) => {
tracing::debug!(
@@ -495,6 +503,18 @@ impl Engine {
ToolError::NotAvailable { .. } => "not_available",
ToolError::PermissionDenied { .. } => "permission_denied",
};
// The discriminant and nothing else. `ToolError::PathEscape`'s
// `Display` *is* an absolute path, and several sibling
// variants render a literal source fragment the model emitted.
match err {
ToolError::PermissionDenied { .. } => {
telemetry.bump_error(codewhale_telemetry::ErrorCounter::ToolDeniedByPolicy)
}
ToolError::Timeout { .. } => {
telemetry.bump_error(codewhale_telemetry::ErrorCounter::ToolTimeout);
}
_ => {}
}
tracing::warn!(
target: "engine.tool_execution",
tool = %tool_name,
crates/tui/src/fleet/exact.rs
+4
View File
@@ -785,6 +785,10 @@ pub(crate) fn preflight_route(
} else if crate::config::has_api_key_for(&scoped, identity.provider) {
CredentialReadiness::Configured
} else {
// The discriminant only. `Missing { detail }` names the provider table
// key, which for a custom route is the customer's own string.
codewhale_telemetry::session_counters()
.bump_error(codewhale_telemetry::ErrorCounter::AuthPreflightFailed);
CredentialReadiness::Missing {
detail: format!("no credential configured for `{}`", identity.key),
}
crates/tui/src/fleet/manager.rs
+5
View File
@@ -367,6 +367,11 @@ impl FleetManager {
descriptor: ManagedFleetRunDescriptor,
) -> Result<FleetRunReport> {
validate_task_spec_document(&doc)?;
// The single funnel: `create_run` and `create_queued_run` both land
// here, so counting at either of those would double-count a plain
// `fleet run`. Counted after validation, so a rejected spec is not a
// dispatch.
codewhale_telemetry::session_counters().bump(codewhale_telemetry::Counter::FleetDispatch);
worker_runtime::canonicalize_fleet_task_roles(&mut doc.tasks);
let roster = self.agent_roster();
worker_runtime::validate_task_agent_profiles(&doc.tasks, roster.members())?;
crates/tui/src/main.rs
+64
View File
@@ -17500,3 +17500,67 @@ mod telemetry_surface_tests {
assert!(RunTerminationReason::Resolved.is_success());
}
}
#[cfg(test)]
mod telemetry_counter_tests {
use codewhale_telemetry::counters::http_status_counter;
use codewhale_telemetry::{ErrorCounter, session_counters};
#[test]
fn a_custom_provider_is_recorded_as_the_literal_custom() {
// This is the single most likely leak in the feature: the persistence
// identity, the exec stream meta, and the planned route's effective
// label all return the customer's own `[providers.<name>]` table key
// when the route is custom, and `/status` already prints it. The
// recording API takes a `ProviderKind` by value so none of them fit.
crate::client::record_provider_response(crate::config::ApiProvider::Custom, 200);
let providers = session_counters().providers();
assert!(
providers.iter().any(|name| name == "custom"),
"expected the literal `custom`, got {providers:?}"
);
let closed: std::collections::BTreeSet<&str> = codewhale_config::ProviderKind::ALL
.iter()
.map(|kind| kind.as_str())
.collect();
for name in &providers {
assert!(
closed.contains(name.as_str()),
"`{name}` is not a ProviderKind; a table name reached the provider set"
);
}
}
#[test]
fn http_status_classes_land_in_the_right_error_counter() {
// Captured from the response, before any `LlmError` is built: every
// variant of that error carries the raw provider body verbatim.
assert_eq!(http_status_counter(200), None);
assert_eq!(http_status_counter(304), None);
assert_eq!(
http_status_counter(429),
Some(ErrorCounter::ProviderHttp4xx)
);
assert_eq!(
http_status_counter(503),
Some(ErrorCounter::ProviderHttp5xx)
);
}
#[test]
fn the_turn_wall_histogram_buckets_by_wall_clock_not_per_turn_events() {
// A histogram, never a timestamped series: a stream of per-turn
// durations reconstructs a session's working rhythm.
let mut wall = codewhale_telemetry::TurnWall::default();
wall.observe_secs(0);
wall.observe_secs(4);
wall.observe_secs(5);
wall.observe_secs(119);
wall.observe_secs(120);
assert_eq!(wall.lt_5s, 2);
assert_eq!(wall.five_to_thirty, 1);
assert_eq!(wall.thirty_to_onetwenty, 1);
assert_eq!(wall.gte_120s, 1);
}
}
crates/tui/src/mcp.rs
+7
View File
@@ -4502,6 +4502,13 @@ fn snapshot_from_config(
}
if let Some(conn) = pool.connections.get(name) {
snapshot.connected = conn.is_ready();
if snapshot.connected {
// A count of connected servers and nothing else. The
// name, the command or URL, and the error string are
// user-chosen and routinely name internal infra.
codewhale_telemetry::session_counters()
.bump(codewhale_telemetry::Counter::McpServerConnected);
}
snapshot.tools = conn
.tools()
.iter()
crates/tui/src/tools/workflow.rs
+8 -1
View File
@@ -866,7 +866,14 @@ impl ToolSpec for WorkflowTool {
async fn execute(&self, input: Value, context: &ToolContext) -> Result<ToolResult, ToolError> {
let state = shared_workflow_state(&context.workspace);
attach_bound_workflow_lifecycles(context, &state)?;
match parse_workflow_action(&input)? {
// Keyed off the parsed `WorkflowAction` discriminant, never off
// `input["action"]`. The JSON Schema published to the model is a
// declaration, not a guard: the real parse also accepts `spawn`,
// `wait`, `list`, `inspect`, `stop`, and `abort`, and its reject arm
// embeds the model's string verbatim.
let action = parse_workflow_action(&input)?;
codewhale_telemetry::session_counters().bump(codewhale_telemetry::Counter::WorkflowRun);
match action {
WorkflowAction::Start => {
let wait = optional_bool(&input, "wait", false);
start_workflow(
crates/tui/src/tui/mouse_ui.rs
+2
View File
@@ -1324,6 +1324,8 @@ pub(crate) fn handle_context_menu_action(app: &mut App, action: ContextMenuActio
}
}
ContextMenuAction::OpenCommandPalette => {
codewhale_telemetry::session_counters()
.bump(codewhale_telemetry::Counter::CommandPaletteOpen);
app.view_stack.push(CommandPaletteView::new_for_locale(
app.ui_locale,
build_command_palette_entries(
crates/tui/src/tui/ui.rs
+21
View File
@@ -1774,6 +1774,7 @@ fn apply_agent_spawned_status_and_observer(
prompt_summary: &str,
) {
let label = app.ensure_agent_label(agent_id);
codewhale_telemetry::session_counters().bump(codewhale_telemetry::Counter::SubagentSpawn);
app.status_message = Some(format!("{label} starting: {prompt_summary}"));
if let Err(error) =
execute_subagent_observer_hook(app, HookEvent::SubagentSpawn, agent_id, "prompt", prompt)
@@ -4199,6 +4200,19 @@ async fn run_event_loop(
}
}
// Counted here, at the caller, never inside
// `execute_turn_end_observer_hook`: that function's
// first statement returns early for anyone with no
// TurnEnd hooks, and the natural future optimization
// hoists that check up to this call site — which would
// silently zero the counter for every user who does
// not use hooks.
{
let telemetry = codewhale_telemetry::session_counters();
telemetry.bump(codewhale_telemetry::Counter::Turns);
telemetry.observe_turn_secs(turn_elapsed.as_secs());
}
if let Err(error) = execute_turn_end_observer_hook(
app,
completed_turn.as_ref(),
@@ -4683,6 +4697,11 @@ async fn run_event_loop(
intent_summary,
approval_force_prompt,
} => {
// A count and nothing else. The tool name, the
// description, the input, and the matched rule are all
// user- or model-authored strings.
codewhale_telemetry::session_counters()
.bump(codewhale_telemetry::Counter::ApprovalModalShown);
if app.remote_control.blocks_local_input() {
let gate = app.remote_control.record_remote_approval(
&id,
@@ -6203,6 +6222,8 @@ async fn run_event_loop(
if app.view_stack.is_empty() && app.kill_to_end_of_line() {
continue;
}
codewhale_telemetry::session_counters()
.bump(codewhale_telemetry::Counter::CommandPaletteOpen);
app.view_stack.push(CommandPaletteView::new_for_locale(
app.ui_locale,
build_command_palette_entries(