github-featured / comet-ml--opik

发布

0 发布 8,340 Git 标签

RSS 订阅源
  • 2.2.12-6203 a102729409
    比较

    [OPIK-7532] [QA] feat: capability coverage map — taxonomy, spec tags, and tag-lint PR gate (#7678)

    frostbyte_neo 发布于 2026-07-30 10:19:50 +00:00

    • [OPIK-7532] [QA] test: tag e2e and visual specs with @area:/@cap: for the coverage map

    Adds the coverage-map tags to every spec in tests_end_to_end/. Tags only —
    no test logic, no selectors, no assertions changed.

    What the tags mean:
    @area: which product area a spec belongs to (one per spec)
    @cap:. which capabilities a test proves
    @vcap:. same, for visual specs

    Existing tier tags (@t1-smoke/@t2-cuj/@t3-nightly) and suite selectors
    (@t1-stsaas, @provider-sanity) are untouched, so nothing about which tests
    run in which job changes. Verified below.

    Granularity: @area: on the describe; @cap: per test where a describe holds
    several distinct tests, on the describe where its tests are variants of one
    capability (the prompt text/chat loops). Per-test is what lets a red run name
    the capability that regressed rather than just the area.

    Two tags were deliberately left off:

    • The bare @prompts tag on the four prompt-playground specs becomes
      @area:playground, not @area:prompts — those specs drive the Playground and
      only assert on the library as an outcome. Attributing them to prompts
      overcounted that area.
    • trace-explore-smoke asserts the project breadcrumb, but only as project
      context for the trace panel, so it does not claim
      @cap:projects.breadcrumb-selector. A cross-area cap there would report the
      Projects area as covered on the strength of an incidental assertion.

    @trace-explore is retired by this change: it predated the split of the Logs
    surface into distinct traces and threads areas. The three specs carrying it
    resolve to @area:traces (trace-explore-smoke, trace-spans-depth) and
    @area:threads (thread-logging-smoke).

    Co-Authored-By: Claude Opus 5 noreply@anthropic.com

    • [OPIK-7532] [QA] ci: add the capability taxonomy, tag lint, and its PR gate

    Adds the other half of the coverage-map work: the taxonomy that defines the
    denominator, the lint that enforces the tag grammar, the conventions doc, and
    the workflow that runs the lint on every PR touching tests_end_to_end/.

    tests_end_to_end/coverage/taxonomy.yaml 18 areas, ~188 capabilities
    tests_end_to_end/coverage/tag_lint.py 6 rules, exits non-zero on any
    tests_end_to_end/TESTING-TAGS.md the grammar and how to extend it
    .github/workflows/tag_lint.yml PR gate on tests_end_to_end/**

    Why this gates PRs instead of running nightly: tier selection is --grep, so a
    spec with a missing or typo'd tag is not a failure — it is silently never
    selected, and the root npm test even passes --pass-with-no-tests. An untagged
    spec therefore stops running with no signal at all. Catching that at review
    time is the point; a nightly job reports it the morning after.

    Why the taxonomy lives here rather than in comet-automation-tests with the
    other QA tooling: a gate that reads it must work on pull requests from external
    contributors, where no cross-repo checkout token is available. Keeping it beside
    the specs it describes means the check needs no secrets, no internal runner and
    no cross-repo checkout. The coverage builder reads it from this repo.

    Coverage as it stands, reported per dimension and never blended:
    functional 75/188 (41% self-hosted, excluding the two cloud-only areas)
    visual 22/54
    load 0/9 denominator defined, status: planned

    The load dimension is declared but deliberately inert: tests_load/ reports to
    JUnit rather than Allure, so it is not joinable yet, and wiring it up is a
    separate decision.

    Co-Authored-By: Claude Opus 5 noreply@anthropic.com

    • [OPIK-7532] [QA] fix: address review — per-variant caps, cap-required lint rule

    Three findings from the PR review were real:

    • prompt-library-smoke tagged both @cap:prompts.create-text-prompt-ui and
      create-chat-prompt-ui on every iteration of a variant loop, so the text test
      claimed the chat path it never exercises. Split into two explicit tests with
      literal tags. Building the tag array from the variant would have been worse
      than the bug: both the lint and the coverage builder match string literals, so
      a computed tag drops the capability silently instead of mis-attributing it.
      Also added the versioning caps runVersioningSteps genuinely asserts.

    • tag_lint never checked that a non-visual spec has any @cap: at all, so a spec
      could pass the gate while contributing nothing to the map. Now enforced.

    • the module docstring overstated what is enforced: it claimed "exactly one
      tier" (a suite selector is a valid tier-less opt-out) and described the visual
      state check as scanning specs when it validates the taxonomy. Corrected, and
      the deliberately-unenforced tier cardinality now says why: the invariant is
      per-test after describe-inheritance, which needs the TS AST, and four specs
      legitimately carry several tiers across sibling describes.

    Declined: SHA-pinning actions and hash-pinning pyyaml in tag_lint.yml. The job
    has contents:read and no secrets, and every other workflow in the repo floats
    major tags — one pinned file is an inconsistency that rots, not a posture.

    Verified: lint clean (26 specs), tsc clean, selection counts unchanged
    (t1 20/11, all 56/24), coverage still 74/188 with zero flag drift.

    • [OPIK-7532] [QA] feat: nightly job to reconcile the taxonomy with spec tags

    The tags are the source of truth for coverage; the covered:/tier: values in
    taxonomy.yaml are a cache of them. This job refreshes that cache nightly and
    opens a PR when it drifts, so nobody hand-maintains those flags. Authored fields
    — areas, capability keys, notes, cloud_only, state, axes — are never touched.

    Two design points worth knowing:

    Tags are read via playwright test --list --reporter=json, not a regex. Tags
    union from describe to test, and four specs legitimately carry several tiers
    across sibling describes, so the tier covering a given @cap: is only knowable
    after inheritance is resolved. A regex first draft of this job was measured
    "correcting" 5 accurate tier fields to wrong values — dataset-items and
    ollie-agentic caps got attributed to the shallowest tier in their file rather
    than their own. No browser is launched, so playwright install is skipped.

    It edits lines in place rather than round-tripping the YAML. The taxonomy carries
    175 comments and 242 column-aligned flow mappings that safe_load + dump would
    flatten into a ~700-line nightly diff. Verified: the one real drift this found
    produces a one-line diff, and an injected note: flip round-trips
    byte-identically.

    Applied the single genuine drift it found: prompts.version-history is t1-smoke,
    not t2-cuj — the UI prompt tests added in the previous commit carry that cap and
    sit in a @t1-smoke describe.

    Verified: --check exits 0 on the reconciled file; covered-flag flips tested in
    both directions with tier inserted/dropped alongside; orphan-tag path warns,
    refuses to invent a capability, and now exits non-zero; clean-tree CI simulation
    passes with npm ci --ignore-scripts and no browsers; actionlint clean.

    • [OPIK-7532] [QA] fix: reconcile must fail closed, not wipe the taxonomy

    Second review round. Four real findings, all in the new reconcile job.

    The serious one: playwright test --list on an estate with a single broken spec
    exits 1 but still prints valid JSON — suites: [] plus an errors array.
    The old code only guarded against empty stdout, so it parsed that happily, saw
    zero tags, and would have flipped all 74 covered capabilities to covered: false, committed the wipe, and opened a PR. Measured: one bad spec => exit 1,
    3.7 kB of parseable JSON, 0 tests, 1 error. Now checks returncode, unparseable
    JSON, and a non-empty errors array before anything is written, and surfaces the
    collection error (file + line + message) rather than a slice of the JSON header.

    Same fail-open shape for a missing project: a dimension with no package.json was
    silently skipped, leaving its tag map empty — which also reads as "nothing is
    covered". Now a hard error.

    Also:

    • stable PR branch name. The timestamped branch meant create-pull-request opened
      a fresh reconciliation PR every night instead of updating the open one.
    • corrected the docstring's exit-code claim: orphan-tag warnings exit 1 even when
      no derived field drifted.
    • dropped hard-coded spec counts and a spec name from the tag_lint tier-cardinality
      rationale; the reasoning is stable, those numbers are not.
    • RuntimeError now prints as a message, not a traceback — these are operational
      failures with actionable text, not bugs.

    Verified: with a broken spec, reconcile exits 1 and the taxonomy is byte-identical
    (diff -q clean); missing project exits 1; happy path still --check exits 0;
    tag-lint 26/0; actionlint clean.

    • [OPIK-7532] [QA] fix: route operational failures through the error boundary

    Third review round. Both findings are the same real hole, and both are correct:
    narrowing main() to except RuntimeError last round left the operational
    exceptions that are not RuntimeError subclasses escaping as raw tracebacks.

    Reproduced before fixing:
    malformed taxonomy YAML -> 16-line yaml.scanner traceback
    npx not on PATH -> 8-line FileNotFoundError traceback

    Neither TimeoutExpired, OSError nor yaml.YAMLError is a RuntimeError subclass, so
    the boundary never saw them. Now converted at their source, with context the
    message actually needs:

    • subprocess.run: TimeoutExpired -> the timeout; OSError -> "is npm ci done in
      this project?", since that is the usual cause in CI.
    • taxonomy read/parse: OSError -> path; YAMLError -> parser message; plus a
      non-mapping guard, because a scalar document would otherwise fail much later
      inside .get() with a confusing AttributeError.
    • write_text: OSError -> path. This one sits after the boundary in main().

    Deliberately not broadened to except Exception, per the bot's alternative
    suggestion: anything outside this set is a bug, and a stack trace is the correct
    output for a bug. The narrow handler is the point.

    Also stopped re-reading the taxonomy for the line-surgery pass — it now reuses
    the text the parse consumed, so a mid-run edit cannot desynchronise the line
    numbers from the parsed structure.

    Verified: all four operational paths exit 1 with zero traceback lines and an
    actionable first line; broken-spec and missing-project guards unchanged with the
    taxonomy byte-identical; happy path --check exits 0; tag-lint 26/0.

    Co-authored-by: Andrei Căutișanu andreicautisanu@ip-192-168-1-156.eu-west-1.compute.internal
    Co-authored-by: Claude Opus 5 noreply@anthropic.com
    Co-authored-by: Andrei Căutișanu andreicautisanu@Andreis-MacBook-Pro.local

    下载附件