Chris Tate
f8c14c59c7
* Add the POSIX pty primitive: openpty, controlling-terminal fork/exec, resize, reap - One transport for terminal children: parent-built argv/envp (clean env plus TERM), async-signal-safe child path via login_tty/execve, group kill and waitpid decode - macOS and Linux-with-libc report supported; every other target compiles to loud PtyUnsupported stubs - Live tests prove the controlling terminal (test -t 0), exit codes, signal decode, and the env policy * Pin libghostty-vt and stage the terminal example skeleton - examples/terminal owns its build: ghostty pinned at 7aa9591 (the first upstream commit that builds under Zig 0.16.0; v1.3.1 still targets 0.15) with simd off so the vt module stays pure Zig - The pin is proven in-tree: the example's test round-trips VT parsing through the pinned module - zig-pkg/ (the project-local package store) joins .gitignore * Add the pty effect vocabulary: spawn, coalesced output, write, resize, kill, replayable exits - fx.ptySpawn/ptyWrite/ptyResize/ptyKill ride the keyed families' seams: one key space, the spawn argv budgets and env policy, exactly one exit per spawn, rejections staged loud and regenerating - Output coalesces through a per-pty staging ring paced by the drain (lossless back-pressure: a full ring parks the reader, never drops), delivered as bounded batches journaled via the content-addressed blob store; replay never spawns a process - The fake pty scripts the whole vocabulary headless (request/write/resize/kill mirrors plus output/exit feeds), and live POSIX tests pin coalescing, losslessness, controlling-terminal wiring, and teardown convergence * Render the terminal example: libghostty-vt grid on the canvas, keyboard-first, replayable - grid.zig wraps one emulator session (cell state, damage, scrollback, keyboard selection) and paints the viewport as real text: per-row background/text runs, theme-mapped ANSI-16 with exact 256-color and truecolor, wide CJK cells, a scrollback thumb - main.zig drives it on the pty vocabulary: typing rides the IME-correct text channel and the emulator key encoder, selection/scroll/copy chords, a variable-length chrome prefix for the grid, and the frame pump resizes the grid and pty together - UiApp gains on_text (the target-less committed-text seam) and a variable_prefix chrome mode; the gpu-surface layer routes unclaimed text_input to the app, the key_down fallback's twin - Tests pin the emulator round trip, CJK width, scrollback, line/block selection, palette honesty, and a fake-pty session that replays byte-identical offline * Document the terminal recipe, platform matrix, and replay story - docs/terminal: the pty vocabulary, coalesced output, the exit taxonomy, grid rendering, and the byte-identical offline replay story, with the honest macOS/Linux/Windows-staged/null matrix - Registered in the Core Concepts nav beside Dynamic Images - Changelog fragments for the pty vocabulary, the terminal example, on_text, and the TS Cmd family * Harden the pty transport: exec-failure detection, teardown, and write accounting - pty spawn distinguishes a failed exec from a real 127 exit via a close-on-exec self-pipe (a bad shebang interpreter now reports spawn_failed, not exited), and rejects argv entries with embedded NULs instead of silently truncating them at the C boundary - The io loop observes shutdown so a reader parked on a full staging ring exits promptly at teardown instead of forcing the abandon path - dropped_writes now counts outbound bytes lost to a write failure or a child that exits with input still staged; a kill racing a natural exit rewrites the terminal to cancelled with the -1 sentinel code - Bounded per-drain output: a delivered batch re-stamps the remaining backlog past the pass boundary, so a continuously writing child delivers one batch per drain instead of starving other events * Harden pty fd hygiene, the kill/reap races, replay provenance, and grid fidelity - pty spawn aborts if CLOEXEC on the exec self-pipe cannot be set (a leaked write end would hang the parent), and the wake pipe is now CLOEXEC so no descriptor leaks into the child; empty PATH components resolve as the current directory (POSIX) - ptyKill no longer signals a reaped pid (the OS may have reused it) and the io loop winds down on kill even when an escaped descendant holds the pty open, so the exit always delivers; dropped_writes counts each lost payload, not one lumped event - Platform-unsupported rejections are executor truth and feed under replay (a journal recorded where ptys are unsupported replays its rejection verbatim); the regenerating marker rides the record's truncated bit so a pty exit reason stays honest, with a replay damage gate for the pairing - feedPtyOutput refuses an over-bound batch instead of truncating, and every fake pty (not just replay parks) refuses a second feed after its exit is queued - Example: inverse-video cells paint text in the background color (not foreground-on-foreground), the grid self-limits to its chrome command budget so a pathological screen degrades to fewer rows, and the Linux default shell is /bin/sh (present on every install, unlike /bin/bash) * Wire the pty command family into the TypeScript tier - Cmd.ptySpawn/ptyWrite/ptyResize/ptyKill (wire opcodes 0x19-0x1C, additive over the current cmd format) expose the pty vocabulary to transpiled cores, with an event arm matched by field name (key/kind/bytes/code/reason/signal/droppedWrites) - The rt kernel encodes the four ops; the emitter lowers and validates them (argv and grid bounds) with teaching diagnostics; the ts-core host decodes them into the runtime Effects pty calls and dispatches events back to the core's event arm - Posting stays native-only (transpiled cores are single-threaded): the TS tier spawns, writes, resizes, kills, and receives. Covered by packages/core effects/conformance tests and the ts_core_host bridge tests, plus a cmdview decoder arm so the evals harness names the new commands * Harden pty close-on-exec fds, the pid-reuse guard, exact write accounting, and NUL-safe TERM - The pty parent end/child end (and the exec self-pipe read end) are close-on-exec, so a concurrent process or pty spawn on another thread cannot inherit another session's descriptors and hold its pty open; pipePair now aborts if either non-blocking fcntl fails instead of returning a pipe that could hang the UI thread in nudge - Closed the pid-reuse window: the io thread publishes a reaping flag under the mutex before waitpid, and ptyKill and teardown decline to signal once it (or exit_staged/io_done) is set, so a reaped child's reused pid is never signalled - dropped_writes now tracks per-payload boundaries: a fully-sent write pops as its bytes flush, so a discard counts only the writes that never reached the child, not already-delivered ones - ptySpawn rejects argv or TERM containing an embedded NUL (validated at the effect boundary so the fake executor and replay agree), never a silently truncated value * Bound the pty exec probe and reap, make kill signalling atomic, and cap the grid command budget - The exec-status probe polls with a timeout instead of a blocking read: a real exec failure writes its byte instantly (always detected), success is EOF, and the timeout is the net for the residual CLOEXEC window so a concurrent fork inheriting the pipe writer can never hang the spawn. Pipes are created close-on-exec atomically on Linux (pipe2); Darwin keeps the immediate fcntl behind the same timeout net - reapEnding replaces the unbounded reapBlocking after the stream ends: the normal case (child already exited) returns at once with no signal, and a child that closed its terminal but kept running is hung up and escalated to SIGKILL within a bounded window, so the exit always arrives and no kill is stranded - ptyKill and teardown signal the child UNDER the shared mutex, atomic against the io thread's pre-waitpid reaping publish, so a reaped pid's reuse is never signalled - dropped_writes stays exact: the outbound record ring is an admission bound (a write past it is refused and counted once, like a full byte buffer), so every accepted write owns a length record and none is folded - Example: the grid's per-row command reserve accounts for background + text + underline per cell (3/column) so a pathological screen truncates safely under the chrome budget; a resize allocation failure leaves the model dimensions uncommitted so the emulator and pty never disagree and the frame pump retries * Open the pty pair with an atomically close-on-exec child end, reset reused header flags - Replace openpty with posix_openpt + grantpt + unlockpt and open the child end O_CLOEXEC atomically: the child end is the descriptor whose inheritance across a concurrent fork's exec would hold the pty open and starve the exit event, and an atomic open closes that window on both platforms (the parent end's Darwin sub-syscall window is benign — an inherited parent end does not hold the pty open) - Reset every reused PtyShared flag on a new spawn (reaping, the outbound write records) so a prior session's state never steers the next: a stale reaping would make ptyKill skip its immediate SIGKILL - Declare ioctl variadic to match the C ABI — a fixed-arg declaration mis-passed the winsize pointer on arm64, silently dropping the initial grid and every resize; the resize test now verifies the applied size, not just the call * Handle already-reaped children, low child end fds, post-exit writes, and pre-output input - reap reports ECHILD (a child reaped by an embedder's own SIGCHLD handling) as a gone child, never as still-running, so the escalation path cannot signal a reused pid - Relocate a child end that lands on fd 0/1/2 to a high descriptor before login_tty: a same-fd dup2 does not clear close-on-exec, which would make the child's stdio vanish at execve when the host started with standard descriptors closed - ptyWrite drops a write once the reaper has staged the exit: there is no io thread left to send it, and mutating the finalized exit's dropped_writes would race the drain - The terminal example accepts input from spawn onward, not only after the first output batch — a shell with an empty prompt and no banner never flips to live, and gating input on it would strand every keystroke * Relocate low pty fds, wind down orphaned ptys, commit-only text, IME preedit, guarded restart - Relocate the child end AND the exec self-pipe write end above the standard descriptors before the fork: login_tty dup2's the child end onto 0/1/2, and a write end left on fd 1/2 would be clobbered, making an exec failure masquerade as a normal exit when the host began with stdio closed - Bind-time services-snapshot failure now winds down already-running ptys (bounded), not just channels, so their exit still delivers instead of stranding with no wake services - on_text delivers COMMITTED text only: an IME preedit (set_composition) or cancel routed through a focused non-text widget no longer types provisional bytes into a terminal - The targetless text path tracks the IME preedit and delivers it on commit — hosts emit an empty commit for unchanged marked text, so the composed bytes come from the buffered preedit; only committed UTF-8 reaches on_text - The terminal example restarts only a genuinely ended or failed session, never during starting/live, so a quiet shell is not duplicated onto its own occupied key * Non-blocking pty parent end, detach at retire, UTF-8-safe preedit truncation - The pty parent end is opened non-blocking and read/write surface WouldBlock: the sole io thread paces both directions through poll and never blocks inside a write when the child stops reading stdin while producing output, so stdout keeps draining and neither side deadlocks (ptyFlushOutbound writes at most one chunk per POLLOUT, leaving the rest staged) - Retirement detaches the io thread instead of joining it: retirement runs at exit delivery, inside the very dispatch a synchronous-marshal wake hook may be waiting on, and the thread's last act is that host wake — joining would recreate the deadlock ChannelWake forbids. By retirement the thread has published io_done and is past all transport/staging/pipe access, so detaching and reclaiming its fds is safe (conforming enqueue-only wakes have already returned; a violating one lingers against the process-lived header) - A truncated targetless IME preedit now cuts on a UTF-8 boundary, so an empty commit never forwards a split code point as committed text * Quiesce every pty wake at teardown, route IME to editable widgets only, chunk input - Teardown now quiesces the wake header of EVERY pty slot, idle included: the header is process-lifetime and reused, so a retired session's slow wake_fn can still be in flight when the services snapshot and platform are freed. A call still executing at the deadline is abandoned, counted, and the platform signalled to outlive it — the abandoned io-thread path accounts for its quiesce failure the same way, no longer silently - Text and IME route to a focused widget only when it is an editable text-entry widget; a focused non-text widget (a button, a list row) lets committed text fall through to the target-less on_text seam instead of dropping it - The targetless IME preedit buffer holds a full phrase-sized composition and the child's exec-failure report retries EINTR, so an interrupted one-byte report is never mistaken for a successful exec - The terminal example chunks committed text into per-write-bound pieces, so a long paste or IME commit types every byte instead of being refused whole * Free the pty header on clean teardown, honor emulator colors, bound the grid text store - A cleanly quiesced pty header is now freed at teardown: unlike a channel header (which an app thread may post to forever), the pty io thread is the sole toucher and is gone once quiesce confirms no in-flight wake, so repeatedly creating and destroying runtimes that used a pty no longer leaks ~64 KiB each; an abandoned header still leaks deliberately - The grid reads the emulator's resolved foreground, background, and cursor: the theme colors are pushed into the emulator's DEFAULTS so ghostty composes OSC 10/11/12 overrides and DECSCNM reverse-video itself, and the renderer honors an application's requested colors instead of forcing the theme - The grid degrades by whole rows when the display-list text store nears its budget (an emoji-dense screen), never blanking cells mid-frame with a swallowed allocation failure - Documented that an .exited pty terminal carries -1 only when the child was reaped outside the toolkit (an embedder installing SA_NOCLDWAIT / ignoring SIGCHLD / running its own reaper) — an already-broken parent/child contract where the real code is unrecoverable * Keep the pty header process-lived, serialize ptsname, honor OSC 4 by mask - Revert the teardown free of PtyShared: quiescing proves in_flight is zero, not that the io thread has RETURNED (it publishes its exit before calling requestHostWake, and a detached mid-life thread's late or violating wake still locks the header's mutex), so freeing risked a use-after-free. The header is now permanently retained like ChannelShared — a bounded per-slot leak that never grows during a runtime's life - ptsname's shared static buffer is resolved and copied under a process-wide spinlock, so concurrent pty spawns from independent runtimes on different threads never open each other's child end - The grid decides an ANSI slot is untouched by the emulator's override MASK, not RGB equality: a program that OSC-4-sets a slot to exactly the default RGB is honored instead of being replaced by the theme color * Stop macOS Option double-input and stage heavy graphemes whole - On macOS, Option is a compose key: Option+F commits the composed character through the text channel, so the key encoder no longer also treats Alt+printable as a chord there (which sent both the composed character and an Alt-escape to the child). Elsewhere Alt stays Meta - The grid's run staging buffer holds a full row plus a heavy grapheme cluster (8 KiB), so a cell with hundreds of combining marks stages whole; the run-break splits long runs across draw commands rather than overflowing, and any residual cut lands between code points (valid UTF-8) * Block for the exec verdict on Linux, refuse empty output records, make grid rows atomic - The exec-status probe blocks for a reliable verdict on Linux, where the report pipe is atomically close-on-exec (pipe2): a slow execve (an interpreter on a sluggish filesystem) is now awaited and correctly reported as spawn_failed rather than assumed successful. Darwin keeps the bounded poll — it has no atomic-CLOEXEC pipe, so blocking could hang on the fork-inheritance window (the documented residual every macOS terminal shares) - Replay refuses a .output pty record with a zero-length blob: the recorder journals output only for a non-empty batch, so a zero-length one is damage that would otherwise replay a synthetic empty event and diverge the fingerprint - The grid measures each row's exact text bytes and skips the row WHOLE when the display-list text store cannot hold them, and breaks a run before a cell whose grapheme would overflow the run scratch — so a heavy-grapheme row is never torn mid-way and a large cluster landing near the buffer's end keeps all its marks - The terminal-response staging buffer holds a large pipelined query burst (16 KiB); an overflowing reply is still dropped whole (never cut) and counted * Zero the signal on a cancelled pty, chunk query replies, grow the IME preedit to fit - A cancelled pty exit reports signal 0: the toolkit's own SIGKILL is what felled it, and the contract is "signal is nonzero only for a .signaled end", so the raw SIGKILL value no longer leaks into a cancelled terminal - Terminal query replies (DSR/DA/XTVERSION) flush through the chunking write path, so a batch that pipelined more than one write's worth of replies delivers every byte instead of being refused whole and leaving the child waiting - The targetless IME preedit buffer grows to fit the composition instead of truncating at a fixed size: each set_composition replaces it with the host's full string, reallocated only when a larger one arrives and freed at runtime deinit, so an unchanged commit of any length forwards the whole composition * Ignore empty pty output, clear stale preedit on OOM, reset on restart, map function keys - feedPtyOutput ignores an empty batch: the live drain only delivers non-empty output, so journaling a zero-length one would write a blob record replay refuses as damage - A failed IME preedit grow clears the buffer instead of leaving the superseded composition, so a later empty commit never inserts stale text the user has replaced - Restarting the terminal hard-resets the emulator (RIS + a fresh parser), so a shell that exited mid-escape or in a non-default mode does not corrupt the next session's key encoding or first output - The key map covers Insert and F1-F12, which carry no committed text and would otherwise be dropped before reaching the child * Bound the exec probe on both platforms, gate impossible exit records, clear selection on restart - The exec-status probe blocks up to a generous bound on every platform rather than forever on Linux: O_CLOEXEC closes an inherited pipe copy on the concurrent forker's exec, not its fork, so a forker slow to exec would delay EOF and an unbounded wait could hang the spawn thread. The bound caps that pathological wait; a timeout means our own child exec'd (an inherited writer merely delayed EOF) and is success. A slow-failing exec under the bound is still awaited and reported; only one that both blocks and fails past it misclassifies (an extreme filesystem pathology) — a self-pipe cannot distinguish that from a delayed EOF without the bound - Replay refuses a pty exit record whose code/signal contradict the delivered contract (signal nonzero iff .signaled; code -1 for every reason but .exited), so a hand-edited .cancelled with code 0 or signal 9 cannot dispatch a contract-violating event - Restarting the terminal leaves selection mode, so a restart while a selection caret is armed does not reject the new shell's typed input * Name the two ends of the pty pair parent and child - The controlling side the toolkit keeps is the parent end; the process side the spawned program adopts as its controlling terminal is the child end. Rename the transport field, the spawn locals, and the poll helper to parent/child, and rephrase every comment to match. - POSIX's own API names (posix_openpt, ptsname, grantpt/unlockpt) keep their C spellings at the call site; only the words the toolkit owns change. * Carry the session key on pty events, normalize fake exits, keep query replies lossless, and shrink the pty leak - The transpiled pty event arm now carries the app's own session key: two sessions routing one event arm were previously indistinguishable, so the arm gains a `key` field the host fills from the wire key (emitter, host bridge, and SDK type updated together). - `feedPtyExit` clamps its tuple to the event contract the live io loop guarantees — a signal only after a signaled end, a code only after an exited one — so a fake or replay can never stage an exit the recorder journals but replay's damage gate then refuses. - The terminal example feeds output in sub-slices and drains the emulator's query answers between them, so a burst of pipelined replies cannot outrun the write-back buffer and a child blocked on a DSR answer never hangs; a residual overflow surfaces on the status line instead of vanishing. - A pty session's ~64 KiB outbound ring and its per-write length ring move into a heap block freed at retire, so only a small header joins the process-lifetime leak (the channel header invariant), not the buffers; a teardown that abandons a stuck io thread still leaks the block with the thread that can reach it. * Make the pty rejection key self-contained and enforce the signaled-exit contract - A staged spawn-rejection Msg is delivered a frame after it is issued, past a frame-arena reset, so it can no longer copy the wire key into that arena: it carries the empty key (the self-contained-Msg rule the bytes field already follows), leaving the durable engine-key routing for live events untouched. - feedPtyExit now enforces the full exit contract as a biconditional — a signaled end REQUIRES a nonzero signal, not just a signal only on a signaled end — so a fake or replay feeding .signaled with signal 0 is refused loudly instead of journaling a record replay's damage gate would later reject. * Give replayed pty output the -1 code, keep the rejection key, and bound the exec probe by deadline - A replay-fed output batch now carries the -1 code sentinel the live drain delivers (the Entry default was 0), so an app that folds the event code into its model no longer diverges between a live run and its replay. - A staged spawn-rejection keeps the app's requested key: it is copied into a process-static ring (durable across the frame reset the staged Msg outlives) and referenced directly, so a duplicate-key or table-full refusal is correlated with the command that caused it instead of arriving keyless. - The exec-status probe bounds itself by a monotonic DEADLINE rather than a per-poll timeout, so signals interrupting poll (EINTR) can no longer each restart the full wait and defer the timeout indefinitely — the synchronous spawn stays bounded under a signal storm. * Make staged rejection keys grow with the batch, pace pty input to the FIFO, and reserve grid widget text - Staged spawn-rejection keys move from a fixed host ring to a growable engine store (stageLoopKey): each key gets its own stable heap buffer reclaimed when the staged stage next empties, so a Cmd.batch staging more rejections than any fixed ring holds can no longer overwrite a key still awaiting delivery and mis-correlate an exit. - The terminal example buffers typed and pasted input and drains it against the pty stdin FIFO's free space (new ptyOutboundFree query), so a paste larger than the 64 KiB FIFO no longer loses its tail when the child is not reading; it paces out as the child reads, with a per-frame flush covering a child that reads without echoing. Query replies stay direct and the flush reserves FIFO headroom for them, so bulk input never starves an answer the child blocks on. - Grid painting reserves display-list text bytes for the header and status widgets that share the per-view text store, so a grapheme-heavy viewport degrades to a few fewer rows instead of pushing the combined frame past the runtime limit and failing the whole update. - Any residual dropped input surfaces on the status line beside the reply-drop count. * Report pty write acceptance so the terminal never loses input or query replies - ptyWrite now returns whether the whole payload was accepted, since it alone knows the byte-FIFO and 256-record admission limits; a byte-capacity query could report room the record ring would still refuse, misleading a caller into treating a rejected write as sent. The misleading ptyOutboundFree query is removed. - The terminal example drains one stream-ordered outbound ring holding typed keys, pastes, AND emulator query replies, retrying any chunk ptyWrite refuses instead of dropping it. A paste larger than the FIFO paces out as the child reads; a reply refused by a full FIFO stays queued rather than being cleared, so a child blocking on a DSR answer is never stranded. A per-frame flush covers a child that reads without echoing, and a full-ring overflow is counted and shown, never silent. * Journal pty write verdicts for replay and free pty buffers through their allocating seam - A ptyWrite admission verdict is executor truth — whether the outbound FIFO and record ring had room depends on how fast the child was reading — so each one now journals (a .pty/.write record) and a replayed write returns the RECORDED verdict instead of recomputing optimistically against a fake with no child: an app that retains refused bytes takes the identical retain/remove path in both runs, pinned by a recorded session whose refused and accepted writes replay fingerprint-identical. The journal format fingerprint moves with the new record kind, refusing pre-verdict journals honestly. - Pty staging rings and outbound blocks now free through the channel_storage_allocator seam that allocated them, so a swapped-in tracking or arena allocator sees its own frees instead of a mismatched destroy against the default backing. * Scope target-less IME preedit per surface and gate command chords out of target-less text - The target-less preedit buffer now tracks its originating surface (window + view label), matching how a focused widget's editor scopes composition to the widget: only the owning surface's commit consumes the buffered bytes and only its events clear them, so a second surface's empty commit can never insert a composition typed into the first. - Target-less text_input now applies the same command-chord gate focused text widgets use (primary/command/control means shortcut, not typing; Alt stays out — Option and AltGr compose text), so Ctrl+C delivers the encoded chord alone and never a stray literal character; pinned at the runtime layer and against the terminal example's pty input. * Deliver key-carried typing target-less, count post-exit write refusals, and complete the terminal key map - Committed text carried on a key_down (hosts whose plain typing rides the key event, with no separate text event) now reaches the target-less on_text seam under the same command-chord gate the focused-widget path applies — ordinary typing was silently lost there, while specials carry no text on any host so nothing doubles with the key fallback. - A ptyWrite refused in the staged-exit window now counts into dropped_writes (the exit reads the count under the mutex at drain delivery, which has not happened yet), closing the one silent-refusal gap; only a write after the exit delivers finds no slot, the documented cancel race. - Every desktop platform now reports delete/home/end/pageup/pagedown/insert and f1-f12 as named keys (previously private-use strings or nothing), shortcuts and menu accelerators accept them, and the terminal example maps chorded punctuation and supplies the pressed character to the emulator's encoder — Ctrl+\ reaches the child as its C0 byte, Ctrl+[ as the encoder's fixterms CSI-u form, F1 as ESC O P. * Reset OSC colors on shell restart and honor widget precedence in target-less text - Session.reset now clears the palette and dynamic color overrides (OSC 4/10/11/12): the emulator's full reset leaves its color state alone, so a shell that tinted the palette and exited would otherwise color the next session; overrides drop while theme defaults stay. - The target-less committed-text fallback now honors the widget-precedence contract's structural-claim step: a key the focused widget answers as a control intent (Space pressing a focused button) no longer ALSO types its literal character through on_text, on either committed-text carrier, while unclaimed characters keep flowing to a terminal that happens to share focus with a button. * Wire the new named keys through every platform's menu accelerators - GTK translates them to accelerator names (Page_Up, F1, ...) so gtk_application_set_accels_for_action actually binds them; AppKit converts named keys to their NSMenuItem key-equivalent characters (function-key unichars, control characters) instead of passing the canonical string through, which also repairs the pre-existing arrow/escape menu bindings. - The Windows keydown path now dispatches menu-item accelerators alongside registered shortcuts — a menu item's key+modifiers emits its menu command exactly as if clicked; AppKit and GTK get this from their menu systems, and on Win32 the message loop is that system. * Settle replay feeds at the journal's end and make pty record accounting exact - Replay now verifies at the journal's end record that every journaled ptyWrite verdict was consumed and no write ran past them (the new .settle replay control): a write-count divergence changes no state when the caller is fire-and-forget, so fingerprint checkpoints alone could pass a diverged run — the settle check fails it loudly in both directions. - Output records journal the canonical scalar shape from BOTH drain sites (-1 code sentinel, no signal, no drops) and the replay damage gate refuses records claiming anything else — previously replay silently delivered defaults that differed from a damaged record's journaled fields. - A fed exit's dropped_writes now adds the refusals the fake itself counted (an oversized write against a scripted pty), so the counted-refusal contract holds without every script re-deriving the tally; under replay the slot count is always zero and the journaled count delivers verbatim. * Keep an IME sequence with the consumer it started over - A composition buffered by the target-less consumer continues target-less even when a text widget takes focus mid-sequence: routing its empty commit to the newly focused editor would resolve a composition that editor never saw and lose the composed text. Ownership ends with the sequence — the next composition belongs to the focused editor as usual. * Admit outbound terminal payloads whole or not at all - The pending-outbound ring's admission is now all-or-nothing: a payload it cannot hold whole is dropped whole and counted, never cut at the ring edge — a torn query reply or encoded key would feed the child a malformed escape sequence, which is worse than a counted loss. Reaching the drop at all means the child ignored 256 KiB of pending input, and the count stays on the status line. * Key the replay write verdicts by pty and grow the queue with the recording - The verdict queue now spills to the heap past its inline window (the pending-stage growth story): every verdict a dispatch recorded feeds before that dispatch replays, so a fixed bound would reject a valid recording whose one update wrote more times than the bound — a giant paste drained in per-write chunks is exactly that. - Each verdict carries the pty key it was recorded against, and a replayed write consumes only a matching-key verdict: the same call count and acceptance results against a DIFFERENT session is still divergent input, which a global boolean sequence would silently pass; a mismatch counts as divergence, strands its verdict, and fails the end-of-journal settle on both signals. * Canonicalize menu key case, teach the Chromium host the named keys, and state the reaping contract - Menu-item keys canonicalize to lowercase at each host's storage boundary, matching the shortcut stores that already did: key names validate case-insensitively, so a mixed-case spelling must still translate to a valid GTK accelerator name and a correct AppKit key equivalent (single characters lowercase there too — uppercase implies Shift, which the modifier mask already expresses). - The macOS Chromium host's key normalizer gains pageup/pagedown/insert and f1-f12, so shortcuts on those keys fire under that host exactly as under AppKit. - The kill fence's ownership premise is now stated at the signal site and in the recipe: the toolkit forks each pty child, is its sole reaper, and never touches SIGCHLD disposition — an embedder must not either, because POSIX offers no portable way to signal a pid once a third party (including a kernel auto-reap) can free it first; inside the contract the reaping fence is exact. * Settle the clock feed too, and retain query replies a full ring refuses - settleReplayFeeds now holds the clock feed to the write-verdict rule: a wallMs read past the journaled values (answered 0 optimistically) or a journaled value never consumed is divergence no checkpoint need see, and either fails the end-of-journal settle loudly. - A query reply the pending ring cannot take right now stays IN the emulator's buffer — uncleared, uncounted — and retries on the next output, resize, or frame nudge, so a child blocked on a DSR answer behind a full ring is never stranded by a discarded reply. Transient payloads (typed text, encoded keys) keep the counted-drop disposal since their bytes cannot outlive the dispatch; only a payload larger than the whole ring is impossible and counts immediately. * Align start-failure write verdicts, grow the reply buffer, and strip the primary alias from encoder chords - A write against a spawn whose transport failed synchronously now journals its refused verdict: the staged executor-truth terminal keeps the key occupied on the live side exactly as the parked slot does under replay, so the two verdict streams stay aligned where they previously diverged (live refused silently, replay consumed a verdict that was never recorded). - The emulator's query-reply buffer is heap-grown to fit (up to the outbound ring's own capacity): replies retained behind a full ring keep accumulating while further output feeds, instead of overflowing a fixed buffer into counted drops that could strand a blocked child; past the ceiling a reply still drops whole and counted, never cut. - The terminal encoder strips the runtime's primary-into-super fold when Ctrl raises it, so a bare Ctrl chord reaches the emulator clean and Ctrl+C delivers ETX — a stray super demoted it to a CSI-u chord no foreground shell treats as an interrupt. The GUI+Ctrl double chord encodes as plain Ctrl, the convention terminals follow. * Hold stdin order across a retained reply - A query reply retained behind a full outbound ring is older than any later keystroke, so transient input now gives the reply its retry first and refuses to jump the queue while it remains stuck (dropped counted, never reordered) — the child's stdin keeps the order a real terminal delivers: the answer it is parsing toward, then the typing. - The output arm retries retained replies right after its flush frees ring room, so the oldest bytes take that room before anything a later dispatch could enqueue. * Refuse and count fake writes after the staged exit, folding drops at delivery - A fake pty now mirrors the live admission path's staged-exit rule: a write after feedPtyExit staged the terminal refuses and counts as dropped, never a silent acceptance into a session that is already over. - The dropped tally folds the slot's count into the delivered exit AT DELIVERY (both the fed-exit queue drain and the start-failure stage), the same read-at-delivery rule the live staged-exit drain follows — so refusals landing between the feed and the drain still reach the delivered count; under replay the slot count is provably zero and the journaled count delivers verbatim. * Land start-failure write refusals in the staged exit's tally - A write refused in the synchronous start-failure window now counts onto the staged terminal entry itself: that terminal's slot was already released, so no slot tally could carry the drop to delivery, and the exit reported zero despite the refusal. The counting walk is the occupies-key predicate with the bump folded in, so the verdict journal and the delivered tally move together — the exit reports every refusal, never silence. * Settle undelivered pty feeds, serialize the spawn's descriptor window, and state the process-group kill limit - settleReplayFeeds now refuses fed pty results still queued at the journal's end: every result was journaled at its live delivery, inside an event that follows it in the stream, so a leftover means the journal was truncated past its consuming event — succeeding would silently omit a recorded delivery. Regenerating staged rejections stay exempt (the live run could end with one staged too). - The pty spawn's whole descriptor-opening window — posix_openpt through fork — now runs under one process-wide lock, and the parent end's close-on-exec fcntl is verified: Darwin ignores O_CLOEXEC on posix_openpt, so a concurrent toolkit fork landing inside another spawn's flag gap would gift its exec'd child a copy of that pty's parent end for the child's whole life. Our own forks can no longer land there; an embedder fork keeps the documented residual window the exec probe's timeout nets, and the probe still polls outside the lock. - The kill contract now states its real reach at every site: SIGKILL lands on the child's process group (the whole foreground job), and a descendant that re-grouped itself escapes — POSIX has no kill-whole-session primitive, the spawn family's documented limit. * Cover every descriptor window with the spawn lock, bound the surrendered reap, and carry an unresolved exec probe to the reap - Every toolkit fork and every Darwin flag gap now shares one lock: the wake pipe's creation and the job-spawn family's process start hold the pty spawn lock, so no toolkit child can inherit another spawn's not-yet-CLOEXEC parent end, pipe writer, or wake pipe across its exec — only embedder forks on threads the toolkit does not own keep the documented residual window. - reapEnding's final wait is bounded: a SIGKILL'd child the kernel holds in uninterruptible I/O cannot be waited out by any signal, so past a further deadline the reap is surrendered and the kill reported as the ending — the exit always reaches the app, and the eventual zombie is the bounded, documented cost that beats an io thread wedged forever. - An exec probe that times out no longer guesses success: the status pipe rides the transport (non-blocking) and is read at reap time, when a failing exec's byte — written before its _exit — is present by construction, so an executable that blocks past the bound and then fails still delivers spawn_failed, never a masqueraded normal exit. * Intern staged rejection keys for the model's life, bound failure-path reaps, and free the spawn lock from job starts - Staged rejection keys are now INTERNED and instance-lived: the TS commit walker shares non-frame pointers into the committed model rather than copying them, so a reducer that stored a rejection's key held memory the old reclaim could free mid-run. Interning bounds the storage by the app's distinct key vocabulary (repeat rejections of one key cost nothing) and the buffers outlive every model that references them. - The pty thread-start and wake-snapshot failure paths use the escalate-then-surrender reap: a child wedged inside an exec on a stalled mount cannot be waited out, and an unbounded waitpid there froze the UI loop instead of delivering spawn_failed. - The job-spawn process start no longer holds the pty spawn lock — it blocks on its own exec-status pipe, so a stalled execve would have wedged the lock and frozen the UI the moment a pty needed it. A job fork landing in a pty flag gap joins the documented bounded residual (inherited copies die at the job child's exec; a delayed exec-pipe EOF resolves through the carried reap-time verdict, never a guess). * Pin compositions to their starting editor, shorten the exec probe, and re-poll surrendered reaps - An IME sequence now routes its continuations to the text entry it STARTED in: a per-view owner is pinned when a set_composition routes and released when its commit, cancel, or a direct insertion closes the sequence — so a focus move mid-composition no longer strands the starting editor's marked text or leaks the commit through the target-less fallback, the mirror of the target-less ownership rule. - The exec probe's deadline drops to half a second: the unresolved verdict carries to the reap where a late failure still reports spawn_failed exactly, so the synchronous wait buys only the instant-failure nicety and no longer holds the loop for seconds; the PATH walk's access() probes remain the documented stalled-mount residual of the spawn-position verdict the replay contract requires. - A surrendered reap now parks its pid on a process-wide list re-polled (WNOHANG) at later spawns and at teardown, so a child that dies after its device or mount recovers is reaped then instead of zombieing for the process's life. * Decide the text claim before app rebuilds, swallow orphaned compositions, and state the Darwin pipe residual - The target-less committed-text claim is decided against the tree the input actually routed through, BEFORE the app dispatches that may rebuild it: a Space that pressed a focused button whose command removes that button still counts as claimed, so one physical keystroke can never double into a command and a literal space. - A widget-owned composition whose editor vanished mid-sequence resolves nowhere: its commit or cancel is swallowed (the newly focused editor never saw the composition, and the target-less consumer never composed it) and the stale pin clears, so a fresh composition reopens cleanly at the focused editor. - The Darwin pipe-then-fcntl window is stated fully at its site: pty forks are locked out, and the residual — a job or embedder fork landing inside it — is bounded on every axis (the copy dies at that child's exec, a wedged exec costs at most the half-second probe with the carried verdict keeping the outcome correct, a held wake pipe merely outlives its session's close until that exec). * Route converted commits to the composing editor, latch orphaned sequences whole, and keep EINTR'd surrendered pids - Every host encodes a converted commit (the composed result differs from the marked text) as cancel-then-text_input, so a cancel now holds the owner pin one event and arms a one-shot grace: the trailing text_input lands in the editor that composed it, never the newly focused one — while a plain cancel's own key_down disarms the grace before ordinary typing could inherit it. - An orphaned sequence swallows EVERY continuation: the pin holds through preedit updates (the sequence is still open, and the focused editor must not inherit them) and releases only at its commit or cancel, with an orphaned cancel arming the swallow grace so the converted commit's trailing text vanishes with the sequence instead of typing into a stranger. - reapSurrendered keeps a pid whose non-blocking poll was interrupted, rather than abandoning a live child to zombie unreaped; only a reap or ECHILD settles the entry. * Give target-less compositions the converted-commit grace too - A target-less composition's cancel now arms the same one-shot grace its widget twin got: the hosts encode a converted commit as cancel-then-text_input, so the trailing text still belongs to the surface that composed it — delivered target-less on that surface, never inserted into whichever text widget took focus mid-sequence. A plain cancel's own key_down disarms the grace before ordinary typing could inherit it. * Retry the late-failure read, orphan-check the commit grace, and die with the view - lateExecFailure retries an interrupted read instead of mistaking EINTR for success: the failure byte may be sitting right there, and reading past the signal is what keeps a delayed exec failure reporting spawn_failed. - The route-to-owner grace re-checks its owner at consumption: the cancel's own app dispatch can rebuild the tree away from the composing editor, and a dead owner converts the grace to a swallow — the converted commit's text resolves nowhere, never in whichever editor holds focus by then. - A target-less composition dies with its view: removing a surface clears any sequence it owned, so a later view reusing the same window and label never mistakes its first composition for a stale continuation that would bypass its own focused editor. * Disarm IME graces on blur and view removal, and re-anchor selection across a resize - A view losing focus disarms its cancel-to-commit grace (and the target-less twin) and releases the owner pin: hosts emit a standalone cancel when a composing view blurs, and after refocus an IM-consumed keystroke's text_input arrives before its key_down echo — a stale grace would route that fresh commit to the old editor or swallow it. - Removing a view clears the target-less grace even when the preedit length is already zero (cancellation zeroes it before arming), so a converted commit's trailing text can never leak target-less into a surface recreated under the same label. - Resizing the terminal re-anchors an armed selection at the clamped caret: reflow moves every cell, so coordinates into the old grid are dropped rather than copied, and Shift+Arrow keeps operating inside the new grid. * Share the blur-side IME hygiene across every focus path, fix the backspace key equivalent, and range-gate exit records - Every focus-mutation entry point — the pointer-driven focus move, the programmatic focusView, and the window-level clearFocusedView blur — now routes through one shared blur helper that disarms the cancel grace and releases its owner pin, closing the lifecycle class structurally instead of per path; audited all view-focus writers to exactly these three. - The macOS "backspace" menu key equivalent maps to 0x7f, the byte the physical Delete/backspace key actually emits (and the key-event normalizer maps back to "backspace") — the nominal BS control 0x08 never fired; every other named key in the table was audited against the normalizer and matches. - Pty exit records are range-gated to what waitpid's status word can produce — codes 0..255 plus the documented -1 externally-reaped sentinel, signals 1..127 — at both the replay damage gate and the feed boundary, refused as damaged rather than replayed into an event no live run can emit; pinned with hand-damaged records on both axes and both directions. * Honor the requested TERM, AltGr text, IME resolutions past claims, flush-first admission, and a clipped grid - The pty environment now replaces an inherited TERM with the spawn's requested value (the spawn declared what terminal the child is attached to; a stale TERM=dumb from the host would misdeclare it), pinned live; and the restart resets the previous session's copy feedback. - AltGr input survives both layers: the text-chord gate exempts Ctrl+Alt together (hosts represent AltGr that way, so a text event carrying both is composed text, not a shortcut), and the terminal's key mapper stops encoding Ctrl+Alt printables as chords for the same reason — the composed character rides the text channel alone. An IME resolution (an owned empty commit, or a converted commit's grace-ridden text) also bypasses the structural-claim gate: it ends a sequence the focused widget never participated in, so a button's Space claim cannot eat it. - Outbound admission flushes before refusing (stale occupancy must not drop a keystroke the drain would have made room for), with a fake full-FIFO test seam standing in for a child that stopped reading; and the grid paints under a clip to its frame with an exact bottom-row guard, so the one stale pre-resize frame degrades to a cropped grid instead of painting over the status bar and past the right edge. * Deadline the reap graces, carry claims across split events, and finish the exit accounting - reapEnding's grace windows are monotonic deadlines: usleep returns early on EINTR, so iteration counting let a signal-heavy embedder collapse the 500 ms hangup grace into an almost-immediate SIGKILL and shrink the surrender window; the surrendered list is also re-polled at every session end, so a recovered child is reaped at the next pty activity of any kind. - The committed-text claim carries across the event split: hosts that deliver a claimed key_down and its committed character as separate events (unlike the key-with-text shape) would recompute the claim against a tree the activation already rebuilt — a one-shot per-view carry keeps one physical keystroke from doubling into a command and a literal character. View blur now clears the composition owner unconditionally too: an active sequence's pin surviving a blur would redirect post-refocus typing to the stale editor. - The example's AltGr exemption narrows to the host that actually represents AltGr as Ctrl+Alt — elsewhere that combination is a genuine chord that must encode — and session end is accounted honestly: cleared outbound bytes count as dropped, retained replies drop with the dead key instead of retrying against it every frame, transport write refusals reach the status tally, and a signaled or cancelled end says so instead of rendering as exited (-1). * Never strand a pre-bind pty exit, and die the claim carry at blur - A failed bind-site snapshot publication now stages a live pty's cancelled terminal from the loop past the wind-down deadline, so the exit delivers instead of waiting on a wake that can never fire - The split-event claim carry clears at blur with the IME grace: a claimed activation that moves focus no longer swallows the refocused surface's first commit - A restarted shell starts its refused-write tally at zero instead of inheriting the dead session's drops * Give the input method its keys, and make the grid preflight measure what paints - A target-less composition now owns its surface's key_downs (candidate navigation, the trailing resolver key on hosts that run the IM filter first), so confirming a candidate with Enter never also sends CR; the buffered preedit and its graces die at the surface's blur - The split-event claim carry is keyed to the armed activation key's own literal, so a different key's committed text arriving next flows instead of feeding a stale latch - The grid's row preflight now counts only bytes painting emits (invisible cells suppress, clusters cap at the run scratch), a session exit counts retained reply bytes as loss, and each fix is pinned in both tiers * Free pty staging only on the io thread's own completion proof - Retirement defers the staging/outbound frees to the thread's io_done publish (same critical section as its staged exit) and leaks them past an abandon, so a detached thread can never observe freed blocks - The outbound flush re-checks open/generation after its unlocked write before touching the block or folding drop counts, closing the one ungated deref - A superseded thread (slot reused after an abandon) goes silent at its next loop head and never publishes reaping/io_done into the successor; pinned with a deadline-miss abandon carrying an in-flight write and a free-counting seam * Suppress IM-consumed composition keys at the GTK source, and bound the grid by what the renderer can hold - Replace the one-shot resolved-key grace with host-side suppression: GTK surfaces no key_down for a key its input method consumed while composing, so a mouse-committed candidate never costs the next genuine Enter or arrow - A styled wide character's spacer tail extends the primary cell's background run, covering both cells for SGR backgrounds and inverse video - Painting stops row-atomically before crossing a distinct-code-point budget (the glyph-atlas proxy), and the run scratch grows to the full text store so any cluster the emulator holds paints whole; each fix pinned * Let compositions own their keys before widget routing, and put real cells in the terminal's semantic surface - A live target-less composition suppresses unchorded key routing ahead of every widget pass (dismissal, focus moves, activation), so a confirming Enter never presses a freshly focused button; chorded shortcuts stay live - hostRequest's inline pre-flight now rejects keys held by live or staged ptys, matching the shared keyed-effect namespace every other issuer enforces - The grid's accessibility label is the viewport text, carrying real cell state into the session fingerprint (equal byte counters with different screens no longer verify), and a failed selection re-pin clears the emulator selection instead of leaving a stale copyable range * Settle every fed family, refuse fed overflow loudly, and keep the semantic screen honest - The end-of-journal settle now reports ANY fed result left undelivered (every family plus journaled env records), and a fed line against a full queue answers EffectQueueFull for the replay pump's drain-and-retry instead of silently converting a recorded delivery into a drop - GTK swallows a suppressed composition key's release too (no orphan key_up), and Windows lets AltGr-composed WM_CHAR text through the chord gate it raises Ctrl+Alt for - The terminal's semantic screen text is heap-exact (never truncated or cut mid-scalar), refreshes when a scroll moves the viewport, clears to unknown on a failed render instead of going stale, and a failed selection serialization surfaces as a failed copy with the selection kept * Close the key-suppression edges and keep selection, scrollback, and copy coherent - GTK dedupes suppressed composition keys under autorepeat and clears them at focus loss; Windows discards the WM_CHAR a dispatched shortcut already consumed (AltGr chords type only when no accelerator matched) - The surrendered-reap table evicts round-robin on overflow, so every wedged child keeps a retry seat instead of all forgetting behind slot zero - Scrollback chords pause while a keyboard selection is armed (viewport-relative caret vs absolute range), and a copy over a vanished emulator range reports failure instead of quietly keeping stale clipboard content * Track suppressed keys by keycode, disarm graces the suppression starved, and confirm before clearing - GTK tracks composition-suppressed keys by physical keycode (a lifted Shift cannot rename the release) and a plain cancel whose resolving key_down is suppressed emits a synthetic duplicate cancel, so the runtime's cancel-to-commit grace disarms instead of eating the next ordinary commit - The Windows shortcut latch holds across the keystroke's whole translated burst (ligature layouts post several WM_CHARs), disarmed by the message sequence at the next key transition - A terminal selection now anchors at the live cursor (never the last painted snapshot) and survives until the clipboard write CONFIRMS - a failed write keeps it standing for the retry the status promises * Paint the terminal where the user can see it - The grid region spacer is a stack, not a panel: the panel's surface chrome (fill, border, shadow, rounded corners) painted OVER the chrome-prefix grid, blanking the whole terminal - Grid text anchors its BASELINE (origin is not the glyph top), so rows land in their cells instead of one line high with row zero swallowed by the clip - The session-state indicator is a quiet muted label instead of a pill that read as a half-painted toggle, and a painted-output oracle now renders the retained frame to pixels headless and asserts the prompt's ink and the caret's cell * Build the terminal example on hosts with only the command-line tools - Disable ghostty's macOS app and xcframework artifacts in the dependency options: only the vt module is consumed, and their configure step resolves the iOS libc, which aborts without a full Xcode install * Fall back to FIONBIO when Darwin's pty parent end rejects F_SETFL - Some macOS releases return ENOTTY from fcntl(F_SETFL, O_NONBLOCK) on the posix_openpt fd while the FIONBIO ioctl succeeds; the shared setNonblock helper tries fcntl first and falls back, so a spawn no longer fails whole on those kernels * Set the pty parent non-blocking only after the replica opens - Some Darwin kernels answer ENOTTY to both fcntl(F_SETFL) and FIONBIO on a pty parent whose replica has never been opened, and accept the identical call once it has; the spawn now orders the non-blocking step after the child end opens * Scope the AltGr chord exemption to Windows and keep an armed selection on its text - Ctrl+Alt is a genuine command chord on Linux and macOS (AltGr rides its own level-3 shift there), so the text gate exempts the pair on Windows alone - a chorded key mid-composition now reaches the app instead of being swallowed - Output that scrolls the live screen rebases the keyboard selection from the emulator's absolute pins: the caret follows the selected text, and a range that leaves the viewport clears selection mode instead of desynchronizing copy from the caret * Arm the cancel-to-commit grace only while a composition owner is pinned - A duplicate cancel (the hosts' synthetic disarm for a consumed cancelling key) arrives after the grace probe already released the owner; re-arming ownerless converted the next ordinary character into a dead-owner swallow, so the arm now requires a live owner * Honor negotiated kitty modes for committed text and key releases, and admit one copy at a time - Committed single-scalar text routes through the emulator's key encoder: byte-identical raw text under legacy modes, CSI-u under a TUI's report-all; multi-scalar IME commits stay raw per the protocol - Key releases reach the encoder through an opt-in on_key phase (key_release_events), emitting kitty release events when negotiated and nothing under legacy; repeats stay presses, the one event type hosts do not distinguish - A copy while the clipboard write is in flight is a no-op, so a duplicate-key rejection can never overwrite the first copy's success; the non-Latin Ctrl-chord gap is recorded as an accepted tail pending base-layout key data in the platform vocabulary * Wheel scrollback, geometric box drawing, and a window that is just the terminal - Trackpad and wheel scrolls over the grid ride a new on_wheel app channel (the pinch channel's sibling) into whole-row scrollback with fractional accumulation; inert while a selection is armed - U+2500-259F render as geometry at exact cell bounds - lines, tees, crosses, doubles, quarter-arc rounded corners, diagonals, blocks, shades, and quadrants - with identical-piece runs merged into single bars, so borders join seamlessly where font glyphs showed seams - The window is a standard titlebar plus the grid: header and status bar removed, size readout gone, keyboard hints moved to the README * One seamless surface: hidden-inset chrome with the terminal running under the traffic lights - The window drops its title and titlebar strip (hidden_inset): the theme background fills edge-to-edge including the titlebar band, the grid's text starts below the reported chrome inset, and only the traffic lights float over the terminal * Shift the video tests' journal byte offsets past the appended pty trailer - The pty record fields append 33 bytes after the video fields in every effect payload, so the hand-patching damage helpers now offset from the pty trailer * Stitch the pty conformance fixtures after the video fixtures - Two fixture-list merge points kept both families' cases in sequence; the video entries close before the pty entries begin
Native SDK evals
An eval harness for AI-agent authoring of Native SDK apps. It formalizes the "clean-agent trial": give a fresh agent nothing but a scaffolded workspace, the native-ui skill, and a task prompt, then grade what it produced deterministically.
Per case the runner:
- Scaffolds a fresh workspace with the repo's own CLI —
zig buildat the repo root, thenzig-out/bin/native init evals/.workspaces/<case> --frontend native(--template zig-corefor the pre-existing native cases and the zig side of dual cases,--template ts-corefor the ts side) — and delivers each track's skills exactly the way a real user gets them:native skills get <name>written to.claude/skills/<name>/SKILL.md(initdoes not ship skills). The zig track getsnative-ui+zig; the ts app track getsts-core+native-ui; core-only ts-core cases getts-core. The workspace is then pre-warmed (native testonce — workspaces are zero-config, so builds go through the CLI verbs) so the agent's own builds are incremental and its wall-clock isn't spent compiling the SDK. - Runs the agent-under-test:
claude -p "<task prompt>"headless in the workspace, routed through the Vercel AI Gateway, with a per-runCLAUDE_CONFIG_DIRso no user-level memory/plugins/hooks leak in,--max-turns, a wall-clock timeout, and the fullstream-jsontranscript captured toresults/. - Grades with deterministic checks:
native testin the workspace,native markup checkon the.nativefiles, per-case file greps (e.g. "the board uses<template>"), and live automation-snapshot greps (native buildwith-Dautomation=true, launch, wait for the automation snapshot, grep it for expected roles/names). - Judges quality the deterministic checks can't see — idiomatic Model/Msg design, template factoring, test meaningfulness — with an
llm_judgecheck: a judge model called directly through the gateway scores case-specific criteria 0–10 against the task prompt and the agent's code. Advisory by default (the score is recorded and printed but never fails the case); set"advisory": falseon a case to makeminScorea gate. Skipped in--dry-run. - Reports a per-case
result.json(pass/fail per check, judge scores, durations, model, turns, cost) plus a console summary table.
Requirements
- macOS (live snapshot checks launch the app; use
--skip-liveelsewhere, or--sandboxfor the Linux lane), Zig 0.16.0, node >= 24, pnpm 10.x. - The Claude Code CLI (
claude) on PATH. - A Vercel AI Gateway API key for real runs:
export AI_GATEWAY_API_KEY="vck_..." # or VERCEL_AI_GATEWAY_API_KEY
The runner assembles the gateway env for the claude subprocess per Vercel's Claude Code guide:
ANTHROPIC_BASE_URL=https://ai-gateway.vercel.sh
ANTHROPIC_AUTH_TOKEN=$AI_GATEWAY_API_KEY
ANTHROPIC_API_KEY= # empty string on purpose: a non-empty value would win over the auth token
Models are gateway slugs. The coder (agent-under-test) defaults to anthropic/claude-sonnet-5 (override with --model or NATIVE_SDK_EVAL_MODEL); the judge defaults to anthropic/claude-opus-4.8 (override with --judge-model or NATIVE_SDK_EVAL_JUDGE_MODEL). Both tracks of a dual case always run with the SAME coder model — fairness is a property of the run, so set it once at run time.
Usage
cd evals
pnpm install
pnpm eval --list # list cases
pnpm eval --dry-run # everything except the model call (no key needed):
# scaffold + skill delivery, print env + claude argv,
# run the graders against the untouched scaffold
# (grader FAILs are expected there and exit 0)
pnpm eval templates-settings-app # one real run
pnpm eval # the whole suite
pnpm eval --model anthropic/claude-opus-4.8 templates-settings-app
pnpm eval --judge-model anthropic/claude-fable-5 templates-settings-app
pnpm eval --skip-live # skip snapshot checks (no app launch / non-macOS)
pnpm eval --track ts dual-feed-table # one track of a dual-track case (default: both)
pnpm eval --keep-workspaces # keep .workspaces/<case> around for inspection
pnpm eval --trials 5 expenses-table # 5 independent trials per case; report pass rates
pnpm eval --concurrency 3 # run up to 3 case trials in parallel (default 2 locally)
pnpm eval --sandbox # run each case in its own Vercel Sandbox microVM
pnpm eval --sandbox --dry-run # full sandbox path minus the model call
pnpm eval --sandbox --sandbox-vcpus 8 # bigger sandboxes (2048 MB RAM per vCPU)
pnpm typecheck
Cases run in parallel (log lines are prefixed [case-name]); --concurrency caps how many at once — locally the default is 2 to keep zig builds from thrashing, with --sandbox the default is 4 (each case has its own VM, but plans rate-limit vCPU allocation).
Trials
Model runs are stochastic; a single pass or fail is weak evidence. --trials <n> runs each case n times, each trial fully independent — its own scaffolded workspace (.workspaces/<case>-trial-<n>), its own agent run, its own checks and judge call — and reports per-case pass rates (e.g. 3/5), per-check pass counts, and the mean judge score. Trials share the --concurrency pool (log lines are prefixed [case-name#trial]), and with --sandbox each trial gets its own microVM.
With --trials 1 (the default) the behavior and file layout are exactly the single-run layout described above. With --trials > 1 each case directory nests per-trial results plus an aggregate:
results/<stamp>/
summary.json # array of per-case aggregates
<case>/
aggregate.json # pass rate, per-check pass counts, mean judge score, per-trial results
trial-1/result.json # exactly a single-run result.json (plus a "trial" field)
trial-1/transcript.jsonl
trial-2/...
The summary table swaps the PASS/FAIL column for a pass rate column, the checks column shows per-check pass counts (3/3 2/3 ..., s = skipped in every trial), judge is the mean score, cost/time are totals across trials, and a per-check breakdown is printed under the table. A real run exits non-zero if any trial failed.
Lanes
Every result carries a lane — where the case ran and got graded — and the summary table has a lane column:
macos-local(default): the run described above; live snapshot checks launch the app directly and require macOS.linux-sandbox(--sandbox): each case trial runs in its own isolated Vercel Sandbox microVM booted from a pre-baked Linux image, including the live checks — the app builds with-Dplatform=linux -Dweb-engine=system -Dautomation=true, launches under Xvfb, and is driven through the same automation dropbox the macOS lane uses. An engine screenshot of the app's gpu_surface is captured through the dropbox and pulled back with the results.
A check that greps a surface which exists on only one OS can declare "lanes": ["macos-local"] in eval.json; on other lanes it reports skipped (not graded on the ... lane) instead of failing, so the summary distinguishes "fails" from "not applicable on this lane". Audited 2026-07-05: none of the ten shipped cases needs a lane annotation — every snapshot pattern asserts roles/names from the SDK's own automation snapshot, and the surfaces they cover (secondary windows, gpu charts, trees, tables) are proven on Linux by tools/linux-truth/. A Linux-lane failure is therefore a real failure until a case says otherwise.
Vercel Sandbox mode
--sandbox boots each case trial from a custom image (see evals/sandbox/) that bakes the Linux GUI stack (GTK4 + WebKitGTK + Xvfb), zig, node/pnpm, the Claude Code CLI, and a pre-warmed build layer: the repo at a pinned ref with the CLI, workspace-test, and automation build graphs already compiled into fixed cache paths. Per case the runner then:
- uploads the repo working tree as a tarball and rsyncs it over the baked repo — deletions propagate, caches survive, so builds against the current tip are incremental on top of the bake;
- starts Xvfb and re-invokes this same harness inside (
pnpm eval --skip-permissions --lane linux-sandbox <case>) with the gateway env assembled exactly like a local run; - pulls the whole case results directory home —
result.json,transcript.jsonl,live-*.pngengine screenshots — before the microVM is destroyed.
--dangerously-skip-permissions is safe there because the whole VM is the throwaway. --sandbox --dry-run exercises the entire path — provisioning, refresh, scaffold, graders (which FAIL against the untouched scaffold, as designed), artifact pull — without the model call, and exits 0.
Image: build and push once with evals/sandbox/build-image.sh (needs a Docker login to the registry — the two token variants are in the script header). Rebuild when the Dockerfile changes, when zig bumps, or when the tip has drifted far enough from the baked ref that in-sandbox builds stop feeling incremental; runs stay correct without a rebuild because of the working-tree refresh. After a push the registry prepares the image for a few minutes; the runner retries image_not_ready for up to 10 minutes. --sandbox-image overrides the default reference (eval-sandbox, resolved in the linked project).
Auth (checked before any sandbox work): either an OIDC token — one-time setup in evals/:
vercel link --scope vercel-labs --project zero-native
vercel env pull .env.local # the runner auto-loads VERCEL_OIDC_TOKEN from .env.local
(the token expires ~12h; re-run vercel env pull .env.local when sandbox auth fails) — or VERCEL_TOKEN + VERCEL_TEAM_ID + VERCEL_PROJECT_ID for environments where OIDC is unavailable. Real runs additionally need AI_GATEWAY_API_KEY as usual.
Cost and limits: sandbox compute is metered (active CPU + provisioned memory) — a 4-vCPU case trial that runs 20-30 minutes lands around $0.20-0.35 plus the model tokens through the gateway. Sandboxes cap at 45 minutes wall clock on the Hobby plan (the runner's per-sandbox timeout), and vCPU allocation is rate-limited per plan, which is why the default --concurrency in sandbox mode is 4.
Real runs exit non-zero if any case fails. Workspaces live in .workspaces/ and results in results/<timestamp>/<case>/ (result.json, transcript.jsonl, the isolated claude-config/ — kept in-VM and not pulled for sandbox runs, plus live-*.png screenshots from the Linux lane); both directories are gitignored.
Permissions for the agent-under-test
By default the agent runs with --permission-mode acceptEdits plus an allowlist covering zig ..., native-sdk ..., and basic file commands — enough for unattended edit/build/test loops without granting arbitrary shell. --skip-permissions switches to --dangerously-skip-permissions; only use it if the default allowlist blocks a case, and remember the workspace is a throwaway dir but the process is not otherwise sandboxed.
In both modes the runner passes --disallowedTools deny rules for evals/cases/** and evals/results/**: the workspace references the SDK repo by path, so the harness itself is reachable from the agent's cwd, and agents exploring the repo for docs/examples were observed reading their own grading config (3/20 runs of the 2026-07-04 suite). SDK source and examples stay readable on purpose — a real user has the repo.
Cases
templates-settings-app— validates the new grammar: repeated grouped toggle sections where<template>/<use>is the natural shape, plus token style attributes (muted headers, surface cards). Checks: build+tests, markup check,<template>/<use>greps, token-attribute greps, snapshot roles.kanban-board— port of the manual builder trial; card identity must survive moving between columns (global-key).habits-tracker— port of the manual markup trial; text entry (elm-style mirror), derived/filtered lists, enum filters.expenses-table— exercises the newest grammar (every built-in component markup-expressible): an expense ledger whose natural shape istable>table-row>table-cellwith<for>rows, an exclusive category filter, and an alert-shaped empty state. The prompt describes only requirements (rows-and-columns of data, a callout, pinned display strings); the greps assert the agent reached the table grammar from the skill alone.process-monitor— exercises the effects surface: a long-running local command (a harmlesssh -ctick loop the prompt pins exactly) spawned from update through the effects channel, lines streaming into a bounded 12-line list, cancel by model-owned key, and status/counts derived from the line/exit Msgs. Greps assert.update_fxwiring,fx.spawn/fx.cancel, fake-executor tests, and the absence of hand-rolledstd.process.Child/std.Thread; the live snapshot asserts the idle state (Start/Cancel, "Status: idle", "0 lines · 0 dropped") so nothing spawns during grading. The judge scores effect-key discipline, non-blocking behavior, honest drop accounting, and derive-don't-store.release-dashboard— exercises the pipeline composites and markdown-in-markup: a release dashboard whose natural shape is<stepper>for the five-stage track (starting on "Canary"),<timeline>/<timeline-item>for the seeded event history, and a<markdown>-sourced notes panel containing a GFM table. Greps assert all three elements plus<for>events; the snapshot asserts the composite semantics ("Canary (active)"stepper labels, timeline items by title,role=gridcellmarkdown table cells, the pinned status summary). The judge scores house-style composition (no ad-hoc reimplementations) and declarative markup.settings-picker— exercises the anchored floating surface pattern: two picker rows (Theme/Accent) whose natural shape is a<select>trigger +<if>-mounted<dropdown-menu anchor=...>of<menu-item>s withon-dismiss, model-owned open state, and an exclusive-open rule. Greps assert the select/anchored-dropdown/on-dismiss/menu-item composition; the snapshot asserts the closed idle state and the pinned status line (the trigger's accessible name may be its value text or an explicitlabel— both legit, the pattern allows either).file-browser-panes— exercises split panes and the tree keymap: a resizable two-pane browser (<split value= on-resize=>with panemin-widthfloors) whose sidebar is a<tree>ofrole="treeitem"rows with model-ownedexpandedstate driving a detail pane. The snapshot assertsrole=separator/role=tree/role=treeitem, the seeded rows, and the pinned selection status. The judge is explicitly told the tree keymap is engine-provided so it never expects app-side keyboard Msgs.metrics-dashboard— exercises the markup chart element and icon-in-button: the whole view is markup (header with<button icon=...>, the<chart>with a<series values="{binding}">, status bar) and the Zig side owns only model/update logic. Greps assert<chart,icon=", and the pinnedy-min="0"; the snapshot assertsrole=chartplus the seeded summary.inspector-window— exercises model-declared secondary windows: a task list whose inspector opens as a real second OS window throughOptions.windows_fn+window_view, live-updates with the selection, reflects a user close viaon_close, and never duplicates. A negative grep rejects faking it with<dialog>/<sheet>/<drawer>.disk-status-spans— exercises inline text spans: one wrapped paragraph mixing bold, muted, and monospace runs (announcing as a single text run), with bound values interpolated inside the spans and a derived percentage summary. Greps assert the three span styles inside one enclosing<text>; the snapshot asserts the paragraph as ONE combined text run with resolved bindings.chat-composer— exercises the composer composites: an<input-group>wrapping a<textarea>plus an in-border<input-group-actions>row, an attached<button-group>cluster, a derived disabled Send, and a<for>message list. The snapshot asserts the composer textbox, the attached cluster, Send disabled on the empty draft, the seeded messages, and the live count.brand-icon-package— exercises the one-image app-icon pipeline throughnative package: the manifest sources icons from a single square PNG at the brand path, no prebuilt icon containers anywhere, and the packaged macOS bundle carries a generatedAppIcon.icnswired into its metadata.playlist-row-actions— exercises the interaction seams: row-level Enter as a list row's primary action (on-submitonlist-item, distinct from select-on-press) and the app-level key fallback (Options.on_key) for Space and the arrows when nothing is focused. Greps assert the row-level submit binding and the fallback wiring; the snapshot asserts the seeded rows, the model-driven selection, and the idle derived status.
Add a case by creating cases/<name>/eval.json (see src/types.ts for the schema). Prompts describe app requirements, never the solution — the point is to see whether a fresh agent reaches the intended grammar from the skill alone.
The TS-authoring track
Cases with "frontend": "ts-core" measure the other authoring surface: the app core written in the TypeScript subset and compiled by packages/core. The scaffold is not an app — it is src/core.ts (the case's starter/ overlay when it ships one, else a minimal counter core), a README with the check loop, and the ts-core skill delivered via native skills get ts-core. Two graders replace build/markup/snapshot:
ts_transpile— the transpiler must exit clean onsrc/core.ts: tsc-semantics typecheck, every subset rule (NS1001-NS1050), Zig emission. Failing diagnostics stay in the result as the violation evidence.ts_harness— behavioral grading: transpile the core, assemble a scratch dir with the emittedcore.zig, the rt kernel, and the case'sharness.zig, thenzig test harness.zig. The harness drives the real dispatch cycle (update→commitModelRoot→frameReset) and asserts the prompt's requirements, so the case prompt pins the Model/Msg/export contract exactly (an API spec, not a solution).
Because the grading harness compiles against the agent's code, ts-core prompts pin names; behavior stays requirements-only. The transpiler compiles pure update(model, msg): Model cores and effectful Model | [Model, Cmd<Msg>] pair-returns (the Cmd surface); the wave-2 dual-track cases below are the effects coverage.
The ts-core cases:
ts-habits-core— a fresh core from a requirements description (the TS mirror ofhabits-tracker): seeded habits, toggle-with-streak math, a trimmed draft add path, exclusive filters, derived counts.ts-expenses-filter— a feature-add to an existing subset file: extend a working expense ledger (shipped asstarter/) with an exclusiveCategory | nullfilter and derived filtered views, without breaking the existing behavior.ts-countdown-bugfix— a bug-fix in an existing subset file: a countdown core whose tick/reset/set_duration transitions drifted from the spec (unguarded ticks, no completion stop/floor, hardcoded reset); the prompt states the intended behavior, the harness isolates each drift.ts-tag-input-core— the text seam under pressure: comma-separated tag parsing with byte-level trim, empty-drop, and case-insensitive dedupe, where string methods are the reflex and NS1004 says no.
The dual-track cases (wave 2)
Wave 1's four ts-* cases measured subset compliance on toy katas and compared against Zig numbers gathered before the zig 0.16-idioms skill existed. Wave 2 replaces that comparison with an honest, contemporaneous one: six realistic asks, each ONE language-blind spec ("frontend": "app-dual") that runs on both authoring tracks — <case>@ts scaffolds a full TypeScript app (native init --frontend native --template ts-core), <case>@zig the Zig app template (--template zig-core). Identical prompt, identical shared checks, one behavioral spec asserted by two thin per-track harnesses; --track ts|zig selects a lane, the default runs both.
Grading per track: the shared checks (native test -Dplatform=null, markup check, view greps, the judge with a case rubric) plus the track's behavioral harness. On the ts track, ts_harness transpiles src/core.ts (its whole import graph) and zig tests the case's harness-ts.zig against the emitted core, the rt kernel, and harness-lib/cmdview.zig — a decoder over the Cmd/Sub wire format, so harnesses assert effects semantically ("one GET to the pinned URL", "the delay re-armed on the same key"). On the zig track, zig_harness injects the case's harness-zig.zig into the workspace as src/eval_behavior_spec.zig (a test import appended to src/main.zig, restored afterward) and runs native test, so it compiles against the agent's real Model/Msg/update and drives the SDK's deterministic fake effects executor (fx.executor = .fake, pendingSpawnAt/feedLine/feedExit/fireTimer/feedResponse/feedFileResult).
The cases — every prompt reads like a real user ask, and every effect result is fed by the harness (no network, no processes, no clocks during grading):
dual-feed-table— "fetch JSON from this API and show it in a sortable table": the HTTP effect (pinned URL, buffered GET, keyed), JSON parsing of a pinned flat schema, two sort orders derived at view time, and the five honest states (idle / loading with re-entry guarded / loaded / explicit empty / failed keeping previous rows) across non-200, transport-failure, and malformed-body deliveries.dual-notes-autosave— "add debounced autosave to this notes app" (starter provided per track, both riding the SDK text-input engines): the keyed one-shot re-arm debounce (Cmd.delay / fx.startTimer replace), byte-exact notes.tsv serialization, Save now with the pending autosave cancelled, a five-state save lifecycle, and the late-result race (a save landing after newer edits must not mark them saved).dual-pomodoro-timer— "build a pomodoro timer with a completion sound": the recurring timer surface (Sub.timer declared from the model / fx timer armed-cancelled through the channel — armed exactly while running), auto-advancing focus/rest state machine, work-only completion counting, stale-tick guards, and the audio surface at completions.dual-list-delete-fix— "this list doesn't update after delete — fix it": a realistic seeded bug (a hand-maintained visible-list cache that delete forgot) in a medium app with a markup view; the harness pins list correctness across delete/toggle/add/filter, the judge scores whether the fix is the derive-don't-store root cause or a symptom patch.dual-ledger-csv-split— "split this core into modules and add export-to-CSV in the new module": multi-file authoring (relative imports on ts,@importon zig, both grepped), byte-exact CSV quoting (commas and doubled quotes exercised by the seeds), the file-write effect, and a four-state export lifecycle.dual-sysinfo-panel— "show system info from a shell command in the UI": a collect-mode background spawn of a pinned argv, byte parsing of the output line, re-entry guarded while probing, and honest ok / failed-with-code / never-started states.
Starters (starter-ts/, starter-zig/) overlay the scaffold for the feature-add, bug-fix, and split cases — written in each track's idiom (immutable spread cores over the SDK text engine on ts; bounded buffers, canvas.TextBuffer, and Model-method bindings on zig) so neither track starts from translated code.
Fairness and methodology. Both tracks get identical prompts (language-neutral asks; the pinned contract names arms/fields in the wire spelling and states each track's idiom for the few surfaces that necessarily differ — routed err arms on ts vs payload-carried outcomes on zig). Each track gets its CURRENT skill set by the documented delivery path and nothing extra: ts → ts-core + native-ui; zig → native-ui + zig (the 0.16-idioms skill). The same coder model grades both tracks of a run. Known caveats to carry into any writeup: wave-1's Zig numbers predate the zig skill (this suite is the corrected baseline — the pre-existing native cases now also receive it, so their history splits at this commit); the behavioral harnesses are compiled contracts, so a run can fail for contract-shape reasons the prompt pins explicitly; and the two tracks' harness glue differs by necessity (wire-format decoding vs fake-executor introspection) while asserting the same spec, test for test.
Authoring metrics
pnpm metrics results/<stamp> [...] post-processes finished runs' transcripts into the agent-authoring metrics the checks cannot see, per case and per track — ts (the ts-core cases and the @ts side of dual cases) vs zig (the pre-existing native cases and the @zig side): first-pass compliance (did the agent's first compliance check after touching sources pass — the transpiler run, native check, native test, or native build on the ts track; native test / native build / native check / native markup check on the zig track), retries-to-green (failing compliance runs before the first green), teaching-error encounters (failing compliance runs that carried a teaching diagnostic — the "did the diagnostics work" round-trip count wave 2 compares across tracks), violation taxonomy (NS/TS rule IDs on the ts track; zig error lines on the zig track, with no member named 'X' bucketed by member so the 0.16-idiom class is visible) raw and per 1k generated LOC (lines written through Write/Edit to source files, .native markup included), and task success (the run's own pass verdict). Harness friction — permission-refused commands, errored compounds with no diagnostic in the output — is dropped from the event stream so it never masquerades as an authoring failure. It writes authoring-metrics.json next to each summary.json.
CI
CI only typechecks the harness (evals-typecheck in .github/workflows/ci.yml). Eval runs are local/manual — no model calls in CI.