Wes Mason
ca9a74e84a
A static observability scorer for the webapp's route entry points,
Lighthouse-style. The idea comes from evlog's `map` command, but that
tool has no Remix adapter and checks for its own logging API, so the
idea is ported rather than the tool.
It scans all 427 loader/action entry points in `apps/webapp/app/routes`
with the TypeScript compiler API and scores each against five checks:
error-classification, auth-boundary, auth-scope, request-context and
audit-trail. Current output on the real tree is **19/100** over 412
measured entry points.
```
cd internal-packages/observability-map
pnpm exec tsx src/cli.ts # terminal report
pnpm exec tsx src/cli.ts --json # machine output
pnpm exec tsx src/cli.ts api/v1/token # one entry, per-check detail
```
The two findings at the top of the fix list are real: `/auth/sso` and
`/api/v1/authorization-code` mint or exchange credentials
unauthenticated, and `/_app/orgs/:organizationSlug/settings/team`
resolves its org from a URL slug and gates each mutating branch on an
RBAC check alone, which per `apps/webapp/CLAUDE.md` is not the tenant
floor on self-hosted.
Decisions worth knowing, all with the reasoning in the README:
- The score started at 83 during development and fell to 19. Every drop
was a perverse incentive being removed, not a regression: routes were
being paid for having no error handling, two checks were reading the
same fact, suppressing a failure raised the score, and a no-op `catch
(e) { throw e }` was worth 50 points a route.
- **A mutation corpus is the tool's main defence.** 44 entries apply
semantics-preserving edits to a copy of the real route tree and assert
the score cannot rise, per route as well as globally, because a mean can
hide one route going up by taking another down. One entry runs as a live
expected failure: `try { String(0); }` with a deciding catch is a known
open hole worth 19 to 44, and it is disclosed rather than quietly
excluded.
- `audit-trail` and `request-context` are reported as headline figures
rather than one finding repeated hundreds of times. Both still count in
full where they should.
- A cohort change moves the number without anything in the codebase
getting better. Widening the sensitive cohort from 26 to 67 took the
global from 15 to 19 with no webapp change at all, so the report prints
per-check applicability and what the global would be without each one.
CI: a report-only job posts a sticky comment when a PR moves the report,
and says nothing when it does not. The package's own tests gate through
`pr_checks.yml`. The diff-scoped merge gate is still deferred until the
report has been used in anger.
524 tests plus the corpus. No runtime or dependency changes to anything
that ships.
<!-- GitButler Footer Boundary Top -->
---
This is **part 1 of 4 in a stack** made with GitButler:
- <kbd> 4 </kbd> #4485
- <kbd> 3 </kbd> #4484
- <kbd> 2 </kbd> #4483
- <kbd> 1 </kbd> #4455 👈
<!-- GitButler Footer Boundary Bottom -->
32 lines
1.6 KiB
JSON
32 lines
1.6 KiB
JSON
{
|
|
"$schema": "https://turborepo.org/schema.json",
|
|
"extends": ["//"],
|
|
"pipeline": {
|
|
// Uncacheable on purpose. This suite's real inputs live outside the package: it scans
|
|
// `apps/webapp/app`, `packages/plugins/src`, `internal-packages/rbac/src` and four files under
|
|
// `.github/workflows`. The root `test` task keys the cache on this package's own files, so a
|
|
// cached pass replayed after a route change broke the scan, which is a guard that stops
|
|
// guarding while still reading green.
|
|
//
|
|
// `inputs` was tried and rejected rather than assumed unworkable. Turbo 1.x does accept `..`
|
|
// in an input glob, and `../../apps/webapp/app/**` did bust the cache on a route change. What
|
|
// it also does is replace the default file set rather than add to it, so the same config
|
|
// silently dropped this package's own `vitest.config.ts` from the hash: editing it replayed a
|
|
// cached pass. The `$TURBO_DEFAULT$` token that adds rather than replaces is turbo 2.x only
|
|
// and matches nothing on the 1.10.3 here. Trading a stale-on-routes hole for a
|
|
// stale-on-own-config hole is not a fix, and an inputs list mirroring what the tests read is
|
|
// one more thing that drifts out of sync without saying so.
|
|
//
|
|
// Cost is about 23s per run, and no CI job pays it: the dedicated workflow calls vitest
|
|
// without turbo, and `unit-tests-internal.yml` runs cold.
|
|
//
|
|
// `it("keeps its test task out of the turbo cache")` in `src/integration.test.ts` fails if
|
|
// this is removed.
|
|
"test": {
|
|
"dependsOn": ["^build"],
|
|
"outputs": [],
|
|
"cache": false
|
|
}
|
|
}
|
|
}
|