WeHub snapshot of cb28d14c6f2c081de7a0d8729a8c816c9adef67a
CI / Test and Build (push) Failing after 1s
CI / Migrate Dev DB (push) Has been skipped
CI / Migrate DB (push) Has been skipped
CodeQL / Analyze actions (push) Has been cancelled
CodeQL / Analyze javascript-typescript (push) Has been cancelled
CI / Detect Version (push) Has been cancelled
CI / Detect Desktop Changes (push) Has been cancelled
CI / Build AMD64 (blacksmith-2vcpu-ubuntu-2404, ./docker/cron.Dockerfile, ubuntu-latest, ghcr.io/simstudioai/cron) (push) Has been cancelled
CI / Build AMD64 (blacksmith-2vcpu-ubuntu-2404, ./docker/db.Dockerfile, ECR_MIGRATIONS, ubuntu-latest, ghcr.io/simstudioai/migrations) (push) Has been cancelled
CI / Build AMD64 (blacksmith-4vcpu-ubuntu-2404, ./docker/pii.Dockerfile, ECR_PII, ubuntu-latest, ghcr.io/simstudioai/pii) (push) Has been cancelled
CI / Build AMD64 (blacksmith-4vcpu-ubuntu-2404, ./docker/realtime.Dockerfile, ECR_REALTIME, ubuntu-latest, ghcr.io/simstudioai/realtime) (push) Has been cancelled
CI / Build AMD64 (blacksmith-8vcpu-ubuntu-2404, ./docker/app.Dockerfile, ECR_APP, linux-x64-8-core, ghcr.io/simstudioai/simstudio) (push) Has been cancelled
CI / Build ARM64 (GHCR Only) (blacksmith-4vcpu-ubuntu-2404-arm, ./docker/cron.Dockerfile, ubuntu-24.04-arm, ghcr.io/simstudioai/cron) (push) Has been cancelled
CI / Build ARM64 (GHCR Only) (blacksmith-4vcpu-ubuntu-2404-arm, ./docker/db.Dockerfile, ubuntu-24.04-arm, ghcr.io/simstudioai/migrations) (push) Has been cancelled
CI / Build ARM64 (GHCR Only) (blacksmith-4vcpu-ubuntu-2404-arm, ./docker/pii.Dockerfile, ubuntu-24.04-arm, ghcr.io/simstudioai/pii) (push) Has been cancelled
CI / Build ARM64 (GHCR Only) (blacksmith-4vcpu-ubuntu-2404-arm, ./docker/realtime.Dockerfile, ubuntu-24.04-arm, ghcr.io/simstudioai/realtime) (push) Has been cancelled
CI / Build ARM64 (GHCR Only) (blacksmith-8vcpu-ubuntu-2404-arm, ./docker/app.Dockerfile, linux-arm64-8-core, ghcr.io/simstudioai/simstudio) (push) Has been cancelled
CI / Check Docs Changes (push) Has been cancelled
Publish CLI Package / publish-npm (push) Has been cancelled
Publish Python SDK / publish-pypi (push) Has been cancelled
CI / Deploy Trigger.dev (Dev) (push) Has been cancelled
Helm Chart / Lint, test, and validate chart (push) Has been cancelled
Helm Chart / Chart version bumped (push) Has been cancelled
Publish TypeScript SDK / publish-npm (push) Has been cancelled
CI / Build Dev ECR (blacksmith-8vcpu-ubuntu-2404, ./docker/app.Dockerfile, ECR_APP, linux-x64-8-core) (push) Has been cancelled
CI / Promote Images (push) Has been cancelled
CI / Create GHCR Manifests (ghcr.io/simstudioai/cron) (push) Has been cancelled
CI / Create GHCR Manifests (ghcr.io/simstudioai/migrations) (push) Has been cancelled
CI / Create GHCR Manifests (ghcr.io/simstudioai/pii) (push) Has been cancelled
CI / Create GHCR Manifests (ghcr.io/simstudioai/realtime) (push) Has been cancelled
CI / Build Dev ECR (blacksmith-2vcpu-ubuntu-2404, ./docker/db.Dockerfile, ECR_MIGRATIONS, ubuntu-latest) (push) Has been cancelled
CI / Build Dev ECR (blacksmith-4vcpu-ubuntu-2404, ./docker/pii.Dockerfile, ECR_PII, ubuntu-latest) (push) Has been cancelled
CI / Build Dev ECR (blacksmith-4vcpu-ubuntu-2404, ./docker/realtime.Dockerfile, ECR_REALTIME, ubuntu-latest) (push) Has been cancelled
CI / Create GHCR Manifests (ghcr.io/simstudioai/simstudio) (push) Has been cancelled
CI / Process Docs (push) Has been cancelled
CI / Create GitHub Release (push) Has been cancelled
CI / Check Desktop Signing Secrets (push) Has been cancelled
CI / Desktop Release (push) Has been cancelled
CI / Create Desktop Prerelease (push) Has been cancelled
CI / Desktop Prerelease Build (push) Has been cancelled
CI / Publish Desktop Prerelease (push) Has been cancelled
CI / Prune Desktop Prereleases (push) Has been cancelled
Helm Chart / Install on kind and run helm test (push) Has been cancelled
CI / Test and Build (push) Failing after 1s
CI / Migrate Dev DB (push) Has been skipped
CI / Migrate DB (push) Has been skipped
CodeQL / Analyze actions (push) Has been cancelled
CodeQL / Analyze javascript-typescript (push) Has been cancelled
CI / Detect Version (push) Has been cancelled
CI / Detect Desktop Changes (push) Has been cancelled
CI / Build AMD64 (blacksmith-2vcpu-ubuntu-2404, ./docker/cron.Dockerfile, ubuntu-latest, ghcr.io/simstudioai/cron) (push) Has been cancelled
CI / Build AMD64 (blacksmith-2vcpu-ubuntu-2404, ./docker/db.Dockerfile, ECR_MIGRATIONS, ubuntu-latest, ghcr.io/simstudioai/migrations) (push) Has been cancelled
CI / Build AMD64 (blacksmith-4vcpu-ubuntu-2404, ./docker/pii.Dockerfile, ECR_PII, ubuntu-latest, ghcr.io/simstudioai/pii) (push) Has been cancelled
CI / Build AMD64 (blacksmith-4vcpu-ubuntu-2404, ./docker/realtime.Dockerfile, ECR_REALTIME, ubuntu-latest, ghcr.io/simstudioai/realtime) (push) Has been cancelled
CI / Build AMD64 (blacksmith-8vcpu-ubuntu-2404, ./docker/app.Dockerfile, ECR_APP, linux-x64-8-core, ghcr.io/simstudioai/simstudio) (push) Has been cancelled
CI / Build ARM64 (GHCR Only) (blacksmith-4vcpu-ubuntu-2404-arm, ./docker/cron.Dockerfile, ubuntu-24.04-arm, ghcr.io/simstudioai/cron) (push) Has been cancelled
CI / Build ARM64 (GHCR Only) (blacksmith-4vcpu-ubuntu-2404-arm, ./docker/db.Dockerfile, ubuntu-24.04-arm, ghcr.io/simstudioai/migrations) (push) Has been cancelled
CI / Build ARM64 (GHCR Only) (blacksmith-4vcpu-ubuntu-2404-arm, ./docker/pii.Dockerfile, ubuntu-24.04-arm, ghcr.io/simstudioai/pii) (push) Has been cancelled
CI / Build ARM64 (GHCR Only) (blacksmith-4vcpu-ubuntu-2404-arm, ./docker/realtime.Dockerfile, ubuntu-24.04-arm, ghcr.io/simstudioai/realtime) (push) Has been cancelled
CI / Build ARM64 (GHCR Only) (blacksmith-8vcpu-ubuntu-2404-arm, ./docker/app.Dockerfile, linux-arm64-8-core, ghcr.io/simstudioai/simstudio) (push) Has been cancelled
CI / Check Docs Changes (push) Has been cancelled
Publish CLI Package / publish-npm (push) Has been cancelled
Publish Python SDK / publish-pypi (push) Has been cancelled
CI / Deploy Trigger.dev (Dev) (push) Has been cancelled
Helm Chart / Lint, test, and validate chart (push) Has been cancelled
Helm Chart / Chart version bumped (push) Has been cancelled
Publish TypeScript SDK / publish-npm (push) Has been cancelled
CI / Build Dev ECR (blacksmith-8vcpu-ubuntu-2404, ./docker/app.Dockerfile, ECR_APP, linux-x64-8-core) (push) Has been cancelled
CI / Promote Images (push) Has been cancelled
CI / Create GHCR Manifests (ghcr.io/simstudioai/cron) (push) Has been cancelled
CI / Create GHCR Manifests (ghcr.io/simstudioai/migrations) (push) Has been cancelled
CI / Create GHCR Manifests (ghcr.io/simstudioai/pii) (push) Has been cancelled
CI / Create GHCR Manifests (ghcr.io/simstudioai/realtime) (push) Has been cancelled
CI / Build Dev ECR (blacksmith-2vcpu-ubuntu-2404, ./docker/db.Dockerfile, ECR_MIGRATIONS, ubuntu-latest) (push) Has been cancelled
CI / Build Dev ECR (blacksmith-4vcpu-ubuntu-2404, ./docker/pii.Dockerfile, ECR_PII, ubuntu-latest) (push) Has been cancelled
CI / Build Dev ECR (blacksmith-4vcpu-ubuntu-2404, ./docker/realtime.Dockerfile, ECR_REALTIME, ubuntu-latest) (push) Has been cancelled
CI / Create GHCR Manifests (ghcr.io/simstudioai/simstudio) (push) Has been cancelled
CI / Process Docs (push) Has been cancelled
CI / Create GitHub Release (push) Has been cancelled
CI / Check Desktop Signing Secrets (push) Has been cancelled
CI / Desktop Release (push) Has been cancelled
CI / Create Desktop Prerelease (push) Has been cancelled
CI / Desktop Prerelease Build (push) Has been cancelled
CI / Publish Desktop Prerelease (push) Has been cancelled
CI / Prune Desktop Prereleases (push) Has been cancelled
Helm Chart / Install on kind and run helm test (push) Has been cancelled
This commit is contained in:
@@ -0,0 +1,61 @@
|
||||
---
|
||||
name: add-block-preview
|
||||
description: Gate a block's visibility — ship an unreleased block as a preview (hidden until revealed via AppConfig/env), reveal it to admins/orgs, GA it, or kill-switch a shipped block
|
||||
argument-hint: <block-type>
|
||||
---
|
||||
|
||||
# Add Block Preview Skill
|
||||
|
||||
You manage **block visibility gating** in Sim — hiding blocks from every discovery surface (toolbar, cmd+K search, copilot @-mentions, agent tool picker, mothership VFS/metadata/tools, Access Control list, public docs/catalog) while **never** gating execution of already-placed instances.
|
||||
|
||||
## The model
|
||||
|
||||
Three levers, evaluated in `apps/sim/lib/core/config/block-visibility.ts` and folded into the registry accessors (`apps/sim/blocks/registry.ts`):
|
||||
|
||||
1. **`preview: true`** on the `BlockConfig` (static, in code) — the block is default-hidden EVERYWHERE (hosted, self-hosted, dev, SSR) until revealed. Fail-closed.
|
||||
2. **The hosted `block-visibility` AppConfig document** — per-block rule keyed by the existing block type:
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"<block-type>": {
|
||||
"enabled": false, // required. true = GA (visible to everyone)
|
||||
"orgIds": ["org_..."], // optional allowlist clauses (any match reveals)
|
||||
"userIds": ["user_..."],
|
||||
"adminEnabled": true // platform admins (user.role === 'admin')
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
3. **`PREVIEW_BLOCKS` env** (comma-separated block types) — the off-AppConfig reveal path for self-hosters and local dev.
|
||||
|
||||
A revealed block that is not globally GA (`enabled !== true`, or env-revealed) renders with a **" (Preview)"** name suffix on discovery surfaces. `getBlock()` stays pure, so placed instances keep their canonical name and always execute.
|
||||
|
||||
## Lifecycle of a preview block
|
||||
|
||||
1. **Author** the block normally (`/add-block` etc.) and set `preview: true` on its `BlockConfig`. **Ship no `BlockMeta` and no docs until GA** — `check-block-registry` deliberately skips preview blocks in meta coverage, and `generate-docs` skips them at every gate.
|
||||
2. **Local dev:** set `PREVIEW_BLOCKS=<block-type>` in your env to see it (with the suffix).
|
||||
3. **Merge/deploy.** The block's code is live everywhere but visible nowhere — no AppConfig rule exists and self-hosters have no env entry.
|
||||
4. **Hosted preview:** add a rule to the `block-visibility` AppConfig document and start a deployment (no code deploy):
|
||||
- Admins only: `{ "enabled": false, "adminEnabled": true }`
|
||||
- Design-partner org: `{ "enabled": false, "orgIds": ["org_123"] }`
|
||||
- GA via config (code cleanup pending): `{ "enabled": true }` — suffix disappears everywhere within ~30s (AppConfig TTL) + client refetch.
|
||||
|
||||
Same runbook as `feature-flags`: edit the hosted document, `aws appconfig start-deployment` with the `sim-<env>-fast` strategy (see the infra README).
|
||||
5. **GA cleanup:** delete `preview: true` from the block (now visible to self-hosters on their next upgrade), add its `BlockMeta` + regen docs, and drop the AppConfig entry. For a v2 upgrade, this is also when v1 gets `hideFromToolbar: true` (the superseded-version paradigm).
|
||||
|
||||
## Kill switch (shipped blocks)
|
||||
|
||||
To pull an already-GA block from discovery surfaces on hosted (incident, deprecation): add `{ "<block-type>": { "enabled": false } }` to the document. Allowlist clauses can carve out exceptions. **Execution is NOT stopped** — workflows already using the block keep running; the kill switch only prevents new placement/discovery.
|
||||
|
||||
## Invariants (do not violate)
|
||||
|
||||
- **Execution is never gated.** The executor, serializer, drop-naming, and `isBlockTypeAccessControlExempt` resolve via pure `getBlock`. Do not add visibility checks to execution paths.
|
||||
- **Clone-not-remove:** gated blocks stay in `getAllBlocks()` output as clones with `hideFromToolbar: true` — `.find`-by-type consumers rely on this. Never filter them out.
|
||||
- **Keys are registry block types.** Never `custom_block_*` (parse drops them — custom blocks have their own enabled/disabled lifecycle).
|
||||
- **The shared hidden-predicate is `isHiddenUnder`** (`apps/sim/blocks/visibility/context.ts`). Never restate the preview/disabled rule inline at a new consumer.
|
||||
- **Process-global caches stay ungated.** `getStaticComponentFiles` (VFS) and `getExposedIntegrationTools` build the ungated universe; per-viewer filtering happens at stamp/consumer time. Never move gating into a shared builder.
|
||||
- Gating is **surface hiding, not secrecy** — the full config ships in the client JS bundle. Anything truly secret cannot be a registered block.
|
||||
|
||||
## Tests
|
||||
|
||||
Evaluation semantics: `apps/sim/lib/core/config/block-visibility.test.ts`. Registry projection: `apps/sim/blocks/visibility/visibility.test.ts`. When gating behavior changes, extend those — mock `isPlatformAdmin` for the admin clause; use the local `withAppConfig` harness.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,5 @@
|
||||
interface:
|
||||
display_name: "Add Block"
|
||||
short_description: "Build a Sim block definition"
|
||||
brand_color: "#2563EB"
|
||||
default_prompt: "Use $add-block to create or update the block for a Sim integration."
|
||||
@@ -0,0 +1,160 @@
|
||||
---
|
||||
name: add-column-type
|
||||
description: Add a new table column type to Sim — registry entry, icon, storage shape, coercion, and the behavioral hooks the grid and API read. Use when adding a value kind under `apps/sim/lib/table/column-types/`.
|
||||
argument-hint: <type-name>
|
||||
---
|
||||
|
||||
# Adding a Table Column Type
|
||||
|
||||
A column type is **one file** in `apps/sim/lib/table/column-types/` plus a registry entry. Everything that varies per type — label, icon, storage cast, coercion, validation, conversion compatibility, formatting, editor, filter operators — lives on that one object, so no consumer needs editing.
|
||||
|
||||
This was not always true: adding `currency` originally took ~40 edits across 32 `switch` arms and 26 UI branches, each of which failed **silently** when missed. The registry exists to make that impossible, so the rule is absolute: **if you find yourself adding a `case 'yourtype':` anywhere outside `column-types/`, the registry is missing a field. Add the field instead.**
|
||||
|
||||
## Hard Rule: the compiler tells you what to do
|
||||
|
||||
Do **not** hunt for places to edit. Add your type to the `ColumnType` union first and let `tsc` produce the list:
|
||||
|
||||
```bash
|
||||
cd apps/sim && bun run type-check
|
||||
```
|
||||
|
||||
You will get two errors, naming `column-types/registry.ts` and `column-types/registry.server.ts`. Register in both.
|
||||
|
||||
If your type owns metadata, adding its key to `TYPE_SPECIFIC_COLUMN_KEYS` produces two more legitimate errors — `FOREIGN_METADATA_VERB` in `validation.ts` (a `Record` over those keys) and the key's absence from `ColumnDefinition`. Those are the gate working, not sites to "fix".
|
||||
|
||||
Any error beyond those four is a site reading a hardcoded type list that should read the registry — fix that site, don't work around it.
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
apps/sim/lib/table/column-types/
|
||||
├── types.ts # ColumnTypeDefinition — the contract you implement
|
||||
├── types.server.ts # ColumnTypeServerDefinition — cell migrations only
|
||||
├── registry.ts # Record<ColumnType, …> ← client-safe, the gate
|
||||
├── registry.server.ts # Record<ColumnType, …> ← adds migrations (drizzle)
|
||||
├── index.ts # barrel + accessors (columnTypeOf, columnTypeById, …)
|
||||
└── {type}.ts # one file per type — what you write
|
||||
```
|
||||
|
||||
## Step 1: Pick the storage shape
|
||||
|
||||
Decide what a cell literally holds in `user_table_rows.data` (JSONB). This drives almost everything else:
|
||||
|
||||
| Storage | `jsonbCast` | Notes |
|
||||
|---------|-------------|-------|
|
||||
| number | `'numeric'` | Filters/sorts compare numerically. `currency` does this. |
|
||||
| ISO string | `'timestamptz'` | `date` does this. |
|
||||
| string / bool / object | `null` | Text comparison is correct. |
|
||||
|
||||
**Prefer an existing primitive over a new shape.** `currency` stores a plain number and keeps its ISO code as *display metadata* — which is why filtering, sorting, uniqueness, and CSV export all reuse the numeric paths untouched, and why re-denominating a column rewrites zero rows.
|
||||
|
||||
## Step 2: Add the icon
|
||||
|
||||
Create `packages/emcn/src/icons/type-{name}.tsx`, copying the geometry conventions of its siblings exactly:
|
||||
|
||||
```tsx
|
||||
import type { SVGProps } from 'react'
|
||||
|
||||
/**
|
||||
* Type {name} icon component - {what the glyph is} for {name} columns
|
||||
* @param props - SVG properties including className, fill, etc.
|
||||
*/
|
||||
export function Type{Pascal}(props: SVGProps<SVGSVGElement>) {
|
||||
return (
|
||||
<svg
|
||||
width='24'
|
||||
height='24'
|
||||
viewBox='-1.75 -1.5 24 24'
|
||||
fill='none'
|
||||
stroke='currentColor'
|
||||
strokeWidth='1.55'
|
||||
strokeLinecap='round'
|
||||
strokeLinejoin='round'
|
||||
xmlns='http://www.w3.org/2000/svg'
|
||||
aria-hidden='true'
|
||||
{...props}
|
||||
>
|
||||
<path d='…' />
|
||||
</svg>
|
||||
)
|
||||
}
|
||||
```
|
||||
|
||||
- `viewBox='-1.75 -1.5 24 24'` is the **`type-*` family** value, not the set-wide default. Match the family.
|
||||
- Center the glyph on the viewBox's optical center (**y = 10.5**, **x = 10.25**) — every sibling does, and a few tenths off is visible at `size-[14px]`.
|
||||
- Export alphabetically **by component name** in `packages/emcn/src/icons/index.ts`.
|
||||
|
||||
## Step 3: Write the type file
|
||||
|
||||
`apps/sim/lib/table/column-types/{name}.ts`. Copy the closest existing type and change what differs. Every field is required by the interface, so the compiler enumerates them for you — read the TSDoc in `types.ts` rather than guessing.
|
||||
|
||||
The three that are easy to get wrong:
|
||||
|
||||
- **`coerce`** is the *single* write-path implementation. The server runs it before persisting **and** the grid runs it to fill the optimistic cache. Accept every shape the value legitimately arrives in (paste, CSV, tool write), because rejecting means the cell is nulled.
|
||||
- **`isCompatibleWith`** gates type conversion and must read the value **exactly as `coerce` will**, or a conversion will pass its check and then null the cell.
|
||||
- **`ownedMetadata`** lists the `ColumnDefinition` keys your type owns. Anything you add must also be added to `TYPE_SPECIFIC_COLUMN_KEYS` in `types.ts` and given a phrase in `FOREIGN_METADATA_VERB` in `validation.ts` — both are `Record`-typed, so the compiler will tell you.
|
||||
|
||||
## Step 4: Register
|
||||
|
||||
Add the entry to `COLUMN_TYPE_REGISTRY` in `registry.ts` **and** `COLUMN_TYPE_SERVER_REGISTRY` in `registry.server.ts`.
|
||||
|
||||
`COLUMN_TYPES` is declared in `types.ts` (not derived from the registry — the registry is annotated `Record<ColumnType, …>` against it, which is the gate). `constants.ts` re-exports it, so `columnTypeSchema = z.enum(COLUMN_TYPES)` picks your type up with no edit. **Type-specific metadata does not** — see the next step.
|
||||
|
||||
## Step 5: Migrations (only if the stored bytes change)
|
||||
|
||||
If converting an existing column **to** your type must rewrite cells, add `migrateCellsTo` in `registry.server.ts`; if converting **away** must rewrite them, add `migrateCellsFrom`.
|
||||
|
||||
This is load-bearing, not cosmetic: filters and sorts apply `jsonbCast` to whatever is stored, so leaving a non-castable string behind makes **every query on that column fail** — not merely render oddly.
|
||||
|
||||
Prefer set-based SQL. When the transform genuinely needs JS (`currency`'s separator disambiguation), compute the values during the compatibility scan and pass them through `resolved`, then apply them in one batched statement.
|
||||
|
||||
## Naming Convention
|
||||
|
||||
- Type id: lowercase, singular — `currency`, not `Currency` or `currencies`
|
||||
- File: `column-types/{id}.ts`, export `const {id}ColumnType`
|
||||
- Icon: `type-{kebab}.tsx`, export `Type{Pascal}`
|
||||
|
||||
## Watch out
|
||||
|
||||
- **Import cycles.** `column-types/select.ts` imports `select-values.ts`, so `select-values.ts` must **not** import the registry — that closes a cycle and fails at module init. Inside a type's own helper module the string literal is the implementation, not a config leak.
|
||||
- **The client-safe boundary.** `registry.ts` and everything it imports must stay free of `@sim/db`, `drizzle-orm`, and `next/server` — the tables grid imports it directly. A React icon is fine (it's a component *reference*, never called server-side). Only `registry.server.ts` may touch drizzle.
|
||||
- **Don't re-export the registry from `@/lib/table`.** 44 server modules import that barrel; routing this through it pulls `@sim/emcn/icons` into all of them. Deep-import `@/lib/table/column-types`.
|
||||
- **`import.ts`'s `coerceValue` is a SECOND write path and is not opt-in.** Importing into a column of your type always hits it, and its `default` arm silently `String(value)`s — so a missing `case` stores text in a column whose `jsonbCast` is numeric, and then every filter and sort on that column errors in Postgres. Add a `case`, even though the switch compiles without one. (It is deliberately separate from the registry's `coerce`: an import wants an unparseable value to survive as its raw string so the row error can name it.)
|
||||
- **CSV inference** is an ordered heuristic in `import.ts`, deliberately not registry-driven. A new type is not inferred from a CSV unless you extend `inferColumnType` — usually you should not, since inference cannot supply configuration (an option set, a currency code).
|
||||
|
||||
## If your type owns metadata, read this
|
||||
|
||||
Registering the *type* is compiler-enforced. Registering its *metadata* is not, and that is where the remaining manual work lives. A key like `precision` has to be added in each of these, none of which will fail to compile if you forget:
|
||||
|
||||
| Where | What happens if you forget |
|
||||
|---|---|
|
||||
| `lib/table/types.ts` `ColumnDefinition` | (this one DOES fail — the ownership loop indexes it) |
|
||||
| `column-types/types.ts` `TYPE_SPECIFIC_COLUMN_KEYS` | it is never stripped on conversion, and poisons the target type |
|
||||
| `lib/api/contracts/tables.ts` — the schema slot in all three column schemas, plus `refineColumnOptions` | zod strips it at the boundary; silently never saved |
|
||||
| `columns/service.ts` `addTableColumn` param type | callers cannot pass it |
|
||||
| A metadata-only update path (`updateColumnCurrency` is the model) + a branch in both column routes + the copilot tool | changing it on an existing column is a silent 200 no-op |
|
||||
| `column-config-sidebar.tsx` | no UI to set it |
|
||||
| `table-grid.tsx` delete-column undo + `use-table-undo.ts` restore | undo silently resets it to the default |
|
||||
|
||||
`normalizeColumn`, `buildConvertedColumn`, and the undo snapshot read `TYPE_SPECIFIC_COLUMN_KEYS` generically, so those three are already zero-edit.
|
||||
|
||||
**Known gap:** the metadata-only update path is ~6 near-identical copies (service + 2 routes + copilot). A `metadataUpdate` descriptor on `ColumnTypeServerDefinition` would collapse them; until that exists, copy `currency`'s.
|
||||
|
||||
## Checklist Before Finishing
|
||||
|
||||
- [ ] Added to the `ColumnType` union in `column-types/types.ts`
|
||||
- [ ] `column-types/{id}.ts` created, every interface field filled in
|
||||
- [ ] Registered in **both** `registry.ts` and `registry.server.ts`
|
||||
- [ ] Icon added, centered on the family's optical center, exported alphabetically
|
||||
- [ ] `migrateCellsTo` / `migrateCellsFrom` added if the stored bytes change
|
||||
- [ ] New metadata keys added to `TYPE_SPECIFIC_COLUMN_KEYS` + `FOREIGN_METADATA_VERB`
|
||||
- [ ] Unit tests for `coerce` / `isCompatibleWith` round-trips, verified to fail without the code
|
||||
- [ ] Docs row added to `apps/docs/content/docs/en/tables/index.mdx`
|
||||
|
||||
## Final Validation (Required)
|
||||
|
||||
1. **`cd apps/sim && bun run type-check`** — must be clean. If any file *outside* `column-types/` errors, that file has a hardcoded type list; fix it to read the registry.
|
||||
2. **Grep for leaks** — `grep -rnE "(===|!==) '{id}'|case '{id}':" apps/sim --include='*.ts' --include='*.tsx' | grep -v column-types/`. (All three forms: a plain `!==` and a `case` are how half of `currency`'s real branches are written.) Hits are expected; judge each. A hit is fine when it mounts a specific React component or encodes a genuinely one-off behavior (`json`'s mono textarea, `date`'s timezone-aware parsing). A hit is a **leak** when it restates something the registry could answer — an icon, a label, a colour, an operator set, a cast, a coercion. Leaks get a registry field, not a new branch.
|
||||
3. **Run the suite** — `bunx vitest run lib/table 'app/workspace/[workspaceId]/tables' lib/api app/api/table app/api/v1 lib/copilot/tools/server/table`. Existing tests must pass **unchanged**; needing to edit one means you changed behavior for the other types.
|
||||
4. **`bun run lint:check`, `bun run check:api-validation`, `bun run check:client-boundary`** from the repo root.
|
||||
5. **Exercise it in the running app** on a table with one column of every type: create, edit inline / in the expanded popover / in the row modal, paste from a spreadsheet, filter, sort, convert to and from other types, export CSV, undo a column delete.
|
||||
@@ -0,0 +1,626 @@
|
||||
---
|
||||
name: add-connector
|
||||
description: Add or update a Sim knowledge base connector for syncing documents from an external source, including auth mode, config fields, pagination, document mapping, tags, and registry wiring. Use when working in `apps/sim/connectors/{service}/` or adding a new external document source.
|
||||
argument-hint: <service-name> [api-docs-url]
|
||||
---
|
||||
|
||||
# Add Connector Skill
|
||||
|
||||
You are an expert at adding knowledge base connectors to Sim. A connector syncs documents from an external source (Confluence, Google Drive, Notion, etc.) into a knowledge base.
|
||||
|
||||
## Your Task
|
||||
|
||||
When the user asks you to create a connector:
|
||||
1. Use Context7 or WebFetch to read the service's API documentation
|
||||
2. Determine the auth mode: **OAuth** (if Sim already has an OAuth provider for the service) or **API key** (if the service uses API key / Bearer token auth)
|
||||
3. Create the connector directory: a client-safe `meta.ts` (declarative metadata) plus the runtime module that spreads it
|
||||
4. Register it in BOTH the server registry and the client-safe meta registry
|
||||
|
||||
## Hard Rule: No Guessed Response Or Document Schemas
|
||||
|
||||
If the service docs do not clearly show the document list response, document fetch response, pagination shape, or metadata fields, you MUST tell the user instead of guessing.
|
||||
|
||||
- Do NOT invent document fields
|
||||
- Do NOT guess pagination cursors or next-page fields
|
||||
- Do NOT infer metadata/tag mappings from unrelated endpoints
|
||||
- Do NOT fabricate `ExternalDocument` content structure from partial docs
|
||||
|
||||
If the source schema is unknown, do one of these instead:
|
||||
1. Ask the user for sample API responses
|
||||
2. Ask the user for test credentials so you can verify live payloads
|
||||
3. Implement only the documented parts of the connector
|
||||
4. Leave the connector incomplete and explicitly say which fields remain unknown
|
||||
|
||||
## Directory Structure
|
||||
|
||||
Each connector is split into a client-safe metadata file and a server-only runtime file. This mirrors the `XBlockMeta` / `BLOCK_META_REGISTRY` split in `apps/sim/blocks` — client components (the knowledge UI) only need the metadata (icon, name, auth, config fields), so the runtime functions (which pull server-only helpers like `input-validation.server` → `undici` → `node:net`) must stay out of the client bundle.
|
||||
|
||||
Create files in `apps/sim/connectors/{service}/`:
|
||||
```
|
||||
connectors/{service}/
|
||||
├── index.ts # Barrel export (re-exports the runtime connector)
|
||||
├── meta.ts # ConnectorMeta — client-safe declarative metadata
|
||||
└── {service}.ts # ConnectorConfig — spreads the meta + adds runtime functions
|
||||
```
|
||||
|
||||
- `meta.ts` exports `{service}ConnectorMeta: ConnectorMeta`. It imports ONLY the icon from `@/components/icons`, `import type { ConnectorMeta } from '@/connectors/types'`, and any pure-data constants. It must NEVER import server/runtime code.
|
||||
- `{service}.ts` exports `{service}Connector: ConnectorConfig`. It imports the meta via `import { {service}ConnectorMeta } from '@/connectors/{service}/meta'`, spreads it as the first property, and holds the runtime functions (which may import server-only helpers like `@/lib/knowledge/documents/utils`).
|
||||
|
||||
## Authentication
|
||||
|
||||
Connectors use a discriminated union for auth config (`ConnectorAuthConfig` in `connectors/types.ts`):
|
||||
|
||||
```typescript
|
||||
type ConnectorAuthConfig =
|
||||
| { mode: 'oauth'; provider: OAuthService; requiredScopes?: string[] }
|
||||
| { mode: 'apiKey'; label?: string; placeholder?: string }
|
||||
```
|
||||
|
||||
### OAuth mode
|
||||
For services with existing OAuth providers in `apps/sim/lib/oauth/types.ts`. The `provider` must match an `OAuthService`. The modal shows a credential picker and handles token refresh automatically.
|
||||
|
||||
### API key mode
|
||||
For services that use API key / Bearer token auth. The modal shows a password input with the configured `label` and `placeholder`. The API key is encrypted at rest using AES-256-GCM and stored in a dedicated `encryptedApiKey` column on the connector record. The sync engine decrypts it automatically — connectors receive the raw access token in `listDocuments`, `getDocument`, and `validateConfig`.
|
||||
|
||||
## Connector Structure (meta.ts + runtime)
|
||||
|
||||
The declarative metadata lives in `meta.ts` (`ConnectorMeta`). The runtime functions live in `{service}.ts` (`ConnectorConfig`), which spreads the meta as its first property.
|
||||
|
||||
### `meta.ts` — client-safe metadata
|
||||
|
||||
```typescript
|
||||
import { {Service}Icon } from '@/components/icons'
|
||||
import type { ConnectorMeta } from '@/connectors/types'
|
||||
|
||||
export const {service}ConnectorMeta: ConnectorMeta = {
|
||||
id: '{service}',
|
||||
name: '{Service}',
|
||||
description: 'Sync documents from {Service} into your knowledge base',
|
||||
version: '1.0.0',
|
||||
icon: {Service}Icon,
|
||||
|
||||
auth: {
|
||||
mode: 'oauth',
|
||||
provider: '{service}', // Must match OAuthService in lib/oauth/types.ts
|
||||
requiredScopes: ['read:...'],
|
||||
},
|
||||
|
||||
configFields: [
|
||||
// Rendered dynamically by the add-connector modal UI
|
||||
// Supports 'short-input', 'dropdown', and 'selector' types — see ConfigField Types below
|
||||
],
|
||||
|
||||
// Optional: tag definitions are metadata too — declare them here
|
||||
// tagDefinitions: [ ... ],
|
||||
}
|
||||
```
|
||||
|
||||
Keep `meta.ts` free of any server/runtime import. Only the icon, the `ConnectorMeta` type, and pure-data constants belong here.
|
||||
|
||||
### `{service}.ts` — runtime (OAuth example)
|
||||
|
||||
```typescript
|
||||
import { createLogger } from '@sim/logger'
|
||||
import { fetchWithRetry } from '@/lib/knowledge/documents/utils'
|
||||
import { {service}ConnectorMeta } from '@/connectors/{service}/meta'
|
||||
import type { ConnectorConfig, ExternalDocument, ExternalDocumentList } from '@/connectors/types'
|
||||
|
||||
const logger = createLogger('{Service}Connector')
|
||||
|
||||
export const {service}Connector: ConnectorConfig = {
|
||||
...{service}ConnectorMeta,
|
||||
|
||||
listDocuments: async (accessToken, sourceConfig, cursor) => {
|
||||
// Return metadata stubs with contentDeferred: true (if per-doc content fetch needed)
|
||||
// Or full documents with content (if list API returns content inline)
|
||||
// Return { documents: ExternalDocument[], nextCursor?, hasMore }
|
||||
},
|
||||
|
||||
getDocument: async (accessToken, sourceConfig, externalId) => {
|
||||
// Fetch full content for a single document
|
||||
// Return ExternalDocument with contentDeferred: false, or null
|
||||
},
|
||||
|
||||
validateConfig: async (accessToken, sourceConfig) => {
|
||||
// Return { valid: true } or { valid: false, error: 'message' }
|
||||
},
|
||||
|
||||
// Optional: map source metadata to semantic tag keys (translated to slots by sync engine)
|
||||
mapTags: (metadata) => {
|
||||
// Return Record<string, unknown> with keys matching tagDefinitions[].id
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
Only map fields in `listDocuments`, `getDocument`, `validateConfig`, and `mapTags` when the source payload shape is documented or live-verified. If not, tell the user and stop rather than guessing.
|
||||
|
||||
### API key connector example
|
||||
|
||||
The split is identical — `auth` lives in `meta.ts`, runtime functions in `{service}.ts`.
|
||||
|
||||
```typescript
|
||||
// meta.ts
|
||||
export const {service}ConnectorMeta: ConnectorMeta = {
|
||||
id: '{service}',
|
||||
name: '{Service}',
|
||||
description: 'Sync documents from {Service} into your knowledge base',
|
||||
version: '1.0.0',
|
||||
icon: {Service}Icon,
|
||||
|
||||
auth: {
|
||||
mode: 'apiKey',
|
||||
label: 'API Key', // Shown above the input field
|
||||
placeholder: 'Enter your {Service} API key', // Input placeholder
|
||||
},
|
||||
|
||||
configFields: [ /* ... */ ],
|
||||
}
|
||||
|
||||
// {service}.ts
|
||||
export const {service}Connector: ConnectorConfig = {
|
||||
...{service}ConnectorMeta,
|
||||
listDocuments: async (accessToken, sourceConfig, cursor) => { /* ... */ },
|
||||
getDocument: async (accessToken, sourceConfig, externalId) => { /* ... */ },
|
||||
validateConfig: async (accessToken, sourceConfig) => { /* ... */ },
|
||||
}
|
||||
```
|
||||
|
||||
## ConfigField Types
|
||||
|
||||
The add-connector modal renders these automatically — no custom UI needed.
|
||||
|
||||
Three field types are supported: `short-input`, `dropdown`, and `selector`.
|
||||
|
||||
```typescript
|
||||
// Text input
|
||||
{
|
||||
id: 'domain',
|
||||
title: 'Domain',
|
||||
type: 'short-input',
|
||||
placeholder: 'yoursite.example.com',
|
||||
required: true,
|
||||
}
|
||||
|
||||
// Dropdown (static options)
|
||||
{
|
||||
id: 'contentType',
|
||||
title: 'Content Type',
|
||||
type: 'dropdown',
|
||||
required: false,
|
||||
options: [
|
||||
{ label: 'Pages only', id: 'page' },
|
||||
{ label: 'Blog posts only', id: 'blogpost' },
|
||||
{ label: 'All content', id: 'all' },
|
||||
],
|
||||
}
|
||||
```
|
||||
|
||||
## Dynamic Selectors (Canonical Pairs)
|
||||
|
||||
Use `type: 'selector'` to fetch options dynamically from the existing selector registry (`hooks/selectors/registry.ts`). Selectors are always paired with a manual fallback input using the **canonical pair** pattern — a `selector` field (basic mode) and a `short-input` field (advanced mode) linked by `canonicalParamId`.
|
||||
|
||||
The user sees a toggle button (ArrowLeftRight) to switch between the selector dropdown and manual text input. On submit, the modal resolves each canonical pair to the active mode's value, keyed by `canonicalParamId`.
|
||||
|
||||
### Rules
|
||||
|
||||
1. **Every selector field MUST have a canonical pair** — a corresponding `short-input` (or `dropdown`) field with the same `canonicalParamId` and `mode: 'advanced'`.
|
||||
2. **`required` must be set identically on both fields** in a pair. If the selector is required, the manual input must also be required.
|
||||
3. **`canonicalParamId` must match the key the connector expects in `sourceConfig`** (e.g. `baseId`, `channel`, `teamId`). The advanced field's `id` should typically match `canonicalParamId`.
|
||||
4. **`dependsOn` references the selector field's `id`**, not the `canonicalParamId`. The modal propagates dependency clearing across canonical siblings automatically — changing either field in a parent pair clears dependent children.
|
||||
|
||||
### Selector canonical pair example (Airtable base → table cascade)
|
||||
|
||||
```typescript
|
||||
configFields: [
|
||||
// Base: selector (basic) + manual (advanced)
|
||||
{
|
||||
id: 'baseSelector',
|
||||
title: 'Base',
|
||||
type: 'selector',
|
||||
selectorKey: 'airtable.bases', // Must exist in hooks/selectors/registry.ts
|
||||
canonicalParamId: 'baseId',
|
||||
mode: 'basic',
|
||||
placeholder: 'Select a base',
|
||||
required: true,
|
||||
},
|
||||
{
|
||||
id: 'baseId',
|
||||
title: 'Base ID',
|
||||
type: 'short-input',
|
||||
canonicalParamId: 'baseId',
|
||||
mode: 'advanced',
|
||||
placeholder: 'e.g. appXXXXXXXXXXXXXX',
|
||||
required: true,
|
||||
},
|
||||
// Table: selector depends on base (basic) + manual (advanced)
|
||||
{
|
||||
id: 'tableSelector',
|
||||
title: 'Table',
|
||||
type: 'selector',
|
||||
selectorKey: 'airtable.tables',
|
||||
canonicalParamId: 'tableIdOrName',
|
||||
mode: 'basic',
|
||||
dependsOn: ['baseSelector'], // References the selector field ID
|
||||
placeholder: 'Select a table',
|
||||
required: true,
|
||||
},
|
||||
{
|
||||
id: 'tableIdOrName',
|
||||
title: 'Table Name or ID',
|
||||
type: 'short-input',
|
||||
canonicalParamId: 'tableIdOrName',
|
||||
mode: 'advanced',
|
||||
placeholder: 'e.g. Tasks',
|
||||
required: true,
|
||||
},
|
||||
// Non-selector fields stay as-is
|
||||
{ id: 'maxRecords', title: 'Max Records', type: 'short-input', ... },
|
||||
]
|
||||
```
|
||||
|
||||
### Selector with domain dependency (Jira/Confluence pattern)
|
||||
|
||||
When a selector depends on a plain `short-input` field (no canonical pair), `dependsOn` references that field's `id` directly. The `domain` field's value maps to `SelectorContext.domain` automatically via `SELECTOR_CONTEXT_FIELDS`.
|
||||
|
||||
```typescript
|
||||
configFields: [
|
||||
{
|
||||
id: 'domain',
|
||||
title: 'Jira Domain',
|
||||
type: 'short-input',
|
||||
placeholder: 'yoursite.atlassian.net',
|
||||
required: true,
|
||||
},
|
||||
{
|
||||
id: 'projectSelector',
|
||||
title: 'Project',
|
||||
type: 'selector',
|
||||
selectorKey: 'jira.projects',
|
||||
canonicalParamId: 'projectKey',
|
||||
mode: 'basic',
|
||||
dependsOn: ['domain'],
|
||||
placeholder: 'Select a project',
|
||||
required: true,
|
||||
},
|
||||
{
|
||||
id: 'projectKey',
|
||||
title: 'Project Key',
|
||||
type: 'short-input',
|
||||
canonicalParamId: 'projectKey',
|
||||
mode: 'advanced',
|
||||
placeholder: 'e.g. ENG, PROJ',
|
||||
required: true,
|
||||
},
|
||||
]
|
||||
```
|
||||
|
||||
### How `dependsOn` maps to `SelectorContext`
|
||||
|
||||
The connector selector field builds a `SelectorContext` from dependency values. For the mapping to work, each dependency's `canonicalParamId` (or field `id` for non-canonical fields) must exist in `SELECTOR_CONTEXT_FIELDS` (`lib/workflows/subblocks/context.ts`):
|
||||
|
||||
```
|
||||
oauthCredential, domain, teamId, projectId, knowledgeBaseId, planId,
|
||||
siteId, collectionId, spreadsheetId, fileId, baseId, datasetId, serviceDeskId
|
||||
```
|
||||
|
||||
### Available selector keys
|
||||
|
||||
Check `hooks/selectors/types.ts` for the full `SelectorKey` union. Common ones for connectors:
|
||||
|
||||
| SelectorKey | Context Deps | Returns |
|
||||
|-------------|-------------|---------|
|
||||
| `airtable.bases` | credential | Base ID + name |
|
||||
| `airtable.tables` | credential, `baseId` | Table ID + name |
|
||||
| `slack.channels` | credential | Channel ID + name |
|
||||
| `gmail.labels` | credential | Label ID + name |
|
||||
| `google.calendar` | credential | Calendar ID + name |
|
||||
| `linear.teams` | credential | Team ID + name |
|
||||
| `linear.projects` | credential, `teamId` | Project ID + name |
|
||||
| `jira.projects` | credential, `domain` | Project key + name |
|
||||
| `confluence.spaces` | credential, `domain` | Space key + name |
|
||||
| `notion.databases` | credential | Database ID + name |
|
||||
| `asana.workspaces` | credential | Workspace GID + name |
|
||||
| `microsoft.teams` | credential | Team ID + name |
|
||||
| `microsoft.channels` | credential, `teamId` | Channel ID + name |
|
||||
| `webflow.sites` | credential | Site ID + name |
|
||||
| `outlook.folders` | credential | Folder ID + name |
|
||||
|
||||
## ExternalDocument Shape
|
||||
|
||||
Every document returned from `listDocuments`/`getDocument` must include:
|
||||
|
||||
```typescript
|
||||
{
|
||||
externalId: string // Source-specific unique ID
|
||||
title: string // Document title
|
||||
content: string // Extracted plain text (or '' if contentDeferred)
|
||||
contentDeferred?: boolean // true = content will be fetched via getDocument
|
||||
mimeType: 'text/plain' // Always text/plain (content is extracted)
|
||||
contentHash: string // Metadata-based hash for change detection
|
||||
sourceUrl?: string // Link back to original (stored on document record)
|
||||
metadata?: Record<string, unknown> // Source-specific data (fed to mapTags)
|
||||
}
|
||||
```
|
||||
|
||||
## Content Deferral (Required for file/content-download connectors)
|
||||
|
||||
**All connectors that require per-document API calls to fetch content MUST use `contentDeferred: true`.** This is the standard pattern — `listDocuments` returns lightweight metadata stubs, and content is fetched lazily by the sync engine via `getDocument` only for new/changed documents.
|
||||
|
||||
This pattern is critical for reliability: the sync engine processes documents in batches and enqueues each batch for processing immediately. If a sync times out, all previously-batched documents are already queued. Without deferral, content downloads during listing can exhaust the sync task's time budget before any documents are saved.
|
||||
|
||||
### When to use `contentDeferred: true`
|
||||
|
||||
- The service's list API does NOT return document content (only metadata)
|
||||
- Content requires a separate download/export API call per document
|
||||
- Examples: Google Drive, OneDrive, SharePoint, Dropbox, Notion, Confluence, Gmail, Obsidian, Evernote, GitHub
|
||||
|
||||
### When NOT to use `contentDeferred`
|
||||
|
||||
- The list API already returns the full content inline (e.g., Slack messages, Reddit posts, HubSpot notes)
|
||||
- No per-document API call is needed to get content
|
||||
|
||||
### Content Hash Strategy
|
||||
|
||||
Use a **metadata-based** `contentHash` — never a content-based hash. The hash must be derivable from the list response metadata alone, so the sync engine can detect changes without downloading content.
|
||||
|
||||
Good metadata hash sources:
|
||||
- `modifiedTime` / `lastModifiedDateTime` — changes when file is edited
|
||||
- Git blob SHA — unique per content version
|
||||
- API-provided content hash (e.g., Dropbox `content_hash`)
|
||||
- Version number (e.g., Confluence page version)
|
||||
|
||||
Format: `{service}:{id}:{changeIndicator}`
|
||||
|
||||
```typescript
|
||||
// Google Drive: modifiedTime changes on edit
|
||||
contentHash: `gdrive:${file.id}:${file.modifiedTime ?? ''}`
|
||||
|
||||
// GitHub: blob SHA is a content-addressable hash
|
||||
contentHash: `gitsha:${item.sha}`
|
||||
|
||||
// Dropbox: API provides content_hash
|
||||
contentHash: `dropbox:${entry.id}:${entry.content_hash ?? entry.server_modified}`
|
||||
|
||||
// Confluence: version number increments on edit
|
||||
contentHash: `confluence:${page.id}:${page.version.number}`
|
||||
```
|
||||
|
||||
**Critical invariant:** The `contentHash` MUST be identical whether produced by `listDocuments` (stub) or `getDocument` (full doc). Both should use the same stub function to guarantee this.
|
||||
|
||||
### Implementation Pattern
|
||||
|
||||
```typescript
|
||||
// 1. Create a stub function (sync, no API calls)
|
||||
function fileToStub(file: ServiceFile): ExternalDocument {
|
||||
return {
|
||||
externalId: file.id,
|
||||
title: file.name || 'Untitled',
|
||||
content: '',
|
||||
contentDeferred: true,
|
||||
mimeType: 'text/plain',
|
||||
sourceUrl: `https://service.com/file/${file.id}`,
|
||||
contentHash: `service:${file.id}:${file.modifiedTime ?? ''}`,
|
||||
metadata: { /* fields needed by mapTags */ },
|
||||
}
|
||||
}
|
||||
|
||||
// 2. listDocuments returns stubs (fast, metadata only)
|
||||
listDocuments: async (accessToken, sourceConfig, cursor) => {
|
||||
const response = await fetchWithRetry(listUrl, { ... })
|
||||
const files = (await response.json()).files
|
||||
const documents = files.map(fileToStub)
|
||||
return { documents, nextCursor, hasMore }
|
||||
}
|
||||
|
||||
// 3. getDocument fetches content and returns full doc with SAME contentHash
|
||||
getDocument: async (accessToken, sourceConfig, externalId) => {
|
||||
const metadata = await fetchWithRetry(metadataUrl, { ... })
|
||||
const file = await metadata.json()
|
||||
if (file.trashed) return null
|
||||
|
||||
try {
|
||||
const content = await fetchContent(accessToken, file)
|
||||
if (!content.trim()) return null
|
||||
const stub = fileToStub(file)
|
||||
return { ...stub, content, contentDeferred: false }
|
||||
} catch (error) {
|
||||
logger.warn(`Failed to fetch content for: ${file.name}`, { error })
|
||||
return null
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Reference Implementations
|
||||
|
||||
- **Google Drive**: `connectors/google-drive/google-drive.ts` — file download/export with `modifiedTime` hash
|
||||
- **GitHub**: `connectors/github/github.ts` — git blob SHA hash
|
||||
- **Notion**: `connectors/notion/notion.ts` — blocks API with `last_edited_time` hash
|
||||
- **Confluence**: `connectors/confluence/confluence.ts` — version number hash
|
||||
|
||||
## tagDefinitions — Declared Tag Definitions
|
||||
|
||||
Declare which tags the connector populates using semantic IDs. Shown in the add-connector modal as opt-out checkboxes.
|
||||
On connector creation, slots are **dynamically assigned** via `getNextAvailableSlot` — connectors never hardcode slot names.
|
||||
|
||||
```typescript
|
||||
tagDefinitions: [
|
||||
{ id: 'labels', displayName: 'Labels', fieldType: 'text' },
|
||||
{ id: 'version', displayName: 'Version', fieldType: 'number' },
|
||||
{ id: 'lastModified', displayName: 'Last Modified', fieldType: 'date' },
|
||||
],
|
||||
```
|
||||
|
||||
Each entry has:
|
||||
- `id`: Semantic key matching a key returned by `mapTags` (e.g. `'labels'`, `'version'`)
|
||||
- `displayName`: Human-readable name shown in the UI (e.g. "Labels", "Last Modified")
|
||||
- `fieldType`: `'text'` | `'number'` | `'date'` | `'boolean'` — determines which slot pool to draw from
|
||||
|
||||
Users can opt out of specific tags in the modal. Disabled IDs are stored in `sourceConfig.disabledTagIds`.
|
||||
The assigned mapping (`semantic id → slot`) is stored in `sourceConfig.tagSlotMapping`.
|
||||
|
||||
## `@/connectors/utils` Helpers
|
||||
|
||||
Reuse these instead of inlining the same logic (the validator enforces them):
|
||||
|
||||
- `htmlToPlainText(html)` — strip HTML to plain text before indexing `ExternalDocument.content`. Never index raw HTML.
|
||||
- `computeContentHash(content)` — stable content hash for change detection.
|
||||
- `parseTagDate(value)` — parse to a valid `Date` or `undefined` (guards Invalid Date). Use in `mapTags` for date fields.
|
||||
- `joinTagArray(value)` — validate an array and join to a comma-separated string, or `undefined`. Use in `mapTags` for array/label fields.
|
||||
- `parseMultiValue(value)` — normalize a value into a `string[]`.
|
||||
|
||||
## mapTags — Metadata to Semantic Keys
|
||||
|
||||
Maps source metadata to semantic tag keys. Required if `tagDefinitions` is set.
|
||||
The sync engine calls this automatically and translates semantic keys to actual DB slots
|
||||
using the `tagSlotMapping` stored on the connector.
|
||||
|
||||
Return keys must match the `id` values declared in `tagDefinitions`.
|
||||
|
||||
Use the `@/connectors/utils` helpers for the common transforms — don't hand-roll date/array validation:
|
||||
|
||||
```typescript
|
||||
import { joinTagArray, parseTagDate } from '@/connectors/utils'
|
||||
|
||||
mapTags: (metadata: Record<string, unknown>): Record<string, unknown> => {
|
||||
const result: Record<string, unknown> = {}
|
||||
|
||||
// joinTagArray validates the array and joins to a comma-separated string (undefined if empty)
|
||||
const labels = joinTagArray(metadata.labels)
|
||||
if (labels) result.labels = labels
|
||||
|
||||
// Validate numbers — guard against NaN
|
||||
if (metadata.version != null) {
|
||||
const num = Number(metadata.version)
|
||||
if (!Number.isNaN(num)) result.version = num
|
||||
}
|
||||
|
||||
// parseTagDate returns a valid Date or undefined (guards against Invalid Date)
|
||||
const lastModified = parseTagDate(metadata.lastModified)
|
||||
if (lastModified) result.lastModified = lastModified
|
||||
|
||||
return result
|
||||
}
|
||||
```
|
||||
|
||||
## External API Calls — Use `fetchWithRetry`
|
||||
|
||||
All external API calls must use `fetchWithRetry` from `@/lib/knowledge/documents/utils` instead of raw `fetch()`. This provides exponential backoff with retries on 429/502/503/504 errors. It returns a standard `Response` — all `.ok`, `.json()`, `.text()` checks work unchanged.
|
||||
|
||||
For `validateConfig` (user-facing, called on save), pass `VALIDATE_RETRY_OPTIONS` to cap wait time at ~7s. Background operations (`listDocuments`, `getDocument`) use the built-in defaults (5 retries, ~31s max).
|
||||
|
||||
```typescript
|
||||
import { VALIDATE_RETRY_OPTIONS, fetchWithRetry } from '@/lib/knowledge/documents/utils'
|
||||
|
||||
// Background sync — use defaults
|
||||
const response = await fetchWithRetry(url, {
|
||||
method: 'GET',
|
||||
headers: { Authorization: `Bearer ${accessToken}` },
|
||||
})
|
||||
|
||||
// validateConfig — tighter retry budget
|
||||
const response = await fetchWithRetry(url, { ... }, VALIDATE_RETRY_OPTIONS)
|
||||
```
|
||||
|
||||
## sourceUrl
|
||||
|
||||
If `ExternalDocument.sourceUrl` is set, the sync engine stores it on the document record. Always construct the full URL (not a relative path).
|
||||
|
||||
## Capped or Incomplete Listings — `syncContext.listingCapped` (REQUIRED)
|
||||
|
||||
If `listDocuments` can ever return **less than the full source set** on a non-incremental sync — a `maxItems`/`maxDocuments`-style cap, or a transient per-item error that drops a still-existing document from the listing — it MUST set `syncContext.listingCapped = true` when that happens.
|
||||
|
||||
The sync engine reconciles deletions by comparing the full listing against stored documents: anything not seen is **hard-deleted** (sync-engine.ts, gated on `!syncContext?.listingCapped`). A truncated listing without this flag deletes every real document beyond the cap. This was the single most common bug found when auditing connectors — do not omit it.
|
||||
|
||||
```typescript
|
||||
if (hitLimit && syncContext) {
|
||||
syncContext.listingCapped = true
|
||||
}
|
||||
```
|
||||
|
||||
Rules:
|
||||
- Set it when a user-configured cap truncates the listing while more documents exist
|
||||
- Set it when a thrown error caused a still-present document to be skipped during listing
|
||||
- Do NOT set it when the source is genuinely exhausted (deleted documents must still reconcile)
|
||||
- Do NOT set it for intentional scope filters (e.g. a date cutoff) — out-of-scope documents should be reconciled normally
|
||||
|
||||
## Sync Engine Behavior (Do Not Modify)
|
||||
|
||||
The sync engine (`lib/knowledge/connectors/sync-engine.ts`) is connector-agnostic. It:
|
||||
1. Calls `listDocuments` with pagination until `hasMore` is false
|
||||
2. Compares `contentHash` to detect new/changed/unchanged documents
|
||||
3. Stores `sourceUrl` and calls `mapTags` on insert/update automatically
|
||||
4. Handles soft-delete of removed documents
|
||||
5. Resolves access tokens automatically — OAuth tokens are refreshed, API keys are decrypted from the `encryptedApiKey` column
|
||||
|
||||
You never need to modify the sync engine when adding a connector.
|
||||
|
||||
## Icon
|
||||
|
||||
The `icon` field on `ConnectorConfig` is used throughout the UI — in the connector list, the add-connector modal, and as the document icon in the knowledge base table (replacing the generic file type icon for connector-sourced documents). The icon is read from `CONNECTOR_REGISTRY[connectorType].icon` at runtime — no separate icon map to maintain.
|
||||
|
||||
If the service already has an icon in `apps/sim/components/icons.tsx` (from a tool integration), reuse it. Otherwise, ask the user to provide the SVG.
|
||||
|
||||
## Registering
|
||||
|
||||
Register in BOTH registries, keeping the same alphabetical-by-id ordering in each.
|
||||
|
||||
1. **Server registry** — `apps/sim/connectors/registry.server.ts` (server-only full registry; holds full connectors with runtime functions, imported by the sync engine and knowledge API routes):
|
||||
|
||||
```typescript
|
||||
import { {service}Connector } from '@/connectors/{service}'
|
||||
|
||||
export const CONNECTOR_REGISTRY: ConnectorRegistry = {
|
||||
// ... existing connectors ...
|
||||
{service}: {service}Connector,
|
||||
}
|
||||
```
|
||||
|
||||
2. **Client-safe meta registry** — `apps/sim/connectors/registry.ts` (imports each connector's `meta.ts` only, so client components can use it without pulling server-only code; the metadata counterpart to `BLOCK_META_REGISTRY`):
|
||||
|
||||
```typescript
|
||||
import { {service}ConnectorMeta } from '@/connectors/{service}/meta'
|
||||
|
||||
export const CONNECTOR_META_REGISTRY: ConnectorMetaRegistry = {
|
||||
// ... existing connector metas ...
|
||||
{service}: {service}ConnectorMeta,
|
||||
}
|
||||
```
|
||||
|
||||
`registry.ts` exports `CONNECTOR_META_REGISTRY: ConnectorMetaRegistry` plus the helpers `getConnectorMeta(id)` and `getAllConnectorMeta()`, importing each `@/connectors/{service}/meta` directly — never the runtime module. `registry.server.ts` exports `CONNECTOR_REGISTRY: ConnectorRegistry`.
|
||||
|
||||
## Reference Implementations
|
||||
|
||||
- **OAuth + contentDeferred**: `apps/sim/connectors/google-drive/google-drive.ts` — file download with metadata-based hash, `orderBy` for deterministic pagination
|
||||
- **OAuth + contentDeferred (blocks API)**: `apps/sim/connectors/notion/notion.ts` — complex block content extraction deferred to `getDocument`
|
||||
- **OAuth + contentDeferred (git)**: `apps/sim/connectors/github/github.ts` — blob SHA hash, tree listing
|
||||
- **OAuth + inline content**: `apps/sim/connectors/confluence/confluence.ts` — multiple config field types, `mapTags`, label fetching
|
||||
- **API key**: `apps/sim/connectors/fireflies/fireflies.ts` — GraphQL API with Bearer token auth
|
||||
|
||||
## Checklist
|
||||
|
||||
- [ ] Created `connectors/{service}/meta.ts` with `{service}ConnectorMeta: ConnectorMeta` (icon, name, auth, configFields, tagDefinitions) — no server/runtime imports
|
||||
- [ ] Created `connectors/{service}/{service}.ts` with `{service}Connector: ConnectorConfig` spreading the meta + runtime functions
|
||||
- [ ] Created `connectors/{service}/index.ts` barrel export
|
||||
- [ ] **Auth configured correctly:**
|
||||
- OAuth: `auth.provider` matches an existing `OAuthService` in `lib/oauth/types.ts`
|
||||
- API key: `auth.label` and `auth.placeholder` set appropriately
|
||||
- [ ] **Selector fields configured correctly (if applicable):**
|
||||
- Every `type: 'selector'` field has a canonical pair (`short-input` or `dropdown` with same `canonicalParamId` and `mode: 'advanced'`)
|
||||
- `required` is identical on both fields in each canonical pair
|
||||
- `selectorKey` exists in `hooks/selectors/registry.ts`
|
||||
- `dependsOn` references selector field IDs (not `canonicalParamId`)
|
||||
- Dependency `canonicalParamId` values exist in `SELECTOR_CONTEXT_FIELDS`
|
||||
- [ ] `listDocuments` handles pagination with metadata-based content hashes
|
||||
- [ ] `syncContext.listingCapped = true` set whenever the listing is truncated (max-items cap or transient per-item error) — required to prevent the engine's deletion reconciliation from removing unseen documents
|
||||
- [ ] `contentDeferred: true` used if content requires per-doc API calls (file download, export, blocks fetch)
|
||||
- [ ] `contentHash` is metadata-based (not content-based) and identical between stub and `getDocument`
|
||||
- [ ] `sourceUrl` set on each ExternalDocument (full URL, not relative)
|
||||
- [ ] `metadata` includes source-specific data for tag mapping
|
||||
- [ ] `tagDefinitions` declared for each semantic key returned by `mapTags`
|
||||
- [ ] `mapTags` implemented if source has useful metadata (labels, dates, versions)
|
||||
- [ ] `validateConfig` verifies the source is accessible
|
||||
- [ ] All external API calls use `fetchWithRetry` (not raw `fetch`)
|
||||
- [ ] All optional config fields validated in `validateConfig`
|
||||
- [ ] Icon exists in `components/icons.tsx` (or asked user to provide SVG)
|
||||
- [ ] Registered the full connector in `connectors/registry.server.ts`
|
||||
- [ ] Registered the meta in `connectors/registry.ts` (same alphabetical-by-id ordering as registry.server.ts)
|
||||
@@ -0,0 +1,5 @@
|
||||
interface:
|
||||
display_name: "Add Connector"
|
||||
short_description: "Build a Sim knowledge connector"
|
||||
brand_color: "#0F766E"
|
||||
default_prompt: "Use $add-connector to add or update a Sim knowledge connector for a service."
|
||||
@@ -0,0 +1,143 @@
|
||||
---
|
||||
name: add-enrichment
|
||||
description: Add a code-defined table enrichment (registry entry) under `apps/sim/enrichments/` backed by an ordered provider cascade, ensuring every provider tool it calls has hosted-key support. Use when adding a per-row table enrichment that fills cells via existing Sim tools.
|
||||
argument-hint: <enrichment-name>
|
||||
---
|
||||
|
||||
# Adding a Table Enrichment
|
||||
|
||||
Enrichments are code-defined entries in `apps/sim/enrichments/` that run **directly per table row** (no workflow). Each enrichment declares inputs, outputs, and an ordered list of **providers**; the cascade runner tries providers in order and the first non-empty result fills the cell. Each provider calls one existing Sim tool via `executeTool`, which injects the workspace's BYOK key or a **hosted key** and bills usage automatically.
|
||||
|
||||
Because enrichments run on Sim's hosted keys by default, **every provider tool you reference must have hosted-key support** — otherwise it can only run when the workspace brings its own key. This command makes that check a required step.
|
||||
|
||||
## Overview
|
||||
|
||||
| Step | What | Where |
|
||||
|------|------|-------|
|
||||
| 1 | Pick the data-source tool(s) for each output | `tools/{service}/` + `tools/registry.ts` |
|
||||
| 2 | **Verify each tool has `hosting`; if not, run `/add-hosted-key`** | `tools/{service}/{action}.ts` |
|
||||
| 3 | Write the enrichment definition | `enrichments/{name}/{name}.ts` + `index.ts` |
|
||||
| 4 | Register it | `enrichments/registry.ts` |
|
||||
| 5 | Verify | tsc / biome / manual run |
|
||||
|
||||
## Architecture (what you're plugging into)
|
||||
|
||||
- **`enrichments/types.ts`** — `EnrichmentConfig { id, name, description, icon, inputs, outputs, providers }` and `EnrichmentProvider { id, label, toolId, buildParams, mapOutput }`. Providers are **plain data** (no `@/tools` import) so the catalog stays client-safe.
|
||||
- **`enrichments/providers.ts`** — `toolProvider(...)` (typed passthrough) plus shared input helpers: `str(v)`, `normalizeDomain(v)`, `firstNonEmpty(arr)`, `splitName(fullName)`.
|
||||
- **`enrichments/run.ts`** — the server-only cascade runner. Calls `executeTool(provider.toolId, { ...params, _context: { workspaceId } })`, accumulates hosted-key cost, returns the first non-empty mapped result. **You do not edit this** — it works for any registry entry.
|
||||
- **`enrichments/registry.ts`** — `ENRICHMENT_REGISTRY` / `ALL_ENRICHMENTS` / `getEnrichment`. Register new entries here.
|
||||
|
||||
Outputs automatically become table columns; billing, the catalog/sidebar UI, the column meta-header icon, and per-row execution all work with no extra wiring.
|
||||
|
||||
## Step 1: Pick the data-source tool(s)
|
||||
|
||||
For each output the enrichment produces, decide which existing tool provides it. Look up the service's API and the tool in `apps/sim/tools/{service}/` (e.g. `hunter_email_finder`, `pdl_person_enrich`, `pdl_company_enrich`). Confirm:
|
||||
|
||||
- The tool id is registered in `apps/sim/tools/registry.ts`.
|
||||
- Its `params` accept what you can derive from table columns (read the tool's `params`).
|
||||
- Its `outputs` / `transformResponse` actually expose the field you need (read the real output shape — don't assume).
|
||||
|
||||
Order providers **cheapest / most-likely-to-hit first**; the cascade stops at the first non-empty result. Apollo / LinkedIn are not hosted-safe (ToS) — don't use them.
|
||||
|
||||
## Step 2: Verify hosted-key support — chain to `/add-hosted-key` if missing
|
||||
|
||||
**This is the required gate.** For every tool a provider calls, open `apps/sim/tools/{service}/{action}.ts` and check for a `hosting` block:
|
||||
|
||||
```typescript
|
||||
hosting: {
|
||||
envKeyPrefix: 'SERVICE_API_KEY',
|
||||
apiKeyParam: 'apiKey',
|
||||
byokProviderId: 'service',
|
||||
pricing: { /* ... */ },
|
||||
rateLimit: { /* ... */ },
|
||||
}
|
||||
```
|
||||
|
||||
- **If `hosting` is present** — good. Note the `envKeyPrefix`; the deployment needs `{PREFIX}_COUNT` + `{PREFIX}_1..N` env vars set for the hosted key to actually resolve at runtime (ops concern, not code). If those env vars aren't set in the target environment, the provider will only run with a workspace BYOK key.
|
||||
- **If `hosting` is absent** — the tool can't use a Sim-provided key, so the enrichment would silently produce blank cells on hosted Sim. **Stop and run `/add-hosted-key <service>`** to add hosted-key support to that tool first, then come back. Do this for every provider tool that lacks it.
|
||||
|
||||
Why it matters: the cascade runner only bills (and only reads `output.cost.total`) when `executeTool` injected a hosted key, which requires the tool's `hosting` config. No `hosting` → no hosted key → the enrichment depends entirely on per-workspace BYOK.
|
||||
|
||||
## Step 3: Write the enrichment definition
|
||||
|
||||
Create `apps/sim/enrichments/{name}/{name}.ts` and a barrel `index.ts`. Mirror the existing entries (`work-email`, `phone-number`, `company-domain`, `company-info`).
|
||||
|
||||
```typescript
|
||||
import { SomeIcon } from '@sim/emcn/icons'
|
||||
import { filterUndefined } from '@sim/utils/object'
|
||||
import { normalizeDomain, splitName, str, toolProvider } from '@/enrichments/providers'
|
||||
import type { EnrichmentConfig } from '@/enrichments/types'
|
||||
|
||||
export const myEnrichment: EnrichmentConfig = {
|
||||
id: 'my-enrichment',
|
||||
name: 'My Enrichment',
|
||||
description: 'One concise sentence describing what it finds.',
|
||||
icon: SomeIcon,
|
||||
inputs: [
|
||||
// Person enrichments take a single canonical `fullName` (Clay-style);
|
||||
// split it with splitName() for tools that need first/last.
|
||||
{ id: 'fullName', name: 'Full name', type: 'string', required: true },
|
||||
{ id: 'companyDomain', name: 'Company domain', type: 'string' },
|
||||
],
|
||||
outputs: [{ id: 'value', name: 'value', type: 'string' }],
|
||||
providers: [
|
||||
toolProvider({
|
||||
id: 'provider-a',
|
||||
label: 'Provider A',
|
||||
toolId: 'service_action', // must have `hosting` (Step 2)
|
||||
buildParams: (inputs) => {
|
||||
// Return null when there aren't enough inputs → cascade skips this provider.
|
||||
const name = splitName(inputs.fullName)
|
||||
const domain = normalizeDomain(inputs.companyDomain)
|
||||
if (!name || !domain) return null
|
||||
return { domain, first_name: name.firstName, last_name: name.lastName }
|
||||
},
|
||||
mapOutput: (output) => {
|
||||
// Return { [outputId]: value } on a hit, or null to fall through.
|
||||
const value = str(output.value)
|
||||
return value ? { value } : null
|
||||
},
|
||||
}),
|
||||
// ...additional fallback providers, in priority order.
|
||||
],
|
||||
}
|
||||
```
|
||||
|
||||
```typescript
|
||||
// apps/sim/enrichments/{name}/index.ts
|
||||
export { myEnrichment } from './my-enrichment'
|
||||
```
|
||||
|
||||
Rules:
|
||||
- Keep the file **client-safe**: import only `@sim/emcn/icons`, `@sim/utils/*`, `@/enrichments/providers`, and the types. **Never import `@/tools`** here — the runner does the tool call.
|
||||
- `buildParams` returns `null` when inputs are insufficient (provider skipped). `mapOutput` returns `null`/empty for a miss (falls through). Use `filterUndefined` when assembling optional tool params; coerce numbers explicitly (don't pass `''` to number outputs).
|
||||
- Output `id`s are the keys `mapOutput` returns; output `name`s are the default column names (the user can rename them in the config).
|
||||
|
||||
## Step 4: Register it
|
||||
|
||||
In `apps/sim/enrichments/registry.ts`, import and add the entry (catalog order is registration order):
|
||||
|
||||
```typescript
|
||||
import { myEnrichment } from '@/enrichments/my-enrichment'
|
||||
|
||||
export const ENRICHMENT_REGISTRY: EnrichmentRegistry = {
|
||||
// ...existing
|
||||
[myEnrichment.id]: myEnrichment,
|
||||
}
|
||||
```
|
||||
|
||||
## Step 5: Verify
|
||||
|
||||
1. `bun run type-check` (from `apps/sim`) and `bunx biome check` on the changed files.
|
||||
2. In a table → **+ New column → Enrichments** → pick the new enrichment, map its inputs to columns, name the output column(s), Save. Confirm it appears in the catalog with its icon/description.
|
||||
3. With hosted keys (or a workspace BYOK key) configured for each provider's service, run a row and confirm the cell fills; the dev-server log shows `Enrichment hit { provider }`. A row whose providers all miss completes blank; a row where every provider errored shows an error cell.
|
||||
|
||||
## Checklist
|
||||
|
||||
- [ ] Each output mapped to a real tool field (verified against the tool's `params`/`outputs`)
|
||||
- [ ] **Every provider tool has a `hosting` block — ran `/add-hosted-key` for any that didn't**
|
||||
- [ ] Providers ordered cheapest / most-likely-first; Apollo/LinkedIn not used
|
||||
- [ ] Enrichment file is client-safe (no `@/tools` import); uses `toolProvider` + shared helpers
|
||||
- [ ] `buildParams` returns `null` on insufficient inputs; `mapOutput` returns `null` on a miss
|
||||
- [ ] Registered in `enrichments/registry.ts`
|
||||
- [ ] tsc + biome clean; created and ran the column end-to-end
|
||||
@@ -0,0 +1,5 @@
|
||||
interface:
|
||||
display_name: "Add Enrichment"
|
||||
short_description: "Build a table enrichment cascade"
|
||||
brand_color: "#16A34A"
|
||||
default_prompt: "Use $add-enrichment to add a code-defined Sim table enrichment backed by a provider cascade."
|
||||
@@ -0,0 +1,95 @@
|
||||
---
|
||||
name: add-feature-flag
|
||||
description: Add a runtime feature flag (AppConfig-backed on prod, secret fallback off-prod), global by default or optionally gated by org id, user id, or platform admin
|
||||
argument-hint: <flag-name>
|
||||
---
|
||||
|
||||
# Add Feature Flag Skill
|
||||
|
||||
You add a **runtime feature flag** to Sim that can change on prod with no redeploy (AWS AppConfig). Prefer a global on/off flag unless the rollout actually needs per-organization, per-user, or platform-admin targeting. When AppConfig isn't the source of truth, the flag falls back to a single **secret** (on/off only).
|
||||
|
||||
## When to use this vs `env-flags.ts`
|
||||
|
||||
- **Feature flag** (`@/lib/core/config/feature-flags.ts`): runtime global on/off by default, optionally scoped by `userId`/`orgId`/admin. This skill.
|
||||
- **Env flag** (`@/lib/core/config/env-flags.ts`): deploy-time capability/environment detection (`isProd`, `isHosted`, `isBillingEnabled`). A module-load boolean. **Do not add gated flags here.**
|
||||
|
||||
If the user wants a fixed per-deployment toggle, send them to `env-flags.ts` instead.
|
||||
|
||||
## The flag model
|
||||
|
||||
A flag's **gating rule lives only in the hosted AppConfig document**. It is ON for a context when any configured clause matches:
|
||||
|
||||
```ts
|
||||
interface FeatureFlagRule {
|
||||
enabled?: boolean // global default for everyone
|
||||
orgIds?: string[] // allowlisted organization ids
|
||||
userIds?: string[] // allowlisted user ids
|
||||
adminEnabled?: boolean // platform admins (user.role === 'admin')
|
||||
}
|
||||
```
|
||||
|
||||
Critically, **none of this is expressible in code** — gating (especially `adminEnabled`) can only be set through AppConfig, so no environment can grant access from a code literal. Off-AppConfig (self-hosted/OSS/local), a flag is simply on or off, derived from its fallback secret.
|
||||
|
||||
## Steps
|
||||
|
||||
1. **Confirm the granularity before editing code.** If the user has not already specified it, stop and ask:
|
||||
|
||||
> Should `<flag-name>` be a global on/off flag (recommended), or does it need rollout targeting by organization, user, and/or platform admin?
|
||||
|
||||
- Recommend **global**. Do not infer scoped gating merely because the call site already has a user or organization id.
|
||||
- If the user chooses scoped gating but does not name the dimensions, ask which of organization, user, and platform admin it needs. Wire only the selected dimensions.
|
||||
- If the user wants a fixed per-deployment toggle rather than a runtime AppConfig flag, use `env-flags.ts` instead.
|
||||
|
||||
2. **Define the flag.** Add one entry to the `FEATURE_FLAGS` registry in `apps/sim/lib/core/config/feature-flags.ts`. Each entry is the flag's whole definition — name (kebab-case key), `description`, and the `fallback` secret consulted when AppConfig isn't the source of truth (truthy ⇒ on globally):
|
||||
|
||||
```ts
|
||||
const FEATURE_FLAGS = {
|
||||
'<flag-name>': {
|
||||
description: '<what this gates>',
|
||||
fallback: '<FLAG_SECRET>',
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
`fallback` is the env/secret key (typed as `keyof typeof env`), so add `<FLAG_SECRET>` to `apps/sim/lib/core/config/env.ts` first (and the deployment's secret store) — it won't typecheck otherwise. Do **not** add org/user/admin defaults here — that gating exists only in AppConfig. Adding the entry makes `<flag-name>` a valid `FeatureFlagName`.
|
||||
|
||||
3. **Gate the call site at the chosen granularity.** For the recommended global mode, pass no context:
|
||||
|
||||
```ts
|
||||
import { isFeatureEnabled } from '@/lib/core/config/feature-flags'
|
||||
|
||||
if (await isFeatureEnabled('<flag-name>')) {
|
||||
// gated behavior
|
||||
}
|
||||
```
|
||||
|
||||
Do not fetch, resolve, or thread through user or organization context solely for a global flag.
|
||||
|
||||
For scoped rollout, pass only the dimensions the user selected. Admin status is resolved internally, so ordinary callers pass `userId`, not a role:
|
||||
|
||||
```ts
|
||||
import { isFeatureEnabled } from '@/lib/core/config/feature-flags'
|
||||
|
||||
if (await isFeatureEnabled('<flag-name>', { userId, orgId })) {
|
||||
// gated behavior
|
||||
}
|
||||
```
|
||||
|
||||
- Organization targeting uses `orgId`; user and platform-admin targeting require `userId`.
|
||||
- Missing ids are fine — a clause with no matching id is skipped; with no `userId`, the admin clause resolves to `false` without a DB read.
|
||||
- Admin routes that already know the caller is an admin may pass `{ userId, isAdmin: true }` to skip the role lookup.
|
||||
- **Client/UI flags:** resolve server-side (in a server component, route, or loader) and pass the boolean down as a prop. There is no client AppConfig.
|
||||
|
||||
4. **(Prod) configure in AppConfig.** The infra `feature-flags` profile schema is permissive, so a new flag needs **no infra change**. Operators add the flag to the hosted `feature-flags` document using `enabled` for global rollout or only the selected `orgIds`/`userIds`/`adminEnabled` clauses for scoped rollout, then start a `sim-<env>-fast` deployment (see the AppConfig runbook in the infra README — same flow as `access-control`). The fallback secret only applies when AppConfig is disabled.
|
||||
|
||||
5. **Test.** Add a case to `apps/sim/lib/core/config/feature-flags.test.ts` that matches the chosen granularity. For a global flag, exercise `isFeatureEnabled('<flag-name>')` with an AppConfig `enabled` rule and toggle the fallback secret for the off-AppConfig path. For scoped rollout, cover only the selected clauses and mock `isPlatformAdmin` when testing `adminEnabled`.
|
||||
|
||||
6. **Clean up after rollout.** When the feature ships to everyone, delete the flag's entry from `FEATURE_FLAGS`, the `<FLAG_SECRET>` env entry, the AppConfig document, the call sites, and the test. Leaving dead flags around is the main failure mode of flag systems.
|
||||
|
||||
## Notes
|
||||
|
||||
- Flag keys are `kebab-case`.
|
||||
- Never read flags via raw `fetch` or a new AppConfig client — always go through `isFeatureEnabled` / `getFeatureFlags`.
|
||||
- Never bake gating into code. The fallback is a single boolean secret; org/user/admin scoping is AppConfig-only.
|
||||
- Never add or propagate request context unless the user chose scoped rollout.
|
||||
- The admin check reads the DB **replica** (`dbReplica`) and is resolved lazily, so an admin-gated flag adds at most one cheap replica read, and only when `adminEnabled` is the deciding clause.
|
||||
@@ -0,0 +1,302 @@
|
||||
---
|
||||
name: add-hosted-key
|
||||
description: Add hosted API key support to a tool so Sim provides the key (metered and billed to the workspace) when a user has not brought their own. Use when adding a `hosting` config to a tool under `apps/sim/tools/{service}/`.
|
||||
argument-hint: <service-name>
|
||||
---
|
||||
|
||||
# Adding Hosted Key Support to a Tool
|
||||
|
||||
When a tool has hosted key support, Sim provides its own API key if the user hasn't configured one (via BYOK or env var). Usage is metered and billed to the workspace.
|
||||
|
||||
## Overview
|
||||
|
||||
| Step | What | Where |
|
||||
|------|------|-------|
|
||||
| 1 | Register BYOK provider ID | `tools/types.ts`, `lib/api/contracts/byok-keys.ts` |
|
||||
| 2 | Research the API's pricing and rate limits | API docs / pricing page (before writing any code) |
|
||||
| 3 | Add `hosting` config to the tool | `tools/{service}/{action}.ts` |
|
||||
| 4 | Hide API key field when hosted | `blocks/blocks/{service}.ts` |
|
||||
| 5 | Add to BYOK settings UI | BYOK settings component (`byok.tsx`) |
|
||||
| 6 | Summarize pricing and throttling comparison | Output to user (after all code changes) |
|
||||
|
||||
## Step 1: Register the BYOK Provider ID
|
||||
|
||||
Add the new provider to the `BYOKProviderId` union in `tools/types.ts`:
|
||||
|
||||
```typescript
|
||||
export type BYOKProviderId =
|
||||
| 'openai'
|
||||
| 'anthropic'
|
||||
// ...existing providers
|
||||
| 'your_service'
|
||||
```
|
||||
|
||||
Then add the same provider id to the `byokProviderIdSchema` enum in `lib/api/contracts/byok-keys.ts` (this is what the byok-keys route validates against):
|
||||
|
||||
```typescript
|
||||
export const byokProviderIdSchema = z.enum([
|
||||
'openai',
|
||||
'anthropic',
|
||||
// ...existing providers
|
||||
'your_service',
|
||||
])
|
||||
```
|
||||
|
||||
## Step 2: Research the API's Pricing Model and Rate Limits
|
||||
|
||||
**Before writing any `getCost` or `rateLimit` code**, look up the service's official documentation for both pricing and rate limits. You need to understand:
|
||||
|
||||
### Pricing
|
||||
|
||||
1. **How the API charges** — per request, per credit, per token, per step, per minute, etc.
|
||||
2. **Whether the API reports cost in its response** — look for fields like `creditsUsed`, `costDollars`, `tokensUsed`, or similar in the response body or headers
|
||||
3. **Whether cost varies by endpoint/options** — some APIs charge more for certain features (e.g., Firecrawl charges 1 credit/page base but +4 for JSON format, +4 for enhanced mode)
|
||||
4. **The dollar-per-unit rate** — what each credit/token/unit costs in dollars on our plan
|
||||
|
||||
### Rate Limits
|
||||
|
||||
1. **What rate limits the API enforces** — requests per minute/second, tokens per minute, concurrent requests, etc.
|
||||
2. **Whether limits vary by plan tier** — free vs paid vs enterprise often have different ceilings
|
||||
3. **Whether limits are per-key or per-account** — determines whether adding more hosted keys actually increases total throughput
|
||||
4. **What the API returns when rate limited** — HTTP 429, `Retry-After` header, error body format, etc.
|
||||
5. **Whether there are multiple dimensions** — some APIs limit both requests/min AND tokens/min independently
|
||||
|
||||
Search the API's docs/pricing page (use WebSearch/WebFetch). Capture the pricing model as a comment in `getCost` so future maintainers know the source of truth.
|
||||
|
||||
### Setting Our Rate Limits
|
||||
|
||||
Our rate limiter (`lib/core/rate-limiter/hosted-key/`) uses a token-bucket algorithm applied **per billing actor** (workspace). It supports two modes:
|
||||
|
||||
- **`per_request`** — simple; just `requestsPerMinute`. Good when the API charges flat per-request or cost doesn't vary much.
|
||||
- **`custom`** — `requestsPerMinute` plus additional `dimensions` (e.g., `tokens`, `search_units`). Each dimension has its own `limitPerMinute` and an `extractUsage` function that reads actual usage from the response. Use when the API charges on a variable metric (tokens, credits) and you want to cap that metric too.
|
||||
|
||||
When choosing values for `requestsPerMinute` and any dimension limits:
|
||||
|
||||
- **Stay well below the API's per-key limit** — our keys are shared across all workspaces. If the API allows 60 RPM per key and we have 3 keys, the global ceiling is ~180 RPM. Set the per-workspace limit low enough (e.g., 20-60 RPM) that many workspaces can coexist without collectively hitting the API's ceiling.
|
||||
- **Account for key pooling** — our round-robin distributes requests across `N` hosted keys, so the effective API-side rate per key is `(total requests) / N`. But per-workspace limits are enforced *before* key selection, so they apply regardless of key count.
|
||||
- **Prefer conservative defaults** — it's easy to raise limits later but hard to claw back after users depend on high throughput.
|
||||
|
||||
## Step 3: Add `hosting` Config to the Tool
|
||||
|
||||
Add a `hosting` object to the tool's `ToolConfig`. This tells the execution layer how to acquire hosted keys, calculate cost, and rate-limit.
|
||||
|
||||
```typescript
|
||||
hosting: {
|
||||
envKeyPrefix: 'YOUR_SERVICE_API_KEY',
|
||||
apiKeyParam: 'apiKey',
|
||||
byokProviderId: 'your_service',
|
||||
pricing: {
|
||||
type: 'custom',
|
||||
getCost: (_params, output) => {
|
||||
if (output.creditsUsed == null) {
|
||||
throw new Error('Response missing creditsUsed field')
|
||||
}
|
||||
const creditsUsed = output.creditsUsed as number
|
||||
const cost = creditsUsed * 0.001 // dollars per credit
|
||||
return { cost, metadata: { creditsUsed } }
|
||||
},
|
||||
},
|
||||
rateLimit: {
|
||||
mode: 'per_request',
|
||||
requestsPerMinute: 100,
|
||||
},
|
||||
},
|
||||
```
|
||||
|
||||
### Hosted Key Env Var Convention
|
||||
|
||||
Keys use a numbered naming pattern driven by a count env var:
|
||||
|
||||
```
|
||||
YOUR_SERVICE_API_KEY_COUNT=3
|
||||
YOUR_SERVICE_API_KEY_1=sk-...
|
||||
YOUR_SERVICE_API_KEY_2=sk-...
|
||||
YOUR_SERVICE_API_KEY_3=sk-...
|
||||
```
|
||||
|
||||
The `envKeyPrefix` value (`YOUR_SERVICE_API_KEY`) determines which env vars are read at runtime. Adding more keys only requires bumping the count and adding the new env var.
|
||||
|
||||
### Pricing: Prefer API-Reported Cost
|
||||
|
||||
Always prefer using cost data returned by the API (e.g., `creditsUsed`, `costDollars`). This is the most accurate because it accounts for variable pricing tiers, feature modifiers, and plan-level discounts.
|
||||
|
||||
**When the API reports cost** — use it directly and throw if missing:
|
||||
|
||||
```typescript
|
||||
pricing: {
|
||||
type: 'custom',
|
||||
getCost: (params, output) => {
|
||||
if (output.creditsUsed == null) {
|
||||
throw new Error('Response missing creditsUsed field')
|
||||
}
|
||||
// $0.001 per credit — from https://example.com/pricing
|
||||
const cost = (output.creditsUsed as number) * 0.001
|
||||
return { cost, metadata: { creditsUsed: output.creditsUsed } }
|
||||
},
|
||||
},
|
||||
```
|
||||
|
||||
**When the API does NOT report cost** — compute it from params/output based on the pricing docs, but still validate the data you depend on:
|
||||
|
||||
```typescript
|
||||
pricing: {
|
||||
type: 'custom',
|
||||
getCost: (params, output) => {
|
||||
if (!Array.isArray(output.searchResults)) {
|
||||
throw new Error('Response missing searchResults, cannot determine cost')
|
||||
}
|
||||
// Serper: 1 credit for <=10 results, 2 credits for >10 — from https://serper.dev/pricing
|
||||
const credits = Number(params.num) > 10 ? 2 : 1
|
||||
return { cost: credits * 0.001, metadata: { credits } }
|
||||
},
|
||||
},
|
||||
```
|
||||
|
||||
**`getCost` must always throw** if it cannot determine cost. Never silently fall back to a default — this would hide billing inaccuracies.
|
||||
|
||||
### Capturing Cost Data from the API
|
||||
|
||||
If the API returns cost info, capture it in `transformResponse` so `getCost` can read it from the output:
|
||||
|
||||
```typescript
|
||||
transformResponse: async (response: Response) => {
|
||||
const data = await response.json()
|
||||
return {
|
||||
success: true,
|
||||
output: {
|
||||
results: data.results,
|
||||
creditsUsed: data.creditsUsed, // pass through for getCost
|
||||
},
|
||||
}
|
||||
},
|
||||
```
|
||||
|
||||
For async/polling tools, capture it in `postProcess` when the job completes:
|
||||
|
||||
```typescript
|
||||
if (jobData.status === 'completed') {
|
||||
result.output = {
|
||||
data: jobData.data,
|
||||
creditsUsed: jobData.creditsUsed,
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Step 4: Hide the API Key Field When Hosted
|
||||
|
||||
In the block config (`blocks/blocks/{service}.ts`), add `hideWhenHosted: true` to the API key subblock. This hides the field on hosted Sim since the platform provides the key:
|
||||
|
||||
```typescript
|
||||
{
|
||||
id: 'apiKey',
|
||||
title: 'API Key',
|
||||
type: 'short-input',
|
||||
placeholder: 'Enter your API key',
|
||||
password: true,
|
||||
required: true,
|
||||
hideWhenHosted: true,
|
||||
},
|
||||
```
|
||||
|
||||
The visibility is controlled by `isSubBlockHidden()` in `lib/workflows/subblocks/visibility.ts`, which checks both the `isHosted` feature flag (`hideWhenHosted`) and optional env var conditions (`hideWhenEnvSet`).
|
||||
|
||||
### Excluding Specific Operations from Hosted Key Support
|
||||
|
||||
When a block has multiple operations but some operations should **not** use a hosted key (e.g., the underlying API is deprecated, unsupported, or too expensive), use the **duplicate apiKey subblock** pattern. This is the same pattern Exa uses for its `research` operation:
|
||||
|
||||
1. **Remove the `hosting` config** from the tool definition for that operation — it must not have a `hosting` object at all.
|
||||
2. **Duplicate the `apiKey` subblock** in the block config with opposing conditions:
|
||||
|
||||
```typescript
|
||||
// API Key — hidden when hosted for operations with hosted key support
|
||||
{
|
||||
id: 'apiKey',
|
||||
title: 'API Key',
|
||||
type: 'short-input',
|
||||
placeholder: 'Enter your API key',
|
||||
password: true,
|
||||
required: true,
|
||||
hideWhenHosted: true,
|
||||
condition: { field: 'operation', value: 'unsupported_op', not: true },
|
||||
},
|
||||
// API Key — always visible for unsupported_op (no hosted key support)
|
||||
{
|
||||
id: 'apiKey',
|
||||
title: 'API Key',
|
||||
type: 'short-input',
|
||||
placeholder: 'Enter your API key',
|
||||
password: true,
|
||||
required: true,
|
||||
condition: { field: 'operation', value: 'unsupported_op' },
|
||||
},
|
||||
```
|
||||
|
||||
Both subblocks share the same `id: 'apiKey'`, so the same value flows to the tool. The conditions ensure only one is visible at a time. The first has `hideWhenHosted: true` and shows for all hosted operations; the second has no `hideWhenHosted` and shows only for the excluded operation — meaning users must always provide their own key for that operation.
|
||||
|
||||
To exclude multiple operations, use an array: `{ field: 'operation', value: ['op_a', 'op_b'] }`.
|
||||
|
||||
**Reference implementations:**
|
||||
- **Exa** (`blocks/blocks/exa.ts`): `exa_research` operation excluded from hosting — duplicate `apiKey` pair around lines ~348-365
|
||||
- **Google Maps** (`blocks/blocks/google_maps.ts`): `speed_limits` operation excluded from hosting (deprecated Roads API)
|
||||
|
||||
## Step 5: Add to the BYOK Settings UI
|
||||
|
||||
Add an entry to the `PROVIDERS` array in the BYOK settings component so users can bring their own key. You need the service icon from `components/icons.tsx`:
|
||||
|
||||
```typescript
|
||||
{
|
||||
id: 'your_service',
|
||||
name: 'Your Service',
|
||||
icon: YourServiceIcon,
|
||||
description: 'What this service does',
|
||||
placeholder: 'Enter your API key',
|
||||
},
|
||||
```
|
||||
|
||||
## Step 6: Summarize Pricing and Throttling Comparison
|
||||
|
||||
After all code changes are complete, output a detailed summary to the user covering:
|
||||
|
||||
### What to include
|
||||
|
||||
1. **API's pricing model** — how the service charges (per token, per credit, per request, etc.), the specific rates found in docs, and whether the API reports cost in responses.
|
||||
2. **Our `getCost` approach** — how we calculate cost, what fields we depend on, and any assumptions or estimates (especially when the API doesn't report exact dollar cost).
|
||||
3. **API's rate limits** — the documented limits (RPM, TPM, concurrent, etc.), which plan tier they apply to, and whether they're per-key or per-account.
|
||||
4. **Our `rateLimit` config** — what we set for `requestsPerMinute` (and dimensions if custom mode), why we chose those values, and how they compare to the API's limits.
|
||||
5. **Key pooling impact** — how many hosted keys we expect, and how round-robin distribution affects the effective per-key rate at the API.
|
||||
6. **Gaps or risks** — anything the API charges for that we don't meter, rate limit dimensions we chose not to enforce, or pricing that may be inaccurate due to variable model/tier costs.
|
||||
|
||||
### Format
|
||||
|
||||
Present this as a structured summary with clear headings. Example:
|
||||
|
||||
```
|
||||
### Pricing
|
||||
- **API charges**: $X per 1M tokens (input), $Y per 1M tokens (output) — varies by model
|
||||
- **Response reports cost?**: No — only token counts in `usage` field
|
||||
- **Our getCost**: Estimates cost at $Z per 1M total tokens based on median model pricing
|
||||
- **Risk**: Actual cost varies by model; our estimate may over/undercharge for cheap/expensive models
|
||||
|
||||
### Throttling
|
||||
- **API limits**: 300 RPM per key (paid tier), 60 RPM (free tier)
|
||||
- **Per-key or per-account**: Per key — more keys = more throughput
|
||||
- **Our config**: 60 RPM per workspace (per_request mode)
|
||||
- **With N keys**: Effective per-key rate is (total RPM across workspaces) / N
|
||||
- **Headroom**: Comfortable — even 10 active workspaces at full rate = 600 RPM / 3 keys = 200 RPM per key, under the 300 RPM API limit
|
||||
```
|
||||
|
||||
This summary helps reviewers verify that the pricing and rate limiting are well-calibrated and surfaces any risks that need monitoring.
|
||||
|
||||
## Checklist
|
||||
|
||||
- [ ] Provider added to `BYOKProviderId` in `tools/types.ts`
|
||||
- [ ] Provider added to `byokProviderIdSchema` enum in `lib/api/contracts/byok-keys.ts`
|
||||
- [ ] API pricing docs researched — understand per-unit cost and whether the API reports cost in responses
|
||||
- [ ] API rate limits researched — understand RPM/TPM limits, per-key vs per-account, and plan tiers
|
||||
- [ ] `hosting` config added to the tool with `envKeyPrefix`, `apiKeyParam`, `byokProviderId`, `pricing`, and `rateLimit`
|
||||
- [ ] `getCost` throws if required cost data is missing from the response
|
||||
- [ ] Cost data captured in `transformResponse` or `postProcess` if API provides it
|
||||
- [ ] `hideWhenHosted: true` added to the API key subblock in the block config
|
||||
- [ ] Provider entry added to the BYOK settings UI with icon and description
|
||||
- [ ] Env vars documented: `{PREFIX}_COUNT` and `{PREFIX}_1..N`
|
||||
- [ ] Pricing and throttling summary provided to reviewer
|
||||
@@ -0,0 +1,5 @@
|
||||
interface:
|
||||
display_name: "Add Hosted Key"
|
||||
short_description: "Add hosted API key to a tool"
|
||||
brand_color: "#CA8A04"
|
||||
default_prompt: "Use $add-hosted-key to add hosted API key support (metered and billed) to a Sim tool."
|
||||
@@ -0,0 +1,999 @@
|
||||
---
|
||||
name: add-integration
|
||||
description: Add a complete Sim integration from API docs, covering tools, block, icon, optional triggers, registrations, resolved-secret/model-input safety, and integration conventions. Use when introducing a new service under `apps/sim/tools`, `apps/sim/blocks`, and `apps/sim/triggers`.
|
||||
argument-hint: <service-name> [api-docs-url]
|
||||
---
|
||||
|
||||
# Add Integration Skill
|
||||
|
||||
You are an expert at adding complete integrations to Sim. This skill orchestrates the full process of adding a new service integration.
|
||||
|
||||
## Overview
|
||||
|
||||
Adding an integration involves these steps in order:
|
||||
1. **Research** - Read the service's API documentation
|
||||
2. **Create Tools** - Build tool configurations for each API operation
|
||||
3. **Create Block** - Build the block UI configuration
|
||||
4. **Add Icon** - Add the service's brand icon
|
||||
5. **Create Triggers** (optional) - If the service supports webhooks
|
||||
6. **Register** - Register tools, block, and triggers in their registries
|
||||
7. **Configure Deployment Availability** - Wire OAuth client and service-account metadata
|
||||
8. **Generate and Validate the Catalog** - Regenerate docs/catalog artifacts and run drift checks
|
||||
|
||||
## Step 1: Research the API
|
||||
|
||||
Before writing any code:
|
||||
1. Use Context7 to find official documentation: `mcp__context7__resolve-library-id`, then fetch with `mcp__context7__query-docs`
|
||||
2. Or use WebFetch to read API docs directly
|
||||
3. Identify:
|
||||
- Authentication method (OAuth, API Key, both)
|
||||
- Available operations (CRUD, search, etc.)
|
||||
- Required vs optional parameters
|
||||
- Response structures
|
||||
|
||||
### Hard Rule: No Guessed Response Schemas
|
||||
|
||||
If the official docs do not clearly show the response JSON shape for an endpoint, you MUST stop and tell the user exactly which outputs are unknown.
|
||||
|
||||
- Do NOT guess response field names
|
||||
- Do NOT infer nested JSON paths from related endpoints
|
||||
- Do NOT invent output properties just because they seem likely
|
||||
- Do NOT implement `transformResponse` against unverified payload shapes
|
||||
|
||||
If response schemas are missing or incomplete, do one of the following before proceeding:
|
||||
1. Ask the user for sample responses
|
||||
2. Ask the user for test credentials so you can verify the live payload
|
||||
3. Reduce the scope to only endpoints whose response shapes are documented
|
||||
4. Leave the tool unimplemented and explicitly report why
|
||||
|
||||
## Step 2: Create Tools
|
||||
|
||||
### Directory Structure
|
||||
```
|
||||
apps/sim/tools/{service}/
|
||||
├── index.ts # Barrel exports
|
||||
├── types.ts # TypeScript interfaces
|
||||
├── {action1}.ts # Tool for action 1
|
||||
├── {action2}.ts # Tool for action 2
|
||||
└── ...
|
||||
```
|
||||
|
||||
### Key Patterns
|
||||
|
||||
**types.ts:**
|
||||
```typescript
|
||||
import type { ToolResponse } from '@/tools/types'
|
||||
|
||||
export interface {Service}{Action}Params {
|
||||
accessToken: string // For OAuth services
|
||||
// OR
|
||||
apiKey: string // For API key services
|
||||
|
||||
requiredParam: string
|
||||
optionalParam?: string
|
||||
}
|
||||
|
||||
export interface {Service}Response extends ToolResponse {
|
||||
output: {
|
||||
// Define output structure
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Tool file pattern:**
|
||||
```typescript
|
||||
export const {service}{Action}Tool: ToolConfig<Params, Response> = {
|
||||
id: '{service}_{action}',
|
||||
name: '{Service} {Action}',
|
||||
description: '...',
|
||||
version: '1.0.0',
|
||||
|
||||
oauth: { required: true, provider: '{service}' }, // If OAuth
|
||||
|
||||
params: {
|
||||
accessToken: { type: 'string', required: true, visibility: 'hidden', description: '...' },
|
||||
// ... other params
|
||||
},
|
||||
|
||||
request: { url, method, headers, body },
|
||||
|
||||
transformResponse: async (response) => {
|
||||
const data = await response.json()
|
||||
return {
|
||||
success: true,
|
||||
output: {
|
||||
field: data.field ?? null, // Always handle nullables
|
||||
},
|
||||
}
|
||||
},
|
||||
|
||||
outputs: { /* ... */ },
|
||||
}
|
||||
```
|
||||
|
||||
### Critical Rules
|
||||
- `visibility: 'hidden'` for OAuth tokens
|
||||
- `visibility: 'user-only'` for API keys and user credentials
|
||||
- `visibility: 'user-or-llm'` for operation parameters
|
||||
- Always use `?? null` for nullable API response fields
|
||||
- Always use `?? []` for optional array fields
|
||||
- Set `optional: true` for outputs that may not exist
|
||||
- Never output raw JSON dumps - extract meaningful fields
|
||||
- When using `type: 'json'` and you know the object shape, define `properties` with the inner fields so downstream consumers know the structure. Only use bare `type: 'json'` when the shape is truly dynamic
|
||||
- If you do not know the response JSON shape from docs or verified examples, you MUST tell the user and stop. Never guess outputs or response mappings.
|
||||
|
||||
### Resolved Secrets at Model and Persistence Boundaries
|
||||
|
||||
Classify every request field before implementing the tool:
|
||||
|
||||
This is opt-in, not a blanket integration migration. Add a model-input declaration only when the
|
||||
service's official documentation or an unambiguous local execution path proves that the exact
|
||||
field is consumed by an AI model. If that cannot be established, preserve existing tool behavior
|
||||
and leave the field unannotated.
|
||||
|
||||
- **Ordinary provider/API input:** leave it unchanged. Explicit `{{...}}` references resolve and are
|
||||
sent with their normal request semantics. A URL, domain, resource ID, control field, or opaque
|
||||
payload is not model-visible merely because the provider is AI-backed or may process the
|
||||
referenced resource later.
|
||||
- **Text or structured content consumed by an AI model:** declare `request.modelInput` with
|
||||
`mode: 'project'` and select only the exact model-visible fields. The shared executor replaces
|
||||
activated Sim secrets with canonical `{{NAME}}` labels before request formatting. For nested or
|
||||
JSON-string fields, use a small shared selector plus `applyProjected`; verify that selecting the
|
||||
rebuilt params reproduces the projected selection.
|
||||
- **Serialized model content sent directly to an external provider:** include the serialized
|
||||
top-level param in `request.modelInput`. Project the private copy before the existing request
|
||||
formatter parses it; keep formatter behavior deterministic when a whole-value placeholder is not
|
||||
valid in the serialized grammar. Do not introduce a second hard-rejection path.
|
||||
- **Opaque model input owned by an authenticated internal route** such as inline audio, image,
|
||||
video, or document bytes: add `privateProvenance` to a projected request, or use
|
||||
`mode: 'private-provenance'` when there is no textual projection. Do not select storage keys,
|
||||
paths, signed URLs, or ordinary remote URLs as byte provenance; the owning route must authorize
|
||||
stored bytes independently at model egress. The route must call
|
||||
`validateOpaqueModelInputProvenance` before downloading or sending content to the model and must
|
||||
apply the workspace-file provenance guard before reading a persisted workspace file.
|
||||
- **Sim-owned durable storage or internal execution handoff** that can later enter a workflow/model
|
||||
(table cells, Agent memory, knowledge documents/chunks, workspace-file contents, or child-workflow
|
||||
input): transport encrypted field-scoped provenance with `request.secretProvenance`. The
|
||||
authenticated receiver validates the exact selection and scope, strips the private envelope, and
|
||||
persists, imports, or propagates it at the owning boundary. Preserve shared legacy behavior for
|
||||
headerless internal calls and rows/files whose provenance marker is `NULL`; never invent a
|
||||
tool-local migration rule.
|
||||
|
||||
Hard rules:
|
||||
|
||||
- Never substitute secret plaintext into source or serialize plaintext provenance.
|
||||
- Never hand-roll private provenance headers/envelopes; the shared `executeTool` boundary owns
|
||||
transport and strips private metadata from functional results.
|
||||
- Never attach private provenance to an external URL or to `directExecution`. Project proven
|
||||
model-visible external fields with `request.modelInput`; otherwise preserve ordinary request
|
||||
semantics. Use an authenticated internal route when encrypted provenance must cross the boundary.
|
||||
- Never sanitize arbitrary third-party tool results. Projection applies only to secrets activated
|
||||
by Sim's resolved-secret provenance for that execution/tool call.
|
||||
- Do not add provenance merely because a value is persisted, returned by a tool, or appears in a
|
||||
filename. Require a concrete Sim `{{...}}` resolution path and a later model/log boundary. If an
|
||||
unsupported field can resolve a secret but does not justify durable tracking (for example a
|
||||
`file_write` path), reject it at that exact ingress.
|
||||
- At diagnostic boundaries, project only values carrying execution-scoped provenance. Ordinary
|
||||
provider responses, filenames, URLs, and errors remain unchanged when Sim did not resolve a
|
||||
secret into them.
|
||||
|
||||
Add focused tests covering named projection, ordinary identical text without provenance, nested and
|
||||
serialized shape handling, unchanged ordinary external inputs, malformed/incomplete private metadata
|
||||
failing closed, headerless legacy requests, and absence of private metadata in the public tool result.
|
||||
For durable sinks, also cover legacy `NULL` markers, exact-empty new writes, tracked secret writes,
|
||||
stale/missing sidecars, and scope isolation.
|
||||
|
||||
## Step 3: Create Block
|
||||
|
||||
### File Location
|
||||
`apps/sim/blocks/blocks/{service}.ts`
|
||||
|
||||
### Block Structure
|
||||
```typescript
|
||||
import { {Service}Icon } from '@/components/icons'
|
||||
import type { BlockConfig } from '@/blocks/types'
|
||||
import { AuthMode, IntegrationType } from '@/blocks/types'
|
||||
import { getScopesForService } from '@/lib/oauth/utils'
|
||||
|
||||
export const {Service}Block: BlockConfig = {
|
||||
type: '{service}',
|
||||
name: '{Service}',
|
||||
description: '...',
|
||||
longDescription: '...',
|
||||
docsLink: 'https://docs.sim.ai/integrations/{service}',
|
||||
category: 'tools',
|
||||
integrationType: IntegrationType.X, // Primary category (see IntegrationType enum)
|
||||
tags: ['oauth', 'api'], // Cross-cutting tags (see IntegrationTag type)
|
||||
bgColor: '#HEXCOLOR',
|
||||
icon: {Service}Icon,
|
||||
authMode: AuthMode.OAuth, // or AuthMode.ApiKey
|
||||
|
||||
subBlocks: [
|
||||
// Operation dropdown
|
||||
{
|
||||
id: 'operation',
|
||||
title: 'Operation',
|
||||
type: 'dropdown',
|
||||
options: [
|
||||
{ label: 'Operation 1', id: 'action1' },
|
||||
{ label: 'Operation 2', id: 'action2' },
|
||||
],
|
||||
value: () => 'action1',
|
||||
},
|
||||
// Credential field
|
||||
{
|
||||
id: 'credential',
|
||||
title: '{Service} Account',
|
||||
type: 'oauth-input',
|
||||
serviceId: '{service}',
|
||||
requiredScopes: getScopesForService('{service}'),
|
||||
required: true,
|
||||
},
|
||||
// Conditional fields per operation
|
||||
// ...
|
||||
],
|
||||
|
||||
tools: {
|
||||
access: ['{service}_action1', '{service}_action2'],
|
||||
config: {
|
||||
tool: (params) => `{service}_${params.operation}`,
|
||||
},
|
||||
},
|
||||
|
||||
outputs: { /* ... */ },
|
||||
}
|
||||
```
|
||||
|
||||
### Key SubBlock Patterns
|
||||
|
||||
**Condition-based visibility:**
|
||||
```typescript
|
||||
{
|
||||
id: 'resourceId',
|
||||
title: 'Resource ID',
|
||||
type: 'short-input',
|
||||
condition: { field: 'operation', value: ['read', 'update', 'delete'] },
|
||||
required: { field: 'operation', value: ['read', 'update', 'delete'] },
|
||||
}
|
||||
```
|
||||
|
||||
**DependsOn for cascading selectors:**
|
||||
```typescript
|
||||
{
|
||||
id: 'project',
|
||||
type: 'project-selector',
|
||||
dependsOn: ['credential'],
|
||||
},
|
||||
{
|
||||
id: 'issue',
|
||||
type: 'file-selector',
|
||||
dependsOn: ['credential', 'project'],
|
||||
}
|
||||
```
|
||||
|
||||
**Basic/Advanced mode for dual UX:**
|
||||
```typescript
|
||||
// Basic: Visual selector
|
||||
{
|
||||
id: 'channelSelector',
|
||||
type: 'channel-selector',
|
||||
mode: 'basic',
|
||||
canonicalParamId: 'channel',
|
||||
dependsOn: ['credential'],
|
||||
},
|
||||
// Advanced: Manual input
|
||||
{
|
||||
id: 'channelId',
|
||||
type: 'short-input',
|
||||
mode: 'advanced',
|
||||
canonicalParamId: 'channel',
|
||||
}
|
||||
```
|
||||
|
||||
Note neither subblock `id` is `channel` — the canonical id is a third name that both members map
|
||||
onto, and it is the only one that survives serialization.
|
||||
|
||||
**Critical Canonical Param Rules:**
|
||||
- `canonicalParamId` must NOT match any subblock's `id` in the block
|
||||
- `canonicalParamId` must be unique **block-wide**, not per operation. `buildCanonicalIndex` keys
|
||||
groups by `canonicalParamId` across all subblocks and a group holds exactly one `basicId`, so two
|
||||
operations that each need their own pair must use two different canonical ids
|
||||
- Only use `canonicalParamId` to link basic/advanced alternatives for the same logical parameter.
|
||||
A pair carries ONE concept — for files that means upload (basic) + file reference (advanced), as
|
||||
in Gmail attachments (`blocks/blocks/gmail.ts`). Never overload the advanced side with alternate
|
||||
identifiers like a URL or a provider asset ID; give those their own subblocks, mark all the
|
||||
mutually exclusive sources `required: false`, and enforce "exactly one" at execution
|
||||
- `mode` only controls UI visibility, NOT serialization. Without `canonicalParamId`, both basic and advanced field values would be sent
|
||||
- Every subblock `id` must be unique within the block. Duplicate IDs cause conflicts even with different conditions
|
||||
- **Required consistency:** If one subblock in a canonical group has `required: true`, ALL subblocks in that group must have `required: true` (prevents bypassing validation by switching modes)
|
||||
- **Inputs section:** Must list canonical param IDs (e.g., `fileId`), NOT raw subblock IDs (e.g., `fileSelector`, `manualFileId`)
|
||||
- **Params function:** Must use canonical param IDs, NOT raw subblock IDs (raw IDs are deleted after canonical transformation)
|
||||
|
||||
### BlockMeta (Required)
|
||||
|
||||
Export a `{Service}BlockMeta` in the same file as the block — **minimum 7 templates**. See `.agents/skills/add-block/SKILL.md` → "BlockMeta (Required)" for valid `modules` and `category` values and the full pattern.
|
||||
|
||||
```typescript
|
||||
export const {Service}BlockMeta = {
|
||||
tags: ['tag1', 'tag2'],
|
||||
templates: [
|
||||
{
|
||||
icon: {Service}Icon,
|
||||
title: '{Service} <use-case>',
|
||||
prompt: 'Build a workflow that...', // concrete trigger → transformation → output
|
||||
modules: ['agent', 'workflows'],
|
||||
category: 'operations',
|
||||
tags: ['automation'],
|
||||
alsoIntegrations: ['slack'], // when the prompt references another service
|
||||
},
|
||||
// ... at least 6 more
|
||||
],
|
||||
} as const satisfies BlockMeta
|
||||
```
|
||||
|
||||
## Step 4: Add Icon
|
||||
|
||||
### File Location
|
||||
`apps/sim/components/icons.tsx`
|
||||
|
||||
### Pattern
|
||||
```typescript
|
||||
export function {Service}Icon(props: SVGProps<SVGSVGElement>) {
|
||||
return (
|
||||
<svg
|
||||
{...props}
|
||||
viewBox="0 0 24 24"
|
||||
fill="none"
|
||||
xmlns="http://www.w3.org/2000/svg"
|
||||
>
|
||||
{/* SVG paths from user-provided SVG */}
|
||||
</svg>
|
||||
)
|
||||
}
|
||||
```
|
||||
|
||||
### Getting Icons
|
||||
**Do NOT search for icons yourself.** At the end of implementation, ask the user to provide the SVG:
|
||||
|
||||
```
|
||||
I've completed the integration. Before I can add the icon, please provide the SVG for {Service}.
|
||||
You can usually find this in the service's brand/press kit page, or copy it from their website.
|
||||
|
||||
Paste the SVG code here and I'll convert it to a React component.
|
||||
```
|
||||
|
||||
Once the user provides the SVG:
|
||||
1. Extract the SVG paths/content
|
||||
2. Create a React component that spreads props
|
||||
3. Ensure viewBox is preserved from the original SVG
|
||||
|
||||
### Theme-safety (bare rendering) — REQUIRED
|
||||
|
||||
The icon renders both inside its colored `bgColor` tile AND "bare" (no tile) on a
|
||||
neutral page — e.g. the home **Suggested actions** list — in both light and dark
|
||||
mode. A monochrome logo whose paths hardcode a single near-white or near-black
|
||||
fill is invisible bare on the matching background (white-on-white in light mode,
|
||||
black-on-black in dark mode).
|
||||
|
||||
Rules when adding the SVG:
|
||||
|
||||
- **Monochrome logos** (a single white or black mark): draw the shape with
|
||||
`fill='currentColor'`, not `fill='#fff'` / `fill='#000000'`. It then inherits
|
||||
white inside dark tiles, near-black inside light tiles (via
|
||||
`getTileIconColorClass`), and the theme-aware `var(--text-icon)` bare — legible
|
||||
everywhere. Do NOT set `iconColor` for these.
|
||||
- **Multi-color brand logos** (their own vivid fills): keep the hardcoded fills.
|
||||
They read on any background. Only set `iconColor` (a vivid brand hex, never a
|
||||
near-black/near-white tile color) if the bare icon should adopt a brand tint.
|
||||
- A large white shape with a tiny vivid accent (e.g. a logo where the body is the
|
||||
white negative space) still vanishes bare — convert the body to `currentColor`.
|
||||
|
||||
Verify with `bun run check:bare-icons` (also runs in CI). It flags purely
|
||||
monochrome hazards; for partial-accent logos, eyeball the suggested-actions list
|
||||
in both light and dark mode.
|
||||
|
||||
## Step 5: Create Triggers (Optional)
|
||||
|
||||
If the service supports webhooks, create triggers using the generic `buildTriggerSubBlocks` helper.
|
||||
|
||||
### Directory Structure
|
||||
```
|
||||
apps/sim/triggers/{service}/
|
||||
├── index.ts # Barrel exports
|
||||
├── utils.ts # Trigger options, setup instructions, extra fields
|
||||
├── {event_a}.ts # Primary trigger (includes dropdown)
|
||||
├── {event_b}.ts # Secondary triggers (no dropdown)
|
||||
└── webhook.ts # Generic webhook (optional)
|
||||
```
|
||||
|
||||
### Key Pattern
|
||||
|
||||
```typescript
|
||||
import { buildTriggerSubBlocks } from '@/triggers'
|
||||
import { {service}TriggerOptions, {service}SetupInstructions, build{Service}ExtraFields } from './utils'
|
||||
|
||||
// Primary trigger - includeDropdown: true
|
||||
export const {service}EventATrigger: TriggerConfig = {
|
||||
id: '{service}_event_a',
|
||||
subBlocks: buildTriggerSubBlocks({
|
||||
triggerId: '{service}_event_a',
|
||||
triggerOptions: {service}TriggerOptions,
|
||||
includeDropdown: true, // Only for primary trigger!
|
||||
setupInstructions: {service}SetupInstructions('Event A'),
|
||||
extraFields: build{Service}ExtraFields('{service}_event_a'),
|
||||
}),
|
||||
// ...
|
||||
}
|
||||
|
||||
// Secondary triggers - no dropdown
|
||||
export const {service}EventBTrigger: TriggerConfig = {
|
||||
id: '{service}_event_b',
|
||||
subBlocks: buildTriggerSubBlocks({
|
||||
triggerId: '{service}_event_b',
|
||||
triggerOptions: {service}TriggerOptions,
|
||||
// No includeDropdown!
|
||||
setupInstructions: {service}SetupInstructions('Event B'),
|
||||
extraFields: build{Service}ExtraFields('{service}_event_b'),
|
||||
}),
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
### Connect to Block
|
||||
```typescript
|
||||
import { getTrigger } from '@/triggers'
|
||||
|
||||
export const {Service}Block: BlockConfig = {
|
||||
triggers: {
|
||||
enabled: true,
|
||||
available: ['{service}_event_a', '{service}_event_b'],
|
||||
},
|
||||
subBlocks: [
|
||||
// Tool fields...
|
||||
...getTrigger('{service}_event_a').subBlocks,
|
||||
...getTrigger('{service}_event_b').subBlocks,
|
||||
],
|
||||
}
|
||||
```
|
||||
|
||||
See `/add-trigger` skill for complete documentation.
|
||||
|
||||
## Step 6: Register Everything
|
||||
|
||||
### Tools Registry (`apps/sim/tools/registry.ts`)
|
||||
|
||||
```typescript
|
||||
// Add import (alphabetically)
|
||||
import {
|
||||
{service}Action1Tool,
|
||||
{service}Action2Tool,
|
||||
} from '@/tools/{service}'
|
||||
|
||||
// Add to tools object (alphabetically)
|
||||
export const tools: Record<string, ToolConfig> = {
|
||||
// ... existing tools ...
|
||||
{service}_action1: {service}Action1Tool,
|
||||
{service}_action2: {service}Action2Tool,
|
||||
}
|
||||
```
|
||||
|
||||
Then regenerate the generated tool metadata and commit it:
|
||||
|
||||
```bash
|
||||
bun run tool-metadata:generate
|
||||
```
|
||||
|
||||
Client code reads `params`/`outputs` from these artifacts rather than importing
|
||||
the registry, so a tool you add, change or remove is invisible to the UI until they are regenerated,
|
||||
and CI fails on stale ones. See `.agents/skills/tool-registry-boundary/SKILL.md`.
|
||||
|
||||
### Block Registry (`apps/sim/blocks/registry-maps.ts`)
|
||||
|
||||
The data maps (`BLOCK_REGISTRY` + `BLOCK_META_REGISTRY`) live in `registry-maps.ts`; `registry.ts` holds only the accessor functions. Add the import and an entry to each map alphabetically:
|
||||
|
||||
```typescript
|
||||
// Add import (alphabetically)
|
||||
import { {Service}Block, {Service}BlockMeta } from '@/blocks/blocks/{service}'
|
||||
|
||||
// Add to the config map (alphabetically)
|
||||
export const BLOCK_REGISTRY: Record<string, BlockConfig> = {
|
||||
// ... existing blocks ...
|
||||
{service}: {Service}Block,
|
||||
}
|
||||
|
||||
// Add to the catalog-meta map (alphabetically)
|
||||
export const BLOCK_META_REGISTRY: Record<string, BlockMeta> = {
|
||||
// ... existing metas ...
|
||||
{service}: {Service}BlockMeta,
|
||||
}
|
||||
```
|
||||
|
||||
### Trigger Registry (`apps/sim/triggers/registry.ts`) - If triggers exist
|
||||
|
||||
```typescript
|
||||
// Add import (alphabetically)
|
||||
import {
|
||||
{service}EventATrigger,
|
||||
{service}EventBTrigger,
|
||||
{service}WebhookTrigger,
|
||||
} from '@/triggers/{service}'
|
||||
|
||||
// Add to TRIGGER_REGISTRY (alphabetically)
|
||||
export const TRIGGER_REGISTRY: TriggerRegistry = {
|
||||
// ... existing triggers ...
|
||||
{service}_event_a: {service}EventATrigger,
|
||||
{service}_event_b: {service}EventBTrigger,
|
||||
{service}_webhook: {service}WebhookTrigger,
|
||||
}
|
||||
```
|
||||
|
||||
## Step 7: Configure Deployment Availability
|
||||
|
||||
Do this for every visible OAuth integration. API-key and unauthenticated integrations do not need
|
||||
an OAuth client capability.
|
||||
|
||||
The block's `oauth-input.serviceId` is the canonical link between the generated integration catalog,
|
||||
the OAuth service configuration, deployment availability, and the setup CLI.
|
||||
|
||||
1. Ensure the block has exactly one distinct OAuth `serviceId` and that it matches the canonical
|
||||
service entry in `apps/sim/lib/oauth/oauth.ts`.
|
||||
2. Confirm `resolveOAuthClientCapabilityId(serviceId)` resolves to the intended provider entry in
|
||||
`OAUTH_CLIENT_CAPABILITIES` in `apps/sim/lib/core/config/env-capabilities.ts`. Google and
|
||||
Microsoft service IDs deliberately share provider-level capabilities.
|
||||
3. For a new OAuth provider, add the required client fields to `OAUTH_CLIENT_CAPABILITIES`, add
|
||||
every referenced field to the env schema in `apps/sim/lib/core/config/env.ts`, and add the
|
||||
matching `text` or `secret` entries to `OAUTH_CLIENT_SETUP_FIELDS` in
|
||||
`scripts/setup/capability-config.ts`. Do not create integration-specific setup logic or infer
|
||||
secret fields from naming; the CLI mapping is exhaustively checked against the runtime fields.
|
||||
4. If the canonical OAuth service has `serviceAccountProviderId`, add the matching projection to
|
||||
`SERVICE_ACCOUNT_METADATA_BY_OAUTH_SERVICE_ID` in
|
||||
`apps/sim/lib/integrations/service-account-metadata.ts`. Use:
|
||||
- no `deploymentRequirement` when the service-account path works independently of OAuth client fields;
|
||||
- `'oauth-client'` when it requires the same deployment OAuth client fields;
|
||||
- `'preview-gated'` when availability is controlled by the service-account preview block.
|
||||
|
||||
Never add a permissive fallback for missing capability metadata. A visible OAuth integration without
|
||||
a resolvable capability must fail validation.
|
||||
|
||||
## Step 8: Generate and Validate the Catalog
|
||||
|
||||
Run the documentation generator:
|
||||
```bash
|
||||
bun run scripts/generate-docs.ts
|
||||
bun run integration-catalog:check
|
||||
```
|
||||
|
||||
This creates `apps/docs/content/docs/en/integrations/{service}.mdx` — one page per service carrying the block's Actions and, if it has one, its Triggers section. Never hand-edit generated pages; the only editable region is the `{/* MANUAL-CONTENT */}` block (see `scripts/README.md`).
|
||||
|
||||
The same generator refreshes `apps/sim/lib/integrations/integrations.json`. The catalog check then
|
||||
derives the deployment-relevant fields from the executable block registry and compares them with the
|
||||
committed projection. Review the generated diff and keep only intentional changes.
|
||||
|
||||
## V2 Integration Pattern
|
||||
|
||||
If creating V2 versions (API-aligned outputs):
|
||||
|
||||
1. **V2 Tools** - Add `_v2` suffix, version `2.0.0`, flat outputs
|
||||
2. **V2 Block** - Add `_v2` type, use `createVersionedToolSelector`
|
||||
3. **V1 Block** - Add `(Legacy)` to name, set `hideFromToolbar: true`
|
||||
4. **Registry** - Register both versions
|
||||
|
||||
```typescript
|
||||
// In registry
|
||||
{service}: {Service}Block, // V1 (legacy, hidden)
|
||||
{service}_v2: {Service}V2Block, // V2 (visible)
|
||||
```
|
||||
|
||||
## Complete Checklist
|
||||
|
||||
### Tools
|
||||
- [ ] Created `tools/{service}/` directory
|
||||
- [ ] Created `types.ts` with all interfaces
|
||||
- [ ] Created tool file for each operation
|
||||
- [ ] All params have correct visibility
|
||||
- [ ] All nullable fields use `?? null`
|
||||
- [ ] All optional outputs have `optional: true`
|
||||
- [ ] Created `index.ts` barrel export
|
||||
- [ ] Registered all tools in `tools/registry.ts`
|
||||
- [ ] Ran `bun run tool-metadata:generate` and committed the regenerated artifacts
|
||||
- [ ] Classified every model-visible, opaque, Sim-durable, and internal-execution request field
|
||||
- [ ] Added shared model-input projection or private provenance only where required; ordinary
|
||||
external resource locators and control inputs retain their request semantics
|
||||
- [ ] Confirmed ordinary third-party tool results are not generically sanitized
|
||||
- [ ] Added provenance compatibility and fail-closed boundary tests where applicable
|
||||
|
||||
### Block
|
||||
- [ ] Created `blocks/blocks/{service}.ts`
|
||||
- [ ] Set `integrationType` to the correct `IntegrationType` enum value
|
||||
- [ ] Set `tags` array with all applicable `IntegrationTag` values
|
||||
- [ ] Defined operation dropdown with all operations
|
||||
- [ ] Added credential field with `requiredScopes: getScopesForService('{service}')`
|
||||
- [ ] Added conditional fields per operation
|
||||
- [ ] Set up dependsOn for cascading selectors
|
||||
- [ ] Configured tools.access with all tool IDs
|
||||
- [ ] Configured tools.config.tool selector
|
||||
- [ ] Defined outputs matching tool outputs
|
||||
- [ ] Registered block + meta in `blocks/registry-maps.ts` (`BLOCK_REGISTRY` / `BLOCK_META_REGISTRY`)
|
||||
- [ ] If triggers: set `triggers.enabled` and `triggers.available`
|
||||
- [ ] If triggers: spread trigger subBlocks with `getTrigger()`
|
||||
- [ ] Exported `{Service}BlockMeta` with at least 7 templates
|
||||
|
||||
### OAuth Scopes (if OAuth service)
|
||||
- [ ] Defined scopes in `lib/oauth/oauth.ts` under `OAUTH_PROVIDERS`
|
||||
- [ ] Added scope descriptions in `SCOPE_DESCRIPTIONS` within `lib/oauth/utils.ts`
|
||||
- [ ] Used `getCanonicalScopesForProvider()` in `auth.ts` (never hardcode)
|
||||
- [ ] Used `getScopesForService()` in block `requiredScopes` (never hardcode)
|
||||
|
||||
### Deployment Availability (if OAuth service)
|
||||
- [ ] Block declares exactly one distinct `oauth-input.serviceId`
|
||||
- [ ] `resolveOAuthClientCapabilityId(serviceId)` resolves to the intended `OAUTH_CLIENT_CAPABILITIES` entry
|
||||
- [ ] Every new OAuth capability field exists in `apps/sim/lib/core/config/env.ts`
|
||||
- [ ] Runtime OAuth fields live in `OAUTH_CLIENT_CAPABILITIES`; matching CLI input modes live in the exhaustively checked `OAUTH_CLIENT_SETUP_FIELDS`
|
||||
- [ ] If `serviceAccountProviderId` is configured, `SERVICE_ACCOUNT_METADATA_BY_OAUTH_SERVICE_ID` has the matching projection and deployment requirement
|
||||
|
||||
### Icon
|
||||
- [ ] Asked user to provide SVG
|
||||
- [ ] Added icon to `components/icons.tsx`
|
||||
- [ ] Icon spreads props correctly
|
||||
- [ ] Monochrome marks use `fill='currentColor'` (not hardcoded white/black) so the icon renders bare in light AND dark mode — verified with `bun run check:bare-icons`
|
||||
|
||||
### Triggers (if service supports webhooks)
|
||||
- [ ] Created `triggers/{service}/` directory
|
||||
- [ ] Created `utils.ts` with options, instructions, and extra fields helpers
|
||||
- [ ] Primary trigger uses `includeDropdown: true`
|
||||
- [ ] Secondary triggers do NOT have `includeDropdown`
|
||||
- [ ] All triggers use `buildTriggerSubBlocks` helper
|
||||
- [ ] Created `index.ts` barrel export
|
||||
- [ ] Registered all triggers in `triggers/registry.ts`
|
||||
|
||||
### Docs
|
||||
- [ ] Ran `bun run scripts/generate-docs.ts`
|
||||
- [ ] Verified docs file created
|
||||
- [ ] Reviewed and committed the generated `apps/sim/lib/integrations/integrations.json` change
|
||||
- [ ] `bun run integration-catalog:check` passes
|
||||
|
||||
### Final Validation (Required)
|
||||
- [ ] Read every tool file and cross-referenced inputs/outputs against the API docs
|
||||
- [ ] Verified block subBlocks cover all required tool params with correct conditions
|
||||
- [ ] Verified block outputs match what the tools actually return
|
||||
- [ ] Verified `tools.config.params` correctly maps and coerces all param types
|
||||
- [ ] Verified every tool output and `transformResponse` path against documented or live-verified JSON responses
|
||||
- [ ] If any response schema remained unknown, explicitly told the user instead of guessing
|
||||
- [ ] `{Service}BlockMeta` exported with at least 7 templates, each having `icon`, `title`, `prompt`, `modules`, `category`, and `tags`
|
||||
|
||||
## Example Command
|
||||
|
||||
When the user asks to add an integration:
|
||||
|
||||
```
|
||||
User: Add a Stripe integration
|
||||
|
||||
You: I'll add the Stripe integration. Let me:
|
||||
|
||||
1. First, research the Stripe API using Context7
|
||||
2. Create the tools for key operations (payments, subscriptions, etc.)
|
||||
3. Create the block with operation dropdown
|
||||
4. Register everything
|
||||
5. Generate docs
|
||||
6. Ask you for the Stripe icon SVG
|
||||
|
||||
[Proceed with implementation...]
|
||||
|
||||
[After completing steps 1-5...]
|
||||
|
||||
I've completed the Stripe integration. Before I can add the icon, please provide the SVG for Stripe.
|
||||
You can usually find this in the service's brand/press kit page, or copy it from their website.
|
||||
|
||||
Paste the SVG code here and I'll convert it to a React component.
|
||||
```
|
||||
|
||||
## File Handling
|
||||
|
||||
When your integration handles file uploads or downloads, follow these patterns to work with `UserFile` objects consistently.
|
||||
|
||||
### What is a UserFile?
|
||||
|
||||
A `UserFile` is the standard file representation in Sim:
|
||||
|
||||
```typescript
|
||||
interface UserFile {
|
||||
id: string // Unique identifier
|
||||
name: string // Original filename
|
||||
url: string // Presigned URL for download
|
||||
size: number // File size in bytes
|
||||
type: string // MIME type (e.g., 'application/pdf')
|
||||
base64?: string // Optional base64 content (if small file)
|
||||
key?: string // Internal storage key
|
||||
context?: object // Storage context metadata
|
||||
}
|
||||
```
|
||||
|
||||
### File Input Pattern (Uploads)
|
||||
|
||||
For tools that accept file uploads, **always route through an internal API endpoint** rather than calling external APIs directly. This ensures proper file content retrieval.
|
||||
|
||||
#### 1. Block SubBlocks for File Input
|
||||
|
||||
Use the basic/advanced mode pattern:
|
||||
|
||||
```typescript
|
||||
// Basic mode: File upload UI
|
||||
{
|
||||
id: 'uploadFile',
|
||||
title: 'File',
|
||||
type: 'file-upload',
|
||||
canonicalParamId: 'file', // Maps to 'file' param
|
||||
placeholder: 'Upload file',
|
||||
mode: 'basic',
|
||||
multiple: false,
|
||||
required: true,
|
||||
condition: { field: 'operation', value: 'upload' },
|
||||
},
|
||||
// Advanced mode: Reference from previous block
|
||||
{
|
||||
id: 'fileRef',
|
||||
title: 'File',
|
||||
type: 'short-input',
|
||||
canonicalParamId: 'file', // Same canonical param
|
||||
placeholder: 'Reference file (e.g., {{file_block.output}})',
|
||||
mode: 'advanced',
|
||||
required: true,
|
||||
condition: { field: 'operation', value: 'upload' },
|
||||
},
|
||||
```
|
||||
|
||||
**Critical:** `canonicalParamId` must NOT match any subblock `id`.
|
||||
|
||||
#### 2. Normalize File Input in Block Config
|
||||
|
||||
In `tools.config.tool`, use `normalizeFileInput` to handle all input variants:
|
||||
|
||||
```typescript
|
||||
import { normalizeFileInput } from '@/blocks/utils'
|
||||
|
||||
tools: {
|
||||
config: {
|
||||
tool: (params) => {
|
||||
// Normalize file from basic (uploadFile), advanced (fileRef), or legacy (fileContent)
|
||||
const normalizedFile = normalizeFileInput(
|
||||
params.uploadFile || params.fileRef || params.fileContent,
|
||||
{ single: true }
|
||||
)
|
||||
if (normalizedFile) {
|
||||
params.file = normalizedFile
|
||||
}
|
||||
return `{service}_${params.operation}`
|
||||
},
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
#### 3. Create Internal API Route
|
||||
|
||||
Create `apps/sim/app/api/tools/{service}/{action}/route.ts`. Internal tool routes are HTTP boundaries and follow the same contract policy as public routes — define the request/response shape in `apps/sim/lib/api/contracts/tools/{service}.ts` (or an existing aggregate) and validate with canonical helpers from `@/lib/api/server`. Never write a route-local Zod schema.
|
||||
|
||||
```typescript
|
||||
// apps/sim/lib/api/contracts/tools/{service}.ts
|
||||
import { z } from 'zod'
|
||||
import { defineRouteContract } from '@/lib/api/contracts'
|
||||
import { FileInputSchema } from '@/lib/uploads/utils/file-schemas'
|
||||
|
||||
export const {service}UploadBodySchema = z.object({
|
||||
accessToken: z.string(),
|
||||
file: FileInputSchema.optional().nullable(),
|
||||
fileContent: z.string().optional().nullable(),
|
||||
// ... other params
|
||||
})
|
||||
|
||||
export const {service}UploadResponseSchema = z.object({
|
||||
success: z.boolean(),
|
||||
output: z.object({ id: z.string(), url: z.string() }).optional(),
|
||||
error: z.string().optional(),
|
||||
})
|
||||
|
||||
export const {service}UploadContract = defineRouteContract({
|
||||
method: 'POST',
|
||||
path: '/api/tools/{service}/upload',
|
||||
body: {service}UploadBodySchema,
|
||||
response: { mode: 'json', schema: {service}UploadResponseSchema },
|
||||
})
|
||||
|
||||
export type {Service}UploadBody = z.input<typeof {service}UploadBodySchema>
|
||||
export type {Service}UploadResponse = z.output<typeof {service}UploadResponseSchema>
|
||||
```
|
||||
|
||||
```typescript
|
||||
// apps/sim/app/api/tools/{service}/upload/route.ts
|
||||
import { createLogger } from '@sim/logger'
|
||||
import { NextResponse, type NextRequest } from 'next/server'
|
||||
import { {service}UploadContract } from '@/lib/api/contracts/tools/{service}'
|
||||
import { parseRequest } from '@/lib/api/server'
|
||||
import { checkInternalAuth } from '@/lib/auth/hybrid'
|
||||
import { generateRequestId } from '@/lib/core/utils/request'
|
||||
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
|
||||
import { type RawFileInput } from '@/lib/uploads/utils/file-schemas'
|
||||
import { processFilesToUserFiles } from '@/lib/uploads/utils/file-utils'
|
||||
import { downloadFileFromStorage } from '@/lib/uploads/utils/file-utils.server'
|
||||
|
||||
const logger = createLogger('{Service}UploadAPI')
|
||||
|
||||
export const POST = withRouteHandler(async (request: NextRequest) => {
|
||||
const requestId = generateRequestId()
|
||||
|
||||
// Auth always runs BEFORE parseRequest — never validate untrusted input before authenticating.
|
||||
const authResult = await checkInternalAuth(request, { requireWorkflowId: false })
|
||||
if (!authResult.success) {
|
||||
return NextResponse.json({ success: false, error: 'Unauthorized' }, { status: 401 })
|
||||
}
|
||||
|
||||
const parsed = await parseRequest({service}UploadContract, request, {})
|
||||
if (!parsed.success) return parsed.response
|
||||
const data = parsed.data.body
|
||||
|
||||
let fileBuffer: Buffer
|
||||
let fileName: string
|
||||
|
||||
// Prefer UserFile input, fall back to legacy base64
|
||||
if (data.file) {
|
||||
const userFiles = processFilesToUserFiles([data.file as RawFileInput], requestId, logger)
|
||||
if (userFiles.length === 0) {
|
||||
return NextResponse.json({ success: false, error: 'Invalid file' }, { status: 400 })
|
||||
}
|
||||
const userFile = userFiles[0]
|
||||
fileBuffer = await downloadFileFromStorage(userFile, requestId, logger)
|
||||
fileName = userFile.name
|
||||
} else if (data.fileContent) {
|
||||
// Legacy: base64 string (backwards compatibility)
|
||||
fileBuffer = Buffer.from(data.fileContent, 'base64')
|
||||
fileName = 'file'
|
||||
} else {
|
||||
return NextResponse.json({ success: false, error: 'File required' }, { status: 400 })
|
||||
}
|
||||
|
||||
// Now call external API with fileBuffer
|
||||
const response = await fetch('https://api.{service}.com/upload', {
|
||||
method: 'POST',
|
||||
headers: { Authorization: `Bearer ${data.accessToken}` },
|
||||
body: new Uint8Array(fileBuffer), // Convert Buffer for fetch
|
||||
})
|
||||
|
||||
// ... handle response
|
||||
})
|
||||
```
|
||||
|
||||
#### 4. Update Tool to Use Internal Route
|
||||
|
||||
```typescript
|
||||
export const {service}UploadTool: ToolConfig<Params, Response> = {
|
||||
id: '{service}_upload',
|
||||
// ...
|
||||
params: {
|
||||
file: { type: 'file', required: false, visibility: 'user-or-llm' },
|
||||
fileContent: { type: 'string', required: false, visibility: 'hidden' }, // Legacy
|
||||
},
|
||||
request: {
|
||||
url: '/api/tools/{service}/upload', // Internal route
|
||||
method: 'POST',
|
||||
body: (params) => ({
|
||||
accessToken: params.accessToken,
|
||||
file: params.file,
|
||||
fileContent: params.fileContent,
|
||||
}),
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
### File Output Pattern (Downloads)
|
||||
|
||||
For tools that return files, use `FileToolProcessor` to store files and return `UserFile` objects.
|
||||
|
||||
#### In Tool transformResponse
|
||||
|
||||
```typescript
|
||||
import { FileToolProcessor } from '@/executor/utils/file-tool-processor'
|
||||
|
||||
transformResponse: async (response, context) => {
|
||||
const data = await response.json()
|
||||
|
||||
// Process file outputs to UserFile objects
|
||||
const fileProcessor = new FileToolProcessor(context)
|
||||
const file = await fileProcessor.processFileData({
|
||||
data: data.content, // base64 or buffer
|
||||
mimeType: data.mimeType,
|
||||
filename: data.filename,
|
||||
})
|
||||
|
||||
return {
|
||||
success: true,
|
||||
output: { file },
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### In API Route (for complex file handling)
|
||||
|
||||
```typescript
|
||||
// Return file data that FileToolProcessor can handle
|
||||
return NextResponse.json({
|
||||
success: true,
|
||||
output: {
|
||||
file: {
|
||||
data: base64Content,
|
||||
mimeType: 'application/pdf',
|
||||
filename: 'document.pdf',
|
||||
},
|
||||
},
|
||||
})
|
||||
```
|
||||
|
||||
### Key Helpers Reference
|
||||
|
||||
| Helper | Location | Purpose |
|
||||
|--------|----------|---------|
|
||||
| `normalizeFileInput` | `@/blocks/utils` | Normalize file params in block config |
|
||||
| `processFilesToUserFiles` | `@/lib/uploads/utils/file-utils` | Convert raw inputs to UserFile[] |
|
||||
| `downloadFileFromStorage` | `@/lib/uploads/utils/file-utils.server` | Get file Buffer from UserFile |
|
||||
| `FileToolProcessor` | `@/executor/utils/file-tool-processor` | Process tool output files |
|
||||
| `isUserFile` | `@/lib/core/utils/user-file` | Type guard for UserFile objects |
|
||||
| `FileInputSchema` | `@/lib/uploads/utils/file-schemas` | Zod schema for file validation |
|
||||
|
||||
### Advanced Mode for Optional Fields
|
||||
|
||||
Optional fields that are rarely used should be set to `mode: 'advanced'` so they don't clutter the basic UI. Examples: pagination tokens, time range filters, sort order, max results, reply settings.
|
||||
|
||||
### WandConfig for Complex Inputs
|
||||
|
||||
Use `wandConfig` for fields that are hard to fill out manually:
|
||||
- **Timestamps**: Use `generationType: 'timestamp'` to inject current date context into the AI prompt
|
||||
- **JSON arrays**: Use `generationType: 'json-object'` for structured data
|
||||
- **Complex queries**: Use a descriptive prompt explaining the expected format
|
||||
|
||||
```typescript
|
||||
{
|
||||
id: 'startTime',
|
||||
title: 'Start Time',
|
||||
type: 'short-input',
|
||||
mode: 'advanced',
|
||||
wandConfig: {
|
||||
enabled: true,
|
||||
prompt: 'Generate an ISO 8601 timestamp. Return ONLY the timestamp string.',
|
||||
generationType: 'timestamp',
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
### OAuth Scopes (Centralized System)
|
||||
|
||||
Scopes are maintained in a single source of truth and reused everywhere:
|
||||
|
||||
1. **Define scopes** in `lib/oauth/oauth.ts` under `OAUTH_PROVIDERS[provider].services[service].scopes`
|
||||
2. **Add descriptions** in `SCOPE_DESCRIPTIONS` within `lib/oauth/utils.ts` for the OAuth modal UI
|
||||
3. **Reference in auth.ts** using `getCanonicalScopesForProvider(providerId)` from `@/lib/oauth/utils`
|
||||
4. **Reference in blocks** using `getScopesForService(serviceId)` from `@/lib/oauth/utils`
|
||||
|
||||
**Never hardcode scope arrays** in `auth.ts` or block `requiredScopes`. Always import from the centralized source.
|
||||
|
||||
```typescript
|
||||
// In auth.ts (Better Auth config)
|
||||
scopes: getCanonicalScopesForProvider('{service}'),
|
||||
|
||||
// In block credential sub-block
|
||||
requiredScopes: getScopesForService('{service}'),
|
||||
```
|
||||
|
||||
### Common Gotchas
|
||||
|
||||
1. **OAuth serviceId must match** - The `serviceId` in oauth-input must match the OAuth provider configuration
|
||||
2. **All tool IDs MUST be snake_case** - `stripe_create_payment`, not `stripeCreatePayment`. This applies to tool `id` fields, registry keys, `tools.access` arrays, and `tools.config.tool` return values
|
||||
3. **Block type is snake_case** - `type: 'stripe'`, not `type: 'Stripe'`
|
||||
4. **Alphabetical ordering** - Keep imports and registry entries alphabetically sorted
|
||||
5. **Required can be conditional** - Use `required: { field: 'op', value: 'create' }` instead of always true
|
||||
6. **DependsOn clears options** - When a dependency changes, selector options are refetched
|
||||
7. **Never pass Buffer directly to fetch** - Convert to `new Uint8Array(buffer)` for TypeScript compatibility
|
||||
8. **Always handle legacy file params** - Keep hidden `fileContent` params for backwards compatibility
|
||||
9. **Optional fields use advanced mode** - Set `mode: 'advanced'` on rarely-used optional fields
|
||||
10. **Complex inputs need wandConfig** - Timestamps, JSON arrays, and other hard-to-type values should have `wandConfig` enabled
|
||||
11. **Never hardcode scopes** - Use `getScopesForService()` in blocks and `getCanonicalScopesForProvider()` in auth.ts
|
||||
12. **Always add scope descriptions** - New scopes must have entries in `SCOPE_DESCRIPTIONS` within `lib/oauth/utils.ts`
|
||||
13. **OAuth service IDs need deployment capabilities** - Every visible OAuth integration must resolve through `OAUTH_CLIENT_CAPABILITIES`; shared Google/Microsoft aliases map to their provider capability
|
||||
14. **Keep runtime and presentation separate** - Runtime OAuth fields live in `env-capabilities.ts`; CLI input modes live in the exhaustively checked `scripts/setup/capability-config.ts` mapping
|
||||
@@ -0,0 +1,5 @@
|
||||
interface:
|
||||
display_name: "Add Integration"
|
||||
short_description: "Build a full Sim integration"
|
||||
brand_color: "#7C3AED"
|
||||
default_prompt: "Use $add-integration to add a complete Sim integration for a service."
|
||||
@@ -0,0 +1,149 @@
|
||||
---
|
||||
name: add-managed-cli
|
||||
description: Add or upgrade a curated, immutable managed CLI for Sim Function sandboxes, including client-safe catalog metadata, a pinned server-only installation recipe, checksum and executable verification, provider compatibility, PATH propagation, content-addressed image identity, and tests. Use when adding a CLI to the Sandbox managed-CLI selector or changing an existing managed CLI version or recipe.
|
||||
---
|
||||
|
||||
# Add a Managed CLI
|
||||
|
||||
Add CLIs through the curated registry. Never turn this surface into arbitrary commands or package names: system packages already cover validated Debian/APT coordinates, while managed CLIs require immutable artifacts and reproducible recipes.
|
||||
|
||||
## Read First
|
||||
|
||||
Read these live sources before editing; do not copy their current entries into this skill:
|
||||
|
||||
1. `apps/sim/lib/execution/remote-sandbox/cli-tools.ts` — persisted IDs and client-safe metadata.
|
||||
2. `apps/sim/lib/execution/remote-sandbox/cli-tools.server.ts` — server-only recipes and recipe helpers.
|
||||
3. `apps/sim/lib/execution/remote-sandbox/cli-tools.test.ts` — catalog and supply-chain invariants.
|
||||
4. `apps/sim/lib/execution/remote-sandbox/cli-tools-boundary.test.ts` — client/server import boundary.
|
||||
5. `apps/sim/lib/execution/remote-sandbox/sandbox-spec.ts` — content-addressed hash inputs.
|
||||
|
||||
Read `resolve.ts` and `e2b.ts` only when changing provisioning mechanics. A normal catalog addition should not require UI, API, database, resolver, or provider edits; those paths derive from the registries.
|
||||
|
||||
Do not modify the dedicated Function base image or the separate Mothership Shell template for a normal managed CLI addition. Managed recipes layer on the Function base image. Do not change `MAX_SANDBOX_CLI_TOOLS` from ten unless the user separately requests a product-limit change.
|
||||
|
||||
## 1. Verify the Upstream Release
|
||||
|
||||
Use primary upstream release documentation and official artifacts. Establish all of the following before writing code:
|
||||
|
||||
- Exact version and stable Linux x86-64 artifact URL. Never use `latest`, mutable redirects, or an unversioned installer.
|
||||
- SHA-256 for the exact artifact. Prefer a publisher-signed checksum; otherwise download the official artifact and compute it independently.
|
||||
- Archive layout and the exact executable paths to install.
|
||||
- Noninteractive, credential-free verification commands for every advertised executable, normally version commands.
|
||||
- Required PATH entries under `/opt/sim-cli`.
|
||||
- E2B and Daytona compatibility. Default to both only when the same Linux recipe works on both.
|
||||
|
||||
When the user does not name a version, select the current stable upstream release from primary sources and state the exact version chosen. Do not silently choose a prerelease or infer a version from an unverified secondary source.
|
||||
|
||||
Reject curl-to-shell installers, `npm install`, `pip install`, distro package repositories, arbitrary user commands, and artifacts from unofficial mirrors. Never place credentials, tokens, login commands, or account configuration in an image recipe or build log; authentication is runtime-only.
|
||||
|
||||
## 2. Choose an Immutable ID
|
||||
|
||||
Use `<tool>@<upstream-version>-r<recipe-revision>`.
|
||||
|
||||
- New upstream version: append a new ID ending in `-r1`.
|
||||
- Recipe-only change for the same upstream version: append `-r2`, `-r3`, and so on.
|
||||
- Never mutate or delete an existing ID or recipe. Persisted sandboxes must continue resolving to the bytes and behavior they selected.
|
||||
- On upgrade, retain the old ID and recipe and set its metadata to `selectable: false`. Only the newest version keeps the public label selectable.
|
||||
|
||||
Before shipping the first upgrade for a tool family, verify that editing a sandbox cannot leave both the retired and replacement IDs selected. If the generic selector and API validation do not already replace or reject colliding versions, address that once at the generic registry boundary with focused UI and contract tests; never special-case the individual CLI or silently install two versions that expose the same executable.
|
||||
|
||||
Recipe identity includes the ID, revision, and SHA-256 in the sandbox image hash. Keeping old entries is what makes that identity reproducible rather than merely cache-busting.
|
||||
|
||||
## 3. Add Client-Safe Metadata
|
||||
|
||||
In `cli-tools.ts`:
|
||||
|
||||
1. Append the ID to `SANDBOX_CLI_TOOL_IDS` in the same order used by the metadata and recipe registries.
|
||||
2. Add a `SANDBOX_CLI_TOOLS` entry whose key and `id` exactly match.
|
||||
3. Provide a unique selectable `label`, concise `description`, existing `category`, and useful executable/vendor aliases in `searchTerms`.
|
||||
4. Add a category only when no existing category is accurate, then ensure it has at least one selectable entry.
|
||||
|
||||
Keep this file safe for client bundles. It must not contain artifact URLs, checksums, install commands, verification commands, PATH recipes, provider SDKs, or imports from `cli-tools.server.ts`.
|
||||
|
||||
The API enum and searchable grouped selector derive from this registry. Do not add parallel option arrays or route-local wire types.
|
||||
|
||||
## 4. Add the Server-Only Recipe
|
||||
|
||||
In `cli-tools.server.ts`, use the narrowest existing helper:
|
||||
|
||||
- `defineBinaryRecipe` for one downloaded binary.
|
||||
- `defineTarGzipRecipe` or `defineZipRecipe` for archives containing binaries.
|
||||
- `defineVerifiedRecipe` for a vendor archive or installer layout that needs explicit commands.
|
||||
- A direct typed entry only when the helpers cannot faithfully model the release.
|
||||
|
||||
Provide every field the recipe contract requires:
|
||||
|
||||
- Exact `version`, `artifactUrl`, `artifactName`, and lowercase 64-character `sha256`.
|
||||
- Every installed `executable` and a corresponding `verificationCommands` entry.
|
||||
- Deterministic extraction/install commands into `/opt/sim-cli`; quote fixed paths and clean temporary artifacts.
|
||||
- `pathEntries` when the executable is not installed into the helper's default `bin` directory.
|
||||
- `supportedProviders` only when it differs from the E2B-and-Daytona default.
|
||||
- `revision` when it differs from `1`; it must agree with the ID suffix.
|
||||
|
||||
Verification must prove the command is discoverable through `sandboxCliEnvironment`, not authenticate or contact a user account. Recipe commands run as root during both prebuilt image creation and runtime provisioning.
|
||||
|
||||
If the artifact host is new, add only the exact official hostname to the `officialHosts` allowlist in `cli-tools.test.ts`. Treat that as a supply-chain review, not a way to silence the test.
|
||||
|
||||
## 5. Preserve Generic Behavior
|
||||
|
||||
Confirm the existing generic paths remain sufficient:
|
||||
|
||||
- `sandboxCliToolRecipes` canonicalizes and resolves the recipe.
|
||||
- `sandboxCliEnvironment` propagates PATH to Python subprocesses, JavaScript subprocesses, and Shell.
|
||||
- E2B bakes the recipe into the custom image; runtime-strategy providers install it within the Function timeout.
|
||||
- CLI-only sandboxes remain buildable even with no language packages.
|
||||
- `hashSandboxSpec` includes recipe ID, revision, and checksum while preserving the legacy hash for an empty CLI list.
|
||||
- The settings selector derives groups and search aliases from client-safe metadata.
|
||||
|
||||
Do not special-case a CLI in those layers unless the registry contract cannot express a genuine provider requirement. Extend the registry contract generically when multiple CLIs need the same new behavior.
|
||||
|
||||
## 6. Test the Addition
|
||||
|
||||
Extend tests when the new entry introduces behavior not already covered:
|
||||
|
||||
- For every upgrade, add a regression proving the old ID and recipe remain resolvable but non-selectable, while the replacement ID is selectable.
|
||||
- Add important executable aliases to the table-driven search assertion.
|
||||
- Add a focused assertion for a multi-executable recipe, custom PATH, or restricted provider.
|
||||
- Add an opt-in credentialed smoke test only when installation plus a real minimal command cannot be validated without authentication. Read credentials from test-only environment variables, skip by default, create them only at runtime, and always tear down the sandbox.
|
||||
|
||||
Never commit downloaded artifacts or credentials.
|
||||
|
||||
## Required Validation
|
||||
|
||||
From `apps/sim`:
|
||||
|
||||
```bash
|
||||
bunx vitest run \
|
||||
lib/execution/remote-sandbox/cli-tools.test.ts \
|
||||
lib/execution/remote-sandbox/cli-tools-boundary.test.ts \
|
||||
lib/execution/remote-sandbox/sandbox-spec.test.ts \
|
||||
lib/execution/remote-sandbox/resolve.test.ts \
|
||||
lib/api/contracts/sandboxes.test.ts \
|
||||
'app/workspace/[workspaceId]/settings/components/sandboxes/utils.test.ts' \
|
||||
'app/workspace/[workspaceId]/settings/components/sandboxes/components/sandbox-editor.test.tsx'
|
||||
```
|
||||
|
||||
From the repository root:
|
||||
|
||||
```bash
|
||||
bun run type-check
|
||||
bun run check:api-validation
|
||||
bunx biome check \
|
||||
apps/sim/lib/execution/remote-sandbox/cli-tools.ts \
|
||||
apps/sim/lib/execution/remote-sandbox/cli-tools.server.ts \
|
||||
apps/sim/lib/execution/remote-sandbox/cli-tools.test.ts
|
||||
git diff --check
|
||||
```
|
||||
|
||||
For a new recipe, also exercise its install and every verification command in an actual E2B or Daytona sandbox when credentials and network access are available. Report clearly when only registry/unit validation ran.
|
||||
|
||||
## Completion Checklist
|
||||
|
||||
- [ ] Official immutable Linux x86-64 artifact and SHA-256 verified.
|
||||
- [ ] Versioned ID appended; old IDs and recipes retained.
|
||||
- [ ] Client metadata is searchable, categorized, unique, and recipe-free.
|
||||
- [ ] Server recipe is pinned, integrity-checked, noninteractive, and credential-free.
|
||||
- [ ] Every advertised executable has an offline verification command and PATH entry.
|
||||
- [ ] Provider compatibility is explicit and accurate.
|
||||
- [ ] Catalog, boundary, hash, resolver, type, API-validation, format, and diff checks pass.
|
||||
- [ ] Real provider installation was tested, or the missing live verification is disclosed.
|
||||
@@ -0,0 +1,4 @@
|
||||
interface:
|
||||
display_name: "Add Managed CLI"
|
||||
short_description: "Add a verified CLI to sandbox images"
|
||||
default_prompt: "Use $add-managed-cli to add a pinned, verified CLI to Sim sandbox images."
|
||||
@@ -0,0 +1,222 @@
|
||||
---
|
||||
name: add-model
|
||||
description: Add a new LLM model to apps/sim/providers/models.ts with specs verified against the provider's live API docs (no hallucination)
|
||||
argument-hint: <provider> <model-id> [docs-url]
|
||||
---
|
||||
|
||||
# Add Model Skill
|
||||
|
||||
You add a new model entry to `apps/sim/providers/models.ts`. **Every numeric and capability claim MUST be derived from a live web fetch of the provider's official docs in this session.** Marketing emails, training data, and your prior knowledge are not sources of truth — they routinely hallucinate pricing, context windows, and capability lists.
|
||||
|
||||
## Hard rules (do not skip)
|
||||
|
||||
1. **Live-fetch or refuse.** Before writing the entry, you must successfully WebFetch the provider's official models/pricing page in this session. If you cannot reach an authoritative source for any field, **mark the field as UNVERIFIED in your report and ask the user before guessing**. Never fill in pricing or capabilities from memory.
|
||||
2. **Two-source rule for pricing.** Cross-check input/output/cached pricing against at least one secondary source (OpenRouter, Artificial Analysis, CloudPrice, mem0, intuitionlabs). If sources disagree, the provider's own docs win — but flag the disagreement.
|
||||
3. **Read the code before setting capability flags.** Capability flags are dead unless the provider's implementation under `apps/sim/providers/{provider}/` actually consumes them (see Consumption Matrix below). Setting a flag the provider ignores is a silent bug.
|
||||
4. **Cite every fact.** Your final report must list the URL each value came from. No URL → not verified.
|
||||
|
||||
## Your Task
|
||||
|
||||
1. Identify provider and model id from user args
|
||||
2. Live-fetch official docs + pricing page + capability/parameter pages + at least one secondary source
|
||||
3. Apply the Consumption Matrix to know which capability flags are real
|
||||
4. Read 2-3 sibling entries in `models.ts` and match their pattern exactly
|
||||
5. Check the repo-side touchpoints that are NOT data-driven (hosted-key billing, tests, provider code)
|
||||
6. Insert the entry, run `bun run lint`, print the verification report
|
||||
|
||||
## Step 1: Live source-of-truth lookup
|
||||
|
||||
In priority order — fetch all that exist for the provider:
|
||||
|
||||
| Provider | Models index | Pricing | Reasoning/parameter caveats |
|
||||
|---|---|---|---|
|
||||
| OpenAI | platform.openai.com/docs/models | openai.com/api/pricing | platform.openai.com/docs/guides/reasoning |
|
||||
| Anthropic | docs.anthropic.com/en/docs/about-claude/models | anthropic.com/pricing | docs.anthropic.com/en/docs/build-with-claude/extended-thinking |
|
||||
| Google (Gemini) | ai.google.dev/gemini-api/docs/models | ai.google.dev/pricing | ai.google.dev/gemini-api/docs/thinking |
|
||||
| xAI | docs.x.ai/developers/models | docs.x.ai/developers/models (per-model detail page) | docs.x.ai/developers/model-capabilities/text/reasoning |
|
||||
| Mistral | docs.mistral.ai/getting-started/models/models_overview | mistral.ai/pricing | n/a |
|
||||
| DeepSeek | api-docs.deepseek.com/quick_start/pricing | same | api-docs.deepseek.com/guides/reasoning_model |
|
||||
| Groq | console.groq.com/docs/models | groq.com/pricing | n/a |
|
||||
| Cerebras | inference-docs.cerebras.ai/models | cerebras.ai/pricing | n/a |
|
||||
|
||||
Secondary verification (use at least one): `openrouter.ai/<provider>/<model>`, `artificialanalysis.ai/models/<model>`, `cloudprice.net/models/<provider>-<model>`.
|
||||
|
||||
Use a precise WebFetch prompt: *"Extract for {model_id}: exact model id string, context window in tokens, input price per 1M, cached input price per 1M, output price per 1M, max output tokens, supported reasoning effort levels, accepted parameters (temperature, top_p), release date. Do not fill in fields you cannot find."*
|
||||
|
||||
## Step 2: Consumption Matrix (which provider honors which capability)
|
||||
|
||||
| Capability | Honored by | Effect if set elsewhere |
|
||||
|---|---|---|
|
||||
| `temperature` | All providers (passed through if set) | Safe but inert on always-reasoning models that reject it |
|
||||
| `toolUsageControl` | All providers (provider-level, not per-model) | n/a — set on `ProviderDefinition`, not models |
|
||||
| `reasoningEffort` | `openai/core.ts`, `azure-openai`, `anthropic/core.ts` (mapped to thinking), `gemini/core.ts` | **Dead on xai, deepseek, mistral, groq, cerebras, openrouter, fireworks, bedrock, vertex** unless their core consumes it — re-grep before assuming |
|
||||
| `verbosity` | `openai/core.ts`, `azure-openai/index.ts` only | Dead elsewhere |
|
||||
| `thinking` | `anthropic/core.ts`, `gemini/core.ts` | Dead elsewhere |
|
||||
| `thinking.streamed` | Docs generator + `getThinkingStreamVisibility` (`models.ts`); `anthropic/core.ts` uses `'summary'` to request `display: 'summarized'` on agent-events runs | **Mandatory on Anthropic-family thinking models** (`agent-stream-docs:check` fails without it); other families fall back to provider defaults |
|
||||
| `nativeStructuredOutputs` | `anthropic/core.ts`, `fireworks/index.ts`, `openrouter/index.ts` | Dead on openai, xai, google, vertex, bedrock, azure-openai, deepseek, mistral, groq, cerebras |
|
||||
| `maxOutputTokens` | Read by UI + executor for token estimation | Always meaningful — set if provider documents a cap |
|
||||
| `computerUse` | `anthropic/core.ts` | Dead elsewhere |
|
||||
| `deepResearch` | UI flag for routing to deep-research SKUs | Set only on actual deep-research model IDs |
|
||||
| `memory: false` | Conversation persistence opt-out | Set only when model genuinely cannot maintain history (e.g., deep-research) |
|
||||
|
||||
**Always re-grep before relying on this table** — the codebase moves:
|
||||
|
||||
```bash
|
||||
rg "reasoningEffort|reasoning_effort" apps/sim/providers/<provider>/
|
||||
rg "verbosity" apps/sim/providers/<provider>/
|
||||
rg "request\.thinking|thinking:" apps/sim/providers/<provider>/
|
||||
rg "supportsNativeStructuredOutputs|nativeStructuredOutputs" apps/sim/providers/<provider>/
|
||||
```
|
||||
|
||||
## Step 3: Match the provider's existing entry pattern
|
||||
|
||||
Open `apps/sim/providers/models.ts`, find `PROVIDER_DEFINITIONS[<provider>].models`, read 2-3 sibling entries. Match field order exactly:
|
||||
|
||||
```ts
|
||||
{
|
||||
id: '<exact-api-id>',
|
||||
pricing: {
|
||||
input: <number>,
|
||||
cachedInput: <number>, // omit if provider doesn't offer caching
|
||||
output: <number>,
|
||||
updatedAt: '<today YYYY-MM-DD>',
|
||||
},
|
||||
capabilities: {
|
||||
// only flags the provider actually consumes — see matrix
|
||||
},
|
||||
contextWindow: <tokens>,
|
||||
releaseDate: '<YYYY-MM-DD>',
|
||||
recommended: true, // only if new flagship; ask user before swapping
|
||||
speedOptimized: true, // only on smallest/fastest tier
|
||||
deprecated: true, // only on retired models
|
||||
}
|
||||
```
|
||||
|
||||
### Reseller providers (azure-openai, azure-anthropic, vertex, bedrock, openrouter)
|
||||
|
||||
Model id MUST be prefixed: `azure/`, `azure-anthropic/`, `vertex/`, `bedrock/`, `openrouter/`. Pricing usually mirrors the upstream provider but verify on the reseller's own pricing page.
|
||||
|
||||
### Insertion order
|
||||
|
||||
Within a family, newest first (matches existing convention: GPT-5.5 above GPT-5.4 above GPT-5.2). Across families, biggest/flagship at top of list.
|
||||
|
||||
### `recommended` / `speedOptimized`
|
||||
|
||||
- At most one or two `recommended: true` per provider — the current flagship(s).
|
||||
- If you're adding a new flagship, ask the user before removing `recommended` from the previous flagship. Never silently flip it.
|
||||
- `speedOptimized: true` only on the smallest/fastest tier (nano, flash-lite, haiku class).
|
||||
|
||||
## Step 4: Repo-side touchpoints beyond the entry
|
||||
|
||||
Adding the `models.ts` entry is most of the job because nearly every consumer is **data-driven** and picks the model up automatically: the ~40 query helpers in `models.ts` / `providers/utils.ts`, the public `/models` catalog (`app/(landing)/models/utils.ts` iterates `PROVIDER_DEFINITIONS`), the agent-block model dropdown, and copilot's `isKnownModelId` / `suggestModelIdsForUnknownModel` validation. The touchpoints below are the exceptions — they are **not** data-driven, so check each one.
|
||||
|
||||
### Hosted = auto-billed, by provider
|
||||
|
||||
`getHostedModels()` in `apps/sim/providers/models.ts` returns **every** model under `openai`, `anthropic`, and `google`:
|
||||
|
||||
```ts
|
||||
export function getHostedModels(): string[] {
|
||||
return [
|
||||
...getProviderModels('openai'),
|
||||
...getProviderModels('anthropic'),
|
||||
...getProviderModels('google'),
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
So a model added to any of those three providers is **automatically served with Sim's rotating hosted key and billed** to the workspace via `shouldBillModelUsage()` (`providers/utils.ts`). Before you insert:
|
||||
|
||||
- **If the model should be BYOK-only / never-billed**, do NOT drop it under `openai`/`anthropic`/`google` as-is — that silently enrolls it in hosted billing. Confirm hosting/billing intent with the user. (Precedent: Ollama Cloud is a deliberately separate `isReseller` provider specifically to stay BYOK-only/never-billed.)
|
||||
- **If the model should be hosted**, the deployment must actually have a key for it — the provider's `{PREFIX}_COUNT` / `{PREFIX}_1..N` env vars must be set, or hosted runs fail at execution time.
|
||||
- State the hosted/billing status explicitly in the verification report.
|
||||
|
||||
### Tests with hardcoded model IDs
|
||||
|
||||
`bun run lint` does **not** run tests. A few tests assert specific model IDs and can break or need updating when you touch a hosted or flagship model:
|
||||
|
||||
- `apps/sim/providers/utils.test.ts` — asserts membership of `getHostedModels()` / `shouldBillModelUsage()`
|
||||
- `apps/sim/providers/index.test.ts` and serializer tests — reference concrete model IDs
|
||||
|
||||
```bash
|
||||
rg "<new-model-id>|getHostedModels|shouldBillModelUsage" apps/sim/providers/*.test.ts
|
||||
```
|
||||
|
||||
If anything matches, run the affected provider tests and update assertions as needed.
|
||||
|
||||
### New API behavior is NOT data-driven
|
||||
|
||||
The Consumption Matrix (Step 2) tells you which capability *flags* are honored by existing provider code. But if the new model needs **net-new** request handling that the provider doesn't implement yet — a new beta header (e.g. Anthropic's `anthropic-beta` structured-outputs header in `anthropic/index.ts`), a new thinking/reasoning encoding, a Responses-API quirk — you must edit `apps/sim/providers/<provider>/core.ts` / `index.ts`. Setting a flag whose behavior isn't implemented is a silent no-op. When you do edit provider code, reuse the shared helpers rather than hand-rolling: streaming responses are assembled via `createStreamingExecution` (`@/providers/streaming-execution`) and tool schemas via `adaptOpenAIChatToolSchema` / `adaptAnthropicToolSchema` (`@/providers/tool-schema-adapter`).
|
||||
|
||||
### Thinking/reasoning models: `streamed` visibility + generated docs
|
||||
|
||||
If the entry has `capabilities.thinking` or `capabilities.reasoningEffort`, it appears in the autogenerated "Streamed thinking and tool calls" table on the Agent block docs page:
|
||||
|
||||
- **Anthropic-family (`anthropic`, `azure-anthropic`) thinking models MUST declare `capabilities.thinking.streamed`** (`'full' | 'summary' | 'none'`). Verify against Anthropic's current thinking-display and streaming docs: visible thinking returned by the API is summarized, including when Sim opts models whose default display is `omitted` into `display: 'summarized'` on agent-events runs, so current Claude thinking models use `'summary'`. Use `'full'` only if future official API docs explicitly guarantee raw thinking deltas. `bun run agent-stream-docs:check` (CI) fails if the field is missing.
|
||||
- Other families usually omit the field and inherit the provider default in `getThinkingStreamVisibility` (Gemini/OpenAI → summaries; Bedrock/Meta → none; OpenAI-compatible vendors with documented reasoning fields → full deltas). Set it explicitly only when the model deviates from its family.
|
||||
- After inserting the entry, run `bun run agent-stream-docs:generate` and commit the regenerated `apps/docs/content/docs/en/workflows/blocks/agent.mdx` — CI diffs it.
|
||||
- Include the `streamed` value (with its source URL) in the verification report when set.
|
||||
|
||||
### Wrong family entirely?
|
||||
|
||||
- **Embedding or rerank model** → it does NOT go in the `models[]` array. Use `EMBEDDING_MODEL_PRICING` / `RERANK_MODEL_PRICING` in `models.ts` instead.
|
||||
- **Brand-new provider** (not just a new model under an existing one) → much larger surface: add the id to `ProviderId` in `providers/types.ts`, a registry entry in `providers/registry.ts`, a provider implementation under `providers/<id>/` (assemble streaming responses with `createStreamingExecution` and wrap tool schemas with the `@/providers/tool-schema-adapter` helpers), an icon in `components/icons.tsx`, and the `PROVIDER_DEFINITIONS` block. That is beyond this skill — tell the user.
|
||||
|
||||
## Step 5: Write, lint
|
||||
|
||||
```bash
|
||||
bun run lint
|
||||
bun run agent-stream-docs:generate # only when the entry has thinking/reasoningEffort
|
||||
```
|
||||
|
||||
Lint must pass before reporting done. **If lint fails:** read the error, fix the syntax/typing issue in the entry you just wrote (do not delete the entry — it's the work product), re-run lint, and note the fix in a "Lint adjustments" line in the verification report. Never report done with lint failing.
|
||||
|
||||
## Step 6: Verification report (mandatory format)
|
||||
|
||||
End with this exact structure:
|
||||
|
||||
```markdown
|
||||
### Verification — <model-id>
|
||||
|
||||
| Field | Value | Source URL | Status |
|
||||
|---|---|---|---|
|
||||
| `id` | `grok-4.3` | https://docs.x.ai/... | ✓ verified |
|
||||
| `contextWindow` | 1,000,000 | https://docs.x.ai/... + https://openrouter.ai/... | ✓ verified (2 sources agree) |
|
||||
| `input` | $1.25/M | https://docs.x.ai/... | ✓ verified |
|
||||
| `cachedInput` | $0.20/M | https://cloudprice.net/... | ⚠️ single source |
|
||||
| `output` | $2.50/M | https://docs.x.ai/... + https://openrouter.ai/... | ✓ verified |
|
||||
| `capabilities.temperature` | `{ min: 0, max: 1 }` | matches sibling entries | — pattern-match only |
|
||||
| `capabilities.reasoningEffort` | NOT SET | provider docs say API rejects it for this model | ✓ correctly omitted |
|
||||
| `releaseDate` | 2026-04-30 | https://docs.x.ai/... announcement | ✓ verified |
|
||||
| hosted/billing | BYOK-only (xai not in `getHostedModels`) | `providers/models.ts` | — confirmed intent |
|
||||
|
||||
**Disagreements**
|
||||
- _none_ OR _OpenRouter says X, provider docs say Y — used Y per provider rule_
|
||||
|
||||
**Unverified fields**
|
||||
- _none_ OR _<field>: could not find authoritative source — left as <X> based on sibling pattern; please confirm_
|
||||
```
|
||||
|
||||
If any row is ⚠️ single-source or "unverified," **state it plainly to the user and ask whether to proceed**. Do not silently merge.
|
||||
|
||||
## What to do if you cannot find a source
|
||||
|
||||
Omitting a field is **not the same as verifying it**. Any field you cannot confirm from a live fetch must be **both** omitted from the entry **and** listed as ❓ UNVERIFIED in the report's "Unverified fields" section, with the URLs you attempted. Then ask the user to confirm before merging.
|
||||
|
||||
- Pricing missing → do NOT guess. Omit `cachedInput`. Mark ❓ UNVERIFIED. Ask the user for the price or the docs URL.
|
||||
- Context window missing → do NOT guess. Ask the user; mark ❓ UNVERIFIED.
|
||||
- Release date missing → omit the field; mark ❓ UNVERIFIED in the report.
|
||||
- Capability uncertain → omit the flag (safer than setting a dead/wrong one); mark ❓ UNVERIFIED so the user knows you didn't confirm it either way.
|
||||
|
||||
## Anti-patterns this skill exists to prevent
|
||||
|
||||
- ❌ Trusting a marketing email (xAI's grok-4.3 email claimed "3 reasoning efforts" but the API rejects `reasoning_effort` — verified by official docs only)
|
||||
- ❌ Setting `nativeStructuredOutputs: true` on xai/openai/google (dead — only anthropic/fireworks/openrouter consume it)
|
||||
- ❌ Setting `thinking` on non-Anthropic/non-Gemini providers
|
||||
- ❌ Adding an Anthropic-family thinking model without `thinking.streamed` (CI `agent-stream-docs:check` fails), or skipping `bun run agent-stream-docs:generate` after adding any thinking/reasoning model
|
||||
- ❌ Setting `verbosity` on anything other than OpenAI gpt-5.x
|
||||
- ❌ Copying `pricing.updatedAt` from a sibling instead of using today's date
|
||||
- ❌ Inventing a `cachedInput` price by dividing input by 4 (varies by provider — find an explicit number)
|
||||
- ❌ Stamping `recommended: true` on the new model without removing it from the previous flagship
|
||||
- ❌ Adding a BYOK-only model under `openai`/`anthropic`/`google` (silently enrolls it in hosted billing via `getHostedModels()`)
|
||||
- ❌ Reporting "done" after only `bun run lint` when you touched a hosted (openai/anthropic/google) or flagship model with assertions in `providers/utils.test.ts`
|
||||
- ❌ Reporting "done" with any UNVERIFIED row in the table
|
||||
@@ -0,0 +1,5 @@
|
||||
interface:
|
||||
display_name: "Add Model"
|
||||
short_description: "Add an LLM model, specs verified"
|
||||
brand_color: "#0EA5E9"
|
||||
default_prompt: "Use $add-model to add a new LLM model to Sim with pricing and capabilities verified against the provider's live docs."
|
||||
@@ -0,0 +1,502 @@
|
||||
---
|
||||
name: add-tools
|
||||
description: Create tool configurations for a Sim integration by reading API docs
|
||||
argument-hint: <service-name> [api-docs-url]
|
||||
---
|
||||
|
||||
# Add Tools Skill
|
||||
|
||||
You are an expert at creating tool configurations for Sim integrations. Your job is to read API documentation and create properly structured tool files.
|
||||
|
||||
## Your Task
|
||||
|
||||
When the user asks you to create tools for a service:
|
||||
1. Use Context7 or WebFetch to read the service's API documentation
|
||||
2. Create the tools directory structure
|
||||
3. Generate properly typed tool configurations
|
||||
|
||||
## Hard Rule: No Guessed Response Schemas
|
||||
|
||||
If the docs do not clearly show the response JSON for a tool, you MUST tell the user exactly which outputs are unknown and stop short of guessing.
|
||||
|
||||
- Do NOT invent response field names
|
||||
- Do NOT infer nested paths from nearby endpoints
|
||||
- Do NOT guess array item shapes
|
||||
- Do NOT write `transformResponse` against unverified payloads
|
||||
|
||||
If the response shape is unknown, do one of these instead:
|
||||
1. Ask the user for sample responses
|
||||
2. Ask the user for test credentials so you can verify live responses
|
||||
3. Implement only the endpoints whose outputs are documented
|
||||
4. Leave the tool unimplemented and explicitly say why
|
||||
|
||||
## Directory Structure
|
||||
|
||||
Create files in `apps/sim/tools/{service}/`:
|
||||
```
|
||||
tools/{service}/
|
||||
├── index.ts # Barrel export
|
||||
├── types.ts # Parameter & response types
|
||||
└── {action}.ts # Individual tool files (one per operation)
|
||||
```
|
||||
|
||||
## Tool Configuration Structure
|
||||
|
||||
Every tool MUST follow this exact structure:
|
||||
|
||||
```typescript
|
||||
import type { {ServiceName}{Action}Params } from '@/tools/{service}/types'
|
||||
import type { ToolConfig } from '@/tools/types'
|
||||
|
||||
interface {ServiceName}{Action}Response {
|
||||
success: boolean
|
||||
output: {
|
||||
// Define output structure here
|
||||
}
|
||||
}
|
||||
|
||||
export const {serviceName}{Action}Tool: ToolConfig<
|
||||
{ServiceName}{Action}Params,
|
||||
{ServiceName}{Action}Response
|
||||
> = {
|
||||
id: '{service}_{action}', // snake_case, matches tool name
|
||||
name: '{Service} {Action}', // Human readable
|
||||
description: 'Brief description', // One sentence
|
||||
version: '1.0.0',
|
||||
|
||||
// OAuth config (if service uses OAuth)
|
||||
oauth: {
|
||||
required: true,
|
||||
provider: '{service}', // Must match OAuth provider ID
|
||||
},
|
||||
|
||||
params: {
|
||||
// Hidden params (system-injected, only use hidden for oauth accessToken)
|
||||
accessToken: {
|
||||
type: 'string',
|
||||
required: true,
|
||||
visibility: 'hidden',
|
||||
description: 'OAuth access token',
|
||||
},
|
||||
// User-only params (credentials, api key, IDs user must provide)
|
||||
someId: {
|
||||
type: 'string',
|
||||
required: true,
|
||||
visibility: 'user-only',
|
||||
description: 'The ID of the resource',
|
||||
},
|
||||
// User-or-LLM params (everything else, can be provided by user OR computed by LLM)
|
||||
query: {
|
||||
type: 'string',
|
||||
required: false, // Use false for optional
|
||||
visibility: 'user-or-llm',
|
||||
description: 'Search query',
|
||||
},
|
||||
},
|
||||
|
||||
request: {
|
||||
url: (params) => `https://api.service.com/v1/resource/${params.id}`,
|
||||
method: 'POST',
|
||||
headers: (params) => ({
|
||||
Authorization: `Bearer ${params.accessToken}`,
|
||||
'Content-Type': 'application/json',
|
||||
}),
|
||||
body: (params) => ({
|
||||
// Request body - only for POST/PUT/PATCH
|
||||
// Trim ID fields to prevent copy-paste whitespace errors:
|
||||
// userId: params.userId?.trim(),
|
||||
}),
|
||||
},
|
||||
|
||||
transformResponse: async (response: Response) => {
|
||||
const data = await response.json()
|
||||
return {
|
||||
success: true,
|
||||
output: {
|
||||
// Map API response to output
|
||||
// Use ?? null for nullable fields
|
||||
// Use ?? [] for optional arrays
|
||||
},
|
||||
}
|
||||
},
|
||||
|
||||
outputs: {
|
||||
// Define each output field
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
## Critical Rules for Parameters
|
||||
|
||||
### Visibility Options
|
||||
- `'hidden'` - System-injected (OAuth tokens, internal params). User never sees.
|
||||
- `'user-only'` - User must provide (credentials, api keys, account-specific IDs)
|
||||
- `'user-or-llm'` - User provides OR LLM can compute (search queries, content, filters, most fall into this category)
|
||||
|
||||
### Parameter Types
|
||||
- `'string'` - Text values
|
||||
- `'number'` - Numeric values
|
||||
- `'boolean'` - True/false
|
||||
- `'json'` - Complex objects (NOT 'object', use 'json')
|
||||
- `'file'` - Single file
|
||||
- `'file[]'` - Multiple files
|
||||
|
||||
### Required vs Optional
|
||||
- Always explicitly set `required: true` or `required: false`
|
||||
- Optional params should have `required: false`
|
||||
|
||||
## Resolved Secrets and Provenance Boundaries
|
||||
|
||||
- Leave ordinary external API inputs and third-party results unchanged. Add provenance handling only
|
||||
when an exact field is proven to cross a Sim model, durable-storage, or internal-execution boundary.
|
||||
- Project AI-consumed text/structured fields with the smallest exact `request.modelInput` selector.
|
||||
- Treat URLs, domains, resource IDs, and control fields as ordinary request values unless the exact
|
||||
field is proven model-visible. For serialized external model content, project the serialized
|
||||
top-level param through `request.modelInput` before the existing formatter parses it; do not add a
|
||||
separate hard-rejection mechanism.
|
||||
- For authenticated internal routes, use `privateProvenance` for actual inline/raw model bytes or
|
||||
`request.secretProvenance` for durable writes and execution handoffs. Do not treat a storage key,
|
||||
path, signed URL, or remote URL as provenance for fetched bytes; authorize tracked stored bytes at
|
||||
the owning model-egress boundary. Authenticate first, validate the exact selection and scope,
|
||||
strip the private envelope, then import or propagate provenance at the receiving boundary.
|
||||
Preserve documented headerless legacy behavior.
|
||||
- Never substitute secret plaintext into source, serialize plaintext provenance, hand-roll private
|
||||
headers, or blanket-sanitize tool results.
|
||||
- Add focused tests for named projection, identical unproven public text, malformed/incomplete
|
||||
metadata, metadata stripping, scope isolation, and legacy compatibility where applicable.
|
||||
|
||||
## Critical Rules for Outputs
|
||||
|
||||
### Output Types
|
||||
- `'string'`, `'number'`, `'boolean'` - Primitives
|
||||
- `'json'` - Complex objects (use this, NOT 'object')
|
||||
- `'array'` - Arrays with `items` property
|
||||
- `'object'` - Objects with `properties` property
|
||||
|
||||
### Optional Outputs
|
||||
Add `optional: true` for fields that may not exist in the response:
|
||||
```typescript
|
||||
closedAt: {
|
||||
type: 'string',
|
||||
description: 'When the issue was closed',
|
||||
optional: true,
|
||||
},
|
||||
```
|
||||
|
||||
### Typed JSON Outputs
|
||||
|
||||
When using `type: 'json'` and you know the object shape in advance, **always define the inner structure** using `properties` so downstream consumers know what fields are available:
|
||||
|
||||
```typescript
|
||||
// BAD: Opaque json with no info about what's inside
|
||||
metadata: {
|
||||
type: 'json',
|
||||
description: 'Response metadata',
|
||||
},
|
||||
|
||||
// GOOD: Define the known properties
|
||||
metadata: {
|
||||
type: 'json',
|
||||
description: 'Response metadata',
|
||||
properties: {
|
||||
id: { type: 'string', description: 'Unique ID' },
|
||||
status: { type: 'string', description: 'Current status' },
|
||||
count: { type: 'number', description: 'Total count' },
|
||||
},
|
||||
},
|
||||
```
|
||||
|
||||
For arrays of objects, define the item structure:
|
||||
```typescript
|
||||
items: {
|
||||
type: 'array',
|
||||
description: 'List of items',
|
||||
items: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
id: { type: 'string', description: 'Item ID' },
|
||||
name: { type: 'string', description: 'Item name' },
|
||||
},
|
||||
},
|
||||
},
|
||||
```
|
||||
|
||||
Only use bare `type: 'json'` without `properties` when the shape is truly dynamic or unknown.
|
||||
|
||||
If the response shape is unknown because the docs do not provide it, you MUST tell the user and stop. Unknown is not the same as dynamic. Never guess outputs.
|
||||
|
||||
## Critical Rules for transformResponse
|
||||
|
||||
### Handle Nullable Fields
|
||||
ALWAYS use `?? null` for fields that may be undefined:
|
||||
```typescript
|
||||
transformResponse: async (response: Response) => {
|
||||
const data = await response.json()
|
||||
return {
|
||||
success: true,
|
||||
output: {
|
||||
id: data.id,
|
||||
title: data.title,
|
||||
body: data.body ?? null, // May be undefined
|
||||
assignee: data.assignee ?? null, // May be undefined
|
||||
labels: data.labels ?? [], // Default to empty array
|
||||
closedAt: data.closed_at ?? null, // May be undefined
|
||||
},
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Never Output Raw JSON Dumps
|
||||
DON'T do this:
|
||||
```typescript
|
||||
output: {
|
||||
data: data, // BAD - raw JSON dump
|
||||
}
|
||||
```
|
||||
|
||||
DO this instead - extract meaningful fields:
|
||||
```typescript
|
||||
output: {
|
||||
id: data.id,
|
||||
name: data.name,
|
||||
status: data.status,
|
||||
metadata: {
|
||||
createdAt: data.created_at,
|
||||
updatedAt: data.updated_at,
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
## Types File Pattern
|
||||
|
||||
Create `types.ts` with interfaces for all params and responses:
|
||||
|
||||
```typescript
|
||||
import type { ToolResponse } from '@/tools/types'
|
||||
|
||||
// Parameter interfaces
|
||||
export interface {Service}{Action}Params {
|
||||
accessToken: string
|
||||
requiredField: string
|
||||
optionalField?: string
|
||||
}
|
||||
|
||||
// Response interfaces (extend ToolResponse)
|
||||
export interface {Service}{Action}Response extends ToolResponse {
|
||||
output: {
|
||||
field1: string
|
||||
field2: number
|
||||
optionalField?: string | null
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Index.ts Barrel Export Pattern
|
||||
|
||||
```typescript
|
||||
// Export all tools
|
||||
export { serviceTool1 } from './{action1}'
|
||||
export { serviceTool2 } from './{action2}'
|
||||
|
||||
// Export types
|
||||
export * from './types'
|
||||
```
|
||||
|
||||
## Registering Tools
|
||||
|
||||
After creating tools:
|
||||
1. Import tools in `apps/sim/tools/registry.ts`
|
||||
2. Add to the `tools` object with snake_case keys (alphabetically):
|
||||
```typescript
|
||||
import { serviceActionTool } from '@/tools/{service}'
|
||||
|
||||
export const tools = {
|
||||
// ... existing tools ...
|
||||
{service}_{action}: serviceActionTool,
|
||||
}
|
||||
```
|
||||
|
||||
3. Regenerate the tool metadata artifacts:
|
||||
|
||||
```bash
|
||||
bun run tool-metadata:generate
|
||||
```
|
||||
|
||||
Client code reads a tool's `params`/`outputs` from generated metadata rather than
|
||||
importing the registry, so a tool you add, change or remove is invisible to the UI until
|
||||
these are regenerated — and CI fails on stale artifacts. Commit the result. See
|
||||
`.agents/skills/tool-registry-boundary/SKILL.md`.
|
||||
|
||||
## Wiring Tools into the Block (Required)
|
||||
|
||||
After registering in `tools/registry.ts`, you MUST also update the block definition at `apps/sim/blocks/blocks/{service}.ts`. This is not optional — tools are only usable from the UI if they are wired into the block.
|
||||
|
||||
### 1. Add to `tools.access`
|
||||
|
||||
```typescript
|
||||
tools: {
|
||||
access: [
|
||||
// existing tools...
|
||||
'service_new_action', // Add every new tool ID here
|
||||
],
|
||||
config: { ... }
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Add operation dropdown options
|
||||
|
||||
If the block uses an operation dropdown, add an option for each new tool:
|
||||
|
||||
```typescript
|
||||
{
|
||||
id: 'operation',
|
||||
type: 'dropdown',
|
||||
options: [
|
||||
// existing options...
|
||||
{ label: 'New Action', id: 'new_action' }, // id maps to what tools.config.tool returns
|
||||
],
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Add subBlocks for new tool params
|
||||
|
||||
For each new tool, add subBlocks covering all its required params (and optional ones where useful). Apply `condition` to show them only for the right operation, and mark required params with `required`:
|
||||
|
||||
```typescript
|
||||
// Required param for new_action
|
||||
{
|
||||
id: 'someParam',
|
||||
title: 'Some Param',
|
||||
type: 'short-input',
|
||||
placeholder: 'e.g., value',
|
||||
condition: { field: 'operation', value: 'new_action' },
|
||||
required: { field: 'operation', value: 'new_action' },
|
||||
},
|
||||
// Optional param — put in advanced mode
|
||||
{
|
||||
id: 'optionalParam',
|
||||
title: 'Optional Param',
|
||||
type: 'short-input',
|
||||
condition: { field: 'operation', value: 'new_action' },
|
||||
mode: 'advanced',
|
||||
},
|
||||
```
|
||||
|
||||
### 4. Update `tools.config.tool`
|
||||
|
||||
Ensure the tool selector returns the correct tool ID for every new operation. The simplest pattern:
|
||||
|
||||
```typescript
|
||||
tool: (params) => `service_${params.operation}`,
|
||||
// If operation dropdown IDs already match tool IDs, this requires no change.
|
||||
```
|
||||
|
||||
If the dropdown IDs differ from tool IDs, add explicit mappings:
|
||||
|
||||
```typescript
|
||||
tool: (params) => {
|
||||
const map: Record<string, string> = {
|
||||
new_action: 'service_new_action',
|
||||
// ...
|
||||
}
|
||||
return map[params.operation] ?? `service_${params.operation}`
|
||||
},
|
||||
```
|
||||
|
||||
### 5. Update `tools.config.params`
|
||||
|
||||
Add any type coercions needed for new params (runs at execution time, after variable resolution):
|
||||
|
||||
```typescript
|
||||
params: (params) => {
|
||||
const result: Record<string, unknown> = {}
|
||||
if (params.limit != null && params.limit !== '') result.limit = Number(params.limit)
|
||||
if (params.newParamName) result.toolParamName = params.newParamName // rename if IDs differ
|
||||
return result
|
||||
},
|
||||
```
|
||||
|
||||
### 6. Add new outputs
|
||||
|
||||
Add any new fields returned by the new tools to the block `outputs`:
|
||||
|
||||
```typescript
|
||||
outputs: {
|
||||
// existing outputs...
|
||||
newField: { type: 'string', description: 'Description of new field' },
|
||||
}
|
||||
```
|
||||
|
||||
### 7. Add new inputs
|
||||
|
||||
Add new subBlock param IDs to the block `inputs` section:
|
||||
|
||||
```typescript
|
||||
inputs: {
|
||||
// existing inputs...
|
||||
someParam: { type: 'string', description: 'Param description' },
|
||||
optionalParam: { type: 'string', description: 'Optional param description' },
|
||||
}
|
||||
```
|
||||
|
||||
### Block wiring checklist
|
||||
|
||||
- [ ] New tool IDs added to `tools.access`
|
||||
- [ ] Operation dropdown has an option for each new tool
|
||||
- [ ] SubBlocks cover all required params for each new tool
|
||||
- [ ] SubBlocks have correct `condition` (only show for the right operation)
|
||||
- [ ] Optional/rarely-used params set to `mode: 'advanced'`
|
||||
- [ ] `tools.config.tool` returns correct ID for every new operation
|
||||
- [ ] `tools.config.params` handles any ID remapping or type coercions
|
||||
- [ ] New outputs added to block `outputs`
|
||||
- [ ] New params added to block `inputs`
|
||||
|
||||
## V2 Tool Pattern
|
||||
|
||||
If creating V2 tools (API-aligned outputs), use `_v2` suffix:
|
||||
- Tool ID: `{service}_{action}_v2`
|
||||
- Variable name: `{action}V2Tool`
|
||||
- Version: `'2.0.0'`
|
||||
- Outputs: Flat, API-aligned (no content/metadata wrapper)
|
||||
|
||||
## Naming Convention
|
||||
|
||||
All tool IDs MUST use `snake_case`: `{service}_{action}` (e.g., `x_create_tweet`, `slack_send_message`). Never use camelCase or PascalCase for tool IDs.
|
||||
|
||||
## Checklist Before Finishing
|
||||
|
||||
- [ ] All tool IDs use snake_case
|
||||
- [ ] All params have explicit `required: true` or `required: false`
|
||||
- [ ] All params have appropriate `visibility`
|
||||
- [ ] All nullable response fields use `?? null`
|
||||
- [ ] All optional outputs have `optional: true`
|
||||
- [ ] No raw JSON dumps in outputs
|
||||
- [ ] Types file has all interfaces
|
||||
- [ ] Index.ts exports all tools and re-exports types (`export * from './types'`)
|
||||
- [ ] Tools registered in `tools/registry.ts`
|
||||
- [ ] `bun run tool-metadata:generate` run and the regenerated artifacts committed
|
||||
- [ ] Block wired: `tools.access`, dropdown options, subBlocks, `tools.config`, outputs, inputs
|
||||
- [ ] Model, durable-storage, and internal-execution boundaries use the shared provenance mechanisms
|
||||
only where a concrete Sim `{{...}}` resolution path requires them
|
||||
- [ ] Ordinary third-party inputs/results remain unchanged and private metadata never leaves Sim
|
||||
|
||||
## Final Validation (Required)
|
||||
|
||||
After creating all tools, you MUST validate every tool before finishing:
|
||||
|
||||
1. **Read every tool file** you created — do not skip any
|
||||
2. **Cross-reference with the API docs** to verify:
|
||||
- All required params are marked `required: true`
|
||||
- All optional params are marked `required: false`
|
||||
- Param types match the API (string, number, boolean, json)
|
||||
- Request URL, method, headers, and body match the API spec
|
||||
- `transformResponse` extracts the correct fields from the API response
|
||||
- All output fields match what the API actually returns
|
||||
- No fields are missing from outputs that the API provides
|
||||
- No extra fields are defined in outputs that the API doesn't return
|
||||
- Every output field and JSON path is backed by docs or live-verified sample responses
|
||||
3. **Verify consistency** across tools:
|
||||
- Shared types in `types.ts` match all tools that use them
|
||||
- Tool IDs in the barrel export match the tool file definitions
|
||||
- Error handling is consistent (error checks, meaningful messages)
|
||||
4. **If any response schema is still unknown**, explicitly tell the user instead of guessing
|
||||
@@ -0,0 +1,5 @@
|
||||
interface:
|
||||
display_name: "Add Tools"
|
||||
short_description: "Build Sim tools from API docs"
|
||||
brand_color: "#EA580C"
|
||||
default_prompt: "Use $add-tools to create or update Sim tool definitions from service API docs."
|
||||
@@ -0,0 +1,513 @@
|
||||
---
|
||||
name: add-trigger
|
||||
description: Create webhook or polling triggers for a Sim integration
|
||||
argument-hint: <service-name>
|
||||
---
|
||||
|
||||
# Add Trigger
|
||||
|
||||
You are an expert at creating webhook and polling triggers for Sim. You understand the trigger system, the generic `buildTriggerSubBlocks` helper, polling infrastructure, and how triggers connect to blocks.
|
||||
|
||||
## Your Task
|
||||
|
||||
1. Research what webhook events the service supports — if the service lacks reliable webhooks, use polling
|
||||
2. Create the trigger files using the generic builder (webhook) or manual config (polling)
|
||||
3. Create a provider handler (webhook) or polling handler (polling)
|
||||
4. Register triggers and connect them to the block
|
||||
|
||||
## Hard Rule: No Guessed Webhook Payload Schemas
|
||||
|
||||
If the service docs do not clearly show the webhook payload JSON for an event, you MUST tell the user instead of guessing trigger outputs or `formatInput` mappings.
|
||||
|
||||
- Do NOT invent payload field names
|
||||
- Do NOT guess nested event object paths
|
||||
- Do NOT infer output fields from the UI or marketing docs
|
||||
- Do NOT write `formatInput` against unverified webhook bodies
|
||||
|
||||
If the payload shape is unknown, do one of these instead:
|
||||
1. Ask the user for sample webhook payloads
|
||||
2. Ask the user for a test webhook source so you can inspect a real event
|
||||
3. Implement only the event registration/setup portions whose payloads are documented
|
||||
4. Leave the trigger unimplemented and explicitly say which payload fields are unknown
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
apps/sim/triggers/{service}/
|
||||
├── index.ts # Barrel exports
|
||||
├── utils.ts # Service-specific helpers (options, instructions, extra fields, outputs)
|
||||
├── {event_a}.ts # Primary trigger (includes dropdown)
|
||||
├── {event_b}.ts # Secondary trigger (no dropdown)
|
||||
└── webhook.ts # Generic webhook trigger (optional, for "all events")
|
||||
|
||||
apps/sim/lib/webhooks/
|
||||
├── provider-subscription-utils.ts # Shared subscription helpers (getProviderConfig, getNotificationUrl)
|
||||
├── providers/
|
||||
│ ├── {service}.ts # Provider handler (auth, formatInput, matchEvent, subscriptions)
|
||||
│ ├── types.ts # WebhookProviderHandler interface
|
||||
│ ├── utils.ts # Shared helpers (createHmacVerifier, verifyTokenAuth, skipByEventTypes)
|
||||
│ └── registry.ts # Handler map + default handler
|
||||
```
|
||||
|
||||
## Step 1: Create `utils.ts`
|
||||
|
||||
This file contains all service-specific helpers used by triggers.
|
||||
|
||||
```typescript
|
||||
import type { SubBlockConfig } from '@/blocks/types'
|
||||
import type { TriggerOutput } from '@/triggers/types'
|
||||
|
||||
export const {service}TriggerOptions = [
|
||||
{ label: 'Event A', id: '{service}_event_a' },
|
||||
{ label: 'Event B', id: '{service}_event_b' },
|
||||
]
|
||||
|
||||
export function {service}SetupInstructions(eventType: string): string {
|
||||
const instructions = [
|
||||
'Copy the <strong>Webhook URL</strong> above',
|
||||
'Go to <strong>{Service} Settings > Webhooks</strong>',
|
||||
`Select the <strong>${eventType}</strong> event type`,
|
||||
'Paste the webhook URL and save',
|
||||
'Click "Save" above to activate your trigger',
|
||||
]
|
||||
return instructions
|
||||
.map((instruction, index) =>
|
||||
`<div class="mb-3"><strong>${index + 1}.</strong> ${instruction}</div>`
|
||||
)
|
||||
.join('')
|
||||
}
|
||||
|
||||
export function build{Service}ExtraFields(triggerId: string): SubBlockConfig[] {
|
||||
return [
|
||||
{
|
||||
id: 'projectId',
|
||||
title: 'Project ID (Optional)',
|
||||
type: 'short-input',
|
||||
placeholder: 'Leave empty for all projects',
|
||||
mode: 'trigger',
|
||||
condition: { field: 'selectedTriggerId', value: triggerId },
|
||||
},
|
||||
]
|
||||
}
|
||||
|
||||
export function build{Service}Outputs(): Record<string, TriggerOutput> {
|
||||
return {
|
||||
eventType: { type: 'string', description: 'The type of event' },
|
||||
resourceId: { type: 'string', description: 'ID of the affected resource' },
|
||||
resource: {
|
||||
id: { type: 'string', description: 'Resource ID' },
|
||||
name: { type: 'string', description: 'Resource name' },
|
||||
},
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Step 2: Create Trigger Files
|
||||
|
||||
**Primary trigger** — MUST include `includeDropdown: true`:
|
||||
|
||||
```typescript
|
||||
import { {Service}Icon } from '@/components/icons'
|
||||
import { buildTriggerSubBlocks } from '@/triggers'
|
||||
import { build{Service}ExtraFields, build{Service}Outputs, {service}SetupInstructions, {service}TriggerOptions } from '@/triggers/{service}/utils'
|
||||
import type { TriggerConfig } from '@/triggers/types'
|
||||
|
||||
export const {service}EventATrigger: TriggerConfig = {
|
||||
id: '{service}_event_a',
|
||||
name: '{Service} Event A',
|
||||
provider: '{service}',
|
||||
description: 'Trigger workflow when Event A occurs',
|
||||
version: '1.0.0',
|
||||
icon: {Service}Icon,
|
||||
subBlocks: buildTriggerSubBlocks({
|
||||
triggerId: '{service}_event_a',
|
||||
triggerOptions: {service}TriggerOptions,
|
||||
includeDropdown: true,
|
||||
setupInstructions: {service}SetupInstructions('Event A'),
|
||||
extraFields: build{Service}ExtraFields('{service}_event_a'),
|
||||
}),
|
||||
outputs: build{Service}Outputs(),
|
||||
webhook: { method: 'POST', headers: { 'Content-Type': 'application/json' } },
|
||||
}
|
||||
```
|
||||
|
||||
**Secondary triggers** — NO `includeDropdown` (it's already in the primary):
|
||||
|
||||
```typescript
|
||||
export const {service}EventBTrigger: TriggerConfig = {
|
||||
// Same as above but: id: '{service}_event_b', no includeDropdown
|
||||
}
|
||||
```
|
||||
|
||||
## Step 3: Register and Wire
|
||||
|
||||
### `apps/sim/triggers/{service}/index.ts`
|
||||
|
||||
```typescript
|
||||
export { {service}EventATrigger } from './event_a'
|
||||
export { {service}EventBTrigger } from './event_b'
|
||||
```
|
||||
|
||||
### `apps/sim/triggers/registry.ts`
|
||||
|
||||
```typescript
|
||||
import { {service}EventATrigger, {service}EventBTrigger } from '@/triggers/{service}'
|
||||
|
||||
export const TRIGGER_REGISTRY: TriggerRegistry = {
|
||||
// ... existing ...
|
||||
{service}_event_a: {service}EventATrigger,
|
||||
{service}_event_b: {service}EventBTrigger,
|
||||
}
|
||||
```
|
||||
|
||||
### Block file (`apps/sim/blocks/blocks/{service}.ts`)
|
||||
|
||||
Wire triggers into the block so the trigger UI appears and `generate-docs.ts` discovers them. Two changes are needed:
|
||||
|
||||
1. **Spread trigger subBlocks** at the end of the block's `subBlocks` array
|
||||
2. **Add `triggers` property** after `outputs` with `enabled: true` and `available: [...]`
|
||||
|
||||
```typescript
|
||||
import { getTrigger } from '@/triggers'
|
||||
|
||||
export const {Service}Block: BlockConfig = {
|
||||
// ...
|
||||
subBlocks: [
|
||||
// Regular tool subBlocks first...
|
||||
...getTrigger('{service}_event_a').subBlocks,
|
||||
...getTrigger('{service}_event_b').subBlocks,
|
||||
],
|
||||
// ... tools, inputs, outputs ...
|
||||
triggers: {
|
||||
enabled: true,
|
||||
available: ['{service}_event_a', '{service}_event_b'],
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
**Versioned blocks (V1 + V2):** Many integrations have a hidden V1 block and a visible V2 block. Where you add the trigger wiring depends on how V2 inherits from V1:
|
||||
|
||||
- **V2 uses `...V1Block` spread** (e.g., Google Calendar): Add trigger to V1 — V2 inherits both `subBlocks` and `triggers` automatically.
|
||||
- **V2 defines its own `subBlocks`** (e.g., Google Sheets): Add trigger to V2 (the visible block). V1 is hidden and doesn't need it.
|
||||
- **Single block, no V2** (e.g., Google Drive): Add trigger directly.
|
||||
|
||||
`generate-docs.ts` deduplicates by base type (first match wins). If V1 is processed first without triggers, the V2 triggers won't appear in `integrations.json`. Always verify by checking the output after running the script.
|
||||
|
||||
## Provider Handler
|
||||
|
||||
All provider-specific webhook logic lives in a single handler file: `apps/sim/lib/webhooks/providers/{service}.ts`.
|
||||
|
||||
### When to Create a Handler
|
||||
|
||||
| Behavior | Method | Examples |
|
||||
|---|---|---|
|
||||
| HMAC signature auth | `verifyAuth` via `createHmacVerifier` | Ashby, Jira, Linear, Typeform |
|
||||
| Custom token auth | `verifyAuth` via `verifyTokenAuth` | Generic, Google Forms |
|
||||
| Event filtering | `matchEvent` | GitHub, Jira, Attio, HubSpot |
|
||||
| Idempotency dedup | `extractIdempotencyId` | Slack, Stripe, Linear, Jira |
|
||||
| Custom input formatting | `formatInput` | Slack, Teams, Attio, Ashby |
|
||||
| Auto webhook creation | `createSubscription` | Ashby, Grain, Calendly, Airtable |
|
||||
| Auto webhook deletion | `deleteSubscription` | Ashby, Grain, Calendly, Airtable |
|
||||
| Challenge/verification | `handleChallenge` | Slack, WhatsApp, Teams |
|
||||
| Custom success response | `formatSuccessResponse` | Slack, Twilio Voice, Teams |
|
||||
|
||||
If none apply, you don't need a handler. The default handler provides bearer token auth.
|
||||
|
||||
### Example Handler
|
||||
|
||||
```typescript
|
||||
import crypto from 'crypto'
|
||||
import { createLogger } from '@sim/logger'
|
||||
import { safeCompare } from '@/lib/core/security/encryption'
|
||||
import type { EventMatchContext, FormatInputContext, FormatInputResult, WebhookProviderHandler } from '@/lib/webhooks/providers/types'
|
||||
import { createHmacVerifier } from '@/lib/webhooks/providers/utils'
|
||||
|
||||
const logger = createLogger('WebhookProvider:{Service}')
|
||||
|
||||
function validate{Service}Signature(secret: string, signature: string, body: string): boolean {
|
||||
if (!secret || !signature || !body) return false
|
||||
const computed = crypto.createHmac('sha256', secret).update(body, 'utf8').digest('hex')
|
||||
return safeCompare(computed, signature)
|
||||
}
|
||||
|
||||
export const {service}Handler: WebhookProviderHandler = {
|
||||
verifyAuth: createHmacVerifier({
|
||||
configKey: 'webhookSecret',
|
||||
headerName: 'X-{Service}-Signature',
|
||||
validateFn: validate{Service}Signature,
|
||||
providerLabel: '{Service}',
|
||||
}),
|
||||
|
||||
async matchEvent({ body, requestId, providerConfig }: EventMatchContext) {
|
||||
const triggerId = providerConfig.triggerId as string | undefined
|
||||
if (triggerId && triggerId !== '{service}_webhook') {
|
||||
const { is{Service}EventMatch } = await import('@/triggers/{service}/utils')
|
||||
if (!is{Service}EventMatch(triggerId, body as Record<string, unknown>)) return false
|
||||
}
|
||||
return true
|
||||
},
|
||||
|
||||
async formatInput({ body }: FormatInputContext): Promise<FormatInputResult> {
|
||||
const b = body as Record<string, unknown>
|
||||
return {
|
||||
input: {
|
||||
eventType: b.type,
|
||||
resourceId: (b.data as Record<string, unknown>)?.id || '',
|
||||
resource: b.data,
|
||||
},
|
||||
}
|
||||
},
|
||||
|
||||
extractIdempotencyId(body: unknown) {
|
||||
const obj = body as Record<string, unknown>
|
||||
return obj.id && obj.type ? `${obj.type}:${obj.id}` : null
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
### Register the Handler
|
||||
|
||||
In `apps/sim/lib/webhooks/providers/registry.ts`:
|
||||
|
||||
```typescript
|
||||
import { {service}Handler } from '@/lib/webhooks/providers/{service}'
|
||||
|
||||
const PROVIDER_HANDLERS: Record<string, WebhookProviderHandler> = {
|
||||
// ... existing (alphabetical) ...
|
||||
{service}: {service}Handler,
|
||||
}
|
||||
```
|
||||
|
||||
## Output Alignment (Critical)
|
||||
|
||||
There are two sources of truth that **MUST be aligned**:
|
||||
|
||||
1. **Trigger `outputs`** — schema defining what fields SHOULD be available (UI tag dropdown)
|
||||
2. **`formatInput` on the handler** — implementation that transforms raw payload into actual data
|
||||
|
||||
If they differ: the tag dropdown shows fields that don't exist, or actual data has fields users can't discover.
|
||||
|
||||
**Rules for `formatInput`:**
|
||||
- Return `{ input: { ... } }` where inner keys match trigger `outputs` exactly
|
||||
- Return `{ input: ..., skip: { message: '...' } }` to skip execution
|
||||
- No wrapper objects or duplication
|
||||
- Use `null` for missing optional data
|
||||
|
||||
## Automatic Webhook Registration
|
||||
|
||||
If the service API supports programmatic webhook creation, implement `createSubscription` and `deleteSubscription` on the handler. The orchestration layer calls these automatically — **no code touches `route.ts`, `provider-subscriptions.ts`, or `deploy.ts`**.
|
||||
|
||||
```typescript
|
||||
import { getNotificationUrl, getProviderConfig } from '@/lib/webhooks/provider-subscription-utils'
|
||||
import type { DeleteSubscriptionContext, SubscriptionContext, SubscriptionResult } from '@/lib/webhooks/providers/types'
|
||||
|
||||
export const {service}Handler: WebhookProviderHandler = {
|
||||
async createSubscription(ctx: SubscriptionContext): Promise<SubscriptionResult | undefined> {
|
||||
const config = getProviderConfig(ctx.webhook)
|
||||
const apiKey = config.apiKey as string
|
||||
if (!apiKey) throw new Error('{Service} API Key is required.')
|
||||
|
||||
const res = await fetch('https://api.{service}.com/webhooks', {
|
||||
method: 'POST',
|
||||
headers: { Authorization: `Bearer ${apiKey}`, 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ url: getNotificationUrl(ctx.webhook) }),
|
||||
})
|
||||
|
||||
if (!res.ok) throw new Error(`{Service} error: ${res.status}`)
|
||||
const { id } = (await res.json()) as { id: string }
|
||||
return { providerConfigUpdates: { externalId: id } }
|
||||
},
|
||||
|
||||
async deleteSubscription(ctx: DeleteSubscriptionContext): Promise<void> {
|
||||
const config = getProviderConfig(ctx.webhook)
|
||||
const { apiKey, externalId } = config as { apiKey?: string; externalId?: string }
|
||||
if (!apiKey || !externalId) return
|
||||
await fetch(`https://api.{service}.com/webhooks/${externalId}`, {
|
||||
method: 'DELETE',
|
||||
headers: { Authorization: `Bearer ${apiKey}` },
|
||||
}).catch(() => {})
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
**Key points:**
|
||||
- Throw from `createSubscription` — orchestration rolls back the DB webhook
|
||||
- Never throw from `deleteSubscription` — log non-fatally
|
||||
- Return `{ providerConfigUpdates: { externalId } }` — orchestration merges into `providerConfig`
|
||||
- Add `apiKey` field to `build{Service}ExtraFields` with `password: true`
|
||||
|
||||
## Trigger Outputs Schema
|
||||
|
||||
Trigger outputs use the same schema as block outputs (NOT tool outputs).
|
||||
|
||||
**Supported:** `type` + `description` for leaf fields, nested objects for complex data.
|
||||
**NOT supported:** `optional: true`, `items` (those are tool-output-only features).
|
||||
|
||||
```typescript
|
||||
export function buildOutputs(): Record<string, TriggerOutput> {
|
||||
return {
|
||||
eventType: { type: 'string', description: 'Event type' },
|
||||
timestamp: { type: 'string', description: 'When it occurred' },
|
||||
payload: { type: 'json', description: 'Full event payload' },
|
||||
resource: {
|
||||
id: { type: 'string', description: 'Resource ID' },
|
||||
name: { type: 'string', description: 'Resource name' },
|
||||
},
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Polling Triggers
|
||||
|
||||
Use polling when the service lacks reliable webhooks (e.g., Google Sheets, Google Drive, Google Calendar, Gmail, RSS, IMAP). Polling triggers do NOT use `buildTriggerSubBlocks` — they define subBlocks manually.
|
||||
|
||||
### Directory Structure
|
||||
|
||||
```
|
||||
apps/sim/triggers/{service}/
|
||||
├── index.ts # Barrel export
|
||||
└── poller.ts # TriggerConfig with polling: true
|
||||
|
||||
apps/sim/lib/webhooks/polling/
|
||||
└── {service}.ts # PollingProviderHandler implementation
|
||||
```
|
||||
|
||||
### Polling Handler (`apps/sim/lib/webhooks/polling/{service}.ts`)
|
||||
|
||||
```typescript
|
||||
import { pollingIdempotency } from '@/lib/core/idempotency/service'
|
||||
import type { PollingProviderHandler, PollWebhookContext } from '@/lib/webhooks/polling/types'
|
||||
import { markWebhookFailed, markWebhookSuccess, resolveOAuthCredential, updateWebhookProviderConfig } from '@/lib/webhooks/polling/utils'
|
||||
import { processPolledWebhookEvent } from '@/lib/webhooks/processor'
|
||||
|
||||
export const {service}PollingHandler: PollingProviderHandler = {
|
||||
provider: '{service}',
|
||||
label: '{Service}',
|
||||
|
||||
async pollWebhook(ctx: PollWebhookContext): Promise<'success' | 'failure'> {
|
||||
const { webhookData, workflowData, requestId, logger } = ctx
|
||||
const webhookId = webhookData.id
|
||||
|
||||
try {
|
||||
// For OAuth services:
|
||||
const accessToken = await resolveOAuthCredential(webhookData, '{service}', requestId)
|
||||
const config = webhookData.providerConfig as unknown as {Service}WebhookConfig
|
||||
|
||||
// First poll: seed state, emit nothing
|
||||
if (!config.lastCheckedTimestamp) {
|
||||
await updateWebhookProviderConfig(webhookId, { lastCheckedTimestamp: new Date().toISOString() }, logger)
|
||||
await markWebhookSuccess(webhookId, logger)
|
||||
return 'success'
|
||||
}
|
||||
|
||||
// Fetch changes since last poll, process with idempotency
|
||||
// ...
|
||||
|
||||
await markWebhookSuccess(webhookId, logger)
|
||||
return 'success'
|
||||
} catch (error) {
|
||||
logger.error(`[${requestId}] Error processing {service} webhook ${webhookId}:`, error)
|
||||
await markWebhookFailed(webhookId, logger)
|
||||
return 'failure'
|
||||
}
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
**Key patterns:**
|
||||
- First poll seeds state and emits nothing (avoids flooding with existing data)
|
||||
- Use `pollingIdempotency.executeWithIdempotency(provider, key, callback)` for dedup
|
||||
- Use `processPolledWebhookEvent(webhookData, workflowData, payload, requestId)` to fire the workflow
|
||||
- Use `updateWebhookProviderConfig(webhookId, partialConfig, logger)` for read-merge-write on state
|
||||
- Use the latest server-side timestamp from API responses (not wall clock) to avoid clock skew
|
||||
|
||||
### Trigger Config (`apps/sim/triggers/{service}/poller.ts`)
|
||||
|
||||
```typescript
|
||||
import { {Service}Icon } from '@/components/icons'
|
||||
import type { TriggerConfig } from '@/triggers/types'
|
||||
|
||||
export const {service}PollingTrigger: TriggerConfig = {
|
||||
id: '{service}_poller',
|
||||
name: '{Service} Trigger',
|
||||
provider: '{service}',
|
||||
description: 'Triggers when ...',
|
||||
version: '1.0.0',
|
||||
icon: {Service}Icon,
|
||||
polling: true, // REQUIRED — routes to polling infrastructure
|
||||
|
||||
subBlocks: [
|
||||
{ id: 'triggerCredentials', type: 'oauth-input', title: 'Credentials', serviceId: '{service}', requiredScopes: [], required: true, mode: 'trigger' },
|
||||
// ... service-specific config fields (dropdowns, inputs, switches) ...
|
||||
{ id: 'triggerInstructions', type: 'text', title: 'Setup Instructions', hideFromPreview: true, mode: 'trigger', defaultValue: '...' },
|
||||
],
|
||||
|
||||
outputs: {
|
||||
// Must match the payload shape from processPolledWebhookEvent
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
### Registration (3 places)
|
||||
|
||||
1. **`apps/sim/triggers/constants.ts`** — add provider to `POLLING_PROVIDERS` Set
|
||||
2. **`apps/sim/lib/webhooks/polling/registry.ts`** — import handler, add to `POLLING_HANDLERS`
|
||||
3. **`apps/sim/triggers/registry.ts`** — import trigger config, add to `TRIGGER_REGISTRY`
|
||||
|
||||
### Helm Cron Job
|
||||
|
||||
Add to `helm/sim/values.yaml` under the existing polling cron jobs:
|
||||
|
||||
```yaml
|
||||
{service}WebhookPoll:
|
||||
schedule: "*/1 * * * *"
|
||||
concurrencyPolicy: Forbid
|
||||
url: "http://sim:3000/api/webhooks/poll/{service}"
|
||||
```
|
||||
|
||||
### Reference Implementations
|
||||
|
||||
- Simple: `apps/sim/lib/webhooks/polling/rss.ts` + `apps/sim/triggers/rss/poller.ts`
|
||||
- Complex (OAuth, attachments): `apps/sim/lib/webhooks/polling/gmail.ts` + `apps/sim/triggers/gmail/poller.ts`
|
||||
- Cursor-based (changes API): `apps/sim/lib/webhooks/polling/google-drive.ts`
|
||||
- Timestamp-based: `apps/sim/lib/webhooks/polling/google-calendar.ts`
|
||||
|
||||
## Checklist
|
||||
|
||||
### Trigger Definition
|
||||
- [ ] Created `utils.ts` with options, instructions, extra fields, and output builders
|
||||
- [ ] Primary trigger has `includeDropdown: true`; secondary triggers do NOT
|
||||
- [ ] All triggers use `buildTriggerSubBlocks` helper
|
||||
- [ ] Created `index.ts` barrel export
|
||||
|
||||
### Registration
|
||||
- [ ] All triggers in `triggers/registry.ts` → `TRIGGER_REGISTRY`
|
||||
- [ ] Block has `triggers.enabled: true` and lists all trigger IDs in `triggers.available`
|
||||
- [ ] Block spreads all trigger subBlocks: `...getTrigger('id').subBlocks`
|
||||
|
||||
### Provider Handler (if needed)
|
||||
- [ ] Handler file at `apps/sim/lib/webhooks/providers/{service}.ts`
|
||||
- [ ] Registered in `providers/registry.ts` (alphabetical)
|
||||
- [ ] Signature validator is a private function inside the handler file
|
||||
- [ ] `formatInput` output keys match trigger `outputs` exactly
|
||||
- [ ] Event matching uses dynamic `await import()` for trigger utils
|
||||
|
||||
### Auto Registration (if supported)
|
||||
- [ ] `createSubscription` and `deleteSubscription` on the handler
|
||||
- [ ] NO changes to `route.ts`, `provider-subscriptions.ts`, or `deploy.ts`
|
||||
- [ ] API key field uses `password: true`
|
||||
|
||||
### Polling Trigger (if applicable)
|
||||
- [ ] Handler implements `PollingProviderHandler` at `lib/webhooks/polling/{service}.ts`
|
||||
- [ ] Trigger config has `polling: true` and defines subBlocks manually (no `buildTriggerSubBlocks`)
|
||||
- [ ] Provider string matches across: trigger config, handler, `POLLING_PROVIDERS`, polling registry
|
||||
- [ ] First poll seeds state and emits nothing
|
||||
- [ ] Added provider to `POLLING_PROVIDERS` in `triggers/constants.ts`
|
||||
- [ ] Added handler to `POLLING_HANDLERS` in `lib/webhooks/polling/registry.ts`
|
||||
- [ ] Added cron job to `helm/sim/values.yaml`
|
||||
- [ ] Payload shape matches trigger `outputs` schema
|
||||
|
||||
### Testing
|
||||
- [ ] `bun run type-check` passes
|
||||
- [ ] Manually verify output keys match trigger `outputs` keys
|
||||
- [ ] Trigger UI shows correctly in the block
|
||||
@@ -0,0 +1,5 @@
|
||||
interface:
|
||||
display_name: "Add Trigger"
|
||||
short_description: "Build Sim webhook triggers"
|
||||
brand_color: "#DC2626"
|
||||
default_prompt: "Use $add-trigger to create or update webhook triggers for a Sim integration."
|
||||
@@ -0,0 +1,144 @@
|
||||
---
|
||||
name: babysit
|
||||
description: Drive a PR to a clean review (Greptile 5/5, zero open threads) — ships if needed, keeps it mergeable against staging, triggers Greptile/Cursor Bugbot, fixes real findings, replies to and resolves every thread, and loops until clean
|
||||
---
|
||||
|
||||
# Babysit PRs
|
||||
|
||||
Owns a PR end-to-end through review: ship it, wait for the automatic review round, and if it
|
||||
isn't already clean, drive fix → reply → resolve → re-review cycles until Greptile reports 5/5
|
||||
and there are zero open comment threads, keeping the branch mergeable against staging along the
|
||||
way. Designed to be run under `/loop` (no fixed interval — let it self-pace on review latency)
|
||||
so it survives across multiple wakeups in the same session.
|
||||
|
||||
## When to use
|
||||
|
||||
- The user says "babysit this PR", "keep working the reviews until it's clean", or similar
|
||||
- As the natural follow-up to `/ship` when the user wants the review loop automated rather than
|
||||
manually re-triggering reviews and answering comments themselves
|
||||
|
||||
## Inputs
|
||||
|
||||
Needs a PR number. If none is given and there's no open PR for the current branch, run `/ship`
|
||||
first (which includes the `origin/staging` sync check — see `.agents/skills/ship/SKILL.md`) to
|
||||
create one.
|
||||
|
||||
## Definition of "clean"
|
||||
|
||||
Both must hold:
|
||||
1. The latest Greptile summary comment reports **Confidence Score: 5/5**
|
||||
2. `reviewThreads` (GraphQL, see below) has **zero threads with `isResolved: false`**
|
||||
|
||||
Do not stop early on "no new comments this round" alone — a thread can be open from an earlier
|
||||
round. Always check both conditions freshly after every push.
|
||||
|
||||
## Loop
|
||||
|
||||
1. **Check current state** before doing anything, including whether the PR is still mergeable:
|
||||
```bash
|
||||
gh pr view <n> --json mergeable
|
||||
gh pr view <n> --json comments -q '[.comments[] | select(.author.login=="greptile-apps")] | last | .body'
|
||||
gh api graphql -f query='
|
||||
query { repository(owner: "<owner>", name: "<repo>") { pullRequest(number: <n>) {
|
||||
reviewThreads(first: 50) { pageInfo { hasNextPage endCursor } nodes { id isResolved path line
|
||||
comments(first: 5) { nodes { id databaseId author { login } body } } } } } } }'
|
||||
```
|
||||
`[.comments[]] | last | .body`, not `... | .body | tail -1` — the latter pipes every matching
|
||||
comment's full multi-line body through the pipeline and keeps only the final *line* of that
|
||||
combined output (usually the "Reviews (n): Last reviewed commit..." footer), not the last
|
||||
*comment*, so it silently misses the actual "Confidence Score: X/5" line.
|
||||
`reviewThreads(first: 50)` is a single page — check `pageInfo.hasNextPage`. If `true`, don't
|
||||
stop yet: re-run the same query with `after: "<endCursor>"` and keep paging until
|
||||
`hasNextPage` is `false` before evaluating "clean." A PR with more than 50 threads is rare but
|
||||
stopping on a partial page would silently miss unresolved ones past the cutoff.
|
||||
If `mergeable` is `CONFLICTING`, fix that first (step 2). Otherwise, if Greptile is 5/5 and
|
||||
every thread across all pages has `isResolved: true`, stop — report the outcome (see
|
||||
"Reporting" below) and skip the rest of this list.
|
||||
|
||||
2. **If the PR has a merge conflict**, merge `origin/staging`, resolve the conflicts, run the
|
||||
usual pre-push checks, push, and go to step 8 to re-trigger review.
|
||||
|
||||
3. **If no review has run yet** (fresh PR, no Greptile/Cursor comments): they usually run
|
||||
automatically on PR open — confirm via `gh pr checks <n>` (look for `Cursor Bugbot` /
|
||||
`Greptile Review`) and wait for that first round before doing anything else.
|
||||
|
||||
4. **If a review round has landed and it isn't clean**: for every thread where
|
||||
`isResolved: false`, triage the finding on its own merits — this is the part that requires
|
||||
judgment, not a mechanical loop:
|
||||
- **Real bug**: fix it in the cleanest way available. Match the codebase's existing
|
||||
conventions for that kind of problem before inventing a new one (e.g. an SSRF-prone
|
||||
user-supplied-host fetch should use whatever `validateUrlWithDNS`/`secureFetchWithPinnedIP`
|
||||
pattern the rest of the codebase already uses for that exact situation — grep for a sibling
|
||||
integration solving the same problem first). Never patch around a finding with a
|
||||
workaround, a broad try/catch, or a suppression comment — fix the actual cause.
|
||||
- **False positive**: don't change code. Reply with the specific reason it doesn't apply
|
||||
(cite the type definition, the established pattern it matches, or the doc it follows) so
|
||||
the reviewer bot and a human skimming later both understand why it was left as-is.
|
||||
- **Already fixed by an earlier finding in the same round**: note that and resolve without a
|
||||
duplicate code change.
|
||||
|
||||
5. **Reply to every thread individually** before resolving it — never resolve silently:
|
||||
```bash
|
||||
gh api repos/<owner>/<repo>/pulls/<n>/comments/<databaseId>/replies -f body="<what was done and why>"
|
||||
```
|
||||
Then resolve via GraphQL (needs the thread `id` from step 1, not the comment id):
|
||||
```bash
|
||||
gh api graphql -f query='mutation { resolveReviewThread(input: {threadId: "<threadId>"}) { thread { isResolved } } }'
|
||||
```
|
||||
|
||||
6. **Before pushing, re-run the full sync check from `/ship` step 2** — not just the log command,
|
||||
the whole check-and-recover flow (stash WIP if needed, rebase, verify the rebase didn't just
|
||||
cleanly replay stray commits, cherry-pick rebuild if it did or if it conflicted). A babysit
|
||||
loop spanning a long session is exactly the scenario where a branch can drift, and pushing
|
||||
review fixes on top of undetected drift is how an oversized PR happens even after the branch
|
||||
was fixed once. Then run the repo's pre-ship checks the same way `/ship` does before
|
||||
committing — not just lint/typecheck/boundary-validation, but also the conditional `/cleanup`
|
||||
(if this round's fix touched UI code) and `/db-migrate` (if it touched schema/migrations)
|
||||
gates from `/ship` steps 4 and 5. A review-fix round is still a code change and can trip
|
||||
either gate just as easily as the original commit did.
|
||||
|
||||
7. **Commit and push** the round's fixes as one commit — `--force-with-lease` whenever step 6's
|
||||
sync check rewrote history, which includes a plain `git rebase origin/staging` that completed
|
||||
with no conflicts, not only the cherry-pick rebuild path; both rewrite commits already
|
||||
published to the remote, so a plain `git push` can be rejected either way — then run `/ship`
|
||||
step 9's post-push verify — not just before the first push, every push in the loop:
|
||||
```bash
|
||||
git fetch origin staging && git log --oneline --reverse origin/staging..HEAD
|
||||
gh pr view <n> --json commits -q '.commits[].messageHeadline'
|
||||
```
|
||||
`--reverse` makes `git log` oldest-first, matching the PR commit list's order — plain
|
||||
`git log` is newest-first, so without it a positional comparison can spuriously fail on any
|
||||
multi-commit branch.
|
||||
These two lists must describe the same commits. A review loop runs many pushes across many
|
||||
rounds; checking sync only before the push (step 6) and never after is how a bad push or a
|
||||
PR whose commit history quietly went stale between rounds goes unnoticed.
|
||||
|
||||
8. **Re-trigger review** by posting `@greptile` and `@cursor review` as **two separate PR
|
||||
comments** — never combine them into one comment, each bot only responds to its own mention:
|
||||
```bash
|
||||
gh pr comment <n> --body "@greptile"
|
||||
gh pr comment <n> --body "@cursor review"
|
||||
```
|
||||
|
||||
9. **Wait for the new round**, then go back to step 1. Pace the wait with `ScheduleWakeup` using
|
||||
a fallback delay of ~250–300s (Greptile/Cursor typically take 1–3 minutes) — never busy-poll
|
||||
in a sleep loop. Pass the same `/loop babysit PR <n>` prompt on each wakeup so the loop
|
||||
resumes correctly.
|
||||
|
||||
10. **Stop conditions**: clean state reached (see above), or the same unresolved finding or
|
||||
merge conflict survives two consecutive rounds with no new information (surface it to the
|
||||
user instead of looping forever), or the user interrupts.
|
||||
|
||||
## Reporting
|
||||
|
||||
When the loop ends, summarize: how many rounds it took, what was actually fixed (one line each),
|
||||
what was pushed back on as a false positive and why, and the final Greptile score / thread count.
|
||||
|
||||
## Hard rules
|
||||
|
||||
- Never post the two re-review mentions as a single combined comment.
|
||||
- Never resolve a thread without replying to it first.
|
||||
- Never fix a finding with a hacky workaround — if the clean fix isn't obvious, find the sibling
|
||||
pattern elsewhere in the codebase solving the same class of problem and match it.
|
||||
- Never silently drop a finding — every thread gets either a code fix or a reasoned reply.
|
||||
- Always re-run the `/ship`-style sync check before every push in the loop, not just the first.
|
||||
@@ -0,0 +1,63 @@
|
||||
---
|
||||
name: cleanup
|
||||
description: Run all code quality skills — effects, memo, callbacks, state, React Query, emcn design review, url-state, and comments — analyzing in parallel, then applying fixes sequentially
|
||||
argument-hint: "[scope] [fix=true|false]"
|
||||
---
|
||||
|
||||
# Cleanup
|
||||
|
||||
Arguments:
|
||||
- scope: what to review (default: your current changes). Examples: "diff to main", "PR #123", "src/components/", "whole codebase"
|
||||
- fix: whether to apply fixes (default: true). Set to false to only propose changes.
|
||||
|
||||
User arguments: $ARGUMENTS
|
||||
|
||||
## Step 1 — Parallel analysis (read-only)
|
||||
|
||||
First parse the user's `$ARGUMENTS` into `scope` and `fix`: extract the `fix=true|false` token wherever it appears in the string (start, middle, or end), and treat everything else — with that token removed — as `scope`. Defaults: `scope` = your current changes, `fix` = true. The `fix` value is consumed by Step 3 — it does NOT propagate to these passes, which always run `fix=false`.
|
||||
|
||||
Spawn all eight passes concurrently as subagents in a **single message** (multiple Agent tool calls). Each runs its skill on the parsed `scope` with `fix=false` — analysis and proposals ONLY, no edits. Instruct each agent to return its findings as a structured list: for every proposed change, the file path, line range, a one-line description of the change, and the exact before/after so the orchestrator can apply it without re-deriving.
|
||||
|
||||
Run these eight in parallel, substituting the parsed `scope` for `<scope>` in each invocation (pass the real scope text, never the literal `<scope>`):
|
||||
|
||||
1. `/you-might-not-need-an-effect <scope> fix=false`
|
||||
2. `/you-might-not-need-a-memo <scope> fix=false`
|
||||
3. `/you-might-not-need-a-callback <scope> fix=false`
|
||||
4. `/you-might-not-need-state <scope> fix=false`
|
||||
5. `/react-query-best-practices <scope> fix=false`
|
||||
6. `/emcn-design-review <scope> fix=false`
|
||||
7. `/you-might-not-need-url-state <scope> fix=false`
|
||||
8. `/you-might-not-need-a-comment <scope> fix=false`
|
||||
|
||||
## Step 2 — Converge
|
||||
|
||||
Collect all findings into one list, **keeping each proposal tagged with the pass that produced it** — do NOT collapse a file's proposals into a single unlabeled patch, because Step 3 applies in pass order and needs those labels. Detect overlaps where two passes touch the same region (common: a state pass and an effect pass on the same block, or a memo and callback pass on the same component). Reconcile only genuine same-region conflicts, and drop proposals a sibling pass has made moot; a reconciled change inherits the pass label of whichever of its passes comes first in the Step 3 dependency order (effects → state → memo → callback → React Query → url-state → emcn → comments), so it is applied at the earliest safe point. Non-overlapping proposals stay as-is with their own labels. The output is a per-pass list of surviving changes, not a per-file patch.
|
||||
|
||||
## Step 3 — Sequential apply
|
||||
|
||||
If `fix=false`, skip this step — just report the proposals from Step 2.
|
||||
|
||||
Otherwise apply the surviving changes yourself (in the main context, not delegated), iterating **pass by pass** in this dependency order so earlier structural changes settle before later passes build on them:
|
||||
|
||||
1. effects → 2. state → 3. memo → 4. callback → 5. React Query → 6. url-state → 7. emcn design → 8. comments
|
||||
|
||||
For each pass in turn, apply all of that pass's changes, then move to the next pass. A file touched by several passes is therefore edited once per pass, in this order — not once as a merged patch. This is what makes the ordering real: a single merged-per-file patch would collapse all passes into one edit and lose it.
|
||||
|
||||
Comments apply last, on purpose: that pass operates on whatever the earlier structural passes settled the code into, so it never edits lines a sibling pass is about to delete or rewrite.
|
||||
|
||||
**Treat every Step 1 proposal as snapshot-relative, not authoritative.** All passes analyzed the *original* files in parallel, so a proposal's line ranges and before/after text describe the code as it was *before* any edits — once an earlier pass has run, a later pass's snippet may no longer match. So for each change, before applying:
|
||||
|
||||
1. Re-read the file and locate the target by its **content** (the proposal's `old_string` snippet), not by its line number — line numbers from Step 1 are only a hint for where to look, since earlier edits shift them.
|
||||
2. If the `old_string` still matches verbatim, apply it — a content-anchored edit is safe even if its line moved.
|
||||
3. If it no longer matches (an earlier pass altered that region), do **not** force the stale patch. Re-derive the change from the current code by re-applying that pass's rule to the construct, or drop it if a prior pass already made it moot. Never apply a proposal against text it wasn't computed from.
|
||||
|
||||
After all edits, run `bun run lint:check` (it runs `turbo run lint:check` across the repo — there is no per-file target, so run the full check).
|
||||
|
||||
## Step 4 — Summary
|
||||
|
||||
Output a summary across all eight passes: what each found, what was applied vs. skipped-as-redundant, and any proposals that need a human decision.
|
||||
|
||||
## Boundary Audit Guidance
|
||||
|
||||
- When removing route-local Zod schemas, replacing raw `fetch(` calls in hooks, or removing `as unknown as X` casts, do not introduce `// boundary-raw-fetch: <reason>` or `// double-cast-allowed: <reason>` annotations to silence the audit. Fix the underlying call instead — adopt a contract from `@/lib/api/contracts/**` and use `requestJson(contract, ...)` from `@/lib/api/client/request`, or refine the type so the double cast is unnecessary.
|
||||
- Annotations are reserved for legitimate exceptions only: streaming responses, binary downloads, multipart uploads, signed-URL flows, OAuth redirects, external-origin requests, and double casts where no narrower type is available. Each annotation requires a non-empty reason; empty reasons fail `bun run check:api-validation:strict`.
|
||||
@@ -0,0 +1,14 @@
|
||||
---
|
||||
name: council
|
||||
description: Spawn parallel task agents to explore a given area of the codebase from multiple angles, then use their findings to answer the question or build a plan. Use when a task needs broad fan-out exploration across many files before acting.
|
||||
argument-hint: <area-of-interest>
|
||||
# No agents/openai.yaml by design: council is a meta/exploration utility (like cleanup, ship, you-might-not-need-*), not a service-integration builder, so it intentionally ships no standalone agent card.
|
||||
---
|
||||
|
||||
Based on the given area of interest, please:
|
||||
|
||||
1. Dig around the codebase in terms of that given area of interest, gather general information such as keywords and architecture overview.
|
||||
2. Spawn off n=10 (unless specified otherwise) task agents to dig deeper into the codebase in terms of that given area of interest, some of them should be out of the box for variance.
|
||||
3. Once the task agents are done, use the information to do what the user wants.
|
||||
|
||||
If user is in plan mode, use the information to create the plan.
|
||||
@@ -0,0 +1,89 @@
|
||||
---
|
||||
name: db-migrate
|
||||
description: Author or review a Drizzle DB migration for zero-downtime safety — expand/contract phasing, backward-compatibility with the deployed app version, and writing the `-- migration-safe` acknowledgment the check:migrations lint requires. Use when adding/editing files under `packages/db/migrations/` or changing `packages/db/schema.ts`.
|
||||
---
|
||||
|
||||
# DB Migrate Skill
|
||||
|
||||
You make schema changes that survive a deploy without downtime. The `check:migrations` lint (`scripts/check-migrations-safety.ts`) is the deterministic gate; you are the judgment that decides whether a flagged change is actually safe and writes the annotation that satisfies it.
|
||||
|
||||
## The window (why this matters)
|
||||
|
||||
A deploy runs the migration, then rolls out the new app image via blue/green. The two are **not atomic and cannot be** — during cutover the old task set keeps serving against the **already-migrated** schema. So:
|
||||
|
||||
> Every migration must be backward-compatible with the app version that is *already deployed*.
|
||||
|
||||
If a migration drops a column the old code still reads, renames one, or adds a `NOT NULL` the old inserts don't populate, the old code throws until traffic fully shifts — the downtime we're guarding against. You can't fix this by reordering the pipeline; the only fix is discipline.
|
||||
|
||||
## Expand / contract
|
||||
|
||||
Split every breaking change across **two deploys**:
|
||||
|
||||
1. **Expand** (this PR): additive, backward-compatible schema + code that tolerates *both* the old and new shape.
|
||||
2. **Contract** (a later PR, after expand is fully deployed): remove the old thing, now that nothing reads it.
|
||||
|
||||
Never put expand and contract in the same PR. If this PR both removes the code that used a column *and* drops the column, the old code is still live during cutover — split it.
|
||||
|
||||
### Per-operation playbook
|
||||
|
||||
| You want to | Do (deploy 1 = expand) | Do (deploy 2 = contract) |
|
||||
|---|---|---|
|
||||
| Add a required column | `ADD COLUMN` nullable or `DEFAULT`; code writes it | backfill, then `SET NOT NULL` |
|
||||
| Rename a column/table | add the new name; code dual-writes / reads new-then-old | drop the old name |
|
||||
| Drop a column/table | stop all reads/writes in code; ship it | `DROP` (annotate) |
|
||||
| Change a column type | add a new column of the new type; dual-write | backfill, swap reads, drop old |
|
||||
| Add FK / CHECK | `ADD CONSTRAINT ... NOT VALID` | `VALIDATE CONSTRAINT` separately |
|
||||
| Index an existing table | `COMMIT;` breakpoint → `SET lock_timeout = 0` → `CREATE INDEX CONCURRENTLY IF NOT EXISTS` (see `packages/db/scripts/migrate.ts`) | — |
|
||||
| Drop an index | `COMMIT;` breakpoint → `DROP INDEX CONCURRENTLY` — plain `DROP INDEX` takes ACCESS EXCLUSIVE on the table | — |
|
||||
| Backfill data | batched + idempotent `UPDATE` (keyset/`WHERE`, bounded) | — |
|
||||
|
||||
A `CREATE INDEX`, `ADD COLUMN`, or `ADD CONSTRAINT` against a table **created in the same migration** is always safe (no rows, no live traffic) — the lint already suppresses those.
|
||||
|
||||
## Tracking the contract (don't let it rot)
|
||||
|
||||
The contract half is deferred to a later deploy — and that is exactly when it gets forgotten, leaving dead columns, orphaned tables, and `NOT NULL`s that never land. Every deferred contract must become a durable, greppable TODO.
|
||||
|
||||
When an expand defers a drop, leave a **`contract-pending`** marker on the legacy column/table in `packages/db/schema.ts` — that is the file you will be editing when you finally do the drop, so the reminder lives where the work happens:
|
||||
|
||||
```ts
|
||||
// contract-pending(after #5035 is fully deployed): drop once permission-check.ts stops reading it
|
||||
workspaceId: text('workspace_id'),
|
||||
```
|
||||
|
||||
Format: `contract-pending(<precondition>): <what to drop> — <why it's safe once the precondition holds>`. The precondition names the PR/release that removes the last reader and **must be fully deployed** before the contract ships.
|
||||
|
||||
- **The TODO list is a grep** — always accurate, never drifts: `grep -rn "contract-pending" packages/db apps/sim`. Run it when starting migration work to see what is owed.
|
||||
- For anything with a real owner or schedule, also open a tracking issue and put its number in the marker.
|
||||
- **Close the loop in the contract PR:** the contract migration's `-- migration-safe:` annotation references the expand, and you **delete the `contract-pending` marker** in the same PR:
|
||||
```sql
|
||||
-- migration-safe: contract of #5035 — workspace_id readers removed there, deployed 2026-06-10
|
||||
ALTER TABLE "permission_group" DROP COLUMN "workspace_id";
|
||||
```
|
||||
- An expand merged **without** a marker for the drop it defers, or a contract merged **without** removing its marker, is a bug — flag it in review.
|
||||
|
||||
## The judgment the lint can't do
|
||||
|
||||
The lint flags risky *shapes*; it cannot know whether a given drop is *safe right now*. For each flagged statement, do the work it can't:
|
||||
|
||||
1. **Is the dependency gone?** Grep the app for the table/column: search `apps/sim` and `packages` for the column name, the Drizzle field (camelCase), and the table object. If any live read/write remains, it is **not** safe — fix the code first.
|
||||
2. **Did the expand already ship?** The removal of that read/write must be in a deploy that is *already out*, not this same PR. If it's in this PR, split: land the code change now, do the destructive migration in a follow-up after it deploys.
|
||||
3. **Backfills:** confirm the `UPDATE`/`DELETE` is batched (bounded `WHERE`/keyset, not a single whole-table statement), idempotent (safe to replay — a failed migration re-runs unjournaled files from the top), and safe under concurrent writes from the still-live old app.
|
||||
|
||||
## Workflow
|
||||
|
||||
1. Edit `packages/db/schema.ts`, then `cd packages/db && bunx drizzle-kit generate` to produce the SQL. If this is an expand that defers a drop, leave a `contract-pending` marker on the legacy column (see "Tracking the contract"). If this is the contract, delete the marker it resolves.
|
||||
2. Hand-edit the generated SQL where the playbook requires it: `CONCURRENTLY` + `COMMIT;` breakpoint for indexes on existing tables, `NOT VALID` for constraints, batching for backfills.
|
||||
3. Run `bun run check:migrations` (base defaults to `origin/staging`).
|
||||
- **Hard errors** (`add-not-null-no-default`, `rename`, `index-not-concurrent`, `constraint-not-valid`, …): rewrite into expand/contract. Do **not** try to annotate them away — the lint won't accept it.
|
||||
- **Annotate tier** (`drop-table`, `drop-column`, `drop-default`, `set-not-null`, `alter-type`, `drop-index`): only after you've confirmed steps 1–3 above, add a comment on the line directly above the statement:
|
||||
```sql
|
||||
-- migration-safe: `secret` read removed in v0.6.1 (#1234), shipped two deploys ago
|
||||
ALTER TABLE "webhook" DROP COLUMN "secret";
|
||||
```
|
||||
The reason must be specific and name the PR/version that removed the dependency. An empty reason fails the lint.
|
||||
- **Warnings** (`data-backfill`): non-blocking, but confirm the batching/idempotency before merging.
|
||||
4. Verify locally: `cd packages/db && bun run db:migrate` against a dev DB.
|
||||
|
||||
## Hard rule
|
||||
|
||||
Never annotate a destructive statement just to make the lint pass. The annotation is a claim that you verified the old code no longer depends on it. If you can't make that claim truthfully, the change belongs in a later deploy — tell the user to split it.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,85 @@
|
||||
---
|
||||
name: emcn-design-review
|
||||
description: Review UI code for alignment with the emcn design system — components, tokens, patterns, and conventions
|
||||
argument-hint: "[scope] [fix=true|false]"
|
||||
---
|
||||
|
||||
# EMCN Design Review
|
||||
|
||||
Arguments:
|
||||
- scope: what to review (default: your current changes). Examples: "diff to main", "PR #123", "src/components/", "whole codebase"
|
||||
- fix: whether to apply fixes (default: true). Set to false to only propose changes.
|
||||
|
||||
User arguments: $ARGUMENTS
|
||||
|
||||
## Context
|
||||
|
||||
This codebase uses **emcn**, a custom component library built on Radix UI primitives with CVA variants and CSS variable design tokens. All UI must use emcn components and tokens.
|
||||
|
||||
## Steps
|
||||
|
||||
1. Read the emcn public barrel at `packages/emcn/src/index.ts` (re-exports components, Calendar, Table*, and icons) to know what's available; for the full icon set read `packages/emcn/src/icons/index.ts`
|
||||
2. Read `apps/sim/app/_styles/globals.css` for CSS variable tokens
|
||||
3. Analyze the specified scope against every rule below
|
||||
4. If fix=true, apply the fixes. If fix=false, propose the fixes without applying.
|
||||
|
||||
---
|
||||
|
||||
## Imports
|
||||
|
||||
- Import from `@/components/emcn` barrel, never subpaths
|
||||
- Icons from `@sim/emcn/icons`
|
||||
- Use `cn` from `@/lib/core/utils/cn` for conditional classes
|
||||
|
||||
## Design Tokens
|
||||
|
||||
Use CSS variable pattern (`text-[var(--text-primary)]`), never Tailwind semantics (`text-muted-foreground`) or hardcoded colors (`text-gray-500`, `#333`).
|
||||
|
||||
**Text**: `--text-primary`, `--text-secondary`, `--text-tertiary`, `--text-muted`, `--text-body` (canonical value text), `--text-icon`, `--text-placeholder`, `--text-subtle`, `--text-inverse`, `--text-error`
|
||||
**Surfaces**: `--bg`, `--surface-1` through `--surface-7`, `--surface-hover`, `--surface-active`
|
||||
**Borders**: `--border`, `--border-1`, `--border-muted`
|
||||
**Brand/accent**: `--brand-secondary`, `--brand-accent`
|
||||
**Z-Index**: `--z-dropdown` (100), `--z-modal` (200), `--z-popover` (300), `--z-tooltip` (400), `--z-toast` (500)
|
||||
**Shadows**: `shadow-subtle`, `shadow-medium`, `shadow-overlay`, `shadow-card`
|
||||
**Badges**: `--badge-*` semantic families (success/error/gray/blue/purple/orange/amber/teal/cyan/pink, each with `-bg`/`-text`)
|
||||
|
||||
## Buttons
|
||||
|
||||
Intent-to-variant mapping (read the actual `buttonVariants` in `packages/emcn/src/components/button/button.tsx` for the full variant set — it exposes more than listed here):
|
||||
|
||||
| Action | Variant |
|
||||
|--------|---------|
|
||||
| Toolbar, icon-only | `ghost` |
|
||||
| Create, save, submit | `primary` |
|
||||
| Cancel, close | `default` |
|
||||
| Delete, remove | `destructive` |
|
||||
| Selected state | `active` |
|
||||
| Toggle | `outline` |
|
||||
|
||||
## Delete/Remove Confirmations
|
||||
|
||||
`ChipModal` `size='sm'`, title "Delete/Remove {ItemType}", destructive confirm button, plain Cancel (follow the chip footer layout in `.claude/rules/emcn-components.md`). Use `text-[var(--text-error)]` for irreversible warnings.
|
||||
|
||||
## Toast
|
||||
|
||||
`toast.success()`, `toast.error()`, `toast()` from `@/components/emcn`. Never custom notification UI.
|
||||
|
||||
## Badges
|
||||
|
||||
`red`=error/failed, `gray-secondary`=metadata/roles, `type`=type annotations, `green`=success/active, `gray`=neutral, `amber`=processing, `orange`=paused, `blue`=info. Use `dot` prop for status indicators.
|
||||
|
||||
## Icons
|
||||
|
||||
Default: `size-[14px]`. Color: `text-[var(--text-icon)]`. Scale: 14px > 16px > 12px > 20px. Use the `size-*` shorthand — flag `h-[Npx] w-[Npx]` and `h-N w-N` pairs as refactor targets.
|
||||
|
||||
## Anti-patterns to flag
|
||||
|
||||
- Raw `<button>`/`<input>`, or legacy `Input`/`Textarea`/`Modal`, instead of the canonical chip components (`ChipInput`/`ChipTextarea`/`ChipModal`)
|
||||
- Hand-rolled field rows inside a `ChipModalBody` instead of `ChipModalField`
|
||||
- Hardcoded colors (`text-gray-*`, `#hex`, `rgb()`)
|
||||
- Tailwind semantics (`text-muted-foreground`) instead of CSS variables
|
||||
- Template literal className instead of `cn()`
|
||||
- Inline styles for colors/static values (dynamic values OK)
|
||||
- Importing from emcn subpaths instead of barrel
|
||||
- Arbitrary z-index instead of tokens
|
||||
- Wrong button variant for action type
|
||||
@@ -0,0 +1,679 @@
|
||||
---
|
||||
name: emil-design-eng
|
||||
description: This skill encodes Emil Kowalski's philosophy on UI polish, component design, animation decisions, and the invisible details that make software feel great.
|
||||
---
|
||||
|
||||
# Design Engineering
|
||||
|
||||
## Initial Response
|
||||
|
||||
When this skill is first invoked without a specific question, respond only with:
|
||||
|
||||
> I'm ready to help you build interfaces that feel right, my knowledge comes from Emil Kowalski's design engineering philosophy. If you want to dive even deeper, check out Emil’s course: [animations.dev](https://animations.dev/).
|
||||
|
||||
Do not provide any other information until the user asks a question.
|
||||
|
||||
You are a design engineer with the craft sensibility. You build interfaces where every detail compounds into something that feels right. You understand that in a world where everyone's software is good enough, taste is the differentiator.
|
||||
|
||||
## Core Philosophy
|
||||
|
||||
### Taste is trained, not innate
|
||||
|
||||
Good taste is not personal preference. It is a trained instinct: the ability to see beyond the obvious and recognize what elevates. You develop it by surrounding yourself with great work, thinking deeply about why something feels good, and practicing relentlessly.
|
||||
|
||||
When building UI, don't just make it work. Study why the best interfaces feel the way they do. Reverse engineer animations. Inspect interactions. Be curious.
|
||||
|
||||
### Unseen details compound
|
||||
|
||||
Most details users never consciously notice. That is the point. When a feature functions exactly as someone assumes it should, they proceed without giving it a second thought. That is the goal.
|
||||
|
||||
> "All those unseen details combine to produce something that's just stunning, like a thousand barely audible voices all singing in tune." - Paul Graham
|
||||
|
||||
Every decision below exists because the aggregate of invisible correctness creates interfaces people love without knowing why.
|
||||
|
||||
### Beauty is leverage
|
||||
|
||||
People select tools based on the overall experience, not just functionality. Good defaults and good animations are real differentiators. Beauty is underutilized in software. Use it as leverage to stand out.
|
||||
|
||||
## Review Format (Required)
|
||||
|
||||
When reviewing UI code, you MUST use a markdown table with Before/After columns. Do NOT use a list with "Before:" and "After:" on separate lines. Always output an actual markdown table like this:
|
||||
|
||||
| Before | After | Why |
|
||||
| --- | --- | --- |
|
||||
| `transition: all 300ms` | `transition: transform 200ms ease-out` | Specify exact properties; avoid `all` |
|
||||
| `transform: scale(0)` | `transform: scale(0.95); opacity: 0` | Nothing in the real world appears from nothing |
|
||||
| `ease-in` on dropdown | `ease-out` with custom curve | `ease-in` feels sluggish; `ease-out` gives instant feedback |
|
||||
| No `:active` state on button | `transform: scale(0.97)` on `:active` | Buttons must feel responsive to press |
|
||||
| `transform-origin: center` on popover | `transform-origin: var(--radix-popover-content-transform-origin)` | Popovers should scale from their trigger (not modals — modals stay centered) |
|
||||
|
||||
Wrong format (never do this):
|
||||
|
||||
```
|
||||
Before: transition: all 300ms
|
||||
After: transition: transform 200ms ease-out
|
||||
────────────────────────────
|
||||
Before: scale(0)
|
||||
After: scale(0.95)
|
||||
```
|
||||
|
||||
Correct format: A single markdown table with | Before | After | Why | columns, one row per issue found. The "Why" column briefly explains the reasoning.
|
||||
|
||||
## The Animation Decision Framework
|
||||
|
||||
Before writing any animation code, answer these questions in order:
|
||||
|
||||
### 1. Should this animate at all?
|
||||
|
||||
**Ask:** How often will users see this animation?
|
||||
|
||||
| Frequency | Decision |
|
||||
| ----------------------------------------------------------- | ---------------------------- |
|
||||
| 100+ times/day (keyboard shortcuts, command palette toggle) | No animation. Ever. |
|
||||
| Tens of times/day (hover effects, list navigation) | Remove or drastically reduce |
|
||||
| Occasional (modals, drawers, toasts) | Standard animation |
|
||||
| Rare/first-time (onboarding, feedback forms, celebrations) | Can add delight |
|
||||
|
||||
**Never animate keyboard-initiated actions.** These actions are repeated hundreds of times daily. Animation makes them feel slow, delayed, and disconnected from the user's actions.
|
||||
|
||||
Raycast has no open/close animation. That is the optimal experience for something used hundreds of times a day.
|
||||
|
||||
### 2. What is the purpose?
|
||||
|
||||
Every animation must have a clear answer to "why does this animate?"
|
||||
|
||||
Valid purposes:
|
||||
|
||||
- **Spatial consistency**: toast enters and exits from the same direction, making swipe-to-dismiss feel intuitive
|
||||
- **State indication**: a morphing feedback button shows the state change
|
||||
- **Explanation**: a marketing animation that shows how a feature works
|
||||
- **Feedback**: a button scales down on press, confirming the interface heard the user
|
||||
- **Preventing jarring changes**: elements appearing or disappearing without transition feel broken
|
||||
|
||||
If the purpose is just "it looks cool" and the user will see it often, don't animate.
|
||||
|
||||
### 3. What easing should it use?
|
||||
|
||||
Is the element entering or exiting?
|
||||
Yes → ease-out (starts fast, feels responsive)
|
||||
No →
|
||||
Is it moving/morphing on screen?
|
||||
Yes → ease-in-out (natural acceleration/deceleration)
|
||||
Is it a hover/color change?
|
||||
Yes → ease
|
||||
Is it constant motion (marquee, progress bar)?
|
||||
Yes → linear
|
||||
Default → ease-out
|
||||
|
||||
**Critical: use custom easing curves.** The built-in CSS easings are too weak. They lack the punch that makes animations feel intentional.
|
||||
|
||||
```css
|
||||
/* Strong ease-out for UI interactions */
|
||||
--ease-out: cubic-bezier(0.23, 1, 0.32, 1);
|
||||
|
||||
/* Strong ease-in-out for on-screen movement */
|
||||
--ease-in-out: cubic-bezier(0.77, 0, 0.175, 1);
|
||||
|
||||
/* iOS-like drawer curve (from Ionic Framework) */
|
||||
--ease-drawer: cubic-bezier(0.32, 0.72, 0, 1);
|
||||
```
|
||||
|
||||
**Never use ease-in for UI animations.** It starts slow, which makes the interface feel sluggish and unresponsive. A dropdown with `ease-in` at 300ms _feels_ slower than `ease-out` at the same 300ms, because ease-in delays the initial movement — the exact moment the user is watching most closely.
|
||||
|
||||
**Easing curve resources:** Don't create curves from scratch. Use [easing.dev](https://easing.dev/) or [easings.co](https://easings.co/) to find stronger custom variants of standard easings.
|
||||
|
||||
### 4. How fast should it be?
|
||||
|
||||
| Element | Duration |
|
||||
| ------------------------ | ------------- |
|
||||
| Button press feedback | 100-160ms |
|
||||
| Tooltips, small popovers | 125-200ms |
|
||||
| Dropdowns, selects | 150-250ms |
|
||||
| Modals, drawers | 200-500ms |
|
||||
| Marketing/explanatory | Can be longer |
|
||||
|
||||
**Rule: UI animations should stay under 300ms.** A 180ms dropdown feels more responsive than a 400ms one. A faster-spinning spinner makes the app feel like it loads faster, even when the load time is identical.
|
||||
|
||||
### Perceived performance
|
||||
|
||||
Speed in animation is not just about feeling snappy — it directly affects how users perceive your app's performance:
|
||||
|
||||
- A **fast-spinning spinner** makes loading feel faster (same load time, different perception)
|
||||
- A **180ms select** animation feels more responsive than a **400ms** one
|
||||
- **Instant tooltips** after the first one is open (skip delay + skip animation) make the whole toolbar feel faster
|
||||
|
||||
The perception of speed matters as much as actual speed. Easing amplifies this: `ease-out` at 200ms _feels_ faster than `ease-in` at 200ms because the user sees immediate movement.
|
||||
|
||||
## Spring Animations
|
||||
|
||||
Springs feel more natural than duration-based animations because they simulate real physics. They don't have fixed durations — they settle based on physical parameters.
|
||||
|
||||
### When to use springs
|
||||
|
||||
- Drag interactions with momentum
|
||||
- Elements that should feel "alive" (like Apple's Dynamic Island)
|
||||
- Gestures that can be interrupted mid-animation
|
||||
- Decorative mouse-tracking interactions
|
||||
|
||||
### Spring-based mouse interactions
|
||||
|
||||
Tying visual changes directly to mouse position feels artificial because it lacks motion. Use `useSpring` from Motion (formerly Framer Motion) to interpolate value changes with spring-like behavior instead of updating immediately.
|
||||
|
||||
```jsx
|
||||
import { useSpring } from 'framer-motion';
|
||||
|
||||
// Without spring: feels artificial, instant
|
||||
const rotation = mouseX * 0.1;
|
||||
|
||||
// With spring: feels natural, has momentum
|
||||
const springRotation = useSpring(mouseX * 0.1, {
|
||||
stiffness: 100,
|
||||
damping: 10,
|
||||
});
|
||||
```
|
||||
|
||||
This works because the animation is **decorative** — it doesn't serve a function. If this were a functional graph in a banking app, no animation would be better. Know when decoration helps and when it hinders.
|
||||
|
||||
### Spring configuration
|
||||
|
||||
**Apple's approach (recommended — easier to reason about):**
|
||||
|
||||
```js
|
||||
{ type: "spring", duration: 0.5, bounce: 0.2 }
|
||||
```
|
||||
|
||||
**Traditional physics (more control):**
|
||||
|
||||
```js
|
||||
{ type: "spring", mass: 1, stiffness: 100, damping: 10 }
|
||||
```
|
||||
|
||||
Keep bounce subtle (0.1-0.3) when used. Avoid bounce in most UI contexts. Use it for drag-to-dismiss and playful interactions.
|
||||
|
||||
### Interruptibility advantage
|
||||
|
||||
Springs maintain velocity when interrupted — CSS animations and keyframes restart from zero. This makes springs ideal for gestures users might change mid-motion. When you click an expanded item and quickly press Escape, a spring-based animation smoothly reverses from its current position.
|
||||
|
||||
## Component Building Principles
|
||||
|
||||
### Buttons must feel responsive
|
||||
|
||||
Add `transform: scale(0.97)` on `:active`. This gives instant feedback, making the UI feel like it is truly listening to the user.
|
||||
|
||||
```css
|
||||
.button {
|
||||
transition: transform 160ms ease-out;
|
||||
}
|
||||
|
||||
.button:active {
|
||||
transform: scale(0.97);
|
||||
}
|
||||
```
|
||||
|
||||
This applies to any pressable element. The scale should be subtle (0.95-0.98).
|
||||
|
||||
### Never animate from scale(0)
|
||||
|
||||
Nothing in the real world disappears and reappears completely. Elements animating from `scale(0)` look like they come out of nowhere.
|
||||
|
||||
Start from `scale(0.9)` or higher, combined with opacity. Even a barely-visible initial scale makes the entrance feel more natural, like a balloon that has a visible shape even when deflated.
|
||||
|
||||
```css
|
||||
/* Bad */
|
||||
.entering {
|
||||
transform: scale(0);
|
||||
}
|
||||
|
||||
/* Good */
|
||||
.entering {
|
||||
transform: scale(0.95);
|
||||
opacity: 0;
|
||||
}
|
||||
```
|
||||
|
||||
### Make popovers origin-aware
|
||||
|
||||
Popovers should scale in from their trigger, not from center. The default `transform-origin: center` is wrong for almost every popover. **Exception: modals.** Modals should keep `transform-origin: center` because they are not anchored to a specific trigger — they appear centered in the viewport.
|
||||
|
||||
```css
|
||||
/* Radix UI */
|
||||
.popover {
|
||||
transform-origin: var(--radix-popover-content-transform-origin);
|
||||
}
|
||||
|
||||
/* Base UI */
|
||||
.popover {
|
||||
transform-origin: var(--transform-origin);
|
||||
}
|
||||
```
|
||||
|
||||
Whether the user notices the difference individually does not matter. In the aggregate, unseen details become visible. They compound.
|
||||
|
||||
### Tooltips: skip delay on subsequent hovers
|
||||
|
||||
Tooltips should delay before appearing to prevent accidental activation. But once one tooltip is open, hovering over adjacent tooltips should open them instantly with no animation. This feels faster without defeating the purpose of the initial delay.
|
||||
|
||||
```css
|
||||
.tooltip {
|
||||
transition: transform 125ms ease-out, opacity 125ms ease-out;
|
||||
transform-origin: var(--transform-origin);
|
||||
}
|
||||
|
||||
.tooltip[data-starting-style],
|
||||
.tooltip[data-ending-style] {
|
||||
opacity: 0;
|
||||
transform: scale(0.97);
|
||||
}
|
||||
|
||||
/* Skip animation on subsequent tooltips */
|
||||
.tooltip[data-instant] {
|
||||
transition-duration: 0ms;
|
||||
}
|
||||
```
|
||||
|
||||
### Use CSS transitions over keyframes for interruptible UI
|
||||
|
||||
CSS transitions can be interrupted and retargeted mid-animation. Keyframes restart from zero. For any interaction that can be triggered rapidly (adding toasts, toggling states), transitions produce smoother results.
|
||||
|
||||
```css
|
||||
/* Interruptible - good for UI */
|
||||
.toast {
|
||||
transition: transform 400ms ease;
|
||||
}
|
||||
|
||||
/* Not interruptible - avoid for dynamic UI */
|
||||
@keyframes slideIn {
|
||||
from {
|
||||
transform: translateY(100%);
|
||||
}
|
||||
to {
|
||||
transform: translateY(0);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Use blur to mask imperfect transitions
|
||||
|
||||
When a crossfade between two states feels off despite trying different easings and durations, add subtle `filter: blur(2px)` during the transition.
|
||||
|
||||
**Why blur works:** Without blur, you see two distinct objects during a crossfade — the old state and the new state overlapping. This looks unnatural. Blur bridges the visual gap by blending the two states together, tricking the eye into perceiving a single smooth transformation instead of two objects swapping.
|
||||
|
||||
Combine blur with scale-on-press (`scale(0.97)`) for a polished button state transition:
|
||||
|
||||
```css
|
||||
.button {
|
||||
transition: transform 160ms ease-out;
|
||||
}
|
||||
|
||||
.button:active {
|
||||
transform: scale(0.97);
|
||||
}
|
||||
|
||||
.button-content {
|
||||
transition: filter 200ms ease, opacity 200ms ease;
|
||||
}
|
||||
|
||||
.button-content.transitioning {
|
||||
filter: blur(2px);
|
||||
opacity: 0.7;
|
||||
}
|
||||
```
|
||||
|
||||
Keep blur under 20px. Heavy blur is expensive, especially in Safari.
|
||||
|
||||
### Animate enter states with @starting-style
|
||||
|
||||
The modern CSS way to animate element entry without JavaScript:
|
||||
|
||||
```css
|
||||
.toast {
|
||||
opacity: 1;
|
||||
transform: translateY(0);
|
||||
transition: opacity 400ms ease, transform 400ms ease;
|
||||
|
||||
@starting-style {
|
||||
opacity: 0;
|
||||
transform: translateY(100%);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
This replaces the common React pattern of using `useEffect` to set `mounted: true` after initial render. Use `@starting-style` when browser support allows; fall back to the `data-mounted` attribute pattern otherwise.
|
||||
|
||||
```jsx
|
||||
// Legacy pattern (still works everywhere)
|
||||
useEffect(() => {
|
||||
setMounted(true);
|
||||
}, []);
|
||||
// <div data-mounted={mounted}>
|
||||
```
|
||||
|
||||
## CSS Transform Mastery
|
||||
|
||||
### translateY with percentages
|
||||
|
||||
Percentage values in `translate()` are relative to the element's own size. Use `translateY(100%)` to move an element by its own height, regardless of actual dimensions. This is how Sonner positions toasts and how Vaul hides the drawer before animating in.
|
||||
|
||||
```css
|
||||
/* Works regardless of drawer height */
|
||||
.drawer-hidden {
|
||||
transform: translateY(100%);
|
||||
}
|
||||
|
||||
/* Works regardless of toast height */
|
||||
.toast-enter {
|
||||
transform: translateY(-100%);
|
||||
}
|
||||
```
|
||||
|
||||
Prefer percentages over hardcoded pixel values. They are less error-prone and adapt to content.
|
||||
|
||||
### scale() scales children too
|
||||
|
||||
Unlike `width`/`height`, `scale()` also scales an element's children. When scaling a button on press, the font size, icons, and content scale proportionally. This is a feature, not a bug.
|
||||
|
||||
### 3D transforms for depth
|
||||
|
||||
`rotateX()`, `rotateY()` with `transform-style: preserve-3d` create real 3D effects in CSS. Orbiting animations, coin flips, and depth effects are all possible without JavaScript.
|
||||
|
||||
```css
|
||||
.wrapper {
|
||||
transform-style: preserve-3d;
|
||||
}
|
||||
|
||||
@keyframes orbit {
|
||||
from {
|
||||
transform: translate(-50%, -50%) rotateY(0deg) translateZ(72px) rotateY(360deg);
|
||||
}
|
||||
to {
|
||||
transform: translate(-50%, -50%) rotateY(360deg) translateZ(72px) rotateY(0deg);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### transform-origin
|
||||
|
||||
Every element has an anchor point from which transforms execute. The default is center. Set it to match where the trigger lives for origin-aware interactions.
|
||||
|
||||
## clip-path for Animation
|
||||
|
||||
`clip-path` is not just for shapes. It is one of the most powerful animation tools in CSS.
|
||||
|
||||
### The inset shape
|
||||
|
||||
`clip-path: inset(top right bottom left)` defines a rectangular clipping region. Each value "eats" into the element from that side.
|
||||
|
||||
```css
|
||||
/* Fully hidden from right */
|
||||
.hidden {
|
||||
clip-path: inset(0 100% 0 0);
|
||||
}
|
||||
|
||||
/* Fully visible */
|
||||
.visible {
|
||||
clip-path: inset(0 0 0 0);
|
||||
}
|
||||
|
||||
/* Reveal from left to right */
|
||||
.overlay {
|
||||
clip-path: inset(0 100% 0 0);
|
||||
transition: clip-path 200ms ease-out;
|
||||
}
|
||||
.button:active .overlay {
|
||||
clip-path: inset(0 0 0 0);
|
||||
transition: clip-path 2s linear;
|
||||
}
|
||||
```
|
||||
|
||||
### Tabs with perfect color transitions
|
||||
|
||||
Duplicate the tab list. Style the copy as "active" (different background, different text color). Clip the copy so only the active tab is visible. Animate the clip on tab change. This creates a seamless color transition that timing individual color transitions can never achieve.
|
||||
|
||||
### Hold-to-delete pattern
|
||||
|
||||
Use `clip-path: inset(0 100% 0 0)` on a colored overlay. On `:active`, transition to `inset(0 0 0 0)` over 2s with linear timing. On release, snap back with 200ms ease-out. Add `scale(0.97)` on the button for press feedback.
|
||||
|
||||
### Image reveals on scroll
|
||||
|
||||
Start with `clip-path: inset(0 0 100% 0)` (hidden from bottom). Animate to `inset(0 0 0 0)` when the element enters the viewport. Use `IntersectionObserver` or Framer Motion's `useInView` with `{ once: true, margin: "-100px" }`.
|
||||
|
||||
### Comparison sliders
|
||||
|
||||
Overlay two images. Clip the top one with `clip-path: inset(0 50% 0 0)`. Adjust the right inset value based on drag position. No extra DOM elements needed, fully hardware-accelerated.
|
||||
|
||||
## Gesture and Drag Interactions
|
||||
|
||||
### Momentum-based dismissal
|
||||
|
||||
Don't require dragging past a threshold. Calculate velocity: `Math.abs(dragDistance) / elapsedTime`. If velocity exceeds ~0.11, dismiss regardless of distance. A quick flick should be enough.
|
||||
|
||||
```js
|
||||
const timeTaken = new Date().getTime() - dragStartTime.current.getTime();
|
||||
const velocity = Math.abs(swipeAmount) / timeTaken;
|
||||
|
||||
if (Math.abs(swipeAmount) >= SWIPE_THRESHOLD || velocity > 0.11) {
|
||||
dismiss();
|
||||
}
|
||||
```
|
||||
|
||||
### Damping at boundaries
|
||||
|
||||
When a user drags past the natural boundary (e.g., dragging a drawer up when already at top), apply damping. The more they drag, the less the element moves. Things in real life don't suddenly stop; they slow down first.
|
||||
|
||||
### Pointer capture for drag
|
||||
|
||||
Once dragging starts, set the element to capture all pointer events. This ensures dragging continues even if the pointer leaves the element bounds.
|
||||
|
||||
### Multi-touch protection
|
||||
|
||||
Ignore additional touch points after the initial drag begins. Without this, switching fingers mid-drag causes the element to jump to the new position.
|
||||
|
||||
```js
|
||||
function onPress() {
|
||||
if (isDragging) return;
|
||||
// Start drag...
|
||||
}
|
||||
```
|
||||
|
||||
### Friction instead of hard stops
|
||||
|
||||
Instead of preventing upward drag entirely, allow it with increasing friction. It feels more natural than hitting an invisible wall.
|
||||
|
||||
## Performance Rules
|
||||
|
||||
### Only animate transform and opacity
|
||||
|
||||
These properties skip layout and paint, running on the GPU. Animating `padding`, `margin`, `height`, or `width` triggers all three rendering steps.
|
||||
|
||||
### CSS variables are inheritable
|
||||
|
||||
Changing a CSS variable on a parent recalculates styles for all children. In a drawer with many items, updating `--swipe-amount` on the container causes expensive style recalculation. Update `transform` directly on the element instead.
|
||||
|
||||
```js
|
||||
// Bad: triggers recalc on all children
|
||||
element.style.setProperty('--swipe-amount', `${distance}px`);
|
||||
|
||||
// Good: only affects this element
|
||||
element.style.transform = `translateY(${distance}px)`;
|
||||
```
|
||||
|
||||
### Framer Motion hardware acceleration caveat
|
||||
|
||||
Framer Motion's shorthand properties (`x`, `y`, `scale`) are NOT hardware-accelerated. They use `requestAnimationFrame` on the main thread. For hardware acceleration, use the full `transform` string:
|
||||
|
||||
```jsx
|
||||
// NOT hardware accelerated (convenient but drops frames under load)
|
||||
<motion.div animate={{ x: 100 }} />
|
||||
|
||||
// Hardware accelerated (stays smooth even when main thread is busy)
|
||||
<motion.div animate={{ transform: "translateX(100px)" }} />
|
||||
```
|
||||
|
||||
This matters when the browser is simultaneously loading content, running scripts, or painting. At Vercel, the dashboard tab animation used Shared Layout Animations and dropped frames during page loads. Switching to CSS animations (off main thread) fixed it.
|
||||
|
||||
### CSS animations beat JS under load
|
||||
|
||||
CSS animations run off the main thread. When the browser is busy loading a new page, Framer Motion animations (using `requestAnimationFrame`) drop frames. CSS animations remain smooth. Use CSS for predetermined animations; JS for dynamic, interruptible ones.
|
||||
|
||||
### Use WAAPI for programmatic CSS animations
|
||||
|
||||
The Web Animations API gives you JavaScript control with CSS performance. Hardware-accelerated, interruptible, and no library needed.
|
||||
|
||||
```js
|
||||
element.animate([{ clipPath: 'inset(0 0 100% 0)' }, { clipPath: 'inset(0 0 0 0)' }], {
|
||||
duration: 1000,
|
||||
fill: 'forwards',
|
||||
easing: 'cubic-bezier(0.77, 0, 0.175, 1)',
|
||||
});
|
||||
```
|
||||
|
||||
## Accessibility
|
||||
|
||||
### prefers-reduced-motion
|
||||
|
||||
Animations can cause motion sickness. Reduced motion means fewer and gentler animations, not zero. Keep opacity and color transitions that aid comprehension. Remove movement and position animations.
|
||||
|
||||
```css
|
||||
@media (prefers-reduced-motion: reduce) {
|
||||
.element {
|
||||
animation: fade 0.2s ease;
|
||||
/* No transform-based motion */
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
```jsx
|
||||
const shouldReduceMotion = useReducedMotion();
|
||||
const closedX = shouldReduceMotion ? 0 : '-100%';
|
||||
```
|
||||
|
||||
### Touch device hover states
|
||||
|
||||
```css
|
||||
@media (hover: hover) and (pointer: fine) {
|
||||
.element:hover {
|
||||
transform: scale(1.05);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Touch devices trigger hover on tap, causing false positives. Gate hover animations behind this media query.
|
||||
|
||||
## The Sonner Principles (Building Loved Components)
|
||||
|
||||
These principles come from building Sonner (13M+ weekly npm downloads) and apply to any component:
|
||||
|
||||
1. **Developer experience is key.** No hooks, no context, no complex setup. Insert `<Toaster />` once, call `toast()` from anywhere. The less friction to adopt, the more people will use it.
|
||||
|
||||
2. **Good defaults matter more than options.** Ship beautiful out of the box. Most users never customize. The default easing, timing, and visual design should be excellent.
|
||||
|
||||
3. **Naming creates identity.** "Sonner" (French for "to ring") feels more elegant than "react-toast". Sacrifice discoverability for memorability when appropriate.
|
||||
|
||||
4. **Handle edge cases invisibly.** Pause toast timers when the tab is hidden. Fill gaps between stacked toasts with pseudo-elements to maintain hover state. Capture pointer events during drag. Users never notice these, and that is exactly right.
|
||||
|
||||
5. **Use transitions, not keyframes, for dynamic UI.** Toasts are added rapidly. Keyframes restart from zero on interruption. Transitions retarget smoothly.
|
||||
|
||||
6. **Build a great documentation site.** Let people touch the product, play with it, and understand it before they use it. Interactive examples with ready-to-use code snippets lower the barrier to adoption.
|
||||
|
||||
### Cohesion matters
|
||||
|
||||
Sonner's animation feels satisfying partly because the whole experience is cohesive. The easing and duration fit the vibe of the library. It is slightly slower than typical UI animations and uses `ease` rather than `ease-out` to feel more elegant. The animation style matches the toast design, the page design, the name — everything is in harmony.
|
||||
|
||||
When choosing animation values, consider the personality of the component. A playful component can be bouncier. A professional dashboard should be crisp and fast. Match the motion to the mood.
|
||||
|
||||
### The opacity + height combination
|
||||
|
||||
When items enter and exit a list (like Family's drawer), the opacity change must work well with the height animation. This is often trial and error. There is no formula — you adjust until it feels right.
|
||||
|
||||
### Review your work the next day
|
||||
|
||||
Review animations with fresh eyes. You notice imperfections the next day that you missed during development. Play animations in slow motion or frame by frame to spot timing issues that are invisible at full speed.
|
||||
|
||||
### Asymmetric enter/exit timing
|
||||
|
||||
Pressing should be slow when it needs to be deliberate (hold-to-delete: 2s linear), but release should always be snappy (200ms ease-out). This pattern applies broadly: slow where the user is deciding, fast where the system is responding.
|
||||
|
||||
```css
|
||||
/* Release: fast */
|
||||
.overlay {
|
||||
transition: clip-path 200ms ease-out;
|
||||
}
|
||||
|
||||
/* Press: slow and deliberate */
|
||||
.button:active .overlay {
|
||||
transition: clip-path 2s linear;
|
||||
}
|
||||
```
|
||||
|
||||
## Stagger Animations
|
||||
|
||||
When multiple elements enter together, stagger their appearance. Each element animates in with a small delay after the previous one. This creates a cascading effect that feels more natural than everything appearing at once.
|
||||
|
||||
```css
|
||||
.item {
|
||||
opacity: 0;
|
||||
transform: translateY(8px);
|
||||
animation: fadeIn 300ms ease-out forwards;
|
||||
}
|
||||
|
||||
.item:nth-child(1) {
|
||||
animation-delay: 0ms;
|
||||
}
|
||||
.item:nth-child(2) {
|
||||
animation-delay: 50ms;
|
||||
}
|
||||
.item:nth-child(3) {
|
||||
animation-delay: 100ms;
|
||||
}
|
||||
.item:nth-child(4) {
|
||||
animation-delay: 150ms;
|
||||
}
|
||||
|
||||
@keyframes fadeIn {
|
||||
to {
|
||||
opacity: 1;
|
||||
transform: translateY(0);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Keep stagger delays short (30-80ms between items). Long delays make the interface feel slow. Stagger is decorative — never block interaction while stagger animations are playing.
|
||||
|
||||
## Debugging Animations
|
||||
|
||||
### Slow motion testing
|
||||
|
||||
Play animations at reduced speed to spot issues invisible at full speed. Temporarily increase duration to 2-5x normal, or use browser DevTools animation inspector to slow playback.
|
||||
|
||||
Things to look for in slow motion:
|
||||
|
||||
- Do colors transition smoothly, or do you see two distinct states overlapping?
|
||||
- Does the easing feel right, or does it start/stop abruptly?
|
||||
- Is the transform-origin correct, or does the element scale from the wrong point?
|
||||
- Are multiple animated properties (opacity, transform, color) in sync?
|
||||
|
||||
### Frame-by-frame inspection
|
||||
|
||||
Step through animations frame by frame in Chrome DevTools (Animations panel). This reveals timing issues between coordinated properties that you cannot see at full speed.
|
||||
|
||||
### Test on real devices
|
||||
|
||||
For touch interactions (drawers, swipe gestures), test on physical devices. Connect your phone via USB, visit your local dev server by IP address, and use Safari's remote devtools. The Xcode Simulator is an alternative but real hardware is better for gesture testing.
|
||||
|
||||
## Review Checklist
|
||||
|
||||
When reviewing UI code, check for:
|
||||
|
||||
| Issue | Fix |
|
||||
| ------------------------------------------ | ---------------------------------------------------------------- |
|
||||
| `transition: all` | Specify exact properties: `transition: transform 200ms ease-out` |
|
||||
| `scale(0)` entry animation | Start from `scale(0.95)` with `opacity: 0` |
|
||||
| `ease-in` on UI element | Switch to `ease-out` or custom curve |
|
||||
| `transform-origin: center` on popover | Set to trigger location or use Radix/Base UI CSS variable (modals are exempt — keep centered) |
|
||||
| Animation on keyboard action | Remove animation entirely |
|
||||
| Duration > 300ms on UI element | Reduce to 150-250ms |
|
||||
| Hover animation without media query | Add `@media (hover: hover) and (pointer: fine)` |
|
||||
| Keyframes on rapidly-triggered element | Use CSS transitions for interruptibility |
|
||||
| Framer Motion `x`/`y` props under load | Use `transform: "translateX()"` for hardware acceleration |
|
||||
| Same enter/exit transition speed | Make exit faster than enter (e.g., enter 2s, exit 200ms) |
|
||||
| Elements all appear at once | Add stagger delay (30-80ms between items) |
|
||||
@@ -0,0 +1,148 @@
|
||||
---
|
||||
name: make-interfaces-feel-better
|
||||
description: Design engineering principles for making interfaces feel polished. Use when building UI components, reviewing frontend code, implementing animations, hover states, shadows, borders, typography, micro-interactions, enter/exit animations, or any visual detail work. Triggers on UI polish, design details, "make it feel better", "feels off", stagger animations, border radius, optical alignment, font smoothing, tabular numbers, image outlines, box shadows.
|
||||
---
|
||||
|
||||
# Details that make interfaces feel better
|
||||
|
||||
Great interfaces rarely come from a single thing. It's usually a collection of small details that compound into a great experience. Apply these principles when building or reviewing UI code.
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Category | When to Use |
|
||||
| --- | --- |
|
||||
| [Typography](typography.md) | Text wrapping, font smoothing, tabular numbers |
|
||||
| [Surfaces](surfaces.md) | Border radius, optical alignment, shadows, image outlines, hit areas |
|
||||
| [Animations](animations.md) | Interruptible animations, enter/exit transitions, icon animations, scale on press |
|
||||
| [Performance](performance.md) | Transition specificity, `will-change` usage |
|
||||
|
||||
## Core Principles
|
||||
|
||||
### 1. Concentric Border Radius
|
||||
|
||||
Outer radius = inner radius + padding. Mismatched radii on nested elements is the most common thing that makes interfaces feel off.
|
||||
|
||||
### 2. Optical Over Geometric Alignment
|
||||
|
||||
When geometric centering looks off, align optically. Buttons with icons, play triangles, and asymmetric icons all need manual adjustment.
|
||||
|
||||
### 3. Shadows Over Borders
|
||||
|
||||
Layer multiple transparent `box-shadow` values for natural depth. Shadows adapt to any background; solid borders don't.
|
||||
|
||||
### 4. Interruptible Animations
|
||||
|
||||
Use CSS transitions for interactive state changes — they can be interrupted mid-animation. Reserve keyframes for staged sequences that run once.
|
||||
|
||||
### 5. Split and Stagger Enter Animations
|
||||
|
||||
Don't animate a single container. Break content into semantic chunks and stagger each with ~100ms delay.
|
||||
|
||||
### 6. Subtle Exit Animations
|
||||
|
||||
Use a small fixed `translateY` instead of full height. Exits should be softer than enters.
|
||||
|
||||
### 7. Contextual Icon Animations
|
||||
|
||||
Animate icons with `opacity`, `scale`, and `blur` instead of toggling visibility. Use exactly these values: scale from `0.25` to `1`, opacity from `0` to `1`, blur from `4px` to `0px`. If the project has `motion` or `framer-motion` in `package.json`, use `transition: { type: "spring", duration: 0.3, bounce: 0 }` — bounce must always be `0`. If no motion library is installed, keep both icons in the DOM (one absolute-positioned) and cross-fade with CSS transitions using `cubic-bezier(0.2, 0, 0, 1)` — this gives both enter and exit animations without any dependency.
|
||||
|
||||
### 8. Font Smoothing
|
||||
|
||||
Apply `-webkit-font-smoothing: antialiased` to the root layout on macOS for crisper text.
|
||||
|
||||
### 9. Tabular Numbers
|
||||
|
||||
Use `font-variant-numeric: tabular-nums` for any dynamically updating numbers to prevent layout shift.
|
||||
|
||||
### 10. Text Wrapping
|
||||
|
||||
Use `text-wrap: balance` on headings. Use `text-wrap: pretty` for body text to avoid orphans.
|
||||
|
||||
### 11. Image Outlines
|
||||
|
||||
Add a subtle `1px` outline with low opacity to images for consistent depth. The color must be pure black in light mode (`rgba(0, 0, 0, 0.1)`) and pure white in dark mode (`rgba(255, 255, 255, 0.1)`) — never a near-black like slate, zinc, or any tinted neutral. A tinted outline picks up the surface color underneath it and reads as dirt on the image edge.
|
||||
|
||||
### 12. Scale on Press
|
||||
|
||||
A subtle `scale(0.96)` on click gives buttons tactile feedback. Always use `0.96`. Never use a value smaller than `0.95` — anything below feels exaggerated. Add a `static` prop to disable it when motion would be distracting.
|
||||
|
||||
### 13. Skip Animation on Page Load
|
||||
|
||||
Use `initial={false}` on `AnimatePresence` to prevent enter animations on first render. Verify it doesn't break intentional entrance animations.
|
||||
|
||||
### 14. Never Use `transition: all`
|
||||
|
||||
Always specify exact properties: `transition-property: scale, opacity`. Tailwind's `transition-transform` covers `transform, translate, scale, rotate`.
|
||||
|
||||
### 15. Use `will-change` Sparingly
|
||||
|
||||
Only for `transform`, `opacity`, `filter` — properties the GPU can composite. Never use `will-change: all`. Only add when you notice first-frame stutter.
|
||||
|
||||
### 16. Minimum Hit Area
|
||||
|
||||
Interactive elements need at least 40×40px hit area. Extend with a pseudo-element if the visible element is smaller. Never let hit areas of two elements overlap.
|
||||
|
||||
## Common Mistakes
|
||||
|
||||
| Mistake | Fix |
|
||||
| --- | --- |
|
||||
| Same border radius on parent and child | Calculate `outerRadius = innerRadius + padding` |
|
||||
| Icons look off-center | Adjust optically with padding or fix SVG directly |
|
||||
| Hard borders between sections | Use layered `box-shadow` with transparency |
|
||||
| Jarring enter/exit animations | Split, stagger, and keep exits subtle |
|
||||
| Numbers cause layout shift | Apply `tabular-nums` |
|
||||
| Heavy text on macOS | Apply `antialiased` to root |
|
||||
| Animation plays on page load | Add `initial={false}` to `AnimatePresence` |
|
||||
| `transition: all` on elements | Specify exact properties |
|
||||
| First-frame animation stutter | Add `will-change: transform` (sparingly) |
|
||||
| Tiny hit areas on small controls | Extend with pseudo-element to 40×40px |
|
||||
|
||||
## Review Output Format
|
||||
|
||||
Always present changes as a markdown table with **Before** and **After** columns. Include every change you made — not just a subset. Never list findings as separate "Before:" / "After:" lines outside of a table. Group changes by principle using a heading above each table, and keep each row focused on a single diff so the reader can scan the whole list quickly.
|
||||
|
||||
### Example
|
||||
|
||||
#### Concentric border radius
|
||||
| Before | After |
|
||||
| --- | --- |
|
||||
| `rounded-xl` on card + `rounded-xl` on inner button (`p-2`) | `rounded-2xl` on card (`12 + 8`), `rounded-lg` on inner button |
|
||||
| `border-radius: 16px` on both nested surfaces | Outer `24px`, inner `16px` with `8px` padding |
|
||||
|
||||
#### Tabular numbers
|
||||
| Before | After |
|
||||
| --- | --- |
|
||||
| `<span>{count}</span>` on animated counter | `<span className="tabular-nums">{count}</span>` |
|
||||
| Default numerals on timer | Added `font-variant-numeric: tabular-nums` to root |
|
||||
|
||||
#### Scale on press
|
||||
| Before | After |
|
||||
| --- | --- |
|
||||
| `<button className="...">` | Added `active:scale-[0.96] transition-transform` |
|
||||
| `scale(0.9)` on press | Raised to `scale(0.96)` — anything below `0.95` feels exaggerated |
|
||||
|
||||
Rows should cite the specific file and the specific property that changed when it isn't obvious from the snippet. If a principle was reviewed but nothing needed to change, omit that table entirely — empty tables add noise.
|
||||
|
||||
## Review Checklist
|
||||
|
||||
- [ ] Nested rounded elements use concentric border radius
|
||||
- [ ] Icons are optically centered, not just geometrically
|
||||
- [ ] Shadows used instead of borders where appropriate
|
||||
- [ ] Enter animations are split and staggered
|
||||
- [ ] Exit animations are subtle
|
||||
- [ ] Dynamic numbers use tabular-nums
|
||||
- [ ] Font smoothing is applied
|
||||
- [ ] Headings use text-wrap: balance
|
||||
- [ ] Images have subtle outlines
|
||||
- [ ] Buttons use scale on press where appropriate
|
||||
- [ ] AnimatePresence uses `initial={false}` for default-state elements
|
||||
- [ ] No `transition: all` — only specific properties
|
||||
- [ ] `will-change` only on transform/opacity/filter, never `all`
|
||||
- [ ] Interactive elements have at least 40×40px hit area
|
||||
|
||||
## Reference Files
|
||||
|
||||
- [typography.md](typography.md) — Text wrapping, font smoothing, tabular numbers
|
||||
- [surfaces.md](surfaces.md) — Border radius, optical alignment, shadows, image outlines
|
||||
- [animations.md](animations.md) — Interruptible animations, enter/exit transitions, icon animations, scale on press
|
||||
- [performance.md](performance.md) — Transition specificity, `will-change` usage
|
||||
@@ -0,0 +1,379 @@
|
||||
# Animations
|
||||
|
||||
Interruptible animations, enter/exit transitions, and contextual icon animations.
|
||||
|
||||
## Interruptible Animations
|
||||
|
||||
Users change intent mid-interaction. If animations aren't interruptible, the interface feels broken.
|
||||
|
||||
### CSS Transitions vs. Keyframes
|
||||
|
||||
| | CSS Transitions | CSS Keyframe Animations |
|
||||
| --- | --- | --- |
|
||||
| **Behavior** | Interpolate toward latest state | Run on a fixed timeline |
|
||||
| **Interruptible** | Yes — retargets mid-animation | No — restarts from beginning |
|
||||
| **Use for** | Interactive state changes (hover, toggle, open/close) | Staged sequences that run once (enter animations, loading) |
|
||||
| **Duration** | Adapts to remaining distance | Fixed regardless of state |
|
||||
|
||||
```css
|
||||
/* Good — interruptible transition for a toggle */
|
||||
.drawer {
|
||||
transform: translateX(-100%);
|
||||
transition: transform 200ms ease-out;
|
||||
}
|
||||
.drawer.open {
|
||||
transform: translateX(0);
|
||||
}
|
||||
|
||||
/* Clicking again mid-animation smoothly reverses — no jank */
|
||||
```
|
||||
|
||||
```css
|
||||
/* Bad — keyframe animation for interactive element */
|
||||
.drawer.open {
|
||||
animation: slideIn 200ms ease-out forwards;
|
||||
}
|
||||
|
||||
/* Closing mid-animation snaps or restarts — feels broken */
|
||||
```
|
||||
|
||||
**Rule:** Always prefer CSS transitions for interactive elements. Reserve keyframes for one-shot sequences.
|
||||
|
||||
## Enter Animations: Split and Stagger
|
||||
|
||||
Don't animate a single large container. Break content into semantic chunks and animate each individually.
|
||||
|
||||
### Step by Step
|
||||
|
||||
1. **Split** into logical groups (title, description, buttons)
|
||||
2. **Stagger** with ~100ms delay between groups
|
||||
3. **For titles**, consider splitting into individual words with ~80ms stagger
|
||||
4. **Combine** `opacity`, `blur`, and `translateY` for the enter effect
|
||||
|
||||
### Code Example
|
||||
|
||||
```tsx
|
||||
// Motion (Framer Motion) — staggered enter
|
||||
function PageHeader() {
|
||||
return (
|
||||
<motion.div
|
||||
initial="hidden"
|
||||
animate="visible"
|
||||
variants={{
|
||||
visible: { transition: { staggerChildren: 0.1 } },
|
||||
}}
|
||||
>
|
||||
<motion.h1
|
||||
variants={{
|
||||
hidden: { opacity: 0, y: 12, filter: "blur(4px)" },
|
||||
visible: { opacity: 1, y: 0, filter: "blur(0px)" },
|
||||
}}
|
||||
>
|
||||
Welcome
|
||||
</motion.h1>
|
||||
|
||||
<motion.p
|
||||
variants={{
|
||||
hidden: { opacity: 0, y: 12, filter: "blur(4px)" },
|
||||
visible: { opacity: 1, y: 0, filter: "blur(0px)" },
|
||||
}}
|
||||
>
|
||||
A description of the page.
|
||||
</motion.p>
|
||||
|
||||
<motion.div
|
||||
variants={{
|
||||
hidden: { opacity: 0, y: 12, filter: "blur(4px)" },
|
||||
visible: { opacity: 1, y: 0, filter: "blur(0px)" },
|
||||
}}
|
||||
>
|
||||
<Button>Get started</Button>
|
||||
</motion.div>
|
||||
</motion.div>
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
### CSS-Only Stagger
|
||||
|
||||
```css
|
||||
.stagger-item {
|
||||
opacity: 0;
|
||||
transform: translateY(12px);
|
||||
filter: blur(4px);
|
||||
animation: fadeInUp 400ms ease-out forwards;
|
||||
}
|
||||
|
||||
.stagger-item:nth-child(1) { animation-delay: 0ms; }
|
||||
.stagger-item:nth-child(2) { animation-delay: 100ms; }
|
||||
.stagger-item:nth-child(3) { animation-delay: 200ms; }
|
||||
|
||||
@keyframes fadeInUp {
|
||||
to {
|
||||
opacity: 1;
|
||||
transform: translateY(0);
|
||||
filter: blur(0);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Exit Animations
|
||||
|
||||
Exit animations should be softer and less attention-grabbing than enter animations. The user's focus is moving to the next thing — don't fight for attention.
|
||||
|
||||
### Subtle Exit (Recommended)
|
||||
|
||||
```tsx
|
||||
// Small fixed translateY — indicates direction without drama
|
||||
<motion.div
|
||||
exit={{
|
||||
opacity: 0,
|
||||
y: -12,
|
||||
filter: "blur(4px)",
|
||||
transition: { duration: 0.15, ease: "easeIn" },
|
||||
}}
|
||||
>
|
||||
{content}
|
||||
</motion.div>
|
||||
```
|
||||
|
||||
### Full Exit (When Context Matters)
|
||||
|
||||
```tsx
|
||||
// Slide fully out — use when spatial context is important
|
||||
// (e.g., a card returning to a list, a drawer closing)
|
||||
<motion.div
|
||||
exit={{
|
||||
opacity: 0,
|
||||
x: "-100%",
|
||||
transition: { duration: 0.2, ease: "easeIn" },
|
||||
}}
|
||||
>
|
||||
{content}
|
||||
</motion.div>
|
||||
```
|
||||
|
||||
### Good vs. Bad
|
||||
|
||||
```css
|
||||
/* Good — subtle exit */
|
||||
.item-exit {
|
||||
opacity: 0;
|
||||
transform: translateY(-12px);
|
||||
transition: opacity 150ms ease-in, transform 150ms ease-in;
|
||||
}
|
||||
|
||||
/* Bad — dramatic exit that steals focus */
|
||||
.item-exit {
|
||||
opacity: 0;
|
||||
transform: translateY(-100%) scale(0.5);
|
||||
transition: all 400ms ease-in;
|
||||
}
|
||||
|
||||
/* Bad — no exit animation at all (element just vanishes) */
|
||||
.item-exit {
|
||||
display: none;
|
||||
}
|
||||
```
|
||||
|
||||
**Key points:**
|
||||
- Use a small fixed `translateY` (e.g., `-12px`) instead of the full container height
|
||||
- Keep some directional movement to indicate where the element went
|
||||
- Exit duration should be shorter than enter duration (150ms vs 300ms)
|
||||
- Don't remove exit animations entirely — subtle motion preserves context
|
||||
|
||||
## Contextual Icon Animations
|
||||
|
||||
When icons appear or disappear contextually (on hover, on state change), animate them with `opacity`, `scale`, and `blur` rather than just toggling visibility.
|
||||
|
||||
### Motion Example
|
||||
|
||||
```tsx
|
||||
import { AnimatePresence, motion } from "motion/react";
|
||||
|
||||
function IconButton({ isActive, icon: Icon }) {
|
||||
return (
|
||||
<button>
|
||||
<AnimatePresence mode="popLayout">
|
||||
<motion.span
|
||||
key={isActive ? "active" : "inactive"}
|
||||
initial={{ opacity: 0, scale: 0.25, filter: "blur(4px)" }}
|
||||
animate={{ opacity: 1, scale: 1, filter: "blur(0px)" }}
|
||||
exit={{ opacity: 0, scale: 0.25, filter: "blur(4px)" }}
|
||||
transition={{ type: "spring", duration: 0.3, bounce: 0 }}
|
||||
>
|
||||
<Icon />
|
||||
</motion.span>
|
||||
</AnimatePresence>
|
||||
</button>
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
### CSS Transition Approach (No Motion)
|
||||
|
||||
If the project doesn't use Motion (Framer Motion), keep both icons in the DOM and cross-fade them with CSS transitions. Because neither icon unmounts, both enter and exit animate smoothly.
|
||||
|
||||
The trick: one icon is absolutely positioned on top of the other. Toggling state cross-fades them — the entering icon scales up from `0.25` while the exiting icon scales down to `0.25`, both with opacity and blur.
|
||||
|
||||
```tsx
|
||||
function IconButton({ isActive, ActiveIcon, InactiveIcon }) {
|
||||
return (
|
||||
<button>
|
||||
<div className="relative">
|
||||
<div
|
||||
className={cn(
|
||||
"absolute inset-0 flex items-center justify-center",
|
||||
"transition-[opacity,filter,scale] duration-300",
|
||||
"cubic-bezier(0.2, 0, 0, 1)",
|
||||
isActive
|
||||
? "scale-100 opacity-100 blur-0"
|
||||
: "scale-[0.25] opacity-0 blur-[4px]"
|
||||
)}
|
||||
>
|
||||
<ActiveIcon />
|
||||
</div>
|
||||
<div
|
||||
className={cn(
|
||||
"transition-[opacity,filter,scale] duration-300",
|
||||
"cubic-bezier(0.2, 0, 0, 1)",
|
||||
isActive
|
||||
? "scale-[0.25] opacity-0 blur-[4px]"
|
||||
: "scale-100 opacity-100 blur-0"
|
||||
)}
|
||||
>
|
||||
<InactiveIcon />
|
||||
</div>
|
||||
</div>
|
||||
</button>
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
The non-absolute icon (InactiveIcon) defines the layout size. The absolute icon (ActiveIcon) overlays it without affecting flow.
|
||||
|
||||
### Choosing Between Motion and CSS
|
||||
|
||||
| | Motion (Framer Motion) | CSS transitions (both icons in DOM) |
|
||||
| --- | --- | --- |
|
||||
| **Enter animation** | Yes | Yes |
|
||||
| **Exit animation** | Yes (via `AnimatePresence`) | Yes (cross-fade — icon never unmounts) |
|
||||
| **Spring physics** | Yes | No — use `cubic-bezier(0.2, 0, 0, 1)` as approximation |
|
||||
| **When to use** | Project already uses `motion/react` | No motion dependency, or keeping bundle small |
|
||||
|
||||
**Rule:** Check the project's `package.json` for `motion` or `framer-motion`. If present, use the Motion approach. If not, use the CSS cross-fade pattern — don't add a dependency just for icon transitions.
|
||||
|
||||
### When to Animate Icons
|
||||
|
||||
| Animate | Don't animate |
|
||||
| --- | --- |
|
||||
| Icons that appear on hover (action buttons) | Static navigation icons |
|
||||
| State change icons (play → pause, like → liked) | Decorative icons |
|
||||
| Icons in contextual toolbars | Icons that are always visible |
|
||||
| Loading/success state indicators | Icon labels (text next to icon) |
|
||||
|
||||
**Important:** Always use exactly these values for contextual icon animations — do not deviate:
|
||||
- `scale`: `0.25` → `1` (never use `0.5` or `0.6`)
|
||||
- `opacity`: `0` → `1`
|
||||
- `filter`: `"blur(4px)"` → `"blur(0px)"`
|
||||
- `transition`: `{ type: "spring", duration: 0.3, bounce: 0 }` — **bounce must always be `0`**, never `0.1` or any other value
|
||||
|
||||
## Scale on Press
|
||||
|
||||
A subtle scale-down on click gives buttons tactile feedback. Always use `scale(0.96)`. Never use a value smaller than `0.95` — anything below feels exaggerated. Use CSS transitions for interruptibility — if the user releases mid-press, it should smoothly return.
|
||||
|
||||
Not every button needs this. Add a `static` prop to your button component that disables the scale effect when the motion would be distracting.
|
||||
|
||||
### CSS Example
|
||||
|
||||
```css
|
||||
.button {
|
||||
transition-property: scale;
|
||||
transition-duration: 150ms;
|
||||
transition-timing-function: ease-out;
|
||||
}
|
||||
|
||||
.button:active {
|
||||
scale: 0.96;
|
||||
}
|
||||
```
|
||||
|
||||
### Tailwind Example
|
||||
|
||||
```tsx
|
||||
<button className="transition-transform duration-150 ease-out active:scale-[0.96]">
|
||||
Click me
|
||||
</button>
|
||||
```
|
||||
|
||||
### Motion Example
|
||||
|
||||
```tsx
|
||||
<motion.button whileTap={{ scale: 0.96 }}>
|
||||
Click me
|
||||
</motion.button>
|
||||
```
|
||||
|
||||
### Static Prop Pattern
|
||||
|
||||
Extract the scale class into a variable and conditionally apply it based on a `static` prop:
|
||||
|
||||
```tsx
|
||||
const tapScale = "active:not-disabled:scale-[0.96]";
|
||||
|
||||
function Button({ static: isStatic, className, children, ...props }) {
|
||||
return (
|
||||
<button
|
||||
className={cn(
|
||||
"transition-transform duration-150 ease-out",
|
||||
!isStatic && tapScale,
|
||||
className,
|
||||
)}
|
||||
{...props}
|
||||
>
|
||||
{children}
|
||||
</button>
|
||||
);
|
||||
}
|
||||
|
||||
// Usage
|
||||
<Button>Click me</Button> {/* scales on press */}
|
||||
<Button static>Submit</Button> {/* no scale */}
|
||||
```
|
||||
|
||||
## Skip Animation on Page Load
|
||||
|
||||
Use `initial={false}` on `AnimatePresence` to prevent enter animations from firing on first render. Elements that are already in their default state shouldn't animate in on page load — only on subsequent state changes.
|
||||
|
||||
### When It Works
|
||||
|
||||
```tsx
|
||||
// Good — icon doesn't animate in on mount, only on state change
|
||||
<AnimatePresence initial={false} mode="popLayout">
|
||||
<motion.span
|
||||
key={isActive ? "active" : "inactive"}
|
||||
initial={{ opacity: 0, scale: 0.25, filter: "blur(4px)" }}
|
||||
animate={{ opacity: 1, scale: 1, filter: "blur(0px)" }}
|
||||
exit={{ opacity: 0, scale: 0.25, filter: "blur(4px)" }}
|
||||
>
|
||||
<Icon />
|
||||
</motion.span>
|
||||
</AnimatePresence>
|
||||
```
|
||||
|
||||
Works well for: icon swaps, toggles, tabs, segmented controls — anything that has a default state on page load.
|
||||
|
||||
### When It Breaks
|
||||
|
||||
Don't use `initial={false}` when the component relies on its `initial` prop to set up a first-time enter animation, like a staggered page hero or a loading state. In those cases, removing the initial animation skips the entire entrance.
|
||||
|
||||
```tsx
|
||||
// Bad — initial={false} would skip the staggered page enter entirely
|
||||
<AnimatePresence initial={false}>
|
||||
<motion.div initial="hidden" animate="visible" variants={...}>
|
||||
...
|
||||
</motion.div>
|
||||
</AnimatePresence>
|
||||
```
|
||||
|
||||
Verify the component still looks right on a full page refresh before applying this.
|
||||
@@ -0,0 +1,88 @@
|
||||
# Performance
|
||||
|
||||
Transition specificity and GPU compositing hints.
|
||||
|
||||
## Transition Only What Changes
|
||||
|
||||
Never use `transition: all` or Tailwind's `transition` shorthand (which maps to `transition-property: all`). Always specify the exact properties that change.
|
||||
|
||||
### Why
|
||||
|
||||
- `transition: all` forces the browser to watch every property for changes
|
||||
- Causes unexpected transitions on properties you didn't intend to animate (colors, padding, shadows)
|
||||
- Prevents browser optimizations
|
||||
|
||||
### CSS Example
|
||||
|
||||
```css
|
||||
/* Good — only transition what changes */
|
||||
.button {
|
||||
transition-property: scale, background-color;
|
||||
transition-duration: 150ms;
|
||||
transition-timing-function: ease-out;
|
||||
}
|
||||
|
||||
/* Bad — transition everything */
|
||||
.button {
|
||||
transition: all 150ms ease-out;
|
||||
}
|
||||
```
|
||||
|
||||
### Tailwind
|
||||
|
||||
```tsx
|
||||
// Good — explicit properties
|
||||
<button className="transition-[scale,background-color] duration-150 ease-out">
|
||||
|
||||
// Bad — transition all
|
||||
<button className="transition duration-150 ease-out">
|
||||
```
|
||||
|
||||
### Tailwind `transition-transform` Note
|
||||
|
||||
`transition-transform` in Tailwind maps to `transition-property: transform, translate, scale, rotate` — it covers all transform-related properties, not just `transform`. Use this when you're only animating transforms. For multiple non-transform properties, use the bracket syntax: `transition-[scale,opacity,filter]`.
|
||||
|
||||
## Use `will-change` Sparingly
|
||||
|
||||
`will-change` hints the browser to pre-promote an element to its own GPU compositing layer. Without it, the browser promotes the element only when the animation starts — that one-time layer promotion can cause a micro-stutter on the first frame.
|
||||
|
||||
This particularly helps when an element is changing `scale`, `rotation`, or moving around with `transform`. For other properties, it doesn't help much — the browser can't composite them on the GPU anyway.
|
||||
|
||||
### Rules
|
||||
|
||||
```css
|
||||
/* Good — specific property that benefits from GPU compositing */
|
||||
.animated-card {
|
||||
will-change: transform;
|
||||
}
|
||||
|
||||
/* Good — multiple compositor-friendly properties */
|
||||
.animated-card {
|
||||
will-change: transform, opacity;
|
||||
}
|
||||
|
||||
/* Bad — never use will-change: all */
|
||||
.animated-card {
|
||||
will-change: all;
|
||||
}
|
||||
|
||||
/* Bad — properties that can't be GPU-composited anyway */
|
||||
.animated-card {
|
||||
will-change: background-color, padding;
|
||||
}
|
||||
```
|
||||
|
||||
### Useful Properties
|
||||
|
||||
| Property | GPU-compositable | Worth using `will-change` |
|
||||
| --- | --- | --- |
|
||||
| `transform` | Yes | Yes |
|
||||
| `opacity` | Yes | Yes |
|
||||
| `filter` (blur, brightness) | Yes | Yes |
|
||||
| `clip-path` | Yes | Yes |
|
||||
| `top`, `left`, `width`, `height` | No | No |
|
||||
| `background`, `border`, `color` | No | No |
|
||||
|
||||
### When to Skip
|
||||
|
||||
Modern browsers are already good at optimizing on their own. Only add `will-change` when you notice first-frame stutter — Safari in particular benefits from it. Don't add it preemptively to every animated element; each extra compositing layer costs memory.
|
||||
@@ -0,0 +1,256 @@
|
||||
# Surfaces
|
||||
|
||||
Border radius, optical alignment, shadows, and image outlines.
|
||||
|
||||
## Concentric Border Radius
|
||||
|
||||
When nesting rounded elements, the outer radius must equal the inner radius plus the padding between them:
|
||||
|
||||
```
|
||||
outerRadius = innerRadius + padding
|
||||
```
|
||||
|
||||
This rule is most useful when nested surfaces are close together. If padding is larger than `24px`, treat the layers as separate surfaces and choose each radius independently instead of forcing strict concentric math.
|
||||
|
||||
### Example
|
||||
|
||||
```css
|
||||
/* Good — concentric radii */
|
||||
.card {
|
||||
border-radius: 20px; /* 12 + 8 */
|
||||
padding: 8px;
|
||||
}
|
||||
.card-inner {
|
||||
border-radius: 12px;
|
||||
}
|
||||
|
||||
/* Bad — same radius on both */
|
||||
.card {
|
||||
border-radius: 12px;
|
||||
padding: 8px;
|
||||
}
|
||||
.card-inner {
|
||||
border-radius: 12px;
|
||||
}
|
||||
```
|
||||
|
||||
### Tailwind Example
|
||||
|
||||
```tsx
|
||||
// Good — outer radius accounts for padding
|
||||
<div className="rounded-2xl p-2"> {/* 16px radius, 8px padding */}
|
||||
<div className="rounded-lg"> {/* 8px radius = 16 - 8 ✓ */}
|
||||
...
|
||||
</div>
|
||||
</div>
|
||||
|
||||
// Bad — same radius on both
|
||||
<div className="rounded-xl p-2">
|
||||
<div className="rounded-xl"> {/* same radius, looks off */}
|
||||
...
|
||||
</div>
|
||||
</div>
|
||||
```
|
||||
|
||||
Mismatched border radii on nested elements is one of the most common things that makes interfaces feel off. Always calculate concentrically.
|
||||
|
||||
## Optical Alignment
|
||||
|
||||
When geometric centering looks off, align optically instead.
|
||||
|
||||
### Buttons with Text + Icon
|
||||
|
||||
Use slightly less padding on the icon side to make the button feel balanced. A reliable rule of thumb is:
|
||||
`icon-side padding = text-side padding - 2px`.
|
||||
|
||||
```css
|
||||
/* Good — less padding on icon side */
|
||||
.button-with-icon {
|
||||
padding-left: 16px;
|
||||
padding-right: 14px; /* icon side = text side - 2px */
|
||||
}
|
||||
|
||||
/* Bad — equal padding looks like icon is pushed too far right */
|
||||
.button-with-icon {
|
||||
padding: 0 16px;
|
||||
}
|
||||
```
|
||||
|
||||
```tsx
|
||||
// Tailwind
|
||||
<button className="pl-4 pr-3.5 flex items-center gap-2">
|
||||
<span>Continue</span>
|
||||
<ArrowRightIcon />
|
||||
</button>
|
||||
```
|
||||
|
||||
### Play Button Triangles
|
||||
|
||||
Play icons are triangular and their geometric center is not their visual center. Shift slightly right:
|
||||
|
||||
```css
|
||||
/* Good — optically centered */
|
||||
.play-button svg {
|
||||
margin-left: 2px; /* shift right to account for triangle shape */
|
||||
}
|
||||
|
||||
/* Bad — geometrically centered but looks off */
|
||||
.play-button svg {
|
||||
/* no adjustment */
|
||||
}
|
||||
```
|
||||
|
||||
### Asymmetric Icons (Stars, Arrows, Carets)
|
||||
|
||||
Some icons have uneven visual weight. The best fix is adjusting the SVG directly so no extra margin/padding is needed in the component code.
|
||||
|
||||
```tsx
|
||||
// Best — fix in the SVG itself
|
||||
// Adjust the viewBox or path to visually center the icon
|
||||
|
||||
// Fallback — adjust with margin
|
||||
<span className="ml-px">
|
||||
<StarIcon />
|
||||
</span>
|
||||
```
|
||||
|
||||
## Shadows Instead of Borders
|
||||
|
||||
For **buttons, cards, and containers** that use a border for depth or elevation, prefer replacing it with a subtle `box-shadow`. Shadows adapt to any background since they use transparency; solid borders don't. This also helps when using images or multiple colors as backgrounds — solid border colors don't work well on backgrounds other than the ones they were designed for.
|
||||
|
||||
**Do not apply this to dividers** (`border-b`, `border-t`, side borders) or any border whose purpose is layout separation rather than element depth. Those should stay as borders.
|
||||
|
||||
### Shadow as Border (Light Mode)
|
||||
|
||||
The shadow is comprised of three layers. The first acts as a 1px border ring, the second adds subtle lift, and the third provides ambient depth:
|
||||
|
||||
```css
|
||||
:root {
|
||||
--shadow-border:
|
||||
0px 0px 0px 1px rgba(0, 0, 0, 0.06),
|
||||
0px 1px 2px -1px rgba(0, 0, 0, 0.06),
|
||||
0px 2px 4px 0px rgba(0, 0, 0, 0.04);
|
||||
--shadow-border-hover:
|
||||
0px 0px 0px 1px rgba(0, 0, 0, 0.08),
|
||||
0px 1px 2px -1px rgba(0, 0, 0, 0.08),
|
||||
0px 2px 4px 0px rgba(0, 0, 0, 0.06);
|
||||
}
|
||||
```
|
||||
|
||||
### Shadow as Border (Dark Mode)
|
||||
|
||||
In dark mode, simplify to a single white ring — layered depth shadows aren't visible on dark backgrounds:
|
||||
|
||||
```css
|
||||
/* Dark mode — adapt to whatever setup the project uses
|
||||
(prefers-color-scheme, class, data attribute, etc.) */
|
||||
--shadow-border: 0 0 0 1px rgba(255, 255, 255, 0.08);
|
||||
--shadow-border-hover: 0 0 0 1px rgba(255, 255, 255, 0.13);
|
||||
```
|
||||
|
||||
### Usage with Hover Transition
|
||||
|
||||
Apply the variable and add `transition-[box-shadow]` for a smooth hover:
|
||||
|
||||
```css
|
||||
.card {
|
||||
box-shadow: var(--shadow-border);
|
||||
transition-property: box-shadow;
|
||||
transition-duration: 150ms;
|
||||
transition-timing-function: ease-out;
|
||||
}
|
||||
|
||||
.card:hover {
|
||||
box-shadow: var(--shadow-border-hover);
|
||||
}
|
||||
```
|
||||
|
||||
### When to Use Shadows vs. Borders
|
||||
|
||||
| Use shadows | Use borders |
|
||||
| --- | --- |
|
||||
| Cards, containers with depth | Dividers between list items |
|
||||
| Buttons with bordered styles | Table cell boundaries |
|
||||
| Elevated elements (dropdowns, modals) | Form input outlines (for accessibility) |
|
||||
| Elements on varied backgrounds | Hairline separators in dense UI |
|
||||
| Hover/focus states for lift effect | |
|
||||
|
||||
## Image Outlines
|
||||
|
||||
Add a subtle `1px` outline with low opacity to images. This creates consistent depth, especially in design systems where other elements use borders or shadows.
|
||||
|
||||
### Color rules (non-negotiable)
|
||||
|
||||
- **Light mode**: pure black — `rgba(0, 0, 0, 0.1)`. Exact values: R=0, G=0, B=0.
|
||||
- **Dark mode**: pure white — `rgba(255, 255, 255, 0.1)`. Exact values: R=255, G=255, B=255.
|
||||
- Never use a near-black or near-white from the project palette (e.g. slate-900, zinc-900, `#0a0a0a`, `#111827`, `#f5f5f7`). Tinted outlines pick up the surrounding surface color and read as dirt on the image edge.
|
||||
- Never match the outline to the project's accent or ink color. The outline is a neutral separator, not a themed element.
|
||||
|
||||
### Light Mode
|
||||
|
||||
```css
|
||||
img {
|
||||
outline: 1px solid rgba(0, 0, 0, 0.1);
|
||||
outline-offset: -1px; /* inset so it doesn't add to layout */
|
||||
}
|
||||
```
|
||||
|
||||
### Dark Mode
|
||||
|
||||
```css
|
||||
img {
|
||||
outline: 1px solid rgba(255, 255, 255, 0.1);
|
||||
outline-offset: -1px;
|
||||
}
|
||||
```
|
||||
|
||||
### Tailwind with Dark Mode
|
||||
|
||||
```tsx
|
||||
<img
|
||||
className="outline outline-1 -outline-offset-1 outline-black/10 dark:outline-white/10"
|
||||
src={src}
|
||||
alt={alt}
|
||||
/>
|
||||
```
|
||||
|
||||
Use `outline-black/10` and `outline-white/10` specifically — not `outline-slate-*`, `outline-zinc-*`, `outline-neutral-*`, or any tinted scale.
|
||||
|
||||
**Why outline instead of border?** `outline` doesn't affect layout (no added width/height), and `outline-offset: -1px` keeps it inset so images stay their intended size.
|
||||
|
||||
## Minimum Hit Area
|
||||
|
||||
Interactive elements should have a minimum hit area of 44×44px (WCAG) or at least 40×40px. If the visible element is smaller (e.g., a 20×20 checkbox), extend the hit area with a pseudo-element.
|
||||
|
||||
### CSS Example
|
||||
|
||||
```css
|
||||
/* Small checkbox with expanded hit area */
|
||||
.checkbox {
|
||||
position: relative;
|
||||
width: 20px;
|
||||
height: 20px;
|
||||
}
|
||||
|
||||
.checkbox::after {
|
||||
content: "";
|
||||
position: absolute;
|
||||
top: 50%;
|
||||
left: 50%;
|
||||
transform: translate(-50%, -50%);
|
||||
width: 40px;
|
||||
height: 40px;
|
||||
}
|
||||
```
|
||||
|
||||
### Tailwind Example
|
||||
|
||||
```tsx
|
||||
<button className="relative size-5 after:absolute after:top-1/2 after:left-1/2 after:size-10 after:-translate-1/2">
|
||||
<CheckIcon />
|
||||
</button>
|
||||
```
|
||||
|
||||
### Collision Rule
|
||||
|
||||
If the extended hit area overlaps another interactive element, shrink the pseudo-element — but make it as large as possible without colliding. Two interactive elements should never have overlapping hit areas.
|
||||
@@ -0,0 +1,135 @@
|
||||
# Typography
|
||||
|
||||
Typography rendering details that make interfaces feel better.
|
||||
|
||||
## Text Wrapping
|
||||
|
||||
### text-wrap: balance
|
||||
|
||||
Distributes text evenly across lines, preventing orphaned words on headings and short text blocks. **Only works on blocks of 6 lines or fewer** (Chromium) or 10 lines or fewer (Firefox) — the balancing algorithm is computationally expensive, so browsers limit it to short text.
|
||||
|
||||
```css
|
||||
/* Good — even line lengths on short text */
|
||||
h1, h2, h3 {
|
||||
text-wrap: balance;
|
||||
}
|
||||
```
|
||||
|
||||
```css
|
||||
/* Bad — default wrapping leaves orphans */
|
||||
h1 {
|
||||
/* no text-wrap rule → "Read our
|
||||
blog" instead of balanced lines */
|
||||
}
|
||||
```
|
||||
|
||||
```css
|
||||
/* Bad — balance on long paragraphs (silently ignored, wastes intent) */
|
||||
.article-body p {
|
||||
text-wrap: balance;
|
||||
}
|
||||
```
|
||||
|
||||
**Tailwind:** `text-balance`
|
||||
|
||||
### text-wrap: pretty
|
||||
|
||||
Prevents orphaned words (a single word dangling on the last line) by adjusting line breaks throughout the paragraph. Unlike `balance`, it doesn't try to equalize line lengths — it just ensures the last line isn't embarrassingly short. Works on text of any length with no line-count limit.
|
||||
|
||||
This should be your **default for short-to-medium text** — paragraphs, descriptions, captions, list items, card text. For very long text (10+ lines), skip both `pretty` and `balance` — the browser's default wrapping is fine and you avoid unnecessary layout cost.
|
||||
|
||||
```css
|
||||
/* Good — descriptions, captions, short paragraphs */
|
||||
p, li, figcaption, blockquote {
|
||||
text-wrap: pretty;
|
||||
}
|
||||
```
|
||||
|
||||
```tsx
|
||||
// Tailwind
|
||||
<p className="text-pretty">
|
||||
A short paragraph that won't leave an orphan on the last line.
|
||||
</p>
|
||||
```
|
||||
|
||||
**Tailwind:** `text-pretty`
|
||||
|
||||
### When to Use Which
|
||||
|
||||
| Scenario | Use |
|
||||
| --- | --- |
|
||||
| Headings, titles where even distribution matters | `text-wrap: balance` |
|
||||
| Short-to-medium text — paragraphs, descriptions, captions, UI text | `text-wrap: pretty` |
|
||||
| Long text (10+ lines), code blocks, pre-formatted text | Neither — leave default |
|
||||
|
||||
## Font Smoothing (macOS)
|
||||
|
||||
On macOS, text renders heavier than intended by default. Apply antialiased smoothing to the root layout so all text renders crisper and thinner.
|
||||
|
||||
```css
|
||||
/* CSS */
|
||||
html {
|
||||
-webkit-font-smoothing: antialiased;
|
||||
-moz-osx-font-smoothing: grayscale;
|
||||
}
|
||||
```
|
||||
|
||||
```tsx
|
||||
// Tailwind — apply to root layout
|
||||
<html className="antialiased">
|
||||
```
|
||||
|
||||
### Good vs. Bad
|
||||
|
||||
```css
|
||||
/* Good — applied once at the root */
|
||||
html {
|
||||
-webkit-font-smoothing: antialiased;
|
||||
}
|
||||
|
||||
/* Bad — applied per-element, inconsistent */
|
||||
.heading {
|
||||
-webkit-font-smoothing: antialiased;
|
||||
}
|
||||
.body {
|
||||
/* no smoothing → heavier than heading */
|
||||
}
|
||||
```
|
||||
|
||||
**Note:** This only affects macOS rendering. Other platforms ignore these properties, so it's safe to apply universally.
|
||||
|
||||
## Tabular Numbers
|
||||
|
||||
When numbers update dynamically (counters, prices, timers, table columns), use tabular-nums to make all digits equal width. This prevents layout shift as values change.
|
||||
|
||||
```css
|
||||
/* CSS */
|
||||
.counter {
|
||||
font-variant-numeric: tabular-nums;
|
||||
}
|
||||
```
|
||||
|
||||
```tsx
|
||||
// Tailwind
|
||||
<span className="tabular-nums">{count}</span>
|
||||
```
|
||||
|
||||
### When to Use
|
||||
|
||||
| Use tabular-nums | Don't use tabular-nums |
|
||||
| --- | --- |
|
||||
| Counters and timers | Static display numbers |
|
||||
| Prices that update | Decorative large numbers |
|
||||
| Table columns with numbers | Phone numbers, zip codes |
|
||||
| Animated number transitions | Version numbers (v2.1.0) |
|
||||
| Scoreboards, dashboards | |
|
||||
|
||||
### Caveat
|
||||
|
||||
Some fonts (like Inter) change the visual appearance of numerals with this property — specifically, the digit `1` becomes wider and centered. This is expected behavior and usually desirable for alignment, but verify it looks right in your specific font.
|
||||
|
||||
```css
|
||||
/* With Inter font:
|
||||
Default: 1234 → proportional, "1" is narrow
|
||||
Tabular: 1234 → all digits equal width, "1" centered */
|
||||
```
|
||||
@@ -0,0 +1,164 @@
|
||||
---
|
||||
name: memory-load-check
|
||||
description: Review PRs and diffs for unbounded memory loading, concurrency explosions, oversized payload materialization, and missing pagination or byte caps. Use when reviewing cleanup jobs, background jobs, data imports/exports, file parsing, API fan-out, workflow execution payloads, large arrays/files, or any change that reads many rows, files, responses, logs, or external API pages into process memory.
|
||||
---
|
||||
|
||||
# Memory Load Check
|
||||
|
||||
Use this skill when a PR or diff could load unbounded data into a Node/Bun process, especially in cron routes, background tasks, API routes, workflow execution, file parsing, cleanup jobs, migrations, import/export flows, and external API integrations.
|
||||
|
||||
## Review Goal
|
||||
|
||||
Prove each changed path has explicit bounds for:
|
||||
- rows held in memory
|
||||
- bytes held in memory
|
||||
- concurrent promises, DB queries, HTTP calls, storage operations, and jobs
|
||||
- number of pages, batches, chunks, retries, and retained intermediate objects
|
||||
|
||||
If any bound depends only on current production size or "probably small" data, treat it as a finding.
|
||||
|
||||
## References
|
||||
|
||||
Read these when doing a deeper pass:
|
||||
- Node.js streams/backpressure: https://nodejs.org/learn/modules/backpressuring-in-streams
|
||||
- Node.js stream usage: https://nodejs.org/en/learn/modules/how-to-use-streams
|
||||
- Keyset/cursor pagination over offset scans: https://blog.sequinstream.com/keyset-cursors-not-offsets-for-postgres-pagination/
|
||||
- Postgres pagination tradeoffs: https://www.citusdata.com/blog/2016/03/30/five-ways-to-paginate/
|
||||
|
||||
## Sim Helpers To Prefer
|
||||
|
||||
- `apps/sim/lib/cleanup/batch-delete.ts`
|
||||
- `chunkedBatchDelete`: bounded SELECT -> optional side effect -> DELETE loop.
|
||||
- `batchDeleteByWorkspaceAndTimestamp`: common workspace/timestamp cleanup wrapper.
|
||||
- `selectRowsByIdChunks`: chunks large ID sets and enforces an overall row cap.
|
||||
- `chunkArray`: use only after the input set itself is already bounded.
|
||||
- `apps/sim/lib/core/utils/stream-limits.ts`
|
||||
- `PayloadSizeLimitError`
|
||||
- `assertKnownSizeWithinLimit`
|
||||
- `assertContentLengthWithinLimit`
|
||||
- `readStreamToBufferWithLimit`
|
||||
- `readNodeStreamToBufferWithLimit`
|
||||
- `readResponseToBufferWithLimit`
|
||||
- `readResponseTextWithLimit`
|
||||
- Cleanup dispatcher pattern in `apps/sim/lib/billing/cleanup-dispatcher.ts`
|
||||
- page active workspaces with `WHERE id > afterId ORDER BY id LIMIT N`
|
||||
- dispatch concrete chunks (`workspaceIds`, retention, label) instead of one giant scope
|
||||
- prefer Trigger.dev queue/concurrency keys when available
|
||||
- execute inline fallback chunks sequentially, not with unbounded `Promise.all`
|
||||
- File parse route pattern in `apps/sim/app/api/files/parse/route.ts`
|
||||
- cap downloads and parsed output separately
|
||||
- preserve partial results when a later item exceeds the cap
|
||||
- never read untrusted response bodies without a byte cap
|
||||
- KB connector file downloads in `apps/sim/connectors/utils.ts`
|
||||
- `CONNECTOR_MAX_FILE_BYTES`: shared per-file cap (aligned with the manual KB upload limit)
|
||||
- `readBodyWithLimit`: stream a download body to a Buffer with a hard byte cap (null on overflow)
|
||||
- `stubOrSkipBySize`: listing-time skip when the reported size exceeds the cap
|
||||
- `markSkipped` / `sizeLimitSkipReason`: surface oversized files as failed (skipped) KB rows
|
||||
- `ConnectorFileTooLargeError`: thrown mid-download when the listing under-reported size
|
||||
- Large workflow value payloads
|
||||
- prefer durable references/manifests over inlining large arrays or files
|
||||
- materialize refs only behind an explicit byte budget
|
||||
|
||||
## KB Connector File Size Handling
|
||||
|
||||
The connector size pattern in `apps/sim/connectors/utils.ts` (`CONNECTOR_MAX_FILE_BYTES` + `readBodyWithLimit` + `stubOrSkipBySize`/`markSkipped`) exists for one risk: a knowledge-base connector downloading **arbitrary, user-controlled file bytes** that the source does not hard-cap. Apply it by that risk, not by the connector's name.
|
||||
|
||||
Use the pattern when the connector downloads file content via a stream/`download_url` where the user controls the size:
|
||||
- file-storage connectors: Dropbox, OneDrive, SharePoint, Google Drive, S3, GitHub, GitLab, Azure DevOps
|
||||
- any connector that fetches a file via a download URL even if it is not a "storage" service (e.g. the Zoom transcript `.vtt`)
|
||||
|
||||
For those, require all three:
|
||||
- stream the body with `readBodyWithLimit(resp, CONNECTOR_MAX_FILE_BYTES)` — never raw `response.text()`/`response.arrayBuffer()`
|
||||
- skip oversize at listing (`stubOrSkipBySize` with the reported size) and again at fetch time (overflow -> `markSkipped`), since the listing size can be missing or under-reported
|
||||
- never drop/truncate silently — oversized files become content-less failed rows carrying `skippedReason`, so they stay visible in the KB UI instead of vanishing from the index
|
||||
|
||||
Skip the pattern when the source already bounds the payload:
|
||||
- pure API/structured-data connectors (Jira, Linear, Notion, Confluence, Sentry, Slack, Zendesk, Gmail, ...) — paginated JSON/text; apply normal pagination + concurrency bounds instead of a per-file byte cap
|
||||
- native-document connectors capped by the platform (Google Docs ~50 MB, Google Sheets via `MAX_ROWS`, Evernote ~25 MB/note) — a 100 MB cap can never fire, and wrapping a `response.json()`/Thrift parse in `readBodyWithLimit` is cargo-culting
|
||||
|
||||
Litmus test: "Can a user make this one fetch arbitrarily large, with nothing upstream stopping it?" Yes -> use the pattern. No (platform hard-cap, or already paginated) -> a per-file byte cap adds noise, not safety. Borderline: a user-configured/self-hosted endpoint with no platform cap (e.g. Obsidian) — bound it only if the content is genuinely unbounded.
|
||||
|
||||
## Review Workflow
|
||||
|
||||
1. Identify every changed data source:
|
||||
- database queries
|
||||
- storage lists/downloads/uploads
|
||||
- external API pagination
|
||||
- file reads and HTTP responses
|
||||
- workflow logs, snapshots, payloads, arrays, and manifests
|
||||
- queues, cron routes, and background jobs
|
||||
2. For each source, write down the maximum cardinality and maximum bytes. If the code does not enforce one, it is unbounded.
|
||||
3. Trace whether data is processed incrementally or accumulated:
|
||||
- arrays from `select`, `findMany`, `Promise.all`, `map`, `filter`, `flatMap`
|
||||
- maps/sets keyed by all users, workspaces, executions, files, or rows
|
||||
- `Buffer.concat`, `response.arrayBuffer()`, `response.text()`, `JSON.stringify`, `JSON.parse`
|
||||
- queues of promises or job payloads built before dispatch
|
||||
4. Check concurrency separately from memory:
|
||||
- no `Promise.all(items.map(...))` unless `items` is already small and bounded
|
||||
- use chunks, sequential loops, queue concurrency, or a concurrency limiter
|
||||
- align concurrency with DB pool size, storage/API limits, and task queue semantics
|
||||
5. Verify SQL shape:
|
||||
- every bulk query has `LIMIT`
|
||||
- large pagination uses cursor/keyset style (`id > afterId`, timestamps plus unique ID), not deep `OFFSET`
|
||||
- `IN (...)` lists are chunked
|
||||
- side-effect rows selected before delete have per-batch and per-run caps
|
||||
6. Verify byte safety:
|
||||
- check `Content-Length` when available
|
||||
- stream with cumulative byte accounting
|
||||
- cap both input bytes and expanded output bytes
|
||||
- reject or reference oversized values before serializing large JSON responses
|
||||
7. Confirm failure behavior:
|
||||
- exceeding a cap should stop before loading more data
|
||||
- partial successful work should be preserved when the API contract expects it
|
||||
- retries should not duplicate huge in-memory state
|
||||
- cleanup jobs should make progress over future runs instead of widening one run
|
||||
|
||||
## Red Flags
|
||||
|
||||
- loads all active workspaces, users, executions, logs, files, messages, or subscriptions before filtering
|
||||
- builds a full `Map` or `Set` for a platform-wide scope
|
||||
- uses `Promise.all` over rows from an unbounded query
|
||||
- fetches all pages from an external API before processing
|
||||
- reads an entire file, HTTP response, or stream without a max byte budget
|
||||
- checks size only after `Buffer.concat`, `arrayBuffer`, `text`, `JSON.parse`, or parse expansion
|
||||
- a KB connector silently drops or truncates an oversized file instead of recording it as a failed (skipped) row
|
||||
- chunks only after loading the complete dataset
|
||||
- paginates with unbounded/deep `OFFSET` on a mutable or large table
|
||||
- creates one queue job per row without batching or a queue-level concurrency key
|
||||
- accumulates per-row errors/results with no maximum
|
||||
- adds a cache, singleton, or module-level collection without eviction or size limits
|
||||
|
||||
## Preferred Fixes
|
||||
|
||||
- Move filters into SQL/API requests and select only needed columns.
|
||||
- Replace full-table loads with cursor/keyset pagination and a deterministic order.
|
||||
- Process one page/batch at a time; do not keep previous pages unless needed.
|
||||
- Add per-batch and per-run row caps so long backlogs drain across repeated jobs.
|
||||
- Split large ID lists with `selectRowsByIdChunks` or `chunkArray` after bounding the source.
|
||||
- Use `chunkedBatchDelete` for cleanup loops with row side effects.
|
||||
- Use stream-limit helpers for file/HTTP/body reads.
|
||||
- Store large workflow values as refs/manifests and materialize only within a caller budget.
|
||||
- Replace unbounded `Promise.all` with sequential chunk loops, queue concurrency, or a small limiter.
|
||||
- Include tests that prove caps stop work early and partial results or progress are preserved.
|
||||
|
||||
## Findings Format
|
||||
|
||||
Lead with concrete findings, ordered by risk:
|
||||
|
||||
```markdown
|
||||
## Findings
|
||||
|
||||
- **P1 Unbounded workspace load in cleanup dispatch** (`path/to/file.ts`)
|
||||
The new path calls `select().from(workspace)` without a limit, then builds maps for every row before dispatch. In production this scales with all active workspaces and can exhaust the app process. Page by `workspace.id` with a fixed limit and dispatch bounded chunks.
|
||||
|
||||
## Good Signals
|
||||
|
||||
- Uses `readResponseToBufferWithLimit` for external downloads.
|
||||
- Inline fallback processes chunks sequentially.
|
||||
|
||||
## Residual Risk
|
||||
|
||||
- The row cap is explicit, but no test currently proves the loop stops at the cap.
|
||||
```
|
||||
|
||||
Only say "good to go" when every changed source has explicit row, byte, and concurrency bounds or the boundedness is proven by a stable invariant.
|
||||
@@ -0,0 +1,55 @@
|
||||
---
|
||||
name: react-query-best-practices
|
||||
description: Audit React Query usage for best practices — key factories, staleTime, mutations, and server state ownership
|
||||
argument-hint: "[scope] [fix=true|false]"
|
||||
---
|
||||
|
||||
# React Query Best Practices
|
||||
|
||||
Arguments:
|
||||
- scope: what to analyze (default: your current changes). Examples: "diff to main", "PR #123", "src/hooks/queries/", "whole codebase"
|
||||
- fix: whether to apply fixes (default: true). Set to false to only propose changes.
|
||||
|
||||
User arguments: $ARGUMENTS
|
||||
|
||||
## Context
|
||||
|
||||
This codebase uses React Query (TanStack Query) as the single source of truth for all server state. All query hooks live in `hooks/queries/`. Zustand is used only for client-only UI state. Server data must never be duplicated into useState or Zustand outside of mutation callbacks that coordinate cross-store state.
|
||||
|
||||
## References
|
||||
|
||||
Read these before analyzing:
|
||||
1. https://tkdodo.eu/blog/practical-react-query — foundational defaults, custom hooks, avoiding local state copies
|
||||
2. https://tkdodo.eu/blog/effective-react-query-keys — key factory pattern, hierarchical keys, fuzzy invalidation
|
||||
3. https://tkdodo.eu/blog/react-query-as-a-state-manager — React Query IS your server state manager
|
||||
|
||||
## Rules to enforce
|
||||
|
||||
### Query key factories
|
||||
- Every file in `hooks/queries/` must have a hierarchical key factory with an `all` root key
|
||||
- Keys must include intermediate plural keys (`lists`, `details`) for prefix invalidation
|
||||
- Key factories are colocated with their query hooks, not in a global keys file
|
||||
|
||||
### Query hooks
|
||||
- Every `queryFn` must forward `signal` for request cancellation
|
||||
- Every query must have an explicit `staleTime` (default 0 is almost never correct), assigned from a named exported constant — never an inline numeric literal. A server-side prefetch hydrating the same query key must import and reuse that constant instead of restating the number
|
||||
- `keepPreviousData` / `placeholderData` only on variable-key queries (where params change), never on static keys
|
||||
- Use `enabled` to prevent queries from running without required params
|
||||
|
||||
### Mutations
|
||||
- Use `onSettled` (not `onSuccess`) for cache reconciliation — it fires on both success and error
|
||||
- For optimistic updates: save previous data in `onMutate`, roll back in `onError`
|
||||
- Use targeted invalidation (`entityKeys.lists()`) not broad (`entityKeys.all`) when possible
|
||||
- Don't include mutation objects in `useCallback` deps — `.mutate()` is stable
|
||||
|
||||
### Server state ownership
|
||||
- Never copy query data into useState. Use query data directly in components.
|
||||
- Never copy query data into Zustand stores (exception: mutation callbacks that coordinate cross-store state like temp ID replacement)
|
||||
- The query cache is not a local state manager — `setQueryData` is for optimistic updates only
|
||||
- Forms are the one deliberate exception: copy server data into local form state with `staleTime: Infinity`
|
||||
|
||||
## Steps
|
||||
|
||||
1. Read the references above to understand the guidelines
|
||||
2. Analyze the specified scope against the rules listed above
|
||||
3. If fix=true, apply the fixes. If fix=false, propose the fixes without applying.
|
||||
@@ -0,0 +1,148 @@
|
||||
---
|
||||
name: ship
|
||||
description: Commit, push, and open a PR to staging in one shot — runs the cleanup pass and, when migrations changed, the db-migrate safety review first
|
||||
argument-hint: "[optional context or scope notes]"
|
||||
---
|
||||
|
||||
# Ship Command
|
||||
|
||||
You help ship code by creating commits, pushing to the remote branch, and creating PRs in the user's voice.
|
||||
|
||||
## Your Task
|
||||
|
||||
When the user runs `/ship`:
|
||||
|
||||
1. **Check git status** - See what files have changed
|
||||
2. **Sync check**: `git fetch origin staging && git log --oneline origin/staging..HEAD`. Read the actual commit list, not just how many there are — it must show ONLY commits you can attribute to this session (recognizable subjects/SHAs). A worktree/branch can silently be cut from a stale local `staging`, dragging in unrelated commits; a corrupted branch's inflated commit *count* can coincidentally match a later check even when the *commits* are wrong, so always compare content, never just a number.
|
||||
- If it shows commits you don't recognize, fix it now, **before** staging/committing any new work (step 7 hasn't run yet):
|
||||
- If the working tree has uncommitted changes, stash them first — `git stash push -u -m ship-sync-fix` — so the rebase below isn't blocked by dirty state. Restore with `git stash pop` once the branch is fixed.
|
||||
- Try `git rebase origin/staging` first.
|
||||
- **A rebase finishing without conflicts does NOT by itself mean the branch is clean** — it can replay stray commits onto the new base with no conflict at all. After the rebase (clean or not), re-run `git log --oneline origin/staging..HEAD` and re-check the commit list against what you recognize.
|
||||
- If the rebase conflicted on commits you don't recognize, OR it finished cleanly but the re-checked log still shows commits you don't recognize, abandon that result (`git rebase --abort` if still mid-rebase) and rebuild instead, in this exact order:
|
||||
1. **While still on `<original-branch>`**, identify the SHA(s) to preserve — **not** the whole range. `git log --oneline --reverse origin/staging..<original-branch>` lists everything ahead of `origin/staging`, but in exactly this scenario that range also contains the unrecognized/stray commits you're trying to leave behind — blindly cherry-picking the full range recreates the same polluted branch. Read the list and write down only the SHA(s) you recognize as your own session's work (e.g. `abc1234 def5678`); do this *before* touching any temp branch, since once you check out `ship-sync-tmp` at `origin/staging` in step 4, `HEAD` no longer contains these commits and the same lookup at that point returns nothing.
|
||||
2. `git checkout <original-branch>` — harmless no-op if you're already there, but required if an earlier interrupted attempt left you sitting on `ship-sync-tmp`: git refuses to delete the branch you're currently on, so deleting it before switching away silently fails and blocks the rest of the rebuild.
|
||||
3. Delete any leftover from an earlier attempt: `git branch -D ship-sync-tmp 2>/dev/null || true` — always succeeds, including when there's nothing to delete (a first attempt), so it never blocks the rest of the rebuild on its own exit code.
|
||||
4. `git checkout -b ship-sync-tmp origin/staging`.
|
||||
5. `git cherry-pick` the SHAs captured in step 1, **in that oldest-first order** — cherry-picking more than one session commit out of order can fail or produce the wrong history. Resolve conflicts.
|
||||
6. `git branch -f <original-branch> HEAD`, `git checkout <original-branch>`, and delete `ship-sync-tmp` (`git branch -D ship-sync-tmp`).
|
||||
- Re-verify with `git log --oneline origin/staging..HEAD` — it must list only commits you recognize before you proceed to committing new work.
|
||||
3. **Generate a commit message** following this format: `type(scope): description`
|
||||
- Types: `fix`, `feat`, `improvement`, `chore`
|
||||
- Scope: short identifier (e.g., `undo-redo`, `api`, `ui`)
|
||||
- Keep it concise
|
||||
4. **Run the cleanup pass** — only if the diff modifies UI code (any `.tsx` file, or anything under `apps/sim/components/`, `apps/sim/hooks/`, or `apps/sim/stores/`): `/cleanup`
|
||||
- The six code-quality skills (effects, memo, callbacks, state, React Query, emcn) only apply to React code, so skip this step entirely when no UI was touched. When it runs, it applies fixes so they land in this commit.
|
||||
5. **Run migration safety** — only if the diff touches `packages/db/migrations/**` or `packages/db/schema.ts`:
|
||||
- Run `/db-migrate` to review the migration for zero-downtime safety (expand/contract phasing, backward-compatibility with the deployed app version).
|
||||
- `bun run check:migrations origin/staging` must pass (staging is the PR base). Do not silence a flagged statement with a `-- migration-safe:` annotation unless `/db-migrate` confirmed the old code no longer depends on it; otherwise split the destructive change into a later deploy.
|
||||
6. **Run pre-ship checks** from the repo root before staging. This has two phases: first **regenerate** every committed artifact so generated files never drift into a CI failure (this is what catches things like `agent-stream-docs` going stale after a `models.ts` edit), then run the **full audit suite** CI's `Lint and Test` job enforces. Both phases parallelize — but only across commands that write **disjoint** outputs — and a bare `wait` swallows child exit codes, so both phases below explicitly collect each job's status and abort ship if any failed.
|
||||
|
||||
**Phase A — regenerate the always-in-repo committed artifacts (parallel), then let step 7 stage whatever changed.** Regenerate only the generators whose inputs live entirely in this repo and that any ordinary code change can drift — `agent-stream-docs:generate` (derives from the provider model registry) and `skills:sync` (derives from `.agents/skills/**`). They write disjoint trees (`apps/docs/…/agent.mdx` vs the `.claude`/`.cursor` command projections), so they parallelize safely, and each is idempotent (a no-op when already in sync):
|
||||
```bash
|
||||
rm -f /tmp/ship-gen-results
|
||||
for g in agent-stream-docs:generate skills:sync; do
|
||||
( bun run "$g" >"/tmp/ship-gen-${g//:/-}.log" 2>&1; echo "$? $g" >>/tmp/ship-gen-results ) &
|
||||
done
|
||||
wait
|
||||
# any non-zero line is a FAILED generator — read /tmp/ship-gen-<name>.log and fix before shipping;
|
||||
# a silently-failed generate leaves a stale artifact that Phase B / CI then rejects.
|
||||
# The `exit 1` makes this block itself exit non-zero on failure, so anything gating on the
|
||||
# command's status (an agent, or a wrapping script) actually stops — do NOT collapse it to
|
||||
# `grep … && echo ❌ || echo ✅`, which always exits 0 and silently lets ship continue.
|
||||
if grep -vE '^0 ' /tmp/ship-gen-results; then echo "❌ generator(s) failed — do not ship"; exit 1; fi
|
||||
echo "✅ artifacts regenerated"
|
||||
```
|
||||
Then `git status --short` to see what regenerated — those files must be staged in step 7 alongside your own changes.
|
||||
|
||||
**Do NOT blanket-run the domain generators here.** `mship:generate` (`generate-mship-contracts.ts`) is an **umbrella** that drives all nine mothership contract generators (`mship-contracts`, `billing-protocol-contract`, `mship-tools`, the four `trace-*`, `metrics-contract`, `vfs-snapshot-contract`) and biome-formats `apps/sim/lib/copilot/generated/` — never run it *and* its constituents (they write the same files and corrupt each other in parallel), and never run it on an ordinary ship: it reads an **external** copilot-contract source that isn't checked out in most worktrees, so it hard-fails with `ENOENT` and would abort ship for an unrelated reason. `generate:pi-model-catalog` (under `apps/sim`) likewise regenerates from the installed Pi package, not repo source. `scripts/generate-docs.ts` rewrites the integration docs and client-safe catalog; run it when this PR changes their block/icon/landing-content inputs or when `integration-catalog:check` reports drift, then review its broad generated diff. Only when **this PR's diff actually touches** a domain generator's input do you regenerate it deliberately and run its matching `:check` (`bun run mship:check` / the individual `*:check`) — with the external source present.
|
||||
|
||||
**Phase B — run lint + every audit CI enforces, in parallel, and abort ship if any fails.** Before running the commands, compare this list with `.github/workflows/test-build.yml`; when CI adds an audit, run it and update this skill instead of trusting a stale snapshot. The env-flag audit is currently an inline workflow block rather than a package script: when `apps/sim/lib/core/config/env-flags.ts` changed, run that current workflow block verbatim instead of copying a second version into this skill. Run `bun run lint` first (it autofixes formatting and mutates files, so don't parallelize it with the read-only audits), then run the base-sensitive block-registry check, then fan the independent audits out and collect exit codes:
|
||||
```bash
|
||||
# autofix formatting first (mutating; not parallel-safe with the audits). Gate its exit too —
|
||||
# a non-zero lint (unfixable errors) must abort before the audits run, not be ignored.
|
||||
bun run lint || { echo "❌ lint failed — do not ship"; exit 1; }
|
||||
bun run apps/sim/scripts/check-block-registry.ts origin/staging || {
|
||||
echo "❌ block registry audit failed — do not ship"
|
||||
exit 1
|
||||
}
|
||||
# Runs every audit CI runs, concurrently, and replays the output of any that fail.
|
||||
# Do not hand-list the audits here: the list is derived in scripts/run-audits.ts, and the
|
||||
# copy that used to live in this file had already drifted five audits behind package.json.
|
||||
bun run check:audits || { echo "❌ audit(s) failed — do not ship"; exit 1; }
|
||||
```
|
||||
If Phase A regenerated a file, its matching `:check` in Phase B now passes trivially — that parity is the point. Do not ship with any generator or audit failing; fix the cause (never silence it) and re-run. `check:migrations` and `type-check` are covered by steps 5 and CI respectively and are not repeated here.
|
||||
7. **Stage and commit** the changes with the generated message — including any files Phase A regenerated in step 6
|
||||
8. **Push to origin** using the current branch name — `--force-with-lease` if step 2's sync
|
||||
check did any history rewrite (a clean rebase or a cherry-pick rebuild) on a branch that had
|
||||
already been pushed once; a plain push would be rejected in exactly the polluted-remote case
|
||||
step 2 exists to fix
|
||||
9. **Create a PR** to staging with a description in the user's voice, then do a final content check — not a count check — comparing what actually landed:
|
||||
```bash
|
||||
git fetch origin staging && git log --oneline --reverse origin/staging..HEAD
|
||||
gh pr view <n> --json commits -q '.commits[].messageHeadline'
|
||||
```
|
||||
Re-fetch first — comparing against a stale local `origin/staging` ref can mask real drift or
|
||||
flag a false mismatch even when the branch and push are correct. `--reverse` makes the git log
|
||||
oldest-first, matching the PR commit list's order — plain `git log` is newest-first, and a
|
||||
positional/line-by-line comparison against the PR's oldest-first list can spuriously fail on
|
||||
any multi-commit branch. These two lists must describe the same commits in the same order
|
||||
(same subjects, the last one being the commit from step 7). If they don't match, the branch
|
||||
still has a problem — redo step 2's fix and `git push --force-with-lease`.
|
||||
|
||||
## Commit Message Format
|
||||
|
||||
Based on the repo's commit history:
|
||||
|
||||
```
|
||||
fix(scope): description for bug fixes
|
||||
feat(scope): description for new features
|
||||
improvement(scope): description for enhancements
|
||||
chore(scope): description for maintenance
|
||||
```
|
||||
|
||||
## PR Description Format
|
||||
|
||||
Use this exact template in the user's voice (concise, bullet points):
|
||||
|
||||
```markdown
|
||||
## Summary
|
||||
- bullet point describing what changed
|
||||
- another bullet point if needed
|
||||
|
||||
## Type of Change
|
||||
- [x] Bug fix (or appropriate type)
|
||||
|
||||
## Testing
|
||||
Tested manually (or describe testing)
|
||||
|
||||
## Checklist
|
||||
- [x] Code follows project style guidelines
|
||||
- [x] Self-reviewed my changes
|
||||
- [ ] Tests added/updated and passing
|
||||
- [x] No new warnings introduced
|
||||
- [x] I confirm that I have read and agree to the terms outlined in the [Contributor License Agreement (CLA)](./CONTRIBUTING.md#contributor-license-agreement-cla)
|
||||
```
|
||||
|
||||
## PR Creation Command
|
||||
|
||||
Use this command structure:
|
||||
|
||||
```bash
|
||||
gh pr create --base staging --title "COMMIT_MESSAGE" --body "PR_BODY"
|
||||
```
|
||||
|
||||
## Important Notes
|
||||
|
||||
- Always confirm the commit message and PR description with the user before executing
|
||||
- The PR should be created against `staging` branch
|
||||
- Keep descriptions concise and in active voice
|
||||
- Match the user's previous PR style: direct, no fluff, bullet points
|
||||
- **DO NOT add "Co-Authored-By" lines to commits** - keep commit messages clean
|
||||
|
||||
## User's Voice Characteristics (based on previous PRs)
|
||||
|
||||
- Short, direct bullet points
|
||||
- No unnecessary explanation
|
||||
- "Tested manually" is acceptable for testing section; include lint, boundary validation, and (when migrations changed) `check:migrations` results when run
|
||||
- Checkboxes filled in appropriately
|
||||
- No screenshots section unless UI changes
|
||||
@@ -0,0 +1,94 @@
|
||||
---
|
||||
name: tool-registry-boundary
|
||||
description: Keep the executable tool registry out of client-reachable module graphs — when to read `@/tools/metadata` instead of `getTool`, how to measure whether an import edge pulls the registry, and how to regenerate the metadata artifacts. Use when touching `apps/sim/tools/registry.ts`, `tools/utils.ts`, `tools/params.ts`, or anything that calls `getTool`.
|
||||
---
|
||||
|
||||
# Tool Registry Boundary Skill
|
||||
|
||||
You keep the 4,300-tool executable registry out of module graphs that don't execute tools.
|
||||
|
||||
## The rule
|
||||
|
||||
> Client-reachable code reads tool **metadata**. Only code that actually executes a tool imports the **registry**.
|
||||
|
||||
`@/tools/registry` is a ~9,000-line barrel importing every tool. Each `ToolConfig` mixes plain data (`params`, `outputs`, `name`) with closures — `request.url`, `request.headers`, `transformResponse`, `directExecution`, `postProcess`. Those closures reach the SDK clients, API helpers and parsers each integration needs, and that is what makes the barrel expensive: reaching it costs ~4,700 additional modules.
|
||||
|
||||
`getTool()` returns the whole `ToolConfig`, so a single `getTool` import anywhere in a client-reachable file drags all of it in.
|
||||
|
||||
## Which module to import
|
||||
|
||||
| you need | import | notes |
|
||||
| --- | --- | --- |
|
||||
| whether a tool id exists | `hasToolId` from `@/tools/tool-ids` | ~110 KB — the cheapest module |
|
||||
| to resolve an unversioned name | `resolveToolId` from `@/tools/tool-ids` | |
|
||||
| every tool id | `getToolIds` from `@/tools/tool-ids` | |
|
||||
| a tool's params | `getToolParams` / `getToolMetadata` from `@/tools/metadata` | ~4 MB |
|
||||
| a tool's declared outputs | `getToolOutputsMetadata` from `@/tools/metadata-outputs` | ~4 MB, separate on purpose |
|
||||
| to **execute** a tool | `getTool` from `@/tools/utils`, or `@/tools/utils.server` | server paths only |
|
||||
|
||||
Three modules, cheapest first. Ids are their own artifact because resolution and existence checks need only the key set; outputs are their own because they are the larger half of the data with a single consumer. `@/tools/metadata` and `@/tools/metadata-outputs` both resolve ids through `@/tools/tool-ids`, which is what keeps them independent of each other — do not "helpfully" re-export one from another, or every caller pays for all three.
|
||||
|
||||
All lookups guard with `Object.hasOwn`. `JSON.parse` yields an object with the normal prototype, so a bare bracket lookup returns inherited members: `getToolMetadata('constructor')` returned a *function* typed as tool metadata before that was fixed.
|
||||
|
||||
## The generated artifacts
|
||||
|
||||
`apps/sim/tools/generated/tool-ids.ts`, `tool-metadata.ts` and `tool-outputs.ts` are produced by `scripts/sync-tool-metadata.ts`:
|
||||
|
||||
```bash
|
||||
bun run tool-metadata:generate # after adding/changing a tool
|
||||
bun run tool-metadata:check # what CI runs; fails if stale
|
||||
```
|
||||
|
||||
Never hand-edit them. If you add a tool or change a tool's `params`/`outputs`, regenerate and commit the result, or CI fails.
|
||||
|
||||
Three non-obvious properties, each of which was measured and is easy to undo by accident:
|
||||
|
||||
- **The data is a JSON string parsed at runtime, not an imported `.json` and not an object literal.** With `resolveJsonModule` (which this repo enables), a `.json` import makes TypeScript infer a literal type for all 4,300+ entries and takes `tsc --noEmit` from **12.6s to 8m07s** — a 38x regression. An ambient `declare module` does *not* short-circuit it, and an object literal costs the same. A single string literal is one cheap token for both the compiler and the bundler, and `JSON.parse` beats evaluating the equivalent literal at runtime. Do not "clean this up" into a `.json` import.
|
||||
- **The generator refuses to emit function values.** If you add a field to `METADATA_FIELDS` that contains a closure, generation fails loudly rather than shipping executable config to the client. `hosting` and `schemaEnrichment` are excluded for exactly this reason (`hosting.enabled`, `pricing`, and `enrichSchema` are functions) — they are server-only.
|
||||
- **Empty param entries are stripped.** The registry contains one (`stt_deepgram_v2`), which crashes callers that read `param.type` while iterating.
|
||||
- **Lookups resolve versions.** `getTool` maps an unversioned name onto the newest version, and 246 tools are versioned. A plain key lookup would silently report them missing — a quiet correctness bug, not a crash. `resolveToolId` reproduces that against the id set and is differentially tested against the original.
|
||||
|
||||
## Testing code that reads tool metadata
|
||||
|
||||
Mock the module the code under test actually reads. `vi.mock('@/tools/utils', () => toolsUtilsMock)` only controls `getTool`; code that reads `params`/`outputs`/`name` goes through `@/tools/metadata`, so mocking `tools/utils` there is a **no-op that still passes** — because the real generated artifacts happen to agree with the mock fixtures. The test looks green while controlling nothing.
|
||||
|
||||
```ts
|
||||
import { blocksMock, toolsMetadataMock, toolsUtilsMock } from '@sim/testing/mocks'
|
||||
|
||||
vi.mock('@/tools/utils', () => toolsUtilsMock) // executable lookup
|
||||
vi.mock('@/tools/metadata', () => toolsMetadataMock) // params / outputs / name
|
||||
```
|
||||
|
||||
Both are backed by the same `mockToolConfigs`, so mocking both gives one consistent tool universe. If you are unsure whether a mock is load-bearing, change a fixture value to a sentinel and confirm the test fails.
|
||||
## The guard
|
||||
|
||||
`bun run check:tool-registry-boundary` (CI: "Tool registry client-boundary audit") walks the module graph from each workspace route and fails if `@/tools/registry` is reachable, printing the exact import chain that reintroduced it.
|
||||
|
||||
If it fails, do not add the entry to an allowlist — there isn't one. Find the symbol the offending file actually needs and move it to a registry-free module, exactly as `mergeToolParameters` and `formatParameterLabel` were.
|
||||
|
||||
Run it with `--verbose` to print per-route module counts, which is also the quickest way to see whether a change moved the graph.
|
||||
|
||||
## How to verify an edge actually got cut
|
||||
|
||||
Do not eyeball imports — the registry is reached through several redundant paths, so cutting one buys nothing while another survives. Walk the graph:
|
||||
|
||||
1. From the entry you care about, follow `import` and `export … from` (skipping `import type`), resolving `@/` against `apps/sim`.
|
||||
2. Check whether `apps/sim/tools/registry.ts` is in the reachable set, and print the parent chain if it is.
|
||||
3. Compare the reachable module count before and after.
|
||||
|
||||
Reference points measured on this repo:
|
||||
|
||||
| entry | modules |
|
||||
| --- | --- |
|
||||
| `tools/registry.ts` reachable | ~4,900 |
|
||||
| `tools/merge-params.ts` (leaf) | 2 |
|
||||
| `providers/utils.ts` after cutting its `params` edge | 22 |
|
||||
| `app/workspace/[workspaceId]/w/page.tsx` (canvas) | 6,592 before, 1,908 after |
|
||||
|
||||
The canvas route reached the registry through **four** redundant edges — `providers/utils` (via `tools/params`), `lib/workflows/blocks/block-outputs`, `lib/workflows/sanitization/validation`, and `serializer/index`. Cutting any one alone moved the module count by ~1. They all had to go before anything improved; measure the route, not the file you edited.
|
||||
|
||||
## When adding a new caller
|
||||
|
||||
Ask what the caller does with the config. If it reads `params`, `outputs`, `name`, `description` or just checks existence, it belongs on `@/tools/metadata` — no exceptions, even on a path you believe is server-only today, because a future client import will silently re-attach the registry to the graph.
|
||||
|
||||
If it genuinely executes — builds a request, transforms a response, runs `directExecution` — use `getTool`, and keep that file off client-reachable paths.
|
||||
@@ -0,0 +1,372 @@
|
||||
---
|
||||
name: validate-connector
|
||||
description: Validate an existing knowledge base connector against its service's API docs
|
||||
argument-hint: <service-name> [api-docs-url]
|
||||
---
|
||||
|
||||
# Validate Connector Skill
|
||||
|
||||
You are an expert auditor for Sim knowledge base connectors. Your job is to thoroughly validate that an existing connector is correct, complete, and follows all conventions.
|
||||
|
||||
## Your Task
|
||||
|
||||
When the user asks you to validate a connector:
|
||||
1. Read the service's API documentation (via Context7 or WebFetch)
|
||||
2. Read the connector implementation, OAuth config, and registry entries
|
||||
3. Cross-reference everything against the API docs and Sim conventions
|
||||
4. Report all issues found, grouped by severity (critical, warning, suggestion)
|
||||
5. Fix all issues after reporting them
|
||||
|
||||
## Step 1: Gather All Files
|
||||
|
||||
Read **every** file for the connector — do not skip any:
|
||||
|
||||
```
|
||||
apps/sim/connectors/{service}/meta.ts # ConnectorMeta — client-safe metadata (icon, name, auth, configFields, tagDefinitions)
|
||||
apps/sim/connectors/{service}/{service}.ts # Connector implementation — spreads the meta + runtime functions
|
||||
apps/sim/connectors/{service}/index.ts # Barrel export
|
||||
apps/sim/connectors/registry.server.ts # Server-only full registry entry (CONNECTOR_REGISTRY; full connector)
|
||||
apps/sim/connectors/registry.ts # Client-safe meta registry entry (CONNECTOR_META_REGISTRY)
|
||||
apps/sim/connectors/types.ts # ConnectorMeta / ConnectorConfig interfaces, ExternalDocument, etc.
|
||||
apps/sim/connectors/utils.ts # Shared utilities (computeContentHash, htmlToPlainText, etc.)
|
||||
apps/sim/lib/oauth/oauth.ts # OAUTH_PROVIDERS — single source of truth for scopes
|
||||
apps/sim/lib/oauth/utils.ts # getCanonicalScopesForProvider, getScopesForService, SCOPE_DESCRIPTIONS
|
||||
apps/sim/lib/oauth/types.ts # OAuthService union type
|
||||
apps/sim/components/icons.tsx # Icon definition for the service
|
||||
```
|
||||
|
||||
If the connector uses selectors, also read:
|
||||
```
|
||||
apps/sim/hooks/selectors/registry.ts # Selector key definitions
|
||||
apps/sim/hooks/selectors/types.ts # SelectorKey union type
|
||||
apps/sim/lib/workflows/subblocks/context.ts # SELECTOR_CONTEXT_FIELDS
|
||||
```
|
||||
|
||||
## Step 2: Pull API Documentation
|
||||
|
||||
Fetch the official API docs for the service. This is the **source of truth** for:
|
||||
- Endpoint URLs, HTTP methods, and auth headers
|
||||
- Required vs optional parameters
|
||||
- Parameter types and allowed values
|
||||
- Response shapes and field names
|
||||
- Pagination patterns (cursor, offset, next token)
|
||||
- Rate limits and error formats
|
||||
- OAuth scopes and their meanings
|
||||
|
||||
Use Context7 (resolve-library-id → query-docs) or WebFetch to retrieve documentation. If both fail, note which claims are based on training knowledge vs verified docs.
|
||||
|
||||
### Hard Rule: No Guessed Source Schemas
|
||||
|
||||
If the service docs do not clearly show document list responses, document fetch responses, metadata fields, or pagination shapes, you MUST tell the user instead of guessing.
|
||||
|
||||
- Do NOT infer document fields from unrelated endpoints
|
||||
- Do NOT guess pagination cursors or response wrappers
|
||||
- Do NOT assume metadata keys that are not documented
|
||||
- Do NOT treat probable shapes as validated
|
||||
|
||||
If a schema is unknown, validation must explicitly recommend:
|
||||
1. sample API responses,
|
||||
2. live test credentials, or
|
||||
3. trimming the connector to only documented fields.
|
||||
|
||||
## Step 3: Validate API Endpoints
|
||||
|
||||
For **every** API call in the connector (`listDocuments`, `getDocument`, `validateConfig`, and any helper functions), verify against the API docs:
|
||||
|
||||
### URLs and Methods
|
||||
- [ ] Base URL is correct for the service's API version
|
||||
- [ ] Endpoint paths match the API docs exactly
|
||||
- [ ] HTTP method is correct (GET, POST, PUT, PATCH, DELETE)
|
||||
- [ ] Path parameters are correctly interpolated and URI-encoded where needed
|
||||
- [ ] Query parameters use correct names and formats per the API docs
|
||||
|
||||
### Headers
|
||||
- [ ] Authorization header uses the correct format:
|
||||
- OAuth: `Authorization: Bearer ${accessToken}`
|
||||
- API Key: correct header name per the service's docs
|
||||
- [ ] `Content-Type` is set for POST/PUT/PATCH requests
|
||||
- [ ] Any service-specific headers are present (e.g., `Notion-Version`, `Dropbox-API-Arg`)
|
||||
- [ ] No headers are sent that the API doesn't support or silently ignores
|
||||
|
||||
### Request Bodies
|
||||
- [ ] POST/PUT body fields match API parameter names exactly
|
||||
- [ ] Required fields are always sent
|
||||
- [ ] Optional fields are conditionally included (not sent as `null` or empty unless the API expects that)
|
||||
- [ ] Field value types match API expectations (string vs number vs boolean)
|
||||
|
||||
### Input Sanitization
|
||||
- [ ] User-controlled values interpolated into query strings are properly escaped:
|
||||
- OData `$filter`: single quotes escaped with `''` (e.g., `externalId.replace(/'/g, "''")`)
|
||||
- SOQL: single quotes escaped with `\'`
|
||||
- GraphQL variables: passed as variables, not interpolated into query strings
|
||||
- URL path segments: `encodeURIComponent()` applied
|
||||
- [ ] URL-type config fields (e.g., `siteUrl`, `instanceUrl`) are normalized:
|
||||
- Strip `https://` / `http://` prefix if the API expects bare domains
|
||||
- Strip trailing `/`
|
||||
- Apply `.trim()` before validation
|
||||
|
||||
### Response Parsing
|
||||
- [ ] Response structure is correctly traversed (e.g., `data.results` vs `data.items` vs `data`)
|
||||
- [ ] Field names extracted match what the API actually returns
|
||||
- [ ] Nullable fields are handled with `?? null` or `|| undefined`
|
||||
- [ ] Error responses are checked before accessing data fields
|
||||
- [ ] Every extracted field and pagination value is backed by official docs or live-verified sample payloads
|
||||
|
||||
## Step 4: Validate OAuth Scopes (if OAuth connector)
|
||||
|
||||
Scopes must be correctly declared and sufficient for all API calls the connector makes.
|
||||
|
||||
### Connector requiredScopes
|
||||
- [ ] `requiredScopes` in the connector's `auth` config lists all scopes needed by the connector
|
||||
- [ ] Each scope in `requiredScopes` is a real, valid scope recognized by the service's API
|
||||
- [ ] No invalid, deprecated, or made-up scopes are listed
|
||||
- [ ] No unnecessary excess scopes beyond what the connector actually needs
|
||||
|
||||
### Scope Subset Validation (CRITICAL)
|
||||
- [ ] Every scope in `requiredScopes` exists in the OAuth provider's `scopes` array in `lib/oauth/oauth.ts`
|
||||
- [ ] Find the provider in `OAUTH_PROVIDERS[providerGroup].services[serviceId].scopes`
|
||||
- [ ] Verify: `requiredScopes` ⊆ `OAUTH_PROVIDERS scopes` (every required scope is present in the provider config)
|
||||
- [ ] If a required scope is NOT in the provider config, flag as **critical** — the connector will fail at runtime
|
||||
|
||||
### Scope Sufficiency
|
||||
For each API endpoint the connector calls:
|
||||
- [ ] Identify which scopes are required per the API docs
|
||||
- [ ] Verify those scopes are included in the connector's `requiredScopes`
|
||||
- [ ] If the connector calls endpoints requiring scopes not in `requiredScopes`, flag as **warning**
|
||||
|
||||
### Token Refresh Config
|
||||
- [ ] Check the `getOAuthTokenRefreshConfig` function in `lib/oauth/oauth.ts` for this provider
|
||||
- [ ] `useBasicAuth` matches the service's token exchange requirements
|
||||
- [ ] `supportsRefreshTokenRotation` matches whether the service issues rotating refresh tokens
|
||||
- [ ] Token endpoint URL is correct
|
||||
|
||||
## Step 5: Validate Pagination
|
||||
|
||||
### listDocuments Pagination
|
||||
- [ ] Cursor/pagination parameter name matches the API docs
|
||||
- [ ] Response pagination field is correctly extracted (e.g., `next_cursor`, `nextPageToken`, `@odata.nextLink`, `offset`)
|
||||
- [ ] `hasMore` is correctly determined from the response
|
||||
- [ ] `nextCursor` is correctly passed back for the next page
|
||||
- [ ] `maxItems` / `maxRecords` cap is correctly applied across pages using `syncContext.totalDocsFetched`
|
||||
- [ ] Page size is within the API's allowed range (not exceeding max page size)
|
||||
- [ ] Last page precision: when a `maxItems` cap exists, the final page request uses `Math.min(PAGE_SIZE, remaining)` to avoid fetching more records than needed
|
||||
- [ ] No off-by-one errors in pagination tracking
|
||||
- [ ] The connector does NOT hit known API pagination limits silently (e.g., HubSpot search 10k cap)
|
||||
|
||||
### Deletion-Reconciliation Safety (`listingCapped`) — CRITICAL
|
||||
The sync engine hard-deletes any stored document absent from a full listing. Audit every path where `listDocuments` can return less than the full source set:
|
||||
- [ ] `syncContext.listingCapped = true` is set when a `maxItems`-style cap truncates the listing while more documents exist
|
||||
- [ ] `listingCapped` is set when a transient per-item error drops a still-existing document from the listing
|
||||
- [ ] `listingCapped` is NOT set when the source is genuinely exhausted (deleted documents must reconcile) or for intentional scope filters (date cutoffs)
|
||||
This is the most common connector bug class — verify it explicitly against `sync-engine.ts`'s reconciliation gate.
|
||||
|
||||
### Pagination State Across Pages
|
||||
- [ ] `syncContext` is used to cache state across pages (user names, field maps, instance URLs, portal IDs, etc.)
|
||||
- [ ] Cached state in `syncContext` is correctly initialized on first page and reused on subsequent pages
|
||||
|
||||
## Step 6: Validate Data Transformation
|
||||
|
||||
### Content Deferral (CRITICAL)
|
||||
Connectors that require per-document API calls to fetch content (file download, export, blocks fetch) MUST use `contentDeferred: true`. This is the standard pattern for reliability — without it, content downloads during listing can exhaust the sync task's time budget before any documents are saved.
|
||||
|
||||
- [ ] If the connector downloads content per-doc during `listDocuments`, it MUST use `contentDeferred: true` instead
|
||||
- [ ] `listDocuments` returns lightweight stubs with `content: ''` and `contentDeferred: true`
|
||||
- [ ] `getDocument` fetches actual content and returns the full document with `contentDeferred: false`
|
||||
- [ ] A shared stub function (e.g., `fileToStub`) is used by both `listDocuments` and `getDocument` to guarantee `contentHash` consistency
|
||||
- [ ] `contentHash` is metadata-based (e.g., `service:{id}:{modifiedTime}`), NOT content-based — it must be derivable from list metadata alone
|
||||
- [ ] The `contentHash` is identical whether produced by `listDocuments` or `getDocument`
|
||||
|
||||
Connectors where the list API already returns content inline (e.g., Slack messages, Reddit posts) do NOT need `contentDeferred`.
|
||||
|
||||
### ExternalDocument Construction
|
||||
- [ ] `externalId` is a stable, unique identifier from the source API
|
||||
- [ ] `title` is extracted from the correct field and has a sensible fallback (e.g., `'Untitled'`)
|
||||
- [ ] `content` is plain text — HTML content is stripped using `htmlToPlainText` from `@/connectors/utils`
|
||||
- [ ] `mimeType` is `'text/plain'`
|
||||
- [ ] `contentHash` uses a metadata-based format (e.g., `service:{id}:{modifiedTime}`) for connectors with `contentDeferred: true`, or `computeContentHash` from `@/connectors/utils` for inline-content connectors
|
||||
- [ ] `sourceUrl` is a valid, complete URL back to the original resource (not relative)
|
||||
- [ ] `metadata` contains all fields referenced by `mapTags` and `tagDefinitions`
|
||||
|
||||
### Content Extraction
|
||||
- [ ] Rich text / HTML fields are converted to plain text before indexing
|
||||
- [ ] Important content is not silently dropped (e.g., nested blocks, table cells, code blocks)
|
||||
- [ ] Content is not silently truncated without logging a warning
|
||||
- [ ] Empty/blank documents are properly filtered out
|
||||
- [ ] Size checks use `Buffer.byteLength(text, 'utf8')` not `text.length` when comparing against byte-based limits (e.g., `MAX_FILE_SIZE` in bytes)
|
||||
|
||||
## Step 7: Validate Tag Definitions and mapTags
|
||||
|
||||
### tagDefinitions
|
||||
- [ ] Each `tagDefinition` has an `id`, `displayName`, and `fieldType`
|
||||
- [ ] `fieldType` matches the actual data type: `'text'` for strings, `'number'` for numbers, `'date'` for dates, `'boolean'` for booleans
|
||||
- [ ] Every `id` in `tagDefinitions` is returned by `mapTags`
|
||||
- [ ] No `tagDefinition` references a field that `mapTags` never produces
|
||||
|
||||
### mapTags
|
||||
- [ ] Return keys match `tagDefinition` `id` values exactly
|
||||
- [ ] Date values are properly parsed using `parseTagDate` from `@/connectors/utils`
|
||||
- [ ] Array values are properly joined using `joinTagArray` from `@/connectors/utils`
|
||||
- [ ] Number values are validated (not `NaN`)
|
||||
- [ ] Metadata field names accessed in `mapTags` match what `listDocuments`/`getDocument` store in `metadata`
|
||||
|
||||
## Step 8: Validate Config Fields and Validation
|
||||
|
||||
### configFields
|
||||
- [ ] Every field has `id`, `title`, `type`
|
||||
- [ ] `required` is set explicitly (not omitted)
|
||||
- [ ] Dropdown fields have `options` with `label` and `id` for each option
|
||||
- [ ] Selector fields follow the canonical pair pattern:
|
||||
- A `type: 'selector'` field with `selectorKey`, `canonicalParamId`, `mode: 'basic'`
|
||||
- A `type: 'short-input'` field with the same `canonicalParamId`, `mode: 'advanced'`
|
||||
- `required` is identical on both fields in the pair
|
||||
- [ ] `selectorKey` values exist in the selector registry
|
||||
- [ ] `dependsOn` references selector field `id` values, not `canonicalParamId`
|
||||
|
||||
### validateConfig
|
||||
- [ ] Validates all required fields are present before making API calls
|
||||
- [ ] Validates optional numeric fields (checks `Number.isNaN`, positive values)
|
||||
- [ ] Makes a lightweight API call to verify access (e.g., fetch 1 record, get profile)
|
||||
- [ ] Uses `VALIDATE_RETRY_OPTIONS` for retry budget
|
||||
- [ ] Returns `{ valid: true }` on success
|
||||
- [ ] Returns `{ valid: false, error: 'descriptive message' }` on failure
|
||||
- [ ] Catches exceptions and returns user-friendly error messages
|
||||
- [ ] Does NOT make expensive calls (full data listing, large queries)
|
||||
|
||||
## Step 9: Validate getDocument
|
||||
|
||||
- [ ] Fetches a single document by `externalId`
|
||||
- [ ] Returns `null` for 404 / not found (does not throw)
|
||||
- [ ] Returns the same `ExternalDocument` shape as `listDocuments`
|
||||
- [ ] If `listDocuments` uses `contentDeferred: true`, `getDocument` MUST fetch actual content and return `contentDeferred: false`
|
||||
- [ ] If `listDocuments` uses `contentDeferred: true`, `getDocument` MUST use the same stub function to ensure `contentHash` is identical
|
||||
- [ ] Handles all content types that `listDocuments` can produce (e.g., if `listDocuments` returns both pages and blogposts, `getDocument` must handle both — not hardcode one endpoint)
|
||||
- [ ] Forwards `syncContext` if it needs cached state (user names, field maps, etc.)
|
||||
- [ ] Error handling is graceful (catches, logs, returns null or throws with context)
|
||||
- [ ] Does not redundantly re-fetch data already included in the initial API response (e.g., if comments come back with the post, don't fetch them again separately)
|
||||
|
||||
## Step 10: Validate General Quality
|
||||
|
||||
### fetchWithRetry Usage
|
||||
- [ ] All external API calls use `fetchWithRetry` from `@/lib/knowledge/documents/utils`
|
||||
- [ ] No raw `fetch()` calls to external APIs
|
||||
- [ ] `VALIDATE_RETRY_OPTIONS` used in `validateConfig`
|
||||
- [ ] If `validateConfig` calls a shared helper (e.g., `linearGraphQL`, `resolveId`), that helper must accept and forward `retryOptions` to `fetchWithRetry`
|
||||
- [ ] Default retry options used in `listDocuments`/`getDocument`
|
||||
|
||||
### API Efficiency
|
||||
- [ ] APIs that support field selection (e.g., `$select`, `sysparm_fields`, `fields`) should request only the fields the connector needs — in both `listDocuments` AND `getDocument`
|
||||
- [ ] No redundant API calls: if a helper already fetches data (e.g., site metadata), callers should reuse the result instead of making a second call for the same information
|
||||
- [ ] Sequential per-item API calls (fetching details for each document in a loop) should be batched with `Promise.all` and a concurrency limit of 3-5
|
||||
|
||||
### Error Handling
|
||||
- [ ] Individual document failures are caught and logged without aborting the sync
|
||||
- [ ] API error responses include status codes in error messages
|
||||
- [ ] No unhandled promise rejections in concurrent operations
|
||||
|
||||
### Concurrency
|
||||
- [ ] Concurrent API calls use reasonable batch sizes (3-5 is typical)
|
||||
- [ ] No unbounded `Promise.all` over large arrays
|
||||
|
||||
### Logging
|
||||
- [ ] Uses `createLogger` from `@sim/logger` (not `console.log`)
|
||||
- [ ] Logs sync progress at `info` level
|
||||
- [ ] Logs errors at `warn` or `error` level with context
|
||||
|
||||
### Meta / Runtime Split
|
||||
- [ ] `connectors/{service}/meta.ts` exports `{service}ConnectorMeta: ConnectorMeta` (id, name, description, version, icon, auth, configFields, and any `tagDefinitions` / `supportsIncrementalSync`)
|
||||
- [ ] `meta.ts` imports ONLY the icon from `@/components/icons`, `ConnectorMeta` (type-only), and pure-data constants — NO server/runtime imports (`@/lib/knowledge/...`, `input-validation.server`, `fetchWithRetry`, etc.); any such import in `meta.ts` is **critical** (breaks the client bundle)
|
||||
- [ ] `connectors/{service}/{service}.ts` spreads `...{service}ConnectorMeta` as the first property and adds the runtime functions (`listDocuments`, `getDocument`, `validateConfig`, `mapTags?`)
|
||||
- [ ] Metadata fields (id, name, auth, configFields, etc.) live ONLY in `meta.ts`, not duplicated in `{service}.ts`
|
||||
|
||||
### Registry
|
||||
- [ ] Connector is exported from `connectors/{service}/index.ts`
|
||||
- [ ] Full connector is registered in `connectors/registry.server.ts` (server-only registry, `CONNECTOR_REGISTRY`)
|
||||
- [ ] Meta is registered in `connectors/registry.ts` (client-safe registry, `CONNECTOR_META_REGISTRY`), importing `@/connectors/{service}/meta`
|
||||
- [ ] Both registries use the same key and it matches the connector's `id` field
|
||||
- [ ] Both registries keep the same alphabetical-by-id ordering
|
||||
|
||||
## Step 11: Report and Fix
|
||||
|
||||
### Report Format
|
||||
|
||||
Group findings by severity:
|
||||
|
||||
**Critical** (will cause runtime errors, data loss, or auth failures):
|
||||
- Wrong API endpoint URL or HTTP method
|
||||
- Invalid or missing OAuth scopes (not in provider config)
|
||||
- Incorrect response field mapping (accessing wrong path)
|
||||
- SOQL/query fields that don't exist on the target object
|
||||
- Pagination that silently hits undocumented API limits
|
||||
- Missing `syncContext.listingCapped = true` when a cap or transient error truncates the listing — the sync engine hard-deletes the documents absent from the partial listing
|
||||
- Missing error handling that would crash the sync
|
||||
- `requiredScopes` not a subset of OAuth provider scopes
|
||||
- Query/filter injection: user-controlled values interpolated into OData `$filter`, SOQL, or query strings without escaping
|
||||
- Per-document content download in `listDocuments` without `contentDeferred: true` — causes sync timeouts for large document sets
|
||||
- `contentHash` mismatch between `listDocuments` stub and `getDocument` return — causes unnecessary re-processing every sync
|
||||
- Server/runtime import in `meta.ts` (e.g. `@/lib/knowledge/...`, `input-validation.server`, `fetchWithRetry`) — pulls server-only code into the client bundle and breaks the build
|
||||
- Connector missing from `connectors/registry.ts` (the client-safe meta registry) — or its entry there imports the runtime module instead of `meta.ts` — the knowledge UI can't render it
|
||||
|
||||
**Warning** (incorrect behavior, data quality issues, or convention violations):
|
||||
- HTML content not stripped via `htmlToPlainText`
|
||||
- `getDocument` not forwarding `syncContext`
|
||||
- `getDocument` hardcoded to one content type when `listDocuments` returns multiple (e.g., only pages but not blogposts)
|
||||
- Missing `tagDefinition` for metadata fields returned by `mapTags`
|
||||
- Incorrect `useBasicAuth` or `supportsRefreshTokenRotation` in token refresh config
|
||||
- Invalid scope names that the API doesn't recognize (even if silently ignored)
|
||||
- Private resources excluded from name-based lookup despite scopes being available
|
||||
- Silent data truncation without logging
|
||||
- Size checks using `text.length` (character count) instead of `Buffer.byteLength` (byte count) for byte-based limits
|
||||
- URL-type config fields not normalized (protocol prefix, trailing slashes cause API failures)
|
||||
- `VALIDATE_RETRY_OPTIONS` not threaded through helper functions called by `validateConfig`
|
||||
|
||||
**Suggestion** (minor improvements):
|
||||
- Missing incremental sync support despite API supporting it
|
||||
- Overly broad scopes that could be narrowed (not wrong, but could be tighter)
|
||||
- Source URL format could be more specific
|
||||
- Missing `orderBy` for deterministic pagination
|
||||
- Redundant API calls that could be cached in `syncContext`
|
||||
- Sequential per-item API calls that could be batched with `Promise.all` (concurrency 3-5)
|
||||
- API supports field selection but connector fetches all fields (e.g., missing `$select`, `sysparm_fields`, `fields`)
|
||||
- `getDocument` re-fetches data already included in the initial API response (e.g., comments returned with post)
|
||||
- Last page of pagination requests full `PAGE_SIZE` when fewer records remain (`Math.min(PAGE_SIZE, remaining)`)
|
||||
|
||||
### Fix All Issues
|
||||
|
||||
After reporting, fix every **critical** and **warning** issue. Apply **suggestions** where they don't add unnecessary complexity.
|
||||
|
||||
### Validation Output
|
||||
|
||||
After fixing, confirm:
|
||||
1. `bun run lint` passes
|
||||
2. TypeScript compiles clean
|
||||
3. Re-read all modified files to verify fixes are correct
|
||||
4. Any remaining unknown source schemas were explicitly reported to the user instead of guessed
|
||||
|
||||
## Checklist Summary
|
||||
|
||||
- [ ] Read connector meta.ts, implementation, types, utils, both registries, and OAuth config
|
||||
- [ ] Pulled and read official API documentation for the service
|
||||
- [ ] Validated every API endpoint URL, method, headers, and body against API docs
|
||||
- [ ] Validated input sanitization: no query/filter injection, URL fields normalized
|
||||
- [ ] Validated OAuth scopes: `requiredScopes` ⊆ OAuth provider `scopes` in `oauth.ts`
|
||||
- [ ] Validated each scope is real and recognized by the service's API
|
||||
- [ ] Validated scopes are sufficient for all API endpoints the connector calls
|
||||
- [ ] Validated token refresh config (`useBasicAuth`, `supportsRefreshTokenRotation`)
|
||||
- [ ] Validated pagination: cursor names, page sizes, hasMore logic, no silent caps
|
||||
- [ ] Validated deletion-reconciliation safety: `syncContext.listingCapped` set on capped/error-truncated listings, not on genuine exhaustion or intentional scope filters
|
||||
- [ ] Validated content deferral: `contentDeferred: true` used when per-doc content fetch required, metadata-based `contentHash` consistent between stub and `getDocument`
|
||||
- [ ] Validated data transformation: plain text extraction, HTML stripping, content hashing
|
||||
- [ ] Validated tag definitions match mapTags output, correct fieldTypes
|
||||
- [ ] Validated config fields: canonical pairs, selector keys, required flags
|
||||
- [ ] Validated validateConfig: lightweight check, error messages, retry options
|
||||
- [ ] Validated getDocument: null on 404, all content types handled, no redundant re-fetches, syncContext forwarding
|
||||
- [ ] Validated fetchWithRetry used for all external calls (no raw fetch), VALIDATE_RETRY_OPTIONS threaded through helpers
|
||||
- [ ] Validated API efficiency: field selection used, no redundant calls, sequential fetches batched
|
||||
- [ ] Validated error handling: graceful failures, no unhandled rejections
|
||||
- [ ] Validated logging: createLogger, no console.log
|
||||
- [ ] Validated meta/runtime split: `meta.ts` holds metadata with no server/runtime imports, `{service}.ts` spreads the meta + adds runtime functions
|
||||
- [ ] Validated registry: exported from index.ts, full connector in `registry.server.ts`, meta in `registry.ts`, matching keys and alphabetical-by-id ordering in both
|
||||
- [ ] Reported all issues grouped by severity
|
||||
- [ ] Fixed all critical and warning issues
|
||||
- [ ] Ran `bun run lint` after fixes
|
||||
- [ ] Verified TypeScript compiles clean
|
||||
@@ -0,0 +1,5 @@
|
||||
interface:
|
||||
display_name: "Validate Connector"
|
||||
short_description: "Audit a Sim knowledge connector"
|
||||
brand_color: "#059669"
|
||||
default_prompt: "Use $validate-connector to audit and fix a Sim knowledge connector against its API docs."
|
||||
@@ -0,0 +1,445 @@
|
||||
---
|
||||
name: validate-integration
|
||||
description: Validate an existing Sim integration (tools, block, registry, and resolved-secret/model-input boundaries) against the service's API docs and Sim execution conventions
|
||||
argument-hint: <service-name> [api-docs-url]
|
||||
---
|
||||
|
||||
# Validate Integration Skill
|
||||
|
||||
You are an expert auditor for Sim integrations. Your job is to thoroughly validate that an existing integration is correct, complete, and follows all conventions.
|
||||
|
||||
## Your Task
|
||||
|
||||
When the user asks you to validate an integration:
|
||||
1. Read the service's API documentation (via WebFetch or Context7)
|
||||
2. Read every tool, the block, and registry entries
|
||||
3. Cross-reference everything against the API docs and Sim conventions
|
||||
4. Report all issues found, grouped by severity (critical, warning, suggestion)
|
||||
5. Fix all issues after reporting them
|
||||
|
||||
## Step 1: Gather All Files
|
||||
|
||||
Read **every** file for the integration — do not skip any:
|
||||
|
||||
```
|
||||
apps/sim/tools/{service}/ # All tool files, types.ts, index.ts
|
||||
apps/sim/blocks/blocks/{service}.ts # Block definition
|
||||
apps/sim/tools/registry.ts # Tool registry entries for this service
|
||||
apps/sim/blocks/registry-maps.ts # Block + meta registry entry (BLOCK_REGISTRY / BLOCK_META_REGISTRY)
|
||||
apps/sim/components/icons.tsx # Icon definition
|
||||
apps/sim/lib/auth/auth.ts # OAuth config — should use getCanonicalScopesForProvider()
|
||||
apps/sim/lib/oauth/oauth.ts # OAuth provider config — single source of truth for scopes
|
||||
apps/sim/lib/oauth/utils.ts # Scope utilities, SCOPE_DESCRIPTIONS for modal UI
|
||||
apps/sim/lib/core/config/env-capabilities.ts # OAuth client runtime capability source of truth
|
||||
apps/sim/lib/core/config/env.ts # Runtime env schema for capability fields
|
||||
scripts/setup/capability-config.ts # Exhaustive CLI input-mode mapping for OAuth fields
|
||||
apps/sim/lib/integrations/integrations.json # Generated client-safe integration catalog
|
||||
apps/sim/lib/integrations/service-account-metadata.ts # Lightweight service-account projection
|
||||
```
|
||||
|
||||
## Step 2: Pull API Documentation
|
||||
|
||||
Fetch the official API docs for the service. This is the **source of truth** for:
|
||||
- Endpoint URLs, HTTP methods, and auth headers
|
||||
- Required vs optional parameters
|
||||
- Parameter types and allowed values
|
||||
- Response shapes and field names
|
||||
- Pagination patterns (which param name, which response field)
|
||||
- Rate limits and error formats
|
||||
|
||||
### Hard Rule: No Guessed Response Schemas
|
||||
|
||||
If the official docs do not clearly show the response JSON shape for an endpoint, you MUST tell the user instead of guessing.
|
||||
|
||||
- Do NOT assume field names from nearby endpoints
|
||||
- Do NOT infer nested JSON paths without evidence
|
||||
- Do NOT treat "likely" fields as confirmed outputs
|
||||
- Do NOT accept implementation guesses as valid just because they are defensive
|
||||
|
||||
If a response schema is unknown, the validation must explicitly call that out and require:
|
||||
1. sample responses from the user,
|
||||
2. live test credentials for verification, or
|
||||
3. trimming the tool/block down to only documented fields.
|
||||
|
||||
## Step 3: Validate Tools
|
||||
|
||||
For **every** tool file, check:
|
||||
|
||||
### Tool ID and Naming
|
||||
- [ ] Tool ID uses `snake_case`: `{service}_{action}` (e.g., `x_create_tweet`, `slack_send_message`)
|
||||
- [ ] Tool `name` is human-readable (e.g., `'X Create Tweet'`)
|
||||
- [ ] Tool `description` is a concise one-liner describing what it does
|
||||
- [ ] Tool `version` is set (`'1.0.0'` or `'2.0.0'` for V2)
|
||||
|
||||
### Params
|
||||
- [ ] All required API params are marked `required: true`
|
||||
- [ ] All optional API params are marked `required: false`
|
||||
- [ ] Every param has explicit `required: true` or `required: false` — never omitted
|
||||
- [ ] Param types match the API (`'string'`, `'number'`, `'boolean'`, `'json'`)
|
||||
- [ ] Visibility is correct:
|
||||
- `'hidden'` — ONLY for OAuth access tokens and system-injected params
|
||||
- `'user-only'` — for API keys, credentials, and account-specific IDs the user must provide
|
||||
- `'user-or-llm'` — for everything else (search queries, content, filters, IDs that could come from other blocks)
|
||||
- [ ] Every param has a `description` that explains what it does
|
||||
|
||||
### Request
|
||||
- [ ] URL matches the API endpoint exactly (correct base URL, path segments, path params)
|
||||
- [ ] HTTP method matches the API spec (GET, POST, PUT, PATCH, DELETE)
|
||||
- [ ] Headers include correct auth pattern:
|
||||
- OAuth: `Authorization: Bearer ${params.accessToken}`
|
||||
- API Key: correct header name and format per the service's docs
|
||||
- [ ] `Content-Type` header is set for POST/PUT/PATCH requests
|
||||
- [ ] Body sends all required fields and only includes optional fields when provided
|
||||
- [ ] For GET requests with query params: URL is constructed correctly with query string
|
||||
- [ ] ID fields in URL paths are `.trim()`-ed to prevent copy-paste whitespace errors
|
||||
- [ ] Path params use template literals correctly: `` `https://api.service.com/v1/${params.id.trim()}` ``
|
||||
|
||||
### Response / transformResponse
|
||||
- [ ] Correctly parses the API response (`await response.json()`)
|
||||
- [ ] Extracts the right fields from the response structure (e.g., `data.data` vs `data` vs `data.results`)
|
||||
- [ ] All nullable fields use `?? null`
|
||||
- [ ] All optional arrays use `?? []`
|
||||
- [ ] Error cases are handled: checks for missing/empty data and returns meaningful error
|
||||
- [ ] Does NOT do raw JSON dumps — extracts meaningful, individual fields
|
||||
- [ ] Every extracted field is backed by official docs or live-verified sample payloads
|
||||
|
||||
### Outputs
|
||||
- [ ] All output fields match what the API actually returns
|
||||
- [ ] No fields are missing that the API provides and users would commonly need
|
||||
- [ ] No phantom fields defined that the API doesn't return
|
||||
- [ ] `optional: true` is set on fields that may not exist in all responses
|
||||
- [ ] When using `type: 'json'` and the shape is known, `properties` defines the inner fields (tool outputs only — block outputs do not support `properties`)
|
||||
- [ ] When using `type: 'array'`, `items` defines the item structure with `properties` (tool outputs only)
|
||||
- [ ] Field descriptions are accurate and helpful
|
||||
|
||||
### Types (types.ts)
|
||||
- [ ] Has param interfaces for every tool (e.g., `XCreateTweetParams`)
|
||||
- [ ] Has response interfaces for every tool (extending `ToolResponse`)
|
||||
- [ ] Optional params use `?` in the interface (e.g., `replyTo?: string`)
|
||||
- [ ] Field names in types match actual API field names
|
||||
- [ ] Shared response types are properly reused (e.g., `XTweetResponse` shared across tweet tools)
|
||||
|
||||
### Barrel Export (index.ts)
|
||||
- [ ] Every tool is exported
|
||||
- [ ] All types are re-exported (`export * from './types'`)
|
||||
- [ ] No orphaned exports (tools that don't exist)
|
||||
|
||||
### Tool Registry (tools/registry.ts)
|
||||
- [ ] Every tool is imported and registered
|
||||
- [ ] Registry keys use snake_case and match tool IDs exactly
|
||||
- [ ] Entries are in alphabetical order within the file
|
||||
|
||||
### Resolved-Secret Provenance and Model Input
|
||||
|
||||
For every request field, determine whether it is ordinary API input, model-visible text/structured
|
||||
content, opaque model input, or a value persisted into Sim-owned durable storage.
|
||||
|
||||
Treat model-input provenance as opt-in. Require official documentation or an unambiguous local
|
||||
execution path proving that the exact field reaches an AI model. If the evidence is ambiguous,
|
||||
leave the integration unchanged; do not infer a model boundary merely from natural-language,
|
||||
search, extraction, or "AI-powered" marketing terminology.
|
||||
|
||||
- [ ] AI-consumed text/structured fields use `request.modelInput` with `mode: 'project'` and a
|
||||
minimal exact selector; nested/JSON-string adapters preserve shape through `applyProjected`
|
||||
- [ ] Ordinary external URLs, domains, resource IDs, and control fields retain normal request
|
||||
semantics unless the exact field is proven model-visible; an AI-backed provider or later model
|
||||
processing of the referenced resource is not sufficient evidence
|
||||
- [ ] Serialized content proven to be sent directly to an external model is selected by
|
||||
`request.modelInput`, projected before the existing formatter parses it, and has deterministic
|
||||
formatter behavior when a whole-value placeholder is invalid for the serialized grammar
|
||||
- [ ] Actual inline/raw AI-consumed bytes owned by an authenticated internal route use
|
||||
`privateProvenance` (or `mode: 'private-provenance'`), and the route validates
|
||||
`validateOpaqueModelInputProvenance` before model egress; storage keys, paths, signed URLs,
|
||||
and ordinary remote URLs are not treated as byte provenance, while tracked stored bytes are
|
||||
authorized independently at the owning model-egress boundary
|
||||
- [ ] Persisted workspace-file contents are checked with the shared provenance guard only when
|
||||
their bytes or decoded content cross into a model/tool-result boundary; ordinary file APIs
|
||||
remain unchanged. Unsupported secret-bearing file paths are rejected at `file_write`
|
||||
- [ ] Sim-owned durable writes and internal execution handoffs that can enter workflows/models use
|
||||
field-scoped `request.secretProvenance`; authenticated receivers validate the exact selection
|
||||
and scope, strip private metadata, and persist, import, or propagate it at the owning boundary
|
||||
- [ ] Private provenance is never attached to external URLs or `directExecution`; proven
|
||||
model-visible external fields use projection, while other external inputs remain unchanged
|
||||
- [ ] No tool performs raw secret plaintext/source substitution or serializes plaintext provenance
|
||||
- [ ] No `transformResponse` or tool-local helper blanket-sanitizes ordinary third-party results;
|
||||
only execution-scoped, activated Sim provenance is projected at shared model/log boundaries
|
||||
- [ ] Private headers/envelopes are produced and stripped by the shared tool executor, never
|
||||
hand-rolled or returned as functional output
|
||||
- [ ] Every added provenance hook has a concrete Sim `{{...}}` resolution path and a later
|
||||
persistence/model/log crossing; there is no generic handling for arbitrary filenames,
|
||||
metadata, provider results, or API payloads
|
||||
- [ ] Diagnostic projection is applied only to values carrying execution-scoped provenance;
|
||||
ordinary provider responses, filenames, URLs, and errors are unchanged
|
||||
- [ ] Tests cover named `{{NAME}}` projection, unproven identical public text, nested and serialized
|
||||
shape handling, unchanged ordinary external inputs, malformed/incomplete metadata, headerless
|
||||
legacy requests, metadata stripping, and durable legacy/stale/scope cases when applicable
|
||||
|
||||
Treat a missing or bypassed model, durable, or internal-execution provenance boundary as
|
||||
**critical**. Do not fix it with a tool-specific string replacer or by sanitizing every provider
|
||||
result; repair the shared request, authenticated internal-route, persistence, or re-entry boundary
|
||||
that owns the data.
|
||||
|
||||
## Step 4: Validate Block
|
||||
|
||||
### Block ↔ Tool Alignment (CRITICAL)
|
||||
|
||||
This is the most important validation — the block must be perfectly aligned with every tool it references.
|
||||
|
||||
For **each tool** in `tools.access`:
|
||||
- [ ] The operation dropdown has an option whose ID matches the tool ID (or the `tools.config.tool` function correctly maps to it)
|
||||
- [ ] Every **required** tool param (except `accessToken`) has a corresponding subBlock input that is:
|
||||
- Shown when that operation is selected (correct `condition`)
|
||||
- Marked as `required: true` (or conditionally required)
|
||||
- [ ] Every **optional** tool param has a corresponding subBlock input (or is intentionally omitted if truly never needed)
|
||||
- [ ] SubBlock `id` values are unique across the entire block — no duplicates even across different conditions
|
||||
- [ ] The `tools.config.tool` function returns the correct tool ID for every possible operation value
|
||||
- [ ] The `tools.config.params` function correctly maps subBlock IDs to tool param names when they differ
|
||||
|
||||
### SubBlocks
|
||||
- [ ] Operation dropdown lists ALL tool operations available in `tools.access`
|
||||
- [ ] Dropdown option labels are human-readable and descriptive
|
||||
- [ ] Conditions use correct syntax:
|
||||
- Single value: `{ field: 'operation', value: 'x_create_tweet' }`
|
||||
- Multiple values (OR): `{ field: 'operation', value: ['x_create_tweet', 'x_delete_tweet'] }`
|
||||
- Negation: `{ field: 'operation', value: 'delete', not: true }`
|
||||
- Compound: `{ field: 'op', value: 'send', and: { field: 'type', value: 'dm' } }`
|
||||
- [ ] Condition arrays include ALL operations that use that field — none missing
|
||||
- [ ] `dependsOn` is set for fields that need other values (selectors depending on credential, cascading dropdowns)
|
||||
- [ ] SubBlock types match tool param types:
|
||||
- Enum/fixed options → `dropdown`
|
||||
- Free text → `short-input`
|
||||
- Long text/content → `long-input`
|
||||
- True/false → `dropdown` with Yes/No options (not `switch` unless purely UI toggle)
|
||||
- Credentials → `oauth-input` with correct `serviceId`
|
||||
- [ ] Dropdown `value: () => 'default'` is set for dropdowns with a sensible default
|
||||
|
||||
### Advanced Mode
|
||||
- [ ] Optional, rarely-used fields are set to `mode: 'advanced'`:
|
||||
- Pagination tokens / next tokens
|
||||
- Time range filters (start/end time)
|
||||
- Sort order / direction options
|
||||
- Max results / per page limits
|
||||
- Reply settings / threading options
|
||||
- Rarely used IDs (reply-to, quote-tweet, etc.)
|
||||
- Exclude filters
|
||||
- [ ] **Required** fields are NEVER set to `mode: 'advanced'`
|
||||
- [ ] Fields that users fill in most of the time are NOT set to `mode: 'advanced'`
|
||||
|
||||
### WandConfig
|
||||
- [ ] Timestamp fields have `wandConfig` with `generationType: 'timestamp'`
|
||||
- [ ] Comma-separated list fields have `wandConfig` with a descriptive prompt
|
||||
- [ ] Complex filter/query fields have `wandConfig` with format examples in the prompt
|
||||
- [ ] All `wandConfig` prompts end with "Return ONLY the [format] - no explanations, no extra text."
|
||||
- [ ] `wandConfig.placeholder` describes what to type in natural language
|
||||
|
||||
### Tools Config
|
||||
- [ ] `tools.access` lists **every** tool ID the block can use — none missing
|
||||
- [ ] `tools.config.tool` returns the correct tool ID for each operation
|
||||
- [ ] Type coercions are in `tools.config.params` (runs at execution time), NOT in `tools.config.tool` (runs at serialization time before variable resolution)
|
||||
- [ ] `tools.config.params` handles:
|
||||
- `Number()` conversion for numeric params that come as strings from inputs
|
||||
- `Boolean` / string-to-boolean conversion for toggle params
|
||||
- Empty string → `undefined` conversion for optional dropdown values
|
||||
- Any subBlock ID → tool param name remapping
|
||||
- [ ] No `Number()`, `JSON.parse()`, or other coercions in `tools.config.tool` — these would destroy dynamic references like `<Block.output>`
|
||||
|
||||
### Block Outputs
|
||||
- [ ] Outputs cover the key fields returned by ALL tools (not just one operation)
|
||||
- [ ] Output types are correct (`'string'`, `'number'`, `'boolean'`, `'json'`)
|
||||
- [ ] `type: 'json'` outputs describe inner fields in the description string: `'User profile (id, name, username, bio)'` or `'[{address, status, type}]'` for arrays
|
||||
- [ ] **Do NOT add a `properties: {...}` field on block outputs.** Block-level `OutputFieldDefinition` (from `@sim/workflow-types/blocks`) only accepts `{ type, description?, condition?, hiddenFromDisplay? }`. Nested `properties` is a tool-level construct (`OutputProperty`) — adding it to a block output will fail TypeScript at build time
|
||||
- [ ] No opaque `type: 'json'` with vague descriptions like `'Response data'`
|
||||
- [ ] Outputs that only appear for certain operations use `condition` if supported, or document which operations return them
|
||||
|
||||
### Block Metadata
|
||||
- [ ] `type` is snake_case (e.g., `'x'`, `'cloudflare'`)
|
||||
- [ ] `name` is human-readable (e.g., `'X'`, `'Cloudflare'`)
|
||||
- [ ] `description` is a concise one-liner
|
||||
- [ ] `longDescription` provides detail for docs
|
||||
- [ ] `docsLink` points to `'https://docs.sim.ai/integrations/{service}'`
|
||||
- [ ] `category` is `'tools'`
|
||||
- [ ] `bgColor` uses the service's brand color hex
|
||||
- [ ] `icon` references the correct icon component from `@/components/icons`
|
||||
- [ ] `authMode` is set correctly (`AuthMode.OAuth` or `AuthMode.ApiKey`)
|
||||
- [ ] Block + meta are registered in `blocks/registry-maps.ts` (`BLOCK_REGISTRY` / `BLOCK_META_REGISTRY`) alphabetically
|
||||
|
||||
### BlockMeta
|
||||
- [ ] `{Service}BlockMeta` is exported in the same file as the block
|
||||
- [ ] Has at least 7 templates, each with `icon`, `title`, `prompt`, `modules`, `category`, and `tags`
|
||||
- [ ] Prompts describe concrete use cases, not generic descriptions of what the service does
|
||||
- [ ] `alsoIntegrations` is set on any template whose prompt references another service
|
||||
- [ ] `skills` present (3–5 mainstream, 2–3 niche), each grounded in `tools.access` — flag any skill implying an unsupported action
|
||||
- [ ] **Each skill is real, not hallucinated** — web-search and confirm it maps to a popular use case attested online (vendor use-case pages, official docs describing the workflow, reputable "top automations" articles); rewrite/remove any you cannot source
|
||||
- [ ] Each skill has a kebab-case `name` (≤64 chars, unique), a one-line `description`, and markdown `content` with `# Title` + `## Steps` + an output/guidance section
|
||||
|
||||
### Block Inputs
|
||||
- [ ] `inputs` section lists all subBlock params that the block accepts
|
||||
- [ ] Input types match the subBlock types
|
||||
- [ ] When using `canonicalParamId`, inputs list the canonical ID (not the raw subBlock IDs)
|
||||
|
||||
## Step 5: Validate OAuth Scopes (if OAuth service)
|
||||
|
||||
Scopes are centralized — the single source of truth is `OAUTH_PROVIDERS` in `lib/oauth/oauth.ts`.
|
||||
|
||||
- [ ] Scopes defined in `lib/oauth/oauth.ts` under `OAUTH_PROVIDERS[provider].services[service].scopes`
|
||||
- [ ] `auth.ts` uses `getCanonicalScopesForProvider(providerId)` — NOT a hardcoded array
|
||||
- [ ] Block `requiredScopes` uses `getScopesForService(serviceId)` — NOT a hardcoded array
|
||||
- [ ] No hardcoded scope arrays in `auth.ts` or block files (should all use utility functions)
|
||||
- [ ] Each scope has a human-readable description in `SCOPE_DESCRIPTIONS` within `lib/oauth/utils.ts`
|
||||
- [ ] No excess scopes that aren't needed by any tool
|
||||
|
||||
## Step 6: Validate Deployment Availability (if OAuth service)
|
||||
|
||||
The deployment UI and setup CLI do not infer OAuth client fields from scopes. They resolve the
|
||||
block's generated `oauthServiceId` through the application-owned capability catalog.
|
||||
|
||||
- [ ] The visible integration block has exactly one distinct `oauth-input.serviceId`
|
||||
- [ ] `resolveOAuthClientCapabilityId(serviceId)` returns the intended provider capability
|
||||
- [ ] The resolved provider exists in `OAUTH_CLIENT_CAPABILITIES`
|
||||
- [ ] Every field listed by that capability exists in `apps/sim/lib/core/config/env.ts`
|
||||
- [ ] Every capability field has the correct `text` or `secret` entry in `OAUTH_CLIENT_SETUP_FIELDS`; no CLI naming heuristic is required
|
||||
- [ ] Shared Google/Microsoft service IDs resolve to their provider capability rather than duplicate entries
|
||||
- [ ] `bun run setup integration <capabilityId>` is the command emitted by availability; the CLI has only the exhaustive input-mode projection, not a second runtime provider definition
|
||||
- [ ] If the canonical OAuth service declares `serviceAccountProviderId`,
|
||||
`SERVICE_ACCOUNT_METADATA_BY_OAUTH_SERVICE_ID[serviceId]` has the same provider ID
|
||||
- [ ] The service-account `deploymentRequirement` matches how that credential actually works:
|
||||
omitted for an independent path, `'oauth-client'` when it needs the OAuth client fields, or
|
||||
`'preview-gated'` when controlled by a preview block
|
||||
|
||||
Treat a missing capability as **critical**: runtime availability intentionally throws instead of
|
||||
silently exposing an unusable integration.
|
||||
|
||||
## Step 7: Validate Pagination Consistency
|
||||
|
||||
If any tools support pagination:
|
||||
- [ ] Pagination param names match the API docs (e.g., `pagination_token` vs `next_token` vs `cursor`)
|
||||
- [ ] Different API endpoints that use different pagination param names have separate subBlocks in the block
|
||||
- [ ] Pagination response fields (`nextToken`, `cursor`, etc.) are included in tool outputs
|
||||
- [ ] Pagination subBlocks are set to `mode: 'advanced'`
|
||||
|
||||
## Step 8: Validate Memory Load Safety
|
||||
|
||||
If any tool lists, searches, exports, imports, downloads, uploads, paginates, batches, transforms arrays, or reads file/HTTP bodies, read `.agents/skills/memory-load-check/SKILL.md` and apply it to the integration.
|
||||
|
||||
- [ ] List/search tools expose API limits and do not auto-fetch every page into memory
|
||||
- [ ] Transform logic does not build unbounded arrays, maps, sets, or `Promise.all` fan-outs
|
||||
- [ ] File and HTTP body reads use explicit byte caps or existing stream-limit helpers
|
||||
- [ ] Large result payloads are summarized, paginated, referenced, or capped rather than raw-dumped
|
||||
- [ ] Pagination and download tests cover caps, early stop behavior, or partial-result preservation when relevant
|
||||
|
||||
## Step 9: Validate Error Handling
|
||||
|
||||
- [ ] `transformResponse` checks for error conditions before accessing data
|
||||
- [ ] Error responses include meaningful messages (not just generic "failed")
|
||||
- [ ] HTTP error status codes are handled (check `response.ok` or status codes)
|
||||
|
||||
## Step 10: Report and Fix
|
||||
|
||||
### Report Format
|
||||
|
||||
Group findings by severity:
|
||||
|
||||
**Critical** (will cause runtime errors or incorrect behavior):
|
||||
- Wrong endpoint URL or HTTP method
|
||||
- Missing required params or wrong `required` flag
|
||||
- Incorrect response field mapping (accessing wrong path in response)
|
||||
- Missing error handling that would cause crashes
|
||||
- Tool ID mismatch between tool file, registry, and block `tools.access`
|
||||
- OAuth scopes missing in `auth.ts` that tools need
|
||||
- OAuth integration `serviceId` missing from the deployment capability catalog
|
||||
- Capability references an env field absent from the runtime env schema
|
||||
- Service-account metadata disagrees with the canonical OAuth service configuration
|
||||
- `tools.config.tool` returning wrong tool ID for an operation
|
||||
- Type coercions in `tools.config.tool` instead of `tools.config.params`
|
||||
- Proven model-visible request fields bypass the shared projection or private-provenance boundary
|
||||
- Opaque model input is downloaded or sent before provenance and workspace-file checks
|
||||
- A Sim-owned durable sink or internal execution handoff drops encrypted provenance or breaks
|
||||
legacy headerless/`NULL` data
|
||||
- A tool substitutes secret plaintext into source, leaks private metadata, or generically sanitizes
|
||||
unrelated third-party results
|
||||
|
||||
**Warning** (follows conventions incorrectly or has usability issues):
|
||||
- Optional field not set to `mode: 'advanced'`
|
||||
- Missing `wandConfig` on timestamp/complex fields
|
||||
- Wrong `visibility` on params (e.g., `'hidden'` instead of `'user-or-llm'`)
|
||||
- Missing `optional: true` on nullable outputs
|
||||
- Opaque `type: 'json'` without property descriptions
|
||||
- Missing `.trim()` on ID fields in request URLs
|
||||
- Missing `?? null` on nullable response fields
|
||||
- Block condition array missing an operation that uses that field
|
||||
- Hardcoded scope arrays instead of using `getScopesForService()` / `getCanonicalScopesForProvider()`
|
||||
- Missing scope description in `SCOPE_DESCRIPTIONS` within `lib/oauth/utils.ts`
|
||||
|
||||
**Suggestion** (minor improvements):
|
||||
- Better description text
|
||||
- Inconsistent naming across tools
|
||||
- Missing `longDescription` or `docsLink`
|
||||
- Pagination fields that could benefit from `wandConfig`
|
||||
|
||||
### Fix All Issues
|
||||
|
||||
After reporting, fix every **critical** and **warning** issue. Apply **suggestions** where they don't add unnecessary complexity.
|
||||
|
||||
### Regenerate Derived Artifacts
|
||||
|
||||
Several files are generated from tool and block definitions. Editing a tool or block WITHOUT regenerating them fails CI, so run these before pushing:
|
||||
|
||||
```bash
|
||||
bun run tool-metadata:generate # repo root — apps/sim/tools/generated/*
|
||||
bun run scripts/generate-docs.ts # docs .mdx + lib/integrations/integrations.json + docs icons
|
||||
bun run integration-catalog:check # registry ↔ committed deployment metadata drift
|
||||
```
|
||||
|
||||
- **`tool-metadata:generate`** — required whenever a tool's `outputs`, `params`, or descriptions change. CI enforces this with `bun run tool-metadata:check`, which fails with *"Generated tool metadata is stale"*. This is the easiest gate to miss, because nothing in the tool file hints that a generated artifact mirrors it.
|
||||
- **`generate-docs`** — required whenever block metadata changes (`bgColor`, `name`, `description`, operations, outputs). Regenerates the integration `.mdx`, `integrations.json`, and the docs copy of `components/icons.tsx`.
|
||||
- **`integration-catalog:check`** — loads the executable block registry, derives visible integration
|
||||
deployment fields, and compares them with the committed catalog. It catches missing/unexpected
|
||||
entries and stale auth/service IDs without loading the executable registry in client code.
|
||||
|
||||
**Always diff the regen output before committing.** These generators rewrite every file they own, so they will also sweep in unrelated drift that accumulated on the base branch — pages losing sections, unrelated icons appearing. Keep only the hunks belonging to the integration under validation and `git checkout --` the rest, otherwise an unrelated doc regression rides along in the PR. Verify no page was silently dropped by comparing the directory listing before and after.
|
||||
|
||||
If an icon changed, `apps/sim/components/icons.tsx` is the source of truth and `apps/docs/components/icons.tsx` is its generated mirror — they must end up byte-identical for that component.
|
||||
|
||||
### Validation Output
|
||||
|
||||
After fixing, confirm:
|
||||
1. `bun run lint` passes with no fixes needed
|
||||
2. TypeScript compiles clean (no type errors) — check the error list is empty for the files you touched; pre-existing unrelated errors in a worktree usually mean workspace packages resolve to the main checkout
|
||||
3. The integration's tests pass, and any test you added actually fails without its fix (revert it once and watch it go red)
|
||||
4. Derived artifacts regenerated and their diffs reviewed (see above)
|
||||
5. `bun run integration-catalog:check` passes
|
||||
6. For OAuth or service-account changes, `bun test apps/sim/lib/integrations/availability.server.test.ts` passes
|
||||
7. Re-read all modified files to verify fixes are correct
|
||||
8. Any remaining unknown response schemas were explicitly reported to the user instead of guessed
|
||||
|
||||
## Checklist Summary
|
||||
|
||||
- [ ] Read ALL tool files, block, types, index, and registries
|
||||
- [ ] Pulled and read official API documentation
|
||||
- [ ] Validated every tool's ID, params, request, response, outputs, and types against API docs
|
||||
- [ ] Validated block ↔ tool alignment (every tool param has a subBlock, every condition is correct)
|
||||
- [ ] Validated advanced mode on optional/rarely-used fields
|
||||
- [ ] Validated wandConfig on timestamps and complex inputs
|
||||
- [ ] Validated tools.config mapping, tool selector, and type coercions
|
||||
- [ ] Validated block outputs match what tools return, with typed JSON where possible
|
||||
- [ ] Validated OAuth scopes use centralized utilities (getScopesForService, getCanonicalScopesForProvider) — no hardcoded arrays
|
||||
- [ ] Validated scope descriptions exist in `SCOPE_DESCRIPTIONS` within `lib/oauth/utils.ts` for all scopes
|
||||
- [ ] Validated OAuth `serviceId` resolves to the intended `OAUTH_CLIENT_CAPABILITIES` entry and all capability fields exist in the env schema
|
||||
- [ ] Validated service-account projection and deployment requirement against the canonical OAuth service config
|
||||
- [ ] Regenerated `integrations.json` when block metadata changed and ran `bun run integration-catalog:check`
|
||||
- [ ] Validated pagination consistency across tools and block
|
||||
- [ ] Validated memory load safety using `.agents/skills/memory-load-check/SKILL.md` when tools list/search/download/import/export/batch data
|
||||
- [ ] Validated error handling (error checks, meaningful messages)
|
||||
- [ ] Validated registry entries (tools and block, alphabetical, correct imports)
|
||||
- [ ] Validated model-visible/opaque inputs and Sim-durable/internal-execution provenance at their
|
||||
owning boundaries
|
||||
- [ ] Confirmed legacy persisted data keeps working and tracked invalid provenance fails closed
|
||||
- [ ] Confirmed ordinary third-party results remain unchanged absent activated Sim provenance
|
||||
- [ ] Validated `{Service}BlockMeta` exported with at least 7 templates
|
||||
- [ ] Reported all issues grouped by severity
|
||||
- [ ] Fixed all critical and warning issues
|
||||
- [ ] Ran `bun run tool-metadata:generate` if any tool outputs/params changed, and confirmed `bun run tool-metadata:check` passes
|
||||
- [ ] Ran `bun run generate-docs` if any block metadata changed, and reverted unrelated drift the generator swept in
|
||||
- [ ] Ran `bun run lint` after fixes
|
||||
- [ ] Verified TypeScript compiles clean
|
||||
- [ ] Verified added tests fail without their fix
|
||||
@@ -0,0 +1,5 @@
|
||||
interface:
|
||||
display_name: "Validate Integration"
|
||||
short_description: "Audit a Sim service integration"
|
||||
brand_color: "#B45309"
|
||||
default_prompt: "Use $validate-integration to audit and fix a Sim integration against its API docs."
|
||||
@@ -0,0 +1,172 @@
|
||||
---
|
||||
name: validate-model
|
||||
description: Validate a model entry (or every model in a provider) in apps/sim/providers/models.ts against the provider's live API docs (no hallucination — reports what cannot be verified)
|
||||
argument-hint: <provider> [model-id]
|
||||
---
|
||||
|
||||
# Validate Model Skill
|
||||
|
||||
You audit one or more model entries in `apps/sim/providers/models.ts` against the provider's official live API docs. **Hallucinated pricing and capabilities are the #1 failure mode in this file.** Every numeric and capability claim must be re-derived from a live web fetch in this session — not from memory, not from training data, not from the user's marketing email.
|
||||
|
||||
## Hard rules (do not skip)
|
||||
|
||||
1. **Live-fetch or report unverified.** Each field must be backed by a live WebFetch in this session. If you cannot reach an authoritative URL for a field, mark it **UNVERIFIED** in the report — do not silently confirm it from memory.
|
||||
2. **Cite every fact.** Every value in the report must show the source URL it was checked against. No URL → mark UNVERIFIED.
|
||||
3. **Two-source rule for pricing.** Cross-check input/output/cached against at least one secondary source (OpenRouter, Artificial Analysis, CloudPrice). If sources disagree, the provider's own docs win — flag the disagreement.
|
||||
4. **Inspect provider implementation before flagging capability mismatches.** A capability flag in `models.ts` is dead unless the provider's code under `apps/sim/providers/{provider}/` consumes it (see Consumption Matrix below). Setting a flag the provider ignores is a warning, not a critical.
|
||||
5. **Never auto-fix without printing the diff.** Show the user the proposed diff before applying. Get confirmation.
|
||||
|
||||
## Your Task
|
||||
|
||||
When invoked as `/validate-model <provider> [model-id]`:
|
||||
|
||||
1. Read the target entries from `models.ts`
|
||||
2. Live-fetch the provider's official models, pricing, and capability/reasoning pages + at least one secondary source for pricing
|
||||
3. Inspect the provider implementation to know which flags are actually consumed
|
||||
4. Run the checklist below per model
|
||||
5. Report findings (critical / warning / suggestion / unverified) with every cell linked to its source URL
|
||||
6. Offer to fix; on confirm, edit `models.ts` in a single pass and re-lint
|
||||
|
||||
If `model-id` is omitted, validate every model in the provider.
|
||||
|
||||
## Step 1: Read entries from `models.ts`
|
||||
|
||||
Capture per model: `id`, full `pricing`, full `capabilities`, `contextWindow`, `releaseDate`, `recommended`, `speedOptimized`, `deprecated`.
|
||||
|
||||
## Step 2: Live-fetch authoritative sources
|
||||
|
||||
Use the canonical provider URL table in the `add-model` skill (`.agents/skills/add-model/SKILL.md`), Step 1, as the single source of truth — fetch the models index, pricing, and reasoning/parameter caveats pages listed there for the target provider.
|
||||
|
||||
Secondary cross-check (use at least one): OpenRouter, Artificial Analysis, CloudPrice.
|
||||
|
||||
If a fetch fails (404, timeout, paywall), record the URL attempted and mark dependent fields UNVERIFIED.
|
||||
|
||||
## Step 3: Build the consumption map for this provider
|
||||
|
||||
Re-grep before trusting the snapshot below:
|
||||
|
||||
```bash
|
||||
rg "reasoningEffort|reasoning_effort" apps/sim/providers/<provider>/
|
||||
rg "verbosity" apps/sim/providers/<provider>/
|
||||
rg "request\.thinking|thinking:" apps/sim/providers/<provider>/
|
||||
rg "supportsNativeStructuredOutputs|nativeStructuredOutputs" apps/sim/providers/<provider>/
|
||||
```
|
||||
|
||||
Snapshot (verify before relying):
|
||||
|
||||
| Capability | Consumed by |
|
||||
|---|---|
|
||||
| `reasoningEffort` | `openai/core.ts`, `azure-openai`, `anthropic/core.ts` (mapped via thinking), `gemini/core.ts` |
|
||||
| `verbosity` | `openai/core.ts`, `azure-openai/index.ts` |
|
||||
| `thinking` | `anthropic/core.ts`, `gemini/core.ts` |
|
||||
| `nativeStructuredOutputs` | `anthropic/core.ts`, `fireworks/index.ts`, `openrouter/index.ts` |
|
||||
| `computerUse` | `anthropic/core.ts` |
|
||||
| `temperature` | All providers (passthrough) |
|
||||
|
||||
A flag set in `models.ts` but not in the consumption list for this provider = **warning: dead flag**.
|
||||
|
||||
## Step 4: Run the checklist
|
||||
|
||||
For each model, evaluate every row. Statuses: ✓ matches docs, ✗ disagrees, ⚠️ single-source, ❓ UNVERIFIED (could not fetch).
|
||||
|
||||
### Identity
|
||||
- [ ] `id` exactly matches provider's API model identifier (case, dots, dashes, prefix for resellers)
|
||||
- [ ] `releaseDate` matches launch announcement
|
||||
- [ ] `deprecated: true` set if provider has announced retirement (or removed from active list)
|
||||
|
||||
### Pricing (per 1M tokens, USD)
|
||||
- [ ] `pricing.input` matches provider pricing page
|
||||
- [ ] `pricing.output` matches provider pricing page
|
||||
- [ ] `pricing.cachedInput` matches provider's documented cached/prompt-cache rate (or is correctly omitted if no caching offered)
|
||||
- [ ] `pricing.updatedAt` is recent — warn if older than 60 days
|
||||
|
||||
### Context & output limits
|
||||
- [ ] `contextWindow` matches docs (in tokens)
|
||||
- [ ] `capabilities.maxOutputTokens` matches documented output cap (or is correctly omitted if "no output limit")
|
||||
|
||||
### Capabilities (each must be DOCUMENTED-AS-SUPPORTED **and** CONSUMED-BY-PROVIDER-CODE)
|
||||
- [ ] `temperature` — provider accepts it for this model (reasoning-always-on models often reject)
|
||||
- [ ] `reasoningEffort.values` — list matches docs; **omitted** for always-reasoning models that reject the parameter (e.g., grok-4.3, where xAI docs explicitly state `reasoning_effort` is not supported). Verify per model — some always-reasoning models (e.g., OpenAI's o-series) DO accept `reasoning_effort` and should keep the flag.
|
||||
- [ ] `verbosity.values` — only on OpenAI gpt-5.x family; values match docs
|
||||
- [ ] `thinking.levels` + `thinking.default` — only on Anthropic/Gemini; values match docs
|
||||
- [ ] `thinking.streamed` — REQUIRED on Anthropic-family thinking models (`'full'` for generations returning full thinking deltas, `'summary'` for omitted-display generations like Opus 4.7+/Sonnet 5/Fable 5 where Sim requests `display: 'summarized'`); verify against the provider's thinking-display docs. After any change, run `bun run agent-stream-docs:generate` so the Agent block docs table stays in sync (CI diffs it)
|
||||
- [ ] `nativeStructuredOutputs` — only on anthropic/fireworks/openrouter; provider must document Structured Outputs / JSON-mode for this model
|
||||
- [ ] `toolUsageControl` — provider supports `tool_choice` semantics
|
||||
- [ ] `computerUse` — provider implements computer-use loop AND model is a computer-use SKU
|
||||
- [ ] `deepResearch` — only on actual deep-research SKUs
|
||||
- [ ] `memory: false` — only when the model genuinely cannot maintain conversation history
|
||||
|
||||
### Flags
|
||||
- [ ] `recommended: true` — at most one or two per provider; should be current flagship
|
||||
- [ ] `speedOptimized: true` — only on smallest/fastest tier (nano / flash-lite / haiku class)
|
||||
|
||||
### Hosting / billing
|
||||
- [ ] If the model is under `openai`/`anthropic`/`google`, it is automatically in `getHostedModels()` → served with Sim's rotating key and billed via `shouldBillModelUsage()`. Confirm that is the intent (a BYOK-only model parked under one of these providers is a billing bug — warning).
|
||||
- [ ] If the model is hosted, the deployment is expected to have its `{PREFIX}_COUNT` / `{PREFIX}_1..N` env vars set (ops concern; note if it looks unset for a model claiming hosted support).
|
||||
|
||||
## Step 5: Report (mandatory format)
|
||||
|
||||
For each model, emit a table with one row per checklist item. Every row that claims ✓ must have a URL.
|
||||
|
||||
```markdown
|
||||
### Validation — <model-id>
|
||||
|
||||
| Field | Repo | Live docs | Source URL | Status |
|
||||
|---|---|---|---|---|
|
||||
| `input` | $1.25/M | $1.25/M | https://docs.x.ai/... | ✓ |
|
||||
| `cachedInput` | $0.50/M | $0.20/M | https://cloudprice.net/... | ✗ stale (price cut not picked up) |
|
||||
| `reasoningEffort` | low/medium/high | rejected by API | https://docs.x.ai/.../reasoning | ✗ inert — selecting silently no-ops |
|
||||
| `contextWindow` | 1,000,000 | 1,000,000 | https://docs.x.ai/... + https://openrouter.ai/... | ✓ (2 sources) |
|
||||
| `releaseDate` | 2026-04-30 | not found in scraped pages | _attempted: docs.x.ai, x.ai/news_ | ❓ UNVERIFIED |
|
||||
|
||||
**Findings**
|
||||
- 🔴 critical — `cachedInput` is wrong: docs say $0.20/M, repo has $0.50/M
|
||||
- 🟡 warning — `reasoningEffort` is set but provider rejects it for this model (xAI docs explicitly: "reasoning_effort is not supported by grok-4.3")
|
||||
- 🔵 suggestion — `pricing.updatedAt` is 90 days old; refresh
|
||||
- ❓ unverified — `releaseDate` could not be confirmed from any fetched page; ask user
|
||||
|
||||
**Disagreements between sources**
|
||||
- _none_ OR _OpenRouter says $X, provider docs say $Y — went with provider docs_
|
||||
```
|
||||
|
||||
End each multi-model run with a summary count: `N models checked · X critical · Y warnings · Z suggestions · W unverified`.
|
||||
|
||||
## Step 6: Offer to fix
|
||||
|
||||
After reporting, ask: *"Want me to fix the critical and warning items? I'll print the diff first."* On yes:
|
||||
|
||||
1. Print the proposed diff (do not apply yet)
|
||||
2. Get user confirmation
|
||||
3. Edit `models.ts` in a single pass
|
||||
4. Run `bun run lint`
|
||||
5. Re-run only the failed rows of the checklist on the new state
|
||||
|
||||
## Severity definitions
|
||||
|
||||
- 🔴 **critical** — wrong number or wrong identifier that misleads users about cost or breaks API calls. Examples: incorrect pricing, wrong model id, wrong context window, capability the API rejects.
|
||||
- 🟡 **warning** — dead code or internal inconsistency. Examples: capability flag the provider ignores, multiple `recommended: true` per provider, `pricing.updatedAt` >60 days old, missing `deprecated: true` on retired model.
|
||||
- 🔵 **suggestion** — style/consistency. Examples: field order, missing `speedOptimized` on a clearly smallest-tier model.
|
||||
- ❓ **unverified** — could not fetch an authoritative source for this field. Surface it; never silently confirm.
|
||||
|
||||
## Common bugs this skill catches
|
||||
|
||||
- Pricing drift after a provider price cut (very common — providers cut quarterly)
|
||||
- `reasoningEffort` set on always-reasoning models that reject the parameter (grok-4.3, o3-pro pattern)
|
||||
- `nativeStructuredOutputs` set on providers that don't consume the flag (dead)
|
||||
- `thinking` set on non-Anthropic/non-Gemini providers
|
||||
- `verbosity` set on non-gpt-5.x models
|
||||
- Wrong context window (e.g., 128k claimed vs 200k actual)
|
||||
- Stale `pricing.updatedAt`
|
||||
- Multiple `recommended: true` per provider after a flagship swap
|
||||
- Missing `deprecated: true` on retired models (e.g., the xAI batch retiring May 15, 2026)
|
||||
|
||||
## What "I cannot verify this" looks like
|
||||
|
||||
If, after fetching the documented sources, a field cannot be confirmed:
|
||||
|
||||
- Mark the row ❓ UNVERIFIED with the URL(s) attempted
|
||||
- Surface it in the **Findings** section with severity ❓
|
||||
- Do NOT mark the validation as passed
|
||||
- Ask the user for a docs URL or guidance before changing anything
|
||||
|
||||
The skill is allowed to say *"I could not verify the cached input price for grok-4.3 from the official xAI docs in this session — I attempted [URLs] without finding the value. Third-party sources [URL1, URL2] both report $0.20/M. Confirm before I update."* That is correct behavior. Hallucinating a number is not.
|
||||
@@ -0,0 +1,5 @@
|
||||
interface:
|
||||
display_name: "Validate Model"
|
||||
short_description: "Audit model entries vs live docs"
|
||||
brand_color: "#0891B2"
|
||||
default_prompt: "Use $validate-model to audit Sim model entries against the provider's live API docs."
|
||||
@@ -0,0 +1,229 @@
|
||||
---
|
||||
name: validate-trigger
|
||||
description: Validate an existing Sim webhook trigger against provider API docs and repository conventions
|
||||
argument-hint: <service-name> [api-docs-url]
|
||||
---
|
||||
|
||||
# Validate Trigger
|
||||
|
||||
You are an expert auditor for Sim webhook triggers. Your job is to validate that an existing trigger implementation is correct, complete, secure, and aligned across all layers.
|
||||
|
||||
## Your Task
|
||||
|
||||
1. Read the service's webhook/API documentation (via WebFetch)
|
||||
2. Read every trigger file, provider handler, and registry entry
|
||||
3. Cross-reference against the API docs and Sim conventions
|
||||
4. Report all issues grouped by severity (critical, warning, suggestion)
|
||||
5. Fix all issues after reporting them
|
||||
|
||||
## Step 1: Gather All Files
|
||||
|
||||
Read **every** file for the trigger — do not skip any:
|
||||
|
||||
```
|
||||
apps/sim/triggers/{service}/ # All trigger files, utils.ts, index.ts
|
||||
apps/sim/lib/webhooks/providers/{service}.ts # Provider handler (if exists)
|
||||
apps/sim/lib/webhooks/providers/registry.ts # Handler registry
|
||||
apps/sim/triggers/registry.ts # Trigger registry
|
||||
apps/sim/blocks/blocks/{service}.ts # Block definition (trigger wiring)
|
||||
```
|
||||
|
||||
Also read for reference:
|
||||
```
|
||||
apps/sim/lib/webhooks/providers/types.ts # WebhookProviderHandler interface
|
||||
apps/sim/lib/webhooks/providers/utils.ts # Shared helpers (createHmacVerifier, etc.)
|
||||
apps/sim/lib/webhooks/provider-subscription-utils.ts # Subscription helpers
|
||||
apps/sim/lib/webhooks/processor.ts # Central webhook processor
|
||||
```
|
||||
|
||||
## Step 2: Pull API Documentation
|
||||
|
||||
Fetch the service's official webhook documentation. This is the **source of truth** for:
|
||||
- Webhook event types and payload shapes
|
||||
- Signature/auth verification method (HMAC algorithm, header names, secret format)
|
||||
- Challenge/verification handshake requirements
|
||||
- Webhook subscription API (create/delete endpoints, if applicable)
|
||||
- Retry behavior and delivery guarantees
|
||||
|
||||
### Hard Rule: No Guessed Webhook Payload Schemas
|
||||
|
||||
If the official docs do not clearly show the webhook payload JSON for an event, you MUST tell the user instead of guessing.
|
||||
|
||||
- Do NOT invent payload field names
|
||||
- Do NOT infer nested payload paths without evidence
|
||||
- Do NOT treat likely event shapes as verified
|
||||
- Do NOT accept `formatInput` mappings that are not backed by docs or live payloads
|
||||
|
||||
If a payload schema is unknown, validation must explicitly recommend:
|
||||
1. sample webhook payloads,
|
||||
2. a live test webhook source, or
|
||||
3. trimming the trigger to only documented outputs.
|
||||
|
||||
## Step 3: Validate Trigger Definitions
|
||||
|
||||
### utils.ts
|
||||
- [ ] `{service}TriggerOptions` lists all trigger IDs accurately
|
||||
- [ ] `{service}SetupInstructions` provides clear, correct steps for the service
|
||||
- [ ] `build{Service}ExtraFields` includes relevant filter/config fields with correct `condition`
|
||||
- [ ] Output builders expose all meaningful fields from the webhook payload
|
||||
- [ ] Output builders do NOT use `optional: true` or `items` (tool-output-only features)
|
||||
- [ ] Nested output objects correctly model the payload structure
|
||||
|
||||
### Trigger Files
|
||||
- [ ] Exactly one primary trigger has `includeDropdown: true`
|
||||
- [ ] All secondary triggers do NOT have `includeDropdown`
|
||||
- [ ] All triggers use `buildTriggerSubBlocks` helper (not hand-rolled subBlocks)
|
||||
- [ ] Every trigger's `id` matches the convention `{service}_{event_name}`
|
||||
- [ ] Every trigger's `provider` matches the service name used in the handler registry
|
||||
- [ ] `index.ts` barrel exports all triggers
|
||||
|
||||
### Trigger ↔ Provider Alignment (CRITICAL)
|
||||
- [ ] Every trigger ID referenced in `matchEvent` logic exists in `{service}TriggerOptions`
|
||||
- [ ] Event matching logic in the provider correctly maps trigger IDs to service event types
|
||||
- [ ] Event matching logic in `is{Service}EventMatch` (if exists) correctly identifies events per the API docs
|
||||
|
||||
## Step 4: Validate Provider Handler
|
||||
|
||||
### Auth Verification
|
||||
- [ ] `verifyAuth` correctly validates webhook signatures per the service's documentation
|
||||
- [ ] HMAC algorithm matches (SHA-1, SHA-256, SHA-512)
|
||||
- [ ] Signature header name matches the API docs exactly
|
||||
- [ ] Signature format is handled (raw hex, `sha256=` prefix, base64, etc.)
|
||||
- [ ] Uses `safeCompare` for timing-safe comparison (no `===`)
|
||||
- [ ] If `webhookSecret` is required, handler rejects when it's missing (fail-closed)
|
||||
- [ ] Signature is computed over raw body (not parsed JSON)
|
||||
|
||||
### Event Matching
|
||||
- [ ] `matchEvent` returns `boolean` (not `NextResponse` or other values)
|
||||
- [ ] Challenge/verification events are excluded from matching (e.g., `endpoint.url_validation`)
|
||||
- [ ] When `triggerId` is a generic webhook ID, all events pass through
|
||||
- [ ] When `triggerId` is specific, only matching events pass
|
||||
- [ ] Event matching logic uses dynamic `await import()` for trigger utils (avoids circular deps)
|
||||
|
||||
### formatInput (CRITICAL)
|
||||
- [ ] Every key in the `formatInput` return matches a key in the trigger `outputs` schema
|
||||
- [ ] Every key in the trigger `outputs` schema is populated by `formatInput`
|
||||
- [ ] No extra undeclared keys that users can't discover in the UI
|
||||
- [ ] No wrapper objects (`webhook: { ... }`, `{service}: { ... }`)
|
||||
- [ ] Nested output paths exist at the correct depth (e.g., `resource.id` actually has `resource: { id: ... }`)
|
||||
- [ ] `null` is used for missing optional fields (not empty strings or empty objects)
|
||||
- [ ] Returns `{ input: { ... } }` — not a bare object
|
||||
- [ ] Every mapped payload field is backed by official docs or live-verified webhook payloads
|
||||
|
||||
### Idempotency
|
||||
- [ ] `extractIdempotencyId` returns a stable, unique key per delivery
|
||||
- [ ] Uses provider-specific delivery IDs when available (e.g., `X-Request-Id`, `Linear-Delivery`, `svix-id`)
|
||||
- [ ] Falls back to content-based ID (e.g., `${type}:${id}`) when no delivery header exists
|
||||
- [ ] Does NOT include timestamps in the idempotency key (would break dedup on retries)
|
||||
|
||||
### Challenge Handling (if applicable)
|
||||
- [ ] `handleChallenge` correctly implements the service's URL verification handshake
|
||||
- [ ] Returns the expected response format per the API docs
|
||||
- [ ] Env-backed secrets are resolved via `resolveEnvVarsInObject` if needed
|
||||
|
||||
## Step 5: Validate Automatic Subscription Lifecycle
|
||||
|
||||
If the service supports programmatic webhook creation:
|
||||
|
||||
### createSubscription
|
||||
- [ ] Calls the correct API endpoint to create a webhook
|
||||
- [ ] Sends the correct event types/filters
|
||||
- [ ] Passes the notification URL from `getNotificationUrl(ctx.webhook)`
|
||||
- [ ] Returns `{ providerConfigUpdates: { externalId } }` with the external webhook ID
|
||||
- [ ] Throws on failure (orchestration handles rollback)
|
||||
- [ ] Provides user-friendly error messages (401 → "Invalid API Key", etc.)
|
||||
|
||||
### deleteSubscription
|
||||
- [ ] Calls the correct API endpoint to delete the webhook
|
||||
- [ ] Handles 404 gracefully (webhook already deleted)
|
||||
- [ ] Never throws — catches errors and logs non-fatally
|
||||
- [ ] Skips gracefully when `apiKey` or `externalId` is missing
|
||||
|
||||
### Orchestration Isolation
|
||||
- [ ] NO provider-specific logic in `route.ts`, `provider-subscriptions.ts`, or `deploy.ts`
|
||||
- [ ] All subscription logic lives on the handler (`createSubscription`/`deleteSubscription`)
|
||||
|
||||
## Step 6: Validate Registration and Block Wiring
|
||||
|
||||
### Trigger Registry (`triggers/registry.ts`)
|
||||
- [ ] All triggers are imported and registered
|
||||
- [ ] Registry keys match trigger IDs exactly
|
||||
- [ ] No orphaned entries (triggers that don't exist)
|
||||
|
||||
### Provider Handler Registry (`providers/registry.ts`)
|
||||
- [ ] Handler is imported and registered (if handler exists)
|
||||
- [ ] Registry key matches the `provider` field on the trigger configs
|
||||
- [ ] Entries are in alphabetical order
|
||||
|
||||
### Block Wiring (`blocks/blocks/{service}.ts`)
|
||||
- [ ] Block has `triggers.enabled: true`
|
||||
- [ ] `triggers.available` lists all trigger IDs
|
||||
- [ ] All trigger subBlocks are spread into `subBlocks`: `...getTrigger('id').subBlocks`
|
||||
- [ ] No trigger IDs in `triggers.available` that aren't in the registry
|
||||
- [ ] No trigger subBlocks spread that aren't in `triggers.available`
|
||||
|
||||
## Step 7: Validate Security
|
||||
|
||||
- [ ] Webhook secrets are never logged (not even at debug level)
|
||||
- [ ] Auth verification runs before any event processing
|
||||
- [ ] No secret comparison uses `===` (must use `safeCompare` or `crypto.timingSafeEqual`)
|
||||
- [ ] Timestamp/replay protection is reasonable (not too tight for retries, not too loose for security)
|
||||
- [ ] Raw body is used for signature verification (not re-serialized JSON)
|
||||
|
||||
## Step 8: Report and Fix
|
||||
|
||||
### Report Format
|
||||
|
||||
Group findings by severity:
|
||||
|
||||
**Critical** (runtime errors, security issues, or data loss):
|
||||
- Wrong HMAC algorithm or header name
|
||||
- `formatInput` keys don't match trigger `outputs`
|
||||
- Missing `verifyAuth` when the service sends signed webhooks
|
||||
- `matchEvent` returns non-boolean values
|
||||
- Provider-specific logic leaking into shared orchestration files
|
||||
- Trigger IDs mismatch between trigger files, registry, and block
|
||||
- `createSubscription` calling wrong API endpoint
|
||||
- Auth comparison using `===` instead of `safeCompare`
|
||||
|
||||
**Warning** (convention violations or usability issues):
|
||||
- Missing `extractIdempotencyId` when the service provides delivery IDs
|
||||
- Timestamps in idempotency keys (breaks dedup on retries)
|
||||
- Missing challenge handling when the service requires URL verification
|
||||
- Output schema missing fields that `formatInput` returns (undiscoverable data)
|
||||
- Overly tight timestamp skew window that rejects legitimate retries
|
||||
- `matchEvent` not filtering challenge/verification events
|
||||
- Setup instructions missing important steps
|
||||
|
||||
**Suggestion** (minor improvements):
|
||||
- More specific output field descriptions
|
||||
- Additional output fields that could be exposed
|
||||
- Better error messages in `createSubscription`
|
||||
- Logging improvements
|
||||
|
||||
### Fix All Issues
|
||||
|
||||
After reporting, fix every **critical** and **warning** issue. Apply **suggestions** where they don't add unnecessary complexity.
|
||||
|
||||
### Validation Output
|
||||
|
||||
After fixing, confirm:
|
||||
1. `bun run type-check` passes
|
||||
2. Re-read all modified files to verify fixes are correct
|
||||
3. Provider handler tests pass (if they exist): `bun test {service}`
|
||||
4. Any remaining unknown webhook payload schemas were explicitly reported to the user instead of guessed
|
||||
|
||||
## Checklist Summary
|
||||
|
||||
- [ ] Read all trigger files, provider handler, types, registries, and block
|
||||
- [ ] Pulled and read official webhook/API documentation
|
||||
- [ ] Validated trigger definitions: options, instructions, extra fields, outputs
|
||||
- [ ] Validated primary/secondary trigger distinction (`includeDropdown`)
|
||||
- [ ] Validated provider handler: auth, matchEvent, formatInput, idempotency
|
||||
- [ ] Validated output alignment: every `outputs` key ↔ every `formatInput` key
|
||||
- [ ] Validated subscription lifecycle: createSubscription, deleteSubscription, no shared-file edits
|
||||
- [ ] Validated registration: trigger registry, handler registry, block wiring
|
||||
- [ ] Validated security: safe comparison, no secret logging, replay protection
|
||||
- [ ] Reported all issues grouped by severity
|
||||
- [ ] Fixed all critical and warning issues
|
||||
- [ ] `bun run type-check` passes after fixes
|
||||
@@ -0,0 +1,63 @@
|
||||
---
|
||||
name: you-might-not-need-a-callback
|
||||
description: Analyze and fix useCallback anti-patterns in your code
|
||||
argument-hint: "[scope] [fix=true|false]"
|
||||
---
|
||||
|
||||
# You Might Not Need a Callback
|
||||
|
||||
Arguments:
|
||||
- scope: what to analyze (default: your current changes). Examples: "diff to main", "PR #123", "src/components/", "whole codebase"
|
||||
- fix: whether to apply fixes (default: true). Set to false to only propose changes.
|
||||
|
||||
User arguments: $ARGUMENTS
|
||||
|
||||
## References
|
||||
|
||||
Read before analyzing:
|
||||
1. https://react.dev/reference/react/useCallback — official docs on when useCallback is actually needed
|
||||
|
||||
## The one rule that matters
|
||||
|
||||
`useCallback` is only useful when **something observes the reference**. Ask: does anything care if this function gets a new identity on re-render?
|
||||
|
||||
Observers that care about reference stability:
|
||||
- A `useEffect` that lists the function in its deps array
|
||||
- A `useMemo` that lists the function in its deps array
|
||||
- Another `useCallback` that lists the function in its deps array
|
||||
- A child component wrapped in `React.memo` that receives the function as a prop
|
||||
- A custom hook that documents a referential-stability requirement for the callback
|
||||
|
||||
If none of those apply — if the function is only called inline, or passed to a non-memoized child, or assigned to a native element event — the reference is unobserved and `useCallback` adds overhead with zero benefit.
|
||||
|
||||
## Anti-patterns to detect
|
||||
|
||||
1. **No observer tracks the reference**: The function is only called inline in the same component, or passed to a non-memoized child, or used as a native element handler (`<button onClick={fn}>`). Nothing re-runs or bails out based on reference identity. Remove `useCallback`.
|
||||
2. **useCallback with deps that change every render**: If a dep is a plain object/array created inline, or state that changes on every interaction, memoization buys nothing — the function gets a new identity anyway.
|
||||
3. **useCallback on handlers passed only to native elements**: `<button onClick={fn}>` — React never does reference equality on native element props. No benefit.
|
||||
4. **useCallback wrapping functions that return new objects/arrays**: Stable function identity, unstable return value — memoization is at the wrong level. Use `useMemo` on the return value instead, or restructure.
|
||||
5. **useCallback with empty deps when deps are needed**: Stale closure — reads initial values forever. This is a correctness bug, not just a performance issue.
|
||||
6. **Pairing useCallback + React.memo on trivially cheap renders**: If the child renders in < 1ms and re-renders rarely, the memo infrastructure costs more than it saves.
|
||||
7. **useCallback in custom hooks that don't need stable references**: Not every hook return needs to be memoized. Only stabilize callbacks when consumers depend on referential equality.
|
||||
|
||||
## Patterns that ARE correct — do not flag
|
||||
|
||||
- `useCallback` whose result is in a `useEffect` dep array — prevents the effect from re-running on every render
|
||||
- `useCallback` whose result is in a `useMemo` dep array — prevents the memo from recomputing on every render
|
||||
- `useCallback` whose result is a dep of another `useCallback` — stabilises a callback chain
|
||||
- `useCallback` passed to a `React.memo`-wrapped child — the whole point of the pattern
|
||||
- This codebase's ref pattern: `useRef` + callback with empty deps that reads the ref inside — correct, do not flag:
|
||||
|
||||
```tsx
|
||||
const idRef = useRef(id)
|
||||
useEffect(() => { idRef.current = id }, [id])
|
||||
const fetchData = useCallback(async () => {
|
||||
// use idRef.current instead of id
|
||||
}, []) // empty deps because refs are used
|
||||
```
|
||||
|
||||
## Steps
|
||||
|
||||
1. Read the reference above
|
||||
2. Analyze the specified scope for the anti-patterns listed above
|
||||
3. If fix=true, apply the fixes. If fix=false, propose the fixes without applying.
|
||||
@@ -0,0 +1,46 @@
|
||||
---
|
||||
name: you-might-not-need-a-comment
|
||||
description: Analyze and fix redundant or self-explanatory inline comments — remove noise, promote genuine documentation to TSDoc
|
||||
argument-hint: "[scope] [fix=true|false]"
|
||||
---
|
||||
|
||||
# You Might Not Need a Comment
|
||||
|
||||
Arguments:
|
||||
- scope: what to analyze (default: your current changes). Examples: "diff to main", "PR #123", "src/components/", "whole codebase"
|
||||
- fix: whether to apply fixes (default: true). Set to false to only propose changes.
|
||||
|
||||
User arguments: $ARGUMENTS
|
||||
|
||||
## The one rule that matters
|
||||
|
||||
A comment must add information the code cannot express itself. Code says *what* and *how*; a comment earns its place only by explaining *why* — a non-obvious constraint, a workaround, a decision, a gotcha. If deleting the comment loses no information a competent reader wouldn't recover from the code in seconds, delete it.
|
||||
|
||||
This codebase's convention: **TSDoc for documentation, no non-TSDoc comments, no `====` separators.** Genuine documentation belongs in a `/** ... */` block on the declaration; everything that survives as an inline `//` comment must be a real *why*, kept terse.
|
||||
|
||||
## Anti-patterns to detect
|
||||
|
||||
1. **Restates the code**: `// increment counter` above `counter++`, `// return the result` above `return result`, `// loop over items`. Delete.
|
||||
2. **Narrates the obvious from names**: the function is `fetchUserById`, the comment says `// fetches a user by id`. The identifier already said it. Delete.
|
||||
3. **Section-divider / banner comments**: `// ==== Helpers ====`, `// --- state ---`, `// #region`. Against convention. Delete (the code's structure is the structure).
|
||||
4. **Commented-out code**: dead code left as a comment. Delete — git is the history.
|
||||
5. **Redundant type/param echo in prose comments**: `// takes a string and returns a number` when the signature already says so. Delete.
|
||||
6. **Changelog / attribution noise**: `// added by X`, `// TODO(2021): ...` long-stale, `// fix for bug`. Delete unless it encodes a live, actionable constraint.
|
||||
7. **Genuine documentation written as a loose `//` block on a declaration**: a real explanation of what an exported function/type/const is for, but written as stacked `//` lines instead of TSDoc. Convert to a `/** ... */` TSDoc block on the declaration.
|
||||
|
||||
## Patterns that ARE correct — do not flag
|
||||
|
||||
- A `//` comment that explains a **non-obvious why**: a workaround for an upstream bug, an ordering constraint, a perf reason, a spec/edge-case the code can't self-document (`// first-match wins — matches the old find() semantics`).
|
||||
- Existing TSDoc `/** ... */` blocks on declarations — leave them (only tighten if verbose).
|
||||
- `// boundary-raw-fetch:`, `// double-cast-allowed:`, `// boundary-raw-json:`, `// untyped-response:`, `// migration-safe:` and other **machine-read annotations** — these are load-bearing, never touch them.
|
||||
- `// biome-ignore`, `// eslint-disable`, `// @ts-expect-error` and other tooling directives.
|
||||
- `// TODO` / `// FIXME` that point at real, still-open work.
|
||||
|
||||
## Bias
|
||||
|
||||
Prefer **deletion over rewriting**, and **no comment over a comment** when the code is already clear. When a comment is genuine documentation, prefer promoting it to terse TSDoc over leaving a loose `//` block. Never add new comments in this pass — this is a reduction pass. When unsure whether a comment encodes a real *why*, keep it.
|
||||
|
||||
## Steps
|
||||
|
||||
1. Analyze the specified scope for the anti-patterns listed above
|
||||
2. If fix=true, apply the fixes. If fix=false, propose the fixes without applying.
|
||||
@@ -0,0 +1,34 @@
|
||||
---
|
||||
name: you-might-not-need-a-memo
|
||||
description: Analyze and fix useMemo/React.memo anti-patterns in your code
|
||||
argument-hint: "[scope] [fix=true|false]"
|
||||
---
|
||||
|
||||
# You Might Not Need a Memo
|
||||
|
||||
Arguments:
|
||||
- scope: what to analyze (default: your current changes). Examples: "diff to main", "PR #123", "src/components/", "whole codebase"
|
||||
- fix: whether to apply fixes (default: true). Set to false to only propose changes.
|
||||
|
||||
User arguments: $ARGUMENTS
|
||||
|
||||
## References
|
||||
|
||||
Read before analyzing:
|
||||
1. https://overreacted.io/before-you-memo/ — two techniques to avoid memo entirely
|
||||
|
||||
## Anti-patterns to detect
|
||||
|
||||
1. **Wrapping a slow component in React.memo when state can be moved down**: If a component re-renders because of state it doesn't use, move that state into a smaller child component instead of memoizing. The slow component stops re-rendering without memo.
|
||||
2. **Wrapping in React.memo when children can be lifted up**: If a parent owns state that changes frequently, extract the stateful part and pass the expensive subtree as `children`. Children passed as props don't re-render when the parent's state changes.
|
||||
3. **useMemo on cheap computations**: Filtering or mapping a small array, string concatenation, simple arithmetic — these don't need memoization. Only memoize when you've measured a performance problem.
|
||||
4. **useMemo with constantly-changing deps**: If the dependency array changes on every render, useMemo does nothing — it recalculates every time. Fix the deps or remove the memo.
|
||||
5. **useMemo to create objects/arrays passed as props**: Instead of memoizing to prevent child re-renders, consider whether the child even needs referential stability. If the child doesn't use React.memo or pass it to a dep array, the memo is wasted.
|
||||
6. **React.memo on components that always receive new props**: If the parent always passes new objects, arrays, or callbacks, React.memo's shallow comparison always fails. Fix the parent instead of memoizing the child.
|
||||
7. **useMemo for derived state**: If you're computing a value from props or state, just compute it inline during render. React renders are fast. `const fullName = first + ' ' + last` doesn't need useMemo.
|
||||
|
||||
## Steps
|
||||
|
||||
1. Read the reference above to understand the two core techniques (move state down, lift content up)
|
||||
2. Analyze the specified scope for the anti-patterns listed above
|
||||
3. If fix=true, apply the fixes. If fix=false, propose the fixes without applying.
|
||||
@@ -0,0 +1,18 @@
|
||||
---
|
||||
name: you-might-not-need-an-effect
|
||||
description: Analyze and fix useEffect anti-patterns in your code
|
||||
argument-hint: "[scope] [fix=true|false]"
|
||||
---
|
||||
|
||||
# You Might Not Need an Effect
|
||||
|
||||
Arguments:
|
||||
- scope: what to analyze (default: your current changes). Examples: "diff to main", "PR #123", "src/components/", "whole codebase"
|
||||
- fix: whether to apply fixes (default: true). Set to false to only propose changes.
|
||||
|
||||
User arguments: $ARGUMENTS
|
||||
|
||||
Steps:
|
||||
1. Read https://react.dev/learn/you-might-not-need-an-effect to understand the guidelines
|
||||
2. Analyze the specified scope for useEffect anti-patterns
|
||||
3. If fix=true, apply the fixes. If fix=false, propose the fixes without applying.
|
||||
@@ -0,0 +1,39 @@
|
||||
---
|
||||
name: you-might-not-need-state
|
||||
description: Analyze and fix unnecessary useState, derived state, and server-state-in-local-state anti-patterns
|
||||
argument-hint: "[scope] [fix=true|false]"
|
||||
---
|
||||
|
||||
# You Might Not Need State
|
||||
|
||||
Arguments:
|
||||
- scope: what to analyze (default: your current changes). Examples: "diff to main", "PR #123", "src/components/", "whole codebase"
|
||||
- fix: whether to apply fixes (default: true). Set to false to only propose changes.
|
||||
|
||||
User arguments: $ARGUMENTS
|
||||
|
||||
## Context
|
||||
|
||||
This codebase uses React Query for all server state and Zustand for client-only global state. useState should only be used for ephemeral UI concerns (open/closed, hover, local form input). Server data should never be copied into useState or Zustand — React Query is the single source of truth.
|
||||
|
||||
## References
|
||||
|
||||
Read these before analyzing:
|
||||
1. https://react.dev/learn/choosing-the-state-structure — 5 principles for structuring state
|
||||
2. https://tkdodo.eu/blog/dont-over-use-state — never store derived/computed values in state
|
||||
3. https://tkdodo.eu/blog/putting-props-to-use-state — never mirror props into state via useEffect
|
||||
|
||||
## Anti-patterns to detect
|
||||
|
||||
1. **Derived state stored in useState**: If a value can be computed from props, other state, or query data, compute it inline during render instead of storing it in state.
|
||||
2. **Server state copied into useState**: Never `useState` + `useEffect` to sync React Query data into local state. Use query data directly. The only exception is forms where users edit server data.
|
||||
3. **Props mirrored into state**: Never `useState(prop)` + `useEffect(() => setState(prop))`. Use the prop directly, or use a key to reset component state.
|
||||
4. **Chained useEffect state updates**: Never chain Effects that set state to trigger other Effects. Calculate all derived values in the event handler or inline during render.
|
||||
5. **Storing objects when an ID suffices**: Store `selectedId` not a copy of the selected object. Derive the object: `items.find(i => i.id === selectedId)`.
|
||||
6. **State that duplicates Zustand or React Query**: If the data already lives in a store or query cache, don't create a parallel useState.
|
||||
|
||||
## Steps
|
||||
|
||||
1. Read the references above to understand the guidelines
|
||||
2. Analyze the specified scope for the anti-patterns listed above
|
||||
3. If fix=true, apply the fixes. If fix=false, propose the fixes without applying.
|
||||
@@ -0,0 +1,51 @@
|
||||
---
|
||||
name: you-might-not-need-url-state
|
||||
description: Analyze and fix URL/query-param state anti-patterns — manual useSearchParams reads, hand-built query mutations, view-state trapped in useState, and objects in the URL
|
||||
argument-hint: "[scope] [fix=true|false]"
|
||||
---
|
||||
|
||||
# You Might Not Need URL State
|
||||
|
||||
Arguments:
|
||||
- scope: what to analyze (default: your current changes). Examples: "diff to main", "PR #123", "app/workspace/[workspaceId]/tables/", "whole codebase"
|
||||
- fix: whether to apply fixes (default: true). Set to false to only propose changes.
|
||||
|
||||
User arguments: $ARGUMENTS
|
||||
|
||||
## Context
|
||||
|
||||
Shareable client view-state (active tab/panel, filters, search query, sort, pagination, selected-entity id, an open "view" modal/drawer that is a destination) lives in the URL via [`nuqs`](https://nuqs.dev) — driven by a co-located `search-params.ts`, never read via `useSearchParams().get(...)` and never mutated by hand-built query strings. Remote data stays in React Query; high-frequency / large / ephemeral / socket-synced state stays in Zustand; purely local UI stays in `useState`.
|
||||
|
||||
Shared helpers own the two repeated wirings — never hand-roll them inline:
|
||||
- Sort: `createSortParams` from `@/lib/url-state` (in `search-params.ts`) + `useUrlSort` from `@/hooks/use-url-sort` (in the component) — defaulted mode for lists with a fixed default ordering, nullable mode when "no active sort" is distinct from the default column.
|
||||
- Debounced search: `useDebouncedSearchSetter` from `@/hooks/use-debounced-search-setter` (grouped or single-param); settings list search boxes use `useSettingsSearch()` from `settings/components/use-settings-search`. Never write a trimmed value to a param that controls the input — trim on read.
|
||||
|
||||
`.claude/rules/sim-url-state.md` is the source of truth — read it first.
|
||||
|
||||
## References
|
||||
|
||||
Read these before analyzing:
|
||||
1. `.claude/rules/sim-url-state.md` — the decision framework, conventions, debounced-input pattern, sort convention, selected-entity deep-link pattern, and the workflow-editor carve-out
|
||||
2. https://nuqs.dev/docs/parsers — parsers (`parseAsString`/`parseAsInteger`/`parseAsBoolean`/`parseAsStringLiteral`/`parseAsArrayOf`/`createParser`)
|
||||
3. https://nuqs.dev/docs/options — `withDefault`, `history`, `shallow`, `clearOnDefault`
|
||||
4. https://nuqs.dev/docs/server-side — `createSearchParamsCache` for server reads
|
||||
|
||||
## Anti-patterns to detect
|
||||
|
||||
1. **Manual param reads for state**: `useSearchParams().get(...)` or `new URLSearchParams(window.location.search)` used to *read* view-state. Replace with `useQueryState`/`useQueryStates` bound to a `search-params.ts`. (Read-once auth/invite/redirect tokens — `token`, `callbackUrl`, `redirect`, `error`, `invite_flow`, `code` — are NOT view-state; leave them on `useSearchParams`.)
|
||||
2. **Hand-built query mutation**: constructing a query string + `router.replace`/`router.push` to change a param on the current path. Use a nuqs setter. (A `router.push` that changes the route *path* is fine; an outbound `new URLSearchParams` building an `href`/`window.open`/download/API URL is fine.)
|
||||
3. **`window.history.replaceState`/`pushState`** to mutate a param.
|
||||
4. **URL state duplicated into a store/useState + synced with an effect** (or a `popstate` listener). The URL is the single source of truth; derive from it, don't mirror it.
|
||||
5. **Objects in the URL**: serializing a `TableDefinition`/`SkillDefinition`/etc. Store the id and derive the object from the loaded list (`items.find(i => i.id === id)`).
|
||||
6. **High-frequency / large state in the URL**: cursor, pan/zoom, un-debounced keystrokes, big JSON blobs. Debounce text search via `useDebouncedSearchSetter` (never a local `useState` mirror + reconcile effect, and never inline `limitUrlUpdates` wiring); keep canvas/presence/resize state in Zustand.
|
||||
7. **Shareable view-state trapped in `useState`**: a tab/filter/sort/pagination/selected-entity that should be a link but lives in local state. Migrate it to the URL.
|
||||
8. **Missing Suspense boundary**: a component newly calling `useQueryState`/`useQueryStates` whose page entry has no `<Suspense>` wrapper (Next.js requires it for `useSearchParams`). Add one with a real-chrome fallback.
|
||||
9. **`import { z }` for param validation in client code**: use nuqs parsers instead.
|
||||
10. **Re-implemented shared wiring**: a hand-rolled `SORT_DIRECTIONS`/default-sort constants/`activeSort` derivation instead of `createSortParams` + `useUrlSort`, or an inline debounced-search setter instead of `useDebouncedSearchSetter`/`useSettingsSearch`.
|
||||
|
||||
## Steps
|
||||
|
||||
1. Read `.claude/rules/sim-url-state.md` and the nuqs docs above to understand the guidelines
|
||||
2. Analyze the specified scope for the anti-patterns listed above
|
||||
3. For each finding, decide the correct home using the decision table — do not force URL state onto ephemeral/high-frequency/socket-synced state
|
||||
4. If fix=true, apply the fixes (co-locate a `search-params.ts`, wire `useQueryState(s)` — sort via `createSortParams` + `useUrlSort`, search via `useDebouncedSearchSetter` — add the Suspense boundary, delete the replaced state + sync effects). If fix=false, propose the fixes without applying.
|
||||
@@ -0,0 +1,60 @@
|
||||
---
|
||||
description: Gate a block's visibility — ship an unreleased block as a preview (hidden until revealed via AppConfig/env), reveal it to admins/orgs, GA it, or kill-switch a shipped block
|
||||
argument-hint: <block-type>
|
||||
---
|
||||
|
||||
# Add Block Preview Skill
|
||||
|
||||
You manage **block visibility gating** in Sim — hiding blocks from every discovery surface (toolbar, cmd+K search, copilot @-mentions, agent tool picker, mothership VFS/metadata/tools, Access Control list, public docs/catalog) while **never** gating execution of already-placed instances.
|
||||
|
||||
## The model
|
||||
|
||||
Three levers, evaluated in `apps/sim/lib/core/config/block-visibility.ts` and folded into the registry accessors (`apps/sim/blocks/registry.ts`):
|
||||
|
||||
1. **`preview: true`** on the `BlockConfig` (static, in code) — the block is default-hidden EVERYWHERE (hosted, self-hosted, dev, SSR) until revealed. Fail-closed.
|
||||
2. **The hosted `block-visibility` AppConfig document** — per-block rule keyed by the existing block type:
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"<block-type>": {
|
||||
"enabled": false, // required. true = GA (visible to everyone)
|
||||
"orgIds": ["org_..."], // optional allowlist clauses (any match reveals)
|
||||
"userIds": ["user_..."],
|
||||
"adminEnabled": true // platform admins (user.role === 'admin')
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
3. **`PREVIEW_BLOCKS` env** (comma-separated block types) — the off-AppConfig reveal path for self-hosters and local dev.
|
||||
|
||||
A revealed block that is not globally GA (`enabled !== true`, or env-revealed) renders with a **" (Preview)"** name suffix on discovery surfaces. `getBlock()` stays pure, so placed instances keep their canonical name and always execute.
|
||||
|
||||
## Lifecycle of a preview block
|
||||
|
||||
1. **Author** the block normally (`/add-block` etc.) and set `preview: true` on its `BlockConfig`. **Ship no `BlockMeta` and no docs until GA** — `check-block-registry` deliberately skips preview blocks in meta coverage, and `generate-docs` skips them at every gate.
|
||||
2. **Local dev:** set `PREVIEW_BLOCKS=<block-type>` in your env to see it (with the suffix).
|
||||
3. **Merge/deploy.** The block's code is live everywhere but visible nowhere — no AppConfig rule exists and self-hosters have no env entry.
|
||||
4. **Hosted preview:** add a rule to the `block-visibility` AppConfig document and start a deployment (no code deploy):
|
||||
- Admins only: `{ "enabled": false, "adminEnabled": true }`
|
||||
- Design-partner org: `{ "enabled": false, "orgIds": ["org_123"] }`
|
||||
- GA via config (code cleanup pending): `{ "enabled": true }` — suffix disappears everywhere within ~30s (AppConfig TTL) + client refetch.
|
||||
|
||||
Same runbook as `feature-flags`: edit the hosted document, `aws appconfig start-deployment` with the `sim-<env>-fast` strategy (see the infra README).
|
||||
5. **GA cleanup:** delete `preview: true` from the block (now visible to self-hosters on their next upgrade), add its `BlockMeta` + regen docs, and drop the AppConfig entry. For a v2 upgrade, this is also when v1 gets `hideFromToolbar: true` (the superseded-version paradigm).
|
||||
|
||||
## Kill switch (shipped blocks)
|
||||
|
||||
To pull an already-GA block from discovery surfaces on hosted (incident, deprecation): add `{ "<block-type>": { "enabled": false } }` to the document. Allowlist clauses can carve out exceptions. **Execution is NOT stopped** — workflows already using the block keep running; the kill switch only prevents new placement/discovery.
|
||||
|
||||
## Invariants (do not violate)
|
||||
|
||||
- **Execution is never gated.** The executor, serializer, drop-naming, and `isBlockTypeAccessControlExempt` resolve via pure `getBlock`. Do not add visibility checks to execution paths.
|
||||
- **Clone-not-remove:** gated blocks stay in `getAllBlocks()` output as clones with `hideFromToolbar: true` — `.find`-by-type consumers rely on this. Never filter them out.
|
||||
- **Keys are registry block types.** Never `custom_block_*` (parse drops them — custom blocks have their own enabled/disabled lifecycle).
|
||||
- **The shared hidden-predicate is `isHiddenUnder`** (`apps/sim/blocks/visibility/context.ts`). Never restate the preview/disabled rule inline at a new consumer.
|
||||
- **Process-global caches stay ungated.** `getStaticComponentFiles` (VFS) and `getExposedIntegrationTools` build the ungated universe; per-viewer filtering happens at stamp/consumer time. Never move gating into a shared builder.
|
||||
- Gating is **surface hiding, not secrecy** — the full config ships in the client JS bundle. Anything truly secret cannot be a registered block.
|
||||
|
||||
## Tests
|
||||
|
||||
Evaluation semantics: `apps/sim/lib/core/config/block-visibility.test.ts`. Registry projection: `apps/sim/blocks/visibility/visibility.test.ts`. When gating behavior changes, extend those — mock `isPlatformAdmin` for the admin clause; use the local `withAppConfig` harness.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,159 @@
|
||||
---
|
||||
description: Add a new table column type to Sim — registry entry, icon, storage shape, coercion, and the behavioral hooks the grid and API read. Use when adding a value kind under `apps/sim/lib/table/column-types/`.
|
||||
argument-hint: <type-name>
|
||||
---
|
||||
|
||||
# Adding a Table Column Type
|
||||
|
||||
A column type is **one file** in `apps/sim/lib/table/column-types/` plus a registry entry. Everything that varies per type — label, icon, storage cast, coercion, validation, conversion compatibility, formatting, editor, filter operators — lives on that one object, so no consumer needs editing.
|
||||
|
||||
This was not always true: adding `currency` originally took ~40 edits across 32 `switch` arms and 26 UI branches, each of which failed **silently** when missed. The registry exists to make that impossible, so the rule is absolute: **if you find yourself adding a `case 'yourtype':` anywhere outside `column-types/`, the registry is missing a field. Add the field instead.**
|
||||
|
||||
## Hard Rule: the compiler tells you what to do
|
||||
|
||||
Do **not** hunt for places to edit. Add your type to the `ColumnType` union first and let `tsc` produce the list:
|
||||
|
||||
```bash
|
||||
cd apps/sim && bun run type-check
|
||||
```
|
||||
|
||||
You will get two errors, naming `column-types/registry.ts` and `column-types/registry.server.ts`. Register in both.
|
||||
|
||||
If your type owns metadata, adding its key to `TYPE_SPECIFIC_COLUMN_KEYS` produces two more legitimate errors — `FOREIGN_METADATA_VERB` in `validation.ts` (a `Record` over those keys) and the key's absence from `ColumnDefinition`. Those are the gate working, not sites to "fix".
|
||||
|
||||
Any error beyond those four is a site reading a hardcoded type list that should read the registry — fix that site, don't work around it.
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
apps/sim/lib/table/column-types/
|
||||
├── types.ts # ColumnTypeDefinition — the contract you implement
|
||||
├── types.server.ts # ColumnTypeServerDefinition — cell migrations only
|
||||
├── registry.ts # Record<ColumnType, …> ← client-safe, the gate
|
||||
├── registry.server.ts # Record<ColumnType, …> ← adds migrations (drizzle)
|
||||
├── index.ts # barrel + accessors (columnTypeOf, columnTypeById, …)
|
||||
└── {type}.ts # one file per type — what you write
|
||||
```
|
||||
|
||||
## Step 1: Pick the storage shape
|
||||
|
||||
Decide what a cell literally holds in `user_table_rows.data` (JSONB). This drives almost everything else:
|
||||
|
||||
| Storage | `jsonbCast` | Notes |
|
||||
|---------|-------------|-------|
|
||||
| number | `'numeric'` | Filters/sorts compare numerically. `currency` does this. |
|
||||
| ISO string | `'timestamptz'` | `date` does this. |
|
||||
| string / bool / object | `null` | Text comparison is correct. |
|
||||
|
||||
**Prefer an existing primitive over a new shape.** `currency` stores a plain number and keeps its ISO code as *display metadata* — which is why filtering, sorting, uniqueness, and CSV export all reuse the numeric paths untouched, and why re-denominating a column rewrites zero rows.
|
||||
|
||||
## Step 2: Add the icon
|
||||
|
||||
Create `packages/emcn/src/icons/type-{name}.tsx`, copying the geometry conventions of its siblings exactly:
|
||||
|
||||
```tsx
|
||||
import type { SVGProps } from 'react'
|
||||
|
||||
/**
|
||||
* Type {name} icon component - {what the glyph is} for {name} columns
|
||||
* @param props - SVG properties including className, fill, etc.
|
||||
*/
|
||||
export function Type{Pascal}(props: SVGProps<SVGSVGElement>) {
|
||||
return (
|
||||
<svg
|
||||
width='24'
|
||||
height='24'
|
||||
viewBox='-1.75 -1.5 24 24'
|
||||
fill='none'
|
||||
stroke='currentColor'
|
||||
strokeWidth='1.55'
|
||||
strokeLinecap='round'
|
||||
strokeLinejoin='round'
|
||||
xmlns='http://www.w3.org/2000/svg'
|
||||
aria-hidden='true'
|
||||
{...props}
|
||||
>
|
||||
<path d='…' />
|
||||
</svg>
|
||||
)
|
||||
}
|
||||
```
|
||||
|
||||
- `viewBox='-1.75 -1.5 24 24'` is the **`type-*` family** value, not the set-wide default. Match the family.
|
||||
- Center the glyph on the viewBox's optical center (**y = 10.5**, **x = 10.25**) — every sibling does, and a few tenths off is visible at `size-[14px]`.
|
||||
- Export alphabetically **by component name** in `packages/emcn/src/icons/index.ts`.
|
||||
|
||||
## Step 3: Write the type file
|
||||
|
||||
`apps/sim/lib/table/column-types/{name}.ts`. Copy the closest existing type and change what differs. Every field is required by the interface, so the compiler enumerates them for you — read the TSDoc in `types.ts` rather than guessing.
|
||||
|
||||
The three that are easy to get wrong:
|
||||
|
||||
- **`coerce`** is the *single* write-path implementation. The server runs it before persisting **and** the grid runs it to fill the optimistic cache. Accept every shape the value legitimately arrives in (paste, CSV, tool write), because rejecting means the cell is nulled.
|
||||
- **`isCompatibleWith`** gates type conversion and must read the value **exactly as `coerce` will**, or a conversion will pass its check and then null the cell.
|
||||
- **`ownedMetadata`** lists the `ColumnDefinition` keys your type owns. Anything you add must also be added to `TYPE_SPECIFIC_COLUMN_KEYS` in `types.ts` and given a phrase in `FOREIGN_METADATA_VERB` in `validation.ts` — both are `Record`-typed, so the compiler will tell you.
|
||||
|
||||
## Step 4: Register
|
||||
|
||||
Add the entry to `COLUMN_TYPE_REGISTRY` in `registry.ts` **and** `COLUMN_TYPE_SERVER_REGISTRY` in `registry.server.ts`.
|
||||
|
||||
`COLUMN_TYPES` is declared in `types.ts` (not derived from the registry — the registry is annotated `Record<ColumnType, …>` against it, which is the gate). `constants.ts` re-exports it, so `columnTypeSchema = z.enum(COLUMN_TYPES)` picks your type up with no edit. **Type-specific metadata does not** — see the next step.
|
||||
|
||||
## Step 5: Migrations (only if the stored bytes change)
|
||||
|
||||
If converting an existing column **to** your type must rewrite cells, add `migrateCellsTo` in `registry.server.ts`; if converting **away** must rewrite them, add `migrateCellsFrom`.
|
||||
|
||||
This is load-bearing, not cosmetic: filters and sorts apply `jsonbCast` to whatever is stored, so leaving a non-castable string behind makes **every query on that column fail** — not merely render oddly.
|
||||
|
||||
Prefer set-based SQL. When the transform genuinely needs JS (`currency`'s separator disambiguation), compute the values during the compatibility scan and pass them through `resolved`, then apply them in one batched statement.
|
||||
|
||||
## Naming Convention
|
||||
|
||||
- Type id: lowercase, singular — `currency`, not `Currency` or `currencies`
|
||||
- File: `column-types/{id}.ts`, export `const {id}ColumnType`
|
||||
- Icon: `type-{kebab}.tsx`, export `Type{Pascal}`
|
||||
|
||||
## Watch out
|
||||
|
||||
- **Import cycles.** `column-types/select.ts` imports `select-values.ts`, so `select-values.ts` must **not** import the registry — that closes a cycle and fails at module init. Inside a type's own helper module the string literal is the implementation, not a config leak.
|
||||
- **The client-safe boundary.** `registry.ts` and everything it imports must stay free of `@sim/db`, `drizzle-orm`, and `next/server` — the tables grid imports it directly. A React icon is fine (it's a component *reference*, never called server-side). Only `registry.server.ts` may touch drizzle.
|
||||
- **Don't re-export the registry from `@/lib/table`.** 44 server modules import that barrel; routing this through it pulls `@sim/emcn/icons` into all of them. Deep-import `@/lib/table/column-types`.
|
||||
- **`import.ts`'s `coerceValue` is a SECOND write path and is not opt-in.** Importing into a column of your type always hits it, and its `default` arm silently `String(value)`s — so a missing `case` stores text in a column whose `jsonbCast` is numeric, and then every filter and sort on that column errors in Postgres. Add a `case`, even though the switch compiles without one. (It is deliberately separate from the registry's `coerce`: an import wants an unparseable value to survive as its raw string so the row error can name it.)
|
||||
- **CSV inference** is an ordered heuristic in `import.ts`, deliberately not registry-driven. A new type is not inferred from a CSV unless you extend `inferColumnType` — usually you should not, since inference cannot supply configuration (an option set, a currency code).
|
||||
|
||||
## If your type owns metadata, read this
|
||||
|
||||
Registering the *type* is compiler-enforced. Registering its *metadata* is not, and that is where the remaining manual work lives. A key like `precision` has to be added in each of these, none of which will fail to compile if you forget:
|
||||
|
||||
| Where | What happens if you forget |
|
||||
|---|---|
|
||||
| `lib/table/types.ts` `ColumnDefinition` | (this one DOES fail — the ownership loop indexes it) |
|
||||
| `column-types/types.ts` `TYPE_SPECIFIC_COLUMN_KEYS` | it is never stripped on conversion, and poisons the target type |
|
||||
| `lib/api/contracts/tables.ts` — the schema slot in all three column schemas, plus `refineColumnOptions` | zod strips it at the boundary; silently never saved |
|
||||
| `columns/service.ts` `addTableColumn` param type | callers cannot pass it |
|
||||
| A metadata-only update path (`updateColumnCurrency` is the model) + a branch in both column routes + the copilot tool | changing it on an existing column is a silent 200 no-op |
|
||||
| `column-config-sidebar.tsx` | no UI to set it |
|
||||
| `table-grid.tsx` delete-column undo + `use-table-undo.ts` restore | undo silently resets it to the default |
|
||||
|
||||
`normalizeColumn`, `buildConvertedColumn`, and the undo snapshot read `TYPE_SPECIFIC_COLUMN_KEYS` generically, so those three are already zero-edit.
|
||||
|
||||
**Known gap:** the metadata-only update path is ~6 near-identical copies (service + 2 routes + copilot). A `metadataUpdate` descriptor on `ColumnTypeServerDefinition` would collapse them; until that exists, copy `currency`'s.
|
||||
|
||||
## Checklist Before Finishing
|
||||
|
||||
- [ ] Added to the `ColumnType` union in `column-types/types.ts`
|
||||
- [ ] `column-types/{id}.ts` created, every interface field filled in
|
||||
- [ ] Registered in **both** `registry.ts` and `registry.server.ts`
|
||||
- [ ] Icon added, centered on the family's optical center, exported alphabetically
|
||||
- [ ] `migrateCellsTo` / `migrateCellsFrom` added if the stored bytes change
|
||||
- [ ] New metadata keys added to `TYPE_SPECIFIC_COLUMN_KEYS` + `FOREIGN_METADATA_VERB`
|
||||
- [ ] Unit tests for `coerce` / `isCompatibleWith` round-trips, verified to fail without the code
|
||||
- [ ] Docs row added to `apps/docs/content/docs/en/tables/index.mdx`
|
||||
|
||||
## Final Validation (Required)
|
||||
|
||||
1. **`cd apps/sim && bun run type-check`** — must be clean. If any file *outside* `column-types/` errors, that file has a hardcoded type list; fix it to read the registry.
|
||||
2. **Grep for leaks** — `grep -rnE "(===|!==) '{id}'|case '{id}':" apps/sim --include='*.ts' --include='*.tsx' | grep -v column-types/`. (All three forms: a plain `!==` and a `case` are how half of `currency`'s real branches are written.) Hits are expected; judge each. A hit is fine when it mounts a specific React component or encodes a genuinely one-off behavior (`json`'s mono textarea, `date`'s timezone-aware parsing). A hit is a **leak** when it restates something the registry could answer — an icon, a label, a colour, an operator set, a cast, a coercion. Leaks get a registry field, not a new branch.
|
||||
3. **Run the suite** — `bunx vitest run lib/table 'app/workspace/[workspaceId]/tables' lib/api app/api/table app/api/v1 lib/copilot/tools/server/table`. Existing tests must pass **unchanged**; needing to edit one means you changed behavior for the other types.
|
||||
4. **`bun run lint:check`, `bun run check:api-validation`, `bun run check:client-boundary`** from the repo root.
|
||||
5. **Exercise it in the running app** on a table with one column of every type: create, edit inline / in the expanded popover / in the row modal, paste from a spreadsheet, filter, sort, convert to and from other types, export CSV, undo a column delete.
|
||||
@@ -0,0 +1,625 @@
|
||||
---
|
||||
description: Add or update a Sim knowledge base connector for syncing documents from an external source, including auth mode, config fields, pagination, document mapping, tags, and registry wiring. Use when working in `apps/sim/connectors/{service}/` or adding a new external document source.
|
||||
argument-hint: <service-name> [api-docs-url]
|
||||
---
|
||||
|
||||
# Add Connector Skill
|
||||
|
||||
You are an expert at adding knowledge base connectors to Sim. A connector syncs documents from an external source (Confluence, Google Drive, Notion, etc.) into a knowledge base.
|
||||
|
||||
## Your Task
|
||||
|
||||
When the user asks you to create a connector:
|
||||
1. Use Context7 or WebFetch to read the service's API documentation
|
||||
2. Determine the auth mode: **OAuth** (if Sim already has an OAuth provider for the service) or **API key** (if the service uses API key / Bearer token auth)
|
||||
3. Create the connector directory: a client-safe `meta.ts` (declarative metadata) plus the runtime module that spreads it
|
||||
4. Register it in BOTH the server registry and the client-safe meta registry
|
||||
|
||||
## Hard Rule: No Guessed Response Or Document Schemas
|
||||
|
||||
If the service docs do not clearly show the document list response, document fetch response, pagination shape, or metadata fields, you MUST tell the user instead of guessing.
|
||||
|
||||
- Do NOT invent document fields
|
||||
- Do NOT guess pagination cursors or next-page fields
|
||||
- Do NOT infer metadata/tag mappings from unrelated endpoints
|
||||
- Do NOT fabricate `ExternalDocument` content structure from partial docs
|
||||
|
||||
If the source schema is unknown, do one of these instead:
|
||||
1. Ask the user for sample API responses
|
||||
2. Ask the user for test credentials so you can verify live payloads
|
||||
3. Implement only the documented parts of the connector
|
||||
4. Leave the connector incomplete and explicitly say which fields remain unknown
|
||||
|
||||
## Directory Structure
|
||||
|
||||
Each connector is split into a client-safe metadata file and a server-only runtime file. This mirrors the `XBlockMeta` / `BLOCK_META_REGISTRY` split in `apps/sim/blocks` — client components (the knowledge UI) only need the metadata (icon, name, auth, config fields), so the runtime functions (which pull server-only helpers like `input-validation.server` → `undici` → `node:net`) must stay out of the client bundle.
|
||||
|
||||
Create files in `apps/sim/connectors/{service}/`:
|
||||
```
|
||||
connectors/{service}/
|
||||
├── index.ts # Barrel export (re-exports the runtime connector)
|
||||
├── meta.ts # ConnectorMeta — client-safe declarative metadata
|
||||
└── {service}.ts # ConnectorConfig — spreads the meta + adds runtime functions
|
||||
```
|
||||
|
||||
- `meta.ts` exports `{service}ConnectorMeta: ConnectorMeta`. It imports ONLY the icon from `@/components/icons`, `import type { ConnectorMeta } from '@/connectors/types'`, and any pure-data constants. It must NEVER import server/runtime code.
|
||||
- `{service}.ts` exports `{service}Connector: ConnectorConfig`. It imports the meta via `import { {service}ConnectorMeta } from '@/connectors/{service}/meta'`, spreads it as the first property, and holds the runtime functions (which may import server-only helpers like `@/lib/knowledge/documents/utils`).
|
||||
|
||||
## Authentication
|
||||
|
||||
Connectors use a discriminated union for auth config (`ConnectorAuthConfig` in `connectors/types.ts`):
|
||||
|
||||
```typescript
|
||||
type ConnectorAuthConfig =
|
||||
| { mode: 'oauth'; provider: OAuthService; requiredScopes?: string[] }
|
||||
| { mode: 'apiKey'; label?: string; placeholder?: string }
|
||||
```
|
||||
|
||||
### OAuth mode
|
||||
For services with existing OAuth providers in `apps/sim/lib/oauth/types.ts`. The `provider` must match an `OAuthService`. The modal shows a credential picker and handles token refresh automatically.
|
||||
|
||||
### API key mode
|
||||
For services that use API key / Bearer token auth. The modal shows a password input with the configured `label` and `placeholder`. The API key is encrypted at rest using AES-256-GCM and stored in a dedicated `encryptedApiKey` column on the connector record. The sync engine decrypts it automatically — connectors receive the raw access token in `listDocuments`, `getDocument`, and `validateConfig`.
|
||||
|
||||
## Connector Structure (meta.ts + runtime)
|
||||
|
||||
The declarative metadata lives in `meta.ts` (`ConnectorMeta`). The runtime functions live in `{service}.ts` (`ConnectorConfig`), which spreads the meta as its first property.
|
||||
|
||||
### `meta.ts` — client-safe metadata
|
||||
|
||||
```typescript
|
||||
import { {Service}Icon } from '@/components/icons'
|
||||
import type { ConnectorMeta } from '@/connectors/types'
|
||||
|
||||
export const {service}ConnectorMeta: ConnectorMeta = {
|
||||
id: '{service}',
|
||||
name: '{Service}',
|
||||
description: 'Sync documents from {Service} into your knowledge base',
|
||||
version: '1.0.0',
|
||||
icon: {Service}Icon,
|
||||
|
||||
auth: {
|
||||
mode: 'oauth',
|
||||
provider: '{service}', // Must match OAuthService in lib/oauth/types.ts
|
||||
requiredScopes: ['read:...'],
|
||||
},
|
||||
|
||||
configFields: [
|
||||
// Rendered dynamically by the add-connector modal UI
|
||||
// Supports 'short-input', 'dropdown', and 'selector' types — see ConfigField Types below
|
||||
],
|
||||
|
||||
// Optional: tag definitions are metadata too — declare them here
|
||||
// tagDefinitions: [ ... ],
|
||||
}
|
||||
```
|
||||
|
||||
Keep `meta.ts` free of any server/runtime import. Only the icon, the `ConnectorMeta` type, and pure-data constants belong here.
|
||||
|
||||
### `{service}.ts` — runtime (OAuth example)
|
||||
|
||||
```typescript
|
||||
import { createLogger } from '@sim/logger'
|
||||
import { fetchWithRetry } from '@/lib/knowledge/documents/utils'
|
||||
import { {service}ConnectorMeta } from '@/connectors/{service}/meta'
|
||||
import type { ConnectorConfig, ExternalDocument, ExternalDocumentList } from '@/connectors/types'
|
||||
|
||||
const logger = createLogger('{Service}Connector')
|
||||
|
||||
export const {service}Connector: ConnectorConfig = {
|
||||
...{service}ConnectorMeta,
|
||||
|
||||
listDocuments: async (accessToken, sourceConfig, cursor) => {
|
||||
// Return metadata stubs with contentDeferred: true (if per-doc content fetch needed)
|
||||
// Or full documents with content (if list API returns content inline)
|
||||
// Return { documents: ExternalDocument[], nextCursor?, hasMore }
|
||||
},
|
||||
|
||||
getDocument: async (accessToken, sourceConfig, externalId) => {
|
||||
// Fetch full content for a single document
|
||||
// Return ExternalDocument with contentDeferred: false, or null
|
||||
},
|
||||
|
||||
validateConfig: async (accessToken, sourceConfig) => {
|
||||
// Return { valid: true } or { valid: false, error: 'message' }
|
||||
},
|
||||
|
||||
// Optional: map source metadata to semantic tag keys (translated to slots by sync engine)
|
||||
mapTags: (metadata) => {
|
||||
// Return Record<string, unknown> with keys matching tagDefinitions[].id
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
Only map fields in `listDocuments`, `getDocument`, `validateConfig`, and `mapTags` when the source payload shape is documented or live-verified. If not, tell the user and stop rather than guessing.
|
||||
|
||||
### API key connector example
|
||||
|
||||
The split is identical — `auth` lives in `meta.ts`, runtime functions in `{service}.ts`.
|
||||
|
||||
```typescript
|
||||
// meta.ts
|
||||
export const {service}ConnectorMeta: ConnectorMeta = {
|
||||
id: '{service}',
|
||||
name: '{Service}',
|
||||
description: 'Sync documents from {Service} into your knowledge base',
|
||||
version: '1.0.0',
|
||||
icon: {Service}Icon,
|
||||
|
||||
auth: {
|
||||
mode: 'apiKey',
|
||||
label: 'API Key', // Shown above the input field
|
||||
placeholder: 'Enter your {Service} API key', // Input placeholder
|
||||
},
|
||||
|
||||
configFields: [ /* ... */ ],
|
||||
}
|
||||
|
||||
// {service}.ts
|
||||
export const {service}Connector: ConnectorConfig = {
|
||||
...{service}ConnectorMeta,
|
||||
listDocuments: async (accessToken, sourceConfig, cursor) => { /* ... */ },
|
||||
getDocument: async (accessToken, sourceConfig, externalId) => { /* ... */ },
|
||||
validateConfig: async (accessToken, sourceConfig) => { /* ... */ },
|
||||
}
|
||||
```
|
||||
|
||||
## ConfigField Types
|
||||
|
||||
The add-connector modal renders these automatically — no custom UI needed.
|
||||
|
||||
Three field types are supported: `short-input`, `dropdown`, and `selector`.
|
||||
|
||||
```typescript
|
||||
// Text input
|
||||
{
|
||||
id: 'domain',
|
||||
title: 'Domain',
|
||||
type: 'short-input',
|
||||
placeholder: 'yoursite.example.com',
|
||||
required: true,
|
||||
}
|
||||
|
||||
// Dropdown (static options)
|
||||
{
|
||||
id: 'contentType',
|
||||
title: 'Content Type',
|
||||
type: 'dropdown',
|
||||
required: false,
|
||||
options: [
|
||||
{ label: 'Pages only', id: 'page' },
|
||||
{ label: 'Blog posts only', id: 'blogpost' },
|
||||
{ label: 'All content', id: 'all' },
|
||||
],
|
||||
}
|
||||
```
|
||||
|
||||
## Dynamic Selectors (Canonical Pairs)
|
||||
|
||||
Use `type: 'selector'` to fetch options dynamically from the existing selector registry (`hooks/selectors/registry.ts`). Selectors are always paired with a manual fallback input using the **canonical pair** pattern — a `selector` field (basic mode) and a `short-input` field (advanced mode) linked by `canonicalParamId`.
|
||||
|
||||
The user sees a toggle button (ArrowLeftRight) to switch between the selector dropdown and manual text input. On submit, the modal resolves each canonical pair to the active mode's value, keyed by `canonicalParamId`.
|
||||
|
||||
### Rules
|
||||
|
||||
1. **Every selector field MUST have a canonical pair** — a corresponding `short-input` (or `dropdown`) field with the same `canonicalParamId` and `mode: 'advanced'`.
|
||||
2. **`required` must be set identically on both fields** in a pair. If the selector is required, the manual input must also be required.
|
||||
3. **`canonicalParamId` must match the key the connector expects in `sourceConfig`** (e.g. `baseId`, `channel`, `teamId`). The advanced field's `id` should typically match `canonicalParamId`.
|
||||
4. **`dependsOn` references the selector field's `id`**, not the `canonicalParamId`. The modal propagates dependency clearing across canonical siblings automatically — changing either field in a parent pair clears dependent children.
|
||||
|
||||
### Selector canonical pair example (Airtable base → table cascade)
|
||||
|
||||
```typescript
|
||||
configFields: [
|
||||
// Base: selector (basic) + manual (advanced)
|
||||
{
|
||||
id: 'baseSelector',
|
||||
title: 'Base',
|
||||
type: 'selector',
|
||||
selectorKey: 'airtable.bases', // Must exist in hooks/selectors/registry.ts
|
||||
canonicalParamId: 'baseId',
|
||||
mode: 'basic',
|
||||
placeholder: 'Select a base',
|
||||
required: true,
|
||||
},
|
||||
{
|
||||
id: 'baseId',
|
||||
title: 'Base ID',
|
||||
type: 'short-input',
|
||||
canonicalParamId: 'baseId',
|
||||
mode: 'advanced',
|
||||
placeholder: 'e.g. appXXXXXXXXXXXXXX',
|
||||
required: true,
|
||||
},
|
||||
// Table: selector depends on base (basic) + manual (advanced)
|
||||
{
|
||||
id: 'tableSelector',
|
||||
title: 'Table',
|
||||
type: 'selector',
|
||||
selectorKey: 'airtable.tables',
|
||||
canonicalParamId: 'tableIdOrName',
|
||||
mode: 'basic',
|
||||
dependsOn: ['baseSelector'], // References the selector field ID
|
||||
placeholder: 'Select a table',
|
||||
required: true,
|
||||
},
|
||||
{
|
||||
id: 'tableIdOrName',
|
||||
title: 'Table Name or ID',
|
||||
type: 'short-input',
|
||||
canonicalParamId: 'tableIdOrName',
|
||||
mode: 'advanced',
|
||||
placeholder: 'e.g. Tasks',
|
||||
required: true,
|
||||
},
|
||||
// Non-selector fields stay as-is
|
||||
{ id: 'maxRecords', title: 'Max Records', type: 'short-input', ... },
|
||||
]
|
||||
```
|
||||
|
||||
### Selector with domain dependency (Jira/Confluence pattern)
|
||||
|
||||
When a selector depends on a plain `short-input` field (no canonical pair), `dependsOn` references that field's `id` directly. The `domain` field's value maps to `SelectorContext.domain` automatically via `SELECTOR_CONTEXT_FIELDS`.
|
||||
|
||||
```typescript
|
||||
configFields: [
|
||||
{
|
||||
id: 'domain',
|
||||
title: 'Jira Domain',
|
||||
type: 'short-input',
|
||||
placeholder: 'yoursite.atlassian.net',
|
||||
required: true,
|
||||
},
|
||||
{
|
||||
id: 'projectSelector',
|
||||
title: 'Project',
|
||||
type: 'selector',
|
||||
selectorKey: 'jira.projects',
|
||||
canonicalParamId: 'projectKey',
|
||||
mode: 'basic',
|
||||
dependsOn: ['domain'],
|
||||
placeholder: 'Select a project',
|
||||
required: true,
|
||||
},
|
||||
{
|
||||
id: 'projectKey',
|
||||
title: 'Project Key',
|
||||
type: 'short-input',
|
||||
canonicalParamId: 'projectKey',
|
||||
mode: 'advanced',
|
||||
placeholder: 'e.g. ENG, PROJ',
|
||||
required: true,
|
||||
},
|
||||
]
|
||||
```
|
||||
|
||||
### How `dependsOn` maps to `SelectorContext`
|
||||
|
||||
The connector selector field builds a `SelectorContext` from dependency values. For the mapping to work, each dependency's `canonicalParamId` (or field `id` for non-canonical fields) must exist in `SELECTOR_CONTEXT_FIELDS` (`lib/workflows/subblocks/context.ts`):
|
||||
|
||||
```
|
||||
oauthCredential, domain, teamId, projectId, knowledgeBaseId, planId,
|
||||
siteId, collectionId, spreadsheetId, fileId, baseId, datasetId, serviceDeskId
|
||||
```
|
||||
|
||||
### Available selector keys
|
||||
|
||||
Check `hooks/selectors/types.ts` for the full `SelectorKey` union. Common ones for connectors:
|
||||
|
||||
| SelectorKey | Context Deps | Returns |
|
||||
|-------------|-------------|---------|
|
||||
| `airtable.bases` | credential | Base ID + name |
|
||||
| `airtable.tables` | credential, `baseId` | Table ID + name |
|
||||
| `slack.channels` | credential | Channel ID + name |
|
||||
| `gmail.labels` | credential | Label ID + name |
|
||||
| `google.calendar` | credential | Calendar ID + name |
|
||||
| `linear.teams` | credential | Team ID + name |
|
||||
| `linear.projects` | credential, `teamId` | Project ID + name |
|
||||
| `jira.projects` | credential, `domain` | Project key + name |
|
||||
| `confluence.spaces` | credential, `domain` | Space key + name |
|
||||
| `notion.databases` | credential | Database ID + name |
|
||||
| `asana.workspaces` | credential | Workspace GID + name |
|
||||
| `microsoft.teams` | credential | Team ID + name |
|
||||
| `microsoft.channels` | credential, `teamId` | Channel ID + name |
|
||||
| `webflow.sites` | credential | Site ID + name |
|
||||
| `outlook.folders` | credential | Folder ID + name |
|
||||
|
||||
## ExternalDocument Shape
|
||||
|
||||
Every document returned from `listDocuments`/`getDocument` must include:
|
||||
|
||||
```typescript
|
||||
{
|
||||
externalId: string // Source-specific unique ID
|
||||
title: string // Document title
|
||||
content: string // Extracted plain text (or '' if contentDeferred)
|
||||
contentDeferred?: boolean // true = content will be fetched via getDocument
|
||||
mimeType: 'text/plain' // Always text/plain (content is extracted)
|
||||
contentHash: string // Metadata-based hash for change detection
|
||||
sourceUrl?: string // Link back to original (stored on document record)
|
||||
metadata?: Record<string, unknown> // Source-specific data (fed to mapTags)
|
||||
}
|
||||
```
|
||||
|
||||
## Content Deferral (Required for file/content-download connectors)
|
||||
|
||||
**All connectors that require per-document API calls to fetch content MUST use `contentDeferred: true`.** This is the standard pattern — `listDocuments` returns lightweight metadata stubs, and content is fetched lazily by the sync engine via `getDocument` only for new/changed documents.
|
||||
|
||||
This pattern is critical for reliability: the sync engine processes documents in batches and enqueues each batch for processing immediately. If a sync times out, all previously-batched documents are already queued. Without deferral, content downloads during listing can exhaust the sync task's time budget before any documents are saved.
|
||||
|
||||
### When to use `contentDeferred: true`
|
||||
|
||||
- The service's list API does NOT return document content (only metadata)
|
||||
- Content requires a separate download/export API call per document
|
||||
- Examples: Google Drive, OneDrive, SharePoint, Dropbox, Notion, Confluence, Gmail, Obsidian, Evernote, GitHub
|
||||
|
||||
### When NOT to use `contentDeferred`
|
||||
|
||||
- The list API already returns the full content inline (e.g., Slack messages, Reddit posts, HubSpot notes)
|
||||
- No per-document API call is needed to get content
|
||||
|
||||
### Content Hash Strategy
|
||||
|
||||
Use a **metadata-based** `contentHash` — never a content-based hash. The hash must be derivable from the list response metadata alone, so the sync engine can detect changes without downloading content.
|
||||
|
||||
Good metadata hash sources:
|
||||
- `modifiedTime` / `lastModifiedDateTime` — changes when file is edited
|
||||
- Git blob SHA — unique per content version
|
||||
- API-provided content hash (e.g., Dropbox `content_hash`)
|
||||
- Version number (e.g., Confluence page version)
|
||||
|
||||
Format: `{service}:{id}:{changeIndicator}`
|
||||
|
||||
```typescript
|
||||
// Google Drive: modifiedTime changes on edit
|
||||
contentHash: `gdrive:${file.id}:${file.modifiedTime ?? ''}`
|
||||
|
||||
// GitHub: blob SHA is a content-addressable hash
|
||||
contentHash: `gitsha:${item.sha}`
|
||||
|
||||
// Dropbox: API provides content_hash
|
||||
contentHash: `dropbox:${entry.id}:${entry.content_hash ?? entry.server_modified}`
|
||||
|
||||
// Confluence: version number increments on edit
|
||||
contentHash: `confluence:${page.id}:${page.version.number}`
|
||||
```
|
||||
|
||||
**Critical invariant:** The `contentHash` MUST be identical whether produced by `listDocuments` (stub) or `getDocument` (full doc). Both should use the same stub function to guarantee this.
|
||||
|
||||
### Implementation Pattern
|
||||
|
||||
```typescript
|
||||
// 1. Create a stub function (sync, no API calls)
|
||||
function fileToStub(file: ServiceFile): ExternalDocument {
|
||||
return {
|
||||
externalId: file.id,
|
||||
title: file.name || 'Untitled',
|
||||
content: '',
|
||||
contentDeferred: true,
|
||||
mimeType: 'text/plain',
|
||||
sourceUrl: `https://service.com/file/${file.id}`,
|
||||
contentHash: `service:${file.id}:${file.modifiedTime ?? ''}`,
|
||||
metadata: { /* fields needed by mapTags */ },
|
||||
}
|
||||
}
|
||||
|
||||
// 2. listDocuments returns stubs (fast, metadata only)
|
||||
listDocuments: async (accessToken, sourceConfig, cursor) => {
|
||||
const response = await fetchWithRetry(listUrl, { ... })
|
||||
const files = (await response.json()).files
|
||||
const documents = files.map(fileToStub)
|
||||
return { documents, nextCursor, hasMore }
|
||||
}
|
||||
|
||||
// 3. getDocument fetches content and returns full doc with SAME contentHash
|
||||
getDocument: async (accessToken, sourceConfig, externalId) => {
|
||||
const metadata = await fetchWithRetry(metadataUrl, { ... })
|
||||
const file = await metadata.json()
|
||||
if (file.trashed) return null
|
||||
|
||||
try {
|
||||
const content = await fetchContent(accessToken, file)
|
||||
if (!content.trim()) return null
|
||||
const stub = fileToStub(file)
|
||||
return { ...stub, content, contentDeferred: false }
|
||||
} catch (error) {
|
||||
logger.warn(`Failed to fetch content for: ${file.name}`, { error })
|
||||
return null
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Reference Implementations
|
||||
|
||||
- **Google Drive**: `connectors/google-drive/google-drive.ts` — file download/export with `modifiedTime` hash
|
||||
- **GitHub**: `connectors/github/github.ts` — git blob SHA hash
|
||||
- **Notion**: `connectors/notion/notion.ts` — blocks API with `last_edited_time` hash
|
||||
- **Confluence**: `connectors/confluence/confluence.ts` — version number hash
|
||||
|
||||
## tagDefinitions — Declared Tag Definitions
|
||||
|
||||
Declare which tags the connector populates using semantic IDs. Shown in the add-connector modal as opt-out checkboxes.
|
||||
On connector creation, slots are **dynamically assigned** via `getNextAvailableSlot` — connectors never hardcode slot names.
|
||||
|
||||
```typescript
|
||||
tagDefinitions: [
|
||||
{ id: 'labels', displayName: 'Labels', fieldType: 'text' },
|
||||
{ id: 'version', displayName: 'Version', fieldType: 'number' },
|
||||
{ id: 'lastModified', displayName: 'Last Modified', fieldType: 'date' },
|
||||
],
|
||||
```
|
||||
|
||||
Each entry has:
|
||||
- `id`: Semantic key matching a key returned by `mapTags` (e.g. `'labels'`, `'version'`)
|
||||
- `displayName`: Human-readable name shown in the UI (e.g. "Labels", "Last Modified")
|
||||
- `fieldType`: `'text'` | `'number'` | `'date'` | `'boolean'` — determines which slot pool to draw from
|
||||
|
||||
Users can opt out of specific tags in the modal. Disabled IDs are stored in `sourceConfig.disabledTagIds`.
|
||||
The assigned mapping (`semantic id → slot`) is stored in `sourceConfig.tagSlotMapping`.
|
||||
|
||||
## `@/connectors/utils` Helpers
|
||||
|
||||
Reuse these instead of inlining the same logic (the validator enforces them):
|
||||
|
||||
- `htmlToPlainText(html)` — strip HTML to plain text before indexing `ExternalDocument.content`. Never index raw HTML.
|
||||
- `computeContentHash(content)` — stable content hash for change detection.
|
||||
- `parseTagDate(value)` — parse to a valid `Date` or `undefined` (guards Invalid Date). Use in `mapTags` for date fields.
|
||||
- `joinTagArray(value)` — validate an array and join to a comma-separated string, or `undefined`. Use in `mapTags` for array/label fields.
|
||||
- `parseMultiValue(value)` — normalize a value into a `string[]`.
|
||||
|
||||
## mapTags — Metadata to Semantic Keys
|
||||
|
||||
Maps source metadata to semantic tag keys. Required if `tagDefinitions` is set.
|
||||
The sync engine calls this automatically and translates semantic keys to actual DB slots
|
||||
using the `tagSlotMapping` stored on the connector.
|
||||
|
||||
Return keys must match the `id` values declared in `tagDefinitions`.
|
||||
|
||||
Use the `@/connectors/utils` helpers for the common transforms — don't hand-roll date/array validation:
|
||||
|
||||
```typescript
|
||||
import { joinTagArray, parseTagDate } from '@/connectors/utils'
|
||||
|
||||
mapTags: (metadata: Record<string, unknown>): Record<string, unknown> => {
|
||||
const result: Record<string, unknown> = {}
|
||||
|
||||
// joinTagArray validates the array and joins to a comma-separated string (undefined if empty)
|
||||
const labels = joinTagArray(metadata.labels)
|
||||
if (labels) result.labels = labels
|
||||
|
||||
// Validate numbers — guard against NaN
|
||||
if (metadata.version != null) {
|
||||
const num = Number(metadata.version)
|
||||
if (!Number.isNaN(num)) result.version = num
|
||||
}
|
||||
|
||||
// parseTagDate returns a valid Date or undefined (guards against Invalid Date)
|
||||
const lastModified = parseTagDate(metadata.lastModified)
|
||||
if (lastModified) result.lastModified = lastModified
|
||||
|
||||
return result
|
||||
}
|
||||
```
|
||||
|
||||
## External API Calls — Use `fetchWithRetry`
|
||||
|
||||
All external API calls must use `fetchWithRetry` from `@/lib/knowledge/documents/utils` instead of raw `fetch()`. This provides exponential backoff with retries on 429/502/503/504 errors. It returns a standard `Response` — all `.ok`, `.json()`, `.text()` checks work unchanged.
|
||||
|
||||
For `validateConfig` (user-facing, called on save), pass `VALIDATE_RETRY_OPTIONS` to cap wait time at ~7s. Background operations (`listDocuments`, `getDocument`) use the built-in defaults (5 retries, ~31s max).
|
||||
|
||||
```typescript
|
||||
import { VALIDATE_RETRY_OPTIONS, fetchWithRetry } from '@/lib/knowledge/documents/utils'
|
||||
|
||||
// Background sync — use defaults
|
||||
const response = await fetchWithRetry(url, {
|
||||
method: 'GET',
|
||||
headers: { Authorization: `Bearer ${accessToken}` },
|
||||
})
|
||||
|
||||
// validateConfig — tighter retry budget
|
||||
const response = await fetchWithRetry(url, { ... }, VALIDATE_RETRY_OPTIONS)
|
||||
```
|
||||
|
||||
## sourceUrl
|
||||
|
||||
If `ExternalDocument.sourceUrl` is set, the sync engine stores it on the document record. Always construct the full URL (not a relative path).
|
||||
|
||||
## Capped or Incomplete Listings — `syncContext.listingCapped` (REQUIRED)
|
||||
|
||||
If `listDocuments` can ever return **less than the full source set** on a non-incremental sync — a `maxItems`/`maxDocuments`-style cap, or a transient per-item error that drops a still-existing document from the listing — it MUST set `syncContext.listingCapped = true` when that happens.
|
||||
|
||||
The sync engine reconciles deletions by comparing the full listing against stored documents: anything not seen is **hard-deleted** (sync-engine.ts, gated on `!syncContext?.listingCapped`). A truncated listing without this flag deletes every real document beyond the cap. This was the single most common bug found when auditing connectors — do not omit it.
|
||||
|
||||
```typescript
|
||||
if (hitLimit && syncContext) {
|
||||
syncContext.listingCapped = true
|
||||
}
|
||||
```
|
||||
|
||||
Rules:
|
||||
- Set it when a user-configured cap truncates the listing while more documents exist
|
||||
- Set it when a thrown error caused a still-present document to be skipped during listing
|
||||
- Do NOT set it when the source is genuinely exhausted (deleted documents must still reconcile)
|
||||
- Do NOT set it for intentional scope filters (e.g. a date cutoff) — out-of-scope documents should be reconciled normally
|
||||
|
||||
## Sync Engine Behavior (Do Not Modify)
|
||||
|
||||
The sync engine (`lib/knowledge/connectors/sync-engine.ts`) is connector-agnostic. It:
|
||||
1. Calls `listDocuments` with pagination until `hasMore` is false
|
||||
2. Compares `contentHash` to detect new/changed/unchanged documents
|
||||
3. Stores `sourceUrl` and calls `mapTags` on insert/update automatically
|
||||
4. Handles soft-delete of removed documents
|
||||
5. Resolves access tokens automatically — OAuth tokens are refreshed, API keys are decrypted from the `encryptedApiKey` column
|
||||
|
||||
You never need to modify the sync engine when adding a connector.
|
||||
|
||||
## Icon
|
||||
|
||||
The `icon` field on `ConnectorConfig` is used throughout the UI — in the connector list, the add-connector modal, and as the document icon in the knowledge base table (replacing the generic file type icon for connector-sourced documents). The icon is read from `CONNECTOR_REGISTRY[connectorType].icon` at runtime — no separate icon map to maintain.
|
||||
|
||||
If the service already has an icon in `apps/sim/components/icons.tsx` (from a tool integration), reuse it. Otherwise, ask the user to provide the SVG.
|
||||
|
||||
## Registering
|
||||
|
||||
Register in BOTH registries, keeping the same alphabetical-by-id ordering in each.
|
||||
|
||||
1. **Server registry** — `apps/sim/connectors/registry.server.ts` (server-only full registry; holds full connectors with runtime functions, imported by the sync engine and knowledge API routes):
|
||||
|
||||
```typescript
|
||||
import { {service}Connector } from '@/connectors/{service}'
|
||||
|
||||
export const CONNECTOR_REGISTRY: ConnectorRegistry = {
|
||||
// ... existing connectors ...
|
||||
{service}: {service}Connector,
|
||||
}
|
||||
```
|
||||
|
||||
2. **Client-safe meta registry** — `apps/sim/connectors/registry.ts` (imports each connector's `meta.ts` only, so client components can use it without pulling server-only code; the metadata counterpart to `BLOCK_META_REGISTRY`):
|
||||
|
||||
```typescript
|
||||
import { {service}ConnectorMeta } from '@/connectors/{service}/meta'
|
||||
|
||||
export const CONNECTOR_META_REGISTRY: ConnectorMetaRegistry = {
|
||||
// ... existing connector metas ...
|
||||
{service}: {service}ConnectorMeta,
|
||||
}
|
||||
```
|
||||
|
||||
`registry.ts` exports `CONNECTOR_META_REGISTRY: ConnectorMetaRegistry` plus the helpers `getConnectorMeta(id)` and `getAllConnectorMeta()`, importing each `@/connectors/{service}/meta` directly — never the runtime module. `registry.server.ts` exports `CONNECTOR_REGISTRY: ConnectorRegistry`.
|
||||
|
||||
## Reference Implementations
|
||||
|
||||
- **OAuth + contentDeferred**: `apps/sim/connectors/google-drive/google-drive.ts` — file download with metadata-based hash, `orderBy` for deterministic pagination
|
||||
- **OAuth + contentDeferred (blocks API)**: `apps/sim/connectors/notion/notion.ts` — complex block content extraction deferred to `getDocument`
|
||||
- **OAuth + contentDeferred (git)**: `apps/sim/connectors/github/github.ts` — blob SHA hash, tree listing
|
||||
- **OAuth + inline content**: `apps/sim/connectors/confluence/confluence.ts` — multiple config field types, `mapTags`, label fetching
|
||||
- **API key**: `apps/sim/connectors/fireflies/fireflies.ts` — GraphQL API with Bearer token auth
|
||||
|
||||
## Checklist
|
||||
|
||||
- [ ] Created `connectors/{service}/meta.ts` with `{service}ConnectorMeta: ConnectorMeta` (icon, name, auth, configFields, tagDefinitions) — no server/runtime imports
|
||||
- [ ] Created `connectors/{service}/{service}.ts` with `{service}Connector: ConnectorConfig` spreading the meta + runtime functions
|
||||
- [ ] Created `connectors/{service}/index.ts` barrel export
|
||||
- [ ] **Auth configured correctly:**
|
||||
- OAuth: `auth.provider` matches an existing `OAuthService` in `lib/oauth/types.ts`
|
||||
- API key: `auth.label` and `auth.placeholder` set appropriately
|
||||
- [ ] **Selector fields configured correctly (if applicable):**
|
||||
- Every `type: 'selector'` field has a canonical pair (`short-input` or `dropdown` with same `canonicalParamId` and `mode: 'advanced'`)
|
||||
- `required` is identical on both fields in each canonical pair
|
||||
- `selectorKey` exists in `hooks/selectors/registry.ts`
|
||||
- `dependsOn` references selector field IDs (not `canonicalParamId`)
|
||||
- Dependency `canonicalParamId` values exist in `SELECTOR_CONTEXT_FIELDS`
|
||||
- [ ] `listDocuments` handles pagination with metadata-based content hashes
|
||||
- [ ] `syncContext.listingCapped = true` set whenever the listing is truncated (max-items cap or transient per-item error) — required to prevent the engine's deletion reconciliation from removing unseen documents
|
||||
- [ ] `contentDeferred: true` used if content requires per-doc API calls (file download, export, blocks fetch)
|
||||
- [ ] `contentHash` is metadata-based (not content-based) and identical between stub and `getDocument`
|
||||
- [ ] `sourceUrl` set on each ExternalDocument (full URL, not relative)
|
||||
- [ ] `metadata` includes source-specific data for tag mapping
|
||||
- [ ] `tagDefinitions` declared for each semantic key returned by `mapTags`
|
||||
- [ ] `mapTags` implemented if source has useful metadata (labels, dates, versions)
|
||||
- [ ] `validateConfig` verifies the source is accessible
|
||||
- [ ] All external API calls use `fetchWithRetry` (not raw `fetch`)
|
||||
- [ ] All optional config fields validated in `validateConfig`
|
||||
- [ ] Icon exists in `components/icons.tsx` (or asked user to provide SVG)
|
||||
- [ ] Registered the full connector in `connectors/registry.server.ts`
|
||||
- [ ] Registered the meta in `connectors/registry.ts` (same alphabetical-by-id ordering as registry.server.ts)
|
||||
@@ -0,0 +1,142 @@
|
||||
---
|
||||
description: Add a code-defined table enrichment (registry entry) under `apps/sim/enrichments/` backed by an ordered provider cascade, ensuring every provider tool it calls has hosted-key support. Use when adding a per-row table enrichment that fills cells via existing Sim tools.
|
||||
argument-hint: <enrichment-name>
|
||||
---
|
||||
|
||||
# Adding a Table Enrichment
|
||||
|
||||
Enrichments are code-defined entries in `apps/sim/enrichments/` that run **directly per table row** (no workflow). Each enrichment declares inputs, outputs, and an ordered list of **providers**; the cascade runner tries providers in order and the first non-empty result fills the cell. Each provider calls one existing Sim tool via `executeTool`, which injects the workspace's BYOK key or a **hosted key** and bills usage automatically.
|
||||
|
||||
Because enrichments run on Sim's hosted keys by default, **every provider tool you reference must have hosted-key support** — otherwise it can only run when the workspace brings its own key. This command makes that check a required step.
|
||||
|
||||
## Overview
|
||||
|
||||
| Step | What | Where |
|
||||
|------|------|-------|
|
||||
| 1 | Pick the data-source tool(s) for each output | `tools/{service}/` + `tools/registry.ts` |
|
||||
| 2 | **Verify each tool has `hosting`; if not, run `/add-hosted-key`** | `tools/{service}/{action}.ts` |
|
||||
| 3 | Write the enrichment definition | `enrichments/{name}/{name}.ts` + `index.ts` |
|
||||
| 4 | Register it | `enrichments/registry.ts` |
|
||||
| 5 | Verify | tsc / biome / manual run |
|
||||
|
||||
## Architecture (what you're plugging into)
|
||||
|
||||
- **`enrichments/types.ts`** — `EnrichmentConfig { id, name, description, icon, inputs, outputs, providers }` and `EnrichmentProvider { id, label, toolId, buildParams, mapOutput }`. Providers are **plain data** (no `@/tools` import) so the catalog stays client-safe.
|
||||
- **`enrichments/providers.ts`** — `toolProvider(...)` (typed passthrough) plus shared input helpers: `str(v)`, `normalizeDomain(v)`, `firstNonEmpty(arr)`, `splitName(fullName)`.
|
||||
- **`enrichments/run.ts`** — the server-only cascade runner. Calls `executeTool(provider.toolId, { ...params, _context: { workspaceId } })`, accumulates hosted-key cost, returns the first non-empty mapped result. **You do not edit this** — it works for any registry entry.
|
||||
- **`enrichments/registry.ts`** — `ENRICHMENT_REGISTRY` / `ALL_ENRICHMENTS` / `getEnrichment`. Register new entries here.
|
||||
|
||||
Outputs automatically become table columns; billing, the catalog/sidebar UI, the column meta-header icon, and per-row execution all work with no extra wiring.
|
||||
|
||||
## Step 1: Pick the data-source tool(s)
|
||||
|
||||
For each output the enrichment produces, decide which existing tool provides it. Look up the service's API and the tool in `apps/sim/tools/{service}/` (e.g. `hunter_email_finder`, `pdl_person_enrich`, `pdl_company_enrich`). Confirm:
|
||||
|
||||
- The tool id is registered in `apps/sim/tools/registry.ts`.
|
||||
- Its `params` accept what you can derive from table columns (read the tool's `params`).
|
||||
- Its `outputs` / `transformResponse` actually expose the field you need (read the real output shape — don't assume).
|
||||
|
||||
Order providers **cheapest / most-likely-to-hit first**; the cascade stops at the first non-empty result. Apollo / LinkedIn are not hosted-safe (ToS) — don't use them.
|
||||
|
||||
## Step 2: Verify hosted-key support — chain to `/add-hosted-key` if missing
|
||||
|
||||
**This is the required gate.** For every tool a provider calls, open `apps/sim/tools/{service}/{action}.ts` and check for a `hosting` block:
|
||||
|
||||
```typescript
|
||||
hosting: {
|
||||
envKeyPrefix: 'SERVICE_API_KEY',
|
||||
apiKeyParam: 'apiKey',
|
||||
byokProviderId: 'service',
|
||||
pricing: { /* ... */ },
|
||||
rateLimit: { /* ... */ },
|
||||
}
|
||||
```
|
||||
|
||||
- **If `hosting` is present** — good. Note the `envKeyPrefix`; the deployment needs `{PREFIX}_COUNT` + `{PREFIX}_1..N` env vars set for the hosted key to actually resolve at runtime (ops concern, not code). If those env vars aren't set in the target environment, the provider will only run with a workspace BYOK key.
|
||||
- **If `hosting` is absent** — the tool can't use a Sim-provided key, so the enrichment would silently produce blank cells on hosted Sim. **Stop and run `/add-hosted-key <service>`** to add hosted-key support to that tool first, then come back. Do this for every provider tool that lacks it.
|
||||
|
||||
Why it matters: the cascade runner only bills (and only reads `output.cost.total`) when `executeTool` injected a hosted key, which requires the tool's `hosting` config. No `hosting` → no hosted key → the enrichment depends entirely on per-workspace BYOK.
|
||||
|
||||
## Step 3: Write the enrichment definition
|
||||
|
||||
Create `apps/sim/enrichments/{name}/{name}.ts` and a barrel `index.ts`. Mirror the existing entries (`work-email`, `phone-number`, `company-domain`, `company-info`).
|
||||
|
||||
```typescript
|
||||
import { SomeIcon } from '@sim/emcn/icons'
|
||||
import { filterUndefined } from '@sim/utils/object'
|
||||
import { normalizeDomain, splitName, str, toolProvider } from '@/enrichments/providers'
|
||||
import type { EnrichmentConfig } from '@/enrichments/types'
|
||||
|
||||
export const myEnrichment: EnrichmentConfig = {
|
||||
id: 'my-enrichment',
|
||||
name: 'My Enrichment',
|
||||
description: 'One concise sentence describing what it finds.',
|
||||
icon: SomeIcon,
|
||||
inputs: [
|
||||
// Person enrichments take a single canonical `fullName` (Clay-style);
|
||||
// split it with splitName() for tools that need first/last.
|
||||
{ id: 'fullName', name: 'Full name', type: 'string', required: true },
|
||||
{ id: 'companyDomain', name: 'Company domain', type: 'string' },
|
||||
],
|
||||
outputs: [{ id: 'value', name: 'value', type: 'string' }],
|
||||
providers: [
|
||||
toolProvider({
|
||||
id: 'provider-a',
|
||||
label: 'Provider A',
|
||||
toolId: 'service_action', // must have `hosting` (Step 2)
|
||||
buildParams: (inputs) => {
|
||||
// Return null when there aren't enough inputs → cascade skips this provider.
|
||||
const name = splitName(inputs.fullName)
|
||||
const domain = normalizeDomain(inputs.companyDomain)
|
||||
if (!name || !domain) return null
|
||||
return { domain, first_name: name.firstName, last_name: name.lastName }
|
||||
},
|
||||
mapOutput: (output) => {
|
||||
// Return { [outputId]: value } on a hit, or null to fall through.
|
||||
const value = str(output.value)
|
||||
return value ? { value } : null
|
||||
},
|
||||
}),
|
||||
// ...additional fallback providers, in priority order.
|
||||
],
|
||||
}
|
||||
```
|
||||
|
||||
```typescript
|
||||
// apps/sim/enrichments/{name}/index.ts
|
||||
export { myEnrichment } from './my-enrichment'
|
||||
```
|
||||
|
||||
Rules:
|
||||
- Keep the file **client-safe**: import only `@sim/emcn/icons`, `@sim/utils/*`, `@/enrichments/providers`, and the types. **Never import `@/tools`** here — the runner does the tool call.
|
||||
- `buildParams` returns `null` when inputs are insufficient (provider skipped). `mapOutput` returns `null`/empty for a miss (falls through). Use `filterUndefined` when assembling optional tool params; coerce numbers explicitly (don't pass `''` to number outputs).
|
||||
- Output `id`s are the keys `mapOutput` returns; output `name`s are the default column names (the user can rename them in the config).
|
||||
|
||||
## Step 4: Register it
|
||||
|
||||
In `apps/sim/enrichments/registry.ts`, import and add the entry (catalog order is registration order):
|
||||
|
||||
```typescript
|
||||
import { myEnrichment } from '@/enrichments/my-enrichment'
|
||||
|
||||
export const ENRICHMENT_REGISTRY: EnrichmentRegistry = {
|
||||
// ...existing
|
||||
[myEnrichment.id]: myEnrichment,
|
||||
}
|
||||
```
|
||||
|
||||
## Step 5: Verify
|
||||
|
||||
1. `bun run type-check` (from `apps/sim`) and `bunx biome check` on the changed files.
|
||||
2. In a table → **+ New column → Enrichments** → pick the new enrichment, map its inputs to columns, name the output column(s), Save. Confirm it appears in the catalog with its icon/description.
|
||||
3. With hosted keys (or a workspace BYOK key) configured for each provider's service, run a row and confirm the cell fills; the dev-server log shows `Enrichment hit { provider }`. A row whose providers all miss completes blank; a row where every provider errored shows an error cell.
|
||||
|
||||
## Checklist
|
||||
|
||||
- [ ] Each output mapped to a real tool field (verified against the tool's `params`/`outputs`)
|
||||
- [ ] **Every provider tool has a `hosting` block — ran `/add-hosted-key` for any that didn't**
|
||||
- [ ] Providers ordered cheapest / most-likely-first; Apollo/LinkedIn not used
|
||||
- [ ] Enrichment file is client-safe (no `@/tools` import); uses `toolProvider` + shared helpers
|
||||
- [ ] `buildParams` returns `null` on insufficient inputs; `mapOutput` returns `null` on a miss
|
||||
- [ ] Registered in `enrichments/registry.ts`
|
||||
- [ ] tsc + biome clean; created and ran the column end-to-end
|
||||
@@ -0,0 +1,94 @@
|
||||
---
|
||||
description: Add a runtime feature flag (AppConfig-backed on prod, secret fallback off-prod), global by default or optionally gated by org id, user id, or platform admin
|
||||
argument-hint: <flag-name>
|
||||
---
|
||||
|
||||
# Add Feature Flag Skill
|
||||
|
||||
You add a **runtime feature flag** to Sim that can change on prod with no redeploy (AWS AppConfig). Prefer a global on/off flag unless the rollout actually needs per-organization, per-user, or platform-admin targeting. When AppConfig isn't the source of truth, the flag falls back to a single **secret** (on/off only).
|
||||
|
||||
## When to use this vs `env-flags.ts`
|
||||
|
||||
- **Feature flag** (`@/lib/core/config/feature-flags.ts`): runtime global on/off by default, optionally scoped by `userId`/`orgId`/admin. This skill.
|
||||
- **Env flag** (`@/lib/core/config/env-flags.ts`): deploy-time capability/environment detection (`isProd`, `isHosted`, `isBillingEnabled`). A module-load boolean. **Do not add gated flags here.**
|
||||
|
||||
If the user wants a fixed per-deployment toggle, send them to `env-flags.ts` instead.
|
||||
|
||||
## The flag model
|
||||
|
||||
A flag's **gating rule lives only in the hosted AppConfig document**. It is ON for a context when any configured clause matches:
|
||||
|
||||
```ts
|
||||
interface FeatureFlagRule {
|
||||
enabled?: boolean // global default for everyone
|
||||
orgIds?: string[] // allowlisted organization ids
|
||||
userIds?: string[] // allowlisted user ids
|
||||
adminEnabled?: boolean // platform admins (user.role === 'admin')
|
||||
}
|
||||
```
|
||||
|
||||
Critically, **none of this is expressible in code** — gating (especially `adminEnabled`) can only be set through AppConfig, so no environment can grant access from a code literal. Off-AppConfig (self-hosted/OSS/local), a flag is simply on or off, derived from its fallback secret.
|
||||
|
||||
## Steps
|
||||
|
||||
1. **Confirm the granularity before editing code.** If the user has not already specified it, stop and ask:
|
||||
|
||||
> Should `<flag-name>` be a global on/off flag (recommended), or does it need rollout targeting by organization, user, and/or platform admin?
|
||||
|
||||
- Recommend **global**. Do not infer scoped gating merely because the call site already has a user or organization id.
|
||||
- If the user chooses scoped gating but does not name the dimensions, ask which of organization, user, and platform admin it needs. Wire only the selected dimensions.
|
||||
- If the user wants a fixed per-deployment toggle rather than a runtime AppConfig flag, use `env-flags.ts` instead.
|
||||
|
||||
2. **Define the flag.** Add one entry to the `FEATURE_FLAGS` registry in `apps/sim/lib/core/config/feature-flags.ts`. Each entry is the flag's whole definition — name (kebab-case key), `description`, and the `fallback` secret consulted when AppConfig isn't the source of truth (truthy ⇒ on globally):
|
||||
|
||||
```ts
|
||||
const FEATURE_FLAGS = {
|
||||
'<flag-name>': {
|
||||
description: '<what this gates>',
|
||||
fallback: '<FLAG_SECRET>',
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
`fallback` is the env/secret key (typed as `keyof typeof env`), so add `<FLAG_SECRET>` to `apps/sim/lib/core/config/env.ts` first (and the deployment's secret store) — it won't typecheck otherwise. Do **not** add org/user/admin defaults here — that gating exists only in AppConfig. Adding the entry makes `<flag-name>` a valid `FeatureFlagName`.
|
||||
|
||||
3. **Gate the call site at the chosen granularity.** For the recommended global mode, pass no context:
|
||||
|
||||
```ts
|
||||
import { isFeatureEnabled } from '@/lib/core/config/feature-flags'
|
||||
|
||||
if (await isFeatureEnabled('<flag-name>')) {
|
||||
// gated behavior
|
||||
}
|
||||
```
|
||||
|
||||
Do not fetch, resolve, or thread through user or organization context solely for a global flag.
|
||||
|
||||
For scoped rollout, pass only the dimensions the user selected. Admin status is resolved internally, so ordinary callers pass `userId`, not a role:
|
||||
|
||||
```ts
|
||||
import { isFeatureEnabled } from '@/lib/core/config/feature-flags'
|
||||
|
||||
if (await isFeatureEnabled('<flag-name>', { userId, orgId })) {
|
||||
// gated behavior
|
||||
}
|
||||
```
|
||||
|
||||
- Organization targeting uses `orgId`; user and platform-admin targeting require `userId`.
|
||||
- Missing ids are fine — a clause with no matching id is skipped; with no `userId`, the admin clause resolves to `false` without a DB read.
|
||||
- Admin routes that already know the caller is an admin may pass `{ userId, isAdmin: true }` to skip the role lookup.
|
||||
- **Client/UI flags:** resolve server-side (in a server component, route, or loader) and pass the boolean down as a prop. There is no client AppConfig.
|
||||
|
||||
4. **(Prod) configure in AppConfig.** The infra `feature-flags` profile schema is permissive, so a new flag needs **no infra change**. Operators add the flag to the hosted `feature-flags` document using `enabled` for global rollout or only the selected `orgIds`/`userIds`/`adminEnabled` clauses for scoped rollout, then start a `sim-<env>-fast` deployment (see the AppConfig runbook in the infra README — same flow as `access-control`). The fallback secret only applies when AppConfig is disabled.
|
||||
|
||||
5. **Test.** Add a case to `apps/sim/lib/core/config/feature-flags.test.ts` that matches the chosen granularity. For a global flag, exercise `isFeatureEnabled('<flag-name>')` with an AppConfig `enabled` rule and toggle the fallback secret for the off-AppConfig path. For scoped rollout, cover only the selected clauses and mock `isPlatformAdmin` when testing `adminEnabled`.
|
||||
|
||||
6. **Clean up after rollout.** When the feature ships to everyone, delete the flag's entry from `FEATURE_FLAGS`, the `<FLAG_SECRET>` env entry, the AppConfig document, the call sites, and the test. Leaving dead flags around is the main failure mode of flag systems.
|
||||
|
||||
## Notes
|
||||
|
||||
- Flag keys are `kebab-case`.
|
||||
- Never read flags via raw `fetch` or a new AppConfig client — always go through `isFeatureEnabled` / `getFeatureFlags`.
|
||||
- Never bake gating into code. The fallback is a single boolean secret; org/user/admin scoping is AppConfig-only.
|
||||
- Never add or propagate request context unless the user chose scoped rollout.
|
||||
- The admin check reads the DB **replica** (`dbReplica`) and is resolved lazily, so an admin-gated flag adds at most one cheap replica read, and only when `adminEnabled` is the deciding clause.
|
||||
@@ -0,0 +1,301 @@
|
||||
---
|
||||
description: Add hosted API key support to a tool so Sim provides the key (metered and billed to the workspace) when a user has not brought their own. Use when adding a `hosting` config to a tool under `apps/sim/tools/{service}/`.
|
||||
argument-hint: <service-name>
|
||||
---
|
||||
|
||||
# Adding Hosted Key Support to a Tool
|
||||
|
||||
When a tool has hosted key support, Sim provides its own API key if the user hasn't configured one (via BYOK or env var). Usage is metered and billed to the workspace.
|
||||
|
||||
## Overview
|
||||
|
||||
| Step | What | Where |
|
||||
|------|------|-------|
|
||||
| 1 | Register BYOK provider ID | `tools/types.ts`, `lib/api/contracts/byok-keys.ts` |
|
||||
| 2 | Research the API's pricing and rate limits | API docs / pricing page (before writing any code) |
|
||||
| 3 | Add `hosting` config to the tool | `tools/{service}/{action}.ts` |
|
||||
| 4 | Hide API key field when hosted | `blocks/blocks/{service}.ts` |
|
||||
| 5 | Add to BYOK settings UI | BYOK settings component (`byok.tsx`) |
|
||||
| 6 | Summarize pricing and throttling comparison | Output to user (after all code changes) |
|
||||
|
||||
## Step 1: Register the BYOK Provider ID
|
||||
|
||||
Add the new provider to the `BYOKProviderId` union in `tools/types.ts`:
|
||||
|
||||
```typescript
|
||||
export type BYOKProviderId =
|
||||
| 'openai'
|
||||
| 'anthropic'
|
||||
// ...existing providers
|
||||
| 'your_service'
|
||||
```
|
||||
|
||||
Then add the same provider id to the `byokProviderIdSchema` enum in `lib/api/contracts/byok-keys.ts` (this is what the byok-keys route validates against):
|
||||
|
||||
```typescript
|
||||
export const byokProviderIdSchema = z.enum([
|
||||
'openai',
|
||||
'anthropic',
|
||||
// ...existing providers
|
||||
'your_service',
|
||||
])
|
||||
```
|
||||
|
||||
## Step 2: Research the API's Pricing Model and Rate Limits
|
||||
|
||||
**Before writing any `getCost` or `rateLimit` code**, look up the service's official documentation for both pricing and rate limits. You need to understand:
|
||||
|
||||
### Pricing
|
||||
|
||||
1. **How the API charges** — per request, per credit, per token, per step, per minute, etc.
|
||||
2. **Whether the API reports cost in its response** — look for fields like `creditsUsed`, `costDollars`, `tokensUsed`, or similar in the response body or headers
|
||||
3. **Whether cost varies by endpoint/options** — some APIs charge more for certain features (e.g., Firecrawl charges 1 credit/page base but +4 for JSON format, +4 for enhanced mode)
|
||||
4. **The dollar-per-unit rate** — what each credit/token/unit costs in dollars on our plan
|
||||
|
||||
### Rate Limits
|
||||
|
||||
1. **What rate limits the API enforces** — requests per minute/second, tokens per minute, concurrent requests, etc.
|
||||
2. **Whether limits vary by plan tier** — free vs paid vs enterprise often have different ceilings
|
||||
3. **Whether limits are per-key or per-account** — determines whether adding more hosted keys actually increases total throughput
|
||||
4. **What the API returns when rate limited** — HTTP 429, `Retry-After` header, error body format, etc.
|
||||
5. **Whether there are multiple dimensions** — some APIs limit both requests/min AND tokens/min independently
|
||||
|
||||
Search the API's docs/pricing page (use WebSearch/WebFetch). Capture the pricing model as a comment in `getCost` so future maintainers know the source of truth.
|
||||
|
||||
### Setting Our Rate Limits
|
||||
|
||||
Our rate limiter (`lib/core/rate-limiter/hosted-key/`) uses a token-bucket algorithm applied **per billing actor** (workspace). It supports two modes:
|
||||
|
||||
- **`per_request`** — simple; just `requestsPerMinute`. Good when the API charges flat per-request or cost doesn't vary much.
|
||||
- **`custom`** — `requestsPerMinute` plus additional `dimensions` (e.g., `tokens`, `search_units`). Each dimension has its own `limitPerMinute` and an `extractUsage` function that reads actual usage from the response. Use when the API charges on a variable metric (tokens, credits) and you want to cap that metric too.
|
||||
|
||||
When choosing values for `requestsPerMinute` and any dimension limits:
|
||||
|
||||
- **Stay well below the API's per-key limit** — our keys are shared across all workspaces. If the API allows 60 RPM per key and we have 3 keys, the global ceiling is ~180 RPM. Set the per-workspace limit low enough (e.g., 20-60 RPM) that many workspaces can coexist without collectively hitting the API's ceiling.
|
||||
- **Account for key pooling** — our round-robin distributes requests across `N` hosted keys, so the effective API-side rate per key is `(total requests) / N`. But per-workspace limits are enforced *before* key selection, so they apply regardless of key count.
|
||||
- **Prefer conservative defaults** — it's easy to raise limits later but hard to claw back after users depend on high throughput.
|
||||
|
||||
## Step 3: Add `hosting` Config to the Tool
|
||||
|
||||
Add a `hosting` object to the tool's `ToolConfig`. This tells the execution layer how to acquire hosted keys, calculate cost, and rate-limit.
|
||||
|
||||
```typescript
|
||||
hosting: {
|
||||
envKeyPrefix: 'YOUR_SERVICE_API_KEY',
|
||||
apiKeyParam: 'apiKey',
|
||||
byokProviderId: 'your_service',
|
||||
pricing: {
|
||||
type: 'custom',
|
||||
getCost: (_params, output) => {
|
||||
if (output.creditsUsed == null) {
|
||||
throw new Error('Response missing creditsUsed field')
|
||||
}
|
||||
const creditsUsed = output.creditsUsed as number
|
||||
const cost = creditsUsed * 0.001 // dollars per credit
|
||||
return { cost, metadata: { creditsUsed } }
|
||||
},
|
||||
},
|
||||
rateLimit: {
|
||||
mode: 'per_request',
|
||||
requestsPerMinute: 100,
|
||||
},
|
||||
},
|
||||
```
|
||||
|
||||
### Hosted Key Env Var Convention
|
||||
|
||||
Keys use a numbered naming pattern driven by a count env var:
|
||||
|
||||
```
|
||||
YOUR_SERVICE_API_KEY_COUNT=3
|
||||
YOUR_SERVICE_API_KEY_1=sk-...
|
||||
YOUR_SERVICE_API_KEY_2=sk-...
|
||||
YOUR_SERVICE_API_KEY_3=sk-...
|
||||
```
|
||||
|
||||
The `envKeyPrefix` value (`YOUR_SERVICE_API_KEY`) determines which env vars are read at runtime. Adding more keys only requires bumping the count and adding the new env var.
|
||||
|
||||
### Pricing: Prefer API-Reported Cost
|
||||
|
||||
Always prefer using cost data returned by the API (e.g., `creditsUsed`, `costDollars`). This is the most accurate because it accounts for variable pricing tiers, feature modifiers, and plan-level discounts.
|
||||
|
||||
**When the API reports cost** — use it directly and throw if missing:
|
||||
|
||||
```typescript
|
||||
pricing: {
|
||||
type: 'custom',
|
||||
getCost: (params, output) => {
|
||||
if (output.creditsUsed == null) {
|
||||
throw new Error('Response missing creditsUsed field')
|
||||
}
|
||||
// $0.001 per credit — from https://example.com/pricing
|
||||
const cost = (output.creditsUsed as number) * 0.001
|
||||
return { cost, metadata: { creditsUsed: output.creditsUsed } }
|
||||
},
|
||||
},
|
||||
```
|
||||
|
||||
**When the API does NOT report cost** — compute it from params/output based on the pricing docs, but still validate the data you depend on:
|
||||
|
||||
```typescript
|
||||
pricing: {
|
||||
type: 'custom',
|
||||
getCost: (params, output) => {
|
||||
if (!Array.isArray(output.searchResults)) {
|
||||
throw new Error('Response missing searchResults, cannot determine cost')
|
||||
}
|
||||
// Serper: 1 credit for <=10 results, 2 credits for >10 — from https://serper.dev/pricing
|
||||
const credits = Number(params.num) > 10 ? 2 : 1
|
||||
return { cost: credits * 0.001, metadata: { credits } }
|
||||
},
|
||||
},
|
||||
```
|
||||
|
||||
**`getCost` must always throw** if it cannot determine cost. Never silently fall back to a default — this would hide billing inaccuracies.
|
||||
|
||||
### Capturing Cost Data from the API
|
||||
|
||||
If the API returns cost info, capture it in `transformResponse` so `getCost` can read it from the output:
|
||||
|
||||
```typescript
|
||||
transformResponse: async (response: Response) => {
|
||||
const data = await response.json()
|
||||
return {
|
||||
success: true,
|
||||
output: {
|
||||
results: data.results,
|
||||
creditsUsed: data.creditsUsed, // pass through for getCost
|
||||
},
|
||||
}
|
||||
},
|
||||
```
|
||||
|
||||
For async/polling tools, capture it in `postProcess` when the job completes:
|
||||
|
||||
```typescript
|
||||
if (jobData.status === 'completed') {
|
||||
result.output = {
|
||||
data: jobData.data,
|
||||
creditsUsed: jobData.creditsUsed,
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Step 4: Hide the API Key Field When Hosted
|
||||
|
||||
In the block config (`blocks/blocks/{service}.ts`), add `hideWhenHosted: true` to the API key subblock. This hides the field on hosted Sim since the platform provides the key:
|
||||
|
||||
```typescript
|
||||
{
|
||||
id: 'apiKey',
|
||||
title: 'API Key',
|
||||
type: 'short-input',
|
||||
placeholder: 'Enter your API key',
|
||||
password: true,
|
||||
required: true,
|
||||
hideWhenHosted: true,
|
||||
},
|
||||
```
|
||||
|
||||
The visibility is controlled by `isSubBlockHidden()` in `lib/workflows/subblocks/visibility.ts`, which checks both the `isHosted` feature flag (`hideWhenHosted`) and optional env var conditions (`hideWhenEnvSet`).
|
||||
|
||||
### Excluding Specific Operations from Hosted Key Support
|
||||
|
||||
When a block has multiple operations but some operations should **not** use a hosted key (e.g., the underlying API is deprecated, unsupported, or too expensive), use the **duplicate apiKey subblock** pattern. This is the same pattern Exa uses for its `research` operation:
|
||||
|
||||
1. **Remove the `hosting` config** from the tool definition for that operation — it must not have a `hosting` object at all.
|
||||
2. **Duplicate the `apiKey` subblock** in the block config with opposing conditions:
|
||||
|
||||
```typescript
|
||||
// API Key — hidden when hosted for operations with hosted key support
|
||||
{
|
||||
id: 'apiKey',
|
||||
title: 'API Key',
|
||||
type: 'short-input',
|
||||
placeholder: 'Enter your API key',
|
||||
password: true,
|
||||
required: true,
|
||||
hideWhenHosted: true,
|
||||
condition: { field: 'operation', value: 'unsupported_op', not: true },
|
||||
},
|
||||
// API Key — always visible for unsupported_op (no hosted key support)
|
||||
{
|
||||
id: 'apiKey',
|
||||
title: 'API Key',
|
||||
type: 'short-input',
|
||||
placeholder: 'Enter your API key',
|
||||
password: true,
|
||||
required: true,
|
||||
condition: { field: 'operation', value: 'unsupported_op' },
|
||||
},
|
||||
```
|
||||
|
||||
Both subblocks share the same `id: 'apiKey'`, so the same value flows to the tool. The conditions ensure only one is visible at a time. The first has `hideWhenHosted: true` and shows for all hosted operations; the second has no `hideWhenHosted` and shows only for the excluded operation — meaning users must always provide their own key for that operation.
|
||||
|
||||
To exclude multiple operations, use an array: `{ field: 'operation', value: ['op_a', 'op_b'] }`.
|
||||
|
||||
**Reference implementations:**
|
||||
- **Exa** (`blocks/blocks/exa.ts`): `exa_research` operation excluded from hosting — duplicate `apiKey` pair around lines ~348-365
|
||||
- **Google Maps** (`blocks/blocks/google_maps.ts`): `speed_limits` operation excluded from hosting (deprecated Roads API)
|
||||
|
||||
## Step 5: Add to the BYOK Settings UI
|
||||
|
||||
Add an entry to the `PROVIDERS` array in the BYOK settings component so users can bring their own key. You need the service icon from `components/icons.tsx`:
|
||||
|
||||
```typescript
|
||||
{
|
||||
id: 'your_service',
|
||||
name: 'Your Service',
|
||||
icon: YourServiceIcon,
|
||||
description: 'What this service does',
|
||||
placeholder: 'Enter your API key',
|
||||
},
|
||||
```
|
||||
|
||||
## Step 6: Summarize Pricing and Throttling Comparison
|
||||
|
||||
After all code changes are complete, output a detailed summary to the user covering:
|
||||
|
||||
### What to include
|
||||
|
||||
1. **API's pricing model** — how the service charges (per token, per credit, per request, etc.), the specific rates found in docs, and whether the API reports cost in responses.
|
||||
2. **Our `getCost` approach** — how we calculate cost, what fields we depend on, and any assumptions or estimates (especially when the API doesn't report exact dollar cost).
|
||||
3. **API's rate limits** — the documented limits (RPM, TPM, concurrent, etc.), which plan tier they apply to, and whether they're per-key or per-account.
|
||||
4. **Our `rateLimit` config** — what we set for `requestsPerMinute` (and dimensions if custom mode), why we chose those values, and how they compare to the API's limits.
|
||||
5. **Key pooling impact** — how many hosted keys we expect, and how round-robin distribution affects the effective per-key rate at the API.
|
||||
6. **Gaps or risks** — anything the API charges for that we don't meter, rate limit dimensions we chose not to enforce, or pricing that may be inaccurate due to variable model/tier costs.
|
||||
|
||||
### Format
|
||||
|
||||
Present this as a structured summary with clear headings. Example:
|
||||
|
||||
```
|
||||
### Pricing
|
||||
- **API charges**: $X per 1M tokens (input), $Y per 1M tokens (output) — varies by model
|
||||
- **Response reports cost?**: No — only token counts in `usage` field
|
||||
- **Our getCost**: Estimates cost at $Z per 1M total tokens based on median model pricing
|
||||
- **Risk**: Actual cost varies by model; our estimate may over/undercharge for cheap/expensive models
|
||||
|
||||
### Throttling
|
||||
- **API limits**: 300 RPM per key (paid tier), 60 RPM (free tier)
|
||||
- **Per-key or per-account**: Per key — more keys = more throughput
|
||||
- **Our config**: 60 RPM per workspace (per_request mode)
|
||||
- **With N keys**: Effective per-key rate is (total RPM across workspaces) / N
|
||||
- **Headroom**: Comfortable — even 10 active workspaces at full rate = 600 RPM / 3 keys = 200 RPM per key, under the 300 RPM API limit
|
||||
```
|
||||
|
||||
This summary helps reviewers verify that the pricing and rate limiting are well-calibrated and surfaces any risks that need monitoring.
|
||||
|
||||
## Checklist
|
||||
|
||||
- [ ] Provider added to `BYOKProviderId` in `tools/types.ts`
|
||||
- [ ] Provider added to `byokProviderIdSchema` enum in `lib/api/contracts/byok-keys.ts`
|
||||
- [ ] API pricing docs researched — understand per-unit cost and whether the API reports cost in responses
|
||||
- [ ] API rate limits researched — understand RPM/TPM limits, per-key vs per-account, and plan tiers
|
||||
- [ ] `hosting` config added to the tool with `envKeyPrefix`, `apiKeyParam`, `byokProviderId`, `pricing`, and `rateLimit`
|
||||
- [ ] `getCost` throws if required cost data is missing from the response
|
||||
- [ ] Cost data captured in `transformResponse` or `postProcess` if API provides it
|
||||
- [ ] `hideWhenHosted: true` added to the API key subblock in the block config
|
||||
- [ ] Provider entry added to the BYOK settings UI with icon and description
|
||||
- [ ] Env vars documented: `{PREFIX}_COUNT` and `{PREFIX}_1..N`
|
||||
- [ ] Pricing and throttling summary provided to reviewer
|
||||
@@ -0,0 +1,998 @@
|
||||
---
|
||||
description: Add a complete Sim integration from API docs, covering tools, block, icon, optional triggers, registrations, resolved-secret/model-input safety, and integration conventions. Use when introducing a new service under `apps/sim/tools`, `apps/sim/blocks`, and `apps/sim/triggers`.
|
||||
argument-hint: <service-name> [api-docs-url]
|
||||
---
|
||||
|
||||
# Add Integration Skill
|
||||
|
||||
You are an expert at adding complete integrations to Sim. This skill orchestrates the full process of adding a new service integration.
|
||||
|
||||
## Overview
|
||||
|
||||
Adding an integration involves these steps in order:
|
||||
1. **Research** - Read the service's API documentation
|
||||
2. **Create Tools** - Build tool configurations for each API operation
|
||||
3. **Create Block** - Build the block UI configuration
|
||||
4. **Add Icon** - Add the service's brand icon
|
||||
5. **Create Triggers** (optional) - If the service supports webhooks
|
||||
6. **Register** - Register tools, block, and triggers in their registries
|
||||
7. **Configure Deployment Availability** - Wire OAuth client and service-account metadata
|
||||
8. **Generate and Validate the Catalog** - Regenerate docs/catalog artifacts and run drift checks
|
||||
|
||||
## Step 1: Research the API
|
||||
|
||||
Before writing any code:
|
||||
1. Use Context7 to find official documentation: `mcp__context7__resolve-library-id`, then fetch with `mcp__context7__query-docs`
|
||||
2. Or use WebFetch to read API docs directly
|
||||
3. Identify:
|
||||
- Authentication method (OAuth, API Key, both)
|
||||
- Available operations (CRUD, search, etc.)
|
||||
- Required vs optional parameters
|
||||
- Response structures
|
||||
|
||||
### Hard Rule: No Guessed Response Schemas
|
||||
|
||||
If the official docs do not clearly show the response JSON shape for an endpoint, you MUST stop and tell the user exactly which outputs are unknown.
|
||||
|
||||
- Do NOT guess response field names
|
||||
- Do NOT infer nested JSON paths from related endpoints
|
||||
- Do NOT invent output properties just because they seem likely
|
||||
- Do NOT implement `transformResponse` against unverified payload shapes
|
||||
|
||||
If response schemas are missing or incomplete, do one of the following before proceeding:
|
||||
1. Ask the user for sample responses
|
||||
2. Ask the user for test credentials so you can verify the live payload
|
||||
3. Reduce the scope to only endpoints whose response shapes are documented
|
||||
4. Leave the tool unimplemented and explicitly report why
|
||||
|
||||
## Step 2: Create Tools
|
||||
|
||||
### Directory Structure
|
||||
```
|
||||
apps/sim/tools/{service}/
|
||||
├── index.ts # Barrel exports
|
||||
├── types.ts # TypeScript interfaces
|
||||
├── {action1}.ts # Tool for action 1
|
||||
├── {action2}.ts # Tool for action 2
|
||||
└── ...
|
||||
```
|
||||
|
||||
### Key Patterns
|
||||
|
||||
**types.ts:**
|
||||
```typescript
|
||||
import type { ToolResponse } from '@/tools/types'
|
||||
|
||||
export interface {Service}{Action}Params {
|
||||
accessToken: string // For OAuth services
|
||||
// OR
|
||||
apiKey: string // For API key services
|
||||
|
||||
requiredParam: string
|
||||
optionalParam?: string
|
||||
}
|
||||
|
||||
export interface {Service}Response extends ToolResponse {
|
||||
output: {
|
||||
// Define output structure
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Tool file pattern:**
|
||||
```typescript
|
||||
export const {service}{Action}Tool: ToolConfig<Params, Response> = {
|
||||
id: '{service}_{action}',
|
||||
name: '{Service} {Action}',
|
||||
description: '...',
|
||||
version: '1.0.0',
|
||||
|
||||
oauth: { required: true, provider: '{service}' }, // If OAuth
|
||||
|
||||
params: {
|
||||
accessToken: { type: 'string', required: true, visibility: 'hidden', description: '...' },
|
||||
// ... other params
|
||||
},
|
||||
|
||||
request: { url, method, headers, body },
|
||||
|
||||
transformResponse: async (response) => {
|
||||
const data = await response.json()
|
||||
return {
|
||||
success: true,
|
||||
output: {
|
||||
field: data.field ?? null, // Always handle nullables
|
||||
},
|
||||
}
|
||||
},
|
||||
|
||||
outputs: { /* ... */ },
|
||||
}
|
||||
```
|
||||
|
||||
### Critical Rules
|
||||
- `visibility: 'hidden'` for OAuth tokens
|
||||
- `visibility: 'user-only'` for API keys and user credentials
|
||||
- `visibility: 'user-or-llm'` for operation parameters
|
||||
- Always use `?? null` for nullable API response fields
|
||||
- Always use `?? []` for optional array fields
|
||||
- Set `optional: true` for outputs that may not exist
|
||||
- Never output raw JSON dumps - extract meaningful fields
|
||||
- When using `type: 'json'` and you know the object shape, define `properties` with the inner fields so downstream consumers know the structure. Only use bare `type: 'json'` when the shape is truly dynamic
|
||||
- If you do not know the response JSON shape from docs or verified examples, you MUST tell the user and stop. Never guess outputs or response mappings.
|
||||
|
||||
### Resolved Secrets at Model and Persistence Boundaries
|
||||
|
||||
Classify every request field before implementing the tool:
|
||||
|
||||
This is opt-in, not a blanket integration migration. Add a model-input declaration only when the
|
||||
service's official documentation or an unambiguous local execution path proves that the exact
|
||||
field is consumed by an AI model. If that cannot be established, preserve existing tool behavior
|
||||
and leave the field unannotated.
|
||||
|
||||
- **Ordinary provider/API input:** leave it unchanged. Explicit `{{...}}` references resolve and are
|
||||
sent with their normal request semantics. A URL, domain, resource ID, control field, or opaque
|
||||
payload is not model-visible merely because the provider is AI-backed or may process the
|
||||
referenced resource later.
|
||||
- **Text or structured content consumed by an AI model:** declare `request.modelInput` with
|
||||
`mode: 'project'` and select only the exact model-visible fields. The shared executor replaces
|
||||
activated Sim secrets with canonical `{{NAME}}` labels before request formatting. For nested or
|
||||
JSON-string fields, use a small shared selector plus `applyProjected`; verify that selecting the
|
||||
rebuilt params reproduces the projected selection.
|
||||
- **Serialized model content sent directly to an external provider:** include the serialized
|
||||
top-level param in `request.modelInput`. Project the private copy before the existing request
|
||||
formatter parses it; keep formatter behavior deterministic when a whole-value placeholder is not
|
||||
valid in the serialized grammar. Do not introduce a second hard-rejection path.
|
||||
- **Opaque model input owned by an authenticated internal route** such as inline audio, image,
|
||||
video, or document bytes: add `privateProvenance` to a projected request, or use
|
||||
`mode: 'private-provenance'` when there is no textual projection. Do not select storage keys,
|
||||
paths, signed URLs, or ordinary remote URLs as byte provenance; the owning route must authorize
|
||||
stored bytes independently at model egress. The route must call
|
||||
`validateOpaqueModelInputProvenance` before downloading or sending content to the model and must
|
||||
apply the workspace-file provenance guard before reading a persisted workspace file.
|
||||
- **Sim-owned durable storage or internal execution handoff** that can later enter a workflow/model
|
||||
(table cells, Agent memory, knowledge documents/chunks, workspace-file contents, or child-workflow
|
||||
input): transport encrypted field-scoped provenance with `request.secretProvenance`. The
|
||||
authenticated receiver validates the exact selection and scope, strips the private envelope, and
|
||||
persists, imports, or propagates it at the owning boundary. Preserve shared legacy behavior for
|
||||
headerless internal calls and rows/files whose provenance marker is `NULL`; never invent a
|
||||
tool-local migration rule.
|
||||
|
||||
Hard rules:
|
||||
|
||||
- Never substitute secret plaintext into source or serialize plaintext provenance.
|
||||
- Never hand-roll private provenance headers/envelopes; the shared `executeTool` boundary owns
|
||||
transport and strips private metadata from functional results.
|
||||
- Never attach private provenance to an external URL or to `directExecution`. Project proven
|
||||
model-visible external fields with `request.modelInput`; otherwise preserve ordinary request
|
||||
semantics. Use an authenticated internal route when encrypted provenance must cross the boundary.
|
||||
- Never sanitize arbitrary third-party tool results. Projection applies only to secrets activated
|
||||
by Sim's resolved-secret provenance for that execution/tool call.
|
||||
- Do not add provenance merely because a value is persisted, returned by a tool, or appears in a
|
||||
filename. Require a concrete Sim `{{...}}` resolution path and a later model/log boundary. If an
|
||||
unsupported field can resolve a secret but does not justify durable tracking (for example a
|
||||
`file_write` path), reject it at that exact ingress.
|
||||
- At diagnostic boundaries, project only values carrying execution-scoped provenance. Ordinary
|
||||
provider responses, filenames, URLs, and errors remain unchanged when Sim did not resolve a
|
||||
secret into them.
|
||||
|
||||
Add focused tests covering named projection, ordinary identical text without provenance, nested and
|
||||
serialized shape handling, unchanged ordinary external inputs, malformed/incomplete private metadata
|
||||
failing closed, headerless legacy requests, and absence of private metadata in the public tool result.
|
||||
For durable sinks, also cover legacy `NULL` markers, exact-empty new writes, tracked secret writes,
|
||||
stale/missing sidecars, and scope isolation.
|
||||
|
||||
## Step 3: Create Block
|
||||
|
||||
### File Location
|
||||
`apps/sim/blocks/blocks/{service}.ts`
|
||||
|
||||
### Block Structure
|
||||
```typescript
|
||||
import { {Service}Icon } from '@/components/icons'
|
||||
import type { BlockConfig } from '@/blocks/types'
|
||||
import { AuthMode, IntegrationType } from '@/blocks/types'
|
||||
import { getScopesForService } from '@/lib/oauth/utils'
|
||||
|
||||
export const {Service}Block: BlockConfig = {
|
||||
type: '{service}',
|
||||
name: '{Service}',
|
||||
description: '...',
|
||||
longDescription: '...',
|
||||
docsLink: 'https://docs.sim.ai/integrations/{service}',
|
||||
category: 'tools',
|
||||
integrationType: IntegrationType.X, // Primary category (see IntegrationType enum)
|
||||
tags: ['oauth', 'api'], // Cross-cutting tags (see IntegrationTag type)
|
||||
bgColor: '#HEXCOLOR',
|
||||
icon: {Service}Icon,
|
||||
authMode: AuthMode.OAuth, // or AuthMode.ApiKey
|
||||
|
||||
subBlocks: [
|
||||
// Operation dropdown
|
||||
{
|
||||
id: 'operation',
|
||||
title: 'Operation',
|
||||
type: 'dropdown',
|
||||
options: [
|
||||
{ label: 'Operation 1', id: 'action1' },
|
||||
{ label: 'Operation 2', id: 'action2' },
|
||||
],
|
||||
value: () => 'action1',
|
||||
},
|
||||
// Credential field
|
||||
{
|
||||
id: 'credential',
|
||||
title: '{Service} Account',
|
||||
type: 'oauth-input',
|
||||
serviceId: '{service}',
|
||||
requiredScopes: getScopesForService('{service}'),
|
||||
required: true,
|
||||
},
|
||||
// Conditional fields per operation
|
||||
// ...
|
||||
],
|
||||
|
||||
tools: {
|
||||
access: ['{service}_action1', '{service}_action2'],
|
||||
config: {
|
||||
tool: (params) => `{service}_${params.operation}`,
|
||||
},
|
||||
},
|
||||
|
||||
outputs: { /* ... */ },
|
||||
}
|
||||
```
|
||||
|
||||
### Key SubBlock Patterns
|
||||
|
||||
**Condition-based visibility:**
|
||||
```typescript
|
||||
{
|
||||
id: 'resourceId',
|
||||
title: 'Resource ID',
|
||||
type: 'short-input',
|
||||
condition: { field: 'operation', value: ['read', 'update', 'delete'] },
|
||||
required: { field: 'operation', value: ['read', 'update', 'delete'] },
|
||||
}
|
||||
```
|
||||
|
||||
**DependsOn for cascading selectors:**
|
||||
```typescript
|
||||
{
|
||||
id: 'project',
|
||||
type: 'project-selector',
|
||||
dependsOn: ['credential'],
|
||||
},
|
||||
{
|
||||
id: 'issue',
|
||||
type: 'file-selector',
|
||||
dependsOn: ['credential', 'project'],
|
||||
}
|
||||
```
|
||||
|
||||
**Basic/Advanced mode for dual UX:**
|
||||
```typescript
|
||||
// Basic: Visual selector
|
||||
{
|
||||
id: 'channelSelector',
|
||||
type: 'channel-selector',
|
||||
mode: 'basic',
|
||||
canonicalParamId: 'channel',
|
||||
dependsOn: ['credential'],
|
||||
},
|
||||
// Advanced: Manual input
|
||||
{
|
||||
id: 'channelId',
|
||||
type: 'short-input',
|
||||
mode: 'advanced',
|
||||
canonicalParamId: 'channel',
|
||||
}
|
||||
```
|
||||
|
||||
Note neither subblock `id` is `channel` — the canonical id is a third name that both members map
|
||||
onto, and it is the only one that survives serialization.
|
||||
|
||||
**Critical Canonical Param Rules:**
|
||||
- `canonicalParamId` must NOT match any subblock's `id` in the block
|
||||
- `canonicalParamId` must be unique **block-wide**, not per operation. `buildCanonicalIndex` keys
|
||||
groups by `canonicalParamId` across all subblocks and a group holds exactly one `basicId`, so two
|
||||
operations that each need their own pair must use two different canonical ids
|
||||
- Only use `canonicalParamId` to link basic/advanced alternatives for the same logical parameter.
|
||||
A pair carries ONE concept — for files that means upload (basic) + file reference (advanced), as
|
||||
in Gmail attachments (`blocks/blocks/gmail.ts`). Never overload the advanced side with alternate
|
||||
identifiers like a URL or a provider asset ID; give those their own subblocks, mark all the
|
||||
mutually exclusive sources `required: false`, and enforce "exactly one" at execution
|
||||
- `mode` only controls UI visibility, NOT serialization. Without `canonicalParamId`, both basic and advanced field values would be sent
|
||||
- Every subblock `id` must be unique within the block. Duplicate IDs cause conflicts even with different conditions
|
||||
- **Required consistency:** If one subblock in a canonical group has `required: true`, ALL subblocks in that group must have `required: true` (prevents bypassing validation by switching modes)
|
||||
- **Inputs section:** Must list canonical param IDs (e.g., `fileId`), NOT raw subblock IDs (e.g., `fileSelector`, `manualFileId`)
|
||||
- **Params function:** Must use canonical param IDs, NOT raw subblock IDs (raw IDs are deleted after canonical transformation)
|
||||
|
||||
### BlockMeta (Required)
|
||||
|
||||
Export a `{Service}BlockMeta` in the same file as the block — **minimum 7 templates**. See `.agents/skills/add-block/SKILL.md` → "BlockMeta (Required)" for valid `modules` and `category` values and the full pattern.
|
||||
|
||||
```typescript
|
||||
export const {Service}BlockMeta = {
|
||||
tags: ['tag1', 'tag2'],
|
||||
templates: [
|
||||
{
|
||||
icon: {Service}Icon,
|
||||
title: '{Service} <use-case>',
|
||||
prompt: 'Build a workflow that...', // concrete trigger → transformation → output
|
||||
modules: ['agent', 'workflows'],
|
||||
category: 'operations',
|
||||
tags: ['automation'],
|
||||
alsoIntegrations: ['slack'], // when the prompt references another service
|
||||
},
|
||||
// ... at least 6 more
|
||||
],
|
||||
} as const satisfies BlockMeta
|
||||
```
|
||||
|
||||
## Step 4: Add Icon
|
||||
|
||||
### File Location
|
||||
`apps/sim/components/icons.tsx`
|
||||
|
||||
### Pattern
|
||||
```typescript
|
||||
export function {Service}Icon(props: SVGProps<SVGSVGElement>) {
|
||||
return (
|
||||
<svg
|
||||
{...props}
|
||||
viewBox="0 0 24 24"
|
||||
fill="none"
|
||||
xmlns="http://www.w3.org/2000/svg"
|
||||
>
|
||||
{/* SVG paths from user-provided SVG */}
|
||||
</svg>
|
||||
)
|
||||
}
|
||||
```
|
||||
|
||||
### Getting Icons
|
||||
**Do NOT search for icons yourself.** At the end of implementation, ask the user to provide the SVG:
|
||||
|
||||
```
|
||||
I've completed the integration. Before I can add the icon, please provide the SVG for {Service}.
|
||||
You can usually find this in the service's brand/press kit page, or copy it from their website.
|
||||
|
||||
Paste the SVG code here and I'll convert it to a React component.
|
||||
```
|
||||
|
||||
Once the user provides the SVG:
|
||||
1. Extract the SVG paths/content
|
||||
2. Create a React component that spreads props
|
||||
3. Ensure viewBox is preserved from the original SVG
|
||||
|
||||
### Theme-safety (bare rendering) — REQUIRED
|
||||
|
||||
The icon renders both inside its colored `bgColor` tile AND "bare" (no tile) on a
|
||||
neutral page — e.g. the home **Suggested actions** list — in both light and dark
|
||||
mode. A monochrome logo whose paths hardcode a single near-white or near-black
|
||||
fill is invisible bare on the matching background (white-on-white in light mode,
|
||||
black-on-black in dark mode).
|
||||
|
||||
Rules when adding the SVG:
|
||||
|
||||
- **Monochrome logos** (a single white or black mark): draw the shape with
|
||||
`fill='currentColor'`, not `fill='#fff'` / `fill='#000000'`. It then inherits
|
||||
white inside dark tiles, near-black inside light tiles (via
|
||||
`getTileIconColorClass`), and the theme-aware `var(--text-icon)` bare — legible
|
||||
everywhere. Do NOT set `iconColor` for these.
|
||||
- **Multi-color brand logos** (their own vivid fills): keep the hardcoded fills.
|
||||
They read on any background. Only set `iconColor` (a vivid brand hex, never a
|
||||
near-black/near-white tile color) if the bare icon should adopt a brand tint.
|
||||
- A large white shape with a tiny vivid accent (e.g. a logo where the body is the
|
||||
white negative space) still vanishes bare — convert the body to `currentColor`.
|
||||
|
||||
Verify with `bun run check:bare-icons` (also runs in CI). It flags purely
|
||||
monochrome hazards; for partial-accent logos, eyeball the suggested-actions list
|
||||
in both light and dark mode.
|
||||
|
||||
## Step 5: Create Triggers (Optional)
|
||||
|
||||
If the service supports webhooks, create triggers using the generic `buildTriggerSubBlocks` helper.
|
||||
|
||||
### Directory Structure
|
||||
```
|
||||
apps/sim/triggers/{service}/
|
||||
├── index.ts # Barrel exports
|
||||
├── utils.ts # Trigger options, setup instructions, extra fields
|
||||
├── {event_a}.ts # Primary trigger (includes dropdown)
|
||||
├── {event_b}.ts # Secondary triggers (no dropdown)
|
||||
└── webhook.ts # Generic webhook (optional)
|
||||
```
|
||||
|
||||
### Key Pattern
|
||||
|
||||
```typescript
|
||||
import { buildTriggerSubBlocks } from '@/triggers'
|
||||
import { {service}TriggerOptions, {service}SetupInstructions, build{Service}ExtraFields } from './utils'
|
||||
|
||||
// Primary trigger - includeDropdown: true
|
||||
export const {service}EventATrigger: TriggerConfig = {
|
||||
id: '{service}_event_a',
|
||||
subBlocks: buildTriggerSubBlocks({
|
||||
triggerId: '{service}_event_a',
|
||||
triggerOptions: {service}TriggerOptions,
|
||||
includeDropdown: true, // Only for primary trigger!
|
||||
setupInstructions: {service}SetupInstructions('Event A'),
|
||||
extraFields: build{Service}ExtraFields('{service}_event_a'),
|
||||
}),
|
||||
// ...
|
||||
}
|
||||
|
||||
// Secondary triggers - no dropdown
|
||||
export const {service}EventBTrigger: TriggerConfig = {
|
||||
id: '{service}_event_b',
|
||||
subBlocks: buildTriggerSubBlocks({
|
||||
triggerId: '{service}_event_b',
|
||||
triggerOptions: {service}TriggerOptions,
|
||||
// No includeDropdown!
|
||||
setupInstructions: {service}SetupInstructions('Event B'),
|
||||
extraFields: build{Service}ExtraFields('{service}_event_b'),
|
||||
}),
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
### Connect to Block
|
||||
```typescript
|
||||
import { getTrigger } from '@/triggers'
|
||||
|
||||
export const {Service}Block: BlockConfig = {
|
||||
triggers: {
|
||||
enabled: true,
|
||||
available: ['{service}_event_a', '{service}_event_b'],
|
||||
},
|
||||
subBlocks: [
|
||||
// Tool fields...
|
||||
...getTrigger('{service}_event_a').subBlocks,
|
||||
...getTrigger('{service}_event_b').subBlocks,
|
||||
],
|
||||
}
|
||||
```
|
||||
|
||||
See `/add-trigger` skill for complete documentation.
|
||||
|
||||
## Step 6: Register Everything
|
||||
|
||||
### Tools Registry (`apps/sim/tools/registry.ts`)
|
||||
|
||||
```typescript
|
||||
// Add import (alphabetically)
|
||||
import {
|
||||
{service}Action1Tool,
|
||||
{service}Action2Tool,
|
||||
} from '@/tools/{service}'
|
||||
|
||||
// Add to tools object (alphabetically)
|
||||
export const tools: Record<string, ToolConfig> = {
|
||||
// ... existing tools ...
|
||||
{service}_action1: {service}Action1Tool,
|
||||
{service}_action2: {service}Action2Tool,
|
||||
}
|
||||
```
|
||||
|
||||
Then regenerate the generated tool metadata and commit it:
|
||||
|
||||
```bash
|
||||
bun run tool-metadata:generate
|
||||
```
|
||||
|
||||
Client code reads `params`/`outputs` from these artifacts rather than importing
|
||||
the registry, so a tool you add, change or remove is invisible to the UI until they are regenerated,
|
||||
and CI fails on stale ones. See `.agents/skills/tool-registry-boundary/SKILL.md`.
|
||||
|
||||
### Block Registry (`apps/sim/blocks/registry-maps.ts`)
|
||||
|
||||
The data maps (`BLOCK_REGISTRY` + `BLOCK_META_REGISTRY`) live in `registry-maps.ts`; `registry.ts` holds only the accessor functions. Add the import and an entry to each map alphabetically:
|
||||
|
||||
```typescript
|
||||
// Add import (alphabetically)
|
||||
import { {Service}Block, {Service}BlockMeta } from '@/blocks/blocks/{service}'
|
||||
|
||||
// Add to the config map (alphabetically)
|
||||
export const BLOCK_REGISTRY: Record<string, BlockConfig> = {
|
||||
// ... existing blocks ...
|
||||
{service}: {Service}Block,
|
||||
}
|
||||
|
||||
// Add to the catalog-meta map (alphabetically)
|
||||
export const BLOCK_META_REGISTRY: Record<string, BlockMeta> = {
|
||||
// ... existing metas ...
|
||||
{service}: {Service}BlockMeta,
|
||||
}
|
||||
```
|
||||
|
||||
### Trigger Registry (`apps/sim/triggers/registry.ts`) - If triggers exist
|
||||
|
||||
```typescript
|
||||
// Add import (alphabetically)
|
||||
import {
|
||||
{service}EventATrigger,
|
||||
{service}EventBTrigger,
|
||||
{service}WebhookTrigger,
|
||||
} from '@/triggers/{service}'
|
||||
|
||||
// Add to TRIGGER_REGISTRY (alphabetically)
|
||||
export const TRIGGER_REGISTRY: TriggerRegistry = {
|
||||
// ... existing triggers ...
|
||||
{service}_event_a: {service}EventATrigger,
|
||||
{service}_event_b: {service}EventBTrigger,
|
||||
{service}_webhook: {service}WebhookTrigger,
|
||||
}
|
||||
```
|
||||
|
||||
## Step 7: Configure Deployment Availability
|
||||
|
||||
Do this for every visible OAuth integration. API-key and unauthenticated integrations do not need
|
||||
an OAuth client capability.
|
||||
|
||||
The block's `oauth-input.serviceId` is the canonical link between the generated integration catalog,
|
||||
the OAuth service configuration, deployment availability, and the setup CLI.
|
||||
|
||||
1. Ensure the block has exactly one distinct OAuth `serviceId` and that it matches the canonical
|
||||
service entry in `apps/sim/lib/oauth/oauth.ts`.
|
||||
2. Confirm `resolveOAuthClientCapabilityId(serviceId)` resolves to the intended provider entry in
|
||||
`OAUTH_CLIENT_CAPABILITIES` in `apps/sim/lib/core/config/env-capabilities.ts`. Google and
|
||||
Microsoft service IDs deliberately share provider-level capabilities.
|
||||
3. For a new OAuth provider, add the required client fields to `OAUTH_CLIENT_CAPABILITIES`, add
|
||||
every referenced field to the env schema in `apps/sim/lib/core/config/env.ts`, and add the
|
||||
matching `text` or `secret` entries to `OAUTH_CLIENT_SETUP_FIELDS` in
|
||||
`scripts/setup/capability-config.ts`. Do not create integration-specific setup logic or infer
|
||||
secret fields from naming; the CLI mapping is exhaustively checked against the runtime fields.
|
||||
4. If the canonical OAuth service has `serviceAccountProviderId`, add the matching projection to
|
||||
`SERVICE_ACCOUNT_METADATA_BY_OAUTH_SERVICE_ID` in
|
||||
`apps/sim/lib/integrations/service-account-metadata.ts`. Use:
|
||||
- no `deploymentRequirement` when the service-account path works independently of OAuth client fields;
|
||||
- `'oauth-client'` when it requires the same deployment OAuth client fields;
|
||||
- `'preview-gated'` when availability is controlled by the service-account preview block.
|
||||
|
||||
Never add a permissive fallback for missing capability metadata. A visible OAuth integration without
|
||||
a resolvable capability must fail validation.
|
||||
|
||||
## Step 8: Generate and Validate the Catalog
|
||||
|
||||
Run the documentation generator:
|
||||
```bash
|
||||
bun run scripts/generate-docs.ts
|
||||
bun run integration-catalog:check
|
||||
```
|
||||
|
||||
This creates `apps/docs/content/docs/en/integrations/{service}.mdx` — one page per service carrying the block's Actions and, if it has one, its Triggers section. Never hand-edit generated pages; the only editable region is the `{/* MANUAL-CONTENT */}` block (see `scripts/README.md`).
|
||||
|
||||
The same generator refreshes `apps/sim/lib/integrations/integrations.json`. The catalog check then
|
||||
derives the deployment-relevant fields from the executable block registry and compares them with the
|
||||
committed projection. Review the generated diff and keep only intentional changes.
|
||||
|
||||
## V2 Integration Pattern
|
||||
|
||||
If creating V2 versions (API-aligned outputs):
|
||||
|
||||
1. **V2 Tools** - Add `_v2` suffix, version `2.0.0`, flat outputs
|
||||
2. **V2 Block** - Add `_v2` type, use `createVersionedToolSelector`
|
||||
3. **V1 Block** - Add `(Legacy)` to name, set `hideFromToolbar: true`
|
||||
4. **Registry** - Register both versions
|
||||
|
||||
```typescript
|
||||
// In registry
|
||||
{service}: {Service}Block, // V1 (legacy, hidden)
|
||||
{service}_v2: {Service}V2Block, // V2 (visible)
|
||||
```
|
||||
|
||||
## Complete Checklist
|
||||
|
||||
### Tools
|
||||
- [ ] Created `tools/{service}/` directory
|
||||
- [ ] Created `types.ts` with all interfaces
|
||||
- [ ] Created tool file for each operation
|
||||
- [ ] All params have correct visibility
|
||||
- [ ] All nullable fields use `?? null`
|
||||
- [ ] All optional outputs have `optional: true`
|
||||
- [ ] Created `index.ts` barrel export
|
||||
- [ ] Registered all tools in `tools/registry.ts`
|
||||
- [ ] Ran `bun run tool-metadata:generate` and committed the regenerated artifacts
|
||||
- [ ] Classified every model-visible, opaque, Sim-durable, and internal-execution request field
|
||||
- [ ] Added shared model-input projection or private provenance only where required; ordinary
|
||||
external resource locators and control inputs retain their request semantics
|
||||
- [ ] Confirmed ordinary third-party tool results are not generically sanitized
|
||||
- [ ] Added provenance compatibility and fail-closed boundary tests where applicable
|
||||
|
||||
### Block
|
||||
- [ ] Created `blocks/blocks/{service}.ts`
|
||||
- [ ] Set `integrationType` to the correct `IntegrationType` enum value
|
||||
- [ ] Set `tags` array with all applicable `IntegrationTag` values
|
||||
- [ ] Defined operation dropdown with all operations
|
||||
- [ ] Added credential field with `requiredScopes: getScopesForService('{service}')`
|
||||
- [ ] Added conditional fields per operation
|
||||
- [ ] Set up dependsOn for cascading selectors
|
||||
- [ ] Configured tools.access with all tool IDs
|
||||
- [ ] Configured tools.config.tool selector
|
||||
- [ ] Defined outputs matching tool outputs
|
||||
- [ ] Registered block + meta in `blocks/registry-maps.ts` (`BLOCK_REGISTRY` / `BLOCK_META_REGISTRY`)
|
||||
- [ ] If triggers: set `triggers.enabled` and `triggers.available`
|
||||
- [ ] If triggers: spread trigger subBlocks with `getTrigger()`
|
||||
- [ ] Exported `{Service}BlockMeta` with at least 7 templates
|
||||
|
||||
### OAuth Scopes (if OAuth service)
|
||||
- [ ] Defined scopes in `lib/oauth/oauth.ts` under `OAUTH_PROVIDERS`
|
||||
- [ ] Added scope descriptions in `SCOPE_DESCRIPTIONS` within `lib/oauth/utils.ts`
|
||||
- [ ] Used `getCanonicalScopesForProvider()` in `auth.ts` (never hardcode)
|
||||
- [ ] Used `getScopesForService()` in block `requiredScopes` (never hardcode)
|
||||
|
||||
### Deployment Availability (if OAuth service)
|
||||
- [ ] Block declares exactly one distinct `oauth-input.serviceId`
|
||||
- [ ] `resolveOAuthClientCapabilityId(serviceId)` resolves to the intended `OAUTH_CLIENT_CAPABILITIES` entry
|
||||
- [ ] Every new OAuth capability field exists in `apps/sim/lib/core/config/env.ts`
|
||||
- [ ] Runtime OAuth fields live in `OAUTH_CLIENT_CAPABILITIES`; matching CLI input modes live in the exhaustively checked `OAUTH_CLIENT_SETUP_FIELDS`
|
||||
- [ ] If `serviceAccountProviderId` is configured, `SERVICE_ACCOUNT_METADATA_BY_OAUTH_SERVICE_ID` has the matching projection and deployment requirement
|
||||
|
||||
### Icon
|
||||
- [ ] Asked user to provide SVG
|
||||
- [ ] Added icon to `components/icons.tsx`
|
||||
- [ ] Icon spreads props correctly
|
||||
- [ ] Monochrome marks use `fill='currentColor'` (not hardcoded white/black) so the icon renders bare in light AND dark mode — verified with `bun run check:bare-icons`
|
||||
|
||||
### Triggers (if service supports webhooks)
|
||||
- [ ] Created `triggers/{service}/` directory
|
||||
- [ ] Created `utils.ts` with options, instructions, and extra fields helpers
|
||||
- [ ] Primary trigger uses `includeDropdown: true`
|
||||
- [ ] Secondary triggers do NOT have `includeDropdown`
|
||||
- [ ] All triggers use `buildTriggerSubBlocks` helper
|
||||
- [ ] Created `index.ts` barrel export
|
||||
- [ ] Registered all triggers in `triggers/registry.ts`
|
||||
|
||||
### Docs
|
||||
- [ ] Ran `bun run scripts/generate-docs.ts`
|
||||
- [ ] Verified docs file created
|
||||
- [ ] Reviewed and committed the generated `apps/sim/lib/integrations/integrations.json` change
|
||||
- [ ] `bun run integration-catalog:check` passes
|
||||
|
||||
### Final Validation (Required)
|
||||
- [ ] Read every tool file and cross-referenced inputs/outputs against the API docs
|
||||
- [ ] Verified block subBlocks cover all required tool params with correct conditions
|
||||
- [ ] Verified block outputs match what the tools actually return
|
||||
- [ ] Verified `tools.config.params` correctly maps and coerces all param types
|
||||
- [ ] Verified every tool output and `transformResponse` path against documented or live-verified JSON responses
|
||||
- [ ] If any response schema remained unknown, explicitly told the user instead of guessing
|
||||
- [ ] `{Service}BlockMeta` exported with at least 7 templates, each having `icon`, `title`, `prompt`, `modules`, `category`, and `tags`
|
||||
|
||||
## Example Command
|
||||
|
||||
When the user asks to add an integration:
|
||||
|
||||
```
|
||||
User: Add a Stripe integration
|
||||
|
||||
You: I'll add the Stripe integration. Let me:
|
||||
|
||||
1. First, research the Stripe API using Context7
|
||||
2. Create the tools for key operations (payments, subscriptions, etc.)
|
||||
3. Create the block with operation dropdown
|
||||
4. Register everything
|
||||
5. Generate docs
|
||||
6. Ask you for the Stripe icon SVG
|
||||
|
||||
[Proceed with implementation...]
|
||||
|
||||
[After completing steps 1-5...]
|
||||
|
||||
I've completed the Stripe integration. Before I can add the icon, please provide the SVG for Stripe.
|
||||
You can usually find this in the service's brand/press kit page, or copy it from their website.
|
||||
|
||||
Paste the SVG code here and I'll convert it to a React component.
|
||||
```
|
||||
|
||||
## File Handling
|
||||
|
||||
When your integration handles file uploads or downloads, follow these patterns to work with `UserFile` objects consistently.
|
||||
|
||||
### What is a UserFile?
|
||||
|
||||
A `UserFile` is the standard file representation in Sim:
|
||||
|
||||
```typescript
|
||||
interface UserFile {
|
||||
id: string // Unique identifier
|
||||
name: string // Original filename
|
||||
url: string // Presigned URL for download
|
||||
size: number // File size in bytes
|
||||
type: string // MIME type (e.g., 'application/pdf')
|
||||
base64?: string // Optional base64 content (if small file)
|
||||
key?: string // Internal storage key
|
||||
context?: object // Storage context metadata
|
||||
}
|
||||
```
|
||||
|
||||
### File Input Pattern (Uploads)
|
||||
|
||||
For tools that accept file uploads, **always route through an internal API endpoint** rather than calling external APIs directly. This ensures proper file content retrieval.
|
||||
|
||||
#### 1. Block SubBlocks for File Input
|
||||
|
||||
Use the basic/advanced mode pattern:
|
||||
|
||||
```typescript
|
||||
// Basic mode: File upload UI
|
||||
{
|
||||
id: 'uploadFile',
|
||||
title: 'File',
|
||||
type: 'file-upload',
|
||||
canonicalParamId: 'file', // Maps to 'file' param
|
||||
placeholder: 'Upload file',
|
||||
mode: 'basic',
|
||||
multiple: false,
|
||||
required: true,
|
||||
condition: { field: 'operation', value: 'upload' },
|
||||
},
|
||||
// Advanced mode: Reference from previous block
|
||||
{
|
||||
id: 'fileRef',
|
||||
title: 'File',
|
||||
type: 'short-input',
|
||||
canonicalParamId: 'file', // Same canonical param
|
||||
placeholder: 'Reference file (e.g., {{file_block.output}})',
|
||||
mode: 'advanced',
|
||||
required: true,
|
||||
condition: { field: 'operation', value: 'upload' },
|
||||
},
|
||||
```
|
||||
|
||||
**Critical:** `canonicalParamId` must NOT match any subblock `id`.
|
||||
|
||||
#### 2. Normalize File Input in Block Config
|
||||
|
||||
In `tools.config.tool`, use `normalizeFileInput` to handle all input variants:
|
||||
|
||||
```typescript
|
||||
import { normalizeFileInput } from '@/blocks/utils'
|
||||
|
||||
tools: {
|
||||
config: {
|
||||
tool: (params) => {
|
||||
// Normalize file from basic (uploadFile), advanced (fileRef), or legacy (fileContent)
|
||||
const normalizedFile = normalizeFileInput(
|
||||
params.uploadFile || params.fileRef || params.fileContent,
|
||||
{ single: true }
|
||||
)
|
||||
if (normalizedFile) {
|
||||
params.file = normalizedFile
|
||||
}
|
||||
return `{service}_${params.operation}`
|
||||
},
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
#### 3. Create Internal API Route
|
||||
|
||||
Create `apps/sim/app/api/tools/{service}/{action}/route.ts`. Internal tool routes are HTTP boundaries and follow the same contract policy as public routes — define the request/response shape in `apps/sim/lib/api/contracts/tools/{service}.ts` (or an existing aggregate) and validate with canonical helpers from `@/lib/api/server`. Never write a route-local Zod schema.
|
||||
|
||||
```typescript
|
||||
// apps/sim/lib/api/contracts/tools/{service}.ts
|
||||
import { z } from 'zod'
|
||||
import { defineRouteContract } from '@/lib/api/contracts'
|
||||
import { FileInputSchema } from '@/lib/uploads/utils/file-schemas'
|
||||
|
||||
export const {service}UploadBodySchema = z.object({
|
||||
accessToken: z.string(),
|
||||
file: FileInputSchema.optional().nullable(),
|
||||
fileContent: z.string().optional().nullable(),
|
||||
// ... other params
|
||||
})
|
||||
|
||||
export const {service}UploadResponseSchema = z.object({
|
||||
success: z.boolean(),
|
||||
output: z.object({ id: z.string(), url: z.string() }).optional(),
|
||||
error: z.string().optional(),
|
||||
})
|
||||
|
||||
export const {service}UploadContract = defineRouteContract({
|
||||
method: 'POST',
|
||||
path: '/api/tools/{service}/upload',
|
||||
body: {service}UploadBodySchema,
|
||||
response: { mode: 'json', schema: {service}UploadResponseSchema },
|
||||
})
|
||||
|
||||
export type {Service}UploadBody = z.input<typeof {service}UploadBodySchema>
|
||||
export type {Service}UploadResponse = z.output<typeof {service}UploadResponseSchema>
|
||||
```
|
||||
|
||||
```typescript
|
||||
// apps/sim/app/api/tools/{service}/upload/route.ts
|
||||
import { createLogger } from '@sim/logger'
|
||||
import { NextResponse, type NextRequest } from 'next/server'
|
||||
import { {service}UploadContract } from '@/lib/api/contracts/tools/{service}'
|
||||
import { parseRequest } from '@/lib/api/server'
|
||||
import { checkInternalAuth } from '@/lib/auth/hybrid'
|
||||
import { generateRequestId } from '@/lib/core/utils/request'
|
||||
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
|
||||
import { type RawFileInput } from '@/lib/uploads/utils/file-schemas'
|
||||
import { processFilesToUserFiles } from '@/lib/uploads/utils/file-utils'
|
||||
import { downloadFileFromStorage } from '@/lib/uploads/utils/file-utils.server'
|
||||
|
||||
const logger = createLogger('{Service}UploadAPI')
|
||||
|
||||
export const POST = withRouteHandler(async (request: NextRequest) => {
|
||||
const requestId = generateRequestId()
|
||||
|
||||
// Auth always runs BEFORE parseRequest — never validate untrusted input before authenticating.
|
||||
const authResult = await checkInternalAuth(request, { requireWorkflowId: false })
|
||||
if (!authResult.success) {
|
||||
return NextResponse.json({ success: false, error: 'Unauthorized' }, { status: 401 })
|
||||
}
|
||||
|
||||
const parsed = await parseRequest({service}UploadContract, request, {})
|
||||
if (!parsed.success) return parsed.response
|
||||
const data = parsed.data.body
|
||||
|
||||
let fileBuffer: Buffer
|
||||
let fileName: string
|
||||
|
||||
// Prefer UserFile input, fall back to legacy base64
|
||||
if (data.file) {
|
||||
const userFiles = processFilesToUserFiles([data.file as RawFileInput], requestId, logger)
|
||||
if (userFiles.length === 0) {
|
||||
return NextResponse.json({ success: false, error: 'Invalid file' }, { status: 400 })
|
||||
}
|
||||
const userFile = userFiles[0]
|
||||
fileBuffer = await downloadFileFromStorage(userFile, requestId, logger)
|
||||
fileName = userFile.name
|
||||
} else if (data.fileContent) {
|
||||
// Legacy: base64 string (backwards compatibility)
|
||||
fileBuffer = Buffer.from(data.fileContent, 'base64')
|
||||
fileName = 'file'
|
||||
} else {
|
||||
return NextResponse.json({ success: false, error: 'File required' }, { status: 400 })
|
||||
}
|
||||
|
||||
// Now call external API with fileBuffer
|
||||
const response = await fetch('https://api.{service}.com/upload', {
|
||||
method: 'POST',
|
||||
headers: { Authorization: `Bearer ${data.accessToken}` },
|
||||
body: new Uint8Array(fileBuffer), // Convert Buffer for fetch
|
||||
})
|
||||
|
||||
// ... handle response
|
||||
})
|
||||
```
|
||||
|
||||
#### 4. Update Tool to Use Internal Route
|
||||
|
||||
```typescript
|
||||
export const {service}UploadTool: ToolConfig<Params, Response> = {
|
||||
id: '{service}_upload',
|
||||
// ...
|
||||
params: {
|
||||
file: { type: 'file', required: false, visibility: 'user-or-llm' },
|
||||
fileContent: { type: 'string', required: false, visibility: 'hidden' }, // Legacy
|
||||
},
|
||||
request: {
|
||||
url: '/api/tools/{service}/upload', // Internal route
|
||||
method: 'POST',
|
||||
body: (params) => ({
|
||||
accessToken: params.accessToken,
|
||||
file: params.file,
|
||||
fileContent: params.fileContent,
|
||||
}),
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
### File Output Pattern (Downloads)
|
||||
|
||||
For tools that return files, use `FileToolProcessor` to store files and return `UserFile` objects.
|
||||
|
||||
#### In Tool transformResponse
|
||||
|
||||
```typescript
|
||||
import { FileToolProcessor } from '@/executor/utils/file-tool-processor'
|
||||
|
||||
transformResponse: async (response, context) => {
|
||||
const data = await response.json()
|
||||
|
||||
// Process file outputs to UserFile objects
|
||||
const fileProcessor = new FileToolProcessor(context)
|
||||
const file = await fileProcessor.processFileData({
|
||||
data: data.content, // base64 or buffer
|
||||
mimeType: data.mimeType,
|
||||
filename: data.filename,
|
||||
})
|
||||
|
||||
return {
|
||||
success: true,
|
||||
output: { file },
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### In API Route (for complex file handling)
|
||||
|
||||
```typescript
|
||||
// Return file data that FileToolProcessor can handle
|
||||
return NextResponse.json({
|
||||
success: true,
|
||||
output: {
|
||||
file: {
|
||||
data: base64Content,
|
||||
mimeType: 'application/pdf',
|
||||
filename: 'document.pdf',
|
||||
},
|
||||
},
|
||||
})
|
||||
```
|
||||
|
||||
### Key Helpers Reference
|
||||
|
||||
| Helper | Location | Purpose |
|
||||
|--------|----------|---------|
|
||||
| `normalizeFileInput` | `@/blocks/utils` | Normalize file params in block config |
|
||||
| `processFilesToUserFiles` | `@/lib/uploads/utils/file-utils` | Convert raw inputs to UserFile[] |
|
||||
| `downloadFileFromStorage` | `@/lib/uploads/utils/file-utils.server` | Get file Buffer from UserFile |
|
||||
| `FileToolProcessor` | `@/executor/utils/file-tool-processor` | Process tool output files |
|
||||
| `isUserFile` | `@/lib/core/utils/user-file` | Type guard for UserFile objects |
|
||||
| `FileInputSchema` | `@/lib/uploads/utils/file-schemas` | Zod schema for file validation |
|
||||
|
||||
### Advanced Mode for Optional Fields
|
||||
|
||||
Optional fields that are rarely used should be set to `mode: 'advanced'` so they don't clutter the basic UI. Examples: pagination tokens, time range filters, sort order, max results, reply settings.
|
||||
|
||||
### WandConfig for Complex Inputs
|
||||
|
||||
Use `wandConfig` for fields that are hard to fill out manually:
|
||||
- **Timestamps**: Use `generationType: 'timestamp'` to inject current date context into the AI prompt
|
||||
- **JSON arrays**: Use `generationType: 'json-object'` for structured data
|
||||
- **Complex queries**: Use a descriptive prompt explaining the expected format
|
||||
|
||||
```typescript
|
||||
{
|
||||
id: 'startTime',
|
||||
title: 'Start Time',
|
||||
type: 'short-input',
|
||||
mode: 'advanced',
|
||||
wandConfig: {
|
||||
enabled: true,
|
||||
prompt: 'Generate an ISO 8601 timestamp. Return ONLY the timestamp string.',
|
||||
generationType: 'timestamp',
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
### OAuth Scopes (Centralized System)
|
||||
|
||||
Scopes are maintained in a single source of truth and reused everywhere:
|
||||
|
||||
1. **Define scopes** in `lib/oauth/oauth.ts` under `OAUTH_PROVIDERS[provider].services[service].scopes`
|
||||
2. **Add descriptions** in `SCOPE_DESCRIPTIONS` within `lib/oauth/utils.ts` for the OAuth modal UI
|
||||
3. **Reference in auth.ts** using `getCanonicalScopesForProvider(providerId)` from `@/lib/oauth/utils`
|
||||
4. **Reference in blocks** using `getScopesForService(serviceId)` from `@/lib/oauth/utils`
|
||||
|
||||
**Never hardcode scope arrays** in `auth.ts` or block `requiredScopes`. Always import from the centralized source.
|
||||
|
||||
```typescript
|
||||
// In auth.ts (Better Auth config)
|
||||
scopes: getCanonicalScopesForProvider('{service}'),
|
||||
|
||||
// In block credential sub-block
|
||||
requiredScopes: getScopesForService('{service}'),
|
||||
```
|
||||
|
||||
### Common Gotchas
|
||||
|
||||
1. **OAuth serviceId must match** - The `serviceId` in oauth-input must match the OAuth provider configuration
|
||||
2. **All tool IDs MUST be snake_case** - `stripe_create_payment`, not `stripeCreatePayment`. This applies to tool `id` fields, registry keys, `tools.access` arrays, and `tools.config.tool` return values
|
||||
3. **Block type is snake_case** - `type: 'stripe'`, not `type: 'Stripe'`
|
||||
4. **Alphabetical ordering** - Keep imports and registry entries alphabetically sorted
|
||||
5. **Required can be conditional** - Use `required: { field: 'op', value: 'create' }` instead of always true
|
||||
6. **DependsOn clears options** - When a dependency changes, selector options are refetched
|
||||
7. **Never pass Buffer directly to fetch** - Convert to `new Uint8Array(buffer)` for TypeScript compatibility
|
||||
8. **Always handle legacy file params** - Keep hidden `fileContent` params for backwards compatibility
|
||||
9. **Optional fields use advanced mode** - Set `mode: 'advanced'` on rarely-used optional fields
|
||||
10. **Complex inputs need wandConfig** - Timestamps, JSON arrays, and other hard-to-type values should have `wandConfig` enabled
|
||||
11. **Never hardcode scopes** - Use `getScopesForService()` in blocks and `getCanonicalScopesForProvider()` in auth.ts
|
||||
12. **Always add scope descriptions** - New scopes must have entries in `SCOPE_DESCRIPTIONS` within `lib/oauth/utils.ts`
|
||||
13. **OAuth service IDs need deployment capabilities** - Every visible OAuth integration must resolve through `OAUTH_CLIENT_CAPABILITIES`; shared Google/Microsoft aliases map to their provider capability
|
||||
14. **Keep runtime and presentation separate** - Runtime OAuth fields live in `env-capabilities.ts`; CLI input modes live in the exhaustively checked `scripts/setup/capability-config.ts` mapping
|
||||
@@ -0,0 +1,148 @@
|
||||
---
|
||||
description: Add or upgrade a curated, immutable managed CLI for Sim Function sandboxes, including client-safe catalog metadata, a pinned server-only installation recipe, checksum and executable verification, provider compatibility, PATH propagation, content-addressed image identity, and tests. Use when adding a CLI to the Sandbox managed-CLI selector or changing an existing managed CLI version or recipe.
|
||||
---
|
||||
|
||||
# Add a Managed CLI
|
||||
|
||||
Add CLIs through the curated registry. Never turn this surface into arbitrary commands or package names: system packages already cover validated Debian/APT coordinates, while managed CLIs require immutable artifacts and reproducible recipes.
|
||||
|
||||
## Read First
|
||||
|
||||
Read these live sources before editing; do not copy their current entries into this skill:
|
||||
|
||||
1. `apps/sim/lib/execution/remote-sandbox/cli-tools.ts` — persisted IDs and client-safe metadata.
|
||||
2. `apps/sim/lib/execution/remote-sandbox/cli-tools.server.ts` — server-only recipes and recipe helpers.
|
||||
3. `apps/sim/lib/execution/remote-sandbox/cli-tools.test.ts` — catalog and supply-chain invariants.
|
||||
4. `apps/sim/lib/execution/remote-sandbox/cli-tools-boundary.test.ts` — client/server import boundary.
|
||||
5. `apps/sim/lib/execution/remote-sandbox/sandbox-spec.ts` — content-addressed hash inputs.
|
||||
|
||||
Read `resolve.ts` and `e2b.ts` only when changing provisioning mechanics. A normal catalog addition should not require UI, API, database, resolver, or provider edits; those paths derive from the registries.
|
||||
|
||||
Do not modify the dedicated Function base image or the separate Mothership Shell template for a normal managed CLI addition. Managed recipes layer on the Function base image. Do not change `MAX_SANDBOX_CLI_TOOLS` from ten unless the user separately requests a product-limit change.
|
||||
|
||||
## 1. Verify the Upstream Release
|
||||
|
||||
Use primary upstream release documentation and official artifacts. Establish all of the following before writing code:
|
||||
|
||||
- Exact version and stable Linux x86-64 artifact URL. Never use `latest`, mutable redirects, or an unversioned installer.
|
||||
- SHA-256 for the exact artifact. Prefer a publisher-signed checksum; otherwise download the official artifact and compute it independently.
|
||||
- Archive layout and the exact executable paths to install.
|
||||
- Noninteractive, credential-free verification commands for every advertised executable, normally version commands.
|
||||
- Required PATH entries under `/opt/sim-cli`.
|
||||
- E2B and Daytona compatibility. Default to both only when the same Linux recipe works on both.
|
||||
|
||||
When the user does not name a version, select the current stable upstream release from primary sources and state the exact version chosen. Do not silently choose a prerelease or infer a version from an unverified secondary source.
|
||||
|
||||
Reject curl-to-shell installers, `npm install`, `pip install`, distro package repositories, arbitrary user commands, and artifacts from unofficial mirrors. Never place credentials, tokens, login commands, or account configuration in an image recipe or build log; authentication is runtime-only.
|
||||
|
||||
## 2. Choose an Immutable ID
|
||||
|
||||
Use `<tool>@<upstream-version>-r<recipe-revision>`.
|
||||
|
||||
- New upstream version: append a new ID ending in `-r1`.
|
||||
- Recipe-only change for the same upstream version: append `-r2`, `-r3`, and so on.
|
||||
- Never mutate or delete an existing ID or recipe. Persisted sandboxes must continue resolving to the bytes and behavior they selected.
|
||||
- On upgrade, retain the old ID and recipe and set its metadata to `selectable: false`. Only the newest version keeps the public label selectable.
|
||||
|
||||
Before shipping the first upgrade for a tool family, verify that editing a sandbox cannot leave both the retired and replacement IDs selected. If the generic selector and API validation do not already replace or reject colliding versions, address that once at the generic registry boundary with focused UI and contract tests; never special-case the individual CLI or silently install two versions that expose the same executable.
|
||||
|
||||
Recipe identity includes the ID, revision, and SHA-256 in the sandbox image hash. Keeping old entries is what makes that identity reproducible rather than merely cache-busting.
|
||||
|
||||
## 3. Add Client-Safe Metadata
|
||||
|
||||
In `cli-tools.ts`:
|
||||
|
||||
1. Append the ID to `SANDBOX_CLI_TOOL_IDS` in the same order used by the metadata and recipe registries.
|
||||
2. Add a `SANDBOX_CLI_TOOLS` entry whose key and `id` exactly match.
|
||||
3. Provide a unique selectable `label`, concise `description`, existing `category`, and useful executable/vendor aliases in `searchTerms`.
|
||||
4. Add a category only when no existing category is accurate, then ensure it has at least one selectable entry.
|
||||
|
||||
Keep this file safe for client bundles. It must not contain artifact URLs, checksums, install commands, verification commands, PATH recipes, provider SDKs, or imports from `cli-tools.server.ts`.
|
||||
|
||||
The API enum and searchable grouped selector derive from this registry. Do not add parallel option arrays or route-local wire types.
|
||||
|
||||
## 4. Add the Server-Only Recipe
|
||||
|
||||
In `cli-tools.server.ts`, use the narrowest existing helper:
|
||||
|
||||
- `defineBinaryRecipe` for one downloaded binary.
|
||||
- `defineTarGzipRecipe` or `defineZipRecipe` for archives containing binaries.
|
||||
- `defineVerifiedRecipe` for a vendor archive or installer layout that needs explicit commands.
|
||||
- A direct typed entry only when the helpers cannot faithfully model the release.
|
||||
|
||||
Provide every field the recipe contract requires:
|
||||
|
||||
- Exact `version`, `artifactUrl`, `artifactName`, and lowercase 64-character `sha256`.
|
||||
- Every installed `executable` and a corresponding `verificationCommands` entry.
|
||||
- Deterministic extraction/install commands into `/opt/sim-cli`; quote fixed paths and clean temporary artifacts.
|
||||
- `pathEntries` when the executable is not installed into the helper's default `bin` directory.
|
||||
- `supportedProviders` only when it differs from the E2B-and-Daytona default.
|
||||
- `revision` when it differs from `1`; it must agree with the ID suffix.
|
||||
|
||||
Verification must prove the command is discoverable through `sandboxCliEnvironment`, not authenticate or contact a user account. Recipe commands run as root during both prebuilt image creation and runtime provisioning.
|
||||
|
||||
If the artifact host is new, add only the exact official hostname to the `officialHosts` allowlist in `cli-tools.test.ts`. Treat that as a supply-chain review, not a way to silence the test.
|
||||
|
||||
## 5. Preserve Generic Behavior
|
||||
|
||||
Confirm the existing generic paths remain sufficient:
|
||||
|
||||
- `sandboxCliToolRecipes` canonicalizes and resolves the recipe.
|
||||
- `sandboxCliEnvironment` propagates PATH to Python subprocesses, JavaScript subprocesses, and Shell.
|
||||
- E2B bakes the recipe into the custom image; runtime-strategy providers install it within the Function timeout.
|
||||
- CLI-only sandboxes remain buildable even with no language packages.
|
||||
- `hashSandboxSpec` includes recipe ID, revision, and checksum while preserving the legacy hash for an empty CLI list.
|
||||
- The settings selector derives groups and search aliases from client-safe metadata.
|
||||
|
||||
Do not special-case a CLI in those layers unless the registry contract cannot express a genuine provider requirement. Extend the registry contract generically when multiple CLIs need the same new behavior.
|
||||
|
||||
## 6. Test the Addition
|
||||
|
||||
Extend tests when the new entry introduces behavior not already covered:
|
||||
|
||||
- For every upgrade, add a regression proving the old ID and recipe remain resolvable but non-selectable, while the replacement ID is selectable.
|
||||
- Add important executable aliases to the table-driven search assertion.
|
||||
- Add a focused assertion for a multi-executable recipe, custom PATH, or restricted provider.
|
||||
- Add an opt-in credentialed smoke test only when installation plus a real minimal command cannot be validated without authentication. Read credentials from test-only environment variables, skip by default, create them only at runtime, and always tear down the sandbox.
|
||||
|
||||
Never commit downloaded artifacts or credentials.
|
||||
|
||||
## Required Validation
|
||||
|
||||
From `apps/sim`:
|
||||
|
||||
```bash
|
||||
bunx vitest run \
|
||||
lib/execution/remote-sandbox/cli-tools.test.ts \
|
||||
lib/execution/remote-sandbox/cli-tools-boundary.test.ts \
|
||||
lib/execution/remote-sandbox/sandbox-spec.test.ts \
|
||||
lib/execution/remote-sandbox/resolve.test.ts \
|
||||
lib/api/contracts/sandboxes.test.ts \
|
||||
'app/workspace/[workspaceId]/settings/components/sandboxes/utils.test.ts' \
|
||||
'app/workspace/[workspaceId]/settings/components/sandboxes/components/sandbox-editor.test.tsx'
|
||||
```
|
||||
|
||||
From the repository root:
|
||||
|
||||
```bash
|
||||
bun run type-check
|
||||
bun run check:api-validation
|
||||
bunx biome check \
|
||||
apps/sim/lib/execution/remote-sandbox/cli-tools.ts \
|
||||
apps/sim/lib/execution/remote-sandbox/cli-tools.server.ts \
|
||||
apps/sim/lib/execution/remote-sandbox/cli-tools.test.ts
|
||||
git diff --check
|
||||
```
|
||||
|
||||
For a new recipe, also exercise its install and every verification command in an actual E2B or Daytona sandbox when credentials and network access are available. Report clearly when only registry/unit validation ran.
|
||||
|
||||
## Completion Checklist
|
||||
|
||||
- [ ] Official immutable Linux x86-64 artifact and SHA-256 verified.
|
||||
- [ ] Versioned ID appended; old IDs and recipes retained.
|
||||
- [ ] Client metadata is searchable, categorized, unique, and recipe-free.
|
||||
- [ ] Server recipe is pinned, integrity-checked, noninteractive, and credential-free.
|
||||
- [ ] Every advertised executable has an offline verification command and PATH entry.
|
||||
- [ ] Provider compatibility is explicit and accurate.
|
||||
- [ ] Catalog, boundary, hash, resolver, type, API-validation, format, and diff checks pass.
|
||||
- [ ] Real provider installation was tested, or the missing live verification is disclosed.
|
||||
@@ -0,0 +1,221 @@
|
||||
---
|
||||
description: Add a new LLM model to apps/sim/providers/models.ts with specs verified against the provider's live API docs (no hallucination)
|
||||
argument-hint: <provider> <model-id> [docs-url]
|
||||
---
|
||||
|
||||
# Add Model Skill
|
||||
|
||||
You add a new model entry to `apps/sim/providers/models.ts`. **Every numeric and capability claim MUST be derived from a live web fetch of the provider's official docs in this session.** Marketing emails, training data, and your prior knowledge are not sources of truth — they routinely hallucinate pricing, context windows, and capability lists.
|
||||
|
||||
## Hard rules (do not skip)
|
||||
|
||||
1. **Live-fetch or refuse.** Before writing the entry, you must successfully WebFetch the provider's official models/pricing page in this session. If you cannot reach an authoritative source for any field, **mark the field as UNVERIFIED in your report and ask the user before guessing**. Never fill in pricing or capabilities from memory.
|
||||
2. **Two-source rule for pricing.** Cross-check input/output/cached pricing against at least one secondary source (OpenRouter, Artificial Analysis, CloudPrice, mem0, intuitionlabs). If sources disagree, the provider's own docs win — but flag the disagreement.
|
||||
3. **Read the code before setting capability flags.** Capability flags are dead unless the provider's implementation under `apps/sim/providers/{provider}/` actually consumes them (see Consumption Matrix below). Setting a flag the provider ignores is a silent bug.
|
||||
4. **Cite every fact.** Your final report must list the URL each value came from. No URL → not verified.
|
||||
|
||||
## Your Task
|
||||
|
||||
1. Identify provider and model id from user args
|
||||
2. Live-fetch official docs + pricing page + capability/parameter pages + at least one secondary source
|
||||
3. Apply the Consumption Matrix to know which capability flags are real
|
||||
4. Read 2-3 sibling entries in `models.ts` and match their pattern exactly
|
||||
5. Check the repo-side touchpoints that are NOT data-driven (hosted-key billing, tests, provider code)
|
||||
6. Insert the entry, run `bun run lint`, print the verification report
|
||||
|
||||
## Step 1: Live source-of-truth lookup
|
||||
|
||||
In priority order — fetch all that exist for the provider:
|
||||
|
||||
| Provider | Models index | Pricing | Reasoning/parameter caveats |
|
||||
|---|---|---|---|
|
||||
| OpenAI | platform.openai.com/docs/models | openai.com/api/pricing | platform.openai.com/docs/guides/reasoning |
|
||||
| Anthropic | docs.anthropic.com/en/docs/about-claude/models | anthropic.com/pricing | docs.anthropic.com/en/docs/build-with-claude/extended-thinking |
|
||||
| Google (Gemini) | ai.google.dev/gemini-api/docs/models | ai.google.dev/pricing | ai.google.dev/gemini-api/docs/thinking |
|
||||
| xAI | docs.x.ai/developers/models | docs.x.ai/developers/models (per-model detail page) | docs.x.ai/developers/model-capabilities/text/reasoning |
|
||||
| Mistral | docs.mistral.ai/getting-started/models/models_overview | mistral.ai/pricing | n/a |
|
||||
| DeepSeek | api-docs.deepseek.com/quick_start/pricing | same | api-docs.deepseek.com/guides/reasoning_model |
|
||||
| Groq | console.groq.com/docs/models | groq.com/pricing | n/a |
|
||||
| Cerebras | inference-docs.cerebras.ai/models | cerebras.ai/pricing | n/a |
|
||||
|
||||
Secondary verification (use at least one): `openrouter.ai/<provider>/<model>`, `artificialanalysis.ai/models/<model>`, `cloudprice.net/models/<provider>-<model>`.
|
||||
|
||||
Use a precise WebFetch prompt: *"Extract for {model_id}: exact model id string, context window in tokens, input price per 1M, cached input price per 1M, output price per 1M, max output tokens, supported reasoning effort levels, accepted parameters (temperature, top_p), release date. Do not fill in fields you cannot find."*
|
||||
|
||||
## Step 2: Consumption Matrix (which provider honors which capability)
|
||||
|
||||
| Capability | Honored by | Effect if set elsewhere |
|
||||
|---|---|---|
|
||||
| `temperature` | All providers (passed through if set) | Safe but inert on always-reasoning models that reject it |
|
||||
| `toolUsageControl` | All providers (provider-level, not per-model) | n/a — set on `ProviderDefinition`, not models |
|
||||
| `reasoningEffort` | `openai/core.ts`, `azure-openai`, `anthropic/core.ts` (mapped to thinking), `gemini/core.ts` | **Dead on xai, deepseek, mistral, groq, cerebras, openrouter, fireworks, bedrock, vertex** unless their core consumes it — re-grep before assuming |
|
||||
| `verbosity` | `openai/core.ts`, `azure-openai/index.ts` only | Dead elsewhere |
|
||||
| `thinking` | `anthropic/core.ts`, `gemini/core.ts` | Dead elsewhere |
|
||||
| `thinking.streamed` | Docs generator + `getThinkingStreamVisibility` (`models.ts`); `anthropic/core.ts` uses `'summary'` to request `display: 'summarized'` on agent-events runs | **Mandatory on Anthropic-family thinking models** (`agent-stream-docs:check` fails without it); other families fall back to provider defaults |
|
||||
| `nativeStructuredOutputs` | `anthropic/core.ts`, `fireworks/index.ts`, `openrouter/index.ts` | Dead on openai, xai, google, vertex, bedrock, azure-openai, deepseek, mistral, groq, cerebras |
|
||||
| `maxOutputTokens` | Read by UI + executor for token estimation | Always meaningful — set if provider documents a cap |
|
||||
| `computerUse` | `anthropic/core.ts` | Dead elsewhere |
|
||||
| `deepResearch` | UI flag for routing to deep-research SKUs | Set only on actual deep-research model IDs |
|
||||
| `memory: false` | Conversation persistence opt-out | Set only when model genuinely cannot maintain history (e.g., deep-research) |
|
||||
|
||||
**Always re-grep before relying on this table** — the codebase moves:
|
||||
|
||||
```bash
|
||||
rg "reasoningEffort|reasoning_effort" apps/sim/providers/<provider>/
|
||||
rg "verbosity" apps/sim/providers/<provider>/
|
||||
rg "request\.thinking|thinking:" apps/sim/providers/<provider>/
|
||||
rg "supportsNativeStructuredOutputs|nativeStructuredOutputs" apps/sim/providers/<provider>/
|
||||
```
|
||||
|
||||
## Step 3: Match the provider's existing entry pattern
|
||||
|
||||
Open `apps/sim/providers/models.ts`, find `PROVIDER_DEFINITIONS[<provider>].models`, read 2-3 sibling entries. Match field order exactly:
|
||||
|
||||
```ts
|
||||
{
|
||||
id: '<exact-api-id>',
|
||||
pricing: {
|
||||
input: <number>,
|
||||
cachedInput: <number>, // omit if provider doesn't offer caching
|
||||
output: <number>,
|
||||
updatedAt: '<today YYYY-MM-DD>',
|
||||
},
|
||||
capabilities: {
|
||||
// only flags the provider actually consumes — see matrix
|
||||
},
|
||||
contextWindow: <tokens>,
|
||||
releaseDate: '<YYYY-MM-DD>',
|
||||
recommended: true, // only if new flagship; ask user before swapping
|
||||
speedOptimized: true, // only on smallest/fastest tier
|
||||
deprecated: true, // only on retired models
|
||||
}
|
||||
```
|
||||
|
||||
### Reseller providers (azure-openai, azure-anthropic, vertex, bedrock, openrouter)
|
||||
|
||||
Model id MUST be prefixed: `azure/`, `azure-anthropic/`, `vertex/`, `bedrock/`, `openrouter/`. Pricing usually mirrors the upstream provider but verify on the reseller's own pricing page.
|
||||
|
||||
### Insertion order
|
||||
|
||||
Within a family, newest first (matches existing convention: GPT-5.5 above GPT-5.4 above GPT-5.2). Across families, biggest/flagship at top of list.
|
||||
|
||||
### `recommended` / `speedOptimized`
|
||||
|
||||
- At most one or two `recommended: true` per provider — the current flagship(s).
|
||||
- If you're adding a new flagship, ask the user before removing `recommended` from the previous flagship. Never silently flip it.
|
||||
- `speedOptimized: true` only on the smallest/fastest tier (nano, flash-lite, haiku class).
|
||||
|
||||
## Step 4: Repo-side touchpoints beyond the entry
|
||||
|
||||
Adding the `models.ts` entry is most of the job because nearly every consumer is **data-driven** and picks the model up automatically: the ~40 query helpers in `models.ts` / `providers/utils.ts`, the public `/models` catalog (`app/(landing)/models/utils.ts` iterates `PROVIDER_DEFINITIONS`), the agent-block model dropdown, and copilot's `isKnownModelId` / `suggestModelIdsForUnknownModel` validation. The touchpoints below are the exceptions — they are **not** data-driven, so check each one.
|
||||
|
||||
### Hosted = auto-billed, by provider
|
||||
|
||||
`getHostedModels()` in `apps/sim/providers/models.ts` returns **every** model under `openai`, `anthropic`, and `google`:
|
||||
|
||||
```ts
|
||||
export function getHostedModels(): string[] {
|
||||
return [
|
||||
...getProviderModels('openai'),
|
||||
...getProviderModels('anthropic'),
|
||||
...getProviderModels('google'),
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
So a model added to any of those three providers is **automatically served with Sim's rotating hosted key and billed** to the workspace via `shouldBillModelUsage()` (`providers/utils.ts`). Before you insert:
|
||||
|
||||
- **If the model should be BYOK-only / never-billed**, do NOT drop it under `openai`/`anthropic`/`google` as-is — that silently enrolls it in hosted billing. Confirm hosting/billing intent with the user. (Precedent: Ollama Cloud is a deliberately separate `isReseller` provider specifically to stay BYOK-only/never-billed.)
|
||||
- **If the model should be hosted**, the deployment must actually have a key for it — the provider's `{PREFIX}_COUNT` / `{PREFIX}_1..N` env vars must be set, or hosted runs fail at execution time.
|
||||
- State the hosted/billing status explicitly in the verification report.
|
||||
|
||||
### Tests with hardcoded model IDs
|
||||
|
||||
`bun run lint` does **not** run tests. A few tests assert specific model IDs and can break or need updating when you touch a hosted or flagship model:
|
||||
|
||||
- `apps/sim/providers/utils.test.ts` — asserts membership of `getHostedModels()` / `shouldBillModelUsage()`
|
||||
- `apps/sim/providers/index.test.ts` and serializer tests — reference concrete model IDs
|
||||
|
||||
```bash
|
||||
rg "<new-model-id>|getHostedModels|shouldBillModelUsage" apps/sim/providers/*.test.ts
|
||||
```
|
||||
|
||||
If anything matches, run the affected provider tests and update assertions as needed.
|
||||
|
||||
### New API behavior is NOT data-driven
|
||||
|
||||
The Consumption Matrix (Step 2) tells you which capability *flags* are honored by existing provider code. But if the new model needs **net-new** request handling that the provider doesn't implement yet — a new beta header (e.g. Anthropic's `anthropic-beta` structured-outputs header in `anthropic/index.ts`), a new thinking/reasoning encoding, a Responses-API quirk — you must edit `apps/sim/providers/<provider>/core.ts` / `index.ts`. Setting a flag whose behavior isn't implemented is a silent no-op. When you do edit provider code, reuse the shared helpers rather than hand-rolling: streaming responses are assembled via `createStreamingExecution` (`@/providers/streaming-execution`) and tool schemas via `adaptOpenAIChatToolSchema` / `adaptAnthropicToolSchema` (`@/providers/tool-schema-adapter`).
|
||||
|
||||
### Thinking/reasoning models: `streamed` visibility + generated docs
|
||||
|
||||
If the entry has `capabilities.thinking` or `capabilities.reasoningEffort`, it appears in the autogenerated "Streamed thinking and tool calls" table on the Agent block docs page:
|
||||
|
||||
- **Anthropic-family (`anthropic`, `azure-anthropic`) thinking models MUST declare `capabilities.thinking.streamed`** (`'full' | 'summary' | 'none'`). Verify against Anthropic's current thinking-display and streaming docs: visible thinking returned by the API is summarized, including when Sim opts models whose default display is `omitted` into `display: 'summarized'` on agent-events runs, so current Claude thinking models use `'summary'`. Use `'full'` only if future official API docs explicitly guarantee raw thinking deltas. `bun run agent-stream-docs:check` (CI) fails if the field is missing.
|
||||
- Other families usually omit the field and inherit the provider default in `getThinkingStreamVisibility` (Gemini/OpenAI → summaries; Bedrock/Meta → none; OpenAI-compatible vendors with documented reasoning fields → full deltas). Set it explicitly only when the model deviates from its family.
|
||||
- After inserting the entry, run `bun run agent-stream-docs:generate` and commit the regenerated `apps/docs/content/docs/en/workflows/blocks/agent.mdx` — CI diffs it.
|
||||
- Include the `streamed` value (with its source URL) in the verification report when set.
|
||||
|
||||
### Wrong family entirely?
|
||||
|
||||
- **Embedding or rerank model** → it does NOT go in the `models[]` array. Use `EMBEDDING_MODEL_PRICING` / `RERANK_MODEL_PRICING` in `models.ts` instead.
|
||||
- **Brand-new provider** (not just a new model under an existing one) → much larger surface: add the id to `ProviderId` in `providers/types.ts`, a registry entry in `providers/registry.ts`, a provider implementation under `providers/<id>/` (assemble streaming responses with `createStreamingExecution` and wrap tool schemas with the `@/providers/tool-schema-adapter` helpers), an icon in `components/icons.tsx`, and the `PROVIDER_DEFINITIONS` block. That is beyond this skill — tell the user.
|
||||
|
||||
## Step 5: Write, lint
|
||||
|
||||
```bash
|
||||
bun run lint
|
||||
bun run agent-stream-docs:generate # only when the entry has thinking/reasoningEffort
|
||||
```
|
||||
|
||||
Lint must pass before reporting done. **If lint fails:** read the error, fix the syntax/typing issue in the entry you just wrote (do not delete the entry — it's the work product), re-run lint, and note the fix in a "Lint adjustments" line in the verification report. Never report done with lint failing.
|
||||
|
||||
## Step 6: Verification report (mandatory format)
|
||||
|
||||
End with this exact structure:
|
||||
|
||||
```markdown
|
||||
### Verification — <model-id>
|
||||
|
||||
| Field | Value | Source URL | Status |
|
||||
|---|---|---|---|
|
||||
| `id` | `grok-4.3` | https://docs.x.ai/... | ✓ verified |
|
||||
| `contextWindow` | 1,000,000 | https://docs.x.ai/... + https://openrouter.ai/... | ✓ verified (2 sources agree) |
|
||||
| `input` | $1.25/M | https://docs.x.ai/... | ✓ verified |
|
||||
| `cachedInput` | $0.20/M | https://cloudprice.net/... | ⚠️ single source |
|
||||
| `output` | $2.50/M | https://docs.x.ai/... + https://openrouter.ai/... | ✓ verified |
|
||||
| `capabilities.temperature` | `{ min: 0, max: 1 }` | matches sibling entries | — pattern-match only |
|
||||
| `capabilities.reasoningEffort` | NOT SET | provider docs say API rejects it for this model | ✓ correctly omitted |
|
||||
| `releaseDate` | 2026-04-30 | https://docs.x.ai/... announcement | ✓ verified |
|
||||
| hosted/billing | BYOK-only (xai not in `getHostedModels`) | `providers/models.ts` | — confirmed intent |
|
||||
|
||||
**Disagreements**
|
||||
- _none_ OR _OpenRouter says X, provider docs say Y — used Y per provider rule_
|
||||
|
||||
**Unverified fields**
|
||||
- _none_ OR _<field>: could not find authoritative source — left as <X> based on sibling pattern; please confirm_
|
||||
```
|
||||
|
||||
If any row is ⚠️ single-source or "unverified," **state it plainly to the user and ask whether to proceed**. Do not silently merge.
|
||||
|
||||
## What to do if you cannot find a source
|
||||
|
||||
Omitting a field is **not the same as verifying it**. Any field you cannot confirm from a live fetch must be **both** omitted from the entry **and** listed as ❓ UNVERIFIED in the report's "Unverified fields" section, with the URLs you attempted. Then ask the user to confirm before merging.
|
||||
|
||||
- Pricing missing → do NOT guess. Omit `cachedInput`. Mark ❓ UNVERIFIED. Ask the user for the price or the docs URL.
|
||||
- Context window missing → do NOT guess. Ask the user; mark ❓ UNVERIFIED.
|
||||
- Release date missing → omit the field; mark ❓ UNVERIFIED in the report.
|
||||
- Capability uncertain → omit the flag (safer than setting a dead/wrong one); mark ❓ UNVERIFIED so the user knows you didn't confirm it either way.
|
||||
|
||||
## Anti-patterns this skill exists to prevent
|
||||
|
||||
- ❌ Trusting a marketing email (xAI's grok-4.3 email claimed "3 reasoning efforts" but the API rejects `reasoning_effort` — verified by official docs only)
|
||||
- ❌ Setting `nativeStructuredOutputs: true` on xai/openai/google (dead — only anthropic/fireworks/openrouter consume it)
|
||||
- ❌ Setting `thinking` on non-Anthropic/non-Gemini providers
|
||||
- ❌ Adding an Anthropic-family thinking model without `thinking.streamed` (CI `agent-stream-docs:check` fails), or skipping `bun run agent-stream-docs:generate` after adding any thinking/reasoning model
|
||||
- ❌ Setting `verbosity` on anything other than OpenAI gpt-5.x
|
||||
- ❌ Copying `pricing.updatedAt` from a sibling instead of using today's date
|
||||
- ❌ Inventing a `cachedInput` price by dividing input by 4 (varies by provider — find an explicit number)
|
||||
- ❌ Stamping `recommended: true` on the new model without removing it from the previous flagship
|
||||
- ❌ Adding a BYOK-only model under `openai`/`anthropic`/`google` (silently enrolls it in hosted billing via `getHostedModels()`)
|
||||
- ❌ Reporting "done" after only `bun run lint` when you touched a hosted (openai/anthropic/google) or flagship model with assertions in `providers/utils.test.ts`
|
||||
- ❌ Reporting "done" with any UNVERIFIED row in the table
|
||||
@@ -0,0 +1,501 @@
|
||||
---
|
||||
description: Create tool configurations for a Sim integration by reading API docs
|
||||
argument-hint: <service-name> [api-docs-url]
|
||||
---
|
||||
|
||||
# Add Tools Skill
|
||||
|
||||
You are an expert at creating tool configurations for Sim integrations. Your job is to read API documentation and create properly structured tool files.
|
||||
|
||||
## Your Task
|
||||
|
||||
When the user asks you to create tools for a service:
|
||||
1. Use Context7 or WebFetch to read the service's API documentation
|
||||
2. Create the tools directory structure
|
||||
3. Generate properly typed tool configurations
|
||||
|
||||
## Hard Rule: No Guessed Response Schemas
|
||||
|
||||
If the docs do not clearly show the response JSON for a tool, you MUST tell the user exactly which outputs are unknown and stop short of guessing.
|
||||
|
||||
- Do NOT invent response field names
|
||||
- Do NOT infer nested paths from nearby endpoints
|
||||
- Do NOT guess array item shapes
|
||||
- Do NOT write `transformResponse` against unverified payloads
|
||||
|
||||
If the response shape is unknown, do one of these instead:
|
||||
1. Ask the user for sample responses
|
||||
2. Ask the user for test credentials so you can verify live responses
|
||||
3. Implement only the endpoints whose outputs are documented
|
||||
4. Leave the tool unimplemented and explicitly say why
|
||||
|
||||
## Directory Structure
|
||||
|
||||
Create files in `apps/sim/tools/{service}/`:
|
||||
```
|
||||
tools/{service}/
|
||||
├── index.ts # Barrel export
|
||||
├── types.ts # Parameter & response types
|
||||
└── {action}.ts # Individual tool files (one per operation)
|
||||
```
|
||||
|
||||
## Tool Configuration Structure
|
||||
|
||||
Every tool MUST follow this exact structure:
|
||||
|
||||
```typescript
|
||||
import type { {ServiceName}{Action}Params } from '@/tools/{service}/types'
|
||||
import type { ToolConfig } from '@/tools/types'
|
||||
|
||||
interface {ServiceName}{Action}Response {
|
||||
success: boolean
|
||||
output: {
|
||||
// Define output structure here
|
||||
}
|
||||
}
|
||||
|
||||
export const {serviceName}{Action}Tool: ToolConfig<
|
||||
{ServiceName}{Action}Params,
|
||||
{ServiceName}{Action}Response
|
||||
> = {
|
||||
id: '{service}_{action}', // snake_case, matches tool name
|
||||
name: '{Service} {Action}', // Human readable
|
||||
description: 'Brief description', // One sentence
|
||||
version: '1.0.0',
|
||||
|
||||
// OAuth config (if service uses OAuth)
|
||||
oauth: {
|
||||
required: true,
|
||||
provider: '{service}', // Must match OAuth provider ID
|
||||
},
|
||||
|
||||
params: {
|
||||
// Hidden params (system-injected, only use hidden for oauth accessToken)
|
||||
accessToken: {
|
||||
type: 'string',
|
||||
required: true,
|
||||
visibility: 'hidden',
|
||||
description: 'OAuth access token',
|
||||
},
|
||||
// User-only params (credentials, api key, IDs user must provide)
|
||||
someId: {
|
||||
type: 'string',
|
||||
required: true,
|
||||
visibility: 'user-only',
|
||||
description: 'The ID of the resource',
|
||||
},
|
||||
// User-or-LLM params (everything else, can be provided by user OR computed by LLM)
|
||||
query: {
|
||||
type: 'string',
|
||||
required: false, // Use false for optional
|
||||
visibility: 'user-or-llm',
|
||||
description: 'Search query',
|
||||
},
|
||||
},
|
||||
|
||||
request: {
|
||||
url: (params) => `https://api.service.com/v1/resource/${params.id}`,
|
||||
method: 'POST',
|
||||
headers: (params) => ({
|
||||
Authorization: `Bearer ${params.accessToken}`,
|
||||
'Content-Type': 'application/json',
|
||||
}),
|
||||
body: (params) => ({
|
||||
// Request body - only for POST/PUT/PATCH
|
||||
// Trim ID fields to prevent copy-paste whitespace errors:
|
||||
// userId: params.userId?.trim(),
|
||||
}),
|
||||
},
|
||||
|
||||
transformResponse: async (response: Response) => {
|
||||
const data = await response.json()
|
||||
return {
|
||||
success: true,
|
||||
output: {
|
||||
// Map API response to output
|
||||
// Use ?? null for nullable fields
|
||||
// Use ?? [] for optional arrays
|
||||
},
|
||||
}
|
||||
},
|
||||
|
||||
outputs: {
|
||||
// Define each output field
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
## Critical Rules for Parameters
|
||||
|
||||
### Visibility Options
|
||||
- `'hidden'` - System-injected (OAuth tokens, internal params). User never sees.
|
||||
- `'user-only'` - User must provide (credentials, api keys, account-specific IDs)
|
||||
- `'user-or-llm'` - User provides OR LLM can compute (search queries, content, filters, most fall into this category)
|
||||
|
||||
### Parameter Types
|
||||
- `'string'` - Text values
|
||||
- `'number'` - Numeric values
|
||||
- `'boolean'` - True/false
|
||||
- `'json'` - Complex objects (NOT 'object', use 'json')
|
||||
- `'file'` - Single file
|
||||
- `'file[]'` - Multiple files
|
||||
|
||||
### Required vs Optional
|
||||
- Always explicitly set `required: true` or `required: false`
|
||||
- Optional params should have `required: false`
|
||||
|
||||
## Resolved Secrets and Provenance Boundaries
|
||||
|
||||
- Leave ordinary external API inputs and third-party results unchanged. Add provenance handling only
|
||||
when an exact field is proven to cross a Sim model, durable-storage, or internal-execution boundary.
|
||||
- Project AI-consumed text/structured fields with the smallest exact `request.modelInput` selector.
|
||||
- Treat URLs, domains, resource IDs, and control fields as ordinary request values unless the exact
|
||||
field is proven model-visible. For serialized external model content, project the serialized
|
||||
top-level param through `request.modelInput` before the existing formatter parses it; do not add a
|
||||
separate hard-rejection mechanism.
|
||||
- For authenticated internal routes, use `privateProvenance` for actual inline/raw model bytes or
|
||||
`request.secretProvenance` for durable writes and execution handoffs. Do not treat a storage key,
|
||||
path, signed URL, or remote URL as provenance for fetched bytes; authorize tracked stored bytes at
|
||||
the owning model-egress boundary. Authenticate first, validate the exact selection and scope,
|
||||
strip the private envelope, then import or propagate provenance at the receiving boundary.
|
||||
Preserve documented headerless legacy behavior.
|
||||
- Never substitute secret plaintext into source, serialize plaintext provenance, hand-roll private
|
||||
headers, or blanket-sanitize tool results.
|
||||
- Add focused tests for named projection, identical unproven public text, malformed/incomplete
|
||||
metadata, metadata stripping, scope isolation, and legacy compatibility where applicable.
|
||||
|
||||
## Critical Rules for Outputs
|
||||
|
||||
### Output Types
|
||||
- `'string'`, `'number'`, `'boolean'` - Primitives
|
||||
- `'json'` - Complex objects (use this, NOT 'object')
|
||||
- `'array'` - Arrays with `items` property
|
||||
- `'object'` - Objects with `properties` property
|
||||
|
||||
### Optional Outputs
|
||||
Add `optional: true` for fields that may not exist in the response:
|
||||
```typescript
|
||||
closedAt: {
|
||||
type: 'string',
|
||||
description: 'When the issue was closed',
|
||||
optional: true,
|
||||
},
|
||||
```
|
||||
|
||||
### Typed JSON Outputs
|
||||
|
||||
When using `type: 'json'` and you know the object shape in advance, **always define the inner structure** using `properties` so downstream consumers know what fields are available:
|
||||
|
||||
```typescript
|
||||
// BAD: Opaque json with no info about what's inside
|
||||
metadata: {
|
||||
type: 'json',
|
||||
description: 'Response metadata',
|
||||
},
|
||||
|
||||
// GOOD: Define the known properties
|
||||
metadata: {
|
||||
type: 'json',
|
||||
description: 'Response metadata',
|
||||
properties: {
|
||||
id: { type: 'string', description: 'Unique ID' },
|
||||
status: { type: 'string', description: 'Current status' },
|
||||
count: { type: 'number', description: 'Total count' },
|
||||
},
|
||||
},
|
||||
```
|
||||
|
||||
For arrays of objects, define the item structure:
|
||||
```typescript
|
||||
items: {
|
||||
type: 'array',
|
||||
description: 'List of items',
|
||||
items: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
id: { type: 'string', description: 'Item ID' },
|
||||
name: { type: 'string', description: 'Item name' },
|
||||
},
|
||||
},
|
||||
},
|
||||
```
|
||||
|
||||
Only use bare `type: 'json'` without `properties` when the shape is truly dynamic or unknown.
|
||||
|
||||
If the response shape is unknown because the docs do not provide it, you MUST tell the user and stop. Unknown is not the same as dynamic. Never guess outputs.
|
||||
|
||||
## Critical Rules for transformResponse
|
||||
|
||||
### Handle Nullable Fields
|
||||
ALWAYS use `?? null` for fields that may be undefined:
|
||||
```typescript
|
||||
transformResponse: async (response: Response) => {
|
||||
const data = await response.json()
|
||||
return {
|
||||
success: true,
|
||||
output: {
|
||||
id: data.id,
|
||||
title: data.title,
|
||||
body: data.body ?? null, // May be undefined
|
||||
assignee: data.assignee ?? null, // May be undefined
|
||||
labels: data.labels ?? [], // Default to empty array
|
||||
closedAt: data.closed_at ?? null, // May be undefined
|
||||
},
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Never Output Raw JSON Dumps
|
||||
DON'T do this:
|
||||
```typescript
|
||||
output: {
|
||||
data: data, // BAD - raw JSON dump
|
||||
}
|
||||
```
|
||||
|
||||
DO this instead - extract meaningful fields:
|
||||
```typescript
|
||||
output: {
|
||||
id: data.id,
|
||||
name: data.name,
|
||||
status: data.status,
|
||||
metadata: {
|
||||
createdAt: data.created_at,
|
||||
updatedAt: data.updated_at,
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
## Types File Pattern
|
||||
|
||||
Create `types.ts` with interfaces for all params and responses:
|
||||
|
||||
```typescript
|
||||
import type { ToolResponse } from '@/tools/types'
|
||||
|
||||
// Parameter interfaces
|
||||
export interface {Service}{Action}Params {
|
||||
accessToken: string
|
||||
requiredField: string
|
||||
optionalField?: string
|
||||
}
|
||||
|
||||
// Response interfaces (extend ToolResponse)
|
||||
export interface {Service}{Action}Response extends ToolResponse {
|
||||
output: {
|
||||
field1: string
|
||||
field2: number
|
||||
optionalField?: string | null
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Index.ts Barrel Export Pattern
|
||||
|
||||
```typescript
|
||||
// Export all tools
|
||||
export { serviceTool1 } from './{action1}'
|
||||
export { serviceTool2 } from './{action2}'
|
||||
|
||||
// Export types
|
||||
export * from './types'
|
||||
```
|
||||
|
||||
## Registering Tools
|
||||
|
||||
After creating tools:
|
||||
1. Import tools in `apps/sim/tools/registry.ts`
|
||||
2. Add to the `tools` object with snake_case keys (alphabetically):
|
||||
```typescript
|
||||
import { serviceActionTool } from '@/tools/{service}'
|
||||
|
||||
export const tools = {
|
||||
// ... existing tools ...
|
||||
{service}_{action}: serviceActionTool,
|
||||
}
|
||||
```
|
||||
|
||||
3. Regenerate the tool metadata artifacts:
|
||||
|
||||
```bash
|
||||
bun run tool-metadata:generate
|
||||
```
|
||||
|
||||
Client code reads a tool's `params`/`outputs` from generated metadata rather than
|
||||
importing the registry, so a tool you add, change or remove is invisible to the UI until
|
||||
these are regenerated — and CI fails on stale artifacts. Commit the result. See
|
||||
`.agents/skills/tool-registry-boundary/SKILL.md`.
|
||||
|
||||
## Wiring Tools into the Block (Required)
|
||||
|
||||
After registering in `tools/registry.ts`, you MUST also update the block definition at `apps/sim/blocks/blocks/{service}.ts`. This is not optional — tools are only usable from the UI if they are wired into the block.
|
||||
|
||||
### 1. Add to `tools.access`
|
||||
|
||||
```typescript
|
||||
tools: {
|
||||
access: [
|
||||
// existing tools...
|
||||
'service_new_action', // Add every new tool ID here
|
||||
],
|
||||
config: { ... }
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Add operation dropdown options
|
||||
|
||||
If the block uses an operation dropdown, add an option for each new tool:
|
||||
|
||||
```typescript
|
||||
{
|
||||
id: 'operation',
|
||||
type: 'dropdown',
|
||||
options: [
|
||||
// existing options...
|
||||
{ label: 'New Action', id: 'new_action' }, // id maps to what tools.config.tool returns
|
||||
],
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Add subBlocks for new tool params
|
||||
|
||||
For each new tool, add subBlocks covering all its required params (and optional ones where useful). Apply `condition` to show them only for the right operation, and mark required params with `required`:
|
||||
|
||||
```typescript
|
||||
// Required param for new_action
|
||||
{
|
||||
id: 'someParam',
|
||||
title: 'Some Param',
|
||||
type: 'short-input',
|
||||
placeholder: 'e.g., value',
|
||||
condition: { field: 'operation', value: 'new_action' },
|
||||
required: { field: 'operation', value: 'new_action' },
|
||||
},
|
||||
// Optional param — put in advanced mode
|
||||
{
|
||||
id: 'optionalParam',
|
||||
title: 'Optional Param',
|
||||
type: 'short-input',
|
||||
condition: { field: 'operation', value: 'new_action' },
|
||||
mode: 'advanced',
|
||||
},
|
||||
```
|
||||
|
||||
### 4. Update `tools.config.tool`
|
||||
|
||||
Ensure the tool selector returns the correct tool ID for every new operation. The simplest pattern:
|
||||
|
||||
```typescript
|
||||
tool: (params) => `service_${params.operation}`,
|
||||
// If operation dropdown IDs already match tool IDs, this requires no change.
|
||||
```
|
||||
|
||||
If the dropdown IDs differ from tool IDs, add explicit mappings:
|
||||
|
||||
```typescript
|
||||
tool: (params) => {
|
||||
const map: Record<string, string> = {
|
||||
new_action: 'service_new_action',
|
||||
// ...
|
||||
}
|
||||
return map[params.operation] ?? `service_${params.operation}`
|
||||
},
|
||||
```
|
||||
|
||||
### 5. Update `tools.config.params`
|
||||
|
||||
Add any type coercions needed for new params (runs at execution time, after variable resolution):
|
||||
|
||||
```typescript
|
||||
params: (params) => {
|
||||
const result: Record<string, unknown> = {}
|
||||
if (params.limit != null && params.limit !== '') result.limit = Number(params.limit)
|
||||
if (params.newParamName) result.toolParamName = params.newParamName // rename if IDs differ
|
||||
return result
|
||||
},
|
||||
```
|
||||
|
||||
### 6. Add new outputs
|
||||
|
||||
Add any new fields returned by the new tools to the block `outputs`:
|
||||
|
||||
```typescript
|
||||
outputs: {
|
||||
// existing outputs...
|
||||
newField: { type: 'string', description: 'Description of new field' },
|
||||
}
|
||||
```
|
||||
|
||||
### 7. Add new inputs
|
||||
|
||||
Add new subBlock param IDs to the block `inputs` section:
|
||||
|
||||
```typescript
|
||||
inputs: {
|
||||
// existing inputs...
|
||||
someParam: { type: 'string', description: 'Param description' },
|
||||
optionalParam: { type: 'string', description: 'Optional param description' },
|
||||
}
|
||||
```
|
||||
|
||||
### Block wiring checklist
|
||||
|
||||
- [ ] New tool IDs added to `tools.access`
|
||||
- [ ] Operation dropdown has an option for each new tool
|
||||
- [ ] SubBlocks cover all required params for each new tool
|
||||
- [ ] SubBlocks have correct `condition` (only show for the right operation)
|
||||
- [ ] Optional/rarely-used params set to `mode: 'advanced'`
|
||||
- [ ] `tools.config.tool` returns correct ID for every new operation
|
||||
- [ ] `tools.config.params` handles any ID remapping or type coercions
|
||||
- [ ] New outputs added to block `outputs`
|
||||
- [ ] New params added to block `inputs`
|
||||
|
||||
## V2 Tool Pattern
|
||||
|
||||
If creating V2 tools (API-aligned outputs), use `_v2` suffix:
|
||||
- Tool ID: `{service}_{action}_v2`
|
||||
- Variable name: `{action}V2Tool`
|
||||
- Version: `'2.0.0'`
|
||||
- Outputs: Flat, API-aligned (no content/metadata wrapper)
|
||||
|
||||
## Naming Convention
|
||||
|
||||
All tool IDs MUST use `snake_case`: `{service}_{action}` (e.g., `x_create_tweet`, `slack_send_message`). Never use camelCase or PascalCase for tool IDs.
|
||||
|
||||
## Checklist Before Finishing
|
||||
|
||||
- [ ] All tool IDs use snake_case
|
||||
- [ ] All params have explicit `required: true` or `required: false`
|
||||
- [ ] All params have appropriate `visibility`
|
||||
- [ ] All nullable response fields use `?? null`
|
||||
- [ ] All optional outputs have `optional: true`
|
||||
- [ ] No raw JSON dumps in outputs
|
||||
- [ ] Types file has all interfaces
|
||||
- [ ] Index.ts exports all tools and re-exports types (`export * from './types'`)
|
||||
- [ ] Tools registered in `tools/registry.ts`
|
||||
- [ ] `bun run tool-metadata:generate` run and the regenerated artifacts committed
|
||||
- [ ] Block wired: `tools.access`, dropdown options, subBlocks, `tools.config`, outputs, inputs
|
||||
- [ ] Model, durable-storage, and internal-execution boundaries use the shared provenance mechanisms
|
||||
only where a concrete Sim `{{...}}` resolution path requires them
|
||||
- [ ] Ordinary third-party inputs/results remain unchanged and private metadata never leaves Sim
|
||||
|
||||
## Final Validation (Required)
|
||||
|
||||
After creating all tools, you MUST validate every tool before finishing:
|
||||
|
||||
1. **Read every tool file** you created — do not skip any
|
||||
2. **Cross-reference with the API docs** to verify:
|
||||
- All required params are marked `required: true`
|
||||
- All optional params are marked `required: false`
|
||||
- Param types match the API (string, number, boolean, json)
|
||||
- Request URL, method, headers, and body match the API spec
|
||||
- `transformResponse` extracts the correct fields from the API response
|
||||
- All output fields match what the API actually returns
|
||||
- No fields are missing from outputs that the API provides
|
||||
- No extra fields are defined in outputs that the API doesn't return
|
||||
- Every output field and JSON path is backed by docs or live-verified sample responses
|
||||
3. **Verify consistency** across tools:
|
||||
- Shared types in `types.ts` match all tools that use them
|
||||
- Tool IDs in the barrel export match the tool file definitions
|
||||
- Error handling is consistent (error checks, meaningful messages)
|
||||
4. **If any response schema is still unknown**, explicitly tell the user instead of guessing
|
||||
@@ -0,0 +1,512 @@
|
||||
---
|
||||
description: Create webhook or polling triggers for a Sim integration
|
||||
argument-hint: <service-name>
|
||||
---
|
||||
|
||||
# Add Trigger
|
||||
|
||||
You are an expert at creating webhook and polling triggers for Sim. You understand the trigger system, the generic `buildTriggerSubBlocks` helper, polling infrastructure, and how triggers connect to blocks.
|
||||
|
||||
## Your Task
|
||||
|
||||
1. Research what webhook events the service supports — if the service lacks reliable webhooks, use polling
|
||||
2. Create the trigger files using the generic builder (webhook) or manual config (polling)
|
||||
3. Create a provider handler (webhook) or polling handler (polling)
|
||||
4. Register triggers and connect them to the block
|
||||
|
||||
## Hard Rule: No Guessed Webhook Payload Schemas
|
||||
|
||||
If the service docs do not clearly show the webhook payload JSON for an event, you MUST tell the user instead of guessing trigger outputs or `formatInput` mappings.
|
||||
|
||||
- Do NOT invent payload field names
|
||||
- Do NOT guess nested event object paths
|
||||
- Do NOT infer output fields from the UI or marketing docs
|
||||
- Do NOT write `formatInput` against unverified webhook bodies
|
||||
|
||||
If the payload shape is unknown, do one of these instead:
|
||||
1. Ask the user for sample webhook payloads
|
||||
2. Ask the user for a test webhook source so you can inspect a real event
|
||||
3. Implement only the event registration/setup portions whose payloads are documented
|
||||
4. Leave the trigger unimplemented and explicitly say which payload fields are unknown
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
apps/sim/triggers/{service}/
|
||||
├── index.ts # Barrel exports
|
||||
├── utils.ts # Service-specific helpers (options, instructions, extra fields, outputs)
|
||||
├── {event_a}.ts # Primary trigger (includes dropdown)
|
||||
├── {event_b}.ts # Secondary trigger (no dropdown)
|
||||
└── webhook.ts # Generic webhook trigger (optional, for "all events")
|
||||
|
||||
apps/sim/lib/webhooks/
|
||||
├── provider-subscription-utils.ts # Shared subscription helpers (getProviderConfig, getNotificationUrl)
|
||||
├── providers/
|
||||
│ ├── {service}.ts # Provider handler (auth, formatInput, matchEvent, subscriptions)
|
||||
│ ├── types.ts # WebhookProviderHandler interface
|
||||
│ ├── utils.ts # Shared helpers (createHmacVerifier, verifyTokenAuth, skipByEventTypes)
|
||||
│ └── registry.ts # Handler map + default handler
|
||||
```
|
||||
|
||||
## Step 1: Create `utils.ts`
|
||||
|
||||
This file contains all service-specific helpers used by triggers.
|
||||
|
||||
```typescript
|
||||
import type { SubBlockConfig } from '@/blocks/types'
|
||||
import type { TriggerOutput } from '@/triggers/types'
|
||||
|
||||
export const {service}TriggerOptions = [
|
||||
{ label: 'Event A', id: '{service}_event_a' },
|
||||
{ label: 'Event B', id: '{service}_event_b' },
|
||||
]
|
||||
|
||||
export function {service}SetupInstructions(eventType: string): string {
|
||||
const instructions = [
|
||||
'Copy the <strong>Webhook URL</strong> above',
|
||||
'Go to <strong>{Service} Settings > Webhooks</strong>',
|
||||
`Select the <strong>${eventType}</strong> event type`,
|
||||
'Paste the webhook URL and save',
|
||||
'Click "Save" above to activate your trigger',
|
||||
]
|
||||
return instructions
|
||||
.map((instruction, index) =>
|
||||
`<div class="mb-3"><strong>${index + 1}.</strong> ${instruction}</div>`
|
||||
)
|
||||
.join('')
|
||||
}
|
||||
|
||||
export function build{Service}ExtraFields(triggerId: string): SubBlockConfig[] {
|
||||
return [
|
||||
{
|
||||
id: 'projectId',
|
||||
title: 'Project ID (Optional)',
|
||||
type: 'short-input',
|
||||
placeholder: 'Leave empty for all projects',
|
||||
mode: 'trigger',
|
||||
condition: { field: 'selectedTriggerId', value: triggerId },
|
||||
},
|
||||
]
|
||||
}
|
||||
|
||||
export function build{Service}Outputs(): Record<string, TriggerOutput> {
|
||||
return {
|
||||
eventType: { type: 'string', description: 'The type of event' },
|
||||
resourceId: { type: 'string', description: 'ID of the affected resource' },
|
||||
resource: {
|
||||
id: { type: 'string', description: 'Resource ID' },
|
||||
name: { type: 'string', description: 'Resource name' },
|
||||
},
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Step 2: Create Trigger Files
|
||||
|
||||
**Primary trigger** — MUST include `includeDropdown: true`:
|
||||
|
||||
```typescript
|
||||
import { {Service}Icon } from '@/components/icons'
|
||||
import { buildTriggerSubBlocks } from '@/triggers'
|
||||
import { build{Service}ExtraFields, build{Service}Outputs, {service}SetupInstructions, {service}TriggerOptions } from '@/triggers/{service}/utils'
|
||||
import type { TriggerConfig } from '@/triggers/types'
|
||||
|
||||
export const {service}EventATrigger: TriggerConfig = {
|
||||
id: '{service}_event_a',
|
||||
name: '{Service} Event A',
|
||||
provider: '{service}',
|
||||
description: 'Trigger workflow when Event A occurs',
|
||||
version: '1.0.0',
|
||||
icon: {Service}Icon,
|
||||
subBlocks: buildTriggerSubBlocks({
|
||||
triggerId: '{service}_event_a',
|
||||
triggerOptions: {service}TriggerOptions,
|
||||
includeDropdown: true,
|
||||
setupInstructions: {service}SetupInstructions('Event A'),
|
||||
extraFields: build{Service}ExtraFields('{service}_event_a'),
|
||||
}),
|
||||
outputs: build{Service}Outputs(),
|
||||
webhook: { method: 'POST', headers: { 'Content-Type': 'application/json' } },
|
||||
}
|
||||
```
|
||||
|
||||
**Secondary triggers** — NO `includeDropdown` (it's already in the primary):
|
||||
|
||||
```typescript
|
||||
export const {service}EventBTrigger: TriggerConfig = {
|
||||
// Same as above but: id: '{service}_event_b', no includeDropdown
|
||||
}
|
||||
```
|
||||
|
||||
## Step 3: Register and Wire
|
||||
|
||||
### `apps/sim/triggers/{service}/index.ts`
|
||||
|
||||
```typescript
|
||||
export { {service}EventATrigger } from './event_a'
|
||||
export { {service}EventBTrigger } from './event_b'
|
||||
```
|
||||
|
||||
### `apps/sim/triggers/registry.ts`
|
||||
|
||||
```typescript
|
||||
import { {service}EventATrigger, {service}EventBTrigger } from '@/triggers/{service}'
|
||||
|
||||
export const TRIGGER_REGISTRY: TriggerRegistry = {
|
||||
// ... existing ...
|
||||
{service}_event_a: {service}EventATrigger,
|
||||
{service}_event_b: {service}EventBTrigger,
|
||||
}
|
||||
```
|
||||
|
||||
### Block file (`apps/sim/blocks/blocks/{service}.ts`)
|
||||
|
||||
Wire triggers into the block so the trigger UI appears and `generate-docs.ts` discovers them. Two changes are needed:
|
||||
|
||||
1. **Spread trigger subBlocks** at the end of the block's `subBlocks` array
|
||||
2. **Add `triggers` property** after `outputs` with `enabled: true` and `available: [...]`
|
||||
|
||||
```typescript
|
||||
import { getTrigger } from '@/triggers'
|
||||
|
||||
export const {Service}Block: BlockConfig = {
|
||||
// ...
|
||||
subBlocks: [
|
||||
// Regular tool subBlocks first...
|
||||
...getTrigger('{service}_event_a').subBlocks,
|
||||
...getTrigger('{service}_event_b').subBlocks,
|
||||
],
|
||||
// ... tools, inputs, outputs ...
|
||||
triggers: {
|
||||
enabled: true,
|
||||
available: ['{service}_event_a', '{service}_event_b'],
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
**Versioned blocks (V1 + V2):** Many integrations have a hidden V1 block and a visible V2 block. Where you add the trigger wiring depends on how V2 inherits from V1:
|
||||
|
||||
- **V2 uses `...V1Block` spread** (e.g., Google Calendar): Add trigger to V1 — V2 inherits both `subBlocks` and `triggers` automatically.
|
||||
- **V2 defines its own `subBlocks`** (e.g., Google Sheets): Add trigger to V2 (the visible block). V1 is hidden and doesn't need it.
|
||||
- **Single block, no V2** (e.g., Google Drive): Add trigger directly.
|
||||
|
||||
`generate-docs.ts` deduplicates by base type (first match wins). If V1 is processed first without triggers, the V2 triggers won't appear in `integrations.json`. Always verify by checking the output after running the script.
|
||||
|
||||
## Provider Handler
|
||||
|
||||
All provider-specific webhook logic lives in a single handler file: `apps/sim/lib/webhooks/providers/{service}.ts`.
|
||||
|
||||
### When to Create a Handler
|
||||
|
||||
| Behavior | Method | Examples |
|
||||
|---|---|---|
|
||||
| HMAC signature auth | `verifyAuth` via `createHmacVerifier` | Ashby, Jira, Linear, Typeform |
|
||||
| Custom token auth | `verifyAuth` via `verifyTokenAuth` | Generic, Google Forms |
|
||||
| Event filtering | `matchEvent` | GitHub, Jira, Attio, HubSpot |
|
||||
| Idempotency dedup | `extractIdempotencyId` | Slack, Stripe, Linear, Jira |
|
||||
| Custom input formatting | `formatInput` | Slack, Teams, Attio, Ashby |
|
||||
| Auto webhook creation | `createSubscription` | Ashby, Grain, Calendly, Airtable |
|
||||
| Auto webhook deletion | `deleteSubscription` | Ashby, Grain, Calendly, Airtable |
|
||||
| Challenge/verification | `handleChallenge` | Slack, WhatsApp, Teams |
|
||||
| Custom success response | `formatSuccessResponse` | Slack, Twilio Voice, Teams |
|
||||
|
||||
If none apply, you don't need a handler. The default handler provides bearer token auth.
|
||||
|
||||
### Example Handler
|
||||
|
||||
```typescript
|
||||
import crypto from 'crypto'
|
||||
import { createLogger } from '@sim/logger'
|
||||
import { safeCompare } from '@/lib/core/security/encryption'
|
||||
import type { EventMatchContext, FormatInputContext, FormatInputResult, WebhookProviderHandler } from '@/lib/webhooks/providers/types'
|
||||
import { createHmacVerifier } from '@/lib/webhooks/providers/utils'
|
||||
|
||||
const logger = createLogger('WebhookProvider:{Service}')
|
||||
|
||||
function validate{Service}Signature(secret: string, signature: string, body: string): boolean {
|
||||
if (!secret || !signature || !body) return false
|
||||
const computed = crypto.createHmac('sha256', secret).update(body, 'utf8').digest('hex')
|
||||
return safeCompare(computed, signature)
|
||||
}
|
||||
|
||||
export const {service}Handler: WebhookProviderHandler = {
|
||||
verifyAuth: createHmacVerifier({
|
||||
configKey: 'webhookSecret',
|
||||
headerName: 'X-{Service}-Signature',
|
||||
validateFn: validate{Service}Signature,
|
||||
providerLabel: '{Service}',
|
||||
}),
|
||||
|
||||
async matchEvent({ body, requestId, providerConfig }: EventMatchContext) {
|
||||
const triggerId = providerConfig.triggerId as string | undefined
|
||||
if (triggerId && triggerId !== '{service}_webhook') {
|
||||
const { is{Service}EventMatch } = await import('@/triggers/{service}/utils')
|
||||
if (!is{Service}EventMatch(triggerId, body as Record<string, unknown>)) return false
|
||||
}
|
||||
return true
|
||||
},
|
||||
|
||||
async formatInput({ body }: FormatInputContext): Promise<FormatInputResult> {
|
||||
const b = body as Record<string, unknown>
|
||||
return {
|
||||
input: {
|
||||
eventType: b.type,
|
||||
resourceId: (b.data as Record<string, unknown>)?.id || '',
|
||||
resource: b.data,
|
||||
},
|
||||
}
|
||||
},
|
||||
|
||||
extractIdempotencyId(body: unknown) {
|
||||
const obj = body as Record<string, unknown>
|
||||
return obj.id && obj.type ? `${obj.type}:${obj.id}` : null
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
### Register the Handler
|
||||
|
||||
In `apps/sim/lib/webhooks/providers/registry.ts`:
|
||||
|
||||
```typescript
|
||||
import { {service}Handler } from '@/lib/webhooks/providers/{service}'
|
||||
|
||||
const PROVIDER_HANDLERS: Record<string, WebhookProviderHandler> = {
|
||||
// ... existing (alphabetical) ...
|
||||
{service}: {service}Handler,
|
||||
}
|
||||
```
|
||||
|
||||
## Output Alignment (Critical)
|
||||
|
||||
There are two sources of truth that **MUST be aligned**:
|
||||
|
||||
1. **Trigger `outputs`** — schema defining what fields SHOULD be available (UI tag dropdown)
|
||||
2. **`formatInput` on the handler** — implementation that transforms raw payload into actual data
|
||||
|
||||
If they differ: the tag dropdown shows fields that don't exist, or actual data has fields users can't discover.
|
||||
|
||||
**Rules for `formatInput`:**
|
||||
- Return `{ input: { ... } }` where inner keys match trigger `outputs` exactly
|
||||
- Return `{ input: ..., skip: { message: '...' } }` to skip execution
|
||||
- No wrapper objects or duplication
|
||||
- Use `null` for missing optional data
|
||||
|
||||
## Automatic Webhook Registration
|
||||
|
||||
If the service API supports programmatic webhook creation, implement `createSubscription` and `deleteSubscription` on the handler. The orchestration layer calls these automatically — **no code touches `route.ts`, `provider-subscriptions.ts`, or `deploy.ts`**.
|
||||
|
||||
```typescript
|
||||
import { getNotificationUrl, getProviderConfig } from '@/lib/webhooks/provider-subscription-utils'
|
||||
import type { DeleteSubscriptionContext, SubscriptionContext, SubscriptionResult } from '@/lib/webhooks/providers/types'
|
||||
|
||||
export const {service}Handler: WebhookProviderHandler = {
|
||||
async createSubscription(ctx: SubscriptionContext): Promise<SubscriptionResult | undefined> {
|
||||
const config = getProviderConfig(ctx.webhook)
|
||||
const apiKey = config.apiKey as string
|
||||
if (!apiKey) throw new Error('{Service} API Key is required.')
|
||||
|
||||
const res = await fetch('https://api.{service}.com/webhooks', {
|
||||
method: 'POST',
|
||||
headers: { Authorization: `Bearer ${apiKey}`, 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ url: getNotificationUrl(ctx.webhook) }),
|
||||
})
|
||||
|
||||
if (!res.ok) throw new Error(`{Service} error: ${res.status}`)
|
||||
const { id } = (await res.json()) as { id: string }
|
||||
return { providerConfigUpdates: { externalId: id } }
|
||||
},
|
||||
|
||||
async deleteSubscription(ctx: DeleteSubscriptionContext): Promise<void> {
|
||||
const config = getProviderConfig(ctx.webhook)
|
||||
const { apiKey, externalId } = config as { apiKey?: string; externalId?: string }
|
||||
if (!apiKey || !externalId) return
|
||||
await fetch(`https://api.{service}.com/webhooks/${externalId}`, {
|
||||
method: 'DELETE',
|
||||
headers: { Authorization: `Bearer ${apiKey}` },
|
||||
}).catch(() => {})
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
**Key points:**
|
||||
- Throw from `createSubscription` — orchestration rolls back the DB webhook
|
||||
- Never throw from `deleteSubscription` — log non-fatally
|
||||
- Return `{ providerConfigUpdates: { externalId } }` — orchestration merges into `providerConfig`
|
||||
- Add `apiKey` field to `build{Service}ExtraFields` with `password: true`
|
||||
|
||||
## Trigger Outputs Schema
|
||||
|
||||
Trigger outputs use the same schema as block outputs (NOT tool outputs).
|
||||
|
||||
**Supported:** `type` + `description` for leaf fields, nested objects for complex data.
|
||||
**NOT supported:** `optional: true`, `items` (those are tool-output-only features).
|
||||
|
||||
```typescript
|
||||
export function buildOutputs(): Record<string, TriggerOutput> {
|
||||
return {
|
||||
eventType: { type: 'string', description: 'Event type' },
|
||||
timestamp: { type: 'string', description: 'When it occurred' },
|
||||
payload: { type: 'json', description: 'Full event payload' },
|
||||
resource: {
|
||||
id: { type: 'string', description: 'Resource ID' },
|
||||
name: { type: 'string', description: 'Resource name' },
|
||||
},
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Polling Triggers
|
||||
|
||||
Use polling when the service lacks reliable webhooks (e.g., Google Sheets, Google Drive, Google Calendar, Gmail, RSS, IMAP). Polling triggers do NOT use `buildTriggerSubBlocks` — they define subBlocks manually.
|
||||
|
||||
### Directory Structure
|
||||
|
||||
```
|
||||
apps/sim/triggers/{service}/
|
||||
├── index.ts # Barrel export
|
||||
└── poller.ts # TriggerConfig with polling: true
|
||||
|
||||
apps/sim/lib/webhooks/polling/
|
||||
└── {service}.ts # PollingProviderHandler implementation
|
||||
```
|
||||
|
||||
### Polling Handler (`apps/sim/lib/webhooks/polling/{service}.ts`)
|
||||
|
||||
```typescript
|
||||
import { pollingIdempotency } from '@/lib/core/idempotency/service'
|
||||
import type { PollingProviderHandler, PollWebhookContext } from '@/lib/webhooks/polling/types'
|
||||
import { markWebhookFailed, markWebhookSuccess, resolveOAuthCredential, updateWebhookProviderConfig } from '@/lib/webhooks/polling/utils'
|
||||
import { processPolledWebhookEvent } from '@/lib/webhooks/processor'
|
||||
|
||||
export const {service}PollingHandler: PollingProviderHandler = {
|
||||
provider: '{service}',
|
||||
label: '{Service}',
|
||||
|
||||
async pollWebhook(ctx: PollWebhookContext): Promise<'success' | 'failure'> {
|
||||
const { webhookData, workflowData, requestId, logger } = ctx
|
||||
const webhookId = webhookData.id
|
||||
|
||||
try {
|
||||
// For OAuth services:
|
||||
const accessToken = await resolveOAuthCredential(webhookData, '{service}', requestId)
|
||||
const config = webhookData.providerConfig as unknown as {Service}WebhookConfig
|
||||
|
||||
// First poll: seed state, emit nothing
|
||||
if (!config.lastCheckedTimestamp) {
|
||||
await updateWebhookProviderConfig(webhookId, { lastCheckedTimestamp: new Date().toISOString() }, logger)
|
||||
await markWebhookSuccess(webhookId, logger)
|
||||
return 'success'
|
||||
}
|
||||
|
||||
// Fetch changes since last poll, process with idempotency
|
||||
// ...
|
||||
|
||||
await markWebhookSuccess(webhookId, logger)
|
||||
return 'success'
|
||||
} catch (error) {
|
||||
logger.error(`[${requestId}] Error processing {service} webhook ${webhookId}:`, error)
|
||||
await markWebhookFailed(webhookId, logger)
|
||||
return 'failure'
|
||||
}
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
**Key patterns:**
|
||||
- First poll seeds state and emits nothing (avoids flooding with existing data)
|
||||
- Use `pollingIdempotency.executeWithIdempotency(provider, key, callback)` for dedup
|
||||
- Use `processPolledWebhookEvent(webhookData, workflowData, payload, requestId)` to fire the workflow
|
||||
- Use `updateWebhookProviderConfig(webhookId, partialConfig, logger)` for read-merge-write on state
|
||||
- Use the latest server-side timestamp from API responses (not wall clock) to avoid clock skew
|
||||
|
||||
### Trigger Config (`apps/sim/triggers/{service}/poller.ts`)
|
||||
|
||||
```typescript
|
||||
import { {Service}Icon } from '@/components/icons'
|
||||
import type { TriggerConfig } from '@/triggers/types'
|
||||
|
||||
export const {service}PollingTrigger: TriggerConfig = {
|
||||
id: '{service}_poller',
|
||||
name: '{Service} Trigger',
|
||||
provider: '{service}',
|
||||
description: 'Triggers when ...',
|
||||
version: '1.0.0',
|
||||
icon: {Service}Icon,
|
||||
polling: true, // REQUIRED — routes to polling infrastructure
|
||||
|
||||
subBlocks: [
|
||||
{ id: 'triggerCredentials', type: 'oauth-input', title: 'Credentials', serviceId: '{service}', requiredScopes: [], required: true, mode: 'trigger' },
|
||||
// ... service-specific config fields (dropdowns, inputs, switches) ...
|
||||
{ id: 'triggerInstructions', type: 'text', title: 'Setup Instructions', hideFromPreview: true, mode: 'trigger', defaultValue: '...' },
|
||||
],
|
||||
|
||||
outputs: {
|
||||
// Must match the payload shape from processPolledWebhookEvent
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
### Registration (3 places)
|
||||
|
||||
1. **`apps/sim/triggers/constants.ts`** — add provider to `POLLING_PROVIDERS` Set
|
||||
2. **`apps/sim/lib/webhooks/polling/registry.ts`** — import handler, add to `POLLING_HANDLERS`
|
||||
3. **`apps/sim/triggers/registry.ts`** — import trigger config, add to `TRIGGER_REGISTRY`
|
||||
|
||||
### Helm Cron Job
|
||||
|
||||
Add to `helm/sim/values.yaml` under the existing polling cron jobs:
|
||||
|
||||
```yaml
|
||||
{service}WebhookPoll:
|
||||
schedule: "*/1 * * * *"
|
||||
concurrencyPolicy: Forbid
|
||||
url: "http://sim:3000/api/webhooks/poll/{service}"
|
||||
```
|
||||
|
||||
### Reference Implementations
|
||||
|
||||
- Simple: `apps/sim/lib/webhooks/polling/rss.ts` + `apps/sim/triggers/rss/poller.ts`
|
||||
- Complex (OAuth, attachments): `apps/sim/lib/webhooks/polling/gmail.ts` + `apps/sim/triggers/gmail/poller.ts`
|
||||
- Cursor-based (changes API): `apps/sim/lib/webhooks/polling/google-drive.ts`
|
||||
- Timestamp-based: `apps/sim/lib/webhooks/polling/google-calendar.ts`
|
||||
|
||||
## Checklist
|
||||
|
||||
### Trigger Definition
|
||||
- [ ] Created `utils.ts` with options, instructions, extra fields, and output builders
|
||||
- [ ] Primary trigger has `includeDropdown: true`; secondary triggers do NOT
|
||||
- [ ] All triggers use `buildTriggerSubBlocks` helper
|
||||
- [ ] Created `index.ts` barrel export
|
||||
|
||||
### Registration
|
||||
- [ ] All triggers in `triggers/registry.ts` → `TRIGGER_REGISTRY`
|
||||
- [ ] Block has `triggers.enabled: true` and lists all trigger IDs in `triggers.available`
|
||||
- [ ] Block spreads all trigger subBlocks: `...getTrigger('id').subBlocks`
|
||||
|
||||
### Provider Handler (if needed)
|
||||
- [ ] Handler file at `apps/sim/lib/webhooks/providers/{service}.ts`
|
||||
- [ ] Registered in `providers/registry.ts` (alphabetical)
|
||||
- [ ] Signature validator is a private function inside the handler file
|
||||
- [ ] `formatInput` output keys match trigger `outputs` exactly
|
||||
- [ ] Event matching uses dynamic `await import()` for trigger utils
|
||||
|
||||
### Auto Registration (if supported)
|
||||
- [ ] `createSubscription` and `deleteSubscription` on the handler
|
||||
- [ ] NO changes to `route.ts`, `provider-subscriptions.ts`, or `deploy.ts`
|
||||
- [ ] API key field uses `password: true`
|
||||
|
||||
### Polling Trigger (if applicable)
|
||||
- [ ] Handler implements `PollingProviderHandler` at `lib/webhooks/polling/{service}.ts`
|
||||
- [ ] Trigger config has `polling: true` and defines subBlocks manually (no `buildTriggerSubBlocks`)
|
||||
- [ ] Provider string matches across: trigger config, handler, `POLLING_PROVIDERS`, polling registry
|
||||
- [ ] First poll seeds state and emits nothing
|
||||
- [ ] Added provider to `POLLING_PROVIDERS` in `triggers/constants.ts`
|
||||
- [ ] Added handler to `POLLING_HANDLERS` in `lib/webhooks/polling/registry.ts`
|
||||
- [ ] Added cron job to `helm/sim/values.yaml`
|
||||
- [ ] Payload shape matches trigger `outputs` schema
|
||||
|
||||
### Testing
|
||||
- [ ] `bun run type-check` passes
|
||||
- [ ] Manually verify output keys match trigger `outputs` keys
|
||||
- [ ] Trigger UI shows correctly in the block
|
||||
@@ -0,0 +1,143 @@
|
||||
---
|
||||
description: Drive a PR to a clean review (Greptile 5/5, zero open threads) — ships if needed, keeps it mergeable against staging, triggers Greptile/Cursor Bugbot, fixes real findings, replies to and resolves every thread, and loops until clean
|
||||
---
|
||||
|
||||
# Babysit PRs
|
||||
|
||||
Owns a PR end-to-end through review: ship it, wait for the automatic review round, and if it
|
||||
isn't already clean, drive fix → reply → resolve → re-review cycles until Greptile reports 5/5
|
||||
and there are zero open comment threads, keeping the branch mergeable against staging along the
|
||||
way. Designed to be run under `/loop` (no fixed interval — let it self-pace on review latency)
|
||||
so it survives across multiple wakeups in the same session.
|
||||
|
||||
## When to use
|
||||
|
||||
- The user says "babysit this PR", "keep working the reviews until it's clean", or similar
|
||||
- As the natural follow-up to `/ship` when the user wants the review loop automated rather than
|
||||
manually re-triggering reviews and answering comments themselves
|
||||
|
||||
## Inputs
|
||||
|
||||
Needs a PR number. If none is given and there's no open PR for the current branch, run `/ship`
|
||||
first (which includes the `origin/staging` sync check — see `.agents/skills/ship/SKILL.md`) to
|
||||
create one.
|
||||
|
||||
## Definition of "clean"
|
||||
|
||||
Both must hold:
|
||||
1. The latest Greptile summary comment reports **Confidence Score: 5/5**
|
||||
2. `reviewThreads` (GraphQL, see below) has **zero threads with `isResolved: false`**
|
||||
|
||||
Do not stop early on "no new comments this round" alone — a thread can be open from an earlier
|
||||
round. Always check both conditions freshly after every push.
|
||||
|
||||
## Loop
|
||||
|
||||
1. **Check current state** before doing anything, including whether the PR is still mergeable:
|
||||
```bash
|
||||
gh pr view <n> --json mergeable
|
||||
gh pr view <n> --json comments -q '[.comments[] | select(.author.login=="greptile-apps")] | last | .body'
|
||||
gh api graphql -f query='
|
||||
query { repository(owner: "<owner>", name: "<repo>") { pullRequest(number: <n>) {
|
||||
reviewThreads(first: 50) { pageInfo { hasNextPage endCursor } nodes { id isResolved path line
|
||||
comments(first: 5) { nodes { id databaseId author { login } body } } } } } } }'
|
||||
```
|
||||
`[.comments[]] | last | .body`, not `... | .body | tail -1` — the latter pipes every matching
|
||||
comment's full multi-line body through the pipeline and keeps only the final *line* of that
|
||||
combined output (usually the "Reviews (n): Last reviewed commit..." footer), not the last
|
||||
*comment*, so it silently misses the actual "Confidence Score: X/5" line.
|
||||
`reviewThreads(first: 50)` is a single page — check `pageInfo.hasNextPage`. If `true`, don't
|
||||
stop yet: re-run the same query with `after: "<endCursor>"` and keep paging until
|
||||
`hasNextPage` is `false` before evaluating "clean." A PR with more than 50 threads is rare but
|
||||
stopping on a partial page would silently miss unresolved ones past the cutoff.
|
||||
If `mergeable` is `CONFLICTING`, fix that first (step 2). Otherwise, if Greptile is 5/5 and
|
||||
every thread across all pages has `isResolved: true`, stop — report the outcome (see
|
||||
"Reporting" below) and skip the rest of this list.
|
||||
|
||||
2. **If the PR has a merge conflict**, merge `origin/staging`, resolve the conflicts, run the
|
||||
usual pre-push checks, push, and go to step 8 to re-trigger review.
|
||||
|
||||
3. **If no review has run yet** (fresh PR, no Greptile/Cursor comments): they usually run
|
||||
automatically on PR open — confirm via `gh pr checks <n>` (look for `Cursor Bugbot` /
|
||||
`Greptile Review`) and wait for that first round before doing anything else.
|
||||
|
||||
4. **If a review round has landed and it isn't clean**: for every thread where
|
||||
`isResolved: false`, triage the finding on its own merits — this is the part that requires
|
||||
judgment, not a mechanical loop:
|
||||
- **Real bug**: fix it in the cleanest way available. Match the codebase's existing
|
||||
conventions for that kind of problem before inventing a new one (e.g. an SSRF-prone
|
||||
user-supplied-host fetch should use whatever `validateUrlWithDNS`/`secureFetchWithPinnedIP`
|
||||
pattern the rest of the codebase already uses for that exact situation — grep for a sibling
|
||||
integration solving the same problem first). Never patch around a finding with a
|
||||
workaround, a broad try/catch, or a suppression comment — fix the actual cause.
|
||||
- **False positive**: don't change code. Reply with the specific reason it doesn't apply
|
||||
(cite the type definition, the established pattern it matches, or the doc it follows) so
|
||||
the reviewer bot and a human skimming later both understand why it was left as-is.
|
||||
- **Already fixed by an earlier finding in the same round**: note that and resolve without a
|
||||
duplicate code change.
|
||||
|
||||
5. **Reply to every thread individually** before resolving it — never resolve silently:
|
||||
```bash
|
||||
gh api repos/<owner>/<repo>/pulls/<n>/comments/<databaseId>/replies -f body="<what was done and why>"
|
||||
```
|
||||
Then resolve via GraphQL (needs the thread `id` from step 1, not the comment id):
|
||||
```bash
|
||||
gh api graphql -f query='mutation { resolveReviewThread(input: {threadId: "<threadId>"}) { thread { isResolved } } }'
|
||||
```
|
||||
|
||||
6. **Before pushing, re-run the full sync check from `/ship` step 2** — not just the log command,
|
||||
the whole check-and-recover flow (stash WIP if needed, rebase, verify the rebase didn't just
|
||||
cleanly replay stray commits, cherry-pick rebuild if it did or if it conflicted). A babysit
|
||||
loop spanning a long session is exactly the scenario where a branch can drift, and pushing
|
||||
review fixes on top of undetected drift is how an oversized PR happens even after the branch
|
||||
was fixed once. Then run the repo's pre-ship checks the same way `/ship` does before
|
||||
committing — not just lint/typecheck/boundary-validation, but also the conditional `/cleanup`
|
||||
(if this round's fix touched UI code) and `/db-migrate` (if it touched schema/migrations)
|
||||
gates from `/ship` steps 4 and 5. A review-fix round is still a code change and can trip
|
||||
either gate just as easily as the original commit did.
|
||||
|
||||
7. **Commit and push** the round's fixes as one commit — `--force-with-lease` whenever step 6's
|
||||
sync check rewrote history, which includes a plain `git rebase origin/staging` that completed
|
||||
with no conflicts, not only the cherry-pick rebuild path; both rewrite commits already
|
||||
published to the remote, so a plain `git push` can be rejected either way — then run `/ship`
|
||||
step 9's post-push verify — not just before the first push, every push in the loop:
|
||||
```bash
|
||||
git fetch origin staging && git log --oneline --reverse origin/staging..HEAD
|
||||
gh pr view <n> --json commits -q '.commits[].messageHeadline'
|
||||
```
|
||||
`--reverse` makes `git log` oldest-first, matching the PR commit list's order — plain
|
||||
`git log` is newest-first, so without it a positional comparison can spuriously fail on any
|
||||
multi-commit branch.
|
||||
These two lists must describe the same commits. A review loop runs many pushes across many
|
||||
rounds; checking sync only before the push (step 6) and never after is how a bad push or a
|
||||
PR whose commit history quietly went stale between rounds goes unnoticed.
|
||||
|
||||
8. **Re-trigger review** by posting `@greptile` and `@cursor review` as **two separate PR
|
||||
comments** — never combine them into one comment, each bot only responds to its own mention:
|
||||
```bash
|
||||
gh pr comment <n> --body "@greptile"
|
||||
gh pr comment <n> --body "@cursor review"
|
||||
```
|
||||
|
||||
9. **Wait for the new round**, then go back to step 1. Pace the wait with `ScheduleWakeup` using
|
||||
a fallback delay of ~250–300s (Greptile/Cursor typically take 1–3 minutes) — never busy-poll
|
||||
in a sleep loop. Pass the same `/loop babysit PR <n>` prompt on each wakeup so the loop
|
||||
resumes correctly.
|
||||
|
||||
10. **Stop conditions**: clean state reached (see above), or the same unresolved finding or
|
||||
merge conflict survives two consecutive rounds with no new information (surface it to the
|
||||
user instead of looping forever), or the user interrupts.
|
||||
|
||||
## Reporting
|
||||
|
||||
When the loop ends, summarize: how many rounds it took, what was actually fixed (one line each),
|
||||
what was pushed back on as a false positive and why, and the final Greptile score / thread count.
|
||||
|
||||
## Hard rules
|
||||
|
||||
- Never post the two re-review mentions as a single combined comment.
|
||||
- Never resolve a thread without replying to it first.
|
||||
- Never fix a finding with a hacky workaround — if the clean fix isn't obvious, find the sibling
|
||||
pattern elsewhere in the codebase solving the same class of problem and match it.
|
||||
- Never silently drop a finding — every thread gets either a code fix or a reasoned reply.
|
||||
- Always re-run the `/ship`-style sync check before every push in the loop, not just the first.
|
||||
@@ -0,0 +1,62 @@
|
||||
---
|
||||
description: Run all code quality skills — effects, memo, callbacks, state, React Query, emcn design review, url-state, and comments — analyzing in parallel, then applying fixes sequentially
|
||||
argument-hint: "[scope] [fix=true|false]"
|
||||
---
|
||||
|
||||
# Cleanup
|
||||
|
||||
Arguments:
|
||||
- scope: what to review (default: your current changes). Examples: "diff to main", "PR #123", "src/components/", "whole codebase"
|
||||
- fix: whether to apply fixes (default: true). Set to false to only propose changes.
|
||||
|
||||
User arguments: $ARGUMENTS
|
||||
|
||||
## Step 1 — Parallel analysis (read-only)
|
||||
|
||||
First parse the user's `$ARGUMENTS` into `scope` and `fix`: extract the `fix=true|false` token wherever it appears in the string (start, middle, or end), and treat everything else — with that token removed — as `scope`. Defaults: `scope` = your current changes, `fix` = true. The `fix` value is consumed by Step 3 — it does NOT propagate to these passes, which always run `fix=false`.
|
||||
|
||||
Spawn all eight passes concurrently as subagents in a **single message** (multiple Agent tool calls). Each runs its skill on the parsed `scope` with `fix=false` — analysis and proposals ONLY, no edits. Instruct each agent to return its findings as a structured list: for every proposed change, the file path, line range, a one-line description of the change, and the exact before/after so the orchestrator can apply it without re-deriving.
|
||||
|
||||
Run these eight in parallel, substituting the parsed `scope` for `<scope>` in each invocation (pass the real scope text, never the literal `<scope>`):
|
||||
|
||||
1. `/you-might-not-need-an-effect <scope> fix=false`
|
||||
2. `/you-might-not-need-a-memo <scope> fix=false`
|
||||
3. `/you-might-not-need-a-callback <scope> fix=false`
|
||||
4. `/you-might-not-need-state <scope> fix=false`
|
||||
5. `/react-query-best-practices <scope> fix=false`
|
||||
6. `/emcn-design-review <scope> fix=false`
|
||||
7. `/you-might-not-need-url-state <scope> fix=false`
|
||||
8. `/you-might-not-need-a-comment <scope> fix=false`
|
||||
|
||||
## Step 2 — Converge
|
||||
|
||||
Collect all findings into one list, **keeping each proposal tagged with the pass that produced it** — do NOT collapse a file's proposals into a single unlabeled patch, because Step 3 applies in pass order and needs those labels. Detect overlaps where two passes touch the same region (common: a state pass and an effect pass on the same block, or a memo and callback pass on the same component). Reconcile only genuine same-region conflicts, and drop proposals a sibling pass has made moot; a reconciled change inherits the pass label of whichever of its passes comes first in the Step 3 dependency order (effects → state → memo → callback → React Query → url-state → emcn → comments), so it is applied at the earliest safe point. Non-overlapping proposals stay as-is with their own labels. The output is a per-pass list of surviving changes, not a per-file patch.
|
||||
|
||||
## Step 3 — Sequential apply
|
||||
|
||||
If `fix=false`, skip this step — just report the proposals from Step 2.
|
||||
|
||||
Otherwise apply the surviving changes yourself (in the main context, not delegated), iterating **pass by pass** in this dependency order so earlier structural changes settle before later passes build on them:
|
||||
|
||||
1. effects → 2. state → 3. memo → 4. callback → 5. React Query → 6. url-state → 7. emcn design → 8. comments
|
||||
|
||||
For each pass in turn, apply all of that pass's changes, then move to the next pass. A file touched by several passes is therefore edited once per pass, in this order — not once as a merged patch. This is what makes the ordering real: a single merged-per-file patch would collapse all passes into one edit and lose it.
|
||||
|
||||
Comments apply last, on purpose: that pass operates on whatever the earlier structural passes settled the code into, so it never edits lines a sibling pass is about to delete or rewrite.
|
||||
|
||||
**Treat every Step 1 proposal as snapshot-relative, not authoritative.** All passes analyzed the *original* files in parallel, so a proposal's line ranges and before/after text describe the code as it was *before* any edits — once an earlier pass has run, a later pass's snippet may no longer match. So for each change, before applying:
|
||||
|
||||
1. Re-read the file and locate the target by its **content** (the proposal's `old_string` snippet), not by its line number — line numbers from Step 1 are only a hint for where to look, since earlier edits shift them.
|
||||
2. If the `old_string` still matches verbatim, apply it — a content-anchored edit is safe even if its line moved.
|
||||
3. If it no longer matches (an earlier pass altered that region), do **not** force the stale patch. Re-derive the change from the current code by re-applying that pass's rule to the construct, or drop it if a prior pass already made it moot. Never apply a proposal against text it wasn't computed from.
|
||||
|
||||
After all edits, run `bun run lint:check` (it runs `turbo run lint:check` across the repo — there is no per-file target, so run the full check).
|
||||
|
||||
## Step 4 — Summary
|
||||
|
||||
Output a summary across all eight passes: what each found, what was applied vs. skipped-as-redundant, and any proposals that need a human decision.
|
||||
|
||||
## Boundary Audit Guidance
|
||||
|
||||
- When removing route-local Zod schemas, replacing raw `fetch(` calls in hooks, or removing `as unknown as X` casts, do not introduce `// boundary-raw-fetch: <reason>` or `// double-cast-allowed: <reason>` annotations to silence the audit. Fix the underlying call instead — adopt a contract from `@/lib/api/contracts/**` and use `requestJson(contract, ...)` from `@/lib/api/client/request`, or refine the type so the double cast is unnecessary.
|
||||
- Annotations are reserved for legitimate exceptions only: streaming responses, binary downloads, multipart uploads, signed-URL flows, OAuth redirects, external-origin requests, and double casts where no narrower type is available. Each annotation requires a non-empty reason; empty reasons fail `bun run check:api-validation:strict`.
|
||||
@@ -0,0 +1,12 @@
|
||||
---
|
||||
description: Spawn parallel task agents to explore a given area of the codebase from multiple angles, then use their findings to answer the question or build a plan. Use when a task needs broad fan-out exploration across many files before acting.
|
||||
argument-hint: <area-of-interest>
|
||||
---
|
||||
|
||||
Based on the given area of interest, please:
|
||||
|
||||
1. Dig around the codebase in terms of that given area of interest, gather general information such as keywords and architecture overview.
|
||||
2. Spawn off n=10 (unless specified otherwise) task agents to dig deeper into the codebase in terms of that given area of interest, some of them should be out of the box for variance.
|
||||
3. Once the task agents are done, use the information to do what the user wants.
|
||||
|
||||
If user is in plan mode, use the information to create the plan.
|
||||
@@ -0,0 +1,88 @@
|
||||
---
|
||||
description: Author or review a Drizzle DB migration for zero-downtime safety — expand/contract phasing, backward-compatibility with the deployed app version, and writing the `-- migration-safe` acknowledgment the check:migrations lint requires. Use when adding/editing files under `packages/db/migrations/` or changing `packages/db/schema.ts`.
|
||||
---
|
||||
|
||||
# DB Migrate Skill
|
||||
|
||||
You make schema changes that survive a deploy without downtime. The `check:migrations` lint (`scripts/check-migrations-safety.ts`) is the deterministic gate; you are the judgment that decides whether a flagged change is actually safe and writes the annotation that satisfies it.
|
||||
|
||||
## The window (why this matters)
|
||||
|
||||
A deploy runs the migration, then rolls out the new app image via blue/green. The two are **not atomic and cannot be** — during cutover the old task set keeps serving against the **already-migrated** schema. So:
|
||||
|
||||
> Every migration must be backward-compatible with the app version that is *already deployed*.
|
||||
|
||||
If a migration drops a column the old code still reads, renames one, or adds a `NOT NULL` the old inserts don't populate, the old code throws until traffic fully shifts — the downtime we're guarding against. You can't fix this by reordering the pipeline; the only fix is discipline.
|
||||
|
||||
## Expand / contract
|
||||
|
||||
Split every breaking change across **two deploys**:
|
||||
|
||||
1. **Expand** (this PR): additive, backward-compatible schema + code that tolerates *both* the old and new shape.
|
||||
2. **Contract** (a later PR, after expand is fully deployed): remove the old thing, now that nothing reads it.
|
||||
|
||||
Never put expand and contract in the same PR. If this PR both removes the code that used a column *and* drops the column, the old code is still live during cutover — split it.
|
||||
|
||||
### Per-operation playbook
|
||||
|
||||
| You want to | Do (deploy 1 = expand) | Do (deploy 2 = contract) |
|
||||
|---|---|---|
|
||||
| Add a required column | `ADD COLUMN` nullable or `DEFAULT`; code writes it | backfill, then `SET NOT NULL` |
|
||||
| Rename a column/table | add the new name; code dual-writes / reads new-then-old | drop the old name |
|
||||
| Drop a column/table | stop all reads/writes in code; ship it | `DROP` (annotate) |
|
||||
| Change a column type | add a new column of the new type; dual-write | backfill, swap reads, drop old |
|
||||
| Add FK / CHECK | `ADD CONSTRAINT ... NOT VALID` | `VALIDATE CONSTRAINT` separately |
|
||||
| Index an existing table | `COMMIT;` breakpoint → `SET lock_timeout = 0` → `CREATE INDEX CONCURRENTLY IF NOT EXISTS` (see `packages/db/scripts/migrate.ts`) | — |
|
||||
| Drop an index | `COMMIT;` breakpoint → `DROP INDEX CONCURRENTLY` — plain `DROP INDEX` takes ACCESS EXCLUSIVE on the table | — |
|
||||
| Backfill data | batched + idempotent `UPDATE` (keyset/`WHERE`, bounded) | — |
|
||||
|
||||
A `CREATE INDEX`, `ADD COLUMN`, or `ADD CONSTRAINT` against a table **created in the same migration** is always safe (no rows, no live traffic) — the lint already suppresses those.
|
||||
|
||||
## Tracking the contract (don't let it rot)
|
||||
|
||||
The contract half is deferred to a later deploy — and that is exactly when it gets forgotten, leaving dead columns, orphaned tables, and `NOT NULL`s that never land. Every deferred contract must become a durable, greppable TODO.
|
||||
|
||||
When an expand defers a drop, leave a **`contract-pending`** marker on the legacy column/table in `packages/db/schema.ts` — that is the file you will be editing when you finally do the drop, so the reminder lives where the work happens:
|
||||
|
||||
```ts
|
||||
// contract-pending(after #5035 is fully deployed): drop once permission-check.ts stops reading it
|
||||
workspaceId: text('workspace_id'),
|
||||
```
|
||||
|
||||
Format: `contract-pending(<precondition>): <what to drop> — <why it's safe once the precondition holds>`. The precondition names the PR/release that removes the last reader and **must be fully deployed** before the contract ships.
|
||||
|
||||
- **The TODO list is a grep** — always accurate, never drifts: `grep -rn "contract-pending" packages/db apps/sim`. Run it when starting migration work to see what is owed.
|
||||
- For anything with a real owner or schedule, also open a tracking issue and put its number in the marker.
|
||||
- **Close the loop in the contract PR:** the contract migration's `-- migration-safe:` annotation references the expand, and you **delete the `contract-pending` marker** in the same PR:
|
||||
```sql
|
||||
-- migration-safe: contract of #5035 — workspace_id readers removed there, deployed 2026-06-10
|
||||
ALTER TABLE "permission_group" DROP COLUMN "workspace_id";
|
||||
```
|
||||
- An expand merged **without** a marker for the drop it defers, or a contract merged **without** removing its marker, is a bug — flag it in review.
|
||||
|
||||
## The judgment the lint can't do
|
||||
|
||||
The lint flags risky *shapes*; it cannot know whether a given drop is *safe right now*. For each flagged statement, do the work it can't:
|
||||
|
||||
1. **Is the dependency gone?** Grep the app for the table/column: search `apps/sim` and `packages` for the column name, the Drizzle field (camelCase), and the table object. If any live read/write remains, it is **not** safe — fix the code first.
|
||||
2. **Did the expand already ship?** The removal of that read/write must be in a deploy that is *already out*, not this same PR. If it's in this PR, split: land the code change now, do the destructive migration in a follow-up after it deploys.
|
||||
3. **Backfills:** confirm the `UPDATE`/`DELETE` is batched (bounded `WHERE`/keyset, not a single whole-table statement), idempotent (safe to replay — a failed migration re-runs unjournaled files from the top), and safe under concurrent writes from the still-live old app.
|
||||
|
||||
## Workflow
|
||||
|
||||
1. Edit `packages/db/schema.ts`, then `cd packages/db && bunx drizzle-kit generate` to produce the SQL. If this is an expand that defers a drop, leave a `contract-pending` marker on the legacy column (see "Tracking the contract"). If this is the contract, delete the marker it resolves.
|
||||
2. Hand-edit the generated SQL where the playbook requires it: `CONCURRENTLY` + `COMMIT;` breakpoint for indexes on existing tables, `NOT VALID` for constraints, batching for backfills.
|
||||
3. Run `bun run check:migrations` (base defaults to `origin/staging`).
|
||||
- **Hard errors** (`add-not-null-no-default`, `rename`, `index-not-concurrent`, `constraint-not-valid`, …): rewrite into expand/contract. Do **not** try to annotate them away — the lint won't accept it.
|
||||
- **Annotate tier** (`drop-table`, `drop-column`, `drop-default`, `set-not-null`, `alter-type`, `drop-index`): only after you've confirmed steps 1–3 above, add a comment on the line directly above the statement:
|
||||
```sql
|
||||
-- migration-safe: `secret` read removed in v0.6.1 (#1234), shipped two deploys ago
|
||||
ALTER TABLE "webhook" DROP COLUMN "secret";
|
||||
```
|
||||
The reason must be specific and name the PR/version that removed the dependency. An empty reason fails the lint.
|
||||
- **Warnings** (`data-backfill`): non-blocking, but confirm the batching/idempotency before merging.
|
||||
4. Verify locally: `cd packages/db && bun run db:migrate` against a dev DB.
|
||||
|
||||
## Hard rule
|
||||
|
||||
Never annotate a destructive statement just to make the lint pass. The annotation is a claim that you verified the old code no longer depends on it. If you can't make that claim truthfully, the change belongs in a later deploy — tell the user to split it.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,84 @@
|
||||
---
|
||||
description: Review UI code for alignment with the emcn design system — components, tokens, patterns, and conventions
|
||||
argument-hint: "[scope] [fix=true|false]"
|
||||
---
|
||||
|
||||
# EMCN Design Review
|
||||
|
||||
Arguments:
|
||||
- scope: what to review (default: your current changes). Examples: "diff to main", "PR #123", "src/components/", "whole codebase"
|
||||
- fix: whether to apply fixes (default: true). Set to false to only propose changes.
|
||||
|
||||
User arguments: $ARGUMENTS
|
||||
|
||||
## Context
|
||||
|
||||
This codebase uses **emcn**, a custom component library built on Radix UI primitives with CVA variants and CSS variable design tokens. All UI must use emcn components and tokens.
|
||||
|
||||
## Steps
|
||||
|
||||
1. Read the emcn public barrel at `packages/emcn/src/index.ts` (re-exports components, Calendar, Table*, and icons) to know what's available; for the full icon set read `packages/emcn/src/icons/index.ts`
|
||||
2. Read `apps/sim/app/_styles/globals.css` for CSS variable tokens
|
||||
3. Analyze the specified scope against every rule below
|
||||
4. If fix=true, apply the fixes. If fix=false, propose the fixes without applying.
|
||||
|
||||
---
|
||||
|
||||
## Imports
|
||||
|
||||
- Import from `@/components/emcn` barrel, never subpaths
|
||||
- Icons from `@sim/emcn/icons`
|
||||
- Use `cn` from `@/lib/core/utils/cn` for conditional classes
|
||||
|
||||
## Design Tokens
|
||||
|
||||
Use CSS variable pattern (`text-[var(--text-primary)]`), never Tailwind semantics (`text-muted-foreground`) or hardcoded colors (`text-gray-500`, `#333`).
|
||||
|
||||
**Text**: `--text-primary`, `--text-secondary`, `--text-tertiary`, `--text-muted`, `--text-body` (canonical value text), `--text-icon`, `--text-placeholder`, `--text-subtle`, `--text-inverse`, `--text-error`
|
||||
**Surfaces**: `--bg`, `--surface-1` through `--surface-7`, `--surface-hover`, `--surface-active`
|
||||
**Borders**: `--border`, `--border-1`, `--border-muted`
|
||||
**Brand/accent**: `--brand-secondary`, `--brand-accent`
|
||||
**Z-Index**: `--z-dropdown` (100), `--z-modal` (200), `--z-popover` (300), `--z-tooltip` (400), `--z-toast` (500)
|
||||
**Shadows**: `shadow-subtle`, `shadow-medium`, `shadow-overlay`, `shadow-card`
|
||||
**Badges**: `--badge-*` semantic families (success/error/gray/blue/purple/orange/amber/teal/cyan/pink, each with `-bg`/`-text`)
|
||||
|
||||
## Buttons
|
||||
|
||||
Intent-to-variant mapping (read the actual `buttonVariants` in `packages/emcn/src/components/button/button.tsx` for the full variant set — it exposes more than listed here):
|
||||
|
||||
| Action | Variant |
|
||||
|--------|---------|
|
||||
| Toolbar, icon-only | `ghost` |
|
||||
| Create, save, submit | `primary` |
|
||||
| Cancel, close | `default` |
|
||||
| Delete, remove | `destructive` |
|
||||
| Selected state | `active` |
|
||||
| Toggle | `outline` |
|
||||
|
||||
## Delete/Remove Confirmations
|
||||
|
||||
`ChipModal` `size='sm'`, title "Delete/Remove {ItemType}", destructive confirm button, plain Cancel (follow the chip footer layout in `.claude/rules/emcn-components.md`). Use `text-[var(--text-error)]` for irreversible warnings.
|
||||
|
||||
## Toast
|
||||
|
||||
`toast.success()`, `toast.error()`, `toast()` from `@/components/emcn`. Never custom notification UI.
|
||||
|
||||
## Badges
|
||||
|
||||
`red`=error/failed, `gray-secondary`=metadata/roles, `type`=type annotations, `green`=success/active, `gray`=neutral, `amber`=processing, `orange`=paused, `blue`=info. Use `dot` prop for status indicators.
|
||||
|
||||
## Icons
|
||||
|
||||
Default: `size-[14px]`. Color: `text-[var(--text-icon)]`. Scale: 14px > 16px > 12px > 20px. Use the `size-*` shorthand — flag `h-[Npx] w-[Npx]` and `h-N w-N` pairs as refactor targets.
|
||||
|
||||
## Anti-patterns to flag
|
||||
|
||||
- Raw `<button>`/`<input>`, or legacy `Input`/`Textarea`/`Modal`, instead of the canonical chip components (`ChipInput`/`ChipTextarea`/`ChipModal`)
|
||||
- Hand-rolled field rows inside a `ChipModalBody` instead of `ChipModalField`
|
||||
- Hardcoded colors (`text-gray-*`, `#hex`, `rgb()`)
|
||||
- Tailwind semantics (`text-muted-foreground`) instead of CSS variables
|
||||
- Template literal className instead of `cn()`
|
||||
- Inline styles for colors/static values (dynamic values OK)
|
||||
- Importing from emcn subpaths instead of barrel
|
||||
- Arbitrary z-index instead of tokens
|
||||
- Wrong button variant for action type
|
||||
@@ -0,0 +1,678 @@
|
||||
---
|
||||
description: This skill encodes Emil Kowalski's philosophy on UI polish, component design, animation decisions, and the invisible details that make software feel great.
|
||||
---
|
||||
|
||||
# Design Engineering
|
||||
|
||||
## Initial Response
|
||||
|
||||
When this skill is first invoked without a specific question, respond only with:
|
||||
|
||||
> I'm ready to help you build interfaces that feel right, my knowledge comes from Emil Kowalski's design engineering philosophy. If you want to dive even deeper, check out Emil’s course: [animations.dev](https://animations.dev/).
|
||||
|
||||
Do not provide any other information until the user asks a question.
|
||||
|
||||
You are a design engineer with the craft sensibility. You build interfaces where every detail compounds into something that feels right. You understand that in a world where everyone's software is good enough, taste is the differentiator.
|
||||
|
||||
## Core Philosophy
|
||||
|
||||
### Taste is trained, not innate
|
||||
|
||||
Good taste is not personal preference. It is a trained instinct: the ability to see beyond the obvious and recognize what elevates. You develop it by surrounding yourself with great work, thinking deeply about why something feels good, and practicing relentlessly.
|
||||
|
||||
When building UI, don't just make it work. Study why the best interfaces feel the way they do. Reverse engineer animations. Inspect interactions. Be curious.
|
||||
|
||||
### Unseen details compound
|
||||
|
||||
Most details users never consciously notice. That is the point. When a feature functions exactly as someone assumes it should, they proceed without giving it a second thought. That is the goal.
|
||||
|
||||
> "All those unseen details combine to produce something that's just stunning, like a thousand barely audible voices all singing in tune." - Paul Graham
|
||||
|
||||
Every decision below exists because the aggregate of invisible correctness creates interfaces people love without knowing why.
|
||||
|
||||
### Beauty is leverage
|
||||
|
||||
People select tools based on the overall experience, not just functionality. Good defaults and good animations are real differentiators. Beauty is underutilized in software. Use it as leverage to stand out.
|
||||
|
||||
## Review Format (Required)
|
||||
|
||||
When reviewing UI code, you MUST use a markdown table with Before/After columns. Do NOT use a list with "Before:" and "After:" on separate lines. Always output an actual markdown table like this:
|
||||
|
||||
| Before | After | Why |
|
||||
| --- | --- | --- |
|
||||
| `transition: all 300ms` | `transition: transform 200ms ease-out` | Specify exact properties; avoid `all` |
|
||||
| `transform: scale(0)` | `transform: scale(0.95); opacity: 0` | Nothing in the real world appears from nothing |
|
||||
| `ease-in` on dropdown | `ease-out` with custom curve | `ease-in` feels sluggish; `ease-out` gives instant feedback |
|
||||
| No `:active` state on button | `transform: scale(0.97)` on `:active` | Buttons must feel responsive to press |
|
||||
| `transform-origin: center` on popover | `transform-origin: var(--radix-popover-content-transform-origin)` | Popovers should scale from their trigger (not modals — modals stay centered) |
|
||||
|
||||
Wrong format (never do this):
|
||||
|
||||
```
|
||||
Before: transition: all 300ms
|
||||
After: transition: transform 200ms ease-out
|
||||
────────────────────────────
|
||||
Before: scale(0)
|
||||
After: scale(0.95)
|
||||
```
|
||||
|
||||
Correct format: A single markdown table with | Before | After | Why | columns, one row per issue found. The "Why" column briefly explains the reasoning.
|
||||
|
||||
## The Animation Decision Framework
|
||||
|
||||
Before writing any animation code, answer these questions in order:
|
||||
|
||||
### 1. Should this animate at all?
|
||||
|
||||
**Ask:** How often will users see this animation?
|
||||
|
||||
| Frequency | Decision |
|
||||
| ----------------------------------------------------------- | ---------------------------- |
|
||||
| 100+ times/day (keyboard shortcuts, command palette toggle) | No animation. Ever. |
|
||||
| Tens of times/day (hover effects, list navigation) | Remove or drastically reduce |
|
||||
| Occasional (modals, drawers, toasts) | Standard animation |
|
||||
| Rare/first-time (onboarding, feedback forms, celebrations) | Can add delight |
|
||||
|
||||
**Never animate keyboard-initiated actions.** These actions are repeated hundreds of times daily. Animation makes them feel slow, delayed, and disconnected from the user's actions.
|
||||
|
||||
Raycast has no open/close animation. That is the optimal experience for something used hundreds of times a day.
|
||||
|
||||
### 2. What is the purpose?
|
||||
|
||||
Every animation must have a clear answer to "why does this animate?"
|
||||
|
||||
Valid purposes:
|
||||
|
||||
- **Spatial consistency**: toast enters and exits from the same direction, making swipe-to-dismiss feel intuitive
|
||||
- **State indication**: a morphing feedback button shows the state change
|
||||
- **Explanation**: a marketing animation that shows how a feature works
|
||||
- **Feedback**: a button scales down on press, confirming the interface heard the user
|
||||
- **Preventing jarring changes**: elements appearing or disappearing without transition feel broken
|
||||
|
||||
If the purpose is just "it looks cool" and the user will see it often, don't animate.
|
||||
|
||||
### 3. What easing should it use?
|
||||
|
||||
Is the element entering or exiting?
|
||||
Yes → ease-out (starts fast, feels responsive)
|
||||
No →
|
||||
Is it moving/morphing on screen?
|
||||
Yes → ease-in-out (natural acceleration/deceleration)
|
||||
Is it a hover/color change?
|
||||
Yes → ease
|
||||
Is it constant motion (marquee, progress bar)?
|
||||
Yes → linear
|
||||
Default → ease-out
|
||||
|
||||
**Critical: use custom easing curves.** The built-in CSS easings are too weak. They lack the punch that makes animations feel intentional.
|
||||
|
||||
```css
|
||||
/* Strong ease-out for UI interactions */
|
||||
--ease-out: cubic-bezier(0.23, 1, 0.32, 1);
|
||||
|
||||
/* Strong ease-in-out for on-screen movement */
|
||||
--ease-in-out: cubic-bezier(0.77, 0, 0.175, 1);
|
||||
|
||||
/* iOS-like drawer curve (from Ionic Framework) */
|
||||
--ease-drawer: cubic-bezier(0.32, 0.72, 0, 1);
|
||||
```
|
||||
|
||||
**Never use ease-in for UI animations.** It starts slow, which makes the interface feel sluggish and unresponsive. A dropdown with `ease-in` at 300ms _feels_ slower than `ease-out` at the same 300ms, because ease-in delays the initial movement — the exact moment the user is watching most closely.
|
||||
|
||||
**Easing curve resources:** Don't create curves from scratch. Use [easing.dev](https://easing.dev/) or [easings.co](https://easings.co/) to find stronger custom variants of standard easings.
|
||||
|
||||
### 4. How fast should it be?
|
||||
|
||||
| Element | Duration |
|
||||
| ------------------------ | ------------- |
|
||||
| Button press feedback | 100-160ms |
|
||||
| Tooltips, small popovers | 125-200ms |
|
||||
| Dropdowns, selects | 150-250ms |
|
||||
| Modals, drawers | 200-500ms |
|
||||
| Marketing/explanatory | Can be longer |
|
||||
|
||||
**Rule: UI animations should stay under 300ms.** A 180ms dropdown feels more responsive than a 400ms one. A faster-spinning spinner makes the app feel like it loads faster, even when the load time is identical.
|
||||
|
||||
### Perceived performance
|
||||
|
||||
Speed in animation is not just about feeling snappy — it directly affects how users perceive your app's performance:
|
||||
|
||||
- A **fast-spinning spinner** makes loading feel faster (same load time, different perception)
|
||||
- A **180ms select** animation feels more responsive than a **400ms** one
|
||||
- **Instant tooltips** after the first one is open (skip delay + skip animation) make the whole toolbar feel faster
|
||||
|
||||
The perception of speed matters as much as actual speed. Easing amplifies this: `ease-out` at 200ms _feels_ faster than `ease-in` at 200ms because the user sees immediate movement.
|
||||
|
||||
## Spring Animations
|
||||
|
||||
Springs feel more natural than duration-based animations because they simulate real physics. They don't have fixed durations — they settle based on physical parameters.
|
||||
|
||||
### When to use springs
|
||||
|
||||
- Drag interactions with momentum
|
||||
- Elements that should feel "alive" (like Apple's Dynamic Island)
|
||||
- Gestures that can be interrupted mid-animation
|
||||
- Decorative mouse-tracking interactions
|
||||
|
||||
### Spring-based mouse interactions
|
||||
|
||||
Tying visual changes directly to mouse position feels artificial because it lacks motion. Use `useSpring` from Motion (formerly Framer Motion) to interpolate value changes with spring-like behavior instead of updating immediately.
|
||||
|
||||
```jsx
|
||||
import { useSpring } from 'framer-motion';
|
||||
|
||||
// Without spring: feels artificial, instant
|
||||
const rotation = mouseX * 0.1;
|
||||
|
||||
// With spring: feels natural, has momentum
|
||||
const springRotation = useSpring(mouseX * 0.1, {
|
||||
stiffness: 100,
|
||||
damping: 10,
|
||||
});
|
||||
```
|
||||
|
||||
This works because the animation is **decorative** — it doesn't serve a function. If this were a functional graph in a banking app, no animation would be better. Know when decoration helps and when it hinders.
|
||||
|
||||
### Spring configuration
|
||||
|
||||
**Apple's approach (recommended — easier to reason about):**
|
||||
|
||||
```js
|
||||
{ type: "spring", duration: 0.5, bounce: 0.2 }
|
||||
```
|
||||
|
||||
**Traditional physics (more control):**
|
||||
|
||||
```js
|
||||
{ type: "spring", mass: 1, stiffness: 100, damping: 10 }
|
||||
```
|
||||
|
||||
Keep bounce subtle (0.1-0.3) when used. Avoid bounce in most UI contexts. Use it for drag-to-dismiss and playful interactions.
|
||||
|
||||
### Interruptibility advantage
|
||||
|
||||
Springs maintain velocity when interrupted — CSS animations and keyframes restart from zero. This makes springs ideal for gestures users might change mid-motion. When you click an expanded item and quickly press Escape, a spring-based animation smoothly reverses from its current position.
|
||||
|
||||
## Component Building Principles
|
||||
|
||||
### Buttons must feel responsive
|
||||
|
||||
Add `transform: scale(0.97)` on `:active`. This gives instant feedback, making the UI feel like it is truly listening to the user.
|
||||
|
||||
```css
|
||||
.button {
|
||||
transition: transform 160ms ease-out;
|
||||
}
|
||||
|
||||
.button:active {
|
||||
transform: scale(0.97);
|
||||
}
|
||||
```
|
||||
|
||||
This applies to any pressable element. The scale should be subtle (0.95-0.98).
|
||||
|
||||
### Never animate from scale(0)
|
||||
|
||||
Nothing in the real world disappears and reappears completely. Elements animating from `scale(0)` look like they come out of nowhere.
|
||||
|
||||
Start from `scale(0.9)` or higher, combined with opacity. Even a barely-visible initial scale makes the entrance feel more natural, like a balloon that has a visible shape even when deflated.
|
||||
|
||||
```css
|
||||
/* Bad */
|
||||
.entering {
|
||||
transform: scale(0);
|
||||
}
|
||||
|
||||
/* Good */
|
||||
.entering {
|
||||
transform: scale(0.95);
|
||||
opacity: 0;
|
||||
}
|
||||
```
|
||||
|
||||
### Make popovers origin-aware
|
||||
|
||||
Popovers should scale in from their trigger, not from center. The default `transform-origin: center` is wrong for almost every popover. **Exception: modals.** Modals should keep `transform-origin: center` because they are not anchored to a specific trigger — they appear centered in the viewport.
|
||||
|
||||
```css
|
||||
/* Radix UI */
|
||||
.popover {
|
||||
transform-origin: var(--radix-popover-content-transform-origin);
|
||||
}
|
||||
|
||||
/* Base UI */
|
||||
.popover {
|
||||
transform-origin: var(--transform-origin);
|
||||
}
|
||||
```
|
||||
|
||||
Whether the user notices the difference individually does not matter. In the aggregate, unseen details become visible. They compound.
|
||||
|
||||
### Tooltips: skip delay on subsequent hovers
|
||||
|
||||
Tooltips should delay before appearing to prevent accidental activation. But once one tooltip is open, hovering over adjacent tooltips should open them instantly with no animation. This feels faster without defeating the purpose of the initial delay.
|
||||
|
||||
```css
|
||||
.tooltip {
|
||||
transition: transform 125ms ease-out, opacity 125ms ease-out;
|
||||
transform-origin: var(--transform-origin);
|
||||
}
|
||||
|
||||
.tooltip[data-starting-style],
|
||||
.tooltip[data-ending-style] {
|
||||
opacity: 0;
|
||||
transform: scale(0.97);
|
||||
}
|
||||
|
||||
/* Skip animation on subsequent tooltips */
|
||||
.tooltip[data-instant] {
|
||||
transition-duration: 0ms;
|
||||
}
|
||||
```
|
||||
|
||||
### Use CSS transitions over keyframes for interruptible UI
|
||||
|
||||
CSS transitions can be interrupted and retargeted mid-animation. Keyframes restart from zero. For any interaction that can be triggered rapidly (adding toasts, toggling states), transitions produce smoother results.
|
||||
|
||||
```css
|
||||
/* Interruptible - good for UI */
|
||||
.toast {
|
||||
transition: transform 400ms ease;
|
||||
}
|
||||
|
||||
/* Not interruptible - avoid for dynamic UI */
|
||||
@keyframes slideIn {
|
||||
from {
|
||||
transform: translateY(100%);
|
||||
}
|
||||
to {
|
||||
transform: translateY(0);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Use blur to mask imperfect transitions
|
||||
|
||||
When a crossfade between two states feels off despite trying different easings and durations, add subtle `filter: blur(2px)` during the transition.
|
||||
|
||||
**Why blur works:** Without blur, you see two distinct objects during a crossfade — the old state and the new state overlapping. This looks unnatural. Blur bridges the visual gap by blending the two states together, tricking the eye into perceiving a single smooth transformation instead of two objects swapping.
|
||||
|
||||
Combine blur with scale-on-press (`scale(0.97)`) for a polished button state transition:
|
||||
|
||||
```css
|
||||
.button {
|
||||
transition: transform 160ms ease-out;
|
||||
}
|
||||
|
||||
.button:active {
|
||||
transform: scale(0.97);
|
||||
}
|
||||
|
||||
.button-content {
|
||||
transition: filter 200ms ease, opacity 200ms ease;
|
||||
}
|
||||
|
||||
.button-content.transitioning {
|
||||
filter: blur(2px);
|
||||
opacity: 0.7;
|
||||
}
|
||||
```
|
||||
|
||||
Keep blur under 20px. Heavy blur is expensive, especially in Safari.
|
||||
|
||||
### Animate enter states with @starting-style
|
||||
|
||||
The modern CSS way to animate element entry without JavaScript:
|
||||
|
||||
```css
|
||||
.toast {
|
||||
opacity: 1;
|
||||
transform: translateY(0);
|
||||
transition: opacity 400ms ease, transform 400ms ease;
|
||||
|
||||
@starting-style {
|
||||
opacity: 0;
|
||||
transform: translateY(100%);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
This replaces the common React pattern of using `useEffect` to set `mounted: true` after initial render. Use `@starting-style` when browser support allows; fall back to the `data-mounted` attribute pattern otherwise.
|
||||
|
||||
```jsx
|
||||
// Legacy pattern (still works everywhere)
|
||||
useEffect(() => {
|
||||
setMounted(true);
|
||||
}, []);
|
||||
// <div data-mounted={mounted}>
|
||||
```
|
||||
|
||||
## CSS Transform Mastery
|
||||
|
||||
### translateY with percentages
|
||||
|
||||
Percentage values in `translate()` are relative to the element's own size. Use `translateY(100%)` to move an element by its own height, regardless of actual dimensions. This is how Sonner positions toasts and how Vaul hides the drawer before animating in.
|
||||
|
||||
```css
|
||||
/* Works regardless of drawer height */
|
||||
.drawer-hidden {
|
||||
transform: translateY(100%);
|
||||
}
|
||||
|
||||
/* Works regardless of toast height */
|
||||
.toast-enter {
|
||||
transform: translateY(-100%);
|
||||
}
|
||||
```
|
||||
|
||||
Prefer percentages over hardcoded pixel values. They are less error-prone and adapt to content.
|
||||
|
||||
### scale() scales children too
|
||||
|
||||
Unlike `width`/`height`, `scale()` also scales an element's children. When scaling a button on press, the font size, icons, and content scale proportionally. This is a feature, not a bug.
|
||||
|
||||
### 3D transforms for depth
|
||||
|
||||
`rotateX()`, `rotateY()` with `transform-style: preserve-3d` create real 3D effects in CSS. Orbiting animations, coin flips, and depth effects are all possible without JavaScript.
|
||||
|
||||
```css
|
||||
.wrapper {
|
||||
transform-style: preserve-3d;
|
||||
}
|
||||
|
||||
@keyframes orbit {
|
||||
from {
|
||||
transform: translate(-50%, -50%) rotateY(0deg) translateZ(72px) rotateY(360deg);
|
||||
}
|
||||
to {
|
||||
transform: translate(-50%, -50%) rotateY(360deg) translateZ(72px) rotateY(0deg);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### transform-origin
|
||||
|
||||
Every element has an anchor point from which transforms execute. The default is center. Set it to match where the trigger lives for origin-aware interactions.
|
||||
|
||||
## clip-path for Animation
|
||||
|
||||
`clip-path` is not just for shapes. It is one of the most powerful animation tools in CSS.
|
||||
|
||||
### The inset shape
|
||||
|
||||
`clip-path: inset(top right bottom left)` defines a rectangular clipping region. Each value "eats" into the element from that side.
|
||||
|
||||
```css
|
||||
/* Fully hidden from right */
|
||||
.hidden {
|
||||
clip-path: inset(0 100% 0 0);
|
||||
}
|
||||
|
||||
/* Fully visible */
|
||||
.visible {
|
||||
clip-path: inset(0 0 0 0);
|
||||
}
|
||||
|
||||
/* Reveal from left to right */
|
||||
.overlay {
|
||||
clip-path: inset(0 100% 0 0);
|
||||
transition: clip-path 200ms ease-out;
|
||||
}
|
||||
.button:active .overlay {
|
||||
clip-path: inset(0 0 0 0);
|
||||
transition: clip-path 2s linear;
|
||||
}
|
||||
```
|
||||
|
||||
### Tabs with perfect color transitions
|
||||
|
||||
Duplicate the tab list. Style the copy as "active" (different background, different text color). Clip the copy so only the active tab is visible. Animate the clip on tab change. This creates a seamless color transition that timing individual color transitions can never achieve.
|
||||
|
||||
### Hold-to-delete pattern
|
||||
|
||||
Use `clip-path: inset(0 100% 0 0)` on a colored overlay. On `:active`, transition to `inset(0 0 0 0)` over 2s with linear timing. On release, snap back with 200ms ease-out. Add `scale(0.97)` on the button for press feedback.
|
||||
|
||||
### Image reveals on scroll
|
||||
|
||||
Start with `clip-path: inset(0 0 100% 0)` (hidden from bottom). Animate to `inset(0 0 0 0)` when the element enters the viewport. Use `IntersectionObserver` or Framer Motion's `useInView` with `{ once: true, margin: "-100px" }`.
|
||||
|
||||
### Comparison sliders
|
||||
|
||||
Overlay two images. Clip the top one with `clip-path: inset(0 50% 0 0)`. Adjust the right inset value based on drag position. No extra DOM elements needed, fully hardware-accelerated.
|
||||
|
||||
## Gesture and Drag Interactions
|
||||
|
||||
### Momentum-based dismissal
|
||||
|
||||
Don't require dragging past a threshold. Calculate velocity: `Math.abs(dragDistance) / elapsedTime`. If velocity exceeds ~0.11, dismiss regardless of distance. A quick flick should be enough.
|
||||
|
||||
```js
|
||||
const timeTaken = new Date().getTime() - dragStartTime.current.getTime();
|
||||
const velocity = Math.abs(swipeAmount) / timeTaken;
|
||||
|
||||
if (Math.abs(swipeAmount) >= SWIPE_THRESHOLD || velocity > 0.11) {
|
||||
dismiss();
|
||||
}
|
||||
```
|
||||
|
||||
### Damping at boundaries
|
||||
|
||||
When a user drags past the natural boundary (e.g., dragging a drawer up when already at top), apply damping. The more they drag, the less the element moves. Things in real life don't suddenly stop; they slow down first.
|
||||
|
||||
### Pointer capture for drag
|
||||
|
||||
Once dragging starts, set the element to capture all pointer events. This ensures dragging continues even if the pointer leaves the element bounds.
|
||||
|
||||
### Multi-touch protection
|
||||
|
||||
Ignore additional touch points after the initial drag begins. Without this, switching fingers mid-drag causes the element to jump to the new position.
|
||||
|
||||
```js
|
||||
function onPress() {
|
||||
if (isDragging) return;
|
||||
// Start drag...
|
||||
}
|
||||
```
|
||||
|
||||
### Friction instead of hard stops
|
||||
|
||||
Instead of preventing upward drag entirely, allow it with increasing friction. It feels more natural than hitting an invisible wall.
|
||||
|
||||
## Performance Rules
|
||||
|
||||
### Only animate transform and opacity
|
||||
|
||||
These properties skip layout and paint, running on the GPU. Animating `padding`, `margin`, `height`, or `width` triggers all three rendering steps.
|
||||
|
||||
### CSS variables are inheritable
|
||||
|
||||
Changing a CSS variable on a parent recalculates styles for all children. In a drawer with many items, updating `--swipe-amount` on the container causes expensive style recalculation. Update `transform` directly on the element instead.
|
||||
|
||||
```js
|
||||
// Bad: triggers recalc on all children
|
||||
element.style.setProperty('--swipe-amount', `${distance}px`);
|
||||
|
||||
// Good: only affects this element
|
||||
element.style.transform = `translateY(${distance}px)`;
|
||||
```
|
||||
|
||||
### Framer Motion hardware acceleration caveat
|
||||
|
||||
Framer Motion's shorthand properties (`x`, `y`, `scale`) are NOT hardware-accelerated. They use `requestAnimationFrame` on the main thread. For hardware acceleration, use the full `transform` string:
|
||||
|
||||
```jsx
|
||||
// NOT hardware accelerated (convenient but drops frames under load)
|
||||
<motion.div animate={{ x: 100 }} />
|
||||
|
||||
// Hardware accelerated (stays smooth even when main thread is busy)
|
||||
<motion.div animate={{ transform: "translateX(100px)" }} />
|
||||
```
|
||||
|
||||
This matters when the browser is simultaneously loading content, running scripts, or painting. At Vercel, the dashboard tab animation used Shared Layout Animations and dropped frames during page loads. Switching to CSS animations (off main thread) fixed it.
|
||||
|
||||
### CSS animations beat JS under load
|
||||
|
||||
CSS animations run off the main thread. When the browser is busy loading a new page, Framer Motion animations (using `requestAnimationFrame`) drop frames. CSS animations remain smooth. Use CSS for predetermined animations; JS for dynamic, interruptible ones.
|
||||
|
||||
### Use WAAPI for programmatic CSS animations
|
||||
|
||||
The Web Animations API gives you JavaScript control with CSS performance. Hardware-accelerated, interruptible, and no library needed.
|
||||
|
||||
```js
|
||||
element.animate([{ clipPath: 'inset(0 0 100% 0)' }, { clipPath: 'inset(0 0 0 0)' }], {
|
||||
duration: 1000,
|
||||
fill: 'forwards',
|
||||
easing: 'cubic-bezier(0.77, 0, 0.175, 1)',
|
||||
});
|
||||
```
|
||||
|
||||
## Accessibility
|
||||
|
||||
### prefers-reduced-motion
|
||||
|
||||
Animations can cause motion sickness. Reduced motion means fewer and gentler animations, not zero. Keep opacity and color transitions that aid comprehension. Remove movement and position animations.
|
||||
|
||||
```css
|
||||
@media (prefers-reduced-motion: reduce) {
|
||||
.element {
|
||||
animation: fade 0.2s ease;
|
||||
/* No transform-based motion */
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
```jsx
|
||||
const shouldReduceMotion = useReducedMotion();
|
||||
const closedX = shouldReduceMotion ? 0 : '-100%';
|
||||
```
|
||||
|
||||
### Touch device hover states
|
||||
|
||||
```css
|
||||
@media (hover: hover) and (pointer: fine) {
|
||||
.element:hover {
|
||||
transform: scale(1.05);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Touch devices trigger hover on tap, causing false positives. Gate hover animations behind this media query.
|
||||
|
||||
## The Sonner Principles (Building Loved Components)
|
||||
|
||||
These principles come from building Sonner (13M+ weekly npm downloads) and apply to any component:
|
||||
|
||||
1. **Developer experience is key.** No hooks, no context, no complex setup. Insert `<Toaster />` once, call `toast()` from anywhere. The less friction to adopt, the more people will use it.
|
||||
|
||||
2. **Good defaults matter more than options.** Ship beautiful out of the box. Most users never customize. The default easing, timing, and visual design should be excellent.
|
||||
|
||||
3. **Naming creates identity.** "Sonner" (French for "to ring") feels more elegant than "react-toast". Sacrifice discoverability for memorability when appropriate.
|
||||
|
||||
4. **Handle edge cases invisibly.** Pause toast timers when the tab is hidden. Fill gaps between stacked toasts with pseudo-elements to maintain hover state. Capture pointer events during drag. Users never notice these, and that is exactly right.
|
||||
|
||||
5. **Use transitions, not keyframes, for dynamic UI.** Toasts are added rapidly. Keyframes restart from zero on interruption. Transitions retarget smoothly.
|
||||
|
||||
6. **Build a great documentation site.** Let people touch the product, play with it, and understand it before they use it. Interactive examples with ready-to-use code snippets lower the barrier to adoption.
|
||||
|
||||
### Cohesion matters
|
||||
|
||||
Sonner's animation feels satisfying partly because the whole experience is cohesive. The easing and duration fit the vibe of the library. It is slightly slower than typical UI animations and uses `ease` rather than `ease-out` to feel more elegant. The animation style matches the toast design, the page design, the name — everything is in harmony.
|
||||
|
||||
When choosing animation values, consider the personality of the component. A playful component can be bouncier. A professional dashboard should be crisp and fast. Match the motion to the mood.
|
||||
|
||||
### The opacity + height combination
|
||||
|
||||
When items enter and exit a list (like Family's drawer), the opacity change must work well with the height animation. This is often trial and error. There is no formula — you adjust until it feels right.
|
||||
|
||||
### Review your work the next day
|
||||
|
||||
Review animations with fresh eyes. You notice imperfections the next day that you missed during development. Play animations in slow motion or frame by frame to spot timing issues that are invisible at full speed.
|
||||
|
||||
### Asymmetric enter/exit timing
|
||||
|
||||
Pressing should be slow when it needs to be deliberate (hold-to-delete: 2s linear), but release should always be snappy (200ms ease-out). This pattern applies broadly: slow where the user is deciding, fast where the system is responding.
|
||||
|
||||
```css
|
||||
/* Release: fast */
|
||||
.overlay {
|
||||
transition: clip-path 200ms ease-out;
|
||||
}
|
||||
|
||||
/* Press: slow and deliberate */
|
||||
.button:active .overlay {
|
||||
transition: clip-path 2s linear;
|
||||
}
|
||||
```
|
||||
|
||||
## Stagger Animations
|
||||
|
||||
When multiple elements enter together, stagger their appearance. Each element animates in with a small delay after the previous one. This creates a cascading effect that feels more natural than everything appearing at once.
|
||||
|
||||
```css
|
||||
.item {
|
||||
opacity: 0;
|
||||
transform: translateY(8px);
|
||||
animation: fadeIn 300ms ease-out forwards;
|
||||
}
|
||||
|
||||
.item:nth-child(1) {
|
||||
animation-delay: 0ms;
|
||||
}
|
||||
.item:nth-child(2) {
|
||||
animation-delay: 50ms;
|
||||
}
|
||||
.item:nth-child(3) {
|
||||
animation-delay: 100ms;
|
||||
}
|
||||
.item:nth-child(4) {
|
||||
animation-delay: 150ms;
|
||||
}
|
||||
|
||||
@keyframes fadeIn {
|
||||
to {
|
||||
opacity: 1;
|
||||
transform: translateY(0);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Keep stagger delays short (30-80ms between items). Long delays make the interface feel slow. Stagger is decorative — never block interaction while stagger animations are playing.
|
||||
|
||||
## Debugging Animations
|
||||
|
||||
### Slow motion testing
|
||||
|
||||
Play animations at reduced speed to spot issues invisible at full speed. Temporarily increase duration to 2-5x normal, or use browser DevTools animation inspector to slow playback.
|
||||
|
||||
Things to look for in slow motion:
|
||||
|
||||
- Do colors transition smoothly, or do you see two distinct states overlapping?
|
||||
- Does the easing feel right, or does it start/stop abruptly?
|
||||
- Is the transform-origin correct, or does the element scale from the wrong point?
|
||||
- Are multiple animated properties (opacity, transform, color) in sync?
|
||||
|
||||
### Frame-by-frame inspection
|
||||
|
||||
Step through animations frame by frame in Chrome DevTools (Animations panel). This reveals timing issues between coordinated properties that you cannot see at full speed.
|
||||
|
||||
### Test on real devices
|
||||
|
||||
For touch interactions (drawers, swipe gestures), test on physical devices. Connect your phone via USB, visit your local dev server by IP address, and use Safari's remote devtools. The Xcode Simulator is an alternative but real hardware is better for gesture testing.
|
||||
|
||||
## Review Checklist
|
||||
|
||||
When reviewing UI code, check for:
|
||||
|
||||
| Issue | Fix |
|
||||
| ------------------------------------------ | ---------------------------------------------------------------- |
|
||||
| `transition: all` | Specify exact properties: `transition: transform 200ms ease-out` |
|
||||
| `scale(0)` entry animation | Start from `scale(0.95)` with `opacity: 0` |
|
||||
| `ease-in` on UI element | Switch to `ease-out` or custom curve |
|
||||
| `transform-origin: center` on popover | Set to trigger location or use Radix/Base UI CSS variable (modals are exempt — keep centered) |
|
||||
| Animation on keyboard action | Remove animation entirely |
|
||||
| Duration > 300ms on UI element | Reduce to 150-250ms |
|
||||
| Hover animation without media query | Add `@media (hover: hover) and (pointer: fine)` |
|
||||
| Keyframes on rapidly-triggered element | Use CSS transitions for interruptibility |
|
||||
| Framer Motion `x`/`y` props under load | Use `transform: "translateX()"` for hardware acceleration |
|
||||
| Same enter/exit transition speed | Make exit faster than enter (e.g., enter 2s, exit 200ms) |
|
||||
| Elements all appear at once | Add stagger delay (30-80ms between items) |
|
||||
@@ -0,0 +1,147 @@
|
||||
---
|
||||
description: Design engineering principles for making interfaces feel polished. Use when building UI components, reviewing frontend code, implementing animations, hover states, shadows, borders, typography, micro-interactions, enter/exit animations, or any visual detail work. Triggers on UI polish, design details, "make it feel better", "feels off", stagger animations, border radius, optical alignment, font smoothing, tabular numbers, image outlines, box shadows.
|
||||
---
|
||||
|
||||
# Details that make interfaces feel better
|
||||
|
||||
Great interfaces rarely come from a single thing. It's usually a collection of small details that compound into a great experience. Apply these principles when building or reviewing UI code.
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Category | When to Use |
|
||||
| --- | --- |
|
||||
| [Typography](typography.md) | Text wrapping, font smoothing, tabular numbers |
|
||||
| [Surfaces](surfaces.md) | Border radius, optical alignment, shadows, image outlines, hit areas |
|
||||
| [Animations](animations.md) | Interruptible animations, enter/exit transitions, icon animations, scale on press |
|
||||
| [Performance](performance.md) | Transition specificity, `will-change` usage |
|
||||
|
||||
## Core Principles
|
||||
|
||||
### 1. Concentric Border Radius
|
||||
|
||||
Outer radius = inner radius + padding. Mismatched radii on nested elements is the most common thing that makes interfaces feel off.
|
||||
|
||||
### 2. Optical Over Geometric Alignment
|
||||
|
||||
When geometric centering looks off, align optically. Buttons with icons, play triangles, and asymmetric icons all need manual adjustment.
|
||||
|
||||
### 3. Shadows Over Borders
|
||||
|
||||
Layer multiple transparent `box-shadow` values for natural depth. Shadows adapt to any background; solid borders don't.
|
||||
|
||||
### 4. Interruptible Animations
|
||||
|
||||
Use CSS transitions for interactive state changes — they can be interrupted mid-animation. Reserve keyframes for staged sequences that run once.
|
||||
|
||||
### 5. Split and Stagger Enter Animations
|
||||
|
||||
Don't animate a single container. Break content into semantic chunks and stagger each with ~100ms delay.
|
||||
|
||||
### 6. Subtle Exit Animations
|
||||
|
||||
Use a small fixed `translateY` instead of full height. Exits should be softer than enters.
|
||||
|
||||
### 7. Contextual Icon Animations
|
||||
|
||||
Animate icons with `opacity`, `scale`, and `blur` instead of toggling visibility. Use exactly these values: scale from `0.25` to `1`, opacity from `0` to `1`, blur from `4px` to `0px`. If the project has `motion` or `framer-motion` in `package.json`, use `transition: { type: "spring", duration: 0.3, bounce: 0 }` — bounce must always be `0`. If no motion library is installed, keep both icons in the DOM (one absolute-positioned) and cross-fade with CSS transitions using `cubic-bezier(0.2, 0, 0, 1)` — this gives both enter and exit animations without any dependency.
|
||||
|
||||
### 8. Font Smoothing
|
||||
|
||||
Apply `-webkit-font-smoothing: antialiased` to the root layout on macOS for crisper text.
|
||||
|
||||
### 9. Tabular Numbers
|
||||
|
||||
Use `font-variant-numeric: tabular-nums` for any dynamically updating numbers to prevent layout shift.
|
||||
|
||||
### 10. Text Wrapping
|
||||
|
||||
Use `text-wrap: balance` on headings. Use `text-wrap: pretty` for body text to avoid orphans.
|
||||
|
||||
### 11. Image Outlines
|
||||
|
||||
Add a subtle `1px` outline with low opacity to images for consistent depth. The color must be pure black in light mode (`rgba(0, 0, 0, 0.1)`) and pure white in dark mode (`rgba(255, 255, 255, 0.1)`) — never a near-black like slate, zinc, or any tinted neutral. A tinted outline picks up the surface color underneath it and reads as dirt on the image edge.
|
||||
|
||||
### 12. Scale on Press
|
||||
|
||||
A subtle `scale(0.96)` on click gives buttons tactile feedback. Always use `0.96`. Never use a value smaller than `0.95` — anything below feels exaggerated. Add a `static` prop to disable it when motion would be distracting.
|
||||
|
||||
### 13. Skip Animation on Page Load
|
||||
|
||||
Use `initial={false}` on `AnimatePresence` to prevent enter animations on first render. Verify it doesn't break intentional entrance animations.
|
||||
|
||||
### 14. Never Use `transition: all`
|
||||
|
||||
Always specify exact properties: `transition-property: scale, opacity`. Tailwind's `transition-transform` covers `transform, translate, scale, rotate`.
|
||||
|
||||
### 15. Use `will-change` Sparingly
|
||||
|
||||
Only for `transform`, `opacity`, `filter` — properties the GPU can composite. Never use `will-change: all`. Only add when you notice first-frame stutter.
|
||||
|
||||
### 16. Minimum Hit Area
|
||||
|
||||
Interactive elements need at least 40×40px hit area. Extend with a pseudo-element if the visible element is smaller. Never let hit areas of two elements overlap.
|
||||
|
||||
## Common Mistakes
|
||||
|
||||
| Mistake | Fix |
|
||||
| --- | --- |
|
||||
| Same border radius on parent and child | Calculate `outerRadius = innerRadius + padding` |
|
||||
| Icons look off-center | Adjust optically with padding or fix SVG directly |
|
||||
| Hard borders between sections | Use layered `box-shadow` with transparency |
|
||||
| Jarring enter/exit animations | Split, stagger, and keep exits subtle |
|
||||
| Numbers cause layout shift | Apply `tabular-nums` |
|
||||
| Heavy text on macOS | Apply `antialiased` to root |
|
||||
| Animation plays on page load | Add `initial={false}` to `AnimatePresence` |
|
||||
| `transition: all` on elements | Specify exact properties |
|
||||
| First-frame animation stutter | Add `will-change: transform` (sparingly) |
|
||||
| Tiny hit areas on small controls | Extend with pseudo-element to 40×40px |
|
||||
|
||||
## Review Output Format
|
||||
|
||||
Always present changes as a markdown table with **Before** and **After** columns. Include every change you made — not just a subset. Never list findings as separate "Before:" / "After:" lines outside of a table. Group changes by principle using a heading above each table, and keep each row focused on a single diff so the reader can scan the whole list quickly.
|
||||
|
||||
### Example
|
||||
|
||||
#### Concentric border radius
|
||||
| Before | After |
|
||||
| --- | --- |
|
||||
| `rounded-xl` on card + `rounded-xl` on inner button (`p-2`) | `rounded-2xl` on card (`12 + 8`), `rounded-lg` on inner button |
|
||||
| `border-radius: 16px` on both nested surfaces | Outer `24px`, inner `16px` with `8px` padding |
|
||||
|
||||
#### Tabular numbers
|
||||
| Before | After |
|
||||
| --- | --- |
|
||||
| `<span>{count}</span>` on animated counter | `<span className="tabular-nums">{count}</span>` |
|
||||
| Default numerals on timer | Added `font-variant-numeric: tabular-nums` to root |
|
||||
|
||||
#### Scale on press
|
||||
| Before | After |
|
||||
| --- | --- |
|
||||
| `<button className="...">` | Added `active:scale-[0.96] transition-transform` |
|
||||
| `scale(0.9)` on press | Raised to `scale(0.96)` — anything below `0.95` feels exaggerated |
|
||||
|
||||
Rows should cite the specific file and the specific property that changed when it isn't obvious from the snippet. If a principle was reviewed but nothing needed to change, omit that table entirely — empty tables add noise.
|
||||
|
||||
## Review Checklist
|
||||
|
||||
- [ ] Nested rounded elements use concentric border radius
|
||||
- [ ] Icons are optically centered, not just geometrically
|
||||
- [ ] Shadows used instead of borders where appropriate
|
||||
- [ ] Enter animations are split and staggered
|
||||
- [ ] Exit animations are subtle
|
||||
- [ ] Dynamic numbers use tabular-nums
|
||||
- [ ] Font smoothing is applied
|
||||
- [ ] Headings use text-wrap: balance
|
||||
- [ ] Images have subtle outlines
|
||||
- [ ] Buttons use scale on press where appropriate
|
||||
- [ ] AnimatePresence uses `initial={false}` for default-state elements
|
||||
- [ ] No `transition: all` — only specific properties
|
||||
- [ ] `will-change` only on transform/opacity/filter, never `all`
|
||||
- [ ] Interactive elements have at least 40×40px hit area
|
||||
|
||||
## Reference Files
|
||||
|
||||
- [typography.md](typography.md) — Text wrapping, font smoothing, tabular numbers
|
||||
- [surfaces.md](surfaces.md) — Border radius, optical alignment, shadows, image outlines
|
||||
- [animations.md](animations.md) — Interruptible animations, enter/exit transitions, icon animations, scale on press
|
||||
- [performance.md](performance.md) — Transition specificity, `will-change` usage
|
||||
@@ -0,0 +1,163 @@
|
||||
---
|
||||
description: Review PRs and diffs for unbounded memory loading, concurrency explosions, oversized payload materialization, and missing pagination or byte caps. Use when reviewing cleanup jobs, background jobs, data imports/exports, file parsing, API fan-out, workflow execution payloads, large arrays/files, or any change that reads many rows, files, responses, logs, or external API pages into process memory.
|
||||
---
|
||||
|
||||
# Memory Load Check
|
||||
|
||||
Use this skill when a PR or diff could load unbounded data into a Node/Bun process, especially in cron routes, background tasks, API routes, workflow execution, file parsing, cleanup jobs, migrations, import/export flows, and external API integrations.
|
||||
|
||||
## Review Goal
|
||||
|
||||
Prove each changed path has explicit bounds for:
|
||||
- rows held in memory
|
||||
- bytes held in memory
|
||||
- concurrent promises, DB queries, HTTP calls, storage operations, and jobs
|
||||
- number of pages, batches, chunks, retries, and retained intermediate objects
|
||||
|
||||
If any bound depends only on current production size or "probably small" data, treat it as a finding.
|
||||
|
||||
## References
|
||||
|
||||
Read these when doing a deeper pass:
|
||||
- Node.js streams/backpressure: https://nodejs.org/learn/modules/backpressuring-in-streams
|
||||
- Node.js stream usage: https://nodejs.org/en/learn/modules/how-to-use-streams
|
||||
- Keyset/cursor pagination over offset scans: https://blog.sequinstream.com/keyset-cursors-not-offsets-for-postgres-pagination/
|
||||
- Postgres pagination tradeoffs: https://www.citusdata.com/blog/2016/03/30/five-ways-to-paginate/
|
||||
|
||||
## Sim Helpers To Prefer
|
||||
|
||||
- `apps/sim/lib/cleanup/batch-delete.ts`
|
||||
- `chunkedBatchDelete`: bounded SELECT -> optional side effect -> DELETE loop.
|
||||
- `batchDeleteByWorkspaceAndTimestamp`: common workspace/timestamp cleanup wrapper.
|
||||
- `selectRowsByIdChunks`: chunks large ID sets and enforces an overall row cap.
|
||||
- `chunkArray`: use only after the input set itself is already bounded.
|
||||
- `apps/sim/lib/core/utils/stream-limits.ts`
|
||||
- `PayloadSizeLimitError`
|
||||
- `assertKnownSizeWithinLimit`
|
||||
- `assertContentLengthWithinLimit`
|
||||
- `readStreamToBufferWithLimit`
|
||||
- `readNodeStreamToBufferWithLimit`
|
||||
- `readResponseToBufferWithLimit`
|
||||
- `readResponseTextWithLimit`
|
||||
- Cleanup dispatcher pattern in `apps/sim/lib/billing/cleanup-dispatcher.ts`
|
||||
- page active workspaces with `WHERE id > afterId ORDER BY id LIMIT N`
|
||||
- dispatch concrete chunks (`workspaceIds`, retention, label) instead of one giant scope
|
||||
- prefer Trigger.dev queue/concurrency keys when available
|
||||
- execute inline fallback chunks sequentially, not with unbounded `Promise.all`
|
||||
- File parse route pattern in `apps/sim/app/api/files/parse/route.ts`
|
||||
- cap downloads and parsed output separately
|
||||
- preserve partial results when a later item exceeds the cap
|
||||
- never read untrusted response bodies without a byte cap
|
||||
- KB connector file downloads in `apps/sim/connectors/utils.ts`
|
||||
- `CONNECTOR_MAX_FILE_BYTES`: shared per-file cap (aligned with the manual KB upload limit)
|
||||
- `readBodyWithLimit`: stream a download body to a Buffer with a hard byte cap (null on overflow)
|
||||
- `stubOrSkipBySize`: listing-time skip when the reported size exceeds the cap
|
||||
- `markSkipped` / `sizeLimitSkipReason`: surface oversized files as failed (skipped) KB rows
|
||||
- `ConnectorFileTooLargeError`: thrown mid-download when the listing under-reported size
|
||||
- Large workflow value payloads
|
||||
- prefer durable references/manifests over inlining large arrays or files
|
||||
- materialize refs only behind an explicit byte budget
|
||||
|
||||
## KB Connector File Size Handling
|
||||
|
||||
The connector size pattern in `apps/sim/connectors/utils.ts` (`CONNECTOR_MAX_FILE_BYTES` + `readBodyWithLimit` + `stubOrSkipBySize`/`markSkipped`) exists for one risk: a knowledge-base connector downloading **arbitrary, user-controlled file bytes** that the source does not hard-cap. Apply it by that risk, not by the connector's name.
|
||||
|
||||
Use the pattern when the connector downloads file content via a stream/`download_url` where the user controls the size:
|
||||
- file-storage connectors: Dropbox, OneDrive, SharePoint, Google Drive, S3, GitHub, GitLab, Azure DevOps
|
||||
- any connector that fetches a file via a download URL even if it is not a "storage" service (e.g. the Zoom transcript `.vtt`)
|
||||
|
||||
For those, require all three:
|
||||
- stream the body with `readBodyWithLimit(resp, CONNECTOR_MAX_FILE_BYTES)` — never raw `response.text()`/`response.arrayBuffer()`
|
||||
- skip oversize at listing (`stubOrSkipBySize` with the reported size) and again at fetch time (overflow -> `markSkipped`), since the listing size can be missing or under-reported
|
||||
- never drop/truncate silently — oversized files become content-less failed rows carrying `skippedReason`, so they stay visible in the KB UI instead of vanishing from the index
|
||||
|
||||
Skip the pattern when the source already bounds the payload:
|
||||
- pure API/structured-data connectors (Jira, Linear, Notion, Confluence, Sentry, Slack, Zendesk, Gmail, ...) — paginated JSON/text; apply normal pagination + concurrency bounds instead of a per-file byte cap
|
||||
- native-document connectors capped by the platform (Google Docs ~50 MB, Google Sheets via `MAX_ROWS`, Evernote ~25 MB/note) — a 100 MB cap can never fire, and wrapping a `response.json()`/Thrift parse in `readBodyWithLimit` is cargo-culting
|
||||
|
||||
Litmus test: "Can a user make this one fetch arbitrarily large, with nothing upstream stopping it?" Yes -> use the pattern. No (platform hard-cap, or already paginated) -> a per-file byte cap adds noise, not safety. Borderline: a user-configured/self-hosted endpoint with no platform cap (e.g. Obsidian) — bound it only if the content is genuinely unbounded.
|
||||
|
||||
## Review Workflow
|
||||
|
||||
1. Identify every changed data source:
|
||||
- database queries
|
||||
- storage lists/downloads/uploads
|
||||
- external API pagination
|
||||
- file reads and HTTP responses
|
||||
- workflow logs, snapshots, payloads, arrays, and manifests
|
||||
- queues, cron routes, and background jobs
|
||||
2. For each source, write down the maximum cardinality and maximum bytes. If the code does not enforce one, it is unbounded.
|
||||
3. Trace whether data is processed incrementally or accumulated:
|
||||
- arrays from `select`, `findMany`, `Promise.all`, `map`, `filter`, `flatMap`
|
||||
- maps/sets keyed by all users, workspaces, executions, files, or rows
|
||||
- `Buffer.concat`, `response.arrayBuffer()`, `response.text()`, `JSON.stringify`, `JSON.parse`
|
||||
- queues of promises or job payloads built before dispatch
|
||||
4. Check concurrency separately from memory:
|
||||
- no `Promise.all(items.map(...))` unless `items` is already small and bounded
|
||||
- use chunks, sequential loops, queue concurrency, or a concurrency limiter
|
||||
- align concurrency with DB pool size, storage/API limits, and task queue semantics
|
||||
5. Verify SQL shape:
|
||||
- every bulk query has `LIMIT`
|
||||
- large pagination uses cursor/keyset style (`id > afterId`, timestamps plus unique ID), not deep `OFFSET`
|
||||
- `IN (...)` lists are chunked
|
||||
- side-effect rows selected before delete have per-batch and per-run caps
|
||||
6. Verify byte safety:
|
||||
- check `Content-Length` when available
|
||||
- stream with cumulative byte accounting
|
||||
- cap both input bytes and expanded output bytes
|
||||
- reject or reference oversized values before serializing large JSON responses
|
||||
7. Confirm failure behavior:
|
||||
- exceeding a cap should stop before loading more data
|
||||
- partial successful work should be preserved when the API contract expects it
|
||||
- retries should not duplicate huge in-memory state
|
||||
- cleanup jobs should make progress over future runs instead of widening one run
|
||||
|
||||
## Red Flags
|
||||
|
||||
- loads all active workspaces, users, executions, logs, files, messages, or subscriptions before filtering
|
||||
- builds a full `Map` or `Set` for a platform-wide scope
|
||||
- uses `Promise.all` over rows from an unbounded query
|
||||
- fetches all pages from an external API before processing
|
||||
- reads an entire file, HTTP response, or stream without a max byte budget
|
||||
- checks size only after `Buffer.concat`, `arrayBuffer`, `text`, `JSON.parse`, or parse expansion
|
||||
- a KB connector silently drops or truncates an oversized file instead of recording it as a failed (skipped) row
|
||||
- chunks only after loading the complete dataset
|
||||
- paginates with unbounded/deep `OFFSET` on a mutable or large table
|
||||
- creates one queue job per row without batching or a queue-level concurrency key
|
||||
- accumulates per-row errors/results with no maximum
|
||||
- adds a cache, singleton, or module-level collection without eviction or size limits
|
||||
|
||||
## Preferred Fixes
|
||||
|
||||
- Move filters into SQL/API requests and select only needed columns.
|
||||
- Replace full-table loads with cursor/keyset pagination and a deterministic order.
|
||||
- Process one page/batch at a time; do not keep previous pages unless needed.
|
||||
- Add per-batch and per-run row caps so long backlogs drain across repeated jobs.
|
||||
- Split large ID lists with `selectRowsByIdChunks` or `chunkArray` after bounding the source.
|
||||
- Use `chunkedBatchDelete` for cleanup loops with row side effects.
|
||||
- Use stream-limit helpers for file/HTTP/body reads.
|
||||
- Store large workflow values as refs/manifests and materialize only within a caller budget.
|
||||
- Replace unbounded `Promise.all` with sequential chunk loops, queue concurrency, or a small limiter.
|
||||
- Include tests that prove caps stop work early and partial results or progress are preserved.
|
||||
|
||||
## Findings Format
|
||||
|
||||
Lead with concrete findings, ordered by risk:
|
||||
|
||||
```markdown
|
||||
## Findings
|
||||
|
||||
- **P1 Unbounded workspace load in cleanup dispatch** (`path/to/file.ts`)
|
||||
The new path calls `select().from(workspace)` without a limit, then builds maps for every row before dispatch. In production this scales with all active workspaces and can exhaust the app process. Page by `workspace.id` with a fixed limit and dispatch bounded chunks.
|
||||
|
||||
## Good Signals
|
||||
|
||||
- Uses `readResponseToBufferWithLimit` for external downloads.
|
||||
- Inline fallback processes chunks sequentially.
|
||||
|
||||
## Residual Risk
|
||||
|
||||
- The row cap is explicit, but no test currently proves the loop stops at the cap.
|
||||
```
|
||||
|
||||
Only say "good to go" when every changed source has explicit row, byte, and concurrency bounds or the boundedness is proven by a stable invariant.
|
||||
@@ -0,0 +1,54 @@
|
||||
---
|
||||
description: Audit React Query usage for best practices — key factories, staleTime, mutations, and server state ownership
|
||||
argument-hint: "[scope] [fix=true|false]"
|
||||
---
|
||||
|
||||
# React Query Best Practices
|
||||
|
||||
Arguments:
|
||||
- scope: what to analyze (default: your current changes). Examples: "diff to main", "PR #123", "src/hooks/queries/", "whole codebase"
|
||||
- fix: whether to apply fixes (default: true). Set to false to only propose changes.
|
||||
|
||||
User arguments: $ARGUMENTS
|
||||
|
||||
## Context
|
||||
|
||||
This codebase uses React Query (TanStack Query) as the single source of truth for all server state. All query hooks live in `hooks/queries/`. Zustand is used only for client-only UI state. Server data must never be duplicated into useState or Zustand outside of mutation callbacks that coordinate cross-store state.
|
||||
|
||||
## References
|
||||
|
||||
Read these before analyzing:
|
||||
1. https://tkdodo.eu/blog/practical-react-query — foundational defaults, custom hooks, avoiding local state copies
|
||||
2. https://tkdodo.eu/blog/effective-react-query-keys — key factory pattern, hierarchical keys, fuzzy invalidation
|
||||
3. https://tkdodo.eu/blog/react-query-as-a-state-manager — React Query IS your server state manager
|
||||
|
||||
## Rules to enforce
|
||||
|
||||
### Query key factories
|
||||
- Every file in `hooks/queries/` must have a hierarchical key factory with an `all` root key
|
||||
- Keys must include intermediate plural keys (`lists`, `details`) for prefix invalidation
|
||||
- Key factories are colocated with their query hooks, not in a global keys file
|
||||
|
||||
### Query hooks
|
||||
- Every `queryFn` must forward `signal` for request cancellation
|
||||
- Every query must have an explicit `staleTime` (default 0 is almost never correct), assigned from a named exported constant — never an inline numeric literal. A server-side prefetch hydrating the same query key must import and reuse that constant instead of restating the number
|
||||
- `keepPreviousData` / `placeholderData` only on variable-key queries (where params change), never on static keys
|
||||
- Use `enabled` to prevent queries from running without required params
|
||||
|
||||
### Mutations
|
||||
- Use `onSettled` (not `onSuccess`) for cache reconciliation — it fires on both success and error
|
||||
- For optimistic updates: save previous data in `onMutate`, roll back in `onError`
|
||||
- Use targeted invalidation (`entityKeys.lists()`) not broad (`entityKeys.all`) when possible
|
||||
- Don't include mutation objects in `useCallback` deps — `.mutate()` is stable
|
||||
|
||||
### Server state ownership
|
||||
- Never copy query data into useState. Use query data directly in components.
|
||||
- Never copy query data into Zustand stores (exception: mutation callbacks that coordinate cross-store state like temp ID replacement)
|
||||
- The query cache is not a local state manager — `setQueryData` is for optimistic updates only
|
||||
- Forms are the one deliberate exception: copy server data into local form state with `staleTime: Infinity`
|
||||
|
||||
## Steps
|
||||
|
||||
1. Read the references above to understand the guidelines
|
||||
2. Analyze the specified scope against the rules listed above
|
||||
3. If fix=true, apply the fixes. If fix=false, propose the fixes without applying.
|
||||
@@ -0,0 +1,147 @@
|
||||
---
|
||||
description: Commit, push, and open a PR to staging in one shot — runs the cleanup pass and, when migrations changed, the db-migrate safety review first
|
||||
argument-hint: "[optional context or scope notes]"
|
||||
---
|
||||
|
||||
# Ship Command
|
||||
|
||||
You help ship code by creating commits, pushing to the remote branch, and creating PRs in the user's voice.
|
||||
|
||||
## Your Task
|
||||
|
||||
When the user runs `/ship`:
|
||||
|
||||
1. **Check git status** - See what files have changed
|
||||
2. **Sync check**: `git fetch origin staging && git log --oneline origin/staging..HEAD`. Read the actual commit list, not just how many there are — it must show ONLY commits you can attribute to this session (recognizable subjects/SHAs). A worktree/branch can silently be cut from a stale local `staging`, dragging in unrelated commits; a corrupted branch's inflated commit *count* can coincidentally match a later check even when the *commits* are wrong, so always compare content, never just a number.
|
||||
- If it shows commits you don't recognize, fix it now, **before** staging/committing any new work (step 7 hasn't run yet):
|
||||
- If the working tree has uncommitted changes, stash them first — `git stash push -u -m ship-sync-fix` — so the rebase below isn't blocked by dirty state. Restore with `git stash pop` once the branch is fixed.
|
||||
- Try `git rebase origin/staging` first.
|
||||
- **A rebase finishing without conflicts does NOT by itself mean the branch is clean** — it can replay stray commits onto the new base with no conflict at all. After the rebase (clean or not), re-run `git log --oneline origin/staging..HEAD` and re-check the commit list against what you recognize.
|
||||
- If the rebase conflicted on commits you don't recognize, OR it finished cleanly but the re-checked log still shows commits you don't recognize, abandon that result (`git rebase --abort` if still mid-rebase) and rebuild instead, in this exact order:
|
||||
1. **While still on `<original-branch>`**, identify the SHA(s) to preserve — **not** the whole range. `git log --oneline --reverse origin/staging..<original-branch>` lists everything ahead of `origin/staging`, but in exactly this scenario that range also contains the unrecognized/stray commits you're trying to leave behind — blindly cherry-picking the full range recreates the same polluted branch. Read the list and write down only the SHA(s) you recognize as your own session's work (e.g. `abc1234 def5678`); do this *before* touching any temp branch, since once you check out `ship-sync-tmp` at `origin/staging` in step 4, `HEAD` no longer contains these commits and the same lookup at that point returns nothing.
|
||||
2. `git checkout <original-branch>` — harmless no-op if you're already there, but required if an earlier interrupted attempt left you sitting on `ship-sync-tmp`: git refuses to delete the branch you're currently on, so deleting it before switching away silently fails and blocks the rest of the rebuild.
|
||||
3. Delete any leftover from an earlier attempt: `git branch -D ship-sync-tmp 2>/dev/null || true` — always succeeds, including when there's nothing to delete (a first attempt), so it never blocks the rest of the rebuild on its own exit code.
|
||||
4. `git checkout -b ship-sync-tmp origin/staging`.
|
||||
5. `git cherry-pick` the SHAs captured in step 1, **in that oldest-first order** — cherry-picking more than one session commit out of order can fail or produce the wrong history. Resolve conflicts.
|
||||
6. `git branch -f <original-branch> HEAD`, `git checkout <original-branch>`, and delete `ship-sync-tmp` (`git branch -D ship-sync-tmp`).
|
||||
- Re-verify with `git log --oneline origin/staging..HEAD` — it must list only commits you recognize before you proceed to committing new work.
|
||||
3. **Generate a commit message** following this format: `type(scope): description`
|
||||
- Types: `fix`, `feat`, `improvement`, `chore`
|
||||
- Scope: short identifier (e.g., `undo-redo`, `api`, `ui`)
|
||||
- Keep it concise
|
||||
4. **Run the cleanup pass** — only if the diff modifies UI code (any `.tsx` file, or anything under `apps/sim/components/`, `apps/sim/hooks/`, or `apps/sim/stores/`): `/cleanup`
|
||||
- The six code-quality skills (effects, memo, callbacks, state, React Query, emcn) only apply to React code, so skip this step entirely when no UI was touched. When it runs, it applies fixes so they land in this commit.
|
||||
5. **Run migration safety** — only if the diff touches `packages/db/migrations/**` or `packages/db/schema.ts`:
|
||||
- Run `/db-migrate` to review the migration for zero-downtime safety (expand/contract phasing, backward-compatibility with the deployed app version).
|
||||
- `bun run check:migrations origin/staging` must pass (staging is the PR base). Do not silence a flagged statement with a `-- migration-safe:` annotation unless `/db-migrate` confirmed the old code no longer depends on it; otherwise split the destructive change into a later deploy.
|
||||
6. **Run pre-ship checks** from the repo root before staging. This has two phases: first **regenerate** every committed artifact so generated files never drift into a CI failure (this is what catches things like `agent-stream-docs` going stale after a `models.ts` edit), then run the **full audit suite** CI's `Lint and Test` job enforces. Both phases parallelize — but only across commands that write **disjoint** outputs — and a bare `wait` swallows child exit codes, so both phases below explicitly collect each job's status and abort ship if any failed.
|
||||
|
||||
**Phase A — regenerate the always-in-repo committed artifacts (parallel), then let step 7 stage whatever changed.** Regenerate only the generators whose inputs live entirely in this repo and that any ordinary code change can drift — `agent-stream-docs:generate` (derives from the provider model registry) and `skills:sync` (derives from `.agents/skills/**`). They write disjoint trees (`apps/docs/…/agent.mdx` vs the `.claude`/`.cursor` command projections), so they parallelize safely, and each is idempotent (a no-op when already in sync):
|
||||
```bash
|
||||
rm -f /tmp/ship-gen-results
|
||||
for g in agent-stream-docs:generate skills:sync; do
|
||||
( bun run "$g" >"/tmp/ship-gen-${g//:/-}.log" 2>&1; echo "$? $g" >>/tmp/ship-gen-results ) &
|
||||
done
|
||||
wait
|
||||
# any non-zero line is a FAILED generator — read /tmp/ship-gen-<name>.log and fix before shipping;
|
||||
# a silently-failed generate leaves a stale artifact that Phase B / CI then rejects.
|
||||
# The `exit 1` makes this block itself exit non-zero on failure, so anything gating on the
|
||||
# command's status (an agent, or a wrapping script) actually stops — do NOT collapse it to
|
||||
# `grep … && echo ❌ || echo ✅`, which always exits 0 and silently lets ship continue.
|
||||
if grep -vE '^0 ' /tmp/ship-gen-results; then echo "❌ generator(s) failed — do not ship"; exit 1; fi
|
||||
echo "✅ artifacts regenerated"
|
||||
```
|
||||
Then `git status --short` to see what regenerated — those files must be staged in step 7 alongside your own changes.
|
||||
|
||||
**Do NOT blanket-run the domain generators here.** `mship:generate` (`generate-mship-contracts.ts`) is an **umbrella** that drives all nine mothership contract generators (`mship-contracts`, `billing-protocol-contract`, `mship-tools`, the four `trace-*`, `metrics-contract`, `vfs-snapshot-contract`) and biome-formats `apps/sim/lib/copilot/generated/` — never run it *and* its constituents (they write the same files and corrupt each other in parallel), and never run it on an ordinary ship: it reads an **external** copilot-contract source that isn't checked out in most worktrees, so it hard-fails with `ENOENT` and would abort ship for an unrelated reason. `generate:pi-model-catalog` (under `apps/sim`) likewise regenerates from the installed Pi package, not repo source. `scripts/generate-docs.ts` rewrites the integration docs and client-safe catalog; run it when this PR changes their block/icon/landing-content inputs or when `integration-catalog:check` reports drift, then review its broad generated diff. Only when **this PR's diff actually touches** a domain generator's input do you regenerate it deliberately and run its matching `:check` (`bun run mship:check` / the individual `*:check`) — with the external source present.
|
||||
|
||||
**Phase B — run lint + every audit CI enforces, in parallel, and abort ship if any fails.** Before running the commands, compare this list with `.github/workflows/test-build.yml`; when CI adds an audit, run it and update this skill instead of trusting a stale snapshot. The env-flag audit is currently an inline workflow block rather than a package script: when `apps/sim/lib/core/config/env-flags.ts` changed, run that current workflow block verbatim instead of copying a second version into this skill. Run `bun run lint` first (it autofixes formatting and mutates files, so don't parallelize it with the read-only audits), then run the base-sensitive block-registry check, then fan the independent audits out and collect exit codes:
|
||||
```bash
|
||||
# autofix formatting first (mutating; not parallel-safe with the audits). Gate its exit too —
|
||||
# a non-zero lint (unfixable errors) must abort before the audits run, not be ignored.
|
||||
bun run lint || { echo "❌ lint failed — do not ship"; exit 1; }
|
||||
bun run apps/sim/scripts/check-block-registry.ts origin/staging || {
|
||||
echo "❌ block registry audit failed — do not ship"
|
||||
exit 1
|
||||
}
|
||||
# Runs every audit CI runs, concurrently, and replays the output of any that fail.
|
||||
# Do not hand-list the audits here: the list is derived in scripts/run-audits.ts, and the
|
||||
# copy that used to live in this file had already drifted five audits behind package.json.
|
||||
bun run check:audits || { echo "❌ audit(s) failed — do not ship"; exit 1; }
|
||||
```
|
||||
If Phase A regenerated a file, its matching `:check` in Phase B now passes trivially — that parity is the point. Do not ship with any generator or audit failing; fix the cause (never silence it) and re-run. `check:migrations` and `type-check` are covered by steps 5 and CI respectively and are not repeated here.
|
||||
7. **Stage and commit** the changes with the generated message — including any files Phase A regenerated in step 6
|
||||
8. **Push to origin** using the current branch name — `--force-with-lease` if step 2's sync
|
||||
check did any history rewrite (a clean rebase or a cherry-pick rebuild) on a branch that had
|
||||
already been pushed once; a plain push would be rejected in exactly the polluted-remote case
|
||||
step 2 exists to fix
|
||||
9. **Create a PR** to staging with a description in the user's voice, then do a final content check — not a count check — comparing what actually landed:
|
||||
```bash
|
||||
git fetch origin staging && git log --oneline --reverse origin/staging..HEAD
|
||||
gh pr view <n> --json commits -q '.commits[].messageHeadline'
|
||||
```
|
||||
Re-fetch first — comparing against a stale local `origin/staging` ref can mask real drift or
|
||||
flag a false mismatch even when the branch and push are correct. `--reverse` makes the git log
|
||||
oldest-first, matching the PR commit list's order — plain `git log` is newest-first, and a
|
||||
positional/line-by-line comparison against the PR's oldest-first list can spuriously fail on
|
||||
any multi-commit branch. These two lists must describe the same commits in the same order
|
||||
(same subjects, the last one being the commit from step 7). If they don't match, the branch
|
||||
still has a problem — redo step 2's fix and `git push --force-with-lease`.
|
||||
|
||||
## Commit Message Format
|
||||
|
||||
Based on the repo's commit history:
|
||||
|
||||
```
|
||||
fix(scope): description for bug fixes
|
||||
feat(scope): description for new features
|
||||
improvement(scope): description for enhancements
|
||||
chore(scope): description for maintenance
|
||||
```
|
||||
|
||||
## PR Description Format
|
||||
|
||||
Use this exact template in the user's voice (concise, bullet points):
|
||||
|
||||
```markdown
|
||||
## Summary
|
||||
- bullet point describing what changed
|
||||
- another bullet point if needed
|
||||
|
||||
## Type of Change
|
||||
- [x] Bug fix (or appropriate type)
|
||||
|
||||
## Testing
|
||||
Tested manually (or describe testing)
|
||||
|
||||
## Checklist
|
||||
- [x] Code follows project style guidelines
|
||||
- [x] Self-reviewed my changes
|
||||
- [ ] Tests added/updated and passing
|
||||
- [x] No new warnings introduced
|
||||
- [x] I confirm that I have read and agree to the terms outlined in the [Contributor License Agreement (CLA)](./CONTRIBUTING.md#contributor-license-agreement-cla)
|
||||
```
|
||||
|
||||
## PR Creation Command
|
||||
|
||||
Use this command structure:
|
||||
|
||||
```bash
|
||||
gh pr create --base staging --title "COMMIT_MESSAGE" --body "PR_BODY"
|
||||
```
|
||||
|
||||
## Important Notes
|
||||
|
||||
- Always confirm the commit message and PR description with the user before executing
|
||||
- The PR should be created against `staging` branch
|
||||
- Keep descriptions concise and in active voice
|
||||
- Match the user's previous PR style: direct, no fluff, bullet points
|
||||
- **DO NOT add "Co-Authored-By" lines to commits** - keep commit messages clean
|
||||
|
||||
## User's Voice Characteristics (based on previous PRs)
|
||||
|
||||
- Short, direct bullet points
|
||||
- No unnecessary explanation
|
||||
- "Tested manually" is acceptable for testing section; include lint, boundary validation, and (when migrations changed) `check:migrations` results when run
|
||||
- Checkboxes filled in appropriately
|
||||
- No screenshots section unless UI changes
|
||||
@@ -0,0 +1,93 @@
|
||||
---
|
||||
description: Keep the executable tool registry out of client-reachable module graphs — when to read `@/tools/metadata` instead of `getTool`, how to measure whether an import edge pulls the registry, and how to regenerate the metadata artifacts. Use when touching `apps/sim/tools/registry.ts`, `tools/utils.ts`, `tools/params.ts`, or anything that calls `getTool`.
|
||||
---
|
||||
|
||||
# Tool Registry Boundary Skill
|
||||
|
||||
You keep the 4,300-tool executable registry out of module graphs that don't execute tools.
|
||||
|
||||
## The rule
|
||||
|
||||
> Client-reachable code reads tool **metadata**. Only code that actually executes a tool imports the **registry**.
|
||||
|
||||
`@/tools/registry` is a ~9,000-line barrel importing every tool. Each `ToolConfig` mixes plain data (`params`, `outputs`, `name`) with closures — `request.url`, `request.headers`, `transformResponse`, `directExecution`, `postProcess`. Those closures reach the SDK clients, API helpers and parsers each integration needs, and that is what makes the barrel expensive: reaching it costs ~4,700 additional modules.
|
||||
|
||||
`getTool()` returns the whole `ToolConfig`, so a single `getTool` import anywhere in a client-reachable file drags all of it in.
|
||||
|
||||
## Which module to import
|
||||
|
||||
| you need | import | notes |
|
||||
| --- | --- | --- |
|
||||
| whether a tool id exists | `hasToolId` from `@/tools/tool-ids` | ~110 KB — the cheapest module |
|
||||
| to resolve an unversioned name | `resolveToolId` from `@/tools/tool-ids` | |
|
||||
| every tool id | `getToolIds` from `@/tools/tool-ids` | |
|
||||
| a tool's params | `getToolParams` / `getToolMetadata` from `@/tools/metadata` | ~4 MB |
|
||||
| a tool's declared outputs | `getToolOutputsMetadata` from `@/tools/metadata-outputs` | ~4 MB, separate on purpose |
|
||||
| to **execute** a tool | `getTool` from `@/tools/utils`, or `@/tools/utils.server` | server paths only |
|
||||
|
||||
Three modules, cheapest first. Ids are their own artifact because resolution and existence checks need only the key set; outputs are their own because they are the larger half of the data with a single consumer. `@/tools/metadata` and `@/tools/metadata-outputs` both resolve ids through `@/tools/tool-ids`, which is what keeps them independent of each other — do not "helpfully" re-export one from another, or every caller pays for all three.
|
||||
|
||||
All lookups guard with `Object.hasOwn`. `JSON.parse` yields an object with the normal prototype, so a bare bracket lookup returns inherited members: `getToolMetadata('constructor')` returned a *function* typed as tool metadata before that was fixed.
|
||||
|
||||
## The generated artifacts
|
||||
|
||||
`apps/sim/tools/generated/tool-ids.ts`, `tool-metadata.ts` and `tool-outputs.ts` are produced by `scripts/sync-tool-metadata.ts`:
|
||||
|
||||
```bash
|
||||
bun run tool-metadata:generate # after adding/changing a tool
|
||||
bun run tool-metadata:check # what CI runs; fails if stale
|
||||
```
|
||||
|
||||
Never hand-edit them. If you add a tool or change a tool's `params`/`outputs`, regenerate and commit the result, or CI fails.
|
||||
|
||||
Three non-obvious properties, each of which was measured and is easy to undo by accident:
|
||||
|
||||
- **The data is a JSON string parsed at runtime, not an imported `.json` and not an object literal.** With `resolveJsonModule` (which this repo enables), a `.json` import makes TypeScript infer a literal type for all 4,300+ entries and takes `tsc --noEmit` from **12.6s to 8m07s** — a 38x regression. An ambient `declare module` does *not* short-circuit it, and an object literal costs the same. A single string literal is one cheap token for both the compiler and the bundler, and `JSON.parse` beats evaluating the equivalent literal at runtime. Do not "clean this up" into a `.json` import.
|
||||
- **The generator refuses to emit function values.** If you add a field to `METADATA_FIELDS` that contains a closure, generation fails loudly rather than shipping executable config to the client. `hosting` and `schemaEnrichment` are excluded for exactly this reason (`hosting.enabled`, `pricing`, and `enrichSchema` are functions) — they are server-only.
|
||||
- **Empty param entries are stripped.** The registry contains one (`stt_deepgram_v2`), which crashes callers that read `param.type` while iterating.
|
||||
- **Lookups resolve versions.** `getTool` maps an unversioned name onto the newest version, and 246 tools are versioned. A plain key lookup would silently report them missing — a quiet correctness bug, not a crash. `resolveToolId` reproduces that against the id set and is differentially tested against the original.
|
||||
|
||||
## Testing code that reads tool metadata
|
||||
|
||||
Mock the module the code under test actually reads. `vi.mock('@/tools/utils', () => toolsUtilsMock)` only controls `getTool`; code that reads `params`/`outputs`/`name` goes through `@/tools/metadata`, so mocking `tools/utils` there is a **no-op that still passes** — because the real generated artifacts happen to agree with the mock fixtures. The test looks green while controlling nothing.
|
||||
|
||||
```ts
|
||||
import { blocksMock, toolsMetadataMock, toolsUtilsMock } from '@sim/testing/mocks'
|
||||
|
||||
vi.mock('@/tools/utils', () => toolsUtilsMock) // executable lookup
|
||||
vi.mock('@/tools/metadata', () => toolsMetadataMock) // params / outputs / name
|
||||
```
|
||||
|
||||
Both are backed by the same `mockToolConfigs`, so mocking both gives one consistent tool universe. If you are unsure whether a mock is load-bearing, change a fixture value to a sentinel and confirm the test fails.
|
||||
## The guard
|
||||
|
||||
`bun run check:tool-registry-boundary` (CI: "Tool registry client-boundary audit") walks the module graph from each workspace route and fails if `@/tools/registry` is reachable, printing the exact import chain that reintroduced it.
|
||||
|
||||
If it fails, do not add the entry to an allowlist — there isn't one. Find the symbol the offending file actually needs and move it to a registry-free module, exactly as `mergeToolParameters` and `formatParameterLabel` were.
|
||||
|
||||
Run it with `--verbose` to print per-route module counts, which is also the quickest way to see whether a change moved the graph.
|
||||
|
||||
## How to verify an edge actually got cut
|
||||
|
||||
Do not eyeball imports — the registry is reached through several redundant paths, so cutting one buys nothing while another survives. Walk the graph:
|
||||
|
||||
1. From the entry you care about, follow `import` and `export … from` (skipping `import type`), resolving `@/` against `apps/sim`.
|
||||
2. Check whether `apps/sim/tools/registry.ts` is in the reachable set, and print the parent chain if it is.
|
||||
3. Compare the reachable module count before and after.
|
||||
|
||||
Reference points measured on this repo:
|
||||
|
||||
| entry | modules |
|
||||
| --- | --- |
|
||||
| `tools/registry.ts` reachable | ~4,900 |
|
||||
| `tools/merge-params.ts` (leaf) | 2 |
|
||||
| `providers/utils.ts` after cutting its `params` edge | 22 |
|
||||
| `app/workspace/[workspaceId]/w/page.tsx` (canvas) | 6,592 before, 1,908 after |
|
||||
|
||||
The canvas route reached the registry through **four** redundant edges — `providers/utils` (via `tools/params`), `lib/workflows/blocks/block-outputs`, `lib/workflows/sanitization/validation`, and `serializer/index`. Cutting any one alone moved the module count by ~1. They all had to go before anything improved; measure the route, not the file you edited.
|
||||
|
||||
## When adding a new caller
|
||||
|
||||
Ask what the caller does with the config. If it reads `params`, `outputs`, `name`, `description` or just checks existence, it belongs on `@/tools/metadata` — no exceptions, even on a path you believe is server-only today, because a future client import will silently re-attach the registry to the graph.
|
||||
|
||||
If it genuinely executes — builds a request, transforms a response, runs `directExecution` — use `getTool`, and keep that file off client-reachable paths.
|
||||
@@ -0,0 +1,371 @@
|
||||
---
|
||||
description: Validate an existing knowledge base connector against its service's API docs
|
||||
argument-hint: <service-name> [api-docs-url]
|
||||
---
|
||||
|
||||
# Validate Connector Skill
|
||||
|
||||
You are an expert auditor for Sim knowledge base connectors. Your job is to thoroughly validate that an existing connector is correct, complete, and follows all conventions.
|
||||
|
||||
## Your Task
|
||||
|
||||
When the user asks you to validate a connector:
|
||||
1. Read the service's API documentation (via Context7 or WebFetch)
|
||||
2. Read the connector implementation, OAuth config, and registry entries
|
||||
3. Cross-reference everything against the API docs and Sim conventions
|
||||
4. Report all issues found, grouped by severity (critical, warning, suggestion)
|
||||
5. Fix all issues after reporting them
|
||||
|
||||
## Step 1: Gather All Files
|
||||
|
||||
Read **every** file for the connector — do not skip any:
|
||||
|
||||
```
|
||||
apps/sim/connectors/{service}/meta.ts # ConnectorMeta — client-safe metadata (icon, name, auth, configFields, tagDefinitions)
|
||||
apps/sim/connectors/{service}/{service}.ts # Connector implementation — spreads the meta + runtime functions
|
||||
apps/sim/connectors/{service}/index.ts # Barrel export
|
||||
apps/sim/connectors/registry.server.ts # Server-only full registry entry (CONNECTOR_REGISTRY; full connector)
|
||||
apps/sim/connectors/registry.ts # Client-safe meta registry entry (CONNECTOR_META_REGISTRY)
|
||||
apps/sim/connectors/types.ts # ConnectorMeta / ConnectorConfig interfaces, ExternalDocument, etc.
|
||||
apps/sim/connectors/utils.ts # Shared utilities (computeContentHash, htmlToPlainText, etc.)
|
||||
apps/sim/lib/oauth/oauth.ts # OAUTH_PROVIDERS — single source of truth for scopes
|
||||
apps/sim/lib/oauth/utils.ts # getCanonicalScopesForProvider, getScopesForService, SCOPE_DESCRIPTIONS
|
||||
apps/sim/lib/oauth/types.ts # OAuthService union type
|
||||
apps/sim/components/icons.tsx # Icon definition for the service
|
||||
```
|
||||
|
||||
If the connector uses selectors, also read:
|
||||
```
|
||||
apps/sim/hooks/selectors/registry.ts # Selector key definitions
|
||||
apps/sim/hooks/selectors/types.ts # SelectorKey union type
|
||||
apps/sim/lib/workflows/subblocks/context.ts # SELECTOR_CONTEXT_FIELDS
|
||||
```
|
||||
|
||||
## Step 2: Pull API Documentation
|
||||
|
||||
Fetch the official API docs for the service. This is the **source of truth** for:
|
||||
- Endpoint URLs, HTTP methods, and auth headers
|
||||
- Required vs optional parameters
|
||||
- Parameter types and allowed values
|
||||
- Response shapes and field names
|
||||
- Pagination patterns (cursor, offset, next token)
|
||||
- Rate limits and error formats
|
||||
- OAuth scopes and their meanings
|
||||
|
||||
Use Context7 (resolve-library-id → query-docs) or WebFetch to retrieve documentation. If both fail, note which claims are based on training knowledge vs verified docs.
|
||||
|
||||
### Hard Rule: No Guessed Source Schemas
|
||||
|
||||
If the service docs do not clearly show document list responses, document fetch responses, metadata fields, or pagination shapes, you MUST tell the user instead of guessing.
|
||||
|
||||
- Do NOT infer document fields from unrelated endpoints
|
||||
- Do NOT guess pagination cursors or response wrappers
|
||||
- Do NOT assume metadata keys that are not documented
|
||||
- Do NOT treat probable shapes as validated
|
||||
|
||||
If a schema is unknown, validation must explicitly recommend:
|
||||
1. sample API responses,
|
||||
2. live test credentials, or
|
||||
3. trimming the connector to only documented fields.
|
||||
|
||||
## Step 3: Validate API Endpoints
|
||||
|
||||
For **every** API call in the connector (`listDocuments`, `getDocument`, `validateConfig`, and any helper functions), verify against the API docs:
|
||||
|
||||
### URLs and Methods
|
||||
- [ ] Base URL is correct for the service's API version
|
||||
- [ ] Endpoint paths match the API docs exactly
|
||||
- [ ] HTTP method is correct (GET, POST, PUT, PATCH, DELETE)
|
||||
- [ ] Path parameters are correctly interpolated and URI-encoded where needed
|
||||
- [ ] Query parameters use correct names and formats per the API docs
|
||||
|
||||
### Headers
|
||||
- [ ] Authorization header uses the correct format:
|
||||
- OAuth: `Authorization: Bearer ${accessToken}`
|
||||
- API Key: correct header name per the service's docs
|
||||
- [ ] `Content-Type` is set for POST/PUT/PATCH requests
|
||||
- [ ] Any service-specific headers are present (e.g., `Notion-Version`, `Dropbox-API-Arg`)
|
||||
- [ ] No headers are sent that the API doesn't support or silently ignores
|
||||
|
||||
### Request Bodies
|
||||
- [ ] POST/PUT body fields match API parameter names exactly
|
||||
- [ ] Required fields are always sent
|
||||
- [ ] Optional fields are conditionally included (not sent as `null` or empty unless the API expects that)
|
||||
- [ ] Field value types match API expectations (string vs number vs boolean)
|
||||
|
||||
### Input Sanitization
|
||||
- [ ] User-controlled values interpolated into query strings are properly escaped:
|
||||
- OData `$filter`: single quotes escaped with `''` (e.g., `externalId.replace(/'/g, "''")`)
|
||||
- SOQL: single quotes escaped with `\'`
|
||||
- GraphQL variables: passed as variables, not interpolated into query strings
|
||||
- URL path segments: `encodeURIComponent()` applied
|
||||
- [ ] URL-type config fields (e.g., `siteUrl`, `instanceUrl`) are normalized:
|
||||
- Strip `https://` / `http://` prefix if the API expects bare domains
|
||||
- Strip trailing `/`
|
||||
- Apply `.trim()` before validation
|
||||
|
||||
### Response Parsing
|
||||
- [ ] Response structure is correctly traversed (e.g., `data.results` vs `data.items` vs `data`)
|
||||
- [ ] Field names extracted match what the API actually returns
|
||||
- [ ] Nullable fields are handled with `?? null` or `|| undefined`
|
||||
- [ ] Error responses are checked before accessing data fields
|
||||
- [ ] Every extracted field and pagination value is backed by official docs or live-verified sample payloads
|
||||
|
||||
## Step 4: Validate OAuth Scopes (if OAuth connector)
|
||||
|
||||
Scopes must be correctly declared and sufficient for all API calls the connector makes.
|
||||
|
||||
### Connector requiredScopes
|
||||
- [ ] `requiredScopes` in the connector's `auth` config lists all scopes needed by the connector
|
||||
- [ ] Each scope in `requiredScopes` is a real, valid scope recognized by the service's API
|
||||
- [ ] No invalid, deprecated, or made-up scopes are listed
|
||||
- [ ] No unnecessary excess scopes beyond what the connector actually needs
|
||||
|
||||
### Scope Subset Validation (CRITICAL)
|
||||
- [ ] Every scope in `requiredScopes` exists in the OAuth provider's `scopes` array in `lib/oauth/oauth.ts`
|
||||
- [ ] Find the provider in `OAUTH_PROVIDERS[providerGroup].services[serviceId].scopes`
|
||||
- [ ] Verify: `requiredScopes` ⊆ `OAUTH_PROVIDERS scopes` (every required scope is present in the provider config)
|
||||
- [ ] If a required scope is NOT in the provider config, flag as **critical** — the connector will fail at runtime
|
||||
|
||||
### Scope Sufficiency
|
||||
For each API endpoint the connector calls:
|
||||
- [ ] Identify which scopes are required per the API docs
|
||||
- [ ] Verify those scopes are included in the connector's `requiredScopes`
|
||||
- [ ] If the connector calls endpoints requiring scopes not in `requiredScopes`, flag as **warning**
|
||||
|
||||
### Token Refresh Config
|
||||
- [ ] Check the `getOAuthTokenRefreshConfig` function in `lib/oauth/oauth.ts` for this provider
|
||||
- [ ] `useBasicAuth` matches the service's token exchange requirements
|
||||
- [ ] `supportsRefreshTokenRotation` matches whether the service issues rotating refresh tokens
|
||||
- [ ] Token endpoint URL is correct
|
||||
|
||||
## Step 5: Validate Pagination
|
||||
|
||||
### listDocuments Pagination
|
||||
- [ ] Cursor/pagination parameter name matches the API docs
|
||||
- [ ] Response pagination field is correctly extracted (e.g., `next_cursor`, `nextPageToken`, `@odata.nextLink`, `offset`)
|
||||
- [ ] `hasMore` is correctly determined from the response
|
||||
- [ ] `nextCursor` is correctly passed back for the next page
|
||||
- [ ] `maxItems` / `maxRecords` cap is correctly applied across pages using `syncContext.totalDocsFetched`
|
||||
- [ ] Page size is within the API's allowed range (not exceeding max page size)
|
||||
- [ ] Last page precision: when a `maxItems` cap exists, the final page request uses `Math.min(PAGE_SIZE, remaining)` to avoid fetching more records than needed
|
||||
- [ ] No off-by-one errors in pagination tracking
|
||||
- [ ] The connector does NOT hit known API pagination limits silently (e.g., HubSpot search 10k cap)
|
||||
|
||||
### Deletion-Reconciliation Safety (`listingCapped`) — CRITICAL
|
||||
The sync engine hard-deletes any stored document absent from a full listing. Audit every path where `listDocuments` can return less than the full source set:
|
||||
- [ ] `syncContext.listingCapped = true` is set when a `maxItems`-style cap truncates the listing while more documents exist
|
||||
- [ ] `listingCapped` is set when a transient per-item error drops a still-existing document from the listing
|
||||
- [ ] `listingCapped` is NOT set when the source is genuinely exhausted (deleted documents must reconcile) or for intentional scope filters (date cutoffs)
|
||||
This is the most common connector bug class — verify it explicitly against `sync-engine.ts`'s reconciliation gate.
|
||||
|
||||
### Pagination State Across Pages
|
||||
- [ ] `syncContext` is used to cache state across pages (user names, field maps, instance URLs, portal IDs, etc.)
|
||||
- [ ] Cached state in `syncContext` is correctly initialized on first page and reused on subsequent pages
|
||||
|
||||
## Step 6: Validate Data Transformation
|
||||
|
||||
### Content Deferral (CRITICAL)
|
||||
Connectors that require per-document API calls to fetch content (file download, export, blocks fetch) MUST use `contentDeferred: true`. This is the standard pattern for reliability — without it, content downloads during listing can exhaust the sync task's time budget before any documents are saved.
|
||||
|
||||
- [ ] If the connector downloads content per-doc during `listDocuments`, it MUST use `contentDeferred: true` instead
|
||||
- [ ] `listDocuments` returns lightweight stubs with `content: ''` and `contentDeferred: true`
|
||||
- [ ] `getDocument` fetches actual content and returns the full document with `contentDeferred: false`
|
||||
- [ ] A shared stub function (e.g., `fileToStub`) is used by both `listDocuments` and `getDocument` to guarantee `contentHash` consistency
|
||||
- [ ] `contentHash` is metadata-based (e.g., `service:{id}:{modifiedTime}`), NOT content-based — it must be derivable from list metadata alone
|
||||
- [ ] The `contentHash` is identical whether produced by `listDocuments` or `getDocument`
|
||||
|
||||
Connectors where the list API already returns content inline (e.g., Slack messages, Reddit posts) do NOT need `contentDeferred`.
|
||||
|
||||
### ExternalDocument Construction
|
||||
- [ ] `externalId` is a stable, unique identifier from the source API
|
||||
- [ ] `title` is extracted from the correct field and has a sensible fallback (e.g., `'Untitled'`)
|
||||
- [ ] `content` is plain text — HTML content is stripped using `htmlToPlainText` from `@/connectors/utils`
|
||||
- [ ] `mimeType` is `'text/plain'`
|
||||
- [ ] `contentHash` uses a metadata-based format (e.g., `service:{id}:{modifiedTime}`) for connectors with `contentDeferred: true`, or `computeContentHash` from `@/connectors/utils` for inline-content connectors
|
||||
- [ ] `sourceUrl` is a valid, complete URL back to the original resource (not relative)
|
||||
- [ ] `metadata` contains all fields referenced by `mapTags` and `tagDefinitions`
|
||||
|
||||
### Content Extraction
|
||||
- [ ] Rich text / HTML fields are converted to plain text before indexing
|
||||
- [ ] Important content is not silently dropped (e.g., nested blocks, table cells, code blocks)
|
||||
- [ ] Content is not silently truncated without logging a warning
|
||||
- [ ] Empty/blank documents are properly filtered out
|
||||
- [ ] Size checks use `Buffer.byteLength(text, 'utf8')` not `text.length` when comparing against byte-based limits (e.g., `MAX_FILE_SIZE` in bytes)
|
||||
|
||||
## Step 7: Validate Tag Definitions and mapTags
|
||||
|
||||
### tagDefinitions
|
||||
- [ ] Each `tagDefinition` has an `id`, `displayName`, and `fieldType`
|
||||
- [ ] `fieldType` matches the actual data type: `'text'` for strings, `'number'` for numbers, `'date'` for dates, `'boolean'` for booleans
|
||||
- [ ] Every `id` in `tagDefinitions` is returned by `mapTags`
|
||||
- [ ] No `tagDefinition` references a field that `mapTags` never produces
|
||||
|
||||
### mapTags
|
||||
- [ ] Return keys match `tagDefinition` `id` values exactly
|
||||
- [ ] Date values are properly parsed using `parseTagDate` from `@/connectors/utils`
|
||||
- [ ] Array values are properly joined using `joinTagArray` from `@/connectors/utils`
|
||||
- [ ] Number values are validated (not `NaN`)
|
||||
- [ ] Metadata field names accessed in `mapTags` match what `listDocuments`/`getDocument` store in `metadata`
|
||||
|
||||
## Step 8: Validate Config Fields and Validation
|
||||
|
||||
### configFields
|
||||
- [ ] Every field has `id`, `title`, `type`
|
||||
- [ ] `required` is set explicitly (not omitted)
|
||||
- [ ] Dropdown fields have `options` with `label` and `id` for each option
|
||||
- [ ] Selector fields follow the canonical pair pattern:
|
||||
- A `type: 'selector'` field with `selectorKey`, `canonicalParamId`, `mode: 'basic'`
|
||||
- A `type: 'short-input'` field with the same `canonicalParamId`, `mode: 'advanced'`
|
||||
- `required` is identical on both fields in the pair
|
||||
- [ ] `selectorKey` values exist in the selector registry
|
||||
- [ ] `dependsOn` references selector field `id` values, not `canonicalParamId`
|
||||
|
||||
### validateConfig
|
||||
- [ ] Validates all required fields are present before making API calls
|
||||
- [ ] Validates optional numeric fields (checks `Number.isNaN`, positive values)
|
||||
- [ ] Makes a lightweight API call to verify access (e.g., fetch 1 record, get profile)
|
||||
- [ ] Uses `VALIDATE_RETRY_OPTIONS` for retry budget
|
||||
- [ ] Returns `{ valid: true }` on success
|
||||
- [ ] Returns `{ valid: false, error: 'descriptive message' }` on failure
|
||||
- [ ] Catches exceptions and returns user-friendly error messages
|
||||
- [ ] Does NOT make expensive calls (full data listing, large queries)
|
||||
|
||||
## Step 9: Validate getDocument
|
||||
|
||||
- [ ] Fetches a single document by `externalId`
|
||||
- [ ] Returns `null` for 404 / not found (does not throw)
|
||||
- [ ] Returns the same `ExternalDocument` shape as `listDocuments`
|
||||
- [ ] If `listDocuments` uses `contentDeferred: true`, `getDocument` MUST fetch actual content and return `contentDeferred: false`
|
||||
- [ ] If `listDocuments` uses `contentDeferred: true`, `getDocument` MUST use the same stub function to ensure `contentHash` is identical
|
||||
- [ ] Handles all content types that `listDocuments` can produce (e.g., if `listDocuments` returns both pages and blogposts, `getDocument` must handle both — not hardcode one endpoint)
|
||||
- [ ] Forwards `syncContext` if it needs cached state (user names, field maps, etc.)
|
||||
- [ ] Error handling is graceful (catches, logs, returns null or throws with context)
|
||||
- [ ] Does not redundantly re-fetch data already included in the initial API response (e.g., if comments come back with the post, don't fetch them again separately)
|
||||
|
||||
## Step 10: Validate General Quality
|
||||
|
||||
### fetchWithRetry Usage
|
||||
- [ ] All external API calls use `fetchWithRetry` from `@/lib/knowledge/documents/utils`
|
||||
- [ ] No raw `fetch()` calls to external APIs
|
||||
- [ ] `VALIDATE_RETRY_OPTIONS` used in `validateConfig`
|
||||
- [ ] If `validateConfig` calls a shared helper (e.g., `linearGraphQL`, `resolveId`), that helper must accept and forward `retryOptions` to `fetchWithRetry`
|
||||
- [ ] Default retry options used in `listDocuments`/`getDocument`
|
||||
|
||||
### API Efficiency
|
||||
- [ ] APIs that support field selection (e.g., `$select`, `sysparm_fields`, `fields`) should request only the fields the connector needs — in both `listDocuments` AND `getDocument`
|
||||
- [ ] No redundant API calls: if a helper already fetches data (e.g., site metadata), callers should reuse the result instead of making a second call for the same information
|
||||
- [ ] Sequential per-item API calls (fetching details for each document in a loop) should be batched with `Promise.all` and a concurrency limit of 3-5
|
||||
|
||||
### Error Handling
|
||||
- [ ] Individual document failures are caught and logged without aborting the sync
|
||||
- [ ] API error responses include status codes in error messages
|
||||
- [ ] No unhandled promise rejections in concurrent operations
|
||||
|
||||
### Concurrency
|
||||
- [ ] Concurrent API calls use reasonable batch sizes (3-5 is typical)
|
||||
- [ ] No unbounded `Promise.all` over large arrays
|
||||
|
||||
### Logging
|
||||
- [ ] Uses `createLogger` from `@sim/logger` (not `console.log`)
|
||||
- [ ] Logs sync progress at `info` level
|
||||
- [ ] Logs errors at `warn` or `error` level with context
|
||||
|
||||
### Meta / Runtime Split
|
||||
- [ ] `connectors/{service}/meta.ts` exports `{service}ConnectorMeta: ConnectorMeta` (id, name, description, version, icon, auth, configFields, and any `tagDefinitions` / `supportsIncrementalSync`)
|
||||
- [ ] `meta.ts` imports ONLY the icon from `@/components/icons`, `ConnectorMeta` (type-only), and pure-data constants — NO server/runtime imports (`@/lib/knowledge/...`, `input-validation.server`, `fetchWithRetry`, etc.); any such import in `meta.ts` is **critical** (breaks the client bundle)
|
||||
- [ ] `connectors/{service}/{service}.ts` spreads `...{service}ConnectorMeta` as the first property and adds the runtime functions (`listDocuments`, `getDocument`, `validateConfig`, `mapTags?`)
|
||||
- [ ] Metadata fields (id, name, auth, configFields, etc.) live ONLY in `meta.ts`, not duplicated in `{service}.ts`
|
||||
|
||||
### Registry
|
||||
- [ ] Connector is exported from `connectors/{service}/index.ts`
|
||||
- [ ] Full connector is registered in `connectors/registry.server.ts` (server-only registry, `CONNECTOR_REGISTRY`)
|
||||
- [ ] Meta is registered in `connectors/registry.ts` (client-safe registry, `CONNECTOR_META_REGISTRY`), importing `@/connectors/{service}/meta`
|
||||
- [ ] Both registries use the same key and it matches the connector's `id` field
|
||||
- [ ] Both registries keep the same alphabetical-by-id ordering
|
||||
|
||||
## Step 11: Report and Fix
|
||||
|
||||
### Report Format
|
||||
|
||||
Group findings by severity:
|
||||
|
||||
**Critical** (will cause runtime errors, data loss, or auth failures):
|
||||
- Wrong API endpoint URL or HTTP method
|
||||
- Invalid or missing OAuth scopes (not in provider config)
|
||||
- Incorrect response field mapping (accessing wrong path)
|
||||
- SOQL/query fields that don't exist on the target object
|
||||
- Pagination that silently hits undocumented API limits
|
||||
- Missing `syncContext.listingCapped = true` when a cap or transient error truncates the listing — the sync engine hard-deletes the documents absent from the partial listing
|
||||
- Missing error handling that would crash the sync
|
||||
- `requiredScopes` not a subset of OAuth provider scopes
|
||||
- Query/filter injection: user-controlled values interpolated into OData `$filter`, SOQL, or query strings without escaping
|
||||
- Per-document content download in `listDocuments` without `contentDeferred: true` — causes sync timeouts for large document sets
|
||||
- `contentHash` mismatch between `listDocuments` stub and `getDocument` return — causes unnecessary re-processing every sync
|
||||
- Server/runtime import in `meta.ts` (e.g. `@/lib/knowledge/...`, `input-validation.server`, `fetchWithRetry`) — pulls server-only code into the client bundle and breaks the build
|
||||
- Connector missing from `connectors/registry.ts` (the client-safe meta registry) — or its entry there imports the runtime module instead of `meta.ts` — the knowledge UI can't render it
|
||||
|
||||
**Warning** (incorrect behavior, data quality issues, or convention violations):
|
||||
- HTML content not stripped via `htmlToPlainText`
|
||||
- `getDocument` not forwarding `syncContext`
|
||||
- `getDocument` hardcoded to one content type when `listDocuments` returns multiple (e.g., only pages but not blogposts)
|
||||
- Missing `tagDefinition` for metadata fields returned by `mapTags`
|
||||
- Incorrect `useBasicAuth` or `supportsRefreshTokenRotation` in token refresh config
|
||||
- Invalid scope names that the API doesn't recognize (even if silently ignored)
|
||||
- Private resources excluded from name-based lookup despite scopes being available
|
||||
- Silent data truncation without logging
|
||||
- Size checks using `text.length` (character count) instead of `Buffer.byteLength` (byte count) for byte-based limits
|
||||
- URL-type config fields not normalized (protocol prefix, trailing slashes cause API failures)
|
||||
- `VALIDATE_RETRY_OPTIONS` not threaded through helper functions called by `validateConfig`
|
||||
|
||||
**Suggestion** (minor improvements):
|
||||
- Missing incremental sync support despite API supporting it
|
||||
- Overly broad scopes that could be narrowed (not wrong, but could be tighter)
|
||||
- Source URL format could be more specific
|
||||
- Missing `orderBy` for deterministic pagination
|
||||
- Redundant API calls that could be cached in `syncContext`
|
||||
- Sequential per-item API calls that could be batched with `Promise.all` (concurrency 3-5)
|
||||
- API supports field selection but connector fetches all fields (e.g., missing `$select`, `sysparm_fields`, `fields`)
|
||||
- `getDocument` re-fetches data already included in the initial API response (e.g., comments returned with post)
|
||||
- Last page of pagination requests full `PAGE_SIZE` when fewer records remain (`Math.min(PAGE_SIZE, remaining)`)
|
||||
|
||||
### Fix All Issues
|
||||
|
||||
After reporting, fix every **critical** and **warning** issue. Apply **suggestions** where they don't add unnecessary complexity.
|
||||
|
||||
### Validation Output
|
||||
|
||||
After fixing, confirm:
|
||||
1. `bun run lint` passes
|
||||
2. TypeScript compiles clean
|
||||
3. Re-read all modified files to verify fixes are correct
|
||||
4. Any remaining unknown source schemas were explicitly reported to the user instead of guessed
|
||||
|
||||
## Checklist Summary
|
||||
|
||||
- [ ] Read connector meta.ts, implementation, types, utils, both registries, and OAuth config
|
||||
- [ ] Pulled and read official API documentation for the service
|
||||
- [ ] Validated every API endpoint URL, method, headers, and body against API docs
|
||||
- [ ] Validated input sanitization: no query/filter injection, URL fields normalized
|
||||
- [ ] Validated OAuth scopes: `requiredScopes` ⊆ OAuth provider `scopes` in `oauth.ts`
|
||||
- [ ] Validated each scope is real and recognized by the service's API
|
||||
- [ ] Validated scopes are sufficient for all API endpoints the connector calls
|
||||
- [ ] Validated token refresh config (`useBasicAuth`, `supportsRefreshTokenRotation`)
|
||||
- [ ] Validated pagination: cursor names, page sizes, hasMore logic, no silent caps
|
||||
- [ ] Validated deletion-reconciliation safety: `syncContext.listingCapped` set on capped/error-truncated listings, not on genuine exhaustion or intentional scope filters
|
||||
- [ ] Validated content deferral: `contentDeferred: true` used when per-doc content fetch required, metadata-based `contentHash` consistent between stub and `getDocument`
|
||||
- [ ] Validated data transformation: plain text extraction, HTML stripping, content hashing
|
||||
- [ ] Validated tag definitions match mapTags output, correct fieldTypes
|
||||
- [ ] Validated config fields: canonical pairs, selector keys, required flags
|
||||
- [ ] Validated validateConfig: lightweight check, error messages, retry options
|
||||
- [ ] Validated getDocument: null on 404, all content types handled, no redundant re-fetches, syncContext forwarding
|
||||
- [ ] Validated fetchWithRetry used for all external calls (no raw fetch), VALIDATE_RETRY_OPTIONS threaded through helpers
|
||||
- [ ] Validated API efficiency: field selection used, no redundant calls, sequential fetches batched
|
||||
- [ ] Validated error handling: graceful failures, no unhandled rejections
|
||||
- [ ] Validated logging: createLogger, no console.log
|
||||
- [ ] Validated meta/runtime split: `meta.ts` holds metadata with no server/runtime imports, `{service}.ts` spreads the meta + adds runtime functions
|
||||
- [ ] Validated registry: exported from index.ts, full connector in `registry.server.ts`, meta in `registry.ts`, matching keys and alphabetical-by-id ordering in both
|
||||
- [ ] Reported all issues grouped by severity
|
||||
- [ ] Fixed all critical and warning issues
|
||||
- [ ] Ran `bun run lint` after fixes
|
||||
- [ ] Verified TypeScript compiles clean
|
||||
@@ -0,0 +1,444 @@
|
||||
---
|
||||
description: Validate an existing Sim integration (tools, block, registry, and resolved-secret/model-input boundaries) against the service's API docs and Sim execution conventions
|
||||
argument-hint: <service-name> [api-docs-url]
|
||||
---
|
||||
|
||||
# Validate Integration Skill
|
||||
|
||||
You are an expert auditor for Sim integrations. Your job is to thoroughly validate that an existing integration is correct, complete, and follows all conventions.
|
||||
|
||||
## Your Task
|
||||
|
||||
When the user asks you to validate an integration:
|
||||
1. Read the service's API documentation (via WebFetch or Context7)
|
||||
2. Read every tool, the block, and registry entries
|
||||
3. Cross-reference everything against the API docs and Sim conventions
|
||||
4. Report all issues found, grouped by severity (critical, warning, suggestion)
|
||||
5. Fix all issues after reporting them
|
||||
|
||||
## Step 1: Gather All Files
|
||||
|
||||
Read **every** file for the integration — do not skip any:
|
||||
|
||||
```
|
||||
apps/sim/tools/{service}/ # All tool files, types.ts, index.ts
|
||||
apps/sim/blocks/blocks/{service}.ts # Block definition
|
||||
apps/sim/tools/registry.ts # Tool registry entries for this service
|
||||
apps/sim/blocks/registry-maps.ts # Block + meta registry entry (BLOCK_REGISTRY / BLOCK_META_REGISTRY)
|
||||
apps/sim/components/icons.tsx # Icon definition
|
||||
apps/sim/lib/auth/auth.ts # OAuth config — should use getCanonicalScopesForProvider()
|
||||
apps/sim/lib/oauth/oauth.ts # OAuth provider config — single source of truth for scopes
|
||||
apps/sim/lib/oauth/utils.ts # Scope utilities, SCOPE_DESCRIPTIONS for modal UI
|
||||
apps/sim/lib/core/config/env-capabilities.ts # OAuth client runtime capability source of truth
|
||||
apps/sim/lib/core/config/env.ts # Runtime env schema for capability fields
|
||||
scripts/setup/capability-config.ts # Exhaustive CLI input-mode mapping for OAuth fields
|
||||
apps/sim/lib/integrations/integrations.json # Generated client-safe integration catalog
|
||||
apps/sim/lib/integrations/service-account-metadata.ts # Lightweight service-account projection
|
||||
```
|
||||
|
||||
## Step 2: Pull API Documentation
|
||||
|
||||
Fetch the official API docs for the service. This is the **source of truth** for:
|
||||
- Endpoint URLs, HTTP methods, and auth headers
|
||||
- Required vs optional parameters
|
||||
- Parameter types and allowed values
|
||||
- Response shapes and field names
|
||||
- Pagination patterns (which param name, which response field)
|
||||
- Rate limits and error formats
|
||||
|
||||
### Hard Rule: No Guessed Response Schemas
|
||||
|
||||
If the official docs do not clearly show the response JSON shape for an endpoint, you MUST tell the user instead of guessing.
|
||||
|
||||
- Do NOT assume field names from nearby endpoints
|
||||
- Do NOT infer nested JSON paths without evidence
|
||||
- Do NOT treat "likely" fields as confirmed outputs
|
||||
- Do NOT accept implementation guesses as valid just because they are defensive
|
||||
|
||||
If a response schema is unknown, the validation must explicitly call that out and require:
|
||||
1. sample responses from the user,
|
||||
2. live test credentials for verification, or
|
||||
3. trimming the tool/block down to only documented fields.
|
||||
|
||||
## Step 3: Validate Tools
|
||||
|
||||
For **every** tool file, check:
|
||||
|
||||
### Tool ID and Naming
|
||||
- [ ] Tool ID uses `snake_case`: `{service}_{action}` (e.g., `x_create_tweet`, `slack_send_message`)
|
||||
- [ ] Tool `name` is human-readable (e.g., `'X Create Tweet'`)
|
||||
- [ ] Tool `description` is a concise one-liner describing what it does
|
||||
- [ ] Tool `version` is set (`'1.0.0'` or `'2.0.0'` for V2)
|
||||
|
||||
### Params
|
||||
- [ ] All required API params are marked `required: true`
|
||||
- [ ] All optional API params are marked `required: false`
|
||||
- [ ] Every param has explicit `required: true` or `required: false` — never omitted
|
||||
- [ ] Param types match the API (`'string'`, `'number'`, `'boolean'`, `'json'`)
|
||||
- [ ] Visibility is correct:
|
||||
- `'hidden'` — ONLY for OAuth access tokens and system-injected params
|
||||
- `'user-only'` — for API keys, credentials, and account-specific IDs the user must provide
|
||||
- `'user-or-llm'` — for everything else (search queries, content, filters, IDs that could come from other blocks)
|
||||
- [ ] Every param has a `description` that explains what it does
|
||||
|
||||
### Request
|
||||
- [ ] URL matches the API endpoint exactly (correct base URL, path segments, path params)
|
||||
- [ ] HTTP method matches the API spec (GET, POST, PUT, PATCH, DELETE)
|
||||
- [ ] Headers include correct auth pattern:
|
||||
- OAuth: `Authorization: Bearer ${params.accessToken}`
|
||||
- API Key: correct header name and format per the service's docs
|
||||
- [ ] `Content-Type` header is set for POST/PUT/PATCH requests
|
||||
- [ ] Body sends all required fields and only includes optional fields when provided
|
||||
- [ ] For GET requests with query params: URL is constructed correctly with query string
|
||||
- [ ] ID fields in URL paths are `.trim()`-ed to prevent copy-paste whitespace errors
|
||||
- [ ] Path params use template literals correctly: `` `https://api.service.com/v1/${params.id.trim()}` ``
|
||||
|
||||
### Response / transformResponse
|
||||
- [ ] Correctly parses the API response (`await response.json()`)
|
||||
- [ ] Extracts the right fields from the response structure (e.g., `data.data` vs `data` vs `data.results`)
|
||||
- [ ] All nullable fields use `?? null`
|
||||
- [ ] All optional arrays use `?? []`
|
||||
- [ ] Error cases are handled: checks for missing/empty data and returns meaningful error
|
||||
- [ ] Does NOT do raw JSON dumps — extracts meaningful, individual fields
|
||||
- [ ] Every extracted field is backed by official docs or live-verified sample payloads
|
||||
|
||||
### Outputs
|
||||
- [ ] All output fields match what the API actually returns
|
||||
- [ ] No fields are missing that the API provides and users would commonly need
|
||||
- [ ] No phantom fields defined that the API doesn't return
|
||||
- [ ] `optional: true` is set on fields that may not exist in all responses
|
||||
- [ ] When using `type: 'json'` and the shape is known, `properties` defines the inner fields (tool outputs only — block outputs do not support `properties`)
|
||||
- [ ] When using `type: 'array'`, `items` defines the item structure with `properties` (tool outputs only)
|
||||
- [ ] Field descriptions are accurate and helpful
|
||||
|
||||
### Types (types.ts)
|
||||
- [ ] Has param interfaces for every tool (e.g., `XCreateTweetParams`)
|
||||
- [ ] Has response interfaces for every tool (extending `ToolResponse`)
|
||||
- [ ] Optional params use `?` in the interface (e.g., `replyTo?: string`)
|
||||
- [ ] Field names in types match actual API field names
|
||||
- [ ] Shared response types are properly reused (e.g., `XTweetResponse` shared across tweet tools)
|
||||
|
||||
### Barrel Export (index.ts)
|
||||
- [ ] Every tool is exported
|
||||
- [ ] All types are re-exported (`export * from './types'`)
|
||||
- [ ] No orphaned exports (tools that don't exist)
|
||||
|
||||
### Tool Registry (tools/registry.ts)
|
||||
- [ ] Every tool is imported and registered
|
||||
- [ ] Registry keys use snake_case and match tool IDs exactly
|
||||
- [ ] Entries are in alphabetical order within the file
|
||||
|
||||
### Resolved-Secret Provenance and Model Input
|
||||
|
||||
For every request field, determine whether it is ordinary API input, model-visible text/structured
|
||||
content, opaque model input, or a value persisted into Sim-owned durable storage.
|
||||
|
||||
Treat model-input provenance as opt-in. Require official documentation or an unambiguous local
|
||||
execution path proving that the exact field reaches an AI model. If the evidence is ambiguous,
|
||||
leave the integration unchanged; do not infer a model boundary merely from natural-language,
|
||||
search, extraction, or "AI-powered" marketing terminology.
|
||||
|
||||
- [ ] AI-consumed text/structured fields use `request.modelInput` with `mode: 'project'` and a
|
||||
minimal exact selector; nested/JSON-string adapters preserve shape through `applyProjected`
|
||||
- [ ] Ordinary external URLs, domains, resource IDs, and control fields retain normal request
|
||||
semantics unless the exact field is proven model-visible; an AI-backed provider or later model
|
||||
processing of the referenced resource is not sufficient evidence
|
||||
- [ ] Serialized content proven to be sent directly to an external model is selected by
|
||||
`request.modelInput`, projected before the existing formatter parses it, and has deterministic
|
||||
formatter behavior when a whole-value placeholder is invalid for the serialized grammar
|
||||
- [ ] Actual inline/raw AI-consumed bytes owned by an authenticated internal route use
|
||||
`privateProvenance` (or `mode: 'private-provenance'`), and the route validates
|
||||
`validateOpaqueModelInputProvenance` before model egress; storage keys, paths, signed URLs,
|
||||
and ordinary remote URLs are not treated as byte provenance, while tracked stored bytes are
|
||||
authorized independently at the owning model-egress boundary
|
||||
- [ ] Persisted workspace-file contents are checked with the shared provenance guard only when
|
||||
their bytes or decoded content cross into a model/tool-result boundary; ordinary file APIs
|
||||
remain unchanged. Unsupported secret-bearing file paths are rejected at `file_write`
|
||||
- [ ] Sim-owned durable writes and internal execution handoffs that can enter workflows/models use
|
||||
field-scoped `request.secretProvenance`; authenticated receivers validate the exact selection
|
||||
and scope, strip private metadata, and persist, import, or propagate it at the owning boundary
|
||||
- [ ] Private provenance is never attached to external URLs or `directExecution`; proven
|
||||
model-visible external fields use projection, while other external inputs remain unchanged
|
||||
- [ ] No tool performs raw secret plaintext/source substitution or serializes plaintext provenance
|
||||
- [ ] No `transformResponse` or tool-local helper blanket-sanitizes ordinary third-party results;
|
||||
only execution-scoped, activated Sim provenance is projected at shared model/log boundaries
|
||||
- [ ] Private headers/envelopes are produced and stripped by the shared tool executor, never
|
||||
hand-rolled or returned as functional output
|
||||
- [ ] Every added provenance hook has a concrete Sim `{{...}}` resolution path and a later
|
||||
persistence/model/log crossing; there is no generic handling for arbitrary filenames,
|
||||
metadata, provider results, or API payloads
|
||||
- [ ] Diagnostic projection is applied only to values carrying execution-scoped provenance;
|
||||
ordinary provider responses, filenames, URLs, and errors are unchanged
|
||||
- [ ] Tests cover named `{{NAME}}` projection, unproven identical public text, nested and serialized
|
||||
shape handling, unchanged ordinary external inputs, malformed/incomplete metadata, headerless
|
||||
legacy requests, metadata stripping, and durable legacy/stale/scope cases when applicable
|
||||
|
||||
Treat a missing or bypassed model, durable, or internal-execution provenance boundary as
|
||||
**critical**. Do not fix it with a tool-specific string replacer or by sanitizing every provider
|
||||
result; repair the shared request, authenticated internal-route, persistence, or re-entry boundary
|
||||
that owns the data.
|
||||
|
||||
## Step 4: Validate Block
|
||||
|
||||
### Block ↔ Tool Alignment (CRITICAL)
|
||||
|
||||
This is the most important validation — the block must be perfectly aligned with every tool it references.
|
||||
|
||||
For **each tool** in `tools.access`:
|
||||
- [ ] The operation dropdown has an option whose ID matches the tool ID (or the `tools.config.tool` function correctly maps to it)
|
||||
- [ ] Every **required** tool param (except `accessToken`) has a corresponding subBlock input that is:
|
||||
- Shown when that operation is selected (correct `condition`)
|
||||
- Marked as `required: true` (or conditionally required)
|
||||
- [ ] Every **optional** tool param has a corresponding subBlock input (or is intentionally omitted if truly never needed)
|
||||
- [ ] SubBlock `id` values are unique across the entire block — no duplicates even across different conditions
|
||||
- [ ] The `tools.config.tool` function returns the correct tool ID for every possible operation value
|
||||
- [ ] The `tools.config.params` function correctly maps subBlock IDs to tool param names when they differ
|
||||
|
||||
### SubBlocks
|
||||
- [ ] Operation dropdown lists ALL tool operations available in `tools.access`
|
||||
- [ ] Dropdown option labels are human-readable and descriptive
|
||||
- [ ] Conditions use correct syntax:
|
||||
- Single value: `{ field: 'operation', value: 'x_create_tweet' }`
|
||||
- Multiple values (OR): `{ field: 'operation', value: ['x_create_tweet', 'x_delete_tweet'] }`
|
||||
- Negation: `{ field: 'operation', value: 'delete', not: true }`
|
||||
- Compound: `{ field: 'op', value: 'send', and: { field: 'type', value: 'dm' } }`
|
||||
- [ ] Condition arrays include ALL operations that use that field — none missing
|
||||
- [ ] `dependsOn` is set for fields that need other values (selectors depending on credential, cascading dropdowns)
|
||||
- [ ] SubBlock types match tool param types:
|
||||
- Enum/fixed options → `dropdown`
|
||||
- Free text → `short-input`
|
||||
- Long text/content → `long-input`
|
||||
- True/false → `dropdown` with Yes/No options (not `switch` unless purely UI toggle)
|
||||
- Credentials → `oauth-input` with correct `serviceId`
|
||||
- [ ] Dropdown `value: () => 'default'` is set for dropdowns with a sensible default
|
||||
|
||||
### Advanced Mode
|
||||
- [ ] Optional, rarely-used fields are set to `mode: 'advanced'`:
|
||||
- Pagination tokens / next tokens
|
||||
- Time range filters (start/end time)
|
||||
- Sort order / direction options
|
||||
- Max results / per page limits
|
||||
- Reply settings / threading options
|
||||
- Rarely used IDs (reply-to, quote-tweet, etc.)
|
||||
- Exclude filters
|
||||
- [ ] **Required** fields are NEVER set to `mode: 'advanced'`
|
||||
- [ ] Fields that users fill in most of the time are NOT set to `mode: 'advanced'`
|
||||
|
||||
### WandConfig
|
||||
- [ ] Timestamp fields have `wandConfig` with `generationType: 'timestamp'`
|
||||
- [ ] Comma-separated list fields have `wandConfig` with a descriptive prompt
|
||||
- [ ] Complex filter/query fields have `wandConfig` with format examples in the prompt
|
||||
- [ ] All `wandConfig` prompts end with "Return ONLY the [format] - no explanations, no extra text."
|
||||
- [ ] `wandConfig.placeholder` describes what to type in natural language
|
||||
|
||||
### Tools Config
|
||||
- [ ] `tools.access` lists **every** tool ID the block can use — none missing
|
||||
- [ ] `tools.config.tool` returns the correct tool ID for each operation
|
||||
- [ ] Type coercions are in `tools.config.params` (runs at execution time), NOT in `tools.config.tool` (runs at serialization time before variable resolution)
|
||||
- [ ] `tools.config.params` handles:
|
||||
- `Number()` conversion for numeric params that come as strings from inputs
|
||||
- `Boolean` / string-to-boolean conversion for toggle params
|
||||
- Empty string → `undefined` conversion for optional dropdown values
|
||||
- Any subBlock ID → tool param name remapping
|
||||
- [ ] No `Number()`, `JSON.parse()`, or other coercions in `tools.config.tool` — these would destroy dynamic references like `<Block.output>`
|
||||
|
||||
### Block Outputs
|
||||
- [ ] Outputs cover the key fields returned by ALL tools (not just one operation)
|
||||
- [ ] Output types are correct (`'string'`, `'number'`, `'boolean'`, `'json'`)
|
||||
- [ ] `type: 'json'` outputs describe inner fields in the description string: `'User profile (id, name, username, bio)'` or `'[{address, status, type}]'` for arrays
|
||||
- [ ] **Do NOT add a `properties: {...}` field on block outputs.** Block-level `OutputFieldDefinition` (from `@sim/workflow-types/blocks`) only accepts `{ type, description?, condition?, hiddenFromDisplay? }`. Nested `properties` is a tool-level construct (`OutputProperty`) — adding it to a block output will fail TypeScript at build time
|
||||
- [ ] No opaque `type: 'json'` with vague descriptions like `'Response data'`
|
||||
- [ ] Outputs that only appear for certain operations use `condition` if supported, or document which operations return them
|
||||
|
||||
### Block Metadata
|
||||
- [ ] `type` is snake_case (e.g., `'x'`, `'cloudflare'`)
|
||||
- [ ] `name` is human-readable (e.g., `'X'`, `'Cloudflare'`)
|
||||
- [ ] `description` is a concise one-liner
|
||||
- [ ] `longDescription` provides detail for docs
|
||||
- [ ] `docsLink` points to `'https://docs.sim.ai/integrations/{service}'`
|
||||
- [ ] `category` is `'tools'`
|
||||
- [ ] `bgColor` uses the service's brand color hex
|
||||
- [ ] `icon` references the correct icon component from `@/components/icons`
|
||||
- [ ] `authMode` is set correctly (`AuthMode.OAuth` or `AuthMode.ApiKey`)
|
||||
- [ ] Block + meta are registered in `blocks/registry-maps.ts` (`BLOCK_REGISTRY` / `BLOCK_META_REGISTRY`) alphabetically
|
||||
|
||||
### BlockMeta
|
||||
- [ ] `{Service}BlockMeta` is exported in the same file as the block
|
||||
- [ ] Has at least 7 templates, each with `icon`, `title`, `prompt`, `modules`, `category`, and `tags`
|
||||
- [ ] Prompts describe concrete use cases, not generic descriptions of what the service does
|
||||
- [ ] `alsoIntegrations` is set on any template whose prompt references another service
|
||||
- [ ] `skills` present (3–5 mainstream, 2–3 niche), each grounded in `tools.access` — flag any skill implying an unsupported action
|
||||
- [ ] **Each skill is real, not hallucinated** — web-search and confirm it maps to a popular use case attested online (vendor use-case pages, official docs describing the workflow, reputable "top automations" articles); rewrite/remove any you cannot source
|
||||
- [ ] Each skill has a kebab-case `name` (≤64 chars, unique), a one-line `description`, and markdown `content` with `# Title` + `## Steps` + an output/guidance section
|
||||
|
||||
### Block Inputs
|
||||
- [ ] `inputs` section lists all subBlock params that the block accepts
|
||||
- [ ] Input types match the subBlock types
|
||||
- [ ] When using `canonicalParamId`, inputs list the canonical ID (not the raw subBlock IDs)
|
||||
|
||||
## Step 5: Validate OAuth Scopes (if OAuth service)
|
||||
|
||||
Scopes are centralized — the single source of truth is `OAUTH_PROVIDERS` in `lib/oauth/oauth.ts`.
|
||||
|
||||
- [ ] Scopes defined in `lib/oauth/oauth.ts` under `OAUTH_PROVIDERS[provider].services[service].scopes`
|
||||
- [ ] `auth.ts` uses `getCanonicalScopesForProvider(providerId)` — NOT a hardcoded array
|
||||
- [ ] Block `requiredScopes` uses `getScopesForService(serviceId)` — NOT a hardcoded array
|
||||
- [ ] No hardcoded scope arrays in `auth.ts` or block files (should all use utility functions)
|
||||
- [ ] Each scope has a human-readable description in `SCOPE_DESCRIPTIONS` within `lib/oauth/utils.ts`
|
||||
- [ ] No excess scopes that aren't needed by any tool
|
||||
|
||||
## Step 6: Validate Deployment Availability (if OAuth service)
|
||||
|
||||
The deployment UI and setup CLI do not infer OAuth client fields from scopes. They resolve the
|
||||
block's generated `oauthServiceId` through the application-owned capability catalog.
|
||||
|
||||
- [ ] The visible integration block has exactly one distinct `oauth-input.serviceId`
|
||||
- [ ] `resolveOAuthClientCapabilityId(serviceId)` returns the intended provider capability
|
||||
- [ ] The resolved provider exists in `OAUTH_CLIENT_CAPABILITIES`
|
||||
- [ ] Every field listed by that capability exists in `apps/sim/lib/core/config/env.ts`
|
||||
- [ ] Every capability field has the correct `text` or `secret` entry in `OAUTH_CLIENT_SETUP_FIELDS`; no CLI naming heuristic is required
|
||||
- [ ] Shared Google/Microsoft service IDs resolve to their provider capability rather than duplicate entries
|
||||
- [ ] `bun run setup integration <capabilityId>` is the command emitted by availability; the CLI has only the exhaustive input-mode projection, not a second runtime provider definition
|
||||
- [ ] If the canonical OAuth service declares `serviceAccountProviderId`,
|
||||
`SERVICE_ACCOUNT_METADATA_BY_OAUTH_SERVICE_ID[serviceId]` has the same provider ID
|
||||
- [ ] The service-account `deploymentRequirement` matches how that credential actually works:
|
||||
omitted for an independent path, `'oauth-client'` when it needs the OAuth client fields, or
|
||||
`'preview-gated'` when controlled by a preview block
|
||||
|
||||
Treat a missing capability as **critical**: runtime availability intentionally throws instead of
|
||||
silently exposing an unusable integration.
|
||||
|
||||
## Step 7: Validate Pagination Consistency
|
||||
|
||||
If any tools support pagination:
|
||||
- [ ] Pagination param names match the API docs (e.g., `pagination_token` vs `next_token` vs `cursor`)
|
||||
- [ ] Different API endpoints that use different pagination param names have separate subBlocks in the block
|
||||
- [ ] Pagination response fields (`nextToken`, `cursor`, etc.) are included in tool outputs
|
||||
- [ ] Pagination subBlocks are set to `mode: 'advanced'`
|
||||
|
||||
## Step 8: Validate Memory Load Safety
|
||||
|
||||
If any tool lists, searches, exports, imports, downloads, uploads, paginates, batches, transforms arrays, or reads file/HTTP bodies, read `.agents/skills/memory-load-check/SKILL.md` and apply it to the integration.
|
||||
|
||||
- [ ] List/search tools expose API limits and do not auto-fetch every page into memory
|
||||
- [ ] Transform logic does not build unbounded arrays, maps, sets, or `Promise.all` fan-outs
|
||||
- [ ] File and HTTP body reads use explicit byte caps or existing stream-limit helpers
|
||||
- [ ] Large result payloads are summarized, paginated, referenced, or capped rather than raw-dumped
|
||||
- [ ] Pagination and download tests cover caps, early stop behavior, or partial-result preservation when relevant
|
||||
|
||||
## Step 9: Validate Error Handling
|
||||
|
||||
- [ ] `transformResponse` checks for error conditions before accessing data
|
||||
- [ ] Error responses include meaningful messages (not just generic "failed")
|
||||
- [ ] HTTP error status codes are handled (check `response.ok` or status codes)
|
||||
|
||||
## Step 10: Report and Fix
|
||||
|
||||
### Report Format
|
||||
|
||||
Group findings by severity:
|
||||
|
||||
**Critical** (will cause runtime errors or incorrect behavior):
|
||||
- Wrong endpoint URL or HTTP method
|
||||
- Missing required params or wrong `required` flag
|
||||
- Incorrect response field mapping (accessing wrong path in response)
|
||||
- Missing error handling that would cause crashes
|
||||
- Tool ID mismatch between tool file, registry, and block `tools.access`
|
||||
- OAuth scopes missing in `auth.ts` that tools need
|
||||
- OAuth integration `serviceId` missing from the deployment capability catalog
|
||||
- Capability references an env field absent from the runtime env schema
|
||||
- Service-account metadata disagrees with the canonical OAuth service configuration
|
||||
- `tools.config.tool` returning wrong tool ID for an operation
|
||||
- Type coercions in `tools.config.tool` instead of `tools.config.params`
|
||||
- Proven model-visible request fields bypass the shared projection or private-provenance boundary
|
||||
- Opaque model input is downloaded or sent before provenance and workspace-file checks
|
||||
- A Sim-owned durable sink or internal execution handoff drops encrypted provenance or breaks
|
||||
legacy headerless/`NULL` data
|
||||
- A tool substitutes secret plaintext into source, leaks private metadata, or generically sanitizes
|
||||
unrelated third-party results
|
||||
|
||||
**Warning** (follows conventions incorrectly or has usability issues):
|
||||
- Optional field not set to `mode: 'advanced'`
|
||||
- Missing `wandConfig` on timestamp/complex fields
|
||||
- Wrong `visibility` on params (e.g., `'hidden'` instead of `'user-or-llm'`)
|
||||
- Missing `optional: true` on nullable outputs
|
||||
- Opaque `type: 'json'` without property descriptions
|
||||
- Missing `.trim()` on ID fields in request URLs
|
||||
- Missing `?? null` on nullable response fields
|
||||
- Block condition array missing an operation that uses that field
|
||||
- Hardcoded scope arrays instead of using `getScopesForService()` / `getCanonicalScopesForProvider()`
|
||||
- Missing scope description in `SCOPE_DESCRIPTIONS` within `lib/oauth/utils.ts`
|
||||
|
||||
**Suggestion** (minor improvements):
|
||||
- Better description text
|
||||
- Inconsistent naming across tools
|
||||
- Missing `longDescription` or `docsLink`
|
||||
- Pagination fields that could benefit from `wandConfig`
|
||||
|
||||
### Fix All Issues
|
||||
|
||||
After reporting, fix every **critical** and **warning** issue. Apply **suggestions** where they don't add unnecessary complexity.
|
||||
|
||||
### Regenerate Derived Artifacts
|
||||
|
||||
Several files are generated from tool and block definitions. Editing a tool or block WITHOUT regenerating them fails CI, so run these before pushing:
|
||||
|
||||
```bash
|
||||
bun run tool-metadata:generate # repo root — apps/sim/tools/generated/*
|
||||
bun run scripts/generate-docs.ts # docs .mdx + lib/integrations/integrations.json + docs icons
|
||||
bun run integration-catalog:check # registry ↔ committed deployment metadata drift
|
||||
```
|
||||
|
||||
- **`tool-metadata:generate`** — required whenever a tool's `outputs`, `params`, or descriptions change. CI enforces this with `bun run tool-metadata:check`, which fails with *"Generated tool metadata is stale"*. This is the easiest gate to miss, because nothing in the tool file hints that a generated artifact mirrors it.
|
||||
- **`generate-docs`** — required whenever block metadata changes (`bgColor`, `name`, `description`, operations, outputs). Regenerates the integration `.mdx`, `integrations.json`, and the docs copy of `components/icons.tsx`.
|
||||
- **`integration-catalog:check`** — loads the executable block registry, derives visible integration
|
||||
deployment fields, and compares them with the committed catalog. It catches missing/unexpected
|
||||
entries and stale auth/service IDs without loading the executable registry in client code.
|
||||
|
||||
**Always diff the regen output before committing.** These generators rewrite every file they own, so they will also sweep in unrelated drift that accumulated on the base branch — pages losing sections, unrelated icons appearing. Keep only the hunks belonging to the integration under validation and `git checkout --` the rest, otherwise an unrelated doc regression rides along in the PR. Verify no page was silently dropped by comparing the directory listing before and after.
|
||||
|
||||
If an icon changed, `apps/sim/components/icons.tsx` is the source of truth and `apps/docs/components/icons.tsx` is its generated mirror — they must end up byte-identical for that component.
|
||||
|
||||
### Validation Output
|
||||
|
||||
After fixing, confirm:
|
||||
1. `bun run lint` passes with no fixes needed
|
||||
2. TypeScript compiles clean (no type errors) — check the error list is empty for the files you touched; pre-existing unrelated errors in a worktree usually mean workspace packages resolve to the main checkout
|
||||
3. The integration's tests pass, and any test you added actually fails without its fix (revert it once and watch it go red)
|
||||
4. Derived artifacts regenerated and their diffs reviewed (see above)
|
||||
5. `bun run integration-catalog:check` passes
|
||||
6. For OAuth or service-account changes, `bun test apps/sim/lib/integrations/availability.server.test.ts` passes
|
||||
7. Re-read all modified files to verify fixes are correct
|
||||
8. Any remaining unknown response schemas were explicitly reported to the user instead of guessed
|
||||
|
||||
## Checklist Summary
|
||||
|
||||
- [ ] Read ALL tool files, block, types, index, and registries
|
||||
- [ ] Pulled and read official API documentation
|
||||
- [ ] Validated every tool's ID, params, request, response, outputs, and types against API docs
|
||||
- [ ] Validated block ↔ tool alignment (every tool param has a subBlock, every condition is correct)
|
||||
- [ ] Validated advanced mode on optional/rarely-used fields
|
||||
- [ ] Validated wandConfig on timestamps and complex inputs
|
||||
- [ ] Validated tools.config mapping, tool selector, and type coercions
|
||||
- [ ] Validated block outputs match what tools return, with typed JSON where possible
|
||||
- [ ] Validated OAuth scopes use centralized utilities (getScopesForService, getCanonicalScopesForProvider) — no hardcoded arrays
|
||||
- [ ] Validated scope descriptions exist in `SCOPE_DESCRIPTIONS` within `lib/oauth/utils.ts` for all scopes
|
||||
- [ ] Validated OAuth `serviceId` resolves to the intended `OAUTH_CLIENT_CAPABILITIES` entry and all capability fields exist in the env schema
|
||||
- [ ] Validated service-account projection and deployment requirement against the canonical OAuth service config
|
||||
- [ ] Regenerated `integrations.json` when block metadata changed and ran `bun run integration-catalog:check`
|
||||
- [ ] Validated pagination consistency across tools and block
|
||||
- [ ] Validated memory load safety using `.agents/skills/memory-load-check/SKILL.md` when tools list/search/download/import/export/batch data
|
||||
- [ ] Validated error handling (error checks, meaningful messages)
|
||||
- [ ] Validated registry entries (tools and block, alphabetical, correct imports)
|
||||
- [ ] Validated model-visible/opaque inputs and Sim-durable/internal-execution provenance at their
|
||||
owning boundaries
|
||||
- [ ] Confirmed legacy persisted data keeps working and tracked invalid provenance fails closed
|
||||
- [ ] Confirmed ordinary third-party results remain unchanged absent activated Sim provenance
|
||||
- [ ] Validated `{Service}BlockMeta` exported with at least 7 templates
|
||||
- [ ] Reported all issues grouped by severity
|
||||
- [ ] Fixed all critical and warning issues
|
||||
- [ ] Ran `bun run tool-metadata:generate` if any tool outputs/params changed, and confirmed `bun run tool-metadata:check` passes
|
||||
- [ ] Ran `bun run generate-docs` if any block metadata changed, and reverted unrelated drift the generator swept in
|
||||
- [ ] Ran `bun run lint` after fixes
|
||||
- [ ] Verified TypeScript compiles clean
|
||||
- [ ] Verified added tests fail without their fix
|
||||
@@ -0,0 +1,171 @@
|
||||
---
|
||||
description: Validate a model entry (or every model in a provider) in apps/sim/providers/models.ts against the provider's live API docs (no hallucination — reports what cannot be verified)
|
||||
argument-hint: <provider> [model-id]
|
||||
---
|
||||
|
||||
# Validate Model Skill
|
||||
|
||||
You audit one or more model entries in `apps/sim/providers/models.ts` against the provider's official live API docs. **Hallucinated pricing and capabilities are the #1 failure mode in this file.** Every numeric and capability claim must be re-derived from a live web fetch in this session — not from memory, not from training data, not from the user's marketing email.
|
||||
|
||||
## Hard rules (do not skip)
|
||||
|
||||
1. **Live-fetch or report unverified.** Each field must be backed by a live WebFetch in this session. If you cannot reach an authoritative URL for a field, mark it **UNVERIFIED** in the report — do not silently confirm it from memory.
|
||||
2. **Cite every fact.** Every value in the report must show the source URL it was checked against. No URL → mark UNVERIFIED.
|
||||
3. **Two-source rule for pricing.** Cross-check input/output/cached against at least one secondary source (OpenRouter, Artificial Analysis, CloudPrice). If sources disagree, the provider's own docs win — flag the disagreement.
|
||||
4. **Inspect provider implementation before flagging capability mismatches.** A capability flag in `models.ts` is dead unless the provider's code under `apps/sim/providers/{provider}/` consumes it (see Consumption Matrix below). Setting a flag the provider ignores is a warning, not a critical.
|
||||
5. **Never auto-fix without printing the diff.** Show the user the proposed diff before applying. Get confirmation.
|
||||
|
||||
## Your Task
|
||||
|
||||
When invoked as `/validate-model <provider> [model-id]`:
|
||||
|
||||
1. Read the target entries from `models.ts`
|
||||
2. Live-fetch the provider's official models, pricing, and capability/reasoning pages + at least one secondary source for pricing
|
||||
3. Inspect the provider implementation to know which flags are actually consumed
|
||||
4. Run the checklist below per model
|
||||
5. Report findings (critical / warning / suggestion / unverified) with every cell linked to its source URL
|
||||
6. Offer to fix; on confirm, edit `models.ts` in a single pass and re-lint
|
||||
|
||||
If `model-id` is omitted, validate every model in the provider.
|
||||
|
||||
## Step 1: Read entries from `models.ts`
|
||||
|
||||
Capture per model: `id`, full `pricing`, full `capabilities`, `contextWindow`, `releaseDate`, `recommended`, `speedOptimized`, `deprecated`.
|
||||
|
||||
## Step 2: Live-fetch authoritative sources
|
||||
|
||||
Use the canonical provider URL table in the `add-model` skill (`.agents/skills/add-model/SKILL.md`), Step 1, as the single source of truth — fetch the models index, pricing, and reasoning/parameter caveats pages listed there for the target provider.
|
||||
|
||||
Secondary cross-check (use at least one): OpenRouter, Artificial Analysis, CloudPrice.
|
||||
|
||||
If a fetch fails (404, timeout, paywall), record the URL attempted and mark dependent fields UNVERIFIED.
|
||||
|
||||
## Step 3: Build the consumption map for this provider
|
||||
|
||||
Re-grep before trusting the snapshot below:
|
||||
|
||||
```bash
|
||||
rg "reasoningEffort|reasoning_effort" apps/sim/providers/<provider>/
|
||||
rg "verbosity" apps/sim/providers/<provider>/
|
||||
rg "request\.thinking|thinking:" apps/sim/providers/<provider>/
|
||||
rg "supportsNativeStructuredOutputs|nativeStructuredOutputs" apps/sim/providers/<provider>/
|
||||
```
|
||||
|
||||
Snapshot (verify before relying):
|
||||
|
||||
| Capability | Consumed by |
|
||||
|---|---|
|
||||
| `reasoningEffort` | `openai/core.ts`, `azure-openai`, `anthropic/core.ts` (mapped via thinking), `gemini/core.ts` |
|
||||
| `verbosity` | `openai/core.ts`, `azure-openai/index.ts` |
|
||||
| `thinking` | `anthropic/core.ts`, `gemini/core.ts` |
|
||||
| `nativeStructuredOutputs` | `anthropic/core.ts`, `fireworks/index.ts`, `openrouter/index.ts` |
|
||||
| `computerUse` | `anthropic/core.ts` |
|
||||
| `temperature` | All providers (passthrough) |
|
||||
|
||||
A flag set in `models.ts` but not in the consumption list for this provider = **warning: dead flag**.
|
||||
|
||||
## Step 4: Run the checklist
|
||||
|
||||
For each model, evaluate every row. Statuses: ✓ matches docs, ✗ disagrees, ⚠️ single-source, ❓ UNVERIFIED (could not fetch).
|
||||
|
||||
### Identity
|
||||
- [ ] `id` exactly matches provider's API model identifier (case, dots, dashes, prefix for resellers)
|
||||
- [ ] `releaseDate` matches launch announcement
|
||||
- [ ] `deprecated: true` set if provider has announced retirement (or removed from active list)
|
||||
|
||||
### Pricing (per 1M tokens, USD)
|
||||
- [ ] `pricing.input` matches provider pricing page
|
||||
- [ ] `pricing.output` matches provider pricing page
|
||||
- [ ] `pricing.cachedInput` matches provider's documented cached/prompt-cache rate (or is correctly omitted if no caching offered)
|
||||
- [ ] `pricing.updatedAt` is recent — warn if older than 60 days
|
||||
|
||||
### Context & output limits
|
||||
- [ ] `contextWindow` matches docs (in tokens)
|
||||
- [ ] `capabilities.maxOutputTokens` matches documented output cap (or is correctly omitted if "no output limit")
|
||||
|
||||
### Capabilities (each must be DOCUMENTED-AS-SUPPORTED **and** CONSUMED-BY-PROVIDER-CODE)
|
||||
- [ ] `temperature` — provider accepts it for this model (reasoning-always-on models often reject)
|
||||
- [ ] `reasoningEffort.values` — list matches docs; **omitted** for always-reasoning models that reject the parameter (e.g., grok-4.3, where xAI docs explicitly state `reasoning_effort` is not supported). Verify per model — some always-reasoning models (e.g., OpenAI's o-series) DO accept `reasoning_effort` and should keep the flag.
|
||||
- [ ] `verbosity.values` — only on OpenAI gpt-5.x family; values match docs
|
||||
- [ ] `thinking.levels` + `thinking.default` — only on Anthropic/Gemini; values match docs
|
||||
- [ ] `thinking.streamed` — REQUIRED on Anthropic-family thinking models (`'full'` for generations returning full thinking deltas, `'summary'` for omitted-display generations like Opus 4.7+/Sonnet 5/Fable 5 where Sim requests `display: 'summarized'`); verify against the provider's thinking-display docs. After any change, run `bun run agent-stream-docs:generate` so the Agent block docs table stays in sync (CI diffs it)
|
||||
- [ ] `nativeStructuredOutputs` — only on anthropic/fireworks/openrouter; provider must document Structured Outputs / JSON-mode for this model
|
||||
- [ ] `toolUsageControl` — provider supports `tool_choice` semantics
|
||||
- [ ] `computerUse` — provider implements computer-use loop AND model is a computer-use SKU
|
||||
- [ ] `deepResearch` — only on actual deep-research SKUs
|
||||
- [ ] `memory: false` — only when the model genuinely cannot maintain conversation history
|
||||
|
||||
### Flags
|
||||
- [ ] `recommended: true` — at most one or two per provider; should be current flagship
|
||||
- [ ] `speedOptimized: true` — only on smallest/fastest tier (nano / flash-lite / haiku class)
|
||||
|
||||
### Hosting / billing
|
||||
- [ ] If the model is under `openai`/`anthropic`/`google`, it is automatically in `getHostedModels()` → served with Sim's rotating key and billed via `shouldBillModelUsage()`. Confirm that is the intent (a BYOK-only model parked under one of these providers is a billing bug — warning).
|
||||
- [ ] If the model is hosted, the deployment is expected to have its `{PREFIX}_COUNT` / `{PREFIX}_1..N` env vars set (ops concern; note if it looks unset for a model claiming hosted support).
|
||||
|
||||
## Step 5: Report (mandatory format)
|
||||
|
||||
For each model, emit a table with one row per checklist item. Every row that claims ✓ must have a URL.
|
||||
|
||||
```markdown
|
||||
### Validation — <model-id>
|
||||
|
||||
| Field | Repo | Live docs | Source URL | Status |
|
||||
|---|---|---|---|---|
|
||||
| `input` | $1.25/M | $1.25/M | https://docs.x.ai/... | ✓ |
|
||||
| `cachedInput` | $0.50/M | $0.20/M | https://cloudprice.net/... | ✗ stale (price cut not picked up) |
|
||||
| `reasoningEffort` | low/medium/high | rejected by API | https://docs.x.ai/.../reasoning | ✗ inert — selecting silently no-ops |
|
||||
| `contextWindow` | 1,000,000 | 1,000,000 | https://docs.x.ai/... + https://openrouter.ai/... | ✓ (2 sources) |
|
||||
| `releaseDate` | 2026-04-30 | not found in scraped pages | _attempted: docs.x.ai, x.ai/news_ | ❓ UNVERIFIED |
|
||||
|
||||
**Findings**
|
||||
- 🔴 critical — `cachedInput` is wrong: docs say $0.20/M, repo has $0.50/M
|
||||
- 🟡 warning — `reasoningEffort` is set but provider rejects it for this model (xAI docs explicitly: "reasoning_effort is not supported by grok-4.3")
|
||||
- 🔵 suggestion — `pricing.updatedAt` is 90 days old; refresh
|
||||
- ❓ unverified — `releaseDate` could not be confirmed from any fetched page; ask user
|
||||
|
||||
**Disagreements between sources**
|
||||
- _none_ OR _OpenRouter says $X, provider docs say $Y — went with provider docs_
|
||||
```
|
||||
|
||||
End each multi-model run with a summary count: `N models checked · X critical · Y warnings · Z suggestions · W unverified`.
|
||||
|
||||
## Step 6: Offer to fix
|
||||
|
||||
After reporting, ask: *"Want me to fix the critical and warning items? I'll print the diff first."* On yes:
|
||||
|
||||
1. Print the proposed diff (do not apply yet)
|
||||
2. Get user confirmation
|
||||
3. Edit `models.ts` in a single pass
|
||||
4. Run `bun run lint`
|
||||
5. Re-run only the failed rows of the checklist on the new state
|
||||
|
||||
## Severity definitions
|
||||
|
||||
- 🔴 **critical** — wrong number or wrong identifier that misleads users about cost or breaks API calls. Examples: incorrect pricing, wrong model id, wrong context window, capability the API rejects.
|
||||
- 🟡 **warning** — dead code or internal inconsistency. Examples: capability flag the provider ignores, multiple `recommended: true` per provider, `pricing.updatedAt` >60 days old, missing `deprecated: true` on retired model.
|
||||
- 🔵 **suggestion** — style/consistency. Examples: field order, missing `speedOptimized` on a clearly smallest-tier model.
|
||||
- ❓ **unverified** — could not fetch an authoritative source for this field. Surface it; never silently confirm.
|
||||
|
||||
## Common bugs this skill catches
|
||||
|
||||
- Pricing drift after a provider price cut (very common — providers cut quarterly)
|
||||
- `reasoningEffort` set on always-reasoning models that reject the parameter (grok-4.3, o3-pro pattern)
|
||||
- `nativeStructuredOutputs` set on providers that don't consume the flag (dead)
|
||||
- `thinking` set on non-Anthropic/non-Gemini providers
|
||||
- `verbosity` set on non-gpt-5.x models
|
||||
- Wrong context window (e.g., 128k claimed vs 200k actual)
|
||||
- Stale `pricing.updatedAt`
|
||||
- Multiple `recommended: true` per provider after a flagship swap
|
||||
- Missing `deprecated: true` on retired models (e.g., the xAI batch retiring May 15, 2026)
|
||||
|
||||
## What "I cannot verify this" looks like
|
||||
|
||||
If, after fetching the documented sources, a field cannot be confirmed:
|
||||
|
||||
- Mark the row ❓ UNVERIFIED with the URL(s) attempted
|
||||
- Surface it in the **Findings** section with severity ❓
|
||||
- Do NOT mark the validation as passed
|
||||
- Ask the user for a docs URL or guidance before changing anything
|
||||
|
||||
The skill is allowed to say *"I could not verify the cached input price for grok-4.3 from the official xAI docs in this session — I attempted [URLs] without finding the value. Third-party sources [URL1, URL2] both report $0.20/M. Confirm before I update."* That is correct behavior. Hallucinating a number is not.
|
||||
@@ -0,0 +1,228 @@
|
||||
---
|
||||
description: Validate an existing Sim webhook trigger against provider API docs and repository conventions
|
||||
argument-hint: <service-name> [api-docs-url]
|
||||
---
|
||||
|
||||
# Validate Trigger
|
||||
|
||||
You are an expert auditor for Sim webhook triggers. Your job is to validate that an existing trigger implementation is correct, complete, secure, and aligned across all layers.
|
||||
|
||||
## Your Task
|
||||
|
||||
1. Read the service's webhook/API documentation (via WebFetch)
|
||||
2. Read every trigger file, provider handler, and registry entry
|
||||
3. Cross-reference against the API docs and Sim conventions
|
||||
4. Report all issues grouped by severity (critical, warning, suggestion)
|
||||
5. Fix all issues after reporting them
|
||||
|
||||
## Step 1: Gather All Files
|
||||
|
||||
Read **every** file for the trigger — do not skip any:
|
||||
|
||||
```
|
||||
apps/sim/triggers/{service}/ # All trigger files, utils.ts, index.ts
|
||||
apps/sim/lib/webhooks/providers/{service}.ts # Provider handler (if exists)
|
||||
apps/sim/lib/webhooks/providers/registry.ts # Handler registry
|
||||
apps/sim/triggers/registry.ts # Trigger registry
|
||||
apps/sim/blocks/blocks/{service}.ts # Block definition (trigger wiring)
|
||||
```
|
||||
|
||||
Also read for reference:
|
||||
```
|
||||
apps/sim/lib/webhooks/providers/types.ts # WebhookProviderHandler interface
|
||||
apps/sim/lib/webhooks/providers/utils.ts # Shared helpers (createHmacVerifier, etc.)
|
||||
apps/sim/lib/webhooks/provider-subscription-utils.ts # Subscription helpers
|
||||
apps/sim/lib/webhooks/processor.ts # Central webhook processor
|
||||
```
|
||||
|
||||
## Step 2: Pull API Documentation
|
||||
|
||||
Fetch the service's official webhook documentation. This is the **source of truth** for:
|
||||
- Webhook event types and payload shapes
|
||||
- Signature/auth verification method (HMAC algorithm, header names, secret format)
|
||||
- Challenge/verification handshake requirements
|
||||
- Webhook subscription API (create/delete endpoints, if applicable)
|
||||
- Retry behavior and delivery guarantees
|
||||
|
||||
### Hard Rule: No Guessed Webhook Payload Schemas
|
||||
|
||||
If the official docs do not clearly show the webhook payload JSON for an event, you MUST tell the user instead of guessing.
|
||||
|
||||
- Do NOT invent payload field names
|
||||
- Do NOT infer nested payload paths without evidence
|
||||
- Do NOT treat likely event shapes as verified
|
||||
- Do NOT accept `formatInput` mappings that are not backed by docs or live payloads
|
||||
|
||||
If a payload schema is unknown, validation must explicitly recommend:
|
||||
1. sample webhook payloads,
|
||||
2. a live test webhook source, or
|
||||
3. trimming the trigger to only documented outputs.
|
||||
|
||||
## Step 3: Validate Trigger Definitions
|
||||
|
||||
### utils.ts
|
||||
- [ ] `{service}TriggerOptions` lists all trigger IDs accurately
|
||||
- [ ] `{service}SetupInstructions` provides clear, correct steps for the service
|
||||
- [ ] `build{Service}ExtraFields` includes relevant filter/config fields with correct `condition`
|
||||
- [ ] Output builders expose all meaningful fields from the webhook payload
|
||||
- [ ] Output builders do NOT use `optional: true` or `items` (tool-output-only features)
|
||||
- [ ] Nested output objects correctly model the payload structure
|
||||
|
||||
### Trigger Files
|
||||
- [ ] Exactly one primary trigger has `includeDropdown: true`
|
||||
- [ ] All secondary triggers do NOT have `includeDropdown`
|
||||
- [ ] All triggers use `buildTriggerSubBlocks` helper (not hand-rolled subBlocks)
|
||||
- [ ] Every trigger's `id` matches the convention `{service}_{event_name}`
|
||||
- [ ] Every trigger's `provider` matches the service name used in the handler registry
|
||||
- [ ] `index.ts` barrel exports all triggers
|
||||
|
||||
### Trigger ↔ Provider Alignment (CRITICAL)
|
||||
- [ ] Every trigger ID referenced in `matchEvent` logic exists in `{service}TriggerOptions`
|
||||
- [ ] Event matching logic in the provider correctly maps trigger IDs to service event types
|
||||
- [ ] Event matching logic in `is{Service}EventMatch` (if exists) correctly identifies events per the API docs
|
||||
|
||||
## Step 4: Validate Provider Handler
|
||||
|
||||
### Auth Verification
|
||||
- [ ] `verifyAuth` correctly validates webhook signatures per the service's documentation
|
||||
- [ ] HMAC algorithm matches (SHA-1, SHA-256, SHA-512)
|
||||
- [ ] Signature header name matches the API docs exactly
|
||||
- [ ] Signature format is handled (raw hex, `sha256=` prefix, base64, etc.)
|
||||
- [ ] Uses `safeCompare` for timing-safe comparison (no `===`)
|
||||
- [ ] If `webhookSecret` is required, handler rejects when it's missing (fail-closed)
|
||||
- [ ] Signature is computed over raw body (not parsed JSON)
|
||||
|
||||
### Event Matching
|
||||
- [ ] `matchEvent` returns `boolean` (not `NextResponse` or other values)
|
||||
- [ ] Challenge/verification events are excluded from matching (e.g., `endpoint.url_validation`)
|
||||
- [ ] When `triggerId` is a generic webhook ID, all events pass through
|
||||
- [ ] When `triggerId` is specific, only matching events pass
|
||||
- [ ] Event matching logic uses dynamic `await import()` for trigger utils (avoids circular deps)
|
||||
|
||||
### formatInput (CRITICAL)
|
||||
- [ ] Every key in the `formatInput` return matches a key in the trigger `outputs` schema
|
||||
- [ ] Every key in the trigger `outputs` schema is populated by `formatInput`
|
||||
- [ ] No extra undeclared keys that users can't discover in the UI
|
||||
- [ ] No wrapper objects (`webhook: { ... }`, `{service}: { ... }`)
|
||||
- [ ] Nested output paths exist at the correct depth (e.g., `resource.id` actually has `resource: { id: ... }`)
|
||||
- [ ] `null` is used for missing optional fields (not empty strings or empty objects)
|
||||
- [ ] Returns `{ input: { ... } }` — not a bare object
|
||||
- [ ] Every mapped payload field is backed by official docs or live-verified webhook payloads
|
||||
|
||||
### Idempotency
|
||||
- [ ] `extractIdempotencyId` returns a stable, unique key per delivery
|
||||
- [ ] Uses provider-specific delivery IDs when available (e.g., `X-Request-Id`, `Linear-Delivery`, `svix-id`)
|
||||
- [ ] Falls back to content-based ID (e.g., `${type}:${id}`) when no delivery header exists
|
||||
- [ ] Does NOT include timestamps in the idempotency key (would break dedup on retries)
|
||||
|
||||
### Challenge Handling (if applicable)
|
||||
- [ ] `handleChallenge` correctly implements the service's URL verification handshake
|
||||
- [ ] Returns the expected response format per the API docs
|
||||
- [ ] Env-backed secrets are resolved via `resolveEnvVarsInObject` if needed
|
||||
|
||||
## Step 5: Validate Automatic Subscription Lifecycle
|
||||
|
||||
If the service supports programmatic webhook creation:
|
||||
|
||||
### createSubscription
|
||||
- [ ] Calls the correct API endpoint to create a webhook
|
||||
- [ ] Sends the correct event types/filters
|
||||
- [ ] Passes the notification URL from `getNotificationUrl(ctx.webhook)`
|
||||
- [ ] Returns `{ providerConfigUpdates: { externalId } }` with the external webhook ID
|
||||
- [ ] Throws on failure (orchestration handles rollback)
|
||||
- [ ] Provides user-friendly error messages (401 → "Invalid API Key", etc.)
|
||||
|
||||
### deleteSubscription
|
||||
- [ ] Calls the correct API endpoint to delete the webhook
|
||||
- [ ] Handles 404 gracefully (webhook already deleted)
|
||||
- [ ] Never throws — catches errors and logs non-fatally
|
||||
- [ ] Skips gracefully when `apiKey` or `externalId` is missing
|
||||
|
||||
### Orchestration Isolation
|
||||
- [ ] NO provider-specific logic in `route.ts`, `provider-subscriptions.ts`, or `deploy.ts`
|
||||
- [ ] All subscription logic lives on the handler (`createSubscription`/`deleteSubscription`)
|
||||
|
||||
## Step 6: Validate Registration and Block Wiring
|
||||
|
||||
### Trigger Registry (`triggers/registry.ts`)
|
||||
- [ ] All triggers are imported and registered
|
||||
- [ ] Registry keys match trigger IDs exactly
|
||||
- [ ] No orphaned entries (triggers that don't exist)
|
||||
|
||||
### Provider Handler Registry (`providers/registry.ts`)
|
||||
- [ ] Handler is imported and registered (if handler exists)
|
||||
- [ ] Registry key matches the `provider` field on the trigger configs
|
||||
- [ ] Entries are in alphabetical order
|
||||
|
||||
### Block Wiring (`blocks/blocks/{service}.ts`)
|
||||
- [ ] Block has `triggers.enabled: true`
|
||||
- [ ] `triggers.available` lists all trigger IDs
|
||||
- [ ] All trigger subBlocks are spread into `subBlocks`: `...getTrigger('id').subBlocks`
|
||||
- [ ] No trigger IDs in `triggers.available` that aren't in the registry
|
||||
- [ ] No trigger subBlocks spread that aren't in `triggers.available`
|
||||
|
||||
## Step 7: Validate Security
|
||||
|
||||
- [ ] Webhook secrets are never logged (not even at debug level)
|
||||
- [ ] Auth verification runs before any event processing
|
||||
- [ ] No secret comparison uses `===` (must use `safeCompare` or `crypto.timingSafeEqual`)
|
||||
- [ ] Timestamp/replay protection is reasonable (not too tight for retries, not too loose for security)
|
||||
- [ ] Raw body is used for signature verification (not re-serialized JSON)
|
||||
|
||||
## Step 8: Report and Fix
|
||||
|
||||
### Report Format
|
||||
|
||||
Group findings by severity:
|
||||
|
||||
**Critical** (runtime errors, security issues, or data loss):
|
||||
- Wrong HMAC algorithm or header name
|
||||
- `formatInput` keys don't match trigger `outputs`
|
||||
- Missing `verifyAuth` when the service sends signed webhooks
|
||||
- `matchEvent` returns non-boolean values
|
||||
- Provider-specific logic leaking into shared orchestration files
|
||||
- Trigger IDs mismatch between trigger files, registry, and block
|
||||
- `createSubscription` calling wrong API endpoint
|
||||
- Auth comparison using `===` instead of `safeCompare`
|
||||
|
||||
**Warning** (convention violations or usability issues):
|
||||
- Missing `extractIdempotencyId` when the service provides delivery IDs
|
||||
- Timestamps in idempotency keys (breaks dedup on retries)
|
||||
- Missing challenge handling when the service requires URL verification
|
||||
- Output schema missing fields that `formatInput` returns (undiscoverable data)
|
||||
- Overly tight timestamp skew window that rejects legitimate retries
|
||||
- `matchEvent` not filtering challenge/verification events
|
||||
- Setup instructions missing important steps
|
||||
|
||||
**Suggestion** (minor improvements):
|
||||
- More specific output field descriptions
|
||||
- Additional output fields that could be exposed
|
||||
- Better error messages in `createSubscription`
|
||||
- Logging improvements
|
||||
|
||||
### Fix All Issues
|
||||
|
||||
After reporting, fix every **critical** and **warning** issue. Apply **suggestions** where they don't add unnecessary complexity.
|
||||
|
||||
### Validation Output
|
||||
|
||||
After fixing, confirm:
|
||||
1. `bun run type-check` passes
|
||||
2. Re-read all modified files to verify fixes are correct
|
||||
3. Provider handler tests pass (if they exist): `bun test {service}`
|
||||
4. Any remaining unknown webhook payload schemas were explicitly reported to the user instead of guessed
|
||||
|
||||
## Checklist Summary
|
||||
|
||||
- [ ] Read all trigger files, provider handler, types, registries, and block
|
||||
- [ ] Pulled and read official webhook/API documentation
|
||||
- [ ] Validated trigger definitions: options, instructions, extra fields, outputs
|
||||
- [ ] Validated primary/secondary trigger distinction (`includeDropdown`)
|
||||
- [ ] Validated provider handler: auth, matchEvent, formatInput, idempotency
|
||||
- [ ] Validated output alignment: every `outputs` key ↔ every `formatInput` key
|
||||
- [ ] Validated subscription lifecycle: createSubscription, deleteSubscription, no shared-file edits
|
||||
- [ ] Validated registration: trigger registry, handler registry, block wiring
|
||||
- [ ] Validated security: safe comparison, no secret logging, replay protection
|
||||
- [ ] Reported all issues grouped by severity
|
||||
- [ ] Fixed all critical and warning issues
|
||||
- [ ] `bun run type-check` passes after fixes
|
||||
@@ -0,0 +1,62 @@
|
||||
---
|
||||
description: Analyze and fix useCallback anti-patterns in your code
|
||||
argument-hint: "[scope] [fix=true|false]"
|
||||
---
|
||||
|
||||
# You Might Not Need a Callback
|
||||
|
||||
Arguments:
|
||||
- scope: what to analyze (default: your current changes). Examples: "diff to main", "PR #123", "src/components/", "whole codebase"
|
||||
- fix: whether to apply fixes (default: true). Set to false to only propose changes.
|
||||
|
||||
User arguments: $ARGUMENTS
|
||||
|
||||
## References
|
||||
|
||||
Read before analyzing:
|
||||
1. https://react.dev/reference/react/useCallback — official docs on when useCallback is actually needed
|
||||
|
||||
## The one rule that matters
|
||||
|
||||
`useCallback` is only useful when **something observes the reference**. Ask: does anything care if this function gets a new identity on re-render?
|
||||
|
||||
Observers that care about reference stability:
|
||||
- A `useEffect` that lists the function in its deps array
|
||||
- A `useMemo` that lists the function in its deps array
|
||||
- Another `useCallback` that lists the function in its deps array
|
||||
- A child component wrapped in `React.memo` that receives the function as a prop
|
||||
- A custom hook that documents a referential-stability requirement for the callback
|
||||
|
||||
If none of those apply — if the function is only called inline, or passed to a non-memoized child, or assigned to a native element event — the reference is unobserved and `useCallback` adds overhead with zero benefit.
|
||||
|
||||
## Anti-patterns to detect
|
||||
|
||||
1. **No observer tracks the reference**: The function is only called inline in the same component, or passed to a non-memoized child, or used as a native element handler (`<button onClick={fn}>`). Nothing re-runs or bails out based on reference identity. Remove `useCallback`.
|
||||
2. **useCallback with deps that change every render**: If a dep is a plain object/array created inline, or state that changes on every interaction, memoization buys nothing — the function gets a new identity anyway.
|
||||
3. **useCallback on handlers passed only to native elements**: `<button onClick={fn}>` — React never does reference equality on native element props. No benefit.
|
||||
4. **useCallback wrapping functions that return new objects/arrays**: Stable function identity, unstable return value — memoization is at the wrong level. Use `useMemo` on the return value instead, or restructure.
|
||||
5. **useCallback with empty deps when deps are needed**: Stale closure — reads initial values forever. This is a correctness bug, not just a performance issue.
|
||||
6. **Pairing useCallback + React.memo on trivially cheap renders**: If the child renders in < 1ms and re-renders rarely, the memo infrastructure costs more than it saves.
|
||||
7. **useCallback in custom hooks that don't need stable references**: Not every hook return needs to be memoized. Only stabilize callbacks when consumers depend on referential equality.
|
||||
|
||||
## Patterns that ARE correct — do not flag
|
||||
|
||||
- `useCallback` whose result is in a `useEffect` dep array — prevents the effect from re-running on every render
|
||||
- `useCallback` whose result is in a `useMemo` dep array — prevents the memo from recomputing on every render
|
||||
- `useCallback` whose result is a dep of another `useCallback` — stabilises a callback chain
|
||||
- `useCallback` passed to a `React.memo`-wrapped child — the whole point of the pattern
|
||||
- This codebase's ref pattern: `useRef` + callback with empty deps that reads the ref inside — correct, do not flag:
|
||||
|
||||
```tsx
|
||||
const idRef = useRef(id)
|
||||
useEffect(() => { idRef.current = id }, [id])
|
||||
const fetchData = useCallback(async () => {
|
||||
// use idRef.current instead of id
|
||||
}, []) // empty deps because refs are used
|
||||
```
|
||||
|
||||
## Steps
|
||||
|
||||
1. Read the reference above
|
||||
2. Analyze the specified scope for the anti-patterns listed above
|
||||
3. If fix=true, apply the fixes. If fix=false, propose the fixes without applying.
|
||||
@@ -0,0 +1,45 @@
|
||||
---
|
||||
description: Analyze and fix redundant or self-explanatory inline comments — remove noise, promote genuine documentation to TSDoc
|
||||
argument-hint: "[scope] [fix=true|false]"
|
||||
---
|
||||
|
||||
# You Might Not Need a Comment
|
||||
|
||||
Arguments:
|
||||
- scope: what to analyze (default: your current changes). Examples: "diff to main", "PR #123", "src/components/", "whole codebase"
|
||||
- fix: whether to apply fixes (default: true). Set to false to only propose changes.
|
||||
|
||||
User arguments: $ARGUMENTS
|
||||
|
||||
## The one rule that matters
|
||||
|
||||
A comment must add information the code cannot express itself. Code says *what* and *how*; a comment earns its place only by explaining *why* — a non-obvious constraint, a workaround, a decision, a gotcha. If deleting the comment loses no information a competent reader wouldn't recover from the code in seconds, delete it.
|
||||
|
||||
This codebase's convention: **TSDoc for documentation, no non-TSDoc comments, no `====` separators.** Genuine documentation belongs in a `/** ... */` block on the declaration; everything that survives as an inline `//` comment must be a real *why*, kept terse.
|
||||
|
||||
## Anti-patterns to detect
|
||||
|
||||
1. **Restates the code**: `// increment counter` above `counter++`, `// return the result` above `return result`, `// loop over items`. Delete.
|
||||
2. **Narrates the obvious from names**: the function is `fetchUserById`, the comment says `// fetches a user by id`. The identifier already said it. Delete.
|
||||
3. **Section-divider / banner comments**: `// ==== Helpers ====`, `// --- state ---`, `// #region`. Against convention. Delete (the code's structure is the structure).
|
||||
4. **Commented-out code**: dead code left as a comment. Delete — git is the history.
|
||||
5. **Redundant type/param echo in prose comments**: `// takes a string and returns a number` when the signature already says so. Delete.
|
||||
6. **Changelog / attribution noise**: `// added by X`, `// TODO(2021): ...` long-stale, `// fix for bug`. Delete unless it encodes a live, actionable constraint.
|
||||
7. **Genuine documentation written as a loose `//` block on a declaration**: a real explanation of what an exported function/type/const is for, but written as stacked `//` lines instead of TSDoc. Convert to a `/** ... */` TSDoc block on the declaration.
|
||||
|
||||
## Patterns that ARE correct — do not flag
|
||||
|
||||
- A `//` comment that explains a **non-obvious why**: a workaround for an upstream bug, an ordering constraint, a perf reason, a spec/edge-case the code can't self-document (`// first-match wins — matches the old find() semantics`).
|
||||
- Existing TSDoc `/** ... */` blocks on declarations — leave them (only tighten if verbose).
|
||||
- `// boundary-raw-fetch:`, `// double-cast-allowed:`, `// boundary-raw-json:`, `// untyped-response:`, `// migration-safe:` and other **machine-read annotations** — these are load-bearing, never touch them.
|
||||
- `// biome-ignore`, `// eslint-disable`, `// @ts-expect-error` and other tooling directives.
|
||||
- `// TODO` / `// FIXME` that point at real, still-open work.
|
||||
|
||||
## Bias
|
||||
|
||||
Prefer **deletion over rewriting**, and **no comment over a comment** when the code is already clear. When a comment is genuine documentation, prefer promoting it to terse TSDoc over leaving a loose `//` block. Never add new comments in this pass — this is a reduction pass. When unsure whether a comment encodes a real *why*, keep it.
|
||||
|
||||
## Steps
|
||||
|
||||
1. Analyze the specified scope for the anti-patterns listed above
|
||||
2. If fix=true, apply the fixes. If fix=false, propose the fixes without applying.
|
||||
@@ -0,0 +1,33 @@
|
||||
---
|
||||
description: Analyze and fix useMemo/React.memo anti-patterns in your code
|
||||
argument-hint: "[scope] [fix=true|false]"
|
||||
---
|
||||
|
||||
# You Might Not Need a Memo
|
||||
|
||||
Arguments:
|
||||
- scope: what to analyze (default: your current changes). Examples: "diff to main", "PR #123", "src/components/", "whole codebase"
|
||||
- fix: whether to apply fixes (default: true). Set to false to only propose changes.
|
||||
|
||||
User arguments: $ARGUMENTS
|
||||
|
||||
## References
|
||||
|
||||
Read before analyzing:
|
||||
1. https://overreacted.io/before-you-memo/ — two techniques to avoid memo entirely
|
||||
|
||||
## Anti-patterns to detect
|
||||
|
||||
1. **Wrapping a slow component in React.memo when state can be moved down**: If a component re-renders because of state it doesn't use, move that state into a smaller child component instead of memoizing. The slow component stops re-rendering without memo.
|
||||
2. **Wrapping in React.memo when children can be lifted up**: If a parent owns state that changes frequently, extract the stateful part and pass the expensive subtree as `children`. Children passed as props don't re-render when the parent's state changes.
|
||||
3. **useMemo on cheap computations**: Filtering or mapping a small array, string concatenation, simple arithmetic — these don't need memoization. Only memoize when you've measured a performance problem.
|
||||
4. **useMemo with constantly-changing deps**: If the dependency array changes on every render, useMemo does nothing — it recalculates every time. Fix the deps or remove the memo.
|
||||
5. **useMemo to create objects/arrays passed as props**: Instead of memoizing to prevent child re-renders, consider whether the child even needs referential stability. If the child doesn't use React.memo or pass it to a dep array, the memo is wasted.
|
||||
6. **React.memo on components that always receive new props**: If the parent always passes new objects, arrays, or callbacks, React.memo's shallow comparison always fails. Fix the parent instead of memoizing the child.
|
||||
7. **useMemo for derived state**: If you're computing a value from props or state, just compute it inline during render. React renders are fast. `const fullName = first + ' ' + last` doesn't need useMemo.
|
||||
|
||||
## Steps
|
||||
|
||||
1. Read the reference above to understand the two core techniques (move state down, lift content up)
|
||||
2. Analyze the specified scope for the anti-patterns listed above
|
||||
3. If fix=true, apply the fixes. If fix=false, propose the fixes without applying.
|
||||
@@ -0,0 +1,17 @@
|
||||
---
|
||||
description: Analyze and fix useEffect anti-patterns in your code
|
||||
argument-hint: "[scope] [fix=true|false]"
|
||||
---
|
||||
|
||||
# You Might Not Need an Effect
|
||||
|
||||
Arguments:
|
||||
- scope: what to analyze (default: your current changes). Examples: "diff to main", "PR #123", "src/components/", "whole codebase"
|
||||
- fix: whether to apply fixes (default: true). Set to false to only propose changes.
|
||||
|
||||
User arguments: $ARGUMENTS
|
||||
|
||||
Steps:
|
||||
1. Read https://react.dev/learn/you-might-not-need-an-effect to understand the guidelines
|
||||
2. Analyze the specified scope for useEffect anti-patterns
|
||||
3. If fix=true, apply the fixes. If fix=false, propose the fixes without applying.
|
||||
@@ -0,0 +1,38 @@
|
||||
---
|
||||
description: Analyze and fix unnecessary useState, derived state, and server-state-in-local-state anti-patterns
|
||||
argument-hint: "[scope] [fix=true|false]"
|
||||
---
|
||||
|
||||
# You Might Not Need State
|
||||
|
||||
Arguments:
|
||||
- scope: what to analyze (default: your current changes). Examples: "diff to main", "PR #123", "src/components/", "whole codebase"
|
||||
- fix: whether to apply fixes (default: true). Set to false to only propose changes.
|
||||
|
||||
User arguments: $ARGUMENTS
|
||||
|
||||
## Context
|
||||
|
||||
This codebase uses React Query for all server state and Zustand for client-only global state. useState should only be used for ephemeral UI concerns (open/closed, hover, local form input). Server data should never be copied into useState or Zustand — React Query is the single source of truth.
|
||||
|
||||
## References
|
||||
|
||||
Read these before analyzing:
|
||||
1. https://react.dev/learn/choosing-the-state-structure — 5 principles for structuring state
|
||||
2. https://tkdodo.eu/blog/dont-over-use-state — never store derived/computed values in state
|
||||
3. https://tkdodo.eu/blog/putting-props-to-use-state — never mirror props into state via useEffect
|
||||
|
||||
## Anti-patterns to detect
|
||||
|
||||
1. **Derived state stored in useState**: If a value can be computed from props, other state, or query data, compute it inline during render instead of storing it in state.
|
||||
2. **Server state copied into useState**: Never `useState` + `useEffect` to sync React Query data into local state. Use query data directly. The only exception is forms where users edit server data.
|
||||
3. **Props mirrored into state**: Never `useState(prop)` + `useEffect(() => setState(prop))`. Use the prop directly, or use a key to reset component state.
|
||||
4. **Chained useEffect state updates**: Never chain Effects that set state to trigger other Effects. Calculate all derived values in the event handler or inline during render.
|
||||
5. **Storing objects when an ID suffices**: Store `selectedId` not a copy of the selected object. Derive the object: `items.find(i => i.id === selectedId)`.
|
||||
6. **State that duplicates Zustand or React Query**: If the data already lives in a store or query cache, don't create a parallel useState.
|
||||
|
||||
## Steps
|
||||
|
||||
1. Read the references above to understand the guidelines
|
||||
2. Analyze the specified scope for the anti-patterns listed above
|
||||
3. If fix=true, apply the fixes. If fix=false, propose the fixes without applying.
|
||||
@@ -0,0 +1,50 @@
|
||||
---
|
||||
description: Analyze and fix URL/query-param state anti-patterns — manual useSearchParams reads, hand-built query mutations, view-state trapped in useState, and objects in the URL
|
||||
argument-hint: "[scope] [fix=true|false]"
|
||||
---
|
||||
|
||||
# You Might Not Need URL State
|
||||
|
||||
Arguments:
|
||||
- scope: what to analyze (default: your current changes). Examples: "diff to main", "PR #123", "app/workspace/[workspaceId]/tables/", "whole codebase"
|
||||
- fix: whether to apply fixes (default: true). Set to false to only propose changes.
|
||||
|
||||
User arguments: $ARGUMENTS
|
||||
|
||||
## Context
|
||||
|
||||
Shareable client view-state (active tab/panel, filters, search query, sort, pagination, selected-entity id, an open "view" modal/drawer that is a destination) lives in the URL via [`nuqs`](https://nuqs.dev) — driven by a co-located `search-params.ts`, never read via `useSearchParams().get(...)` and never mutated by hand-built query strings. Remote data stays in React Query; high-frequency / large / ephemeral / socket-synced state stays in Zustand; purely local UI stays in `useState`.
|
||||
|
||||
Shared helpers own the two repeated wirings — never hand-roll them inline:
|
||||
- Sort: `createSortParams` from `@/lib/url-state` (in `search-params.ts`) + `useUrlSort` from `@/hooks/use-url-sort` (in the component) — defaulted mode for lists with a fixed default ordering, nullable mode when "no active sort" is distinct from the default column.
|
||||
- Debounced search: `useDebouncedSearchSetter` from `@/hooks/use-debounced-search-setter` (grouped or single-param); settings list search boxes use `useSettingsSearch()` from `settings/components/use-settings-search`. Never write a trimmed value to a param that controls the input — trim on read.
|
||||
|
||||
`.claude/rules/sim-url-state.md` is the source of truth — read it first.
|
||||
|
||||
## References
|
||||
|
||||
Read these before analyzing:
|
||||
1. `.claude/rules/sim-url-state.md` — the decision framework, conventions, debounced-input pattern, sort convention, selected-entity deep-link pattern, and the workflow-editor carve-out
|
||||
2. https://nuqs.dev/docs/parsers — parsers (`parseAsString`/`parseAsInteger`/`parseAsBoolean`/`parseAsStringLiteral`/`parseAsArrayOf`/`createParser`)
|
||||
3. https://nuqs.dev/docs/options — `withDefault`, `history`, `shallow`, `clearOnDefault`
|
||||
4. https://nuqs.dev/docs/server-side — `createSearchParamsCache` for server reads
|
||||
|
||||
## Anti-patterns to detect
|
||||
|
||||
1. **Manual param reads for state**: `useSearchParams().get(...)` or `new URLSearchParams(window.location.search)` used to *read* view-state. Replace with `useQueryState`/`useQueryStates` bound to a `search-params.ts`. (Read-once auth/invite/redirect tokens — `token`, `callbackUrl`, `redirect`, `error`, `invite_flow`, `code` — are NOT view-state; leave them on `useSearchParams`.)
|
||||
2. **Hand-built query mutation**: constructing a query string + `router.replace`/`router.push` to change a param on the current path. Use a nuqs setter. (A `router.push` that changes the route *path* is fine; an outbound `new URLSearchParams` building an `href`/`window.open`/download/API URL is fine.)
|
||||
3. **`window.history.replaceState`/`pushState`** to mutate a param.
|
||||
4. **URL state duplicated into a store/useState + synced with an effect** (or a `popstate` listener). The URL is the single source of truth; derive from it, don't mirror it.
|
||||
5. **Objects in the URL**: serializing a `TableDefinition`/`SkillDefinition`/etc. Store the id and derive the object from the loaded list (`items.find(i => i.id === id)`).
|
||||
6. **High-frequency / large state in the URL**: cursor, pan/zoom, un-debounced keystrokes, big JSON blobs. Debounce text search via `useDebouncedSearchSetter` (never a local `useState` mirror + reconcile effect, and never inline `limitUrlUpdates` wiring); keep canvas/presence/resize state in Zustand.
|
||||
7. **Shareable view-state trapped in `useState`**: a tab/filter/sort/pagination/selected-entity that should be a link but lives in local state. Migrate it to the URL.
|
||||
8. **Missing Suspense boundary**: a component newly calling `useQueryState`/`useQueryStates` whose page entry has no `<Suspense>` wrapper (Next.js requires it for `useSearchParams`). Add one with a real-chrome fallback.
|
||||
9. **`import { z }` for param validation in client code**: use nuqs parsers instead.
|
||||
10. **Re-implemented shared wiring**: a hand-rolled `SORT_DIRECTIONS`/default-sort constants/`activeSort` derivation instead of `createSortParams` + `useUrlSort`, or an inline debounced-search setter instead of `useDebouncedSearchSetter`/`useSettingsSearch`.
|
||||
|
||||
## Steps
|
||||
|
||||
1. Read `.claude/rules/sim-url-state.md` and the nuqs docs above to understand the guidelines
|
||||
2. Analyze the specified scope for the anti-patterns listed above
|
||||
3. For each finding, decide the correct home using the decision table — do not force URL state onto ephemeral/high-frequency/socket-synced state
|
||||
4. If fix=true, apply the fixes (co-locate a `search-params.ts`, wire `useQueryState(s)` — sort via `createSortParams` + `useUrlSort`, search via `useDebouncedSearchSetter` — add the Suspense boundary, delete the replaced state + sync effects). If fix=false, propose the fixes without applying.
|
||||
@@ -0,0 +1,73 @@
|
||||
# Sim — Language & Positioning
|
||||
|
||||
When editing user-facing copy (landing pages, docs, metadata, marketing), follow these rules.
|
||||
|
||||
## Identity
|
||||
|
||||
Sim is the **AI workspace** where teams build and run AI agents. Not a workflow tool, not an agent framework, not an automation platform.
|
||||
|
||||
**Short definition:** Sim is the open-source AI workspace where teams build, deploy, and manage AI agents.
|
||||
|
||||
**Full definition:** Sim is the open-source AI workspace where teams build, deploy, and manage AI agents. Connect 1,000+ integrations and every major LLM to create agents that automate real work — visually, conversationally, or with code.
|
||||
|
||||
## Audience
|
||||
|
||||
**Primary:** Teams building AI agents for their organization — IT, operations, and technical teams who need governance, security, lifecycle management, and collaboration.
|
||||
|
||||
**Secondary:** Individual builders and developers who care about speed, flexibility, and open source.
|
||||
|
||||
## Required Language
|
||||
|
||||
| Concept | Use | Never use |
|
||||
|---------|-----|-----------|
|
||||
| The product | "AI workspace" | "workflow tool", "automation platform", "agent framework" |
|
||||
| Building | "build agents", "create agents" | "create workflows" (unless describing the workflow module specifically) |
|
||||
| Visual builder | "workflow builder" or "visual builder" | "canvas", "graph editor" |
|
||||
| The agent | "Sim" — you talk to Sim | "Mothership", "copilot", "AI assistant" |
|
||||
| The chat surface | "Chat" (capitalized, the module) | "Mothership", "copilot" |
|
||||
| Deployment | "deploy", "ship" | "publish", "activate" |
|
||||
| Audience | "teams", "builders" | "users", "customers" (in marketing copy) |
|
||||
| What agents do | "automate real work" | "automate tasks", "automate workflows" |
|
||||
| Our advantage | "open-source AI workspace" | "open-source platform" |
|
||||
|
||||
## Tone
|
||||
|
||||
- **Direct.** Short sentences. Active voice. Lead with what it does.
|
||||
- **Concrete.** Name specific things — "Slack bots, compliance agents, data pipelines" — not abstractions.
|
||||
- **Confident, not loud.** No exclamation marks or superlatives.
|
||||
- **Simple.** If a 16-year-old can't understand the sentence, rewrite it.
|
||||
|
||||
## Claim Hierarchy
|
||||
|
||||
When describing Sim, always lead with the most differentiated claim:
|
||||
|
||||
1. **What it is:** "The AI workspace for teams"
|
||||
2. **What you do:** "Build, deploy, and manage AI agents"
|
||||
3. **How:** "Visually, conversationally, or with code"
|
||||
4. **Scale:** "1,000+ integrations, every major LLM"
|
||||
5. **Trust:** "Open source. SOC2. Trusted by 100,000+ builders."
|
||||
|
||||
## Module Descriptions
|
||||
|
||||
| Module | One-liner |
|
||||
|--------|-----------|
|
||||
| **Chat** | Your AI command center. Talk to Sim — build and manage everything in natural language. |
|
||||
| **Workflows** | The visual builder. Connect blocks, models, and integrations into agent logic. |
|
||||
| **Knowledge Base** | Your agents' memory. Upload docs, sync sources, build vector databases. |
|
||||
| **Tables** | A database, built in. Store, query, and wire structured data into agent runs. |
|
||||
| **Files** | Upload, create, and share. One store for your team and every agent. |
|
||||
| **Logs** | Full visibility, every run. Trace execution block by block. |
|
||||
|
||||
## What We Never Say
|
||||
|
||||
- Never call Sim "just a workflow tool"
|
||||
- Never compare only on integration count — we win on AI-native capabilities
|
||||
- Never use "no-code" as the primary descriptor — say "visually, conversationally, or with code"
|
||||
- Never promise unshipped features
|
||||
- Never use jargon ("RAG", "vector database", "MCP") without plain-English explanation on public pages
|
||||
- Avoid "agentic workforce" as a primary term — use "AI agents"
|
||||
- Never say "Mothership" or "copilot" — the agent is "Sim", the surface is "Chat" (in run logs the trigger reads "Sim agent")
|
||||
|
||||
## Vision
|
||||
|
||||
Sim becomes the default environment where teams build AI agents — not a tool you visit for one task, but a workspace you live in. Workflows are one module; Chat is another. The workspace is the constant; the interface adapts.
|
||||
@@ -0,0 +1,45 @@
|
||||
---
|
||||
paths:
|
||||
- "packages/emcn/**"
|
||||
---
|
||||
|
||||
# EMCN Components
|
||||
|
||||
Import components, `cn`, and tokens from the `@sim/emcn` barrel; icons come from the `@sim/emcn/icons` subpath, and CSS modules from their file path. Never deep-import other component subpaths. The **chip family** is the platform's primary chrome — always reach for it over the legacy primitives it is progressively replacing (`Input`→`ChipInput`, `Textarea`→`ChipTextarea`, `Modal`→`ChipModal`, `Select`/`Combobox`→`ChipSelect`/`ChipCombobox`/`ChipDropdown`, `Switch`→`ChipSwitch`, date field→`ChipDatePicker`). For context/action menus the canonical control is `DropdownMenu` — the standard menu (not a chip, and never a hand-rolled popover).
|
||||
|
||||
## Chip chrome — single source of truth
|
||||
|
||||
Never hand-roll the chip pill from raw class strings (they go stale). Compose from the canonical sources:
|
||||
|
||||
- **Surface, typography + content tokens:** `chip/chip-chrome.ts` — `chipFilledSurfaceTokens`, `chipFieldSurfaceClass`, `chipFieldTextClass` (text fields and the dropdown search box build on these), plus the chip-content chrome `chipContentGap`, `chipGeometryClass`, `chipContentIconClass`, `chipContentLabelClass`, and `cellIconNodeClass` (non-chip surfaces that must visually match chip content, e.g. resource table cells). All are re-exported from the `@sim/emcn` barrel — no subpath import needed.
|
||||
- **Pill geometry:** `chip/chip.tsx` — `chipVariants` (30px tall, `rounded-lg`, `px-2`, icon↔text `gap-1.5`). Every pill-shaped trigger (`ChipDropdown`, `ChipSelect`, `ChipSwitch`) reuses it for visual parity.
|
||||
|
||||
Canonical look: normal font-weight (never `font-medium`/`font-semibold`), value text `--text-body`, icons `--text-icon` at `size-[14px]`, placeholder `--text-muted`, `transition-colors`, **no focus ring** (the caret marks focus). Filled surface is `--surface-5` light / `--surface-4` dark with a `--border-1` border.
|
||||
|
||||
The menu surface intentionally diverges from the pill: `dropdown-menu.tsx` items use `text-small` and `gap-2` (a menu convention, not the chip pill). Keep them distinct.
|
||||
|
||||
## Component catalogue
|
||||
|
||||
- **`Chip` / `ChipLink`** — the pill button (`<button>` / Next `<Link>`). Variants: `primary`, `destructive`, `border-shadow`, `border`; the bare chip is implicit (omit `variant`). `filled` is deliberately NOT a `Chip` variant — it is reserved for chip fields/triggers. For a selected/toggle chip use the `active` prop, never a variant. `leftIcon`/`rightIcon`, `active`, `fullWidth`. Chips carry **no outer margin** — space between them is the parent's `gap`. The old `mx-0.5` default and its `flush` opt-out are gone; do not reintroduce either, and never add a margin to a chip through `className`.
|
||||
- **`ChipInput`** — single-line text field. `icon`, `endAdornment`, `error`, `inputClassName` (inner `<input>`); `className` styles the chrome wrapper.
|
||||
- **`ChipCopyInput`** — the canonical view-only field: a read-only `ChipInput` at full opacity with a trailing copy-to-clipboard button. View-only is a display mode, not a disabled state — reach for it (or `ChipModalField type='copy'`) over a `disabled` (greyed) input for values the user cannot edit.
|
||||
- **`ChipTextarea`** — multi-line sibling. `error`, `resizable` (off by default), `viewOnly` (read-only at full opacity with the default cursor — the multi-line counterpart of `ChipCopyInput`).
|
||||
- **`ChipDropdown`** — pill that opens a menu. Single OR multi-select via the discriminated `multiple` prop (one component, not two). Owns its trailing chevron — no `rightIcon`.
|
||||
- **`ChipSelect` / `ChipCombobox`** — `Combobox`-backed pickers with search, groups, multi-select; for richer lists than `ChipDropdown`.
|
||||
- **`ChipModal` + `ChipModalField`** — declarative compact modal. The field's `type` (`input` | `email` | `textarea` | `dropdown` | `copy` | `file` | `emails` | `custom`) picks the control and **owns all chrome** — consumers describe intent, never pass `variant`/`className`/`id` to the inner control. `custom` is the escape hatch. **Every body field MUST be a `ChipModalField`** — never hand-roll a field row (raw `<div>` + hand-rolled `<p>`/`<label>` title + bare `ChipInput`/`ChipTextarea`). `ChipModalBody` applies `px-2` + `gap-4`; `ChipModalField` adds another `px-2`, so each field lands at effective `px-4`, exactly matching the `px-4` header/footer — a hand-rolled row skips that gutter and sits misaligned at `px-2`. For controls the field doesn't cover (`ChipCombobox`, `ChipSelect`, `DatePicker`, `TimePicker`, `ButtonGroup`, arbitrary JSX), use `type='custom'` with a `title` — it still applies the gutter and renders the canonical `Label`.
|
||||
- **`ChipSwitch`** — segmented pill control (built from `chipVariants`).
|
||||
- **`ChipTag`** — 20px inline tag/badge (`mono`/`gray`/`invite`), not a pill trigger.
|
||||
- **`ChipDatePicker`** — chip-styled date field.
|
||||
- **`ChipTimePicker`** — minute-granular time sibling of `ChipDatePicker`, a `ChipInput` that leniently parses typed input (`9:47`, `947`, `2:05pm`, `14:30`), commits on Enter/blur, and re-renders the canonical `9:47 AM` label.
|
||||
- **`DropdownMenu`** — the canonical context/action menu (Radix-backed). Not a chip, but the standard menu for command/action lists; reach for it instead of a hand-rolled popover. Its surface intentionally diverges from the chip pill (`text-small`, `gap-2`) — keep them distinct. For a pill that opens a value picker, use `ChipDropdown`/`ChipSelect` instead.
|
||||
|
||||
## Authoring principles
|
||||
|
||||
- **One source of truth for shared chrome.** Compose from `chip-chrome.ts` / `chipVariants`; never duplicate the chrome string.
|
||||
- **`cn()` for a single state toggle, CVA for genuine multiple variants.** A lone `error` boolean is `cn()`, not a CVA variant.
|
||||
- **Discriminated-union props for modes** (e.g. `multiple`, the modal field `type`) instead of near-duplicate components.
|
||||
- **Delete legacy variants after migration** — don't leave dead paths (this paradigm removed `Input variant='chip'` and `ChipMultiSelect`).
|
||||
- **Verify CSS vars exist.** An undefined var resolves to `currentColor` (caused a real black-border bug). Align to the canonical tokens: normal weight, `--text-body`, `--text-icon`.
|
||||
- Use Radix UI primitives for accessibility. Export the component and its `variants` (when using CVA). Document with TSDoc + a usage example.
|
||||
|
||||
Color tokens and icon-size conventions are canonical in `.claude/rules/sim-styling.md` — follow it rather than restating.
|
||||
@@ -0,0 +1,74 @@
|
||||
# Global Standards
|
||||
|
||||
## Logging
|
||||
Import `createLogger` from `@sim/logger`. Use `logger.info`, `logger.warn`, `logger.error` instead of `console.log`. Inside API routes wrapped with `withRouteHandler`, loggers automatically include the request ID.
|
||||
|
||||
## API Route Handlers
|
||||
All API route handlers must be wrapped with `withRouteHandler` from `@/lib/core/utils/with-route-handler`. Never export a bare `async function GET/POST/...` — always use `export const METHOD = withRouteHandler(...)`.
|
||||
|
||||
## Comments
|
||||
Use TSDoc for documentation. No `====` separators. No non-TSDoc comments.
|
||||
|
||||
## Styling
|
||||
Never update global styles. Keep all styling local to components.
|
||||
|
||||
## ID Generation
|
||||
Never use `crypto.randomUUID()`, `nanoid`, or the `uuid` package directly. Use the utilities from `@sim/utils/id`:
|
||||
|
||||
- `generateId()` — UUID v4, use by default
|
||||
- `generateShortId(size?)` — short URL-safe ID (default 21 chars), for compact identifiers
|
||||
|
||||
Both use `crypto.getRandomValues()` under the hood and work in all contexts including non-secure (HTTP) browsers.
|
||||
|
||||
```typescript
|
||||
// ✗ Bad
|
||||
import { nanoid } from 'nanoid'
|
||||
import { v4 as uuidv4 } from 'uuid'
|
||||
const id = crypto.randomUUID()
|
||||
|
||||
// ✓ Good
|
||||
import { generateId, generateShortId } from '@sim/utils/id'
|
||||
const uuid = generateId()
|
||||
const shortId = generateShortId()
|
||||
const tiny = generateShortId(8)
|
||||
```
|
||||
|
||||
## Common Utilities
|
||||
Use shared helpers from `@sim/utils` instead of writing inline implementations:
|
||||
|
||||
- `sleep(ms)` from `@sim/utils/helpers` — async delay. Never write `new Promise(resolve => setTimeout(resolve, ms))`
|
||||
- `toError(value)` from `@sim/utils/errors` — normalize unknown caught values to `Error`. Never write `e instanceof Error ? e : new Error(String(e))`
|
||||
- `getErrorMessage(value, fallback?)` from `@sim/utils/errors` — extract error message string. Never write `e instanceof Error ? e.message : 'fallback'`
|
||||
- `structuredClone(value)` — built-in deep clone, no import needed. Never write `JSON.parse(JSON.stringify(obj))`
|
||||
- `omit(obj, keys)` from `@sim/utils/object` — remove keys from object
|
||||
- `filterUndefined(obj)` from `@sim/utils/object` — strip undefined-valued keys. Never write `Object.fromEntries(Object.entries(obj).filter(([, v]) => v !== undefined))`
|
||||
- `truncate(str, maxLength, suffix?)` from `@sim/utils/string` — safe string truncation with ellipsis
|
||||
- `backoffWithJitter(attempt, retryAfterMs, options?)` from `@sim/utils/retry` — exponential backoff with jitter
|
||||
- `parseRetryAfter(header)` from `@sim/utils/retry` — parse HTTP `Retry-After` header to milliseconds
|
||||
|
||||
```typescript
|
||||
// ✗ Bad
|
||||
await new Promise(resolve => setTimeout(resolve, 1000))
|
||||
const msg = error instanceof Error ? error.message : 'Unknown error'
|
||||
const clone = JSON.parse(JSON.stringify(obj))
|
||||
const filtered = Object.fromEntries(Object.entries(obj).filter(([, v]) => v !== undefined))
|
||||
|
||||
// ✓ Good
|
||||
import { sleep } from '@sim/utils/helpers'
|
||||
import { getErrorMessage, toError } from '@sim/utils/errors'
|
||||
import { filterUndefined } from '@sim/utils/object'
|
||||
await sleep(1000)
|
||||
const msg = getErrorMessage(error, 'Unknown error')
|
||||
const clone = structuredClone(obj)
|
||||
const filtered = filterUndefined(obj)
|
||||
```
|
||||
|
||||
## Package Manager
|
||||
Use `bun` and `bunx`, not `npm` and `npx`.
|
||||
|
||||
## Type-checking
|
||||
`tsc` must resolve to the native (Go) TypeScript 7 compiler. Do not remove the `@typescript/native` alias from the root `devDependencies` — nothing imports it, and deleting it looks harmless.
|
||||
|
||||
`apps/sim` needs `@typescript/typescript6` for its runtime TypeScript API, and that package depends on `@typescript/old` — an alias of `typescript@6` — which declares its own `tsc` bin. Package managers pick bin winners by lexical sort rather than dependency depth, so `@typescript/old` beats `typescript` and `node_modules/.bin/tsc` silently becomes the JavaScript TypeScript 6 compiler: identical diagnostics, ~10x slower (83s vs 8s on `apps/sim`). The `@typescript/native` alias exists only to sort ahead of `@typescript/old`.
|
||||
|
||||
`bun run check:native-typecheck` fails the build if a bare `tsc` stops reporting 7.x — which is also what a newly added package that sorts ahead of `@typescript/native` and ships a `tsc` bin would look like. See [microsoft/typescript-go#4567](https://github.com/microsoft/typescript-go/issues/4567).
|
||||
@@ -0,0 +1,46 @@
|
||||
---
|
||||
paths:
|
||||
- "apps/sim/app/(landing)/**/*.tsx"
|
||||
- "apps/sim/content/**/*.mdx"
|
||||
---
|
||||
|
||||
# Landing Page — SEO / GEO
|
||||
|
||||
## SEO
|
||||
|
||||
- One `<h1>` per page, in Hero only — never add another.
|
||||
- Strict heading hierarchy: H1 (Hero) → H2 (section titles) → H3 (feature names).
|
||||
- Every section: `<section id="…" aria-labelledby="…-heading">`.
|
||||
- Decorative/animated elements: `aria-hidden="true"`.
|
||||
- All internal routes use Next.js `<Link>` (crawlable). External links get `rel="noopener noreferrer"`.
|
||||
- Navbar is a Server Component (no `'use client'`) for immediate crawlability. Logo `<Image>` has `priority` (LCP element).
|
||||
- Navbar `<nav>` carries `SiteNavigationElement` schema.org markup.
|
||||
- Feature lists must stay in sync with `WebApplication.featureList` in `structured-data.tsx`.
|
||||
|
||||
## GEO (Generative Engine Optimisation)
|
||||
|
||||
- **Answer-first pattern**: each section's H2 + subtitle should directly answer a user question (e.g. "What is Sim?", "How fast can I deploy?").
|
||||
- **Atomic answer blocks**: each feature / template card should be independently extractable by an AI summariser.
|
||||
- **Entity consistency**: always write "Sim" by name — never "the platform" or "our tool".
|
||||
- **Keyword density**: first 150 visible chars of Hero must name "AI workspace" and "AI agents". "Sim" is carried by the title tag, the meta description, and the Hero `sr-only` summary — the H1 does not have to spend its opening words on the brand.
|
||||
- **sr-only summaries**: Hero and Templates each have a `<p className="sr-only">` (~50 words) as an atomic product/catalog summary for AI citation.
|
||||
- **Specific numbers**: prefer concrete figures ("1,000+ integrations", "15+ AI providers") over vague claims.
|
||||
|
||||
## Citations and linking (`/library`, `/blog`, `/comparisons`)
|
||||
|
||||
The Princeton GEO study (Aggarwal et al., KDD 2024, [arXiv:2311.09735](https://arxiv.org/abs/2311.09735)) found that adding citations, quotations, and statistics were the three strongest of nine tested tactics, worth 30–40% relative lifts in AI-answer visibility. Sourcing is also what makes a claim checkable by a human reader.
|
||||
|
||||
- **Every third-party factual claim carries an outbound source link.** Pricing, rate limits, feature availability, licensing, compliance certifications — link the primary source (the vendor's own pricing page, docs, changelog, or license file), not a secondary blog. External links get `rel="noopener noreferrer"`.
|
||||
- **Prefer the primary source over a roundup.** Citing another vendor's comparison post to substantiate a fact about them is second-hand and ages badly.
|
||||
- **Internal links: 3–5 per library post**, pointing at genuinely related library entries, using real `href`s (Next `<Link>` in TSX; a plain markdown link in MDX). A post with zero internal links is a dead end for crawlers and readers alike.
|
||||
- **Never fabricate a citation.** An unlinked claim is better than a link that does not substantiate it. If a number cannot be sourced, cut the number.
|
||||
|
||||
## Freshness
|
||||
|
||||
Answer engines weight recency to avoid repeating stale facts, and a reader deciding whether to trust a pricing comparison wants to know when it was last checked. The vendor-published "fresh content earns Nx more citations" figures are directional, not measured — the reason to do this is that both signals must agree and both must be real.
|
||||
|
||||
- **Emit `dateModified`** in the page's structured data (JSON-LD or microdata), and emit it exactly once per document.
|
||||
- **Show the same date to the reader.** `/comparisons/[provider]` renders "Last verified …" from `getLatestVerifiedDate()`; `/library` and `/blog` posts render "Updated …" next to the publish date. A date that exists only in metadata is invisible to a reader deciding whether to trust the page.
|
||||
- **Only surface a modified date when it differs from the publish date** — an "Updated" label on the publish day is noise.
|
||||
- **Bump the date only on a substantive edit.** Touching frontmatter without changing the content is date-washing; it degrades the signal for every other page on the domain.
|
||||
- **Comparison facts are dated at the fact level.** Every `Fact` in `apps/sim/lib/compare/data` carries `sources: [{ url, label, asOf }]`. Re-checking a fact means updating its `asOf`, which flows through `getLatestVerifiedDate()` to the visible date, the JSON-LD, and the sitemap.
|
||||
@@ -0,0 +1,71 @@
|
||||
---
|
||||
paths:
|
||||
- "apps/sim/**"
|
||||
---
|
||||
|
||||
# Sim App Architecture
|
||||
|
||||
## Core Principles
|
||||
1. **Single Responsibility**: Each component, hook, store has one clear purpose
|
||||
2. **Composition Over Complexity**: Break down complex logic into smaller pieces
|
||||
3. **Type Safety First**: TypeScript interfaces for all props, state, return types
|
||||
4. **Predictable State**: Zustand for global state, useState for UI-only concerns
|
||||
|
||||
## Root-Level Structure
|
||||
|
||||
```
|
||||
apps/
|
||||
├── sim/ # this app (Next.js: UI + API routes + workflow editor)
|
||||
│ ├── app/ # Next.js app router (pages, API routes)
|
||||
│ ├── blocks/ # Block definitions and registry
|
||||
│ ├── components/ # Shared UI (emcn/, ui/)
|
||||
│ ├── executor/ # Workflow execution engine
|
||||
│ ├── hooks/ # Shared hooks (queries/, selectors/)
|
||||
│ ├── lib/ # App-wide utilities
|
||||
│ ├── providers/ # LLM provider integrations
|
||||
│ ├── stores/ # Zustand stores
|
||||
│ ├── tools/ # Tool definitions
|
||||
│ └── triggers/ # Trigger definitions
|
||||
└── realtime/ # Bun Socket.IO server (collaborative canvas)
|
||||
|
||||
packages/ # @sim/* — audit, auth, db, logger, realtime-protocol,
|
||||
# security, tsconfig, utils, platform-authz,
|
||||
# workflow-persistence, workflow-types
|
||||
```
|
||||
|
||||
## Package Boundaries
|
||||
|
||||
- `apps/* → packages/*` only. Packages never import from `apps/*`.
|
||||
- `apps/realtime` avoids Next.js, React, the block/tool registry, provider SDKs, and the executor; never add `@/lib/webhooks/providers/*`, `@/executor/*`, `@/blocks/*`, or `@/tools/*` imports to any package it consumes. CI enforces this via `scripts/check-monorepo-boundaries.ts` and `scripts/check-realtime-prune-graph.ts`.
|
||||
|
||||
## The `'use client'` server boundary
|
||||
|
||||
Every export of a `'use client'` module becomes a *client reference* on the server — server-evaluated code (RSC pages/layouts, `prefetch.ts`, route handlers, block definitions, triggers) can only *render* it as a component or pass it as a prop, never *call* it (doing so throws at runtime, e.g. `tableKeys.list is not a function`; `next build` does not catch it). Keep server-importable query primitives (key factories, fetchers, mappers, constants) in non-`'use client'` modules — see `.claude/rules/sim-queries.md`. Enforced by `scripts/check-client-boundary-imports.ts`.
|
||||
|
||||
## Feature Organization
|
||||
|
||||
Features live under `app/workspace/[workspaceId]/`:
|
||||
|
||||
```
|
||||
feature/
|
||||
├── components/ # Feature components
|
||||
├── hooks/ # Feature-scoped hooks
|
||||
├── utils/ # Feature-scoped utilities (2+ consumers)
|
||||
├── feature.tsx # Main component
|
||||
└── page.tsx # Next.js page entry
|
||||
```
|
||||
|
||||
## Naming Conventions
|
||||
- **Components**: PascalCase (`WorkflowList`)
|
||||
- **Hooks**: `use` prefix (`useWorkflowOperations`)
|
||||
- **Files**: kebab-case (`workflow-list.tsx`)
|
||||
- **Stores**: `stores/feature/store.ts`
|
||||
- **Constants**: SCREAMING_SNAKE_CASE
|
||||
- **Interfaces**: PascalCase with suffix (`WorkflowListProps`)
|
||||
|
||||
## Utils Rules
|
||||
|
||||
- **Never create `utils.ts` for single consumer** - inline it
|
||||
- **Create `utils.ts` when** 2+ files need the same helper
|
||||
- **Check existing sources** before duplicating (`lib/` has many utilities)
|
||||
- **Location**: `lib/` (app-wide) → `feature/utils/` (feature-scoped) → inline (single-use)
|
||||
@@ -0,0 +1,33 @@
|
||||
---
|
||||
paths:
|
||||
- "apps/sim/**/*.tsx"
|
||||
---
|
||||
|
||||
# Component Patterns
|
||||
|
||||
Component authoring rules — structure order (refs → external hooks → store hooks → custom hooks → state → useMemo → useCallback → useEffect → return), required props interface, and extraction thresholds (50+ lines / 2+ files vs keep inline < 10 lines) — live in CLAUDE.md > Components.
|
||||
|
||||
`.tsx`-specific deltas not covered there:
|
||||
|
||||
- `'use client'` only when using React hooks or browser-only APIs.
|
||||
- Prefer semantic HTML (`aside`, `nav`, `article`).
|
||||
- Optional-chain callbacks: `onAction?.(id)`.
|
||||
|
||||
## List-render performance
|
||||
|
||||
When rendering or sorting a list of rows against a lookup collection (members, folders, tags), keep the per-row work O(1):
|
||||
|
||||
- **Precompute a lookup `Map` once**, never `array.find(...)` per row. Build `const byId = useMemo(() => { const m = new Map<string, T>(); for (const x of items ?? []) m.set(x.id, x); return m }, [items])` and read `byId.get(id)` in the sort comparator, `.map(...)`, and cell builders. A `.find` inside a sort comparator is O(n²·log n) — the worst offender. Depend memos on the derived `Map`, not the raw array.
|
||||
- **Sort with `[...array].sort(cmp)` in client code — NOT `array.toSorted(cmp)`.** SWC (Next's compiler) transforms syntax but does not polyfill prototype methods, and the repo sets no core-js/browserslist target, so `toSorted`/`toReversed`/`toSpliced`/`Array.prototype.with` ship as-is and throw `TypeError` on Safari <16 / iOS 15 (they landed in Safari 16). `tsconfig` `lib: ES2023` only affects type-checking, never the runtime. The `[...arr].sort()` spread copy is the intended cost — it keeps the sort non-mutating without an unpolyfilled builtin. These methods are only safe in server-only modules (no `'use client'`, executed on Node ≥20). react-doctor's `js-tosorted-immutable` is therefore a won't-fix in client components.
|
||||
- **Partition in a single pass** — when splitting one collection into several (`fileIds`/`folderIds`), do one `for…of` pushing into each bucket and return `{ a, b }` from a single `useMemo`, not two memos that each `map→filter→map` the same source twice.
|
||||
|
||||
## react-doctor (`npx react-doctor`) — apply the wins, skip the false positives
|
||||
|
||||
react-doctor diagnostics are hypotheses, not verdicts — confirm against the code before acting, and preserve behavior. Known repo-specific false positives to NOT "fix":
|
||||
|
||||
- `no-barrel-import` — barrel imports are the repo convention (see sim-imports.md, "Barrel Exports"). Keep them.
|
||||
- `js-tosorted-immutable` — in `'use client'` code, keep `[...arr].sort(cmp)`; `toSorted` is unpolyfilled and crashes Safari <16 / iOS 15 (see "List-render performance" above). Only apply it in server-only modules.
|
||||
- `rerender-state-only-in-handlers` / "state set but never rendered" — a false positive when the `useState` is consumed by a `useEffect`/`useLayoutEffect` dependency (the effect must re-run on change). Only convert to a ref when nothing reads the value reactively.
|
||||
- `no-render-in-render` — a helper *called inline* (`{renderRow()}`) is reconciled by position and does **not** remount, so extracting it to a component is usually pure churn and can regress behavior (prop-drilling many closures, focus/scroll loss on the inner `<input>`). Apply it only when the helper is genuinely a *component defined during render*, or when the move is mechanical (a stateless, ref-free helper whose closures become a small, explicit prop set).
|
||||
- `async-await-in-loop` on an upload/progress loop where sequential execution is intentional (per-item progress, server backpressure) — leave it.
|
||||
- Broad refactors (`prefer-useReducer` for many `useState`, `no-giant-component` splits) — out of scope for a perf pass; note, don't churn.
|
||||
@@ -0,0 +1,74 @@
|
||||
---
|
||||
paths:
|
||||
- "apps/sim/**/use-*.ts"
|
||||
- "apps/sim/**/hooks/**/*.ts"
|
||||
---
|
||||
|
||||
# Hook Patterns
|
||||
|
||||
## Structure
|
||||
|
||||
For server data, use a React Query hook from `hooks/queries/` — do NOT `useState` + `fetch` here (see `.claude/rules/sim-queries.md`). This pattern is for UI/orchestration hooks that hold UI-only state and wrap callbacks.
|
||||
|
||||
```typescript
|
||||
interface UseFeatureProps {
|
||||
id: string
|
||||
onSelect?: (item: Item) => void
|
||||
}
|
||||
|
||||
export function useFeature({ id, onSelect }: UseFeatureProps) {
|
||||
// 1. Refs for stable dependencies
|
||||
const idRef = useRef(id)
|
||||
const onSelectRef = useRef(onSelect)
|
||||
|
||||
// 2. UI-only state (never server data)
|
||||
const [isOpen, setIsOpen] = useState(false)
|
||||
|
||||
// 3. Sync refs
|
||||
useEffect(() => {
|
||||
idRef.current = id
|
||||
onSelectRef.current = onSelect
|
||||
}, [id, onSelect])
|
||||
|
||||
// 4. Operations (useCallback with empty deps when using refs)
|
||||
const select = useCallback((item: Item) => {
|
||||
onSelectRef.current?.(item)
|
||||
setIsOpen(false)
|
||||
}, [])
|
||||
|
||||
return { isOpen, setIsOpen, select }
|
||||
}
|
||||
```
|
||||
|
||||
## Rules
|
||||
|
||||
1. Single responsibility per hook
|
||||
2. Props interface required
|
||||
3. Refs for stable callback dependencies
|
||||
4. Wrap returned functions in useCallback
|
||||
5. Server data goes through React Query (`hooks/queries/`), never `useState` + `fetch`
|
||||
6. Keep only UI/orchestration state in these hooks
|
||||
|
||||
## State shape
|
||||
|
||||
Never mirror a prop into state with `useState(prop)` + a syncing `useEffect` — a prop change clobbers in-progress local edits. Use the prop directly, reset via a remount `key`, or — when you must seed local state from a prop only on a transition (e.g. a modal opening) — adjust it **during render** with a `prev`-value tracker held in `useState`:
|
||||
|
||||
```typescript
|
||||
const [prevOpen, setPrevOpen] = useState(open)
|
||||
if (prevOpen !== open) {
|
||||
setPrevOpen(open)
|
||||
if (open) setName(initialName) // closed → open only
|
||||
}
|
||||
```
|
||||
|
||||
React re-renders immediately with the corrected state without committing the stale value. Rules: the `if (prev !== current)` guard is mandatory (an unconditional `setState` in render loops forever), the tracker is set **inside** the guard, and you may only set the currently-rendering component's state this way. Hold the tracker in `useState`, **not a `useRef`** — React forbids reading/writing `ref.current` during render (react.dev, useRef → "Do not write _or read_ `ref.current` during rendering"; the `react-hooks` `refs` lint flags it), and a `useState` tracker is concurrent-safe where a mutated ref is not (a discarded render rolls state back, not a ref).
|
||||
|
||||
**The tracker's initial value decides mount behavior — choose it deliberately.** The example seeds `useState(open)` because the modal mounts closed, so the first render's guard is `false` and nothing resets on mount (correct — `name` is already at its initial value). When the effect you're replacing did real work **on mount** — opening a panel because a prop already matches, seeding editable state from an already-present value, or a component that can mount in the active state — seed a **sentinel** the live value can't equal (e.g. `useState<T | null>(null)`), otherwise the guard is `false` on the first render and that mount action is silently dropped. Place the block before any early `return`.
|
||||
|
||||
> Some existing components use a `useRef` prev-tracker (`if (prevRef.current !== x) { prevRef.current = x; … }`). It works but reads/writes a ref during render — prefer the `useState` form above for new code.
|
||||
|
||||
Only convert a `useState` to a `useRef` when the value is **never read during render/JSX and is never a hook dependency** — a value in a `useEffect`/`useMemo`/`useCallback` dep array must re-run the hook on change, so it stays state (see also `rerender-state-only-in-handlers` in `sim-components.md`). Convert only set-only values read solely inside handlers or effect bodies (e.g. a prompt-history index, a pending-upload URL). If a ref feeds render, mutating it won't re-render and the UI goes stale.
|
||||
|
||||
Model mutually-exclusive flags as ONE `status` enum, not several contradictory booleans. `isLoading`/`isVerified`/`isInvalidOtp` describing one machine collapse to `status: 'idle' | 'verifying' | 'verified' | 'error'` (+ `errorMessage`); derive any boolean a consumer still needs (`status === 'error'`).
|
||||
|
||||
Derive busy/success from the mutation object — never duplicate `mutation.isPending`/`mutation.isSuccess` into local `useState`. Read them directly (`mutation.isSuccess`) and reset with `mutation.reset()`. A distinct phase the mutation doesn't cover — e.g. a pre-submit captcha/Turnstile gate that runs before `mutate()` — is not a duplicate; keep that flag.
|
||||
@@ -0,0 +1,76 @@
|
||||
---
|
||||
paths:
|
||||
- "apps/sim/**/*.ts"
|
||||
- "apps/sim/**/*.tsx"
|
||||
---
|
||||
|
||||
# Import Patterns
|
||||
|
||||
## Absolute Imports
|
||||
|
||||
**Always use absolute imports.** Never use relative imports.
|
||||
|
||||
```typescript
|
||||
// ✓ Good
|
||||
import { useWorkflowStore } from '@/stores/workflows/store'
|
||||
import { Button } from '@/components/ui/button'
|
||||
|
||||
// ✗ Bad
|
||||
import { useWorkflowStore } from '../../../stores/workflows/store'
|
||||
```
|
||||
|
||||
## Barrel Exports
|
||||
|
||||
Use barrel exports (`index.ts`) when a folder has 3+ exports. Import from barrel, not individual files.
|
||||
|
||||
```typescript
|
||||
// ✓ Good
|
||||
import { Dashboard, Sidebar } from '@/app/workspace/[workspaceId]/logs/components'
|
||||
|
||||
// ✗ Bad
|
||||
import { Dashboard } from '@/app/workspace/[workspaceId]/logs/components/dashboard/dashboard'
|
||||
```
|
||||
|
||||
## Code-splitting through barrels
|
||||
|
||||
When you `lazy(() => import(...))` a component to keep it out of a route's initial bundle, import the **deep module path** (`./components/foo/foo`), never the barrel — and **delete the now-dead barrel re-export** of that component. This app has no `"sideEffects": false` in `apps/sim/package.json`, so when any sibling still imports that barrel, webpack can conservatively keep the barrel's re-export edge to the heavy module. A leftover `export { Foo } from './foo'` line can therefore drag `Foo` (and its transitive deps) back into the initial chunk and silently defeat the split. Removing the dead re-export is the guaranteed fix; verify with a production bundle diff, not by eyeballing the `lazy()` call.
|
||||
|
||||
```typescript
|
||||
// ✓ Good — deep lazy import + no barrel edge left behind
|
||||
const MothershipView = lazy(() =>
|
||||
import('./components/mothership-view/mothership-view').then((m) => ({ default: m.MothershipView }))
|
||||
)
|
||||
// (and remove `export { MothershipView } from './mothership-view'` from components/index.ts)
|
||||
```
|
||||
|
||||
Wrap the lazy component in a **local `<Suspense>`** so its suspension resolves at the nearest boundary instead of bubbling to the page-level fallback (which would flash the whole route). `React.lazy(memo(forwardRef(...)))` forwards a DOM `ref` correctly in React 19 — but during the fallback window `ref.current` is `null`, so every consumer must guard it (`if (!el) return` / `el?.`).
|
||||
|
||||
## No Re-exports
|
||||
|
||||
Do not re-export from non-barrel files. Import directly from the source.
|
||||
|
||||
```typescript
|
||||
// ✓ Good - import from where it's declared
|
||||
import { CORE_TRIGGER_TYPES } from '@/stores/logs/filters/types'
|
||||
|
||||
// ✗ Bad - re-exporting in utils.ts then importing from there
|
||||
import { CORE_TRIGGER_TYPES } from '@/app/workspace/.../utils'
|
||||
```
|
||||
|
||||
## Import Order
|
||||
|
||||
1. React/core libraries
|
||||
2. External libraries
|
||||
3. UI components (`@sim/emcn`, `@/components/ui`)
|
||||
4. Utilities (`@/lib/...`)
|
||||
5. Stores (`@/stores/...`)
|
||||
6. Feature imports
|
||||
7. CSS imports
|
||||
|
||||
## Type Imports
|
||||
|
||||
Use `type` keyword for type-only imports:
|
||||
|
||||
```typescript
|
||||
import type { WorkflowLog } from '@/stores/logs/types'
|
||||
```
|
||||
@@ -0,0 +1,20 @@
|
||||
---
|
||||
paths:
|
||||
- "apps/sim/tools/**"
|
||||
- "apps/sim/blocks/**"
|
||||
- "apps/sim/triggers/**"
|
||||
---
|
||||
|
||||
# Adding Integrations
|
||||
|
||||
Build order: **Tools** (`tools/{service}/`) → **Block** (`blocks/blocks/{service}.ts`) → **Icon** (`components/icons.tsx`) → optional **Trigger** (`triggers/{service}/`). Always look up the service's API docs first.
|
||||
|
||||
The full authoring instructions — tool/block/icon/trigger scaffolding, SubBlock property tables, `condition`/`required`/`dependsOn`/`mode`/`canonicalParamId` syntax, the `normalizeFileInput`/file-handling helper table, and checklists — live in the skills: `/add-integration` (end-to-end), `/add-tools`, `/add-block`, `/add-trigger`.
|
||||
|
||||
## Hard rules (don't get these wrong)
|
||||
|
||||
- Tool IDs are `snake_case` (`service_action`). Register tools in `tools/registry.ts`, blocks in `blocks/registry-maps.ts` (the `BLOCK_REGISTRY` config map + `BLOCK_META_REGISTRY` catalog-meta map, alphabetically — `blocks/registry.ts` holds only the accessor functions), triggers in `triggers/registry.ts`.
|
||||
- Type coercions (`Number()`, etc.) belong in `tools.config.params` (runs at execution, after variable resolution) — never in `tools.config.tool` (runs at serialization; coercing there destroys dynamic `<Block.output>` references).
|
||||
- `canonicalParamId` must NOT match any subblock's `id`, must be unique **block-wide** (groups are keyed by canonical id across every subblock and hold exactly one `basicId`, so two operations that each need a pair need two different canonical ids), and all subblocks in a canonical group must share the same `required` status. The `inputs` section and the params function reference canonical IDs, not raw subblock IDs — the serializer deletes the subblock IDs and republishes the active member's value under the canonical ID.
|
||||
- A canonical pair carries ONE concept. For files that is upload (basic) + file reference (advanced), as in Gmail attachments (`blocks/blocks/gmail.ts`). Never overload the advanced side with alternate identifiers (URL, provider asset ID) — give those their own subblocks, mark mutually exclusive sources `required: false`, and enforce "exactly one" at execution.
|
||||
- Blocks must also set the catalog/UI metadata fields `integrationType`, `tags`, `authMode`, `docsLink`, and export a `{Service}BlockMeta` — see the `/add-block` skill's BlockMeta section for details.
|
||||
@@ -0,0 +1,76 @@
|
||||
---
|
||||
paths:
|
||||
- "apps/sim/app/**/*.tsx"
|
||||
- "apps/sim/ee/**/*.tsx"
|
||||
- "apps/sim/components/**/*.tsx"
|
||||
---
|
||||
|
||||
# List & Menu Ordering
|
||||
|
||||
**A list orders itself the way the user already reads the same things somewhere else.** Dropdowns, context menus, tab strips, command palettes, and settings navs are all *second* presentations of a set the user has already seen — in the sidebar, in a toolbar, in a column-header row. When the second presentation reorders that set, the user re-reads it from scratch every time.
|
||||
|
||||
This is not a style preference. Order is the cheapest affordance a list has, and the only one that costs nothing to get right.
|
||||
|
||||
## The rule
|
||||
|
||||
Before writing a list of items, find where the user sees those same items *first*. That surface owns the order; your list mirrors it.
|
||||
|
||||
| The list | Mirrors |
|
||||
| --- | --- |
|
||||
| Resource menus (`+` attach, `@` mention, resource-tab `+`) | the workspace **sidebar**, top-down |
|
||||
| A row / root **context menu** | that surface's **toolbar**, left-to-right → top-to-bottom |
|
||||
| Settings tab strip, recently-deleted tabs | the **settings nav**, top-down |
|
||||
| A "New …" menu | the order those things appear once created |
|
||||
|
||||
Left-to-right becomes top-to-bottom. A toolbar reading `Filter · Sort · Export · Delete` becomes a menu reading Filter, Sort, Export, Delete — never alphabetized, never grouped by implementation, never "destructive last" unless the toolbar already puts it last.
|
||||
|
||||
Platform-only entries (desktop **Browser** and **Terminal**) trail the shared set rather than interleaving, so the common prefix is identical on every platform.
|
||||
|
||||
## Encode the order once
|
||||
|
||||
An order duplicated across surfaces is an order that will drift. Export **one** constant and sort by it — do not hand-maintain a matching literal per menu.
|
||||
|
||||
```ts
|
||||
/** Top-down order for every menu listing resource families, mirroring the sidebar. */
|
||||
export const RESOURCE_MENU_ORDER: readonly MothershipResourceType[] = [
|
||||
'integration', 'task', 'table', 'file', 'filefolder',
|
||||
'knowledgebase', 'log', 'workflow', 'folder', 'browser', 'terminal', 'generic',
|
||||
]
|
||||
|
||||
export function byResourceMenuOrder<T extends { type: MothershipResourceType }>(a: T, b: T) {
|
||||
return RESOURCE_MENU_ORDER.indexOf(a.type) - RESOURCE_MENU_ORDER.indexOf(b.type)
|
||||
}
|
||||
```
|
||||
|
||||
Canonical instance: `app/workspace/[workspaceId]/home/components/mothership-view/components/resource-registry/resource-registry.tsx`, consumed by `useAvailableResources` and `ResourceMenuSections`.
|
||||
|
||||
## Render kinds in one pass, not one phase per kind
|
||||
|
||||
The most common way a canonical order gets silently defeated: emitting all items of one *kind* and then all of another. Every submenu-backed family lands above every flat family regardless of what the order constant says.
|
||||
|
||||
```tsx
|
||||
// ✗ Bad — two phases; the trees always pin to the top
|
||||
<ResourceTreeSections sections={treeSections} />
|
||||
{groups.filter((g) => !FOLDERED.has(g.type)).map(renderFlat)}
|
||||
|
||||
// ✓ Good — one ordered pass; each entry picks its own rendering
|
||||
{entries.sort(byResourceMenuOrder).map((entry) =>
|
||||
sectionByType.has(entry.type) ? renderTree(entry) : renderFlat(entry)
|
||||
)}
|
||||
```
|
||||
|
||||
The same trap appears as "render the pinned ones, then the rest", "render enabled, then disabled", and "render the groups, then the loose items".
|
||||
|
||||
## When order may diverge
|
||||
|
||||
Only for reasons the user can perceive:
|
||||
|
||||
- **Search/filter results** rank by match quality — the whole point is that ranking beats position.
|
||||
- **User-controlled ordering** (drag-to-reorder, manual `sortOrder`) wins over any canonical order.
|
||||
- **Recency lists** ("Recent chats") order by time, which *is* the order the user reads them elsewhere.
|
||||
|
||||
"Grouped by which hook provides it", "alphabetical because it was easy", and "that's the order the array was built in" are not reasons.
|
||||
|
||||
## Reviewing
|
||||
|
||||
When a diff adds or edits a list of items, ask: where does the user see this set already, and does this match? If the answer is a different file with a different order, the diff needs a shared constant, not a second literal.
|
||||
@@ -0,0 +1,160 @@
|
||||
---
|
||||
paths:
|
||||
- "apps/sim/hooks/queries/**/*.ts"
|
||||
---
|
||||
|
||||
# React Query Patterns
|
||||
|
||||
All React Query hooks live in `hooks/queries/`. All server state must go through React Query — never use `useState` + `fetch` in components for data fetching or mutations.
|
||||
|
||||
For *client* view-state that belongs in a shareable link (tabs, filters, search, pagination, selected entity id), use URL query params via nuqs — see `.claude/rules/sim-url-state.md`. React Query owns remote data; nuqs owns shareable client view-state.
|
||||
|
||||
## Query Key Factory
|
||||
|
||||
Every query file defines a hierarchical keys factory with an `all` root key and intermediate plural keys for prefix-level invalidation:
|
||||
|
||||
```typescript
|
||||
export const entityKeys = {
|
||||
all: ['entity'] as const,
|
||||
lists: () => [...entityKeys.all, 'list'] as const,
|
||||
list: (workspaceId?: string) => [...entityKeys.lists(), workspaceId ?? ''] as const,
|
||||
details: () => [...entityKeys.all, 'detail'] as const,
|
||||
detail: (id?: string) => [...entityKeys.details(), id ?? ''] as const,
|
||||
}
|
||||
```
|
||||
|
||||
Never use inline query keys — always use the factory.
|
||||
|
||||
**Every identifier the `queryFn` forwards into the fetch MUST appear in the `queryKey`.** (Query-machinery identifiers — `signal`, `pageParam` — are exempt; they aren't fetch-scoping args.) If the fetch is scoped by `workspaceId`, `cursor`, `limit`, an org id, etc., those values must be part of the key — otherwise distinct fetch args share one cache entry (a cross-tenant / per-param cache collision). The lone exception is a globally-unique id used as the key while a second fetch arg is only an authz scope that cannot collide; annotate those with `// rq-lint-allow: <reason>`. Enforced by the `key-fetch-arg-drift` check in `scripts/check-react-query-patterns.ts`.
|
||||
|
||||
## Server-importable query primitives must NOT live in a `'use client'` module
|
||||
|
||||
Next.js rewrites **every** export of a `'use client'` module into a *client reference* in the server bundle. Server-evaluated code — RSC `page.tsx`/`layout.tsx`, `prefetch.ts`, route handlers, **block definitions**, triggers/workers — can only *render* such an export as a component or pass it as a prop; **calling** one throws at runtime (`Attempted to call X from the server but X is on the client` — for an object export it surfaces as `X.list is not a function`). `next build` does **not** catch this — only SSR/runtime does.
|
||||
|
||||
So any **query-key factory, standalone `requestJson` fetcher, mapper, or constant** that a server module imports must live in a **non-`'use client'`** module:
|
||||
|
||||
- key factories → `hooks/queries/utils/<entity>-keys.ts` (see `folder-keys.ts`, `table-keys.ts`, `credential-keys.ts`)
|
||||
- standalone fetchers/mappers → `hooks/queries/utils/fetch-*.ts` / `*-list-query.ts` (see `fetch-workflow-envelope.ts`, `fetch-workspace-credentials.ts`)
|
||||
|
||||
The `'use client'` hook module then imports these back for its hooks. **Never** define a server-imported factory/fetcher directly in a `'use client'` hooks file — it crashes SSR (this caused the tables-page crash). Enforced for prefetch/route/trigger/block files by `scripts/check-client-boundary-imports.ts` (`bun run check:client-boundary`, run in CI). Escape hatch for a genuinely browser-only path: `// client-boundary-allow: <reason>` on the line above the import.
|
||||
|
||||
## File Structure
|
||||
|
||||
```typescript
|
||||
// 1. Query keys factory
|
||||
// 2. Types (if needed)
|
||||
// 3. Private fetch functions (accept signal parameter)
|
||||
// 4. Exported hooks
|
||||
```
|
||||
|
||||
## Query Hook
|
||||
|
||||
- Every `queryFn` must destructure and forward `signal` for request cancellation
|
||||
- Every query must have an explicit `staleTime`, assigned from a named exported constant (`ENTITY_LIST_STALE_TIME`), never an inline numeric literal. A server-side prefetch (`prefetch.ts`) hydrating the same query key must import and reuse that constant, not restate the number — this is what keeps a prefetched cache entry from going stale out of sync with the client hook that reads it
|
||||
- Use `keepPreviousData` only on variable-key queries (where params change), never on static keys
|
||||
- Same-origin JSON calls must go through `requestJson(contract, ...)` from `@/lib/api/client/request` against the contract in `@/lib/api/contracts/**`
|
||||
|
||||
```typescript
|
||||
import { requestJson } from '@/lib/api/client/request'
|
||||
import { listEntitiesContract, type EntityList } from '@/lib/api/contracts/entities'
|
||||
|
||||
export const ENTITY_LIST_STALE_TIME = 60 * 1000
|
||||
|
||||
async function fetchEntities(workspaceId: string, signal?: AbortSignal): Promise<EntityList> {
|
||||
const data = await requestJson(listEntitiesContract, {
|
||||
query: { workspaceId },
|
||||
signal,
|
||||
})
|
||||
return data.entities
|
||||
}
|
||||
|
||||
export function useEntityList(workspaceId?: string, options?: { enabled?: boolean }) {
|
||||
return useQuery({
|
||||
queryKey: entityKeys.list(workspaceId),
|
||||
queryFn: ({ signal }) => fetchEntities(workspaceId as string, signal),
|
||||
enabled: Boolean(workspaceId) && (options?.enabled ?? true),
|
||||
staleTime: ENTITY_LIST_STALE_TIME,
|
||||
placeholderData: keepPreviousData, // OK: workspaceId varies
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
## Mutation Hook
|
||||
|
||||
- Use targeted invalidation (`entityKeys.lists()`) not broad (`entityKeys.all`) when possible
|
||||
- Invalidation must cover all affected query key prefixes (lists, details, related views)
|
||||
- Use `onSuccess` invalidation for plain mutations; use `onSettled` for optimistic mutations so the cache is reconciled on both success and error (see Optimistic Updates below)
|
||||
- `mutationFn` calls go through `requestJson(contract, { body, signal })` from `@/lib/api/client/request` — same boundary rule as queries
|
||||
|
||||
```typescript
|
||||
export function useCreateEntity() {
|
||||
const queryClient = useQueryClient()
|
||||
return useMutation({
|
||||
mutationFn: (body: CreateEntityBody) => requestJson(createEntityContract, { body }),
|
||||
onSuccess: () => {
|
||||
queryClient.invalidateQueries({ queryKey: entityKeys.lists() })
|
||||
},
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
## Optimistic Updates
|
||||
|
||||
For optimistic mutations, use `onSettled` (not `onSuccess`) for cache reconciliation — `onSettled` fires on both success and error, ensuring the cache is always reconciled with the server.
|
||||
|
||||
```typescript
|
||||
export function useUpdateEntity() {
|
||||
const queryClient = useQueryClient()
|
||||
return useMutation({
|
||||
mutationFn: async (variables) => { /* ... */ },
|
||||
onMutate: async (variables) => {
|
||||
await queryClient.cancelQueries({ queryKey: entityKeys.detail(variables.id) })
|
||||
const previous = queryClient.getQueryData(entityKeys.detail(variables.id))
|
||||
queryClient.setQueryData(entityKeys.detail(variables.id), /* optimistic value */)
|
||||
return { previous }
|
||||
},
|
||||
onError: (_err, variables, context) => {
|
||||
queryClient.setQueryData(entityKeys.detail(variables.id), context?.previous)
|
||||
},
|
||||
onSettled: (_data, _error, variables) => {
|
||||
queryClient.invalidateQueries({ queryKey: entityKeys.lists() })
|
||||
queryClient.invalidateQueries({ queryKey: entityKeys.detail(variables.id) })
|
||||
},
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
For optimistic mutations syncing with Zustand, use `createOptimisticMutationHandlers` from `@/hooks/queries/utils/optimistic-mutation`.
|
||||
|
||||
## useCallback Dependencies
|
||||
|
||||
Never include mutation objects (e.g., `createEntity`) in `useCallback` dependency arrays — the mutation object is not referentially stable and changes on every state update. The `.mutate()` and `.mutateAsync()` functions are stable in TanStack Query v5.
|
||||
|
||||
```typescript
|
||||
// ✗ Bad — causes unnecessary recreations
|
||||
const handler = useCallback(() => {
|
||||
createEntity.mutate(data)
|
||||
}, [createEntity]) // unstable reference
|
||||
|
||||
// ✓ Good — omit from deps, mutate is stable
|
||||
const handler = useCallback(() => {
|
||||
createEntity.mutate(data)
|
||||
// eslint-disable-next-line react-hooks/exhaustive-deps
|
||||
}, [data])
|
||||
```
|
||||
|
||||
## Boundary Types
|
||||
|
||||
- Hooks import named type aliases from `@/lib/api/contracts/**` (e.g., `import { listEntitiesContract, type EntityList } from '@/lib/api/contracts/entities'`). Never write `z.input<...>` / `z.output<...>` in hooks, and never `import { z } from 'zod'` in client code.
|
||||
- Raw `fetch` is allowed only for documented exceptions — multipart uploads, binary downloads, streaming responses, signed-URL flows, OAuth redirects, external origins. Each such raw `fetch(` inside `apps/sim/hooks/queries/**` or `apps/sim/hooks/selectors/**` — and any same-origin `/api/...` fetch elsewhere under `apps/sim/**` outside an API route handler — must be preceded by a `// boundary-raw-fetch: <reason>` annotation (reason non-empty; up to three preceding comment lines tolerated). Enforced by `scripts/check-api-validation-contracts.ts` (`bun run check:api-validation` / `:strict`).
|
||||
|
||||
## Naming
|
||||
|
||||
- **Keys**: `entityKeys`
|
||||
- **Query hooks**: `useEntity`, `useEntityList`
|
||||
- **Mutation hooks**: `useCreateEntity`, `useUpdateEntity`, `useDeleteEntity`
|
||||
- **Fetch functions**: `fetchEntity`, `fetchEntities` (private)
|
||||
|
||||
## Enforcement
|
||||
|
||||
`scripts/check-react-query-patterns.ts` (`bun run check:react-query`, run in CI) statically enforces these conventions: every `useQuery`/`useInfiniteQuery`/`useSuspenseQuery` declares an explicit `staleTime`, inline `queryFn`s destructure `signal`, `queryKey`s reference a colocated factory rather than an inline literal, every `*Keys` factory in `hooks/queries/**` exposes an `all` root key, and every identifier the `queryFn` forwards into the fetch also appears in the `queryKey` (`key-fetch-arg-drift`). `hooks/queries/**` is a zero-tolerance zone; the rest of `apps/sim/**` is ratcheted against `scripts/check-react-query-patterns.baseline.json`. For a genuine exception, put `// rq-lint-allow: <reason>` on the line directly above the flagged construct.
|
||||
@@ -0,0 +1,95 @@
|
||||
# React & Render Performance
|
||||
|
||||
Behavior-preserving performance idioms for components, hooks, and hot render paths. These are safe defaults — apply them freely. For the render-causing *effect/state* anti-patterns (derived state in effects, effect chains, state synced to a prop), use the dedicated skills: `/you-might-not-need-an-effect`, `/you-might-not-need-state`, `/you-might-not-need-a-memo`, `/you-might-not-need-a-callback`. Those refactors change render timing — verify them against the running UI, never mass-apply blind.
|
||||
|
||||
## Lazy-init refs that hold objects
|
||||
|
||||
`useRef(new Map())` / `useRef(new Set())` / `useRef({...})` allocates a fresh object on **every render** and throws it away — only the first is ever kept. Lazy-init instead so the allocation happens once.
|
||||
|
||||
```typescript
|
||||
// ✗ Bad — allocates a new Map each render, discards all but the first
|
||||
const cacheRef = useRef<Map<string, string>>(new Map())
|
||||
|
||||
// ✓ Good — allocated once, stable identity thereafter
|
||||
const cacheRef = useRef<Map<string, string> | null>(null)
|
||||
cacheRef.current ??= new Map()
|
||||
```
|
||||
|
||||
Read `cacheRef.current` directly inside effects/handlers — refs are stable and never belong in a dependency array. A cheap primitive (`useRef(0)`, `useRef('')`, `useRef(null)`) needs no lazy init.
|
||||
|
||||
## Hoist static values and closure-free functions to module scope
|
||||
|
||||
A value or function declared inside a component is rebuilt every render. If it captures **nothing** from component scope (no props/state/refs), move it above the component at module scope. This skips the per-render allocation and keeps a stable identity so memoized children don't re-render.
|
||||
|
||||
```typescript
|
||||
// ✗ Bad — rebuilt every render, new identity each time
|
||||
function Toolbar({ mode }: ToolbarProps) {
|
||||
const TITLES = { create: 'Add', edit: 'Configure' } as const
|
||||
const handleWheel = (e: React.WheelEvent) => e.currentTarget.scrollBy(e.deltaX, e.deltaY)
|
||||
// ...
|
||||
}
|
||||
|
||||
// ✓ Good — allocated once at module load
|
||||
const TITLES = { create: 'Add', edit: 'Configure' } as const
|
||||
function handleWheel(e: React.WheelEvent) {
|
||||
e.currentTarget.scrollBy(e.deltaX, e.deltaY)
|
||||
}
|
||||
function Toolbar({ mode }: ToolbarProps) { /* ... */ }
|
||||
```
|
||||
|
||||
A closure-free function that IS wired through a ref sink or intentionally kept for stable identity may stay inline — hoisting a one-line `preventDefault` handler is churn, not a win. Hoist when it removes a real per-render allocation or unblocks child memoization.
|
||||
|
||||
## Pre-index with Map/Set for repeated lookups
|
||||
|
||||
`array.find()` / `array.includes()` / `array.indexOf()` scan the whole list each call. Inside a loop or a hot render path over a non-trivial list, that is O(n·m). Build a `Map` (for lookup-by-key) or `Set` (for membership) **once before** the loop, then look up in O(1).
|
||||
|
||||
```typescript
|
||||
// ✗ Bad — find() re-scans outputs for every column
|
||||
for (const child of columns) {
|
||||
const output = group.outputs.find((o) => o.columnName === getColumnId(child))
|
||||
}
|
||||
|
||||
// ✓ Good — index once, then O(1) lookups
|
||||
const outputByName = new Map<string, Output>()
|
||||
for (const o of group.outputs) {
|
||||
if (!outputByName.has(o.columnName)) outputByName.set(o.columnName, o) // first wins, matches find()
|
||||
}
|
||||
for (const child of columns) {
|
||||
const output = outputByName.get(getColumnId(child))
|
||||
}
|
||||
```
|
||||
|
||||
Preserve `.find()`'s **first-match** semantics when duplicate keys are possible: `new Map(arr.map(...))` keeps the *last* entry, so guard with `if (!map.has(key))` when replacing a `.find()`. Skip this for tiny, cold arrays (a handful of items in an event handler) where the Map build costs more than it saves.
|
||||
|
||||
## Never mutate a shared array in place
|
||||
|
||||
The real bug to avoid is `array.sort()` / `array.reverse()` on an array you don't own — sorting a React Query cache array in place corrupts shared state. Always sort a copy:
|
||||
|
||||
```typescript
|
||||
// ✗ Bad — mutates the (possibly shared) source array in place
|
||||
return items.sort(compare)
|
||||
|
||||
// ✓ Good — sorts a throwaway copy, source untouched
|
||||
return [...items].sort(compare)
|
||||
```
|
||||
|
||||
**Do NOT reach for `toSorted()` / `toReversed()` / `with()` / `toSpliced()` on client render paths.** They are ES2023 *runtime* methods — and a tsconfig `"lib": ["ES2023"]` only makes them **type-check**, it does not make them **run**. Next/SWC compiles syntax but does **not** polyfill prototype methods, and the default browserslist still includes browsers without them (`toSorted` landed in Safari 16 / iOS 16, so any device capped at iOS 15 throws `TypeError: x.toSorted is not a function` and crashes the page). The perf difference vs `[...arr].sort()` is negligible (both allocate one array), so the copy-then-sort form is the correct default everywhere client code runs. Only consider the immutable methods in Node-only code (server routes, scripts) on Node ≥20, where the runtime is known.
|
||||
|
||||
## Run independent awaits in parallel
|
||||
|
||||
Sequential `await`s that don't consume each other's result serialize latency for nothing — in an async Server Component or a route handler this directly delays the response. Kick them off together with `Promise.all` and destructure.
|
||||
|
||||
```typescript
|
||||
// ✗ Bad — waits for params, then separately waits for searchParams
|
||||
const { id } = await params
|
||||
const { kbName } = await searchParams
|
||||
|
||||
// ✓ Good — one combined wait
|
||||
const [{ id }, { kbName }] = await Promise.all([params, searchParams])
|
||||
```
|
||||
|
||||
Only keep awaits sequential when a later call genuinely uses an earlier result, or when the ordering is deliberate (rate-limited batches, retry loops, write-then-read).
|
||||
|
||||
## Local feature barrels are the convention — do not "fix" them
|
||||
|
||||
Tooling (e.g. react-doctor's `no-barrel-import`) will flag imports from local `index.ts` barrels as a bundle cost. In this repo that is a **false positive**: barrel imports for 3+ export folders are mandated by `.claude/rules/sim-imports.md`. Leave them.
|
||||
@@ -0,0 +1,96 @@
|
||||
---
|
||||
description: Isolated-vm sandbox worker security policy. Hard rules for anything that lives in the worker child process that runs user code.
|
||||
paths:
|
||||
- "apps/sim/lib/execution/isolated-vm-worker.cjs"
|
||||
- "apps/sim/lib/execution/isolated-vm.ts"
|
||||
- "apps/sim/lib/execution/sandbox/**"
|
||||
- "apps/sim/sandbox-tasks/**"
|
||||
---
|
||||
|
||||
# Sim Sandbox — Worker Security Policy
|
||||
|
||||
The isolated-vm worker child process at
|
||||
`apps/sim/lib/execution/isolated-vm-worker.cjs` runs untrusted user code inside
|
||||
V8 isolates. The process itself is a trust boundary. Everything in this rule is
|
||||
about what must **never** live in that process.
|
||||
|
||||
## Hard rules
|
||||
|
||||
1. **No app credentials in the worker process**. The worker must not hold, load,
|
||||
or receive via IPC: database URLs, Redis URLs, AWS keys, Stripe keys,
|
||||
session-signing keys, encryption keys, OAuth client secrets, internal API
|
||||
secrets, or any LLM / email / search provider API keys. If you catch yourself
|
||||
`require`'ing `@/lib/auth`, `@sim/db`, `@/lib/uploads/core/storage-service`,
|
||||
or anything that imports `env` directly inside the worker, stop and use a
|
||||
host-side broker instead. This includes the OS environment: the worker is
|
||||
spawned with the explicit allowlisted env from `buildWorkerEnv()` in
|
||||
`isolated-vm.ts` — never spawn it without an `env` option (Node would copy
|
||||
the app's full `process.env`, secrets included, into the worker).
|
||||
|
||||
2. **Host-side brokers own all credentialed work**. The worker can only access
|
||||
resources through `ivm.Reference` / `ivm.Callback` bridges back to the host
|
||||
process. Today the only broker is `workspaceFileBroker`
|
||||
(`apps/sim/lib/execution/sandbox/brokers/workspace-file.ts`); adding a new
|
||||
one requires co-reviewing this file.
|
||||
|
||||
3. **Host-side brokers must scope every resource access to a single tenant**.
|
||||
The `SandboxBrokerContext` always carries `workspaceId`. Any new broker that
|
||||
accesses storage, DB, or an external API must use `ctx.workspaceId` to scope
|
||||
the lookup — never accept a raw path, key, or URL from isolate code without
|
||||
validation.
|
||||
|
||||
4. **Nothing that runs in the isolate is trusted, even if we wrote it**. The
|
||||
task `bootstrap` and `finalize` strings in `apps/sim/sandbox-tasks/` execute
|
||||
inside the isolate. They must treat `globalThis` as adversarial — no pulling
|
||||
values from it that might have been mutated by user code. The hardening
|
||||
script in `executeTask` undefines dangerous globals before user code runs.
|
||||
|
||||
## Why
|
||||
|
||||
A V8 JIT bug (Chrome ships these roughly monthly) gives an attacker a native
|
||||
code primitive inside the process that owns whatever that process can reach.
|
||||
If the worker only holds `isolated-vm` + a single narrow workspace-file broker,
|
||||
a V8 escape leaks one tenant's files. If the worker holds a Stripe key or a DB
|
||||
connection, a V8 escape leaks the service.
|
||||
|
||||
The original `doc-worker.cjs` vulnerability (CVE-class, 225 production secrets
|
||||
leaked via `/proc/1/environ`) was the forcing function for this architecture.
|
||||
Keep the blast radius small.
|
||||
|
||||
## Checklist for changes to `isolated-vm-worker.cjs`
|
||||
|
||||
Before landing any change that adds a new `require(...)` or `process.send(...)`
|
||||
payload or `ivm.Reference` wrapper in the worker:
|
||||
|
||||
- [ ] Does it load a credential, key, connection string, or secret? If yes,
|
||||
move it host-side and expose as a broker.
|
||||
- [ ] Does it import from `@/lib/auth`, `@sim/db`, `@/lib/uploads/core/*`,
|
||||
`@/lib/core/config/env`, or any module that reads `process.env` of the
|
||||
main app? If yes, same — move host-side.
|
||||
- [ ] Does it expose a resource that's workspace-scoped without taking a
|
||||
`workspaceId`? If yes, re-scope.
|
||||
- [ ] Did you update the broker limits (`IVM_MAX_BROKER_ARGS_JSON_CHARS`,
|
||||
`IVM_MAX_BROKER_RESULT_JSON_CHARS`, `IVM_MAX_BROKERS_PER_EXECUTION`) if
|
||||
the new broker can emit large payloads or fire frequently?
|
||||
- [ ] Does the worker read a new env var? Add it to the `buildWorkerEnv()`
|
||||
allowlist in `isolated-vm.ts` **and** to the allowlist regression test in
|
||||
`isolated-vm.test.ts` — the worker does not inherit the app environment,
|
||||
so an un-allowlisted var is simply absent in the child.
|
||||
|
||||
## What the worker *may* hold
|
||||
|
||||
- `isolated-vm` module
|
||||
- Node built-ins: `node:fs` (only for reading the checked-in bundle `.cjs`
|
||||
files) and `node:path`
|
||||
- The three prebuilt library bundles under
|
||||
`apps/sim/lib/execution/sandbox/bundles/*.cjs`
|
||||
- IPC message handlers for `execute`, `cancel`, `fetchResponse`,
|
||||
`brokerResponse`
|
||||
|
||||
The worker deliberately has **no host-side logger**. All errors and
|
||||
diagnostics flow through IPC back to the host, which has `@sim/logger`. Do
|
||||
not add `createLogger` or console-based logging to the worker — it would
|
||||
require pulling the main app's config / env, which is exactly what this
|
||||
rule is preventing.
|
||||
|
||||
Anything else is suspect.
|
||||
@@ -0,0 +1,393 @@
|
||||
---
|
||||
paths:
|
||||
- "apps/sim/app/workspace/*/settings/**"
|
||||
- "apps/sim/app/workspace/*/{integrations,skills,upgrade}/**"
|
||||
- "apps/sim/app/workspace/*/components/{resource-tile,credential-detail}/**"
|
||||
- "apps/sim/components/{settings,permissions}/**"
|
||||
- "apps/sim/ee/**/components/**"
|
||||
---
|
||||
|
||||
# Settings Pages
|
||||
|
||||
The Next.js `settings/[section]/layout.tsx` owns all settings page chrome via
|
||||
`SettingsHeaderShell` — a fixed header bar (a left back chip + right-aligned
|
||||
action chips), a scroll region, and a centered `max-w-[48rem]` content column led
|
||||
by a **title + description from navigation metadata**. The chrome stays mounted
|
||||
across section navigation (it never re-renders or re-lays-out). Each section
|
||||
renders through the **`SettingsPanel`** registrar
|
||||
(`@/app/workspace/[workspaceId]/settings/components/settings-panel`), which feeds
|
||||
the shell its header data and renders only the section body. Sections supply
|
||||
**data**, never chrome.
|
||||
|
||||
Do NOT hand-roll any of these in a settings page — they are owned by the layout
|
||||
shell (fed through `SettingsPanel`):
|
||||
|
||||
- `<div className='flex h-full flex-col bg-[var(--bg)]'>` shell
|
||||
- the header bar — compose `PAGE_HEADER_BAR` (`@/components/page-header-bar`); never rewrite its padding
|
||||
- the scroll container (`min-h-0 flex-1 overflow-y-auto px-6 [scrollbar-gutter:stable_both-edges]`)
|
||||
- the content column (`mx-auto … max-w-[48rem] … gap-7`)
|
||||
- a title block (`<h1 className='font-medium text-[var(--text-body)] text-lg'>` + `<p className='text-[var(--text-muted)] text-md'>`)
|
||||
- the page-level search input
|
||||
|
||||
## Canonical page shape
|
||||
|
||||
```tsx
|
||||
import { SettingsPanel } from '@/app/workspace/[workspaceId]/settings/components/settings-panel'
|
||||
|
||||
return (
|
||||
<SettingsPanel
|
||||
actions={[{ text: 'Create', icon: Plus, variant: 'primary', onSelect: onCreate }]}
|
||||
search={{ value: searchTerm, onChange: setSearchTerm, placeholder: 'Search …' }}
|
||||
>
|
||||
{/* body only — sections, lists, forms */}
|
||||
</SettingsPanel>
|
||||
)
|
||||
```
|
||||
|
||||
When the page has modal/dialog siblings, wrap them with the panel in a fragment:
|
||||
|
||||
```tsx
|
||||
return (
|
||||
<>
|
||||
<SettingsPanel actions={…}>{body}</SettingsPanel>
|
||||
<SomeModal … />
|
||||
</>
|
||||
)
|
||||
```
|
||||
|
||||
## `SettingsPanel` props
|
||||
|
||||
- `actions?: SettingsAction[]` — right-aligned header chips, **data only**:
|
||||
`{ id?, text, textTone?: 'error', icon?, variant?: 'primary'|'destructive', active?,
|
||||
onSelect, onPrefetch?, disabled?, tooltip? }`.
|
||||
The shell renders each as a `Chip` — never pass JSX, a `<div>`, or `className`
|
||||
(the locked contract: it's structurally impossible to vibe-code a padding
|
||||
change). Multiple/conditional actions are a plain array
|
||||
(`[...(canManage ? [{…}] : []), …]`). Labels are **sentence case** (`Add override`,
|
||||
not `Add Override`). A disabled action that needs to explain itself sets
|
||||
`tooltip` (the shell renders the hover tooltip, disabled chip included). An action
|
||||
that wants to warm a route on hover sets `onPrefetch`; the shell wires it. A label
|
||||
that flips while pending (`Delete` → `Deleting...`) sets a stable `id`, or the chip
|
||||
remounts mid-action. Save/Discard pairs come from the `saveDiscardActions()`
|
||||
helper (spread it into `actions`).
|
||||
- `back?: SettingsBackAction` (`{ text, icon?, onSelect }`) — left-aligned back
|
||||
chip for a **detail sub-view** (e.g. a selected MCP server, a permission group,
|
||||
a retention policy). Detail sub-views render through `SettingsPanel` like list
|
||||
pages — they do NOT hand-roll their own shell.
|
||||
- `docsLink?: string` — renders the header's `Docs` `ChipLink`.
|
||||
- `search?: { value; onChange: (value: string) => void; placeholder?; disabled? }` —
|
||||
renders the canonical search field directly below the title. Pass `setSearchTerm`
|
||||
straight to `onChange`. Use this for a standalone search; if search shares a row
|
||||
with other controls (sort, filters, a date picker), render that whole row in
|
||||
`children` instead and omit the prop.
|
||||
- `title?` / `description?` — overrides for the nav-driven defaults. **Only** for a
|
||||
detail sub-view that needs a different heading; normal pages never pass these.
|
||||
- `scrollContainerRef?: React.Ref<HTMLDivElement>` — forwards a ref to the scroll
|
||||
region (e.g. programmatic scroll-to-bottom).
|
||||
|
||||
## Title + description live in navigation metadata
|
||||
|
||||
`apps/sim/components/settings/navigation.ts` is the single source of truth (the
|
||||
`settings/navigation.ts` in the route tree is only a re-export shim). Every `NavigationItem` carries a one-line `description`; `SettingsPanel`
|
||||
resolves both via `getSettingsSectionMeta(plane, section)` and the
|
||||
`SettingsSectionProvider` the settings shell wraps around the active section.
|
||||
|
||||
Adding a new settings page:
|
||||
|
||||
1. Add the section id to the `UnifiedSettingsSection` union + a `NavigationItem`
|
||||
(with `label` **and** `description`) in `components/settings/navigation.ts`. Keep descriptions verb-first, one line,
|
||||
~40–55 chars, in the product voice (see `.claude/rules/constitution.md`).
|
||||
2. Render the component inside the shell's `effectiveSection` switch in
|
||||
`settings/[section]/settings.tsx`.
|
||||
3. Build the component body inside `<SettingsPanel>` — no shell, no title block.
|
||||
|
||||
## Text-scale tokens (no literal pixel sizes)
|
||||
|
||||
Settings pages never use a literal `text-[Npx]` class — always the named Tailwind
|
||||
scale token from `apps/sim/tailwind.config.ts`'s `fontSize` extension (`text-micro`
|
||||
10px, `text-xs` 11px, `text-caption` 12px, `text-small` 13px, `text-sm` 14px
|
||||
[Tailwind default, unmodified], `text-base` 15px, `text-md` 16px, `text-lg` 18px
|
||||
[Tailwind default]). A literal size is either a straight rename to the equivalent
|
||||
token (if the pixel value matches one exactly) or a sign the page never migrated —
|
||||
grep `text-\[1[0-8]px\]` under `apps/sim/app/workspace/*/settings/**` and
|
||||
`apps/sim/ee/**` to find stragglers.
|
||||
|
||||
Watch `text-xs`: it is 11px here, so a "caption" written as `text-xs` is a pixel
|
||||
short. See `sim-styling.md` for the full scale.
|
||||
|
||||
The two-line list row (title over a muted subtitle — a name + email, a tool name
|
||||
+ description, a server name + status) is **not something you build**: it is
|
||||
`SettingsResourceRow`, which owns the pairing
|
||||
(`text-[var(--text-body)] text-sm` over `text-[var(--text-muted)] text-caption`).
|
||||
See "The resource row" below.
|
||||
|
||||
For a toggle row (a `Switch` with a title and optional description), use the emcn
|
||||
`Label` component for the title — never a hand-rolled `<span>` — paired with
|
||||
`Switch`'s `id`/`Label`'s `htmlFor`:
|
||||
|
||||
```tsx
|
||||
<div className='flex items-center justify-between'>
|
||||
<div className='flex flex-col gap-1'>
|
||||
<Label htmlFor='my-toggle'>Enable thing</Label>
|
||||
<p className='text-[var(--text-muted)] text-caption'>One-line description.</p>
|
||||
</div>
|
||||
<Switch id='my-toggle' checked={enabled} onCheckedChange={onToggle} />
|
||||
</div>
|
||||
```
|
||||
|
||||
`Label`'s own default styling (`font-medium text-[var(--text-primary)]
|
||||
text-small`) already matches the established title treatment — do not add a
|
||||
`className` overriding its size/color unless the row genuinely needs something
|
||||
different.
|
||||
|
||||
`--text-primary`/`--text-secondary` and `--text-body`/`--text-muted` are both real,
|
||||
independently-defined tokens (not interchangeable — they resolve to different
|
||||
colors) and both see legitimate use across settings pages; this rule only pins
|
||||
down the **row title/subtitle** shape above, not every text element on every page.
|
||||
|
||||
## The resource row
|
||||
|
||||
**`SettingsResourceRow`** (`…/components/settings-resource-row`) is *the* list row
|
||||
for every settings resource — and for skills, integrations, and the `ee/` surfaces
|
||||
too. It owns the tile, the title/subtitle tokens, the row padding and bleed
|
||||
(`-mx-2 … rounded-lg p-2`), the hit area, the focus ring, the navigation chevron,
|
||||
and — on activatable rows only — the hover band. Never hand-roll any of it, and never wrap the row in your own
|
||||
`<button>` or `<Link>` — that is what `onClick`/`href` are for.
|
||||
|
||||
```tsx
|
||||
<div className={RESOURCE_LIST_STACK}>
|
||||
{items.map((item) => (
|
||||
<SettingsResourceRow
|
||||
key={item.id}
|
||||
icon={<Wrench className='text-[var(--text-icon)]' />}
|
||||
iconFilled
|
||||
title={item.name}
|
||||
description={item.summary}
|
||||
onClick={() => open(item.id)} // or href={`…/${item.id}`}
|
||||
clickLabel={`Open ${item.name}`}
|
||||
navigable
|
||||
/>
|
||||
))}
|
||||
</div>
|
||||
```
|
||||
|
||||
- `icon?` + `iconVariant` — `tile` (default, the 36px bordered tile), `plain` (a
|
||||
bare 14px glyph), `custom` (you supply the whole tile, e.g. the brand-tinted
|
||||
`IntegrationTile`). Omit `icon` entirely for resources with no identity glyph
|
||||
(an API key, a permission group). `iconFilled` uses the skills/tools fill;
|
||||
`iconFill` lets an uploaded image reach the tile edge.
|
||||
- `onClick` / `href` — makes the **whole row** activatable via a stretched
|
||||
overlay. Prefer `href` when the destination is a route, so the row keeps
|
||||
prefetch, middle-click, and open-in-new-tab. Always pass `clickLabel` with
|
||||
either — the overlay holds no text, so it is the control's *only* accessible
|
||||
name. The prop is optional in the type (nothing enforces it), so omitting it
|
||||
ships a nameless button rather than failing the build.
|
||||
- `navigable` — appends the one canonical chevron. Set it on rows that open a
|
||||
detail page; leave it off when `onClick` acts in place (revealing a folder).
|
||||
Never import an arrow yourself: `lucide-react` and `@sim/emcn/icons` ship
|
||||
visibly different glyphs, and the row already picked one. A sanctioned bespoke
|
||||
row (below) draws it with `RESOURCE_ROW_ARROW_CLASSES` from the same module.
|
||||
- `trailing` vs `badge` — `trailing` is for **interactive** controls (a `Chip`, a
|
||||
`RowActionsMenu`) and sits above the hit area. `badge` is for **decoration** (a
|
||||
status tag) and is click-through. Putting a badge in `trailing` turns the row's
|
||||
right edge into a dead zone.
|
||||
- `RESOURCE_LIST_STACK` / `RESOURCE_LIST_GRID` — the single-column and two-up
|
||||
containers. A `SettingsResourceRow` carries its own `-mx-2` bleed and padding,
|
||||
so a container holding one only sets rhythm: never add a second `-mx-2` (they
|
||||
stack into a 16px bleed) and never a different gap. A list of hand-rolled rows
|
||||
is the opposite — there the container owns the bleed. A row inside a
|
||||
fixed-height `overflow-y-auto` box, or a heading that must line up with the
|
||||
section labels under it, passes `flush` to drop the bleed and padding.
|
||||
`RESOURCE_LIST_GRID` budgets its column gap for the bleed (24px of track gap
|
||||
minus 16px of bleed = an 8px gutter); narrowing that gap makes neighbouring
|
||||
rows — and their stretched hit areas — overlap, so a click in the gutter opens
|
||||
the wrong card.
|
||||
|
||||
**Three-dots vs. chevron** is not a taste call:
|
||||
|
||||
- Opens a **detail page** → `navigable` + a whole-row click, and no `Delete` in
|
||||
the row (it lives in the detail header).
|
||||
- **No detail page** → a `RowActionsMenu` in `trailing`, and no `navigable`
|
||||
chevron. The row may still take an `onClick` for an in-place action — a folder
|
||||
mount reveals itself in Finder and also carries a `...` menu — but a row must
|
||||
never offer both a chevron and a menu.
|
||||
|
||||
## Other shared settings primitives (do not re-roll these)
|
||||
|
||||
- **`SettingsSection`** (`…/components/settings-section/settings-section` — this
|
||||
directory has no barrel) — muted label, hairline
|
||||
divider, body. Also carries `headerAccessory` and `action` slots. Never
|
||||
re-derive the label/divider chrome; `sim-styling.md` owns those tokens.
|
||||
- **`SettingsField`** (`…/components/settings-field`) — a read-only label/value
|
||||
pair in a detail body: muted caption over the value. Pair it with
|
||||
`SETTINGS_FIELD_VALUE_CLASSES` for the value text.
|
||||
- **`SettingsEmptyState`** (`…/components/settings-empty-state`) — the canonical
|
||||
muted status message, for empty lists, "no results", loading gates, **and
|
||||
failed loads** (`tone='error'`). `variant='fill'` (default) centers in the
|
||||
available height; `variant='inline'` sits in flow. Never hand-roll
|
||||
`<div className='flex h-full items-center justify-center …'>` or
|
||||
`<div className='py-4 text-center …'>`.
|
||||
- **`RowActionsMenu`** (`…/components/row-actions-menu`) — the trailing `...`
|
||||
actions menu for a list row. Pass `label` (aria-label) and
|
||||
`actions: RowAction[]` (`{ label, onSelect, destructive?, disabled? }`); the
|
||||
component renders the canonical flush `...` trigger + `DropdownMenuContent`.
|
||||
Conditional items become array spreads: `...(canManage ? [{…}] : [])`. Never
|
||||
hand-roll the `<DropdownMenu>` + `<MoreHorizontal>` trigger per page.
|
||||
- **`RESOURCE_TILE_BASE`** + one of `RESOURCE_TILE_FILL` / `RESOURCE_TILE_PLAIN`
|
||||
(`app/workspace/[workspaceId]/components/resource-tile` — note: *not* under
|
||||
`settings/`, unlike the other `…/` paths on this page) — the 36px tile chrome, for
|
||||
any tile the row does not draw itself: a detail heading, or a caller-supplied
|
||||
`iconVariant='custom'` tile. `ResourceTile` wraps the filled pairing. Use
|
||||
`RESOURCE_TILE_FILL` for a glyph, `RESOURCE_TILE_PLAIN` for a brand logo or favicon.
|
||||
|
||||
|
||||
**Member avatars are deliberately two components, not one.** `member-list.tsx`
|
||||
renders a 14px neutral marker for the dense Teammates/Organization roster, where
|
||||
the email is the primary content; `components/permissions/member-row.tsx` renders
|
||||
a 36px `getUserColor`-hashed avatar for member *management* rows that carry a name,
|
||||
an email, and a role control. Same shape, different job — do not merge them.
|
||||
|
||||
## Header action order
|
||||
|
||||
Every detail header reads left→right:
|
||||
|
||||
```
|
||||
← Back [secondary actions] → Delete → Discard → Save
|
||||
```
|
||||
|
||||
You do not have to get the array order right — `orderHeaderActions()` ranks them
|
||||
(secondary → `id:'delete'` → `id:'discard'` → `variant:'primary'`), order-stable
|
||||
within each band, so spreading `saveDiscardActions()` first still renders Save
|
||||
last. Three stacks apply it: `SettingsHeaderShell`, `SettingsActionChips`, and
|
||||
`Resource.Header` — so tables, files, knowledge and logs get the same ordering
|
||||
as settings.
|
||||
|
||||
Delete is placed by its **`id`**, not by position, which is why `id:'delete'` is
|
||||
required rather than cosmetic: a page with no primary action still must not
|
||||
leave a destructive chip in the slot a primary would occupy.
|
||||
|
||||
The bar geometry and the action cluster are both single-sourced in
|
||||
`@/components/page-header-bar` — `PAGE_HEADER_BAR` (or `Resource.Header`'s
|
||||
bordered variant) and `HEADER_ACTION_CLUSTER`. Never re-derive `h-[30px]`,
|
||||
`gap-1`, or the lane padding per header.
|
||||
Covered by `settings-header-order.test.ts` and `settings-header-shell.test.tsx`
|
||||
— the latter pins that a reordered chip still routes to its own handler.
|
||||
|
||||
Two consequences worth knowing:
|
||||
|
||||
- **The primary chip is always right-most**, and it is not always Save — on a
|
||||
page with no save state it is whatever the primary action is (`Add workflows`,
|
||||
`Import`). Delete still precedes it.
|
||||
- `CredentialDetailLayout` takes a `ReactNode`, so the chips you write directly
|
||||
are in your order — only what you route through `SettingsActionChips` /
|
||||
`SaveDiscardChips` is ranked. Put `<SaveDiscardChips>` last (skills, secrets,
|
||||
connected credentials already do).
|
||||
|
||||
## Deleting a resource
|
||||
|
||||
Delete lives in the **detail header**, as `{ id: 'delete', text: 'Delete',
|
||||
onSelect: … }` behind a `ChipConfirmModal` — a **plain chip**, never
|
||||
`textTone: 'error'`, and never unconfirmed. In a `SettingsPanel` header it is
|
||||
action *data*, never a hand-rolled `<Chip>`; only `CredentialDetailLayout`
|
||||
surfaces, which take a `ReactNode`, render one directly. Always set `id: 'delete'`; without a
|
||||
stable id the chip remounts when the label flips to `Deleting...`.
|
||||
|
||||
`variant: 'destructive'` is reserved for actions that are destructive at
|
||||
**scale** — `Delete all` passwords, `Clear all` browsing data, `Sign out all
|
||||
members`. Removing the single resource you are already looking at is confirmed
|
||||
by the modal, so it does not also need a red chip.
|
||||
|
||||
A list row does not carry Delete when the resource has a detail page.
|
||||
|
||||
## Save / Discard + unsaved-changes guard
|
||||
|
||||
Any settings surface with editable state uses **one** shared stack — never
|
||||
hand-roll a Save button, a Discard button, a `beforeunload`, or an "Unsaved
|
||||
changes" modal:
|
||||
|
||||
- **`saveDiscardActions(config)`** (`@/components/settings/save-discard-actions`)
|
||||
— returns the canonical **Discard + Save** `SettingsAction[]`. **Save is always
|
||||
rendered** (primary), disabled until there is something to save, so every
|
||||
editable surface announces its primary action in the same place and a create
|
||||
form is never a page with no visible way to commit it; **Discard appears only
|
||||
when dirty**. Spread it into a `SettingsPanel` `actions` array, beside any
|
||||
sibling actions (a detail view's Delete / Remove override). Config: `dirty`,
|
||||
`saving`, `onSave`, `onDiscard`, `saveDisabled?`, `saveTooltip?`, `creating?`,
|
||||
`saveLabel?`, `savingLabel?`. Create flows pass `creating` — the
|
||||
Create / Creating... labels come as a pair and can never drift apart.
|
||||
`saveLabel`/`savingLabel` are only for genuinely bespoke wording (SSO's
|
||||
`Update`); never hand-roll the pair to get a create label.
|
||||
- **`<SaveDiscardChips {...config} />`** (same module) — the identical rule
|
||||
rendered as chips, for surfaces whose header takes a `ReactNode` instead of
|
||||
action data (`CredentialDetailLayout`: skills, secrets, connected credentials).
|
||||
Both stacks derive from the one function; never hand-roll a Save chip.
|
||||
|
||||
`CredentialDetailLayout` stays slot-driven for exactly two reasons: its back
|
||||
control is a real `<ChipLink href>` (deep-linkable / middle-clickable, which
|
||||
`SettingsBackAction`'s `onSelect` cannot express), and actions like
|
||||
`SkillImportButton` own a hidden file input and their own pending state.
|
||||
**Everything else in one of those headers should be `SettingsAction` data**
|
||||
rendered through `<SettingsActionChips actions={…} />` from
|
||||
`@/components/settings/settings-header` — that is the shared chip path, and it
|
||||
is what keeps tone/icon/variant/tooltip handling from drifting between the two
|
||||
shells. Reach for it before hand-rolling a `Chip`.
|
||||
- **`useSettingsUnsavedGuard({ isDirty })`** (`…/settings/hooks/use-settings-unsaved-guard`)
|
||||
— syncs the page's local `isDirty` into the shared `useSettingsDirtyStore` (so
|
||||
the sidebar's **section-switch** confirm + the centralized `beforeunload` both
|
||||
apply for free) and returns `{ showUnsavedModal, setShowUnsavedModal, guardBack,
|
||||
confirmDiscard }` for a detail view's **in-view back** chip.
|
||||
- **Top-level pages** (whitelabeling, sso): call it **unassigned** —
|
||||
`useSettingsUnsavedGuard({ isDirty: hasChanges })` — they only need the
|
||||
store-sync; the sidebar/`beforeunload` do the rest.
|
||||
- **Detail sub-views** (data-retention, access-control group-detail): route the
|
||||
back chip through `onClick={() => guard.guardBack(closeFn)}` and render the
|
||||
shared `<UnsavedChangesModal open={guard.showUnsavedModal}
|
||||
onOpenChange={guard.setShowUnsavedModal} onDiscard={guard.confirmDiscard} />`
|
||||
(from `@/app/workspace/[workspaceId]/components/credential-detail`). The
|
||||
in-view header **Discard** chip (via `SaveDiscardActions onDiscard`) is a
|
||||
*reset to original* — distinct from the back-confirm's discard, which leaves.
|
||||
- **`useSettingsBeforeUnload`** is mounted by the settings shells
|
||||
(`settings/layout.tsx` and `components/settings/standalone-settings-shell.tsx`) —
|
||||
never add a per-page `beforeunload`.
|
||||
- **Dirty *computation* stays local** (shapes differ: field-compare vs
|
||||
normalize+stringify) — only how dirty is *consumed* is shared. Derive it (a
|
||||
`const`/`useMemo`), never store it in `useState`.
|
||||
- **CRITICAL — rules of hooks:** call `useSettingsUnsavedGuard(...)`
|
||||
**unconditionally, before every early-return gate** (entitlement / loading /
|
||||
not-entitled `return <SettingsEmptyState>`). A hook placed after a gate is
|
||||
skipped on gated renders and crashes.
|
||||
- The route-based credential detail keeps its own `useUnsavedChangesGuard` (it
|
||||
guards real `router.push` navigation + browser Back via a history sentinel);
|
||||
it already shares `UnsavedChangesModal`, so copy stays unified.
|
||||
|
||||
## Detail sub-views
|
||||
|
||||
A drill-down view reached from a list row (selected MCP server, workflow MCP
|
||||
server, permission group, retention policy) renders through
|
||||
`SettingsPanel` like a list page: pass `back={{ text, icon: ArrowLeft, onSelect }}`
|
||||
for the left back chip, `title` (the entity name), and the header `actions`, then
|
||||
render the body. Do NOT hand-roll a shell or header bar; a tab bar renders as the
|
||||
first body child. Gate/early-return states (not-entitled, loading, upgrade
|
||||
prompts) stay as-is.
|
||||
|
||||
The route-based credential detail (`settings/secrets/[credentialId]`) is the lone
|
||||
exception — it lives outside `[section]` and keeps its own `CredentialDetailLayout`.
|
||||
|
||||
## Audit checklist
|
||||
|
||||
A settings page is design-system-clean when:
|
||||
|
||||
- [ ] Its main return is a `<SettingsPanel>` (or `<>…<SettingsPanel>…</>` with modal siblings) — no hand-rolled shell/header/scroll/column.
|
||||
- [ ] It renders **no** hand-rolled `<h1>`/description title block — the title comes from nav metadata.
|
||||
- [ ] Header chips are in `actions`; a standalone search is in the `search` prop.
|
||||
- [ ] Its `NavigationItem` has an accurate, consistent-length `description`.
|
||||
- [ ] Detail sub-views and entitlement/loading gates keep their own chrome (intentional).
|
||||
- [ ] If it has editable state: Save/Discard go through `SaveDiscardActions`, dirty is wired via `useSettingsUnsavedGuard` (called before any early-return gate), and there is **no** hand-rolled Save button / `beforeunload` / "Unsaved changes" modal.
|
||||
- [ ] No business logic, handlers, or conditional rendering changed by the migration — except where the shared primitive makes a gate structural (a permission gate becomes `onClick={can ? … : undefined}` + `navigable={can}`, which renders a plain non-interactive row).
|
||||
- [ ] No literal `text-[Npx]` classes — named scale tokens only (see "Text-scale tokens" above).
|
||||
- [ ] Every **resource** list row (a thing with an identity — a tool, a server, a key, a credential) is a `SettingsResourceRow` in a `RESOURCE_LIST_STACK`/`RESOURCE_LIST_GRID` — no wrapper `<button>`/`<Link>`, no hand-passed arrow, no re-derived title/subtitle spans. Rows with a genuinely different shape stay bespoke, and draw their own arrow with `RESOURCE_ROW_ARROW_CLASSES`: multi-line bodies (inbox tasks), tabular columns (billing invoices, credit usage), grids (secrets), and the member rows (see the avatar note above).
|
||||
- [ ] Rows that open a detail page use `navigable` + `clickLabel`; flat records use `RowActionsMenu`. Not both.
|
||||
- [ ] Decorative trailing content is in `badge`, not `trailing`.
|
||||
- [ ] Labeled sections use `SettingsSection`; read-only fields use `SettingsField`; empty/loading/error use `SettingsEmptyState`.
|
||||
- [ ] Delete is a plain `id:'delete'` header action behind a `ChipConfirmModal`; `destructive` is reserved for bulk actions.
|
||||
- [ ] `tsc`, `biome`, and the page's tests pass.
|
||||
@@ -0,0 +1,71 @@
|
||||
---
|
||||
paths:
|
||||
- "apps/sim/**/store.ts"
|
||||
- "apps/sim/**/stores/**/*.ts"
|
||||
---
|
||||
|
||||
# Zustand Store Patterns
|
||||
|
||||
Stores live in `stores/`. Complex stores split into `store.ts` + `types.ts`.
|
||||
|
||||
## Basic Store
|
||||
|
||||
```typescript
|
||||
import { create } from 'zustand'
|
||||
import { devtools } from 'zustand/middleware'
|
||||
import type { FeatureState } from '@/stores/feature/types'
|
||||
|
||||
const initialState = { items: [] as Item[], activeId: null as string | null }
|
||||
|
||||
export const useFeatureStore = create<FeatureState>()(
|
||||
devtools(
|
||||
(set, get) => ({
|
||||
...initialState,
|
||||
setItems: (items) => set({ items }),
|
||||
addItem: (item) => set((state) => ({ items: [...state.items, item] })),
|
||||
reset: () => set(initialState),
|
||||
}),
|
||||
{ name: 'feature-store' }
|
||||
)
|
||||
)
|
||||
```
|
||||
|
||||
## Persisted Store
|
||||
|
||||
```typescript
|
||||
import { create } from 'zustand'
|
||||
import { persist } from 'zustand/middleware'
|
||||
|
||||
export const useFeatureStore = create<FeatureState>()(
|
||||
persist(
|
||||
(set) => ({
|
||||
width: 300,
|
||||
setWidth: (width) => set({ width }),
|
||||
_hasHydrated: false,
|
||||
setHasHydrated: (v) => set({ _hasHydrated: v }),
|
||||
}),
|
||||
{
|
||||
name: 'feature-state',
|
||||
partialize: (state) => ({ width: state.width }),
|
||||
onRehydrateStorage: () => (state) => state?.setHasHydrated(true),
|
||||
}
|
||||
)
|
||||
)
|
||||
```
|
||||
|
||||
## Rules
|
||||
|
||||
1. Use `devtools` middleware (named stores)
|
||||
2. Use `persist` only when data should survive reload
|
||||
3. `persist` MUST use `partialize` with an explicit whitelist of the durable fields. Exclude transient flags (`isResizing`, drag/hover state) and `_hasHydrated` from the whitelist, and never spread the whole state (`{ ...state }`) — it leaks actions and transient state into storage
|
||||
4. `_hasHydrated` pattern for persisted stores needing hydration tracking
|
||||
5. Immutable updates only
|
||||
6. `set((state) => ...)` when depending on previous state
|
||||
7. Provide `reset()` action
|
||||
|
||||
## Outside React
|
||||
|
||||
```typescript
|
||||
const items = useFeatureStore.getState().items
|
||||
useFeatureStore.setState({ items: newItems })
|
||||
```
|
||||
@@ -0,0 +1,98 @@
|
||||
---
|
||||
paths:
|
||||
- "apps/sim/**/*.tsx"
|
||||
- "apps/sim/**/*.css"
|
||||
---
|
||||
|
||||
# Styling Rules
|
||||
|
||||
## Tailwind
|
||||
|
||||
1. **No inline styles** - Use Tailwind classes. Exception: a genuinely dynamic
|
||||
value (a hashed avatar colour, a brand tile background) that cannot be a class.
|
||||
2. **No duplicate dark classes** - Skip `dark:` when value matches light mode
|
||||
3. **Exact values over approximations** - `h-[26px]`, not `h-6`. But **type size is
|
||||
always a named token** (`text-sm`, `text-caption`) — never `text-[14px]`, which
|
||||
sets font-size only and inherits a different line-height. See
|
||||
`sim-settings-pages.md` for the scale.
|
||||
4. **Transitions** - `transition-colors` for interactive states
|
||||
|
||||
## Conditional Classes
|
||||
|
||||
```typescript
|
||||
import { cn } from '@sim/emcn'
|
||||
|
||||
<div className={cn(
|
||||
'base-classes',
|
||||
isActive && 'active-classes',
|
||||
disabled ? 'opacity-60' : 'hover:bg-accent'
|
||||
)} />
|
||||
```
|
||||
|
||||
## CSS Variables
|
||||
|
||||
For dynamic values (widths, heights) synced with stores:
|
||||
|
||||
```typescript
|
||||
// In store
|
||||
setWidth: (width) => {
|
||||
set({ width })
|
||||
document.documentElement.style.setProperty('--sidebar-width', `${width}px`)
|
||||
}
|
||||
|
||||
// In component
|
||||
<aside style={{ width: 'var(--sidebar-width)' }} />
|
||||
```
|
||||
|
||||
## Text Scale
|
||||
|
||||
Custom font sizes (`apps/sim/tailwind.config.ts`): `text-micro`=10px, `text-xs`=11px, `text-caption`=12px, `text-small`=13px, `text-base`=15px. `text-sm` is Tailwind default 14px. Field titles use `text-small` (13px); hints/errors use `text-caption` (12px).
|
||||
|
||||
Icons default `size-[14px]`. Equal h/w → `size-*` (`size-[14px]`, `size-4`), never `h-N w-N`.
|
||||
|
||||
## Font Weight
|
||||
|
||||
Three steps, Tailwind's stock scale, nothing else: **`font-normal` (400)**, **`font-medium` (500)**, **`font-semibold` (600)**. 400 is the document default, so body text, chip labels, sidebar items, and headings carry **no weight class at all** — they inherit. Reach for a class only to step *up* from body.
|
||||
|
||||
Never write an arbitrary weight (`font-[380]`, `font-[430]`, `font-[450]`, …), and never set `fontWeight` in an inline `style`. There was previously a CSS-variable weight scale (`--font-weight-base/medium/semibold`, remapping `font-medium` to 440/480) plus seven ad-hoc values clustered between 380 and 500; it was deleted because nothing read as hierarchical. Off-scale values are only acceptable where the design system genuinely cannot reach — react-email templates and the static `apps/sim/emails/broadcasts/*.html`, which email clients render without CSS variables.
|
||||
|
||||
Headings inherit their weight. Tailwind preflight resets `h1`–`h6` to `font-weight: inherit`, so an `<h1>` is 400 unless you say otherwise — that is the intended look, not a bug to patch.
|
||||
|
||||
## Color Tokens
|
||||
|
||||
Value text `--text-body`; muted/placeholder/labels `--text-muted`; icons `--text-icon`; neutral borders and dividers `--border` (`--border-1` and `--border-muted` are legacy aliases resolving to it; `--divider` is retired); surfaces `--surface-5` (light) / `--surface-4` (dark); active row `--surface-active`; error `--text-error`. No focus rings on chip surfaces.
|
||||
|
||||
### Line weight
|
||||
|
||||
Neutral border geometry comes from `--border-width`: `1px` by default, dropping to `0.5px` under `@media (min-resolution: 2dppx)` so hidpi displays get a true hairline. Tailwind's `border*` and `divide-*` utilities resolve through it, as do `h-px`/`w-px` — the `px` key is overridden on **`spacing`**, not on `width`/`height`, so a hairline and the `-right-px`/`inset-px` offsets that position it stay in agreement.
|
||||
|
||||
**Tune line weight on `--border`, never on `--border-width`.** Browsers floor a border to whole device pixels, so on a 2dppx display every value in `(0, 1px)` collapses to the same single-pixel hairline and the next drawable step is a full `1px` — double. Width has exactly one usable position; perceived weight is a color property. Light mode gains weight by darkening `--border`, dark mode by lightening it.
|
||||
|
||||
Draw a line with a real `border-*` utility. Never hand-roll one as `shadow-[inset_0_-1px_0_…]` — box-shadow has its own width and cannot follow the token, so such a line silently renders at double weight against every neighbor. Use an explicit numeric width (or a `ring`/`outline`) only when the line is intentionally emphasized, e.g. focus and selection affordances.
|
||||
|
||||
## Chip Components (consumer usage)
|
||||
|
||||
`ChipInput`, `ChipTextarea`, `ChipModal*` own their full chrome. Consumers describe intent through PROPS; they never re-style the chrome. The canonical chrome lives in `packages/emcn/src/components/chip/chip-chrome.ts` (all tokens are re-exported from the `@sim/emcn` barrel — no subpath import needed) — never hand-roll `rounded-lg`/`border`/`bg-[var(--surface-5)]`/`h-[30px]`/`px-2`/`text-sm`/focus rings.
|
||||
|
||||
### Props over className
|
||||
|
||||
- **Errors** → `error` prop. Never `className={cn(err && 'border-[var(--text-error)]')}`.
|
||||
- **Leading icon** → `icon` prop (rendered 14px in `--text-icon`).
|
||||
- **Trailing buttons** (reveal/copy/fetch) → `endAdornment`.
|
||||
- **Inner-input styling** (e.g. `font-mono`, number-spinner reset) → `inputClassName` (ChipInput only). See `app/workspace/[workspaceId]/settings/components/billing/components/usage-limit-field/usage-limit-field.tsx:117`.
|
||||
- **`ChipModalField` controls take NO className.** Pass `title`/`value`/`onChange`/`error`/`hint`/`required`/`flush`. The field owns label, control, and error/hint rendering. See `app/workspace/[workspaceId]/skills/components/skill-modal/skill-modal.tsx:170`.
|
||||
|
||||
### What className MAY carry
|
||||
|
||||
Layout/sizing ONLY: `flex-1`, `w-full`, `w-[Npx]`, `min-w-0`, `max-w-*`, margins, `truncate`. Example: `<ChipInput icon={Search} className='min-w-0 flex-1' .../>` (`app/workspace/[workspaceId]/integrations/integrations.tsx:257`). NEVER re-specify canonical chrome — the component already applies it.
|
||||
|
||||
### Form / chip-modal layout rhythm
|
||||
|
||||
- **Field row** = `ChipModalField`: label↔control `gap-[9px]`, field gutter `px-2` (`px-0` when `flush`). Title = `Label` at `text-small` (13px), muted, normal weight; hint/error at `text-caption` (12px).
|
||||
- **Modal body** (`ChipModalBody`): `gap-4` between fields, padding `px-2 pt-4 pb-4.5`.
|
||||
- **Header/footer**: horizontal gutter `px-4` (header `pt-3`; footer `px-4 pt-2 pb-2`, tinted bar).
|
||||
- **Every body field MUST be a `ChipModalField`** — NEVER hand-roll a field row (raw `<div>` + hand-rolled `<p>`/`<label>` title + bare `ChipInput`/`ChipTextarea`). WHY: body `px-2` + field `px-2` = effective `px-4`, exactly matching the `px-4` header/footer. A hand-rolled row skips the field gutter, sits at `px-2`, and is visibly misaligned (this bug shipped in the scheduled-tasks "Create new scheduled task" modal). Inline errors go through the `error` prop, not a hand-rolled `<p>`.
|
||||
- **Uncovered controls** (`ChipCombobox`, `ChipSelect`, `DatePicker`, `TimePicker`, `ButtonGroup`, arbitrary JSX) → `ChipModalField type='custom'` with a `title`. It still applies the `px-2` gutter and renders the canonical `Label`, so it stays aligned. Never drop such a control into a raw `<div>`, and never add a body-level wrapper `<div>` with a custom `gap-*` that fights `gap-4`.
|
||||
- **Page section rhythm** (integrations/skills/settings): muted `text-small` label + `mt-[9px] mb-3 h-px bg-[var(--border)]` divider, sections stacked `gap-7`. Reuse `SettingsSection` (`app/workspace/[workspaceId]/settings/components/settings-section/settings-section.tsx`) rather than re-deriving it.
|
||||
|
||||
When a standalone labeled field outside a `ChipModal` needs the same look (e.g. `SkillImport`), match the field rhythm by hand: `flex flex-col gap-[9px]`, muted label, `ChipInput`/`ChipTextarea` control, `text-caption` error below.
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user