main
8 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 James Russo
|
557d82b6a9 |
fix(producer): validate distributed video metadata (#2839)
## What - enforce a finite, validated `meta/videos.json` contract shared by Plan v1 and Plan v2 - preserve authored finite ends and source-derived trim-aware ends; bound any still-open end at the validated composition end - fail distributed planning when any declared video source did not extract instead of publishing a blank-capable plan - make the v1 chunk reader reject malformed/null video timing before frame injection - route deterministic video-source/metadata failures as non-retryable in AWS and GCP while retaining retries for transient extraction failures ## Why An open-ended video whose remote source could not be resolved retained `Infinity` through planning. Plan v2 correctly rejected that value, while Plan v1 serialized it as `null`; the v1 frame lookup could then suppress injected frames and silently produce incorrect output. The invariant belongs at the shared metadata boundary. Both protocols must receive identical finite timing, and unavailable sources must fail closed before plan publication. ## Test plan - [x] producer distributed planning, metadata, v1 chunk boundary, Plan v2 conversion/materialization, and public exports - [x] core runtime media semantics (authored slots, natural duration, looping, non-looping hold) - [x] engine video extraction and frame lookup - [x] AWS Lambda/CDK/SAM and GCP Cloud Run error normalization/retry classification - [x] producer, core, engine, AWS, and GCP typechecks/builds - [x] formatting, oxlint, tracked-artifact, fallow, and commit hooks - [x] exact incident composition replayed through the AWS Lambda handler's Lambda-local path in a Lambda-like container; Plan v1 and Plan v2 both fail closed as `VIDEO_SOURCE_UNRENDERABLE` during planning, before plan publication - [x] full PR CI, including all nine regression shards and Windows render/tests No production flags or deployment/release workflows are changed. |
||
|
James
|
2a284a8e3a | fix(gcp): enforce effective BeginFrame capture | ||
|
James Russo
|
0499a5cbcb | fix(gcp-cloud-run): normalize v2 integrity codes (#2790) | ||
|
James Russo
|
5bf61d6df0 |
feat(aws-lambda): support plan protocol v2 (#2789)
* feat(aws-lambda): support plan protocol v2 * fix(aws-lambda): align SAM v2 terminal errors |
||
|
James Russo
|
4da567df22 |
feat(gcp-cloud-run): Google Cloud Run + Workflows distributed render adapter (#1253)
* feat(gcp-cloud-run): add Google Cloud Run + Workflows distributed render adapter Adds @hyperframes/gcp-cloud-run, the GCP counterpart to @hyperframes/aws-lambda (issue #932). The OSS distributed primitives (plan, renderChunk x N, assemble) are unchanged; this package is the storage/compute/orchestration glue. Package: Cloud Run handler (one image, three actions), runs under bun; GCS transport; in-image chrome-headless-shell resolver; client SDK (renderToCloudRun, getRenderProgress, deploySite, computeRenderCost); Dockerfile; Cloud Workflows definition; Terraform module; CLI cloudrun deploy|sites|render|render-batch|progress|destroy with --output-resolution and --strict-variables; 62 unit tests + docs + live smoke script. Shared extraction (removes ~640 lines of adapter duplication): move the cloud-agnostic config validator + content-hash into producer/distributed; both adapters import them. Validated end-to-end on GCP at 37.4 dB PSNR vs baseline. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(cli): resolve @hyperframes/gcp-cloud-run in the CLI build + root build The CLI bundle (esbuild) couldn't resolve `@hyperframes/gcp-cloud-run/sdk`, failing Build/Typecheck/CLI-smoke (and the perf/windows/regression jobs that build first). Mirror the aws-lambda handling: mark the gcp adapter + its /sdk subpath external in tsup.config.ts with a source alias, and add gcp-cloud-run to the root `build` filter so its dist exists for publish + runtime. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ci): copy gcp-cloud-run manifest in Dockerfile.test for frozen install The regression test image runs `bun install --frozen-lockfile` after copying each workspace package.json individually. The CLI now depends on @hyperframes/gcp-cloud-run (workspace:*), so the frozen install fails to resolve it unless its manifest is present. Add the COPY line. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(cli): add machine-sizing flags to `cloudrun deploy` Closes the parity gap with `lambda deploy` (which exposes --memory etc.). `cloudrun deploy` now threads --cpu, --memory, --max-instances, and --timeout into the Terraform apply; omitted flags keep the module defaults (4 vCPU / 16Gi / 100 instances / 3600s). For finer control, apply the module directly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(gcp-cloud-run): address PR review (security, waste, limits, alerts) - server.ts: bucket-allowlist guard no longer fails open silently. Unset env logs a one-time WARNING; "*" is an explicit opt-out; otherwise it enforces. - server.ts: stop double-shipping audio.aac. It already rides in the plan tarball every consumer downloads, so drop the redundant standalone upload (plan) + re-download/overwrite (assemble); assemble reads it from the untar, falling back to a supplied AudioGcsUri for compat. - server.ts: chunk extension via path.extname() instead of slice(lastIndexOf). - workflow.yaml: clamp parallel concurrency_limit to math.min(chunkCount, 20) — Cloud Workflows hard-caps concurrent iterations at 20. - Dockerfile: pin bun (bun-v1.3.9) so an interop change can't silently break the image rebuild. - terraform: add min_instances var (default 0); add a workflow-failure alert (finished_execution_count status=FAILED) alongside the request-count one. - costAccounting: document that displayCost excludes GCS storage/egress. Verified against the actual APIs: @google-cloud/workflows@4.4.0 ICreateExecutionRequest has no executionId (so the idempotency-token suggestion isn't available in this client); Workflows concurrency cap is 20; failure metric is workflows.googleapis.com/finished_execution_count (status label). 174 adapter tests pass, fallow/oxlint/oxfmt/terraform clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(gcp-cloud-run): address round-2 review — error code + CFR forwarding - workflow.yaml: rename the zero-chunk failure code PLAN_TOO_LARGE → PLAN_PRODUCED_ZERO_CHUNKS. The old code implied a size-ceiling breach (the opposite cause), misleading anyone triaging the alert. - workflow.yaml: forward Config.cfr to the assemble step (`Cfr: ${("cfr" in config) and config.cfr}`). It was read by the handler but never sent, so exact-CFR was silently off for every Cloud Run render. Uses the same `in`-operator guard already proven in the retryable predicate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(release): include gcp-cloud-run in set-version PACKAGES list set-version.ts (driven by release:prepare) bumps an explicit package list to the shared version on each release. gcp-cloud-run was wired into the build + publish.yml but missing here, so a release would leave it at a stale version and publish.yml would push the wrong version. Add it so the new package version-bumps + publishes in lockstep with the others. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
James Russo
|
b6b6b8ed3b |
docs(lambda): add migration guide + non-Lambda Dockerfile example (#915)
Two adopter-facing artifacts that close out Phase 6b's user-facing
surface:
- docs/deploy/migrating-to-hyperframes-lambda.mdx — side-by-side
concept mapping for users coming from another one-command-deploy
video renderer. Covers the verb mapping (deploy/render/progress/
destroy/sites/policies), composition format (plain HTML vs JSX),
render config, and a handful of intentional differences (no HDR
in distributed mode, no webm, gpu-mode=software requirement,
fail-closed font fetch, local stack-state files, narrow-after-
first-deploy IAM pattern). Closes with a migration checklist.
Per repo convention, no competitor framework is named anywhere
in the source — adopters self-identify.
- examples/k8s-jobs/Dockerfile.example + README.md — reference
Dockerfile for adopters who want to run distributed renders
outside AWS Lambda. Bakes Node 22 + chrome-headless-shell +
ffmpeg + the producer source. Deliberately not published to
a registry; adopters build it themselves so Chrome / ffmpeg /
producer versions stay pinned to the checkout they audited.
The README documents the typical K8s Jobs orchestration shape
that points adopters at packages/aws-lambda/src/handler.ts as
the reference adapter.
Migration guide registered under the existing Deploy group in
docs.json. .gitignore extended to negate the new examples/k8s-jobs/
path the same way examples/aws-lambda/ is negated.
No source code changes.
|
||
|
James Russo
|
aa58ebc048 |
feat(lambda): add real-AWS smoke + benchmark script (#908)
* feat(lambda): add real-AWS smoke + benchmark script
Phase 6.3 of the distributed rendering plan
(DISTRIBUTED-RENDERING-PLAN.md §11 Phase 6a). Local bash script that
deploys the PR 6.2 SAM template to your own AWS account, renders a
fixture composition through the Step Functions state machine at several
chunk counts, PSNR-compares each output against the in-process
baseline, and tears the stack down.
This is the gate that proves the architecture works on real Lambda
infrastructure. After it runs green and the numbers are good, Phase 6b
(CLI, CDK, docs) can proceed with confidence.
Script lives at examples/aws-lambda/scripts/smoke.sh. Defaults:
- fixture: mp4-h264-sdr
- chunk_counts: 2,4,8
- psnr-threshold: 50 dB
- region: us-east-1
- stack-name: hyperframes-lambda-smoke-<timestamp>
AWS credentials come from the standard resolution chain (env vars,
~/.aws/credentials, SSO, IMDS). Pin a specific profile via --profile or
AWS_PROFILE; the script doesn't ship a default.
Workflow:
1. Pre-flight: verify aws/sam/bun/ffmpeg/jq/zip on PATH; check
credentials via sts:GetCallerIdentity.
2. Build the PR 6.1 ZIP and run verify:zip-size.
3. sam validate --lint + sam deploy under a per-run stack name.
4. Zip the fixture's src/ and upload to the render bucket.
5. For each chunk count, start a Step Functions execution, poll for
completion (25-min cap), download the output mp4 and the execution
history JSON, ffmpeg-psnr against the LFS-tracked in-process
baseline, and append to results.json.
6. Gate on the PSNR threshold.
7. Empty the bucket + sam delete (unless --keep-stack).
Outputs land under ./lambda-smoke-artifacts/:
- results.json (chunkCount x wallClockMs x psnrAvgDb)
- renders/N<N>-output.mp4
- renders/N<N>-history.json (full Step Functions execution history)
Per-run stack name with concurrency-safe AWS resource isolation. Run
multiple smokes in parallel without races; teardown guards against
stale stacks via cleanup_and_exit on every failure path.
Distinction from CI: this is a maintainer-run gate, not part of regular
CI. The architecture's per-PR safety net is the local Docker-based
BeginFrame probe (PR 6.1) and the upcoming Lambda RIE smoke mode (PR
6.6). No GitHub Actions / OIDC / cross-account secrets required.
This is part of the 8-PR Phase 6 stack; PR 6.3 of 8 — the last PR of
Phase 6a (validation). Phase 6b (CLI + CDK + docs) starts once 6.3's
benchmark numbers come back.
* fix(lambda): address PR 880 review feedback
- Document wall-clock methodology bias inline (eval.sh header + README):
local timing includes bun + tsx + harness scaffolding while Lambda
timing measures pure SFN execution, so "speedup" is end-to-end CLI
experience, not renderer-vs-renderer.
- Add --iterations N (default 1) with median wall-clock reporting via
awk-side median. Cold-start variance is ±5-10s per chunk; single-
sample readings made the PR-body speedup table not ground truth.
- Add --reserved-concurrency flag to both scripts; default still 16 but
no longer hardcoded. Pass-through to ReservedConcurrency CFN param.
- README: cost-per-pass estimate for both scripts.
- Replace `sed -n '2,30p' "$0"` help with usage() heredoc in both
scripts — fragile to header reflows and didn't survive the comment
expansion this commit adds anyway.
- eval.sh RMS-level parser: add a third fallback (`RMS level:` with no
`dB` suffix) for older ffmpeg builds where astats predates the unit
tag. Word-boundary guards keep `RMS peak level` from being eaten.
* docs(lambda): drop internal plan-doc + Rio refs from smoke/eval scripts
|
||
|
James Russo
|
6664e54053 |
feat(lambda): add SAM template and sample events for AWS deployment (#907)
* feat(lambda): add SAM template and sample events for AWS deployment Phase 6.2 of the distributed rendering plan (DISTRIBUTED-RENDERING-PLAN.md §15). Reference SAM template for deploying HyperFrames distributed rendering on AWS — one Lambda function in three roles, choreographed by a Step Functions standard workflow with a Map state for parallel chunk rendering. Resources created by the template: - Lambda function pointing at the Phase 6.1 ZIP - Step Functions state machine: Plan -> Map(N) RenderChunk -> Assemble - S3 bucket for plan tarballs, chunk outputs, final mp4 - IAM role for the state machine - CloudWatch alarm guarding against runaway chunk invocations Retry policy: 4 attempts, 2s initial, 2x backoff, max 60s, with the typed non-retryable error codes from plan §9.3 explicitly opted out. CodeUri points at packages/aws-lambda/dist/handler.zip; sam deploy resolves the local path and uploads to a SAM-managed bucket on first deploy. Validated: sam validate --lint passes against the template. This is part of the 8-PR Phase 6 stack; PR 6.2 of 8. * fix(lambda): address PR 879 review feedback - Add CloudWatch alarms for Lambda Errors metric (5min window, threshold 1) and Step Functions ExecutionsFailed metric. The existing runaway- invocations alarm catches too-many-calls but missed silent per-chunk failures and retry-exhaustion. - Document VersioningConfiguration: Suspended tradeoff inline. Adopters treating the final mp4 as user-keepable should bump to Enabled. - Cost-allocation Tags on RenderBucket + Lambda Globals. - Lambda Tracing: Active so X-Ray spans don't terminate at the SF→Lambda boundary (the state machine already had tracing). - State-machine top-level TimeoutSeconds: 3600 as defensive ceiling on the whole choreography — catches Plan-retry storms before they hit individual task budgets. - AssertChunkCount Choice state: if Plan ever returns ChunkCount=0 the Map would silently iterate zero times and Assemble would receive an empty ChunkS3Uris[] producing an empty output. Fail-fast with typed PLAN_TOO_LARGE error instead. - Architecture comment: explicit x86_64-only constraint from @sparticuz/chromium so adopters trying Graviton don't get bitten. * docs(lambda): drop internal plan-doc refs from SAM example + template |