Vlad Frangu
dd6e3b9929
Replaces the remaining `ow`-based argument validation with `zod` across all packages and reworks how validation results are consumed and reported. Closes #3716 - **`ow` is gone** — every argument check now goes through `parseArgument(value, schema, label?)` from `@crawlee/utils`, backed by shared zod schemas (`schemas`, exported via `@crawlee/utils/internal`). The `@sapphire/shapeshift` checks in `@crawlee/fs-storage` were converted too, so a single validation library remains. - **Parse results are used everywhere** — option defaults moved from destructuring into the schemas (`.default(...)`), and call sites destructure the typed parse result. `parseArgument` returns `TValue & z.output<TSchema>`, so call sites keep their declared TS types while gaining the defaults. - **Schemas are built once** — all per-call schemas are hoisted to module scope; crawler/launcher classes build their strict options schema once as a `static optionsSchema` next to `optionsShape`. The `urlPatternSchema` for `include`/`exclude` lives in `enqueue_links/shared.ts`, next to the type it validates. - **Specific validators instead of `anyObject`** — class-typed options use `z.instanceof(...)` (`BaseHttpClient`, `Configuration`, `EventManager`), interface-typed ones use duck-typed `objectWithKeys` validators (`storageBackend`, `requestManager`, `logger`, …), and element-typed arrays use the new `schemas.arrayOf(item, 'numbers')`. `ArgumentValidationError` (replacing ow's `ArgumentError`) renders one line per issue: the expected type, the received type and value folded into one clause, the offending field path, and the validated interface: ```text // v3 (ow) — first issue only Expected property `maxRequestRetries` to be of type `number` but received type `string` in object `HttpCrawlerOptions` // v4 (zod) — every issue, one line each Invalid input: expected number, received the string `many` at `maxRequestRetries` in `HttpCrawlerOptions` Invalid input: expected an array of numbers, received the number `500` at `additionalHttpErrorStatusCodes` in `HttpCrawlerOptions` Invalid input: expected boolean, received the string `yes` at `retryOnBlocked` in `HttpCrawlerOptions` ``` Details worth knowing: - Union failures expand into one line per failed arm (zod's own message is a bare "Invalid input"). - `NaN` is named as itself, an empty string renders as `''`, and arrays name their element type (``expected an array of URL patterns``) — none of which ow or stock zod reported. - `new Request('https://…')` gets a targeted hint pointing at the `{ url }` object form. - For programmatic handling, the error exposes zod's structured output: `error.issues` and the raw `ZodError` as a typed `cause`. The migration is documented in the v4 upgrading guide (`docs/upgrading/upgrading_v4.md`), including a rename-cheat-sheet entry. - Custom HTTP clients must now **extend `BaseHttpClient`** from `@crawlee/http-client` rather than just implementing the interface (all shipped clients already do; `LazyDefaultHttpClient` was converted). Same applies to test mocks — `Object.create(BaseHttpClient.prototype)` works. - One caveat of consuming parse results: zod object schemas return a pruned plain copy, so options holding class instances are validated with passthrough schemas (`z.custom`-based) to keep their prototypes — there are comments at the relevant schemas. - Fixes a few latent gaps surfaced along the way: `Request.state` now accepts `RequestState.SKIPPED` (validated via `z.enum(RequestState)`), and the publish-time catalog inlining covers `optionalDependencies`. - `ArgumentValidationError` and its formatter are intentionally kept close to the copy in apify/apify-client-js#986 — a follow-up may extract them into a shared package. --------- Co-authored-by: Martin Adámek <banan23@gmail.com>
93 lines
3.6 KiB
YAML
93 lines
3.6 KiB
YAML
packages:
|
|
- packages/*
|
|
- docs
|
|
- website
|
|
|
|
catalog:
|
|
zod: ^4.4.3
|
|
|
|
minimumReleaseAge: 1440
|
|
minimumReleaseAgeExclude:
|
|
- "@apify/*"
|
|
- "@crawlee/*"
|
|
- "apify-client"
|
|
- "apify"
|
|
- "crawlee"
|
|
- "got-scraping"
|
|
|
|
overrides:
|
|
playwright-core: 1.61.1
|
|
"@browserbasehq/stagehand": 3.0.7
|
|
# Dedup minimatch to v9 everywhere except inside lerna — lerna 9.x's bundled
|
|
# code (`__toESM(require('minimatch')).default(...)`) only works with v3,
|
|
# whose CJS export *is* the function. Pinning v9 there caused the publish
|
|
# step to silently report 0 changed packages.
|
|
minimatch: "^9.0.0"
|
|
tmp: "^0.2.7"
|
|
"@puppeteer/browsers": "^3.0.4"
|
|
form-data: "^4.0.6"
|
|
tar: "^7.5.16"
|
|
"lerna>js-yaml": "^4.2.0"
|
|
"lerna>minimatch": "^3.1.4"
|
|
# The e2e tests run the `apify` SDK against the local 4.x workspace packages.
|
|
# The default `latest`/`next` tags are the 3.x SDK, which is not runtime
|
|
# compatible with `@crawlee/core@4` (e.g. `Configuration.INTEGER_VARS` was
|
|
# removed), so keep the SDK on the latest v4 beta and force it to use the workspace `@crawlee/*`
|
|
# packages (mirroring the per-actor `overrides.apify` used for PLATFORM tests).
|
|
apify: "^4.0.0-beta.24"
|
|
"apify>@crawlee/core": "workspace:*"
|
|
"apify>@crawlee/types": "workspace:*"
|
|
"apify>@crawlee/utils": "workspace:*"
|
|
# header-generator (via fingerprint-generator, a direct dependency of @crawlee/browser-pool)
|
|
# bundles its own ow@0.28.2, while all @crawlee/* packages depend on ow@2.0.0 directly. The two
|
|
# copies' predicate types are nominally distinct (branded symbols), so a type built with one
|
|
# doesn't satisfy a shape built with the other — this broke `@crawlee/stagehand`'s build with
|
|
# `optionsShape`/`exactShape` mismatches. header-generator's ow usage is limited to the
|
|
# long-stable basic predicate API (string/number/object.exactShape/oneOf/ofType), so it's safe
|
|
# to dedupe onto the same major version everyone else uses.
|
|
"header-generator>ow": "^2.0.0"
|
|
|
|
# pnpm 11 replaces `onlyBuiltDependencies` with an explicit `allowBuilds` map.
|
|
# Each entry must be true (build allowed) or false (build skipped) — pnpm 11
|
|
# refuses to install if any dep needs a build decision that isn't in the map
|
|
# (combined with `strictDepBuilds: false` below so the install can still
|
|
# proceed when new build-requesting deps appear without a manual entry).
|
|
allowBuilds:
|
|
"@apify/ui-icons": true
|
|
"@playwright/browser-chromium": true
|
|
"@playwright/browser-firefox": true
|
|
"@playwright/browser-webkit": true
|
|
"@swc/core": true
|
|
better-sqlite3: true
|
|
bufferutil: true
|
|
core-js: true
|
|
core-js-pure: false
|
|
esbuild: true
|
|
nx: true
|
|
protobufjs: true
|
|
puppeteer: true
|
|
unrs-resolver: true
|
|
|
|
strictDepBuilds: false
|
|
|
|
# pnpm 11 wraps every `pnpm run X` with an automatic `pnpm install` (the
|
|
# `runDepsStatusCheck` feature) so scripts always see fresh deps. The
|
|
# publish workflow runs `pnpm turbo copy --force -- --canary=major` which
|
|
# fans `pnpm run copy` out across all 23 workspace packages in parallel —
|
|
# each invocation kicks off its own `pnpm install`, they contend on the
|
|
# store and lockfile, and one of them gets killed with SIGINT, failing the
|
|
# turbo task with "command exited (1)". Disabling the auto-install keeps
|
|
# the parallel scripts deterministic; we run `pnpm install --frozen-lockfile`
|
|
# explicitly in CI before any `pnpm run` step anyway.
|
|
verifyDepsBeforeRun: false
|
|
|
|
nodeLinker: hoisted
|
|
linkWorkspacePackages: true
|
|
preferWorkspacePackages: true
|
|
publicHoistPattern:
|
|
- "*"
|
|
|
|
patchedDependencies:
|
|
'@docusaurus/core@3.10.2': patches/@docusaurus__core@3.10.2.patch
|
|
'@signalwire/docusaurus-plugin-llms-txt@1.2.2': patches/@signalwire__docusaurus-plugin-llms-txt@1.2.2.patch
|