Vlad Frangu
dd6e3b9929
Replaces the remaining `ow`-based argument validation with `zod` across all packages and reworks how validation results are consumed and reported. Closes #3716 - **`ow` is gone** — every argument check now goes through `parseArgument(value, schema, label?)` from `@crawlee/utils`, backed by shared zod schemas (`schemas`, exported via `@crawlee/utils/internal`). The `@sapphire/shapeshift` checks in `@crawlee/fs-storage` were converted too, so a single validation library remains. - **Parse results are used everywhere** — option defaults moved from destructuring into the schemas (`.default(...)`), and call sites destructure the typed parse result. `parseArgument` returns `TValue & z.output<TSchema>`, so call sites keep their declared TS types while gaining the defaults. - **Schemas are built once** — all per-call schemas are hoisted to module scope; crawler/launcher classes build their strict options schema once as a `static optionsSchema` next to `optionsShape`. The `urlPatternSchema` for `include`/`exclude` lives in `enqueue_links/shared.ts`, next to the type it validates. - **Specific validators instead of `anyObject`** — class-typed options use `z.instanceof(...)` (`BaseHttpClient`, `Configuration`, `EventManager`), interface-typed ones use duck-typed `objectWithKeys` validators (`storageBackend`, `requestManager`, `logger`, …), and element-typed arrays use the new `schemas.arrayOf(item, 'numbers')`. `ArgumentValidationError` (replacing ow's `ArgumentError`) renders one line per issue: the expected type, the received type and value folded into one clause, the offending field path, and the validated interface: ```text // v3 (ow) — first issue only Expected property `maxRequestRetries` to be of type `number` but received type `string` in object `HttpCrawlerOptions` // v4 (zod) — every issue, one line each Invalid input: expected number, received the string `many` at `maxRequestRetries` in `HttpCrawlerOptions` Invalid input: expected an array of numbers, received the number `500` at `additionalHttpErrorStatusCodes` in `HttpCrawlerOptions` Invalid input: expected boolean, received the string `yes` at `retryOnBlocked` in `HttpCrawlerOptions` ``` Details worth knowing: - Union failures expand into one line per failed arm (zod's own message is a bare "Invalid input"). - `NaN` is named as itself, an empty string renders as `''`, and arrays name their element type (``expected an array of URL patterns``) — none of which ow or stock zod reported. - `new Request('https://…')` gets a targeted hint pointing at the `{ url }` object form. - For programmatic handling, the error exposes zod's structured output: `error.issues` and the raw `ZodError` as a typed `cause`. The migration is documented in the v4 upgrading guide (`docs/upgrading/upgrading_v4.md`), including a rename-cheat-sheet entry. - Custom HTTP clients must now **extend `BaseHttpClient`** from `@crawlee/http-client` rather than just implementing the interface (all shipped clients already do; `LazyDefaultHttpClient` was converted). Same applies to test mocks — `Object.create(BaseHttpClient.prototype)` works. - One caveat of consuming parse results: zod object schemas return a pruned plain copy, so options holding class instances are validated with passthrough schemas (`z.custom`-based) to keep their prototypes — there are comments at the relevant schemas. - Fixes a few latent gaps surfaced along the way: `Request.state` now accepts `RequestState.SKIPPED` (validated via `z.enum(RequestState)`), and the publish-time catalog inlining covers `optionalDependencies`. - `ArgumentValidationError` and its formatter are intentionally kept close to the copy in apify/apify-client-js#986 — a follow-up may extract them into a shared package. --------- Co-authored-by: Martin Adámek <banan23@gmail.com>
59 lines
1.7 KiB
JSON
59 lines
1.7 KiB
JSON
{
|
|
"name": "@crawlee/basic",
|
|
"version": "4.0.0",
|
|
"description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
|
|
"engines": {
|
|
"node": ">=22.0.0"
|
|
},
|
|
"type": "module",
|
|
"exports": {
|
|
".": "./dist/index.js",
|
|
"./package.json": "./package.json"
|
|
},
|
|
"author": {
|
|
"name": "Apify",
|
|
"email": "support@apify.com",
|
|
"url": "https://apify.com"
|
|
},
|
|
"contributors": [
|
|
"Jan Curn <jan@apify.com>",
|
|
"Marek Trunkat <marek@apify.com>",
|
|
"Ondra Urban <ondra@apify.com>"
|
|
],
|
|
"license": "Apache-2.0",
|
|
"repository": {
|
|
"type": "git",
|
|
"url": "git+https://github.com/apify/crawlee"
|
|
},
|
|
"bugs": {
|
|
"url": "https://github.com/apify/crawlee/issues"
|
|
},
|
|
"homepage": "https://crawlee.dev",
|
|
"scripts": {
|
|
"build": "pnpm clean && pnpm compile && pnpm copy",
|
|
"clean": "rimraf ./dist",
|
|
"compile": "tsc -p tsconfig.build.json",
|
|
"copy": "tsx ../../scripts/copy.ts"
|
|
},
|
|
"publishConfig": {
|
|
"access": "public"
|
|
},
|
|
"dependencies": {
|
|
"@apify/datastructures": "^2.0.0",
|
|
"@apify/timeout": "^0.4.4",
|
|
"@apify/utilities": "^2.15.5",
|
|
"@crawlee/core": "workspace:*",
|
|
"@crawlee/http-client": "workspace:^",
|
|
"@crawlee/types": "workspace:*",
|
|
"@crawlee/utils": "workspace:*",
|
|
"csv-stringify": "^6.5.2",
|
|
"tldts": "^7.0.6",
|
|
"tslib": "^2.8.1",
|
|
"type-fest": "^4.41.0",
|
|
"zod": "catalog:"
|
|
},
|
|
"optionalDependencies": {
|
|
"@crawlee/impit-client": "workspace:^"
|
|
}
|
|
}
|