Bumps typescript to `^7.0.2` (native compiler) for building and
type-aware linting, together with `oxlint-tsgolint@^7.0.2001`, which
version-tracks TS 7.
The docs build stays on TS 6: typedoc needs the TypeScript JS API, which
the native compiler no longer exposes. The website keeps its own
`typescript: ^6.0.0` pin and pnpm nests that copy under
`node_modules/typedoc`, so typedoc resolves 6.x while the packages
compile with TS 7.
TS 7 also caught one real issue in the tests: `RequestList#initialize()`
is private, and the call in `request_list.test.ts` now needs the same
`@ts-expect-error` the surrounding test already uses for the private
constructor.
The api-extractor runner script also imports `typescript` for its report
parsing; that import now goes through a `typescript-v6` alias
(`npm:typescript@^6.0.0`), since the parsing needs the classic JS API
(api-extractor itself bundles its own TypeScript and is unaffected). The
regenerated reports in `docs/public-api/` pick up TS 7's single-quoted
string literal types in three packages, 4 lines total.
Same change as apify/apify-sdk-js#695. Build, docs build, type-aware
lint, test typecheck and the unit suite all pass.
Replaces the remaining `ow`-based argument validation with `zod` across
all packages and reworks how validation results are consumed and
reported.
Closes#3716
- **`ow` is gone** — every argument check now goes through
`parseArgument(value, schema, label?)` from `@crawlee/utils`, backed by
shared zod schemas (`schemas`, exported via `@crawlee/utils/internal`).
The `@sapphire/shapeshift` checks in `@crawlee/fs-storage` were
converted too, so a single validation library remains.
- **Parse results are used everywhere** — option defaults moved from
destructuring into the schemas (`.default(...)`), and call sites
destructure the typed parse result. `parseArgument` returns `TValue &
z.output<TSchema>`, so call sites keep their declared TS types while
gaining the defaults.
- **Schemas are built once** — all per-call schemas are hoisted to
module scope; crawler/launcher classes build their strict options schema
once as a `static optionsSchema` next to `optionsShape`. The
`urlPatternSchema` for `include`/`exclude` lives in
`enqueue_links/shared.ts`, next to the type it validates.
- **Specific validators instead of `anyObject`** — class-typed options
use `z.instanceof(...)` (`BaseHttpClient`, `Configuration`,
`EventManager`), interface-typed ones use duck-typed `objectWithKeys`
validators (`storageBackend`, `requestManager`, `logger`, …), and
element-typed arrays use the new `schemas.arrayOf(item, 'numbers')`.
`ArgumentValidationError` (replacing ow's `ArgumentError`) renders one
line per issue: the expected type, the received type and value folded
into one clause, the offending field path, and the validated interface:
```text
// v3 (ow) — first issue only
Expected property `maxRequestRetries` to be of type `number` but received type `string` in object `HttpCrawlerOptions`
// v4 (zod) — every issue, one line each
Invalid input: expected number, received the string `many` at `maxRequestRetries` in `HttpCrawlerOptions`
Invalid input: expected an array of numbers, received the number `500` at `additionalHttpErrorStatusCodes` in `HttpCrawlerOptions`
Invalid input: expected boolean, received the string `yes` at `retryOnBlocked` in `HttpCrawlerOptions`
```
Details worth knowing:
- Union failures expand into one line per failed arm (zod's own message
is a bare "Invalid input").
- `NaN` is named as itself, an empty string renders as `''`, and arrays
name their element type (``expected an array of URL patterns``) — none
of which ow or stock zod reported.
- `new Request('https://…')` gets a targeted hint pointing at the `{ url
}` object form.
- For programmatic handling, the error exposes zod's structured output:
`error.issues` and the raw `ZodError` as a typed `cause`.
The migration is documented in the v4 upgrading guide
(`docs/upgrading/upgrading_v4.md`), including a rename-cheat-sheet
entry.
- Custom HTTP clients must now **extend `BaseHttpClient`** from
`@crawlee/http-client` rather than just implementing the interface (all
shipped clients already do; `LazyDefaultHttpClient` was converted). Same
applies to test mocks — `Object.create(BaseHttpClient.prototype)` works.
- One caveat of consuming parse results: zod object schemas return a
pruned plain copy, so options holding class instances are validated with
passthrough schemas (`z.custom`-based) to keep their prototypes — there
are comments at the relevant schemas.
- Fixes a few latent gaps surfaced along the way: `Request.state` now
accepts `RequestState.SKIPPED` (validated via `z.enum(RequestState)`),
and the publish-time catalog inlining covers `optionalDependencies`.
- `ArgumentValidationError` and its formatter are intentionally kept
close to the copy in apify/apify-client-js#986 — a follow-up may extract
them into a shared package.
---------
Co-authored-by: Martin Adámek <banan23@gmail.com>
`scripts/copy.ts` rewrites the version of every `@crawlee/*` dependency
(and `crawlee`) to the monorepo version during canary publish and
`pin-versions`. But `@crawlee/fs-storage-native` is an externally
published package (pinned to `0.1.5-beta.18` in `@crawlee/fs-storage`)
that is **not** versioned in lockstep with the monorepo, so rewriting it
produced a non-existent version and broke CI publishing.
This excludes `@crawlee/fs-storage-native` from the version-rewrite loop
in both the `canary` and `pin-versions` branches.
BREAKING CHANGE:
The project is now native ESM without a CJS alternative. This is fine since all supported node versions allow `require(esm)`.
Also all the dependencies are updated to the latest versions, including cheerio v1.
Martin Adámek