Bumps typescript to `^7.0.2` (native compiler) for building and
type-aware linting, together with `oxlint-tsgolint@^7.0.2001`, which
version-tracks TS 7.
The docs build stays on TS 6: typedoc needs the TypeScript JS API, which
the native compiler no longer exposes. The website keeps its own
`typescript: ^6.0.0` pin and pnpm nests that copy under
`node_modules/typedoc`, so typedoc resolves 6.x while the packages
compile with TS 7.
TS 7 also caught one real issue in the tests: `RequestList#initialize()`
is private, and the call in `request_list.test.ts` now needs the same
`@ts-expect-error` the surrounding test already uses for the private
constructor.
The api-extractor runner script also imports `typescript` for its report
parsing; that import now goes through a `typescript-v6` alias
(`npm:typescript@^6.0.0`), since the parsing needs the classic JS API
(api-extractor itself bundles its own TypeScript and is unaffected). The
regenerated reports in `docs/public-api/` pick up TS 7's single-quoted
string literal types in three packages, 4 lines total.
Same change as apify/apify-sdk-js#695. Build, docs build, type-aware
lint, test typecheck and the unit suite all pass.
The statistics record is persisted during crawler teardown and the platform
key-value store is eventually consistent, so reading it immediately after the
run can miss it. That crashed the whole test with a TypeError on
stats.requestsFinished (seen in cheerio-curl-impersonate-ts) even though the
actor run itself succeeded. The lookup now retries for up to ~30 seconds and
falls back to an empty object, so a genuinely missing record fails the
assertions cleanly.
Cloudflare serves its challenge pages with a 403 status. On v3,
handleCloudflareChallenge() received the session and removed 403 from the
session pool's blocked status codes itself; v4 dropped that mechanism when the
hook was redesigned, so challenged requests died in throwOnBlockedRequest()
on every retry and the solver only ever got a single attempt. Solving the
challenge is probabilistic, which is why the fixture passes on master (where
retries reach the solver) and kept failing here.
blockedStatusCodes is a public crawler option in v4, so the fixture opts out
of 403 explicitly. Whether handleCloudflareChallengeHook() should handle this
automatically again is a follow-up design question.
camoufox-js releases bundle a specific Firefox build that must match the one
expected by the pinned playwright version (0.11 ↔ 1.60). The fixture pinned
camoufox-js ^0.12.0 next to playwright 1.60.0, so the Cloudflare challenge
kept failing on the platform even with the updated challenge markup handling
(#4019) in place — master validates that fix with the 0.11/1.60 pairing.
Two upstream v4 changes landed after the last E2E run and broke the suite:
- the enqueueLinks split (#4010) changed the context helper's return value to
the addRequestsBatched result, so the *-enqueue-links fixtures now assert on
`addedRequests` being empty instead of deep-equality with the old shape
- the zod validation unification (#3935) introduced a `catalog:` dependency,
which npm cannot resolve when the platform builds the actor image; the E2E
package-copy step now rewrites catalog deps to their pinned versions from
pnpm-workspace.yaml, the same way it already rewrites `workspace:` deps
The transplanted enqueueLinks split reverted a few master-carried behaviors in
BasicCrawler; this restores them on top of the new design:
- stop capturing statistics before teardown again, so the crawler state is
saved before the final persistence event fires (prevents double persistence)
- teardown() only emits an explicit PERSIST_STATE event for externally-managed
event managers, and tears the owned session pool down with persistState
matching event manager ownership (an unset flag previously fell back to the
`persistState = true` default, double-persisting the pool)
- the enqueue limit log distinguishes an explicit `limit` from the remaining
maxRequestsPerCrawl budget again
- adapt the master-carried tests to the addRequests() API; drop the
explicit-undefined override tests for options that no longer exist on it
Moves the `requestManager`-bound enqueueing logic into
`BasicCrawlerContext.addRequests`, and each DOM-aware crawler now
exposes its own `extractLinks()` plus an `enqueueLinks()` that composes
`extractLinks` + `addRequests`.
This aligns the JS implementation with what Python does, to some extent.
Closes#3081
- markRequestHandled → markRequestAsHandled on SitemapRequestLoader
- await the now-async RequestQueue.getTotalCount()
- transformRequestFunction skips now report the dedicated 'transform' reason
- robots.txt mock needs getCrawlDelay
- statistics/session-pool single-persistence tests observe KeyValueStore.setValue
instead of the persistState methods RecoverableState replaced
- pass an explicit logger to Sitemap.load in the aggregated-warning test
- requestQueue → requestManager in enqueueLinks options
- await the now-async createRequestQueueMock
- sessionPoolOptions → sessionPool instance in the redirect-cookie test
- config → configuration in purgeDefaultStorages options
- SitemapRequestList → SitemapRequestLoader in loader tests
- handleCloudflareChallenge lost its session parameter in v4
- drop duplicate imports
Replaces the remaining `ow`-based argument validation with `zod` across
all packages and reworks how validation results are consumed and
reported.
Closes#3716
- **`ow` is gone** — every argument check now goes through
`parseArgument(value, schema, label?)` from `@crawlee/utils`, backed by
shared zod schemas (`schemas`, exported via `@crawlee/utils/internal`).
The `@sapphire/shapeshift` checks in `@crawlee/fs-storage` were
converted too, so a single validation library remains.
- **Parse results are used everywhere** — option defaults moved from
destructuring into the schemas (`.default(...)`), and call sites
destructure the typed parse result. `parseArgument` returns `TValue &
z.output<TSchema>`, so call sites keep their declared TS types while
gaining the defaults.
- **Schemas are built once** — all per-call schemas are hoisted to
module scope; crawler/launcher classes build their strict options schema
once as a `static optionsSchema` next to `optionsShape`. The
`urlPatternSchema` for `include`/`exclude` lives in
`enqueue_links/shared.ts`, next to the type it validates.
- **Specific validators instead of `anyObject`** — class-typed options
use `z.instanceof(...)` (`BaseHttpClient`, `Configuration`,
`EventManager`), interface-typed ones use duck-typed `objectWithKeys`
validators (`storageBackend`, `requestManager`, `logger`, …), and
element-typed arrays use the new `schemas.arrayOf(item, 'numbers')`.
`ArgumentValidationError` (replacing ow's `ArgumentError`) renders one
line per issue: the expected type, the received type and value folded
into one clause, the offending field path, and the validated interface:
```text
// v3 (ow) — first issue only
Expected property `maxRequestRetries` to be of type `number` but received type `string` in object `HttpCrawlerOptions`
// v4 (zod) — every issue, one line each
Invalid input: expected number, received the string `many` at `maxRequestRetries` in `HttpCrawlerOptions`
Invalid input: expected an array of numbers, received the number `500` at `additionalHttpErrorStatusCodes` in `HttpCrawlerOptions`
Invalid input: expected boolean, received the string `yes` at `retryOnBlocked` in `HttpCrawlerOptions`
```
Details worth knowing:
- Union failures expand into one line per failed arm (zod's own message
is a bare "Invalid input").
- `NaN` is named as itself, an empty string renders as `''`, and arrays
name their element type (``expected an array of URL patterns``) — none
of which ow or stock zod reported.
- `new Request('https://…')` gets a targeted hint pointing at the `{ url
}` object form.
- For programmatic handling, the error exposes zod's structured output:
`error.issues` and the raw `ZodError` as a typed `cause`.
The migration is documented in the v4 upgrading guide
(`docs/upgrading/upgrading_v4.md`), including a rename-cheat-sheet
entry.
- Custom HTTP clients must now **extend `BaseHttpClient`** from
`@crawlee/http-client` rather than just implementing the interface (all
shipped clients already do; `LazyDefaultHttpClient` was converted). Same
applies to test mocks — `Object.create(BaseHttpClient.prototype)` works.
- One caveat of consuming parse results: zod object schemas return a
pruned plain copy, so options holding class instances are validated with
passthrough schemas (`z.custom`-based) to keep their prototypes — there
are comments at the relevant schemas.
- Fixes a few latent gaps surfaced along the way: `Request.state` now
accepts `RequestState.SKIPPED` (validated via `z.enum(RequestState)`),
and the publish-time catalog inlining covers `optionalDependencies`.
- `ArgumentValidationError` and its formatter are intentionally kept
close to the copy in apify/apify-client-js#986 — a follow-up may extract
them into a shared package.
---------
Co-authored-by: Martin Adámek <banan23@gmail.com>
Gets the E2E test suite running against v4. The suite hadn't been run
since the v4 rewrite and everything failed on startup. After these
changes the MEMORY run passes locally end to end, and getting there
surfaced a few real regressions in the packages themselves.
- `LinkeDOMCrawler`'s `enqueueLinks` helper referenced the global
`document` (which doesn't exist in Node) instead of the parsed window,
so every call crashed at runtime.
- `ErrorSnapshotter.saveHTMLSnapshot()` returned the record key with a
v3-style `.html` suffix, so the follow-up `getPublicUrl()` lookup missed
and `firstErrorHtmlUrl` never made it into the crawler statistics.
- `JSDOMCrawlingContext`/`LinkeDOMCrawlingContext` didn't override
`enqueueLinks`, exposing the strict urls-required signature even though
the runtime helper extracts URLs from the parsed document.
- `LinkeDOMCrawler` can now be constructed without arguments, like the
other crawlers.
- Bumped the pinned `apify` SDK to 4.0.0-beta.22 (beta.19 imports
`snakeCaseToCamelCase` from `@crawlee/utils`, which no longer exists
there).
- Adapted `tools.mjs` to the fs-storage on-disk layout (extensionless
key-value records; the short-lived `__default__` directory alias it
originally targeted was a bug, fixed in #4013) and to the
`@crawlee/utils` exports split.
- Migrated test actors to the v4 APIs: the `logger` option with
`ApifyLogAdapter` instead of `log`, hooks reading `gotoOptions` from the
crawling context, `session.setCookie()`, a custom `SessionPool` instead
of `sessionPoolOptions`, the WHATWG `Response` returned by
`sendRequest`, `registerDeferredCleanup` for dataset writes that must
survive a throwing handler, and explicit enqueue strategies now that
`include` globs are ANDed with the default same-hostname strategy.
- The ignore-ssl test now configures TLS verification on the http
client, because the crawler-level `ignoreSslErrors` option is not wired
to the default client in v4. That dangling option deserves a separate
fix or removal, since it currently does nothing.
- The impit test pins session fingerprints, since the random default
fingerprint overrides the client's browser impersonation.
- Added ES2022 to the actor tsconfigs' `lib` (a bare `["DOM"]` drops the
ES lib and broke compilation on `ErrorOptions`).
- Skipped the zero-concurrency queue test: it stages a stuck queue
through the v3 client-side `inProgress` set, which the rewritten queue
doesn't have.
- Fixed the camoufox fetch retry loop fetching 5x even on success, and
removed a duplicate `apify` dependency key that silently downgraded the
curl-impersonate actor to SDK v3.
- Commented out the LOCAL storage matrix entry in the workflow, as
`@apify/storage-local` doesn't support v4.
- closes#3998
Unnamed storages (default and aliased) are now purged on start, named
ones are not — crawlee-python's rule. The fs backend sweeps the storage
directories, so leftovers from a previous process are caught too.
Two adjacent bugs, one commit each:
- `createDatasetBackend()` / `({})` didn't open the default storage,
though `StorageIdentifier` says they do
- the `__default__` sentinel leaked into the directory name, so default
storages lived in `storage/datasets/__default__` rather than `default`
`ThrottlingRequestManager` (#3741) sub-queues are alias-keyed, so they
now only survive a restart with `purgeOnStart` off — as its docs already
said.
The `ignoreSslErrors` option stopped working during the v4 HTTP client
interface rework: it was still folded into got-style request options,
but those never reach `httpClient.sendRequest()`. In v3 the option
works, and actors commonly expose it in their input schemas and pass it
into crawler options (e.g. actor-scraper), so this keeps it working
instead of removing it.
The option is renamed to `ignoreTlsErrors`, matching
`session.proxyInfo.ignoreTlsErrors`, the browser pool, and the impit
client (the old name is dropped, documented in the upgrading guide;
actors migrating to v4 rename it in their own code, the SDK does not
touch this option). The crawler forwards it (still defaulting to `true`,
same as v3) as a new `SendRequestOptions.ignoreTlsErrors` flag, which
`BaseHttpClient` also enables for MITM proxy sessions (previously
equally dead). The impit client honors the flag; for custom clients it
is best effort, and `FetchHttpClient` cannot disable TLS verification at
all. The dead plumbing in `getRequestOptions()` is removed and unit
tests cover the forwarding chain.
Adds a `/internal` subpath export to `@crawlee/utils` for symbols that
are only shared between crawlee packages (CSS selectors, type guards,
URL helpers, `CheerioRoot`, etc.), keeping the root entry point for
user-facing utilities. The `crawlee` meta-package naturally only
re-exports the public surface.
Closes#3079
`PlaywrightBrowser._setBrowserType()` has existed since the initial
crawlee commit but was never called, so the `_browserType` field stayed
`undefined` and `browserType()` returned `undefined` at runtime despite
its non-optional `BrowserType` signature. Persistent contexts are the
default (`useIncognitoPages: false`), so a call like
`browser.browserType().name()` on the wrapper threw a `TypeError`. The
incognito and remote-connection paths return the native Playwright
`Browser` and were not affected.
This wires the existing setter in `PlaywrightPlugin._launch()` where the
wrapper is created. `this.library` is the `BrowserType` that launched
the context, so the wrapper now provides the consistent API with
Playwright's `Browser` that its docblock describes. Also adds a test
covering both the persistent-context wrapper and the native incognito
browser.
Closes#3945.
`@crawlee/types` no longer imports `CookieJar`/`SerializedCookieJar`
from `tough-cookie` - it declares its own structurally-compatible
interfaces instead, so `tough-cookie` is no longer part of its
dependency tree. `tough-cookie` stays a direct dependency everywhere
it's actually used (`core`, `http-client`, `impit-client`).
`closeCookieModals` loads `idcac-playwright` through a guarded dynamic
import, and both crawler packages already declare it as an optional peer
dependency — but it was also listed in `dependencies`, so it got
installed for everyone anyway. The e2e actors that call
`closeCookieModals` now depend on it explicitly.
`playwright-crawler` also had a `^0.1.3` dependency against a `^0.2.0`
peer range.
Adopts native `#` private fields for private class properties across all
packages and removes the remaining `_` prefixes from protected/private
members, as agreed in #3108.
- Around 300 private properties are now native `#` fields, so they are
hard-private at runtime and invisible to spread and `JSON.stringify`.
Private and protected methods keep the `private`/`protected` keyword and
lose the `_` prefix.
- Renamed protected extension points: `_init` → `init`,
`_throwOnBlockedRequest` → `throwOnBlockedRequest`,
`_getMessageFromError` → `getMessageFromError` and
`_getCookieHeaderFromRequest` → `getCookieHeaderFromRequest` on
`BasicCrawler`; `_navigationHandler` → `navigationHandler` on
`BrowserCrawler` and its subclasses; `_addProxyToLaunchOptions` →
`addProxyToLaunchOptions`, `_isChromiumBasedBrowser` →
`isChromiumBasedBrowser`, `_connectToRemoteBrowser` →
`connectToRemoteBrowser` and `_throwAugmentedLaunchError` →
`throwAugmentedLaunchError` on `BrowserPlugin`. The `@internal`
`LaunchContext._remoteToken` and `PlaywrightBrowser._setBrowserType`
became `remoteToken` and `setBrowserType`.
- `BrowserPlugin._launch` and
`BrowserController._close`/`_kill`/`_newPage`/`_getCookies`/`_setCookies`
keep the underscore, since their public wrapper methods own the plain
names. `Readable._read` is a Node contract.
- Tests that used to poke internal state were rewritten against public
API where the test's meaning survives: session pool state via
`getState()`, request list persistence via the exported persistence
keys, proxy rotation via `newUrl()`, autoscaling options via the
constructor, crawler cleanup via `teardown()`. The members those tests
reached are now `#` fields too.
- The members still declared with TypeScript's `private` fall into two
groups, each with a one-line comment saying why: runtime hazards for `#`
(the adaptive crawler's log proxy, two cross-class accesses), and
genuine test injection seams (backend delegation spies, interval
replacement, static counter resets, config-echo assertions with no
public surface).
- `LaunchContext.extend()` now rejects all declared fields and accessors
as reserved names; it previously missed fields declared after the
reserved list was computed. The fingerprinting hook assigns the declared
`fingerprint` field directly instead of going through `extend()`.
- A new `no-underscore-dangle` oxlint rule (with `enforceInClassFields`
and `enforceInMethodNames`) enforces the convention.
- The upgrading guide documents the renames and the `#` semantics
change; API reports regenerated.
Closes#3108
Navigation hooks on browser crawlers could not be typed with custom
`userData`: a hook declared as `(ctx:
PuppeteerCrawlingContext<MyUserData>) => ...` failed to type-check
against `preNavigationHooks`/`postNavigationHooks`, while the HTTP
crawler family already supports this (#2063).
- `BrowserCrawlingContext`, `PlaywrightCrawlingContext`,
`PuppeteerCrawlingContext`, `StagehandCrawlingContext` and
`AdaptivePlaywrightCrawlerContext` now default `UserData` to `any`, the
same pattern (and marker comment) the HTTP family adopted in v4.
- `PlaywrightHook`, `PuppeteerHook` and `StagehandHook` are now type
aliases generic over `UserData` (e.g. `PlaywrightHook<MyUserData>`),
like `CheerioHook` and `HttpHook`.
- `HttpCrawlerOptions.preNavigationHooks` now uses
`CrawlingContext<any>`, so pre-navigation hooks typed with custom user
data (`InternalHttpHook<CrawlingContext<MyUserData>>`) are assignable as
well; the post-navigation option already allowed this.
- The same default change makes request handlers typed with custom user
data assignable to untyped crawler options as well.
- Adds type-level regression tests (hooks and request handlers), an
upgrading guide entry for the switch from interfaces to type aliases,
and regenerated API snapshots.
Closes#2063
The test 'BrowserCrawler > should allow modifying gotoOptions by pre
navigation hooks' intermittently failed with `expected 59999 to deeply
equal 60000`.
The hook set `gotoOptions.timeout = 60000`, which is exactly the default
`navigationTimeoutMillis` for browser crawlers. The crawler detects "the
hook did not override the timeout" by value equality with that default,
so the test's value was treated as untouched and clamped to the
remaining navigation window (`60000 - elapsed ms`). The exact assertion
was therefore a race: a millisecond of scheduling delay before
`navigate()` produced 59999, and adding a 5 ms delay to the hook chain
reproduced the failure on every run.
The hook now sets a value distinct from the default (25000), which the
crawler honors verbatim, so the assertion is deterministic (verified
stable across repeated runs, including with an artificial delay). A
comment in the test explains why the value must differ from the default.
The clamp path itself is already covered by the slow-navigation tests in
`playwright_crawler.test.ts`.
Align `EnqueueLinksOptions` with crawlee-python (#3409):
- Replace `globs`, `regexps`, `pseudoUrls` options with
`include`/`exclude` accepting `UrlPatternInput[]`
- Strip request options (label, method, payload, userData, headers) from
pattern objects — patterns are pure URL matchers
- `transformRequestFunction` is now the only way to customize
per-request options, runs after all filtering
- Add `'skip'` and `'unchanged'` return values to `RequestTransform`
(aligned with Python's `RequestTransformAction`)
- Apply same changes to `enqueueLinksByClickingElements` (Playwright +
Puppeteer) and `SitemapRequestList`
- Remove `@apify/pseudo_url` dependency and `PseudoUrl` re-export
- Update all templates from `globs` to `include`
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Timeout redesign (#2951)
v4 had already dropped the old `navigationTimeoutSecs +
requestHandlerTimeoutSecs + buffer` sum, so the request handler timeout
covers only the user's function and the confusing "timed out after 130
seconds" messages are gone. What it did not do is put back the pieces
that sum incidentally bounded.
- **The navigation phase is one window.** `navigationTimeoutSecs` covers
the `preNavigationHooks`, the navigation itself, and the
`postNavigationHooks` as a single shared budget, matching Crawlee for
Python. A hook that hangs no longer stalls the request forever; it eats
into the same window the navigation uses. (This replaces an earlier
attempt at a separate per-hook `navigationHooksTimeoutSecs`, which is
gone.)
- **A whole-request backstop.** The phases between the timed ones
(`extendContext`, the robots.txt check, response processing) could hang
indefinitely. An internal backstop now bounds the whole request. It is
sized to outlast the phases that have their own timeout (navigation plus
the handler), so a legitimately slow request is never cut short and it
only fires when something is genuinely stuck. It is configured with
`CRAWLEE_INTERNAL_TIMEOUT`, now resolved through `Configuration` like
the other env-backed options. Set it below the phase timeouts and the
crawler raises it per request and warns at startup, rather than cutting
a phase short.
## `context.extendTimeout()`
When the time needed is only apparent once a hook or handler is already
running, `context.extendTimeout(secs)` buys more. From inside the
navigation phase it pushes the shared navigation window; from the
request handler it pushes the handler timeout. Either way it also pushes
the backstop and raises the request-manager reservation, so the extra
time is neither clipped by the backstop nor undone by a locking backend
handing the request out again.
## Per-route timeouts (#1485)
```ts
router.addHandler('LIST', handler, { requestHandlerTimeoutSecs: 120 });
router.addHandler('DETAIL', handler); // keeps the crawler's default
```
`requestHandlerTimeoutSecs` is unchanged and stays the default for
anything a route does not override; a route's value may be longer or
shorter than it. The label is known before the handler starts, so the
timeout is resolved per request and the router never reaches back into
the crawler mid-flight. The backstop and the reservation both account
for the longest route in play.
## Browser navigation
A `preNavigationHooks` hook can still override `gotoOptions.timeout`
(including `0`, Playwright's "no timeout"); the shared window no longer
clamps it to 1ms or discards a larger value. A navigation timeout,
whether ours or the driver's own, is reported as `Navigation timed out
after N seconds` instead of the driver's raw millisecond value.
## Adaptive crawler
`AdaptivePlaywrightCrawler` runs the handler up to twice per request (a
static attempt falling through to the browser). A
`getRequestHandlerRunCount` hook (2 for adaptive, 1 everywhere else)
sizes the whole-request budgets for both runs, while each run keeps its
own handler window. This replaces an earlier one-off doubling that
missed per-route overrides.
## Notes on the implementation
The backstop is a bare timer, not `addTimeoutToPromise`: nested
`addTimeoutToPromise` calls share one `AbortController`, so wrapping the
whole request in one would let the handler timing out abort the outer
context and cancel the error handling that reclaims the request. The
plumbing lives in its own `request-backstop.ts` module.
For the same reason the HTTP navigation binds its request to the
navigation frame's `@apify/timeout` cancel signal rather than a fixed
`AbortSignal.timeout`: the response body is read lazily, after the
post-navigation hooks, so a fixed timer would abort a body a hook is
legitimately still keeping alive via `extendTimeout`. A genuine
navigation timeout still fails the request; the body read is bounded at
the parse step.
Depends on the `extendTimeout` addition in `@apify/timeout`
(apify/apify-shared-js#669), released as 0.4.4 and consumed here.
Closes#1485Closes#2951
These `@internal` async iterable helpers are only used by
`@crawlee/core`, so they no longer need to live in `@crawlee/utils`.
Common symbols used in multiple `@crawlee/` packages have been left in
the `@crawlee/utils` package.
`Cookie.parse` returns `undefined` for unparseable fragments (e.g. a
bare name), which the `!` assertion then dereferenced and threw; skip
and warn instead.
Moves the `@ignore`-d `isStream`, `isBuffer`, `toBuffer` and
`weightedAvg` helpers out of `@crawlee/utils` and into `@crawlee/core`
as package-private internals, since core is their only consumer.
Related to #3079
These symbols were plain re-exports of `@crawlee/types` and duplicates
of helpers already living in `@crawlee/core`, so consumers now import
them from their real home.
v3 used `config` and `configuration` interchangeably across the public
API. This settles on `configuration` everywhere, matching Crawlee for
Python.
Renamed methods:
| Before | After |
| --- | --- |
| `Configuration.getGlobalConfig()` |
`Configuration.getGlobalConfiguration()` |
| `LocalEventManager.fromConfig()` |
`LocalEventManager.fromConfiguration()` |
Renamed `config` options to `configuration` in `StorageOpenOptions`
(`Dataset.open()`, `KeyValueStore.open()`, `RequestQueue.open()`),
`UseStateOptions`, `purgeDefaultStorages()`, `SnapshotterOptions`,
`SaveSnapshotOptions` (Playwright and Puppeteer),
`RecoverableStateOptions`, `RequestListOptions`, `CpuLoadSignalOptions`
and `MemoryLoadSignalOptions`.
Renamed `config` properties to `configuration` on `Dataset`,
`KeyValueStore`, `Snapshotter` and `BrowserLauncher` (plus the
Playwright, Puppeteer and Stagehand launchers).
The `configuration` crawler option and
`serviceLocator.getConfiguration()` / `setConfiguration()` were already
consistent and are unchanged.
The BC is documented in the v4 upgrading guide.
Closes#3706
Infers the router's `label → userData` route map on the crawler and uses
it to type the request **inputs** — `crawler.run` /
`crawler.addRequests` and the `addRequests` / `enqueueLinks` context
helpers. Targets `v4`.
> The router core this builds on — the typed route map (#3747) and
per-label [Standard Schema](https://standardschema.dev) validation with
`RequestValidationError` (#3851) — already merged to `master` and is
present on `v4`. This PR is the remaining `v4`-only piece: propagating
that route map to the crawler and context request methods (a breaking
change to those signatures, hence `v4`).
When a typed router is passed as `requestHandler`, providing a declared
`label` requires the matching `userData` shape and rejects unknown
labels; unlabeled requests keep loose `userData` (they hit the default
handler).
- **Handler context** — `ctx.addRequests` / `ctx.enqueueLinks` are typed
from the route map. Driven by the router itself, so it works for
**every** crawler type.
- **Crawler instance** — `Routes` is inferred from the `requestHandler`
option and types `crawler.addRequests` / `crawler.run`. Threaded through
all crawler classes (Basic/Http/Cheerio/JSDOM/LinkeDOM and the browser
family: Browser/Playwright/Puppeteer/Stagehand/AdaptivePlaywright).
```ts
const crawler = new PlaywrightCrawler({ requestHandler: router });
await crawler.addRequests([{ url, label: 'PRODUCT', userData: { sku: 's', price: 1 } }]); // ✅
await crawler.addRequests([{ url, label: 'PRODUCT', userData: { sku: 1 } }]); // ❌ wrong userData
await crawler.addRequests([{ url, label: 'NOPE' }]); // ❌ unknown label
```
Fully backwards compatible: without a typed router the request inputs
stay loosely typed, exactly as before.
Relates to #3082
Harry