Bumps typescript to `^7.0.2` (native compiler) for building and
type-aware linting, together with `oxlint-tsgolint@^7.0.2001`, which
version-tracks TS 7.
The docs build stays on TS 6: typedoc needs the TypeScript JS API, which
the native compiler no longer exposes. The website keeps its own
`typescript: ^6.0.0` pin and pnpm nests that copy under
`node_modules/typedoc`, so typedoc resolves 6.x while the packages
compile with TS 7.
TS 7 also caught one real issue in the tests: `RequestList#initialize()`
is private, and the call in `request_list.test.ts` now needs the same
`@ts-expect-error` the surrounding test already uses for the private
constructor.
The api-extractor runner script also imports `typescript` for its report
parsing; that import now goes through a `typescript-v6` alias
(`npm:typescript@^6.0.0`), since the parsing needs the classic JS API
(api-extractor itself bundles its own TypeScript and is unaffected). The
regenerated reports in `docs/public-api/` pick up TS 7's single-quoted
string literal types in three packages, 4 lines total.
Same change as apify/apify-sdk-js#695. Build, docs build, type-aware
lint, test typecheck and the unit suite all pass.
Lost in the v4 rebase; matches the createHttpRouter/createCheerioRouter
overload set. Also removes the rebase reconciliation checklist, which is
fully resolved by this commit.
Moves the `requestManager`-bound enqueueing logic into
`BasicCrawlerContext.addRequests`, and each DOM-aware crawler now
exposes its own `extractLinks()` plus an `enqueueLinks()` that composes
`extractLinks` + `addRequests`.
This aligns the JS implementation with what Python does, to some extent.
Closes#3081
Replaces the remaining `ow`-based argument validation with `zod` across
all packages and reworks how validation results are consumed and
reported.
Closes#3716
- **`ow` is gone** — every argument check now goes through
`parseArgument(value, schema, label?)` from `@crawlee/utils`, backed by
shared zod schemas (`schemas`, exported via `@crawlee/utils/internal`).
The `@sapphire/shapeshift` checks in `@crawlee/fs-storage` were
converted too, so a single validation library remains.
- **Parse results are used everywhere** — option defaults moved from
destructuring into the schemas (`.default(...)`), and call sites
destructure the typed parse result. `parseArgument` returns `TValue &
z.output<TSchema>`, so call sites keep their declared TS types while
gaining the defaults.
- **Schemas are built once** — all per-call schemas are hoisted to
module scope; crawler/launcher classes build their strict options schema
once as a `static optionsSchema` next to `optionsShape`. The
`urlPatternSchema` for `include`/`exclude` lives in
`enqueue_links/shared.ts`, next to the type it validates.
- **Specific validators instead of `anyObject`** — class-typed options
use `z.instanceof(...)` (`BaseHttpClient`, `Configuration`,
`EventManager`), interface-typed ones use duck-typed `objectWithKeys`
validators (`storageBackend`, `requestManager`, `logger`, …), and
element-typed arrays use the new `schemas.arrayOf(item, 'numbers')`.
`ArgumentValidationError` (replacing ow's `ArgumentError`) renders one
line per issue: the expected type, the received type and value folded
into one clause, the offending field path, and the validated interface:
```text
// v3 (ow) — first issue only
Expected property `maxRequestRetries` to be of type `number` but received type `string` in object `HttpCrawlerOptions`
// v4 (zod) — every issue, one line each
Invalid input: expected number, received the string `many` at `maxRequestRetries` in `HttpCrawlerOptions`
Invalid input: expected an array of numbers, received the number `500` at `additionalHttpErrorStatusCodes` in `HttpCrawlerOptions`
Invalid input: expected boolean, received the string `yes` at `retryOnBlocked` in `HttpCrawlerOptions`
```
Details worth knowing:
- Union failures expand into one line per failed arm (zod's own message
is a bare "Invalid input").
- `NaN` is named as itself, an empty string renders as `''`, and arrays
name their element type (``expected an array of URL patterns``) — none
of which ow or stock zod reported.
- `new Request('https://…')` gets a targeted hint pointing at the `{ url
}` object form.
- For programmatic handling, the error exposes zod's structured output:
`error.issues` and the raw `ZodError` as a typed `cause`.
The migration is documented in the v4 upgrading guide
(`docs/upgrading/upgrading_v4.md`), including a rename-cheat-sheet
entry.
- Custom HTTP clients must now **extend `BaseHttpClient`** from
`@crawlee/http-client` rather than just implementing the interface (all
shipped clients already do; `LazyDefaultHttpClient` was converted). Same
applies to test mocks — `Object.create(BaseHttpClient.prototype)` works.
- One caveat of consuming parse results: zod object schemas return a
pruned plain copy, so options holding class instances are validated with
passthrough schemas (`z.custom`-based) to keep their prototypes — there
are comments at the relevant schemas.
- Fixes a few latent gaps surfaced along the way: `Request.state` now
accepts `RequestState.SKIPPED` (validated via `z.enum(RequestState)`),
and the publish-time catalog inlining covers `optionalDependencies`.
- `ArgumentValidationError` and its formatter are intentionally kept
close to the copy in apify/apify-client-js#986 — a follow-up may extract
them into a shared package.
---------
Co-authored-by: Martin Adámek <banan23@gmail.com>
Gets the E2E test suite running against v4. The suite hadn't been run
since the v4 rewrite and everything failed on startup. After these
changes the MEMORY run passes locally end to end, and getting there
surfaced a few real regressions in the packages themselves.
- `LinkeDOMCrawler`'s `enqueueLinks` helper referenced the global
`document` (which doesn't exist in Node) instead of the parsed window,
so every call crashed at runtime.
- `ErrorSnapshotter.saveHTMLSnapshot()` returned the record key with a
v3-style `.html` suffix, so the follow-up `getPublicUrl()` lookup missed
and `firstErrorHtmlUrl` never made it into the crawler statistics.
- `JSDOMCrawlingContext`/`LinkeDOMCrawlingContext` didn't override
`enqueueLinks`, exposing the strict urls-required signature even though
the runtime helper extracts URLs from the parsed document.
- `LinkeDOMCrawler` can now be constructed without arguments, like the
other crawlers.
- Bumped the pinned `apify` SDK to 4.0.0-beta.22 (beta.19 imports
`snakeCaseToCamelCase` from `@crawlee/utils`, which no longer exists
there).
- Adapted `tools.mjs` to the fs-storage on-disk layout (extensionless
key-value records; the short-lived `__default__` directory alias it
originally targeted was a bug, fixed in #4013) and to the
`@crawlee/utils` exports split.
- Migrated test actors to the v4 APIs: the `logger` option with
`ApifyLogAdapter` instead of `log`, hooks reading `gotoOptions` from the
crawling context, `session.setCookie()`, a custom `SessionPool` instead
of `sessionPoolOptions`, the WHATWG `Response` returned by
`sendRequest`, `registerDeferredCleanup` for dataset writes that must
survive a throwing handler, and explicit enqueue strategies now that
`include` globs are ANDed with the default same-hostname strategy.
- The ignore-ssl test now configures TLS verification on the http
client, because the crawler-level `ignoreSslErrors` option is not wired
to the default client in v4. That dangling option deserves a separate
fix or removal, since it currently does nothing.
- The impit test pins session fingerprints, since the random default
fingerprint overrides the client's browser impersonation.
- Added ES2022 to the actor tsconfigs' `lib` (a bare `["DOM"]` drops the
ES lib and broke compilation on `ErrorOptions`).
- Skipped the zero-concurrency queue test: it stages a stuck queue
through the v3 client-side `inProgress` set, which the rewritten queue
doesn't have.
- Fixed the camoufox fetch retry loop fetching 5x even on success, and
removed a duplicate `apify` dependency key that silently downgraded the
curl-impersonate actor to SDK v3.
- Commented out the LOCAL storage matrix entry in the workflow, as
`@apify/storage-local` doesn't support v4.
- closes#3998
Unnamed storages (default and aliased) are now purged on start, named
ones are not — crawlee-python's rule. The fs backend sweeps the storage
directories, so leftovers from a previous process are caught too.
Two adjacent bugs, one commit each:
- `createDatasetBackend()` / `({})` didn't open the default storage,
though `StorageIdentifier` says they do
- the `__default__` sentinel leaked into the directory name, so default
storages lived in `storage/datasets/__default__` rather than `default`
`ThrottlingRequestManager` (#3741) sub-queues are alias-keyed, so they
now only survive a restart with `purgeOnStart` off — as its docs already
said.
The `ignoreSslErrors` option stopped working during the v4 HTTP client
interface rework: it was still folded into got-style request options,
but those never reach `httpClient.sendRequest()`. In v3 the option
works, and actors commonly expose it in their input schemas and pass it
into crawler options (e.g. actor-scraper), so this keeps it working
instead of removing it.
The option is renamed to `ignoreTlsErrors`, matching
`session.proxyInfo.ignoreTlsErrors`, the browser pool, and the impit
client (the old name is dropped, documented in the upgrading guide;
actors migrating to v4 rename it in their own code, the SDK does not
touch this option). The crawler forwards it (still defaulting to `true`,
same as v3) as a new `SendRequestOptions.ignoreTlsErrors` flag, which
`BaseHttpClient` also enables for MITM proxy sessions (previously
equally dead). The impit client honors the flag; for custom clients it
is best effort, and `FetchHttpClient` cannot disable TLS verification at
all. The dead plumbing in `getRequestOptions()` is removed and unit
tests cover the forwarding chain.
Converts all `const enum` declarations (`EventType`,
`BROWSER_POOL_EVENTS`, `BROWSER_CONTROLLER_EVENTS`,
`OperatingSystemsName`, `DeviceCategory`) to regular enums. `const enum`
breaks downstream consumers using `isolatedModules` or transpilers that
don't inline them (Babel, esbuild, vitest).
Closes#3125
Types the default cookie jar against `@crawlee/types`' `CookieJar`
interface and dynamically imports `tough-cookie` only when a default jar
actually needs constructing.
Adds a `/internal` subpath export to `@crawlee/utils` for symbols that
are only shared between crawlee packages (CSS selectors, type guards,
URL helpers, `CheerioRoot`, etc.), keeping the root entry point for
user-facing utilities. The `crawlee` meta-package naturally only
re-exports the public surface.
Closes#3079
Closes#3945.
`@crawlee/types` no longer imports `CookieJar`/`SerializedCookieJar`
from `tough-cookie` - it declares its own structurally-compatible
interfaces instead, so `tough-cookie` is no longer part of its
dependency tree. `tough-cookie` stays a direct dependency everywhere
it's actually used (`core`, `http-client`, `impit-client`).
Adopts native `#` private fields for private class properties across all
packages and removes the remaining `_` prefixes from protected/private
members, as agreed in #3108.
- Around 300 private properties are now native `#` fields, so they are
hard-private at runtime and invisible to spread and `JSON.stringify`.
Private and protected methods keep the `private`/`protected` keyword and
lose the `_` prefix.
- Renamed protected extension points: `_init` → `init`,
`_throwOnBlockedRequest` → `throwOnBlockedRequest`,
`_getMessageFromError` → `getMessageFromError` and
`_getCookieHeaderFromRequest` → `getCookieHeaderFromRequest` on
`BasicCrawler`; `_navigationHandler` → `navigationHandler` on
`BrowserCrawler` and its subclasses; `_addProxyToLaunchOptions` →
`addProxyToLaunchOptions`, `_isChromiumBasedBrowser` →
`isChromiumBasedBrowser`, `_connectToRemoteBrowser` →
`connectToRemoteBrowser` and `_throwAugmentedLaunchError` →
`throwAugmentedLaunchError` on `BrowserPlugin`. The `@internal`
`LaunchContext._remoteToken` and `PlaywrightBrowser._setBrowserType`
became `remoteToken` and `setBrowserType`.
- `BrowserPlugin._launch` and
`BrowserController._close`/`_kill`/`_newPage`/`_getCookies`/`_setCookies`
keep the underscore, since their public wrapper methods own the plain
names. `Readable._read` is a Node contract.
- Tests that used to poke internal state were rewritten against public
API where the test's meaning survives: session pool state via
`getState()`, request list persistence via the exported persistence
keys, proxy rotation via `newUrl()`, autoscaling options via the
constructor, crawler cleanup via `teardown()`. The members those tests
reached are now `#` fields too.
- The members still declared with TypeScript's `private` fall into two
groups, each with a one-line comment saying why: runtime hazards for `#`
(the adaptive crawler's log proxy, two cross-class accesses), and
genuine test injection seams (backend delegation spies, interval
replacement, static counter resets, config-echo assertions with no
public surface).
- `LaunchContext.extend()` now rejects all declared fields and accessors
as reserved names; it previously missed fields declared after the
reserved list was computed. The fingerprinting hook assigns the declared
`fingerprint` field directly instead of going through `extend()`.
- A new `no-underscore-dangle` oxlint rule (with `enforceInClassFields`
and `enforceInMethodNames`) enforces the convention.
- The upgrading guide documents the renames and the `#` semantics
change; API reports regenerated.
Closes#3108
Navigation hooks on browser crawlers could not be typed with custom
`userData`: a hook declared as `(ctx:
PuppeteerCrawlingContext<MyUserData>) => ...` failed to type-check
against `preNavigationHooks`/`postNavigationHooks`, while the HTTP
crawler family already supports this (#2063).
- `BrowserCrawlingContext`, `PlaywrightCrawlingContext`,
`PuppeteerCrawlingContext`, `StagehandCrawlingContext` and
`AdaptivePlaywrightCrawlerContext` now default `UserData` to `any`, the
same pattern (and marker comment) the HTTP family adopted in v4.
- `PlaywrightHook`, `PuppeteerHook` and `StagehandHook` are now type
aliases generic over `UserData` (e.g. `PlaywrightHook<MyUserData>`),
like `CheerioHook` and `HttpHook`.
- `HttpCrawlerOptions.preNavigationHooks` now uses
`CrawlingContext<any>`, so pre-navigation hooks typed with custom user
data (`InternalHttpHook<CrawlingContext<MyUserData>>`) are assignable as
well; the post-navigation option already allowed this.
- The same default change makes request handlers typed with custom user
data assignable to untyped crawler options as well.
- Adds type-level regression tests (hooks and request handlers), an
upgrading guide entry for the switch from interfaces to type aliases,
and regenerated API snapshots.
Closes#2063
In v3, browser navigation hooks received the `page.goto()` options as a
second argument (`(crawlingContext, gotoOptions) => ...`). In v4 they
take a single context argument and the options live on
`context.gotoOptions`. The upgrading guide documents the analogous
`HttpCrawler` `gotOptions` change but never mentions the browser-side
one, which every `PlaywrightCrawler`/`PuppeteerCrawler` user with a
two-argument hook hits on upgrade.
Adds a short section with a before/after sample next to the existing
`HttpCrawler` entry.
Align `EnqueueLinksOptions` with crawlee-python (#3409):
- Replace `globs`, `regexps`, `pseudoUrls` options with
`include`/`exclude` accepting `UrlPatternInput[]`
- Strip request options (label, method, payload, userData, headers) from
pattern objects — patterns are pure URL matchers
- `transformRequestFunction` is now the only way to customize
per-request options, runs after all filtering
- Add `'skip'` and `'unchanged'` return values to `RequestTransform`
(aligned with Python's `RequestTransformAction`)
- Apply same changes to `enqueueLinksByClickingElements` (Playwright +
Puppeteer) and `SitemapRequestList`
- Remove `@apify/pseudo_url` dependency and `PseudoUrl` re-export
- Update all templates from `globs` to `include`
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Timeout redesign (#2951)
v4 had already dropped the old `navigationTimeoutSecs +
requestHandlerTimeoutSecs + buffer` sum, so the request handler timeout
covers only the user's function and the confusing "timed out after 130
seconds" messages are gone. What it did not do is put back the pieces
that sum incidentally bounded.
- **The navigation phase is one window.** `navigationTimeoutSecs` covers
the `preNavigationHooks`, the navigation itself, and the
`postNavigationHooks` as a single shared budget, matching Crawlee for
Python. A hook that hangs no longer stalls the request forever; it eats
into the same window the navigation uses. (This replaces an earlier
attempt at a separate per-hook `navigationHooksTimeoutSecs`, which is
gone.)
- **A whole-request backstop.** The phases between the timed ones
(`extendContext`, the robots.txt check, response processing) could hang
indefinitely. An internal backstop now bounds the whole request. It is
sized to outlast the phases that have their own timeout (navigation plus
the handler), so a legitimately slow request is never cut short and it
only fires when something is genuinely stuck. It is configured with
`CRAWLEE_INTERNAL_TIMEOUT`, now resolved through `Configuration` like
the other env-backed options. Set it below the phase timeouts and the
crawler raises it per request and warns at startup, rather than cutting
a phase short.
## `context.extendTimeout()`
When the time needed is only apparent once a hook or handler is already
running, `context.extendTimeout(secs)` buys more. From inside the
navigation phase it pushes the shared navigation window; from the
request handler it pushes the handler timeout. Either way it also pushes
the backstop and raises the request-manager reservation, so the extra
time is neither clipped by the backstop nor undone by a locking backend
handing the request out again.
## Per-route timeouts (#1485)
```ts
router.addHandler('LIST', handler, { requestHandlerTimeoutSecs: 120 });
router.addHandler('DETAIL', handler); // keeps the crawler's default
```
`requestHandlerTimeoutSecs` is unchanged and stays the default for
anything a route does not override; a route's value may be longer or
shorter than it. The label is known before the handler starts, so the
timeout is resolved per request and the router never reaches back into
the crawler mid-flight. The backstop and the reservation both account
for the longest route in play.
## Browser navigation
A `preNavigationHooks` hook can still override `gotoOptions.timeout`
(including `0`, Playwright's "no timeout"); the shared window no longer
clamps it to 1ms or discards a larger value. A navigation timeout,
whether ours or the driver's own, is reported as `Navigation timed out
after N seconds` instead of the driver's raw millisecond value.
## Adaptive crawler
`AdaptivePlaywrightCrawler` runs the handler up to twice per request (a
static attempt falling through to the browser). A
`getRequestHandlerRunCount` hook (2 for adaptive, 1 everywhere else)
sizes the whole-request budgets for both runs, while each run keeps its
own handler window. This replaces an earlier one-off doubling that
missed per-route overrides.
## Notes on the implementation
The backstop is a bare timer, not `addTimeoutToPromise`: nested
`addTimeoutToPromise` calls share one `AbortController`, so wrapping the
whole request in one would let the handler timing out abort the outer
context and cancel the error handling that reclaims the request. The
plumbing lives in its own `request-backstop.ts` module.
For the same reason the HTTP navigation binds its request to the
navigation frame's `@apify/timeout` cancel signal rather than a fixed
`AbortSignal.timeout`: the response body is read lazily, after the
post-navigation hooks, so a fixed timer would abort a body a hook is
legitimately still keeping alive via `extendTimeout`. A genuine
navigation timeout still fails the request; the body read is bounded at
the parse step.
Depends on the `extendTimeout` addition in `@apify/timeout`
(apify/apify-shared-js#669), released as 0.4.4 and consumed here.
Closes#1485Closes#2951
`Cookie.parse` returns `undefined` for unparseable fragments (e.g. a
bare name), which the `!` assertion then dereferenced and threw; skip
and warn instead.
These symbols were plain re-exports of `@crawlee/types` and duplicates
of helpers already living in `@crawlee/core`, so consumers now import
them from their real home.
v3 used `config` and `configuration` interchangeably across the public
API. This settles on `configuration` everywhere, matching Crawlee for
Python.
Renamed methods:
| Before | After |
| --- | --- |
| `Configuration.getGlobalConfig()` |
`Configuration.getGlobalConfiguration()` |
| `LocalEventManager.fromConfig()` |
`LocalEventManager.fromConfiguration()` |
Renamed `config` options to `configuration` in `StorageOpenOptions`
(`Dataset.open()`, `KeyValueStore.open()`, `RequestQueue.open()`),
`UseStateOptions`, `purgeDefaultStorages()`, `SnapshotterOptions`,
`SaveSnapshotOptions` (Playwright and Puppeteer),
`RecoverableStateOptions`, `RequestListOptions`, `CpuLoadSignalOptions`
and `MemoryLoadSignalOptions`.
Renamed `config` properties to `configuration` on `Dataset`,
`KeyValueStore`, `Snapshotter` and `BrowserLauncher` (plus the
Playwright, Puppeteer and Stagehand launchers).
The `configuration` crawler option and
`serviceLocator.getConfiguration()` / `setConfiguration()` were already
consistent and are unchanged.
The BC is documented in the v4 upgrading guide.
Closes#3706
Infers the router's `label → userData` route map on the crawler and uses
it to type the request **inputs** — `crawler.run` /
`crawler.addRequests` and the `addRequests` / `enqueueLinks` context
helpers. Targets `v4`.
> The router core this builds on — the typed route map (#3747) and
per-label [Standard Schema](https://standardschema.dev) validation with
`RequestValidationError` (#3851) — already merged to `master` and is
present on `v4`. This PR is the remaining `v4`-only piece: propagating
that route map to the crawler and context request methods (a breaking
change to those signatures, hence `v4`).
When a typed router is passed as `requestHandler`, providing a declared
`label` requires the matching `userData` shape and rejects unknown
labels; unlabeled requests keep loose `userData` (they hit the default
handler).
- **Handler context** — `ctx.addRequests` / `ctx.enqueueLinks` are typed
from the route map. Driven by the router itself, so it works for
**every** crawler type.
- **Crawler instance** — `Routes` is inferred from the `requestHandler`
option and types `crawler.addRequests` / `crawler.run`. Threaded through
all crawler classes (Basic/Http/Cheerio/JSDOM/LinkeDOM and the browser
family: Browser/Playwright/Puppeteer/Stagehand/AdaptivePlaywright).
```ts
const crawler = new PlaywrightCrawler({ requestHandler: router });
await crawler.addRequests([{ url, label: 'PRODUCT', userData: { sku: 's', price: 1 } }]); // ✅
await crawler.addRequests([{ url, label: 'PRODUCT', userData: { sku: 1 } }]); // ❌ wrong userData
await crawler.addRequests([{ url, label: 'NOPE' }]); // ❌ unknown label
```
Fully backwards compatible: without a typed router the request inputs
stay loosely typed, exactly as before.
Relates to #3082
Moves the single-consumer `createRequestDebugInfo`, `getObjectType` and
`inspectValue` helpers from the public `@crawlee/utils` API to
`@crawlee/core`.
Related to https://github.com/apify/crawlee/issues/3079
Harry