Lost in the v4 rebase; matches the createHttpRouter/createCheerioRouter
overload set. Also removes the rebase reconciliation checklist, which is
fully resolved by this commit.
The 4.0 snapshot only existed so the v4 branch site build had a default
version. On master, the v4 docs are the current (next) version, labeled
"4.0 (RC)", and the real 4.0 snapshot will be generated by the release
workflow when 4.0.0 ships. The default docs version stays 3.18 until then.
The transplanted enqueueLinks split reverted a few master-carried behaviors in
BasicCrawler; this restores them on top of the new design:
- stop capturing statistics before teardown again, so the crawler state is
saved before the final persistence event fires (prevents double persistence)
- teardown() only emits an explicit PERSIST_STATE event for externally-managed
event managers, and tears the owned session pool down with persistState
matching event manager ownership (an unset flag previously fell back to the
`persistState = true` default, double-persisting the pool)
- the enqueue limit log distinguishes an explicit `limit` from the remaining
maxRequestsPerCrawl budget again
- adapt the master-carried tests to the addRequests() API; drop the
explicit-undefined override tests for options that no longer exist on it
Moves the `requestManager`-bound enqueueing logic into
`BasicCrawlerContext.addRequests`, and each DOM-aware crawler now
exposes its own `extractLinks()` plus an `enqueueLinks()` that composes
`extractLinks` + `addRequests`.
This aligns the JS implementation with what Python does, to some extent.
Closes#3081
- markRequestHandled → markRequestAsHandled on SitemapRequestLoader
- await the now-async RequestQueue.getTotalCount()
- transformRequestFunction skips now report the dedicated 'transform' reason
- robots.txt mock needs getCrawlDelay
- statistics/session-pool single-persistence tests observe KeyValueStore.setValue
instead of the persistState methods RecoverableState replaced
- pass an explicit logger to Sitemap.load in the aggregated-warning test
The domain matching in the restored `filterUrl`/`matchesEnqueueStrategy`
helpers (carried over from master) uses tldts, which had been dropped from
the package manifest during the rebase.
The rebase onto master replaced the RobotsTxtFile factory bodies with master's
versions (which keep the URL for enqueue-strategy filtering), dropping the
@ts-ignore comments v4 needs because robots-parser's CJS default export is not
callable under nodenext module resolution.
The rebase onto master carried over master's renovate bump of camoufox-js to
^0.12.0, but the template (and the repo root) still pin the Playwright version
whose bundled Firefox matches camoufox-js 0.11.
Replaces the remaining `ow`-based argument validation with `zod` across
all packages and reworks how validation results are consumed and
reported.
Closes#3716
- **`ow` is gone** — every argument check now goes through
`parseArgument(value, schema, label?)` from `@crawlee/utils`, backed by
shared zod schemas (`schemas`, exported via `@crawlee/utils/internal`).
The `@sapphire/shapeshift` checks in `@crawlee/fs-storage` were
converted too, so a single validation library remains.
- **Parse results are used everywhere** — option defaults moved from
destructuring into the schemas (`.default(...)`), and call sites
destructure the typed parse result. `parseArgument` returns `TValue &
z.output<TSchema>`, so call sites keep their declared TS types while
gaining the defaults.
- **Schemas are built once** — all per-call schemas are hoisted to
module scope; crawler/launcher classes build their strict options schema
once as a `static optionsSchema` next to `optionsShape`. The
`urlPatternSchema` for `include`/`exclude` lives in
`enqueue_links/shared.ts`, next to the type it validates.
- **Specific validators instead of `anyObject`** — class-typed options
use `z.instanceof(...)` (`BaseHttpClient`, `Configuration`,
`EventManager`), interface-typed ones use duck-typed `objectWithKeys`
validators (`storageBackend`, `requestManager`, `logger`, …), and
element-typed arrays use the new `schemas.arrayOf(item, 'numbers')`.
`ArgumentValidationError` (replacing ow's `ArgumentError`) renders one
line per issue: the expected type, the received type and value folded
into one clause, the offending field path, and the validated interface:
```text
// v3 (ow) — first issue only
Expected property `maxRequestRetries` to be of type `number` but received type `string` in object `HttpCrawlerOptions`
// v4 (zod) — every issue, one line each
Invalid input: expected number, received the string `many` at `maxRequestRetries` in `HttpCrawlerOptions`
Invalid input: expected an array of numbers, received the number `500` at `additionalHttpErrorStatusCodes` in `HttpCrawlerOptions`
Invalid input: expected boolean, received the string `yes` at `retryOnBlocked` in `HttpCrawlerOptions`
```
Details worth knowing:
- Union failures expand into one line per failed arm (zod's own message
is a bare "Invalid input").
- `NaN` is named as itself, an empty string renders as `''`, and arrays
name their element type (``expected an array of URL patterns``) — none
of which ow or stock zod reported.
- `new Request('https://…')` gets a targeted hint pointing at the `{ url
}` object form.
- For programmatic handling, the error exposes zod's structured output:
`error.issues` and the raw `ZodError` as a typed `cause`.
The migration is documented in the v4 upgrading guide
(`docs/upgrading/upgrading_v4.md`), including a rename-cheat-sheet
entry.
- Custom HTTP clients must now **extend `BaseHttpClient`** from
`@crawlee/http-client` rather than just implementing the interface (all
shipped clients already do; `LazyDefaultHttpClient` was converted). Same
applies to test mocks — `Object.create(BaseHttpClient.prototype)` works.
- One caveat of consuming parse results: zod object schemas return a
pruned plain copy, so options holding class instances are validated with
passthrough schemas (`z.custom`-based) to keep their prototypes — there
are comments at the relevant schemas.
- Fixes a few latent gaps surfaced along the way: `Request.state` now
accepts `RequestState.SKIPPED` (validated via `z.enum(RequestState)`),
and the publish-time catalog inlining covers `optionalDependencies`.
- `ArgumentValidationError` and its formatter are intentionally kept
close to the copy in apify/apify-client-js#986 — a follow-up may extract
them into a shared package.
---------
Co-authored-by: Martin Adámek <banan23@gmail.com>
Gets the E2E test suite running against v4. The suite hadn't been run
since the v4 rewrite and everything failed on startup. After these
changes the MEMORY run passes locally end to end, and getting there
surfaced a few real regressions in the packages themselves.
- `LinkeDOMCrawler`'s `enqueueLinks` helper referenced the global
`document` (which doesn't exist in Node) instead of the parsed window,
so every call crashed at runtime.
- `ErrorSnapshotter.saveHTMLSnapshot()` returned the record key with a
v3-style `.html` suffix, so the follow-up `getPublicUrl()` lookup missed
and `firstErrorHtmlUrl` never made it into the crawler statistics.
- `JSDOMCrawlingContext`/`LinkeDOMCrawlingContext` didn't override
`enqueueLinks`, exposing the strict urls-required signature even though
the runtime helper extracts URLs from the parsed document.
- `LinkeDOMCrawler` can now be constructed without arguments, like the
other crawlers.
- Bumped the pinned `apify` SDK to 4.0.0-beta.22 (beta.19 imports
`snakeCaseToCamelCase` from `@crawlee/utils`, which no longer exists
there).
- Adapted `tools.mjs` to the fs-storage on-disk layout (extensionless
key-value records; the short-lived `__default__` directory alias it
originally targeted was a bug, fixed in #4013) and to the
`@crawlee/utils` exports split.
- Migrated test actors to the v4 APIs: the `logger` option with
`ApifyLogAdapter` instead of `log`, hooks reading `gotoOptions` from the
crawling context, `session.setCookie()`, a custom `SessionPool` instead
of `sessionPoolOptions`, the WHATWG `Response` returned by
`sendRequest`, `registerDeferredCleanup` for dataset writes that must
survive a throwing handler, and explicit enqueue strategies now that
`include` globs are ANDed with the default same-hostname strategy.
- The ignore-ssl test now configures TLS verification on the http
client, because the crawler-level `ignoreSslErrors` option is not wired
to the default client in v4. That dangling option deserves a separate
fix or removal, since it currently does nothing.
- The impit test pins session fingerprints, since the random default
fingerprint overrides the client's browser impersonation.
- Added ES2022 to the actor tsconfigs' `lib` (a bare `["DOM"]` drops the
ES lib and broke compilation on `ErrorOptions`).
- Skipped the zero-concurrency queue test: it stages a stuck queue
through the v3 client-side `inProgress` set, which the rewritten queue
doesn't have.
- Fixed the camoufox fetch retry loop fetching 5x even on success, and
removed a duplicate `apify` dependency key that silently downgraded the
curl-impersonate actor to SDK v3.
- Commented out the LOCAL storage matrix entry in the workflow, as
`@apify/storage-local` doesn't support v4.
- closes#3998
Unnamed storages (default and aliased) are now purged on start, named
ones are not — crawlee-python's rule. The fs backend sweeps the storage
directories, so leftovers from a previous process are caught too.
Two adjacent bugs, one commit each:
- `createDatasetBackend()` / `({})` didn't open the default storage,
though `StorageIdentifier` says they do
- the `__default__` sentinel leaked into the directory name, so default
storages lived in `storage/datasets/__default__` rather than `default`
`ThrottlingRequestManager` (#3741) sub-queues are alias-keyed, so they
now only survive a restart with `purgeOnStart` off — as its docs already
said.
The `ignoreSslErrors` option stopped working during the v4 HTTP client
interface rework: it was still folded into got-style request options,
but those never reach `httpClient.sendRequest()`. In v3 the option
works, and actors commonly expose it in their input schemas and pass it
into crawler options (e.g. actor-scraper), so this keeps it working
instead of removing it.
The option is renamed to `ignoreTlsErrors`, matching
`session.proxyInfo.ignoreTlsErrors`, the browser pool, and the impit
client (the old name is dropped, documented in the upgrading guide;
actors migrating to v4 rename it in their own code, the SDK does not
touch this option). The crawler forwards it (still defaulting to `true`,
same as v3) as a new `SendRequestOptions.ignoreTlsErrors` flag, which
`BaseHttpClient` also enables for MITM proxy sessions (previously
equally dead). The impit client honors the flag; for custom clients it
is best effort, and `FetchHttpClient` cannot disable TLS verification at
all. The dead plumbing in `getRequestOptions()` is removed and unit
tests cover the forwarding chain.
Converts all `const enum` declarations (`EventType`,
`BROWSER_POOL_EVENTS`, `BROWSER_CONTROLLER_EVENTS`,
`OperatingSystemsName`, `DeviceCategory`) to regular enums. `const enum`
breaks downstream consumers using `isolatedModules` or transpilers that
don't inline them (Babel, esbuild, vitest).
Closes#3125
Types the default cookie jar against `@crawlee/types`' `CookieJar`
interface and dynamically imports `tough-cookie` only when a default jar
actually needs constructing.
Relates to #3549. `sax` was imported eagerly at module scope in
`packages/utils/src/internals/sitemap.ts`, so every consumer of
`@crawlee/utils` paid its load cost even if sitemap parsing was never
used.
`sax.SAXParser` is now loaded via a dynamic `import()` only when an XML
sitemap parser is actually constructed.
No public API changes (this class is only used from the `parseSitemap`
generator).
Adds a `/internal` subpath export to `@crawlee/utils` for symbols that
are only shared between crawlee packages (CSS selectors, type guards,
URL helpers, `CheerioRoot`, etc.), keeping the root entry point for
user-facing utilities. The `crawlee` meta-package naturally only
re-exports the public surface.
Closes#3079
Both are only used on specific code paths (sitemap parsing, CSV export)
but were loaded eagerly by every crawler import, costing ~110ms of the
~500ms `import('@crawlee/http')` on my machine.
Related: #3549
`PlaywrightBrowser._setBrowserType()` has existed since the initial
crawlee commit but was never called, so the `_browserType` field stayed
`undefined` and `browserType()` returned `undefined` at runtime despite
its non-optional `BrowserType` signature. Persistent contexts are the
default (`useIncognitoPages: false`), so a call like
`browser.browserType().name()` on the wrapper threw a `TypeError`. The
incognito and remote-connection paths return the native Playwright
`Browser` and were not affected.
This wires the existing setter in `PlaywrightPlugin._launch()` where the
wrapper is created. `this.library` is the `BrowserType` that launched
the context, so the wrapper now provides the consistent API with
Playwright's `Browser` that its docblock describes. Also adds a test
covering both the persistent-context wrapper and the native incognito
browser.
Closes#3945.
`@crawlee/types` no longer imports `CookieJar`/`SerializedCookieJar`
from `tough-cookie` - it declares its own structurally-compatible
interfaces instead, so `tough-cookie` is no longer part of its
dependency tree. `tough-cookie` stays a direct dependency everywhere
it's actually used (`core`, `http-client`, `impit-client`).
`closeCookieModals` loads `idcac-playwright` through a guarded dynamic
import, and both crawler packages already declare it as an optional peer
dependency — but it was also listed in `dependencies`, so it got
installed for everyone anyway. The e2e actors that call
`closeCookieModals` now depend on it explicitly.
`playwright-crawler` also had a `^0.1.3` dependency against a `^0.2.0`
peer range.
Adopts native `#` private fields for private class properties across all
packages and removes the remaining `_` prefixes from protected/private
members, as agreed in #3108.
- Around 300 private properties are now native `#` fields, so they are
hard-private at runtime and invisible to spread and `JSON.stringify`.
Private and protected methods keep the `private`/`protected` keyword and
lose the `_` prefix.
- Renamed protected extension points: `_init` → `init`,
`_throwOnBlockedRequest` → `throwOnBlockedRequest`,
`_getMessageFromError` → `getMessageFromError` and
`_getCookieHeaderFromRequest` → `getCookieHeaderFromRequest` on
`BasicCrawler`; `_navigationHandler` → `navigationHandler` on
`BrowserCrawler` and its subclasses; `_addProxyToLaunchOptions` →
`addProxyToLaunchOptions`, `_isChromiumBasedBrowser` →
`isChromiumBasedBrowser`, `_connectToRemoteBrowser` →
`connectToRemoteBrowser` and `_throwAugmentedLaunchError` →
`throwAugmentedLaunchError` on `BrowserPlugin`. The `@internal`
`LaunchContext._remoteToken` and `PlaywrightBrowser._setBrowserType`
became `remoteToken` and `setBrowserType`.
- `BrowserPlugin._launch` and
`BrowserController._close`/`_kill`/`_newPage`/`_getCookies`/`_setCookies`
keep the underscore, since their public wrapper methods own the plain
names. `Readable._read` is a Node contract.
- Tests that used to poke internal state were rewritten against public
API where the test's meaning survives: session pool state via
`getState()`, request list persistence via the exported persistence
keys, proxy rotation via `newUrl()`, autoscaling options via the
constructor, crawler cleanup via `teardown()`. The members those tests
reached are now `#` fields too.
- The members still declared with TypeScript's `private` fall into two
groups, each with a one-line comment saying why: runtime hazards for `#`
(the adaptive crawler's log proxy, two cross-class accesses), and
genuine test injection seams (backend delegation spies, interval
replacement, static counter resets, config-echo assertions with no
public surface).
- `LaunchContext.extend()` now rejects all declared fields and accessors
as reserved names; it previously missed fields declared after the
reserved list was computed. The fingerprinting hook assigns the declared
`fingerprint` field directly instead of going through `extend()`.
- A new `no-underscore-dangle` oxlint rule (with `enforceInClassFields`
and `enforceInMethodNames`) enforces the convention.
- The upgrading guide documents the renames and the `#` semantics
change; API reports regenerated.
Closes#3108
Navigation hooks on browser crawlers could not be typed with custom
`userData`: a hook declared as `(ctx:
PuppeteerCrawlingContext<MyUserData>) => ...` failed to type-check
against `preNavigationHooks`/`postNavigationHooks`, while the HTTP
crawler family already supports this (#2063).
- `BrowserCrawlingContext`, `PlaywrightCrawlingContext`,
`PuppeteerCrawlingContext`, `StagehandCrawlingContext` and
`AdaptivePlaywrightCrawlerContext` now default `UserData` to `any`, the
same pattern (and marker comment) the HTTP family adopted in v4.
- `PlaywrightHook`, `PuppeteerHook` and `StagehandHook` are now type
aliases generic over `UserData` (e.g. `PlaywrightHook<MyUserData>`),
like `CheerioHook` and `HttpHook`.
- `HttpCrawlerOptions.preNavigationHooks` now uses
`CrawlingContext<any>`, so pre-navigation hooks typed with custom user
data (`InternalHttpHook<CrawlingContext<MyUserData>>`) are assignable as
well; the post-navigation option already allowed this.
- The same default change makes request handlers typed with custom user
data assignable to untyped crawler options as well.
- Adds type-level regression tests (hooks and request handlers), an
upgrading guide entry for the switch from interfaces to type aliases,
and regenerated API snapshots.
Closes#2063
Extends the fix from #3836 (which already made `@crawlee/utils` and
`http-crawler` lazy-load cheerio) to the two remaining spots that were
still importing it eagerly: `playwright-utils.ts` and
`puppeteer_utils.ts`. Both only use cheerio inside `parseWithCheerio()`,
so every Playwright/Puppeteer crawler paid cheerio's import cost even
when the handler never calls it.
Related to #3549.
fs-extra was only used for `ensureDir` and `writeJSON` in `cli` and
`basic-crawler`, both one-line wrappers over `node:fs/promises`.
Swapped them for `mkdir(recursive)` and `writeFile`, dropping the
dependency.
Related to #3549
Align `EnqueueLinksOptions` with crawlee-python (#3409):
- Replace `globs`, `regexps`, `pseudoUrls` options with
`include`/`exclude` accepting `UrlPatternInput[]`
- Strip request options (label, method, payload, userData, headers) from
pattern objects — patterns are pure URL matchers
- `transformRequestFunction` is now the only way to customize
per-request options, runs after all filtering
- Add `'skip'` and `'unchanged'` return values to `RequestTransform`
(aligned with Python's `RequestTransformAction`)
- Apply same changes to `enqueueLinksByClickingElements` (Playwright +
Puppeteer) and `SitemapRequestList`
- Remove `@apify/pseudo_url` dependency and `PseudoUrl` re-export
- Update all templates from `globs` to `include`
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Timeout redesign (#2951)
v4 had already dropped the old `navigationTimeoutSecs +
requestHandlerTimeoutSecs + buffer` sum, so the request handler timeout
covers only the user's function and the confusing "timed out after 130
seconds" messages are gone. What it did not do is put back the pieces
that sum incidentally bounded.
- **The navigation phase is one window.** `navigationTimeoutSecs` covers
the `preNavigationHooks`, the navigation itself, and the
`postNavigationHooks` as a single shared budget, matching Crawlee for
Python. A hook that hangs no longer stalls the request forever; it eats
into the same window the navigation uses. (This replaces an earlier
attempt at a separate per-hook `navigationHooksTimeoutSecs`, which is
gone.)
- **A whole-request backstop.** The phases between the timed ones
(`extendContext`, the robots.txt check, response processing) could hang
indefinitely. An internal backstop now bounds the whole request. It is
sized to outlast the phases that have their own timeout (navigation plus
the handler), so a legitimately slow request is never cut short and it
only fires when something is genuinely stuck. It is configured with
`CRAWLEE_INTERNAL_TIMEOUT`, now resolved through `Configuration` like
the other env-backed options. Set it below the phase timeouts and the
crawler raises it per request and warns at startup, rather than cutting
a phase short.
## `context.extendTimeout()`
When the time needed is only apparent once a hook or handler is already
running, `context.extendTimeout(secs)` buys more. From inside the
navigation phase it pushes the shared navigation window; from the
request handler it pushes the handler timeout. Either way it also pushes
the backstop and raises the request-manager reservation, so the extra
time is neither clipped by the backstop nor undone by a locking backend
handing the request out again.
## Per-route timeouts (#1485)
```ts
router.addHandler('LIST', handler, { requestHandlerTimeoutSecs: 120 });
router.addHandler('DETAIL', handler); // keeps the crawler's default
```
`requestHandlerTimeoutSecs` is unchanged and stays the default for
anything a route does not override; a route's value may be longer or
shorter than it. The label is known before the handler starts, so the
timeout is resolved per request and the router never reaches back into
the crawler mid-flight. The backstop and the reservation both account
for the longest route in play.
## Browser navigation
A `preNavigationHooks` hook can still override `gotoOptions.timeout`
(including `0`, Playwright's "no timeout"); the shared window no longer
clamps it to 1ms or discards a larger value. A navigation timeout,
whether ours or the driver's own, is reported as `Navigation timed out
after N seconds` instead of the driver's raw millisecond value.
## Adaptive crawler
`AdaptivePlaywrightCrawler` runs the handler up to twice per request (a
static attempt falling through to the browser). A
`getRequestHandlerRunCount` hook (2 for adaptive, 1 everywhere else)
sizes the whole-request budgets for both runs, while each run keeps its
own handler window. This replaces an earlier one-off doubling that
missed per-route overrides.
## Notes on the implementation
The backstop is a bare timer, not `addTimeoutToPromise`: nested
`addTimeoutToPromise` calls share one `AbortController`, so wrapping the
whole request in one would let the handler timing out abort the outer
context and cancel the error handling that reclaims the request. The
plumbing lives in its own `request-backstop.ts` module.
For the same reason the HTTP navigation binds its request to the
navigation frame's `@apify/timeout` cancel signal rather than a fixed
`AbortSignal.timeout`: the response body is read lazily, after the
post-navigation hooks, so a fixed timer would abort a body a hook is
legitimately still keeping alive via `extendTimeout`. A genuine
navigation timeout still fails the request; the body read is bounded at
the parse step.
Depends on the `extendTimeout` addition in `@apify/timeout`
(apify/apify-shared-js#669), released as 0.4.4 and consumed here.
Closes#1485Closes#2951
`bindMethodsToServiceLocator()` wraps every prototype method as an own
property on the crawler instance so that calls resolve the crawler's
scoped service locator. The walk goes derived-first but assigned
wrappers unconditionally, so a method defined on both a subclass and a
base class got wrapped twice, and the base version, assigned last, won.
Any crawler constructed with scoped services (`logger`,
`storageBackend`, `eventManager`, or a custom `configuration`) therefore
lost all its method overrides. `AdaptivePlaywrightCrawler` fell back to
`BasicCrawler.runRequestHandler()`: no rendering type prediction or
detection, no log replay, and an `_init()` that never initialized the
predictor. Every other crawler subclass lost its
`buildContextPipeline()` override the same way.
This also explains why the log replay tests added in #3803 kept passing
with the replay line reverted. They pass a custom `logger`, so the
adaptive code path never ran and the request handler logged straight to
the real logger. With this fix they fail without the replay lines and
pass with them (checked in both directions).
The fix skips a method once a more derived version of it has been
handled. A key seen as a getter or setter at a derived level also blocks
wrapping its base version, since dynamic dispatch would pick the derived
accessor.
Closes#3934
These `@internal` async iterable helpers are only used by
`@crawlee/core`, so they no longer need to live in `@crawlee/utils`.
Common symbols used in multiple `@crawlee/` packages have been left in
the `@crawlee/utils` package.
`Cookie.parse` returns `undefined` for unparseable fragments (e.g. a
bare name), which the `!` assertion then dereferenced and threw; skip
and warn instead.
Moves the `@ignore`-d `isStream`, `isBuffer`, `toBuffer` and
`weightedAvg` helpers out of `@crawlee/utils` and into `@crawlee/core`
as package-private internals, since core is their only consumer.
Related to #3079
These symbols were plain re-exports of `@crawlee/types` and duplicates
of helpers already living in `@crawlee/core`, so consumers now import
them from their real home.
Harry