Bumps typescript to `^7.0.2` (native compiler) for building and
type-aware linting, together with `oxlint-tsgolint@^7.0.2001`, which
version-tracks TS 7.
The docs build stays on TS 6: typedoc needs the TypeScript JS API, which
the native compiler no longer exposes. The website keeps its own
`typescript: ^6.0.0` pin and pnpm nests that copy under
`node_modules/typedoc`, so typedoc resolves 6.x while the packages
compile with TS 7.
TS 7 also caught one real issue in the tests: `RequestList#initialize()`
is private, and the call in `request_list.test.ts` now needs the same
`@ts-expect-error` the surrounding test already uses for the private
constructor.
The api-extractor runner script also imports `typescript` for its report
parsing; that import now goes through a `typescript-v6` alias
(`npm:typescript@^6.0.0`), since the parsing needs the classic JS API
(api-extractor itself bundles its own TypeScript and is unaffected). The
regenerated reports in `docs/public-api/` pick up TS 7's single-quoted
string literal types in three packages, 4 lines total.
Same change as apify/apify-sdk-js#695. Build, docs build, type-aware
lint, test typecheck and the unit suite all pass.
This pins every third-party action in our workflows to a full commit
SHA, keeping the resolved version tag as a trailing comment. Renovate
understands that convention and updates the SHA and comment together.
The trigger: yesterday the `v11` tag of `EndBug/add-and-commit` moved to
the broken v11.1.0 release, whose `action.yml` fails to load
(`Unrecognized named-value: 'github'`), which killed our publish
workflow ([failed
run](https://github.com/apify/crawlee/actions/runs/32255557318)). With
SHA pins, a tag moving under us, by accident or by compromise, can't
break or hijack CI anymore. `EndBug/add-and-commit` is pinned to
v11.0.0, the last working release; the upstream fix is pending in
EndBug/add-and-commit#783.
Own-org references (`apify/workflows`, `apify/actions`,
`apify/setup-apify-cli-action`) stay on floating refs on purpose, since
we control those repos.
Bumps the pinned `pnpm` from 11.0.9 to 11.22.0, keeping all three pins
in sync: `packageManager`, `devEngines.packageManager.version`, and
`volta.pnpm` (Renovate's packageManager update typically bumps only the
first, which leaves corepack failing on a version mismatch).
Routine dependency-manager hygiene; no lockfile or source changes.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Adds a "switching master to the next major" playbook to RELEASE.md while
the v4 transition is fresh. It covers the order of operations
(maintenance branch first, then rebase, validate, and fast-forward
master), the docs snapshot removal and its unversioned `/js/api` route
gotcha, which rulesets block the fast-forward push and how the
`BypassTemporary` team covers them, why open PRs have to be retargeted
by hand, and the required-checks and renovate follow-ups.
Re-applies the puppeteer 25.8.0 bump that renovate landed on the
pre-merge master (the v4 merge brought back the old 25.3.0 pin). The
version change alone is not enough: the lockfile keeps a stale
`fingerprint-injector(puppeteer@25.3.0)` peer instance around, and the
duplicated `puppeteer-core` types break `tsc-check-tests` with 82
errors. A `pnpm dedupe` collapses the graph back to a single instance.
Canaries publish from master under the v4 dist-tag (next once 4.0.0 is
stable), the version-docs snapshot job is guarded to master, and
RELEASE.md now documents the branch/dist-tag matrix including the 3.x
maintenance branch.
Lost in the v4 rebase; matches the createHttpRouter/createCheerioRouter
overload set. Also removes the rebase reconciliation checklist, which is
fully resolved by this commit.
The 4.0 snapshot only existed so the v4 branch site build had a default
version. On master, the v4 docs are the current (next) version, labeled
"4.0 (RC)", and the real 4.0 snapshot will be generated by the release
workflow when 4.0.0 ships. The default docs version stays 3.18 until then.
The statistics record is persisted during crawler teardown and the platform
key-value store is eventually consistent, so reading it immediately after the
run can miss it. That crashed the whole test with a TypeError on
stats.requestsFinished (seen in cheerio-curl-impersonate-ts) even though the
actor run itself succeeded. The lookup now retries for up to ~30 seconds and
falls back to an empty object, so a genuinely missing record fails the
assertions cleanly.
Cloudflare serves its challenge pages with a 403 status. On v3,
handleCloudflareChallenge() received the session and removed 403 from the
session pool's blocked status codes itself; v4 dropped that mechanism when the
hook was redesigned, so challenged requests died in throwOnBlockedRequest()
on every retry and the solver only ever got a single attempt. Solving the
challenge is probabilistic, which is why the fixture passes on master (where
retries reach the solver) and kept failing here.
blockedStatusCodes is a public crawler option in v4, so the fixture opts out
of 403 explicitly. Whether handleCloudflareChallengeHook() should handle this
automatically again is a follow-up design question.
The v3.18 release blog post and the 3.18 versioned upgrading guide linked to
the current (unversioned) API reference. That resolved fine on master, where
the current API was 3.18, but on v4 the docs build fails: StorageClient was
renamed and RequestValidationError's page moved. Version-pinned API links are
the established pattern in versioned content (see the 3.17 guide).
camoufox-js releases bundle a specific Firefox build that must match the one
expected by the pinned playwright version (0.11 ↔ 1.60). The fixture pinned
camoufox-js ^0.12.0 next to playwright 1.60.0, so the Cloudflare challenge
kept failing on the platform even with the updated challenge markup handling
(#4019) in place — master validates that fix with the 0.11/1.60 pairing.
Two upstream v4 changes landed after the last E2E run and broke the suite:
- the enqueueLinks split (#4010) changed the context helper's return value to
the addRequestsBatched result, so the *-enqueue-links fixtures now assert on
`addedRequests` being empty instead of deep-equality with the old shape
- the zod validation unification (#3935) introduced a `catalog:` dependency,
which npm cannot resolve when the platform builds the actor image; the E2E
package-copy step now rewrites catalog deps to their pinned versions from
pnpm-workspace.yaml, the same way it already rewrites `workspace:` deps
The transplanted enqueueLinks split reverted a few master-carried behaviors in
BasicCrawler; this restores them on top of the new design:
- stop capturing statistics before teardown again, so the crawler state is
saved before the final persistence event fires (prevents double persistence)
- teardown() only emits an explicit PERSIST_STATE event for externally-managed
event managers, and tears the owned session pool down with persistState
matching event manager ownership (an unset flag previously fell back to the
`persistState = true` default, double-persisting the pool)
- the enqueue limit log distinguishes an explicit `limit` from the remaining
maxRequestsPerCrawl budget again
- adapt the master-carried tests to the addRequests() API; drop the
explicit-undefined override tests for options that no longer exist on it
Moves the `requestManager`-bound enqueueing logic into
`BasicCrawlerContext.addRequests`, and each DOM-aware crawler now
exposes its own `extractLinks()` plus an `enqueueLinks()` that composes
`extractLinks` + `addRequests`.
This aligns the JS implementation with what Python does, to some extent.
Closes#3081
- markRequestHandled → markRequestAsHandled on SitemapRequestLoader
- await the now-async RequestQueue.getTotalCount()
- transformRequestFunction skips now report the dedicated 'transform' reason
- robots.txt mock needs getCrawlDelay
- statistics/session-pool single-persistence tests observe KeyValueStore.setValue
instead of the persistState methods RecoverableState replaced
- pass an explicit logger to Sitemap.load in the aggregated-warning test
- requestQueue → requestManager in enqueueLinks options
- await the now-async createRequestQueueMock
- sessionPoolOptions → sessionPool instance in the redirect-cookie test
- config → configuration in purgeDefaultStorages options
- SitemapRequestList → SitemapRequestLoader in loader tests
- handleCloudflareChallenge lost its session parameter in v4
- drop duplicate imports
The domain matching in the restored `filterUrl`/`matchesEnqueueStrategy`
helpers (carried over from master) uses tldts, which had been dropped from
the package manifest during the rebase.
The rebase onto master replaced the RobotsTxtFile factory bodies with master's
versions (which keep the URL for enqueue-strategy filtering), dropping the
@ts-ignore comments v4 needs because robots-parser's CJS default export is not
callable under nodenext module resolution.
The rebase onto master carried over master's renovate bump of camoufox-js to
^0.12.0, but the template (and the repo root) still pin the Playwright version
whose bundled Firefox matches camoufox-js 0.11.
Replaces the remaining `ow`-based argument validation with `zod` across
all packages and reworks how validation results are consumed and
reported.
Closes#3716
- **`ow` is gone** — every argument check now goes through
`parseArgument(value, schema, label?)` from `@crawlee/utils`, backed by
shared zod schemas (`schemas`, exported via `@crawlee/utils/internal`).
The `@sapphire/shapeshift` checks in `@crawlee/fs-storage` were
converted too, so a single validation library remains.
- **Parse results are used everywhere** — option defaults moved from
destructuring into the schemas (`.default(...)`), and call sites
destructure the typed parse result. `parseArgument` returns `TValue &
z.output<TSchema>`, so call sites keep their declared TS types while
gaining the defaults.
- **Schemas are built once** — all per-call schemas are hoisted to
module scope; crawler/launcher classes build their strict options schema
once as a `static optionsSchema` next to `optionsShape`. The
`urlPatternSchema` for `include`/`exclude` lives in
`enqueue_links/shared.ts`, next to the type it validates.
- **Specific validators instead of `anyObject`** — class-typed options
use `z.instanceof(...)` (`BaseHttpClient`, `Configuration`,
`EventManager`), interface-typed ones use duck-typed `objectWithKeys`
validators (`storageBackend`, `requestManager`, `logger`, …), and
element-typed arrays use the new `schemas.arrayOf(item, 'numbers')`.
`ArgumentValidationError` (replacing ow's `ArgumentError`) renders one
line per issue: the expected type, the received type and value folded
into one clause, the offending field path, and the validated interface:
```text
// v3 (ow) — first issue only
Expected property `maxRequestRetries` to be of type `number` but received type `string` in object `HttpCrawlerOptions`
// v4 (zod) — every issue, one line each
Invalid input: expected number, received the string `many` at `maxRequestRetries` in `HttpCrawlerOptions`
Invalid input: expected an array of numbers, received the number `500` at `additionalHttpErrorStatusCodes` in `HttpCrawlerOptions`
Invalid input: expected boolean, received the string `yes` at `retryOnBlocked` in `HttpCrawlerOptions`
```
Details worth knowing:
- Union failures expand into one line per failed arm (zod's own message
is a bare "Invalid input").
- `NaN` is named as itself, an empty string renders as `''`, and arrays
name their element type (``expected an array of URL patterns``) — none
of which ow or stock zod reported.
- `new Request('https://…')` gets a targeted hint pointing at the `{ url
}` object form.
- For programmatic handling, the error exposes zod's structured output:
`error.issues` and the raw `ZodError` as a typed `cause`.
The migration is documented in the v4 upgrading guide
(`docs/upgrading/upgrading_v4.md`), including a rename-cheat-sheet
entry.
- Custom HTTP clients must now **extend `BaseHttpClient`** from
`@crawlee/http-client` rather than just implementing the interface (all
shipped clients already do; `LazyDefaultHttpClient` was converted). Same
applies to test mocks — `Object.create(BaseHttpClient.prototype)` works.
- One caveat of consuming parse results: zod object schemas return a
pruned plain copy, so options holding class instances are validated with
passthrough schemas (`z.custom`-based) to keep their prototypes — there
are comments at the relevant schemas.
- Fixes a few latent gaps surfaced along the way: `Request.state` now
accepts `RequestState.SKIPPED` (validated via `z.enum(RequestState)`),
and the publish-time catalog inlining covers `optionalDependencies`.
- `ArgumentValidationError` and its formatter are intentionally kept
close to the copy in apify/apify-client-js#986 — a follow-up may extract
them into a shared package.
---------
Co-authored-by: Martin Adámek <banan23@gmail.com>
Gets the E2E test suite running against v4. The suite hadn't been run
since the v4 rewrite and everything failed on startup. After these
changes the MEMORY run passes locally end to end, and getting there
surfaced a few real regressions in the packages themselves.
- `LinkeDOMCrawler`'s `enqueueLinks` helper referenced the global
`document` (which doesn't exist in Node) instead of the parsed window,
so every call crashed at runtime.
- `ErrorSnapshotter.saveHTMLSnapshot()` returned the record key with a
v3-style `.html` suffix, so the follow-up `getPublicUrl()` lookup missed
and `firstErrorHtmlUrl` never made it into the crawler statistics.
- `JSDOMCrawlingContext`/`LinkeDOMCrawlingContext` didn't override
`enqueueLinks`, exposing the strict urls-required signature even though
the runtime helper extracts URLs from the parsed document.
- `LinkeDOMCrawler` can now be constructed without arguments, like the
other crawlers.
- Bumped the pinned `apify` SDK to 4.0.0-beta.22 (beta.19 imports
`snakeCaseToCamelCase` from `@crawlee/utils`, which no longer exists
there).
- Adapted `tools.mjs` to the fs-storage on-disk layout (extensionless
key-value records; the short-lived `__default__` directory alias it
originally targeted was a bug, fixed in #4013) and to the
`@crawlee/utils` exports split.
- Migrated test actors to the v4 APIs: the `logger` option with
`ApifyLogAdapter` instead of `log`, hooks reading `gotoOptions` from the
crawling context, `session.setCookie()`, a custom `SessionPool` instead
of `sessionPoolOptions`, the WHATWG `Response` returned by
`sendRequest`, `registerDeferredCleanup` for dataset writes that must
survive a throwing handler, and explicit enqueue strategies now that
`include` globs are ANDed with the default same-hostname strategy.
- The ignore-ssl test now configures TLS verification on the http
client, because the crawler-level `ignoreSslErrors` option is not wired
to the default client in v4. That dangling option deserves a separate
fix or removal, since it currently does nothing.
- The impit test pins session fingerprints, since the random default
fingerprint overrides the client's browser impersonation.
- Added ES2022 to the actor tsconfigs' `lib` (a bare `["DOM"]` drops the
ES lib and broke compilation on `ErrorOptions`).
- Skipped the zero-concurrency queue test: it stages a stuck queue
through the v3 client-side `inProgress` set, which the rewritten queue
doesn't have.
- Fixed the camoufox fetch retry loop fetching 5x even on success, and
removed a duplicate `apify` dependency key that silently downgraded the
curl-impersonate actor to SDK v3.
- Commented out the LOCAL storage matrix entry in the workflow, as
`@apify/storage-local` doesn't support v4.
- closes#3998
Unnamed storages (default and aliased) are now purged on start, named
ones are not — crawlee-python's rule. The fs backend sweeps the storage
directories, so leftovers from a previous process are caught too.
Two adjacent bugs, one commit each:
- `createDatasetBackend()` / `({})` didn't open the default storage,
though `StorageIdentifier` says they do
- the `__default__` sentinel leaked into the directory name, so default
storages lived in `storage/datasets/__default__` rather than `default`
`ThrottlingRequestManager` (#3741) sub-queues are alias-keyed, so they
now only survive a restart with `purgeOnStart` off — as its docs already
said.
Adds an `rc` dist-tag option to the `publish-to-npm.yml` workflow so we
can publish v4 release candidates, same setup as in mikro-orm.
Dispatching the workflow with `dist-tag: rc` (from the `v4` branch)
will:
- bump versions via `copy --canary=major --preid=rc`, resolving the next
free `4.0.0-rc.N` from npm (first run produces `4.0.0-rc.0`)
- publish all packages under the `rc` dist-tag (new `publish:rc` script)
- create and push a `v4.0.0-rc.N` git tag pointing at the released
commit
Unlike stable releases, no changelog is generated and no commit is
pushed to the branch. The version bump lives only on npm, and the only
trace in git is the tag (same as mikro-orm's RC flow).
The regular beta canary flow (`dist-tag: v4`/`next`) is unchanged. This
only needs to live on the `v4` branch: `workflow_dispatch` uses the
workflow file from the branch selected in the run dropdown, so `master`
needs no changes.
The `ignoreSslErrors` option stopped working during the v4 HTTP client
interface rework: it was still folded into got-style request options,
but those never reach `httpClient.sendRequest()`. In v3 the option
works, and actors commonly expose it in their input schemas and pass it
into crawler options (e.g. actor-scraper), so this keeps it working
instead of removing it.
The option is renamed to `ignoreTlsErrors`, matching
`session.proxyInfo.ignoreTlsErrors`, the browser pool, and the impit
client (the old name is dropped, documented in the upgrading guide;
actors migrating to v4 rename it in their own code, the SDK does not
touch this option). The crawler forwards it (still defaulting to `true`,
same as v3) as a new `SendRequestOptions.ignoreTlsErrors` flag, which
`BaseHttpClient` also enables for MITM proxy sessions (previously
equally dead). The impit client honors the flag; for custom clients it
is best effort, and `FetchHttpClient` cannot disable TLS verification at
all. The dead plumbing in `getRequestOptions()` is removed and unit
tests cover the forwarding chain.
Converts all `const enum` declarations (`EventType`,
`BROWSER_POOL_EVENTS`, `BROWSER_CONTROLLER_EVENTS`,
`OperatingSystemsName`, `DeviceCategory`) to regular enums. `const enum`
breaks downstream consumers using `isolatedModules` or transpilers that
don't inline them (Babel, esbuild, vitest).
Closes#3125
Types the default cookie jar against `@crawlee/types`' `CookieJar`
interface and dynamically imports `tough-cookie` only when a default jar
actually needs constructing.
Relates to #3549. `sax` was imported eagerly at module scope in
`packages/utils/src/internals/sitemap.ts`, so every consumer of
`@crawlee/utils` paid its load cost even if sitemap parsing was never
used.
`sax.SAXParser` is now loaded via a dynamic `import()` only when an XML
sitemap parser is actually constructed.
No public API changes (this class is only used from the `parseSitemap`
generator).
Adds a `/internal` subpath export to `@crawlee/utils` for symbols that
are only shared between crawlee packages (CSS selectors, type guards,
URL helpers, `CheerioRoot`, etc.), keeping the root entry point for
user-facing utilities. The `crawlee` meta-package naturally only
re-exports the public surface.
Closes#3079
Both are only used on specific code paths (sitemap parsing, CSV export)
but were loaded eagerly by every crawler import, costing ~110ms of the
~500ms `import('@crawlee/http')` on my machine.
Related: #3549
`PlaywrightBrowser._setBrowserType()` has existed since the initial
crawlee commit but was never called, so the `_browserType` field stayed
`undefined` and `browserType()` returned `undefined` at runtime despite
its non-optional `BrowserType` signature. Persistent contexts are the
default (`useIncognitoPages: false`), so a call like
`browser.browserType().name()` on the wrapper threw a `TypeError`. The
incognito and remote-connection paths return the native Playwright
`Browser` and were not affected.
This wires the existing setter in `PlaywrightPlugin._launch()` where the
wrapper is created. `this.library` is the `BrowserType` that launched
the context, so the wrapper now provides the consistent API with
Playwright's `Browser` that its docblock describes. Also adds a test
covering both the persistent-context wrapper and the native incognito
browser.
Closes#3945.
`@crawlee/types` no longer imports `CookieJar`/`SerializedCookieJar`
from `tough-cookie` - it declares its own structurally-compatible
interfaces instead, so `tough-cookie` is no longer part of its
dependency tree. `tough-cookie` stays a direct dependency everywhere
it's actually used (`core`, `http-client`, `impit-client`).
Harry