The self-hosting guide predated the 0.9.0 secure-by-default release and
its quickstarts produced an unreachable server. Verified against a fresh
0.9.2 deployment.
self-hosting.md:
- Require CRAWL4AI_API_TOKEN in every quickstart (docker run, compose,
manual build); explain the loopback-only bind and the resulting
"connection reset" on published ports, plus startup-delay troubleshooting
- Compose: .llm.env is required; the token must be set inside it (host
`export` is not forwarded to the container)
- Replace the removed inline-Python hooks API (~600 lines) with the
declarative hooks reference: CRAWL4AI_HOOKS_ENABLED flag, action table,
/hooks/info, and 0.8.x migration notes
- Rewrite /screenshot and /pdf for the artifact flow; note output_path is
removed and currently silently ignored
- Fix dashboard URL (/dashboard, not /monitor); document the token bar in
the playground and dashboard UIs
- Fix stale versions (0.8.0 -> 0.9.2); drop the SDK function-hooks example
MIGRATION.md:
- Token setup instructions for compose (.llm.env) and docker run (explicit
-e form); clarify the loopback failure symptom
- Document CRAWL4AI_HOOKS_ENABLED and the silent failure modes of legacy
hooks.code and output_path
.llm.env.example:
- Add CRAWL4AI_API_TOKEN with guidance; add CRAWL4AI_HOOKS_ENABLED
The router-level token_dep was intentionally removed (PR #2063) because
it broke WebSocket upgrades. Update the structural test to assert the
dependency is absent — auth is enforced by AuthGateMiddleware.
Playground Advanced Config always 400'd because /config/dump requires
type but pyConfigToJson only sent code. Include #cfg-type, align stream
fallback with dump shape, and teach shouldUseStream both nestings.
Monitor dashboard WS /monitor/ws 500'd because router-level token_dep is
HTTP Request-only and cannot inject into WebSocket scopes. Auth remains
on AuthGateMiddleware; admin routes keep require_admin.
Fixes#2059Fixes#2060
The auth gate fix (cbf5e5c) intentionally makes UI shell pages load
without credentials. Update the posture-gate tests to match: move
/dashboard/ and /playground/ from PROTECTED_ENDPOINTS to a new
PUBLIC_UI_PATHS list with an explicit 200 assertion.
- supervisord: write pidfile to /tmp (tmpfs) instead of read-only /app
- redis: set working dir to /var/lib/redis (tmpfs) for RDB snapshots
Fixes "could not write pidfile" and "Failed opening temp RDB file" errors
in read-only root filesystem deployment (docker-compose.yml security hardening).
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
The AuthGateMiddleware blocked UI static pages with 401 because browsers
cannot attach Authorization headers to top-level navigation. The UI shell
serves no data, so it is safe to load without credentials.
- Add public_prefixes to AuthGateMiddleware for prefix-based path bypass
- Register /dashboard, /playground, /static as public prefixes
- Add token input bar to both playground and dashboard UIs
- Replace all bare fetch() calls with authFetch() that attaches Bearer token
- Append ?token= to monitor WebSocket URL (gate already accepts it for WS)
All API/data routes remain fail-closed behind the auth gate.
Closes#2037
Squash of security/docker-hardening-2026-06 (the full R1-R7 redesign) plus the
three new 0.9.0 security fixes (download path traversal, streaming-path SSRF,
extra_args RCE) and uniform egress pinning across all browser-fetch endpoints.
Version 0.9.0. Supersedes the 0.8.8/0.8.9 non-breaking patches with their
breaking-by-design variants; the 0.8.8/0.8.9 CHANGELOG and credit ledger
entries are preserved. The old exec-based hook_manager.py is removed in favor
of declarative hook_registry.py. Security suite: 309 passed, 1 xfailed.
See deploy/docker/MIGRATION.md for the breaking-change migration guide.
0.8.8's SSRF check validated the crawl target URL but not the proxy address, so
an unauthenticated /crawl, /crawl/stream, or /crawl/job could route the browser
through a proxy pointing at an internal IP and reach internal services / cloud
metadata. Reported by Geo (geo-chen).
Fix (backward compatible): validate every proxy destination with the same
not-is_global check used for crawl URLs, before the browser is built -
browser_config.proxy, browser_config.proxy_config.server,
crawler_config.proxy_config.server - and strip proxy/DNS-redirecting flags
(--proxy-server / --proxy-pac-url / --proxy-bypass-list / --host-resolver-rules)
from extra_args. A legitimate public proxy still works; configure proxies via
proxy_config (validated), not raw extra_args flags. _enforce_proxy_safety is
called in both crawl handlers (and covers /crawl/job transitively); HTTPException
passthrough added so the 400 is not masked as a 500.
Bump 0.8.8 -> 0.8.9 (__version__ + Dockerfile). 20 new tests; full security
suite 161 pass. Changelog, release blog, README, SECURITY-CREDITS updated.
This vector was already fixed in the upcoming secure-by-default release; 0.8.9
brings it forward because it is an unauthenticated SSRF.
Backward-compatible fixes for the Docker server - features keep working, only
the unsafe behavior is closed. (The secure-by-default redesign is the later
major.)
- SSRF: replace the explicit blocklist with the one rule (reject any resolved
IP where not ip.is_global) evaluated on embedded IPv4 transition forms too,
closing the gaps - IPv6 unspecified ::, NAT64 64:ff9b::/96, 6to4 2002::/16,
v4-mapped. Error messages are now opaque (no resolved-IP leak).
- output_path arbitrary write: harden validate_output_path with realpath
containment (defeats a symlinked path component) and write via O_NOFOLLOW
(write_output_file). output_path stays supported.
- LLM base_url key exfil: ignore a request-supplied base_url in /md, /llm,
/llm/job; the endpoint is always server-derived. Field still accepted (no
4xx) for compatibility.
- env:SECRET_KEY exfil gadget: LLMConfig refuses env: resolution of protected
names (SECRET/PASSWORD/PRIVATE substrings, CRAWL4AI*/AWS_SECRET* prefixes,
SECRET_KEY/REDIS_PASSWORD/TOKEN). Normal provider keys (OPENAI_API_KEY, ...)
unaffected.
- CRLF log injection: CRLFSafeFilter strips CR/LF/control from log records.
- Webhook header injection: sanitize_webhook_headers (name pattern, no control
chars, deny hop-by-hop/sensitive) at send time + a WebhookConfig validator
for early 422.
Bump 0.8.7 -> 0.8.8 (__version__ + Dockerfile C4AI_VER). 30 new behavioral
tests; existing 111 security tests + 112 library config tests still pass.
NOT included (breaking -> deferred to the major): auth-by-default, trust
boundary, declarative hooks, output_path removal, base_url/provider removal,
loopback bind, redis password, TLS-verify-on, CORS, bounded queue. The
exec-hook RCE and unauth-by-default criticals have no non-breaking fix and are
closed only in the major (hooks are already off by default).
Caught during internal review. `http://[::ffff:127.0.0.1]/` bypassed
validate_webhook_url because getaddrinfo returns ::ffff:7f00:1, which
is not in any IPv4 blocklist (127.0.0.0/8) nor IPv6 blocklist (::1/128).
Fix: added _expand_ip_candidates() helper that unwraps IPv4 from
IPv4-mapped (::ffff:X.Y.Z.W, via .ipv4_mapped) and IPv4-compatible
(::X.Y.Z.W, via low-32-bits) IPv6 addresses. Blocklist now checks
both the original IP and the unwrapped IPv4 form.
Added 6 new TestIPv6MappedBypass tests covering:
- Loopback, RFC 1918, link-local (cloud metadata) via ::ffff: mapping
- IPv4-compatible variant (::127.0.0.1)
- Regression test that plain ::1 still blocked
Also updated stale test assertion in test_eval_security_adversarial:
hasattr, type, __build_class__ were removed from hook builtins in
batch 2 but the test still expected hasattr to remain.
DO NOT PUSH until release day.
Reported by secsys_codex (2026-04-18): /md, /crawl, /llm endpoints
pass user URLs to crawler.arun() with no private IP validation.
- Add validate_url_destination() to utils.py with opt-out via
CRAWL4AI_ALLOW_INTERNAL_URLS=true env var for users who need
to crawl internal services.
- Integrate into validate_url_scheme() (covers all server.py endpoints).
- Add validation at all 4 URL entry points in api.py (handle_llm_qa,
handle_markdown_request, create_new_task, handle_crawl_request).
- raw: URLs bypass check (inline HTML, no network fetch).
- 16 adversarial + source coverage tests added.
- secsys_codex added to SECURITY-CREDITS.md.
DO NOT PUSH until release day.
Addresses the gi_frame.f_back chain exploit reported by Song Binglin (q1uf3ng).
- Delete _safe_eval_expression() and _SAFE_EVAL_BUILTINS entirely from
extraction_strategy.py. Dead security-sensitive code is a liability.
The eval path was already disabled; this removes the function itself.
- Fix hook_manager.py module injection: replace broken exec("import X", ns)
pattern (silently failed due to missing __import__) with direct module
injection. Sanitize asyncio to strip subprocess access (RCE vector).
- Add startup warning when CRAWL4AI_API_TOKEN is unset (all endpoints
unauthenticated).
- Expand adversarial test suite to 87 tests: hook sandbox escapes,
asyncio.subprocess RCE verification, end-to-end exploit payload from
vuln report, dead code deletion checks, codebase eval/exec audit.
The /crawl endpoint now accepts an optional crawler_configs field
(list of CrawlerRunConfig dicts) alongside the existing crawler_config.
When provided with multiple URLs, each config is deserialized and passed
as a list to arun_many(), enabling per-URL configuration with url_matcher
patterns. Single-URL requests and requests without crawler_configs are
unchanged (backward compatible).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Starlette's Route wraps async functions in request_response(), calling
handler(request) instead of handler(scope, receive, send). This broke
the MCP SSE endpoint which needs raw ASGI access. Fix: use a callable
class instead of an async function — Route passes class instances
through as raw ASGI apps without wrapping.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Bump version to 0.8.5 across all references (Dockerfile, README,
Docker README, blog index, __version__.py).
Add release notes, blog post, demo verification script (13 real-crawl
tests), and releases directory entry.
Key highlights:
- Anti-bot detection with 3-tier proxy escalation
- Shadow DOM flattening
- Deep crawl cancellation
- Config defaults API
- 60+ bug fixes and critical security patches
The monitor's update_timeline(), get_health_summary(), and
get_browser_list() all acquired the crawler pool's global LOCK to read
pool stats. That same lock is held during slow browser start/close
operations (get_crawler, janitor, close_all), causing the monitor to
block indefinitely and the pod to become unresponsive after sustained
crawling.
Replaced all three lock acquisitions in monitor.py with a lock-free
get_pool_snapshot() in crawler_pool.py that returns shallow dict copies.
Under CPython's GIL, dict.copy() and len() are atomic — safe for
read-only monitoring with at most slightly stale counts.
Replace @app.get() with starlette.routing.Route() for the SSE handler.
The MCP SDK's SseServerTransport calls raw ASGI (scope, receive, send)
internally, which conflicts with Starlette's middleware wrapping.
Also update CONTRIBUTORS.md for PR #1829.
- #1611: /llm GET endpoint hardcoded server's LLM_PROVIDER. Added optional
provider, temperature, base_url query params with fallback to server config.
Consistent with /md and /llm/job endpoints.
- #1817: Redis connection used non-existent config["redis"]["uri"]. Now builds
URL from host/port/password/db/ssl config fields with REDIS_HOST, REDIS_PORT,
REDIS_PASSWORD environment variable overrides.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Bug fixes:
- Verify redirect targets are alive before returning from URL seeder (#1622)
- Wire mean_delay/max_range from CrawlerRunConfig into dispatcher rate limiter (#1786)
- Use DOMParser instead of innerHTML in process_iframes to prevent XSS (#1796)
Security/Docker:
- Require api_token for /token endpoint when configured (#1795)
- Deep-crawl streaming now mirrors Python library behavior via arun() (#1798)
CI:
- Bump GitHub Actions to latest versions - checkout v6, setup-python v6,
build-push-action v6, setup-buildx v4, login v4 (#1734)
Features:
- Support type-list pipeline in JsonCssExtractionStrategy for chained
extraction like ["attribute", "regex"] (#1290)
- Add --json-ensure-ascii CLI flag and JSON_ENSURE_ASCII config setting
for Unicode preservation in JSON output (#1668)
Add opt-in BrowserConfig flags (avoid_ads, avoid_css) for blocking ad/tracker
domains and CSS resources at the browser context level. Refactor crawler pool
with release_crawler() and active_requests tracking to prevent janitor from
closing browsers with in-flight requests. Add proper finally blocks to all
Docker API/server handlers. Update docs for new config options.
Inspired by #1689.
The httpx.AsyncClient() default 5s timeout causes TimeoutException on
slow LLM-backed endpoints. The exception bypasses the HTTPStatusError
handler, propagating as an unhandled error to the MCP framework.
- Add `timeout` parameter to `attach_mcp()` (default None = no limit)
- Pass timeout through to `_make_http_proxy()` and `httpx.AsyncClient()`
- Catch `httpx.TimeoutException` and surface it as HTTP 504
Fixes#1769https://claude.ai/code/session_01LpranMwFBtQU7kFrV5EHAB
Soham Kukreti