Nasrin
f6f7f1b551
* Fix: Use correct URL variable for raw HTML extraction (#1116) - Prevents full HTML content from being passed as URL to extraction strategies - Added unit tests to verify raw HTML and regular URL processing Fix: Wrong URL variable used for extraction of raw html * Fix #1181: Preserve whitespace in code blocks during HTML scraping The remove_empty_elements_fast() method was removing whitespace-only span elements inside <pre> and <code> tags, causing import statements like "import torch" to become "importtorch". Now skips elements inside code blocks where whitespace is significant. * Refactor Pydantic model configuration to use ConfigDict for arbitrary types * Fix EmbeddingStrategy: Uncomment response handling for the variations and clean up mock data. ref #1621 * Fix: permission issues with .cache/url_seeder and other runtime cache dirs. ref #1638 * fix: ensure BrowserConfig.to_dict serializes proxy_config * feat: make LLM backoff configurable end-to-end - extend LLMConfig with backoff delay/attempt/factor fields and thread them through LLMExtractionStrategy, LLMContentFilter, table extraction, and Docker API handlers - expose the backoff parameter knobs on perform_completion_with_backoff/aperform_completion_with_backoff and document them in the md_v2 guides * reproduced AttributeError from #1642 * pass timeout parameter to docker client request * added missing deep crawling objects to init * generalized query in ContentRelevanceFilter to be a str or list * import modules from enhanceable deserialization * parameterized tests * Fix: capture current page URL to reflect JavaScript navigation and add test for delayed redirects. ref #1268 * refactor: replace PyPDF2 with pypdf across the codebase. ref #1412 * Add browser_context_id and target_id parameters to BrowserConfig Enable Crawl4AI to connect to pre-created CDP browser contexts, which is essential for cloud browser services that pre-create isolated contexts. Changes: - Add browser_context_id and target_id parameters to BrowserConfig - Update from_kwargs() and to_dict() methods - Modify BrowserManager.start() to use existing context when provided - Add _get_page_by_target_id() helper method - Update get_page() to handle pre-existing targets - Add test for browser_context_id functionality This enables cloud services to: 1. Create isolated CDP contexts before Crawl4AI connects 2. Pass context/target IDs to BrowserConfig 3. Have Crawl4AI reuse existing contexts instead of creating new ones * Add cdp_cleanup_on_close flag to prevent memory leaks in cloud/server scenarios * Fix: add cdp_cleanup_on_close to from_kwargs * Fix: find context by target_id for concurrent CDP connections * Fix: use target_id to find correct page in get_page * Fix: use CDP to find context by browserContextId for concurrent sessions * Revert context matching attempts - Playwright cannot see CDP-created contexts * Add create_isolated_context flag for concurrent CDP crawls When True, forces creation of a new browser context instead of reusing the default context. Essential for concurrent crawls on the same browser to prevent navigation conflicts. * Add context caching to create_isolated_context branch Uses contexts_by_config cache (same as non-CDP mode) to reuse contexts for multiple URLs with same config. Still creates new page per crawl for navigation isolation. Benefits batch/deep crawls. * Add init_scripts support to BrowserConfig for pre-page-load JS injection This adds the ability to inject JavaScript that runs before any page loads, useful for stealth evasions (canvas/audio fingerprinting, userAgentData). - Add init_scripts parameter to BrowserConfig (list of JS strings) - Apply init_scripts in setup_context() via context.add_init_script() - Update from_kwargs() and to_dict() for serialization * Fix CDP connection handling: support WS URLs and proper cleanup Changes to browser_manager.py: 1. _verify_cdp_ready(): Support multiple URL formats - WebSocket URLs (ws://, wss://): Skip HTTP verification, Playwright handles directly - HTTP URLs with query params: Properly parse with urlparse to preserve query string - Fixes issue where naive f"{cdp_url}/json/version" broke WS URLs and query params 2. close(): Proper cleanup when cdp_cleanup_on_close=True - Close all sessions (pages) - Close all contexts - Call browser.close() to disconnect (doesn't terminate browser, just releases connection) - Wait 1 second for CDP connection to fully release - Stop Playwright instance to prevent memory leaks This enables: - Connecting to specific browsers via WS URL - Reusing the same browser with multiple sequential connections - No user wait needed between connections (internal 1s delay handles it) Added tests/browser/test_cdp_cleanup_reuse.py with comprehensive tests. * Update gitignore * Some debugging for caching * Add _generate_screenshot_from_html for raw: and file:// URLs Implements the missing method that was being called but never defined. Now raw: and file:// URLs can generate screenshots by: 1. Loading HTML into a browser page via page.set_content() 2. Taking screenshot using existing take_screenshot() method 3. Cleaning up the page afterward This enables cached HTML to be rendered with screenshots in crawl4ai-cloud. * Add PDF and MHTML support for raw: and file:// URLs - Replace _generate_screenshot_from_html with _generate_media_from_html - New method handles screenshot, PDF, and MHTML in one browser session - Update raw: and file:// URL handlers to use new method - Enables cached HTML to generate all media types * Add crash recovery for deep crawl strategies Add optional resume_state and on_state_change parameters to all deep crawl strategies (BFS, DFS, Best-First) for cloud deployment crash recovery. Features: - resume_state: Pass saved state to resume from checkpoint - on_state_change: Async callback fired after each URL for real-time state persistence to external storage (Redis, DB, etc.) - export_state(): Get last captured state manually - Zero overhead when features are disabled (None defaults) State includes visited URLs, pending queue/stack, depths, and pages_crawled count. All state is JSON-serializable. * Fix: HTTP strategy raw: URL parsing truncates at # character The AsyncHTTPCrawlerStrategy.crawl() method used urlparse() to extract content from raw: URLs. This caused HTML with CSS color codes like #eee to be truncated because # is treated as a URL fragment delimiter. Before: raw:body{background:#eee} -> parsed.path = 'body{background:' After: raw:body{background:#eee} -> raw_content = 'body{background:#eee' Fix: Strip the raw: or raw:// prefix directly instead of using urlparse, matching how the browser strategy handles it. * Add base_url parameter to CrawlerRunConfig for raw HTML processing When processing raw: HTML (e.g., from cache), the URL parameter is meaningless for markdown link resolution. This adds a base_url parameter that can be set explicitly to provide proper URL resolution context. Changes: - Add base_url parameter to CrawlerRunConfig.__init__ - Add base_url to CrawlerRunConfig.from_kwargs - Update aprocess_html to use base_url for markdown generation Usage: config = CrawlerRunConfig(base_url='https://example.com') result = await crawler.arun(url='raw:{html}', config=config) * Add prefetch mode for two-phase deep crawling - Add `prefetch` parameter to CrawlerRunConfig - Add `quick_extract_links()` function for fast link extraction - Add short-circuit in aprocess_html() for prefetch mode - Add 42 tests (unit, integration, regression) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Updates on proxy rotation and proxy configuration * Add proxy support to HTTP crawler strategy * Add browser pipeline support for raw:/file:// URLs - Add process_in_browser parameter to CrawlerRunConfig - Route raw:/file:// URLs through _crawl_web() when browser operations needed - Use page.set_content() instead of goto() for local content - Fix cookie handling for non-HTTP URLs in browser_manager - Auto-detect browser requirements: js_code, wait_for, screenshot, etc. - Maintain fast path for raw:/file:// without browser params Fixes #310 * Add smart TTL cache for sitemap URL seeder - Add cache_ttl_hours and validate_sitemap_lastmod params to SeedingConfig - New JSON cache format with metadata (version, created_at, lastmod, url_count) - Cache validation by TTL expiry and sitemap lastmod comparison - Auto-migration from old .jsonl to new .json format - Fixes bug where incomplete cache was used indefinitely * Update URL seeder docs with smart TTL cache parameters - Add cache_ttl_hours and validate_sitemap_lastmod to parameter table - Document smart TTL cache validation with examples - Add cache-related troubleshooting entries - Update key features summary * Add MEMORY.md to gitignore * Docs: Add multi-sample schema generation section Add documentation explaining how to pass multiple HTML samples to generate_schema() for stable selectors that work across pages with varying DOM structures. Includes: - Problem explanation (fragile nth-child selectors) - Solution with code example - Key points for multi-sample queries - Comparison table of fragile vs stable selectors * Fix critical RCE and LFI vulnerabilities in Docker API deployment Security fixes for vulnerabilities reported by ProjectDiscovery: 1. Remote Code Execution via Hooks (CVE pending) - Remove __import__ from allowed_builtins in hook_manager.py - Prevents arbitrary module imports (os, subprocess, etc.) - Hooks now disabled by default via CRAWL4AI_HOOKS_ENABLED env var 2. Local File Inclusion via file:// URLs (CVE pending) - Add URL scheme validation to /execute_js, /screenshot, /pdf, /html - Block file://, javascript:, data: and other dangerous schemes - Only allow http://, https://, and raw: (where appropriate) 3. Security hardening - Add CRAWL4AI_HOOKS_ENABLED=false as default (opt-in for hooks) - Add security warning comments in config.yml - Add validate_url_scheme() helper for consistent validation Testing: - Add unit tests (test_security_fixes.py) - 16 tests - Add integration tests (run_security_tests.py) for live server Affected endpoints: - POST /crawl (hooks disabled by default) - POST /crawl/stream (hooks disabled by default) - POST /execute_js (URL validation added) - POST /screenshot (URL validation added) - POST /pdf (URL validation added) - POST /html (URL validation added) Breaking changes: - Hooks require CRAWL4AI_HOOKS_ENABLED=true to function - file:// URLs no longer work on API endpoints (use library directly) * Enhance authentication flow by implementing JWT token retrieval and adding authorization headers to API requests * Add release notes for v0.7.9, detailing breaking changes, security fixes, new features, bug fixes, and documentation updates * Add release notes for v0.8.0, detailing breaking changes, security fixes, new features, bug fixes, and documentation updates Documentation for v0.8.0 release: - SECURITY.md: Security policy and vulnerability reporting guidelines - RELEASE_NOTES_v0.8.0.md: Comprehensive release notes - migration/v0.8.0-upgrade-guide.md: Step-by-step migration guide - security/GHSA-DRAFT-RCE-LFI.md: GitHub security advisory drafts - CHANGELOG.md: Updated with v0.8.0 changes Breaking changes documented: - Docker API hooks disabled by default (CRAWL4AI_HOOKS_ENABLED) - file:// URLs blocked on Docker API endpoints Security fixes credited to Neo by ProjectDiscovery * Add examples for deep crawl crash recovery and prefetch mode in documentation * Release v0.8.0: The v0.8.0 Update - Updated version to 0.8.0 - Added comprehensive demo and release notes - Updated all documentation * Update security researcher acknowledgment with a hyperlink for Neo by ProjectDiscovery * Add async agenerate_schema method for schema generation - Extract prompt building to shared _build_schema_prompt() method - Add agenerate_schema() async version using aperform_completion_with_backoff - Refactor generate_schema() to use shared prompt builder - Fixes Gemini/Vertex AI compatibility in async contexts (FastAPI) * Fix: Enable litellm.drop_params for O-series/GPT-5 model compatibility O-series (o1, o3) and GPT-5 models only support temperature=1. Setting litellm.drop_params=True auto-drops unsupported parameters instead of throwing UnsupportedParamsError. Fixes temperature=0.01 error for these models in LLM extraction. --------- Co-authored-by: rbushria <rbushri@gmail.com> Co-authored-by: AHMET YILMAZ <tawfik@kidocode.com> Co-authored-by: Soham Kukreti <kukretisoham@gmail.com> Co-authored-by: Chris Murphy <chris.murphy@klaviyo.com> Co-authored-by: unclecode <unclecode@kidocode.com> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
271 lines
10 KiB
Python
271 lines
10 KiB
Python
"""
|
|
Cache validation using HTTP conditional requests and head fingerprinting.
|
|
|
|
Uses httpx for fast, lightweight HTTP requests (no browser needed).
|
|
This module enables smart cache validation to avoid unnecessary full browser crawls
|
|
when content hasn't changed.
|
|
|
|
Validation Strategy:
|
|
1. Send HEAD request with If-None-Match / If-Modified-Since headers
|
|
2. If server returns 304 Not Modified → cache is FRESH
|
|
3. If server returns 200 → fetch <head> and compare fingerprint
|
|
4. If fingerprint matches → cache is FRESH (minor changes only)
|
|
5. Otherwise → cache is STALE, need full recrawl
|
|
"""
|
|
|
|
import httpx
|
|
from dataclasses import dataclass
|
|
from typing import Optional, Tuple
|
|
from enum import Enum
|
|
|
|
from .utils import compute_head_fingerprint
|
|
|
|
|
|
class CacheValidationResult(Enum):
|
|
"""Result of cache validation check."""
|
|
FRESH = "fresh" # Content unchanged, use cache
|
|
STALE = "stale" # Content changed, need recrawl
|
|
UNKNOWN = "unknown" # Couldn't determine, need recrawl
|
|
ERROR = "error" # Request failed, use cache as fallback
|
|
|
|
|
|
@dataclass
|
|
class ValidationResult:
|
|
"""Detailed result of a cache validation attempt."""
|
|
status: CacheValidationResult
|
|
new_etag: Optional[str] = None
|
|
new_last_modified: Optional[str] = None
|
|
new_head_fingerprint: Optional[str] = None
|
|
reason: str = ""
|
|
|
|
|
|
class CacheValidator:
|
|
"""
|
|
Validates cache freshness using lightweight HTTP requests.
|
|
|
|
This validator uses httpx to make fast HTTP requests without needing
|
|
a full browser. It supports two validation methods:
|
|
|
|
1. HTTP Conditional Requests (Layer 3):
|
|
- Uses If-None-Match with stored ETag
|
|
- Uses If-Modified-Since with stored Last-Modified
|
|
- Server returns 304 if content unchanged
|
|
|
|
2. Head Fingerprinting (Layer 4):
|
|
- Fetches only the <head> section (~5KB)
|
|
- Compares fingerprint of key meta tags
|
|
- Catches changes even without server support for conditional requests
|
|
"""
|
|
|
|
def __init__(self, timeout: float = 10.0, user_agent: Optional[str] = None):
|
|
"""
|
|
Initialize the cache validator.
|
|
|
|
Args:
|
|
timeout: Request timeout in seconds
|
|
user_agent: Custom User-Agent string (optional)
|
|
"""
|
|
self.timeout = timeout
|
|
self.user_agent = user_agent or "Mozilla/5.0 (compatible; Crawl4AI/1.0)"
|
|
self._client: Optional[httpx.AsyncClient] = None
|
|
|
|
async def _get_client(self) -> httpx.AsyncClient:
|
|
"""Get or create the httpx client."""
|
|
if self._client is None:
|
|
self._client = httpx.AsyncClient(
|
|
http2=True,
|
|
timeout=self.timeout,
|
|
follow_redirects=True,
|
|
headers={"User-Agent": self.user_agent}
|
|
)
|
|
return self._client
|
|
|
|
async def validate(
|
|
self,
|
|
url: str,
|
|
stored_etag: Optional[str] = None,
|
|
stored_last_modified: Optional[str] = None,
|
|
stored_head_fingerprint: Optional[str] = None,
|
|
) -> ValidationResult:
|
|
"""
|
|
Validate if cached content is still fresh.
|
|
|
|
Args:
|
|
url: The URL to validate
|
|
stored_etag: Previously stored ETag header value
|
|
stored_last_modified: Previously stored Last-Modified header value
|
|
stored_head_fingerprint: Previously computed head fingerprint
|
|
|
|
Returns:
|
|
ValidationResult with status and any updated metadata
|
|
"""
|
|
client = await self._get_client()
|
|
|
|
# Build conditional request headers
|
|
headers = {}
|
|
if stored_etag:
|
|
headers["If-None-Match"] = stored_etag
|
|
if stored_last_modified:
|
|
headers["If-Modified-Since"] = stored_last_modified
|
|
|
|
try:
|
|
# Step 1: Try HEAD request with conditional headers
|
|
if headers:
|
|
response = await client.head(url, headers=headers)
|
|
|
|
if response.status_code == 304:
|
|
return ValidationResult(
|
|
status=CacheValidationResult.FRESH,
|
|
reason="Server returned 304 Not Modified"
|
|
)
|
|
|
|
# Got 200, extract new headers for potential update
|
|
new_etag = response.headers.get("etag")
|
|
new_last_modified = response.headers.get("last-modified")
|
|
|
|
# If we have fingerprint, compare it
|
|
if stored_head_fingerprint:
|
|
head_html, _, _ = await self._fetch_head(url)
|
|
if head_html:
|
|
new_fingerprint = compute_head_fingerprint(head_html)
|
|
if new_fingerprint and new_fingerprint == stored_head_fingerprint:
|
|
return ValidationResult(
|
|
status=CacheValidationResult.FRESH,
|
|
new_etag=new_etag,
|
|
new_last_modified=new_last_modified,
|
|
new_head_fingerprint=new_fingerprint,
|
|
reason="Head fingerprint matches"
|
|
)
|
|
elif new_fingerprint:
|
|
return ValidationResult(
|
|
status=CacheValidationResult.STALE,
|
|
new_etag=new_etag,
|
|
new_last_modified=new_last_modified,
|
|
new_head_fingerprint=new_fingerprint,
|
|
reason="Head fingerprint changed"
|
|
)
|
|
|
|
# Headers changed and no fingerprint match
|
|
return ValidationResult(
|
|
status=CacheValidationResult.STALE,
|
|
new_etag=new_etag,
|
|
new_last_modified=new_last_modified,
|
|
reason="Server returned 200, content may have changed"
|
|
)
|
|
|
|
# Step 2: No conditional headers available, try fingerprint only
|
|
if stored_head_fingerprint:
|
|
head_html, new_etag, new_last_modified = await self._fetch_head(url)
|
|
|
|
if head_html:
|
|
new_fingerprint = compute_head_fingerprint(head_html)
|
|
|
|
if new_fingerprint and new_fingerprint == stored_head_fingerprint:
|
|
return ValidationResult(
|
|
status=CacheValidationResult.FRESH,
|
|
new_etag=new_etag,
|
|
new_last_modified=new_last_modified,
|
|
new_head_fingerprint=new_fingerprint,
|
|
reason="Head fingerprint matches"
|
|
)
|
|
elif new_fingerprint:
|
|
return ValidationResult(
|
|
status=CacheValidationResult.STALE,
|
|
new_etag=new_etag,
|
|
new_last_modified=new_last_modified,
|
|
new_head_fingerprint=new_fingerprint,
|
|
reason="Head fingerprint changed"
|
|
)
|
|
|
|
# Step 3: No validation data available
|
|
return ValidationResult(
|
|
status=CacheValidationResult.UNKNOWN,
|
|
reason="No validation data available (no etag, last-modified, or fingerprint)"
|
|
)
|
|
|
|
except httpx.TimeoutException:
|
|
return ValidationResult(
|
|
status=CacheValidationResult.ERROR,
|
|
reason="Validation request timed out"
|
|
)
|
|
except httpx.RequestError as e:
|
|
return ValidationResult(
|
|
status=CacheValidationResult.ERROR,
|
|
reason=f"Validation request failed: {type(e).__name__}"
|
|
)
|
|
except Exception as e:
|
|
# On unexpected error, prefer using cache over failing
|
|
return ValidationResult(
|
|
status=CacheValidationResult.ERROR,
|
|
reason=f"Validation error: {str(e)}"
|
|
)
|
|
|
|
async def _fetch_head(self, url: str) -> Tuple[Optional[str], Optional[str], Optional[str]]:
|
|
"""
|
|
Fetch only the <head> section of a page.
|
|
|
|
Uses streaming to stop reading after </head> is found,
|
|
minimizing bandwidth usage.
|
|
|
|
Args:
|
|
url: The URL to fetch
|
|
|
|
Returns:
|
|
Tuple of (head_html, etag, last_modified)
|
|
"""
|
|
client = await self._get_client()
|
|
|
|
try:
|
|
async with client.stream(
|
|
"GET",
|
|
url,
|
|
headers={"Accept-Encoding": "identity"} # Disable compression for easier parsing
|
|
) as response:
|
|
etag = response.headers.get("etag")
|
|
last_modified = response.headers.get("last-modified")
|
|
|
|
if response.status_code != 200:
|
|
return None, etag, last_modified
|
|
|
|
# Read until </head> or max 64KB
|
|
chunks = []
|
|
total_bytes = 0
|
|
max_bytes = 65536
|
|
|
|
async for chunk in response.aiter_bytes(4096):
|
|
chunks.append(chunk)
|
|
total_bytes += len(chunk)
|
|
|
|
content = b''.join(chunks)
|
|
# Check for </head> (case insensitive)
|
|
if b'</head>' in content.lower() or b'</HEAD>' in content:
|
|
break
|
|
if total_bytes >= max_bytes:
|
|
break
|
|
|
|
html = content.decode('utf-8', errors='replace')
|
|
|
|
# Extract just the head section
|
|
head_end = html.lower().find('</head>')
|
|
if head_end != -1:
|
|
html = html[:head_end + 7]
|
|
|
|
return html, etag, last_modified
|
|
|
|
except Exception:
|
|
return None, None, None
|
|
|
|
async def close(self):
|
|
"""Close the HTTP client and release resources."""
|
|
if self._client:
|
|
await self._client.aclose()
|
|
self._client = None
|
|
|
|
async def __aenter__(self):
|
|
"""Async context manager entry."""
|
|
return self
|
|
|
|
async def __aexit__(self, exc_type, exc_val, exc_tb):
|
|
"""Async context manager exit."""
|
|
await self.close()
|