Marco Vinciguerra
adc92f7eff
A page that could not be scraped as intended was indistinguishable from one that could. FetchNode's default path (ChromiumLoader -> ascrape_playwright) dropped the Response returned by page.goto(), so a 404, 403, 500, captcha wall or login redirect reached the LLM as ordinary content and the model answered "NA" with nothing in the logs to explain why. Reported in #1102, where en.wikipedia.org/wiki/Timpson_(company) 404s (the article is at Timpson_(retailer)) and the run still looked clean. Two deterministic, LLM-free guards, both warnings so existing behaviour is unchanged for anyone deliberately scraping error pages: - ChromiumLoader keeps the Response from every page.goto() call site (ascrape_playwright, ascrape_playwright_scroll, ascrape_with_js_support) and warns on status >= 400. This mirrors what the opt-in use_soup=True path in FetchNode has always done. - ParseNode warns when the parsed content contains none of the terms the user asked about — schema field names plus the significant words of the prompt. A 200 response can still reach the LLM without the requested data: content behind JavaScript that never rendered, a field inside a <script> blob the parser drops, or a document truncated beyond the model window. Zero matches is a deliberately conservative bar, so the warning stays quiet when the page simply phrases the answer differently. The graphs now pass their schema to ParseNode so it has the field names available. Verified against the URLs from the issue: the 404 now logs "Received HTTP 404 for .../Timpson_(company); the scraped content is likely an error page" before returning NA, while the corrected URL stays silent and answers 1865. Also drops three dead imports from smart_scraper_multi_batch_graph.py, which ruff blocks on now that the file is touched. Fixes #1102 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Test section
Regarding the tests for the folder graphs and nodes it was created a specific repo as a example (link of the repo). The test website is hosted here. Remember to activating Ollama and having installed the LLM on your pc
For running the tests run the command:
pytest