chore(eval): make the reproduction package self-contained and public-safe (#90)
* chore(eval): make the reproduction package self-contained and public-safe
Reproduction:
- Consolidate to a single entry doc: rename eval/REPRODUCE.md -> eval/README.md
(auto-renders on the eval/ dir) and update all references.
- Document the three ways to supply tile images to the reader: self-hosted serve
with materialized tiles, the public search API, and a self-hosted serve that
renders tiles on demand from a kiwix ZIM. Make TILES_DIR optional so the reader
can use serve-returned base64 tiles instead of a local corpus.
- Remove the reader-side "local-wiki" rendering path entirely so all tile
rendering happens serve-side: drop LocalWikiTiledScreenshotRetriever, the
lookup_reference_url machinery, and the --local-wiki / --local-wiki-screenshot-dir
/ --lookup-reference-url flags (they relied on an out-of-tree module and
hardcoded placeholder paths).
- Run benchmarks on their full (filtered) sets; keep the 1000-example subsample
only for nq and sqa, which is what the paper reports.
Hygiene:
- Read the Jina API key from JINA_API_KEY instead of a hardcoded default.
- Update stale notes now that the indexes and tile corpus are published on HF.
- Drop internal working-notes docs from the repo and scrub leftover references to
old internal repo/module names.
- Tidy .gitignore to use generic patterns and add a scoped eval/.gitignore.
* fix(eval): grade NQ correctly and feed serve-returned base64 tiles to the reader
Two pre-existing bugs that silently broke cells, surfaced by a from-scratch
reproduction of NQ/base via the public API:
- grader: NQ / NQ-Tables store gold answers under `gold_answers`, which
`_golds_for` did not read, so every exact-match cell graded 0. Add the key.
- reader: retrieved tiles were attached only when `os.path.exists(path)` is true,
but in the public-API and on-demand-render modes the serve returns each tile
inline as base64 (the "path" is the base64 string itself). Those tiles were
silently dropped and the reader answered from parametric memory (effectively
naive). Add a `_tile_image_b64` helper that accepts both a local file path and
inline base64, and use it at all four tile-attachment sites (build_messages's
three branches + `_encode_images_to_content`).
Verified: NQ/base smoke (20 examples, public API, base64 tiles) now grades
10/20 = 50% with tiles reaching the reader, vs 0 before.
* chore(eval): relax deps to floors + optional reader extra; document the public-API path
- Loosen eval/pyproject deps from exact `==` pins to `>=` floors. uv.lock is the
reproducibility contract (`uv sync --frozen`); the blanket `==` pins were a freeze
artifact, inconsistent with the rest of the workspace (which floors), and they conflicted
with vLLM (numpy 2.4.6 vs vLLM's numpy<2.3).
- Add an optional `reader` extra (`uv sync --extra reader`) that installs vLLM 0.19.0, so the
Qwen3.5-4B reader can be self-hosted from this package. vLLM needs numpy<2.3, hence the cap
(nothing in the base requires >=2.3).
- README: document installing the reader via the extra; add the command to run a cell against
the public API (reproduce.sh hardcodes localhost, so it can't do the public-API path); note
that public :30001 serves the un-normed base index, which does not match the paper's
base/lora cells (those use search_index_normed_v2).
* fix(eval): grade NQ/NQ-Tables with the LLM judge to match the paper
The paper's published NQ/NQT numbers use the gpt-4.1 LLM judge (semantic match), not strict
exact-match. The reader answers short but paraphrases the gold span, so strict exact-match
scores ~20pp lower (≈ the naive number) even when the answer is correct — measured NQ base
= 29% exact-match vs 53% LLM-judge (paper 57.9).
- grader: add a --llm-judge flag; for nq/nq_tables/triviaqa it grades with the existing
gpt-4.1 judge (ground truth = "Any of: <gold aliases>") instead of exact-match. The
default stays strict exact-match (no API key needed).
- reproduce.sh: pass --llm-judge for nq/nqt so the turnkey reproduction matches the paper.
- README: note NQ/NQT paper numbers are LLM-judge; update the cell table + grader section.
* fix(serve)+docs(eval): add torchvision to the serve extra; document self-hosting the search serve
- The `serve` extra was missing torchvision, which transformers' Qwen3-VL processor imports —
self-hosting the search serve failed until it was added by hand. Add it (cu129 via the
existing tool.uv.sources, matching the embed extra).
- eval/README: add a "self-hosting the search serve" note — the [serve] install, the
~217G index + ~220G RAM, where articles.json lives (the pixelrag-tiles dataset), and the
tiles options (full corpus vs --render-on-demand from a kiwix ZIM).
* fix(serve,render): make Mode-3 on-demand rendering actually work
Two real bugs that made the serve's --render-on-demand path (Mode 3 in the eval README)
return empty tiles, surfaced by a from-scratch reproduction:
- serve: the async /search handler called _ondemand_chunk_b64 synchronously, which runs
render_url -> asyncio.run() inside the running event loop -> RuntimeError, swallowed, so
image_base64 was always empty (the reader fell back to closed-book). Offload it with
asyncio.to_thread so render_url's asyncio.run() runs in a thread with no running loop.
- render: the cdp/fast_cdp backends force GPU rasterization, which needs GPU device access
(the `render` group on lab machines: `sudo usermod -aG render $USER`). On a headless box
without it the renderer crashes and CDP capture hangs forever. Add PIXELSHOT_DISABLE_GPU=1
to fall back to CPU rasterization; the default (GPU) path is unchanged.
* fix(render): default to CPU rasterization (the render machines have no GPU)
Our render/serve machines are GPU-less (the serve runs --device cpu; nvidia-smi has no
driver), and the stable production capture config is GPU-rasterization-incompatible anyway.
Forcing `--enable-gpu-rasterization` by default makes Chrome crash / CDP capture hang on a
box without real GPU device access.
Flip the default to `--disable-gpu` (CPU rasterization — works on GPU-less and headless
boxes, the common case). On a properly configured graphics-GPU render box, set
PIXELSHOT_ENABLE_GPU=1 for ~2x throughput (GPU rasterization can also produce blank
captures, so verify output there).
* revert(render): keep the original GPU-rasterization flags (no-GPU boxes render fine)
A previous commit flipped the cdp/fast_cdp default to --disable-gpu on the theory that
--enable-gpu-rasterization crashes on GPU-less / headless machines. That was wrong: verified
on a no-GPU box, the original flags render correctly and fast (Chrome falls back to software
rasterization, no crash). The crash seen on one borrowed cluster box was specific to that
host (patched headless_shell + datacenter GPU), not a general problem — not a reason to change
the default. Restore cdp.py / fast_cdp.py to the original BROWSER_ARGS.
The genuine Mode-3 bug (the serve's async handler calling render_url -> asyncio.run inside a
running event loop) stays fixed via asyncio.to_thread in serve/api.py — that is unrelated to
the GPU flags.
* fix(eval): configurable retrieval timeout so a slow serve doesn't silently go closed-book
LocalAPIRetriever/TextAPIRetriever POST to the search serve with a hardcoded 600s timeout and
batch_size 32. Against a slow serve (e.g. --render-on-demand, which renders a page per tile),
a batch exceeds 600s, the request times out, the batch is cached EMPTY, and the reader answers
closed-book — a silently invalid run (looks like a bad score, not an error). Make the timeout
configurable via PIXELRAG_RETRIEVAL_TIMEOUT (default 600, unchanged for fast serves); document
raising it for mode 3 in the README.
* fix(serve): render on-demand tiles in a subprocess so the serve doesn't wedge
The on-demand renderer called render_url() in-process. render_url uses asyncio.run() + multiprocessing.Pool (fork); the SECOND in-process call deadlocks (fork-in-threaded-process), so a long-lived serve renders the first retrieved page fine and then wedges on the next request. Verified on a no-GPU box: same-process 2nd render hangs (SIGKILL after 40s), a fresh subprocess per render runs 3/3 clean (~1.85s each). Run each render in a subprocess (timeout via PIXELRAG_RENDER_TIMEOUT, default 120s). Pairs with the to_thread offload in api.py that keeps the blocking subprocess off the event loop.
* refactor(serve): render on-demand tiles via the pixelshot CLI, not inline python -c
The pixelshot CLI already exposes --viewport-width / --tile-height (defaults 875/8192), so use the standard module entry (python -m pixelrag_render.render --output ... --viewport-width ... --tile-height ... --workers 1) instead of an inline python -c string. Same subprocess isolation that fixes the wedge, cleaner invocation. Verified: 2 consecutive renders 1.67s/1.57s, correct *.png.tiles output, no hang.
* fix(render): default GPU rasterization OFF (it was an inherited no-op that crashes some boxes)
git shows --enable-gpu-rasterization/--force-gpu-rasterization have been in the cdp/fast_cdp backends since the initial release (82c5794, then carried through PR #38) — inherited on the assumption GPU rasterization speeds up capture. It doesn't: headless Chrome falls back to the software renderer and ignores the flags (verified on a no-GPU box: enable == disable == 1.7s; the bottleneck is capture IPC, not rasterization — see docs/screenshot-throughput-optimization.md). On a box that has a GPU device but no access (e.g. /dev/dri without the render group), Chrome tries the GPU, the GPU process crashes on init, and capture hangs. Default to --disable-gpu; opt in with PIXELSHOT_ENABLE_GPU=1 only on a real graphics-GPU box with device access. Default render verified working (1.62s/tile).
* docs(render): note throughput is capture-only; measure with the bench harness
Clarify that the documented t/s excludes Chrome startup (the bench timer starts after strategy.setup()). A hand-rolled end-to-end loop that counts the ~49s 48-worker startup reports ~13 t/s, which is the startup tax not the capture rate. Re-measured on the reference EPYC 7763 box with the harness: 130 t/s capture-only (200 maxi-ZIM pages, 48w, raw).
* perf(serve): reuse a persistent Chrome for on-demand render (--cdp-url), ~3x faster
On-demand render started a fresh Chrome per page; the cold start dominates (seconds locally, tens of seconds on a cold/NFS box — that's why a Mode-3 NQ run was ~2.3min/example, ~95% of it in rendering). Keep one headless Chrome alive in OnDemandTiles and render each page in a fresh tab via the pixelshot --cdp-url attach mode (PR #76). Still a separate subprocess per render (render_url's asyncio.run + multiprocessing.Pool would deadlock on a 2nd in-process call), but it attaches to the live browser instead of launching one. Verified on a no-GPU box: 6 consecutive renders 0.47s each (vs ~1.6s/page launching fresh) with no hang; Chrome is auto-restarted if a render fails, killed at exit.
* perf(serve): optional mmap index load (PIXELRAG_INDEX_MMAP=1) for near-instant startup
read_index reads the whole index into RAM — a multi-100G index over NFS takes ~1-2h to load. With PIXELRAG_INDEX_MMAP=1, faiss.read_index uses IO_FLAG_MMAP: startup is near-instant (no full read), inverted lists are paged in on demand at query time, and the OS page cache keeps hot lists resident. Ideal for a few-hundred-query eval that touches only a fraction of the index.
This commit is contained in:
Zhifei Li
-24
@@ -37,33 +37,9 @@ logs/
|
||||
*.log
|
||||
arxiv
|
||||
demos/e2e/output/
|
||||
eval/eval_output/
|
||||
.superpowers/
|
||||
.vercel
|
||||
|
||||
# Large local retrieval artifacts (not committed)
|
||||
eval/tmp_news_state.db
|
||||
eval/live_pixel_full.json
|
||||
eval/live_reader_full.json
|
||||
eval/frozen_reader_full.json
|
||||
eval/mms_base_live.jsonl
|
||||
eval/mms_lora_live.jsonl
|
||||
eval/mms_naive_live.jsonl
|
||||
eval/evqa_base_landmarks_live.jsonl
|
||||
eval/evqa_base_inat_live.jsonl
|
||||
eval/evqa_lora_landmarks_live.jsonl
|
||||
eval/evqa_lora_inat_live.jsonl
|
||||
eval/mms_traf_live.jsonl
|
||||
eval/evqa_traf_landmarks.jsonl
|
||||
eval/evqa_traf_inaturalist.jsonl
|
||||
eval/evqa_naive_landmarks.jsonl
|
||||
eval/evqa_naive_inaturalist.jsonl
|
||||
eval/mms_naive_nothink.jsonl
|
||||
eval/evqa_base_landmarks_nothink.jsonl
|
||||
eval/evqa_base_inat_nothink.jsonl
|
||||
eval/evqa_lora_landmarks_nothink.jsonl
|
||||
eval/evqa_lora_inat_nothink.jsonl
|
||||
eval/paper_grader_out/
|
||||
node_modules/
|
||||
.next/
|
||||
|
||||
|
||||
@@ -1,239 +0,0 @@
|
||||
# Screenshot Throughput Optimization — Working Progress
|
||||
|
||||
## Target: 150 t/s @ 100% correct (8192px tiles, maxi Wikipedia)
|
||||
|
||||
## Current Best
|
||||
|
||||
| Config | t/s | Correct | Notes |
|
||||
|--------|-----|---------|-------|
|
||||
| multi-process 48w (frameStoppedLoading) | **91** | 100% ✓ | Stable, production-ready |
|
||||
| multi-process 48w (frameNavigated) | **98** | 100% ✓ | Stable (igpu incompatible) |
|
||||
| multi-process 48w (2000 art) | **113** | 99.8% ✓ | Steady-state |
|
||||
| igpu 48w + frameStoppedLoading | **117-132** | 90-97% | Fast but 3-10% about:blank |
|
||||
| igpu 48w + directClip | **128-148** | 48-90% | Fastest, worst correctness |
|
||||
|
||||
## Production System Comparison
|
||||
|
||||
The wiki-screenshot production system (`~/pixelrag-src/wiki-screenshot/`) uses:
|
||||
```python
|
||||
wait_fonts = False # for kiwix/ZIM datasource
|
||||
wait_images = False # for kiwix/ZIM datasource
|
||||
pre_screenshot_delay = 0.5 # fixed 500ms sleep, no fonts.ready
|
||||
```
|
||||
- Playwright-based (not CDP websocket)
|
||||
- GPU-accelerated (8× L40S per machine)
|
||||
- Multi-machine: 4 machines × ~70-80 t/s = ~290 t/s total
|
||||
- Full Wikipedia (8.28M articles) processed in ~1 day
|
||||
|
||||
Our optimizations added `fonts.ready + eager images + double-rAF` for pixel-perfect
|
||||
correctness. Production skips these waits entirely (`pre_screenshot_delay=0` in
|
||||
coordinator). This is safe for Kiwix because all assets (including fonts) are served
|
||||
from localhost — they load before `wait_until="load"` fires.
|
||||
|
||||
Gemini Vision validation of 5000 production tiles:
|
||||
- 0% BROKEN_RENDER, 0% ERROR_PAGE (rendering is correct without font wait)
|
||||
- 12% BLANK/PARTIAL_BLANK (tile loop overshoots page height — separate bug)
|
||||
|
||||
**Benchmark result**: Removing font/image wait gives only +4% throughput (99 vs 96 t/s)
|
||||
because nav is not the bottleneck — capture IPC is. The 290 t/s production rate comes
|
||||
from 4 machines × GPU acceleration, not from skipping font waits.
|
||||
|
||||
## Pipeline Bottleneck Analysis
|
||||
|
||||
```
|
||||
Stage Capacity Bottleneck?
|
||||
Nav 430 pg/s No (3.4x headroom)
|
||||
Capture 125 t/s YES (C/T_c = 48/321ms)
|
||||
|
||||
Steady-state theoretical: 125-150 t/s
|
||||
Actual (200 art): 98 t/s (75% utilization, 25% = nav serial)
|
||||
Actual (2000 art): 113 t/s (85% utilization)
|
||||
```
|
||||
|
||||
Per-capture breakdown at 48 concurrent:
|
||||
- IPC roundtrip: 181ms (ForceRedraw browser→renderer→compositor, 8 async hops)
|
||||
- DrawRenderPass: 62ms (composite 136 quads)
|
||||
- CopyDrawnRenderPass: 46ms (memcpy 28MB)
|
||||
|
||||
Throughput = `C / T_c(C)` converges at ~125-130 t/s (USL contention curve).
|
||||
Nav latency (186ms) does not affect steady-state throughput (Little's Law).
|
||||
Minimum workers to saturate capture: `C × (1 + T_nav/T_cap) = 72`.
|
||||
|
||||
## Chromium Patches (in custom build)
|
||||
|
||||
| Patch | File | Impact |
|
||||
|-------|------|--------|
|
||||
| rawFilePath | page_handler.cc + Page.pdl | Async write raw BGRA to /dev/shm (ThreadPool) |
|
||||
| directClip | page_handler.cc + Page.pdl | CopyFromSurface(src_rect) without emulation change |
|
||||
| skipRedraw | page_handler.cc + Page.pdl | ForceRedrawWithCallback → CopyFromSurface |
|
||||
| ForceRedrawWithCallback | render_widget_host_impl.cc | Lightweight ForceRedraw with commit callback |
|
||||
| directClip ForceRedraw fix | page_handler.cc | directClip also does ForceRedraw before copy |
|
||||
|
||||
## Strategy Architecture
|
||||
|
||||
Strategies separated from bench framework:
|
||||
- `pixelrag_render.strategies/` — capture strategies (CDPPhased, CDPSequential, etc.)
|
||||
- `pixelrag_render.bench/` — measurement harness with GT validation + experiment dump
|
||||
- `Bench` class: `bench.run(strategy)` → GT cache + capture + verify + JSON dump
|
||||
|
||||
### CDPPhasedStrategy (best strategy)
|
||||
- Work-stealing queue (asyncio.Queue, not round-robin)
|
||||
- Semaphore-limited concurrent captures
|
||||
- `wait_for_event("Page.frameStoppedLoading")` filtered by main frameId
|
||||
- Per-tile semaphore release (fine-grained pipelining)
|
||||
- Configurable: tile_height, nav_timeout, use_direct_clip, extra_chrome_args
|
||||
|
||||
### WebsocketConnection
|
||||
- Background `_recv_loop` for multiplexed CDP
|
||||
- `wait_for_event(method, timeout, filter_fn)` for async event listening
|
||||
- Supports concurrent `cdp()` calls via pending futures dict
|
||||
|
||||
## What Was Tried
|
||||
|
||||
### Worked
|
||||
- ✅ rawFilePath: async write bypasses PNG encoding (+15%)
|
||||
- ✅ directClip: parallel tile capture within viewport
|
||||
- ✅ Phased strategy: semaphore-limited captures reduce contention (+15%)
|
||||
- ✅ Work-stealing queue: better load balancing
|
||||
- ✅ frameNavigated/frameStoppedLoading wait: fixes igpu about:blank race
|
||||
- ✅ Presentation feedback ForceRedraw: 100% correct (but slower)
|
||||
|
||||
### Partially Worked
|
||||
- ⚠️ --in-process-gpu: 120+ t/s but 5-10% about:blank captures
|
||||
- ⚠️ SwapPromise ForceRedraw: shot_p50 325→303ms (7% gain)
|
||||
- ⚠️ directClip for all tiles: fast but correctness depends on ForceRedraw
|
||||
|
||||
### Did Not Work
|
||||
- ❌ --single-process: 168 t/s but 74% correct
|
||||
- ❌ peekPixels (SkiaRenderer): headless uses SoftwareRenderer
|
||||
- ❌ Immediate BeginFrame feedback flush: breaks frame pipeline
|
||||
- ❌ CDPScreenshotNewSurface: RequestRepaintOnNewSurface overhead
|
||||
- ❌ 2-tab pipelining: Chrome UI thread serializes ForceRedraw
|
||||
- ❌ Chrome flags (disable-lcd-text etc.): ±2%
|
||||
- ❌ headless_shell: slower than chrome (no shared HTTP cache)
|
||||
- ❌ One-shot strategy: launch overhead 1-2s/process
|
||||
- ❌ Firefox Playwright: 2.6x slower than Chrome
|
||||
- ❌ Servo (servoshell 0.1.0): stub package, not ready
|
||||
- ❌ CEF (cefpython3): abandoned, no modern Python wheel
|
||||
- ❌ WebKitGTK snapshot: needs GPU/display access
|
||||
- ❌ RequestRepaintOnNewSurface in skipRedraw: didn't fix igpu race
|
||||
- ❌ Bitmap dimension retry: about:blank renders at full viewport size
|
||||
- ❌ Pixel content retry: can't distinguish white page from about:blank
|
||||
|
||||
## igpu About:blank Root Cause
|
||||
|
||||
Chrome `--in-process-gpu` has two bugs at 48 concurrent workers:
|
||||
1. **frameNavigated event not fired**: Chrome sometimes silently drops
|
||||
`Page.frameNavigated` CDP event under high concurrency.
|
||||
Fix: use `Page.frameStoppedLoading` (always reliable).
|
||||
2. **Compositor surface race**: ForceRedraw's presentation feedback arrives
|
||||
before the new page's CompositorFrame is activated in viz. CopyFromSurface
|
||||
reads the old surface (about:blank at 875×8192, indistinguishable from
|
||||
real page by dimensions). No reliable Python-side detection possible.
|
||||
|
||||
## Key Analysis Methods Used
|
||||
|
||||
- **Pipeline bottleneck analysis** (closed queueing model)
|
||||
- **Little's Law**: steady-state throughput = C/T_c when capture-bound
|
||||
- **USL contention curve**: C/T_c(C) convergence at ~125-130 t/s
|
||||
- **USE method**: Utilization (79%), Saturation (semaphore queue), Errors (0)
|
||||
- **Per-capture breakdown**: DrawRenderPass (57ms) + CopyDrawnRenderPass (18ms)
|
||||
+ IPC overhead (95ms) measured via Chromium instrumentation
|
||||
|
||||
## Scale Estimate
|
||||
|
||||
30M tiles (18.7M articles × ~1.6 tiles/article):
|
||||
- Single machine 98 t/s: 30M/98 = 85 hours = **3.5 days**
|
||||
- Single machine 120 t/s (igpu, 95% correct): 30M/120 = 69 hours = **2.9 days**
|
||||
- 4 machines × 98 t/s = 392 t/s: 30M/392 = 21 hours = **< 1 day**
|
||||
- Production system (290 t/s, 4 machines): ~1 day (matches historical data)
|
||||
|
||||
## Production Pipeline: fast_cdp backend
|
||||
|
||||
```
|
||||
Chrome 48w (capture) → /dev/shm (raw BGRA) → ProcessPool 4w (JPEG) → disk
|
||||
98 t/s 28MB/tile ~100 t/s 100KB/tile
|
||||
```
|
||||
|
||||
Architecture:
|
||||
- `render_articles()` in `pixelrag_render.backends.fast_cdp`
|
||||
- Capture: CDPPhasedStrategy logic (work-stealing, semaphore, frameStoppedLoading)
|
||||
- Compression: `concurrent.futures.ProcessPoolExecutor(4)` — GIL-free, separate cores
|
||||
- Raw files in /dev/shm/pixelrag_render/ — auto-deleted after compression
|
||||
- Output: JPEG tiles + tiles.json manifest per article
|
||||
|
||||
Key: compression never blocks capture. Chrome writes raw → returns immediately.
|
||||
Compression reads raw file asynchronously on different CPU cores.
|
||||
|
||||
128-core machine: 48 cores for Chrome, 4 cores for JPEG, 76 cores idle.
|
||||
JPEG compression of 875×8192 takes ~10-20ms → 4 cores handle 200-400 t/s →
|
||||
plenty of headroom over 98 t/s capture rate.
|
||||
|
||||
Storage: 30M tiles × 100KB JPEG = ~3 TB
|
||||
|
||||
## GPU Acceleration (Brewster H200 findings)
|
||||
|
||||
Lab machines have 8× H200/B200 GPUs but:
|
||||
- `/dev/dri/renderD*` needs `render` group membership (no sudo)
|
||||
- Docker daemon not running; rootless docker lacks nvidia-container-toolkit
|
||||
- SwiftShader (CPU Vulkan) doesn't improve throughput vs software rendering
|
||||
- headless Chrome ignores `--use-gl` flags (GPU process crashes on init)
|
||||
- When GPU DOES init (via Xvfb + ANGLE), missing NVIDIA userspace drivers in container
|
||||
|
||||
To unlock GPU: `sudo usermod -aG render $USER` on lab machine.
|
||||
Expected impact: 4x faster DrawRenderPass based on production system data.
|
||||
|
||||
## Backend reconciliation & SPA-render fix (2026-06-11)
|
||||
|
||||
### The three render code paths (who actually runs what)
|
||||
- `backends/websocket.py` — the **shipped** general-purpose renderer. The `pixelshot`
|
||||
CLI, the `pixelbrowse` skill, and the `pixelrag index` pipeline (`render_urls`,
|
||||
`backend="cdp"`/`"websocket"`) all go through it. Simple: per-worker queue, inline
|
||||
JPEG over CDP, no extra deps.
|
||||
- `backends/fast_cdp.py` — high-throughput batch path (`render_articles`): phased-logic
|
||||
capture + rawFilePath to /dev/shm + ProcessPool JPEG. **No in-repo caller** — invoked
|
||||
only by an out-of-repo ops script. The 8.28M flagship Wikipedia index was built by a
|
||||
*separate* system (Playwright/GPU/4-machine, see "Production System Comparison"), not
|
||||
by either of these.
|
||||
- `strategies/*` — the benchmarking menu; used only by `bench/`. Kept as research scaffolding.
|
||||
|
||||
### Regression fixed: websocket backend rendered SPAs / tall pages wrong
|
||||
`backends/websocket.py` had drifted from the established capture pattern — it had **no
|
||||
nav-completion wait** (fired `document.fonts.ready` immediately after `Page.navigate`)
|
||||
and **no per-tile scroll**, both of which `fast_cdp` and the production strategies have.
|
||||
Consequences:
|
||||
- JS/SPA pages were measured/captured mid-hydration at a transient (often much taller)
|
||||
layout → tiled into mostly-empty space = blank tiles (this is the "tile loop overshoots
|
||||
page height" blank bug noted under "Production System Comparison", here root-caused).
|
||||
- At small `tile_height` (the skill uses 1568) every tile past the first was blank,
|
||||
because content below the short device viewport is never rasterized without scrolling.
|
||||
|
||||
Fix (verified in `bench/` against ground truth at 100% on the smoke set):
|
||||
- Wait for the `load` event before measuring/capturing (`readyState==='complete'`
|
||||
shortcut + 12s cap). SSR pages fire `load` ~as fast as `fonts.ready`, so ~0 cost
|
||||
(measured: Wikipedia render time unchanged).
|
||||
- Scroll each tile into view before capture (mirrors `fast_cdp`).
|
||||
- Optional `--wait-network-idle` (JS PerformanceObserver) for pages that fetch content
|
||||
after load; off by default (costs a quiet window/page), on by default in the skill.
|
||||
|
||||
### Raw vs inline-JPEG is the dominant throughput lever (measured, 48w, N=600, this box)
|
||||
| config | correct | t/s | note |
|
||||
|---|---|---|---|
|
||||
| phased **raw** (fast_cdp config) | 99.7% | **306** | capture-only in bench; JPEG is decoupled/parallel |
|
||||
| phased jpeg (inline) | 98.2% | 182 | Chrome encodes JPEG on the capture critical path |
|
||||
| sequential raw | 99.7% | 221 | |
|
||||
| sequential jpeg (inline) | 98.2% | 142 | |
|
||||
|
||||
Takeaways: (1) **inline JPEG encoding is the bottleneck** — bypassing it with rawFilePath
|
||||
+ parallel compression is ~+56-68%. (2) phased's semaphore/work-stealing buys ~+38% over
|
||||
sequential **in raw mode** (in jpeg mode the encoding bottleneck masks it to ~+8% — an
|
||||
earlier jpeg-only comparison was misleading). So `fast_cdp` is ~2x the simple inline path
|
||||
at batch scale and is **kept**. Absolute t/s here is optimistic (capture-only, short
|
||||
window, 128-core box) vs the ~91-113 production figure; the *ratios* are the point.
|
||||
|
||||
### Design direction
|
||||
Ship **one simple backend** (`websocket.py`, inline JPEG) for the CLI/skill/`pixelrag index`
|
||||
— that scale doesn't need the raw+decoupled machinery, and the flagship index uses the
|
||||
separate system anyway. Keep `fast_cdp` + `strategies/` as batch/research code. The shared
|
||||
capture-readiness logic (load wait, scroll) should eventually live in one place so the
|
||||
shipped backend can't silently drift from the correct pattern again.
|
||||
@@ -1,592 +0,0 @@
|
||||
# Reproducing Paper Results
|
||||
|
||||
> **Paper**: *PixelRAG: Retrieval and Generation in Pixel Space over Millions of Web Screenshots*
|
||||
>
|
||||
> This document maps every table and figure in the paper to the exact commands needed to reproduce the numbers.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### Infrastructure
|
||||
|
||||
| Component | Description | Where |
|
||||
|-----------|-------------|-------|
|
||||
| **Wikipedia tile index (base)** | 28M vectors, Qwen3-VL-Embedding-2B (pretrained) | `pixelrag-data/search_index/` (215 GB FAISS IVF, dim=2048) |
|
||||
| **Wikipedia tile index (fine-tuned)** | 26M vectors, LoRA checkpoint-200 | `pixelrag-data/search_index_lora_vit_ckpt200_v2/` (202 GB) |
|
||||
| **Wikipedia text index** | 15.7M text chunks (1024 tokens, Trafilatura) | `pixelrag-data/text_search_index_1024/` (121 GB) |
|
||||
| **Article metadata** | URL↔tile mapping for 7.1M articles | `pixelrag-data/articles.json` (199 MB) |
|
||||
| **Tile images** | ~30M PNG tiles (1024×1024) | Remote NFS or local SSD (~5.6 TB) |
|
||||
| **News tile index** | 3.6M tiles (BBC/AP/CNN) for LiveVQA | S3: `s3://wiki-screenshot-tiles-backup/kiwix_tiles/news_image_search_index/` |
|
||||
| **News text index** | 866K text chunks for news | S3: `s3://wiki-screenshot-tiles-backup/kiwix_tiles/news_text_search_index/` |
|
||||
| **News tiles** | Raw PNG tiles for news articles | S3: `s3://wiki-screenshot-tiles-backup/kiwix_tiles/news_tiles/` |
|
||||
| **LoRA adapter** | Fine-tuned embedding LoRA weights | S3: `s3://wiki-screenshot-tiles-backup/kiwix_tiles/adapters/lora_vit_ckpt200/` |
|
||||
| **Kiwix ZIM** | Offline Wikipedia for HTML baselines | S3: `s3://wiki-screenshot-tiles-backup/kiwix_tiles/zim/` |
|
||||
|
||||
All S3 paths use AWS profile `leann` (`aws s3 --profile leann ...`).
|
||||
|
||||
### Services to Start
|
||||
|
||||
```bash
|
||||
# 1. Screenshot search API (port 30888) — serves the pixel tile index
|
||||
pixelrag-serve \
|
||||
--index-dir pixelrag-data/search_index \ # or search_index_lora_vit_ckpt200_v2
|
||||
--tiles-dir /path/to/wikipedia_tiles \
|
||||
--articles-json pixelrag-data/articles.json \
|
||||
--model Qwen/Qwen3-VL-Embedding-2B \
|
||||
--device cuda --port 30888
|
||||
|
||||
# 2. Text search API (port 30889) — serves the text chunk index
|
||||
pixelrag-serve \
|
||||
--index-dir pixelrag-data/text_search_index_1024 \
|
||||
--tiles-dir /path/to/text_chunks \
|
||||
--articles-json pixelrag-data/articles.json \
|
||||
--model Qwen/Qwen3-VL-Embedding-2B \
|
||||
--device cuda --port 30889
|
||||
|
||||
# 3. Reader model (port 8000) — vLLM serving Qwen3.5-4B (default reader)
|
||||
vllm serve Qwen/Qwen3.5-4B-Instruct \
|
||||
--port 8000 --tensor-parallel-size 1 \
|
||||
--max-model-len 32768
|
||||
```
|
||||
|
||||
### Environment
|
||||
|
||||
```bash
|
||||
cd ~/pixelrag/eval
|
||||
|
||||
# Install eval dependencies (one-time)
|
||||
uv pip install pandas tqdm trafilatura openai aiohttp datasets huggingface-hub
|
||||
|
||||
# For grading
|
||||
export OPENAI_API_KEY=sk-... # GPT-4.1 judge
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Table 1: Main Results (6 Benchmarks × 4 Methods)
|
||||
|
||||
**Reader**: Qwen3.5-4B, **k=3**, **Grader**: GPT-4.1 judge (except LiveVQA = exact match)
|
||||
|
||||
### No Retrieval (baseline)
|
||||
|
||||
```bash
|
||||
# SimpleQA — no retrieval
|
||||
python run_bench.py \
|
||||
--task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
|
||||
--num-examples 1000 --no-think
|
||||
|
||||
# NQ — no retrieval
|
||||
python run_bench.py \
|
||||
--task nq --model Qwen/Qwen3.5-4B-Instruct \
|
||||
--num-examples 1000 --no-think
|
||||
|
||||
# NQ-Tables — no retrieval
|
||||
python run_bench.py \
|
||||
--task nq_tables --model Qwen/Qwen3.5-4B-Instruct \
|
||||
--num-examples 1000 --no-think
|
||||
|
||||
# MMSearch — no retrieval (300 examples)
|
||||
python run_bench.py \
|
||||
--task mmsearch --model Qwen/Qwen3.5-4B-Instruct \
|
||||
--num-examples 300 --no-think
|
||||
|
||||
# EVQA — no retrieval (landmarks, automatic only, n=749)
|
||||
python run_bench.py \
|
||||
--task encyclopedic_vqa --model Qwen/Qwen3.5-4B-Instruct \
|
||||
--evqa-dataset-filter landmarks --evqa-question-type-filter automatic \
|
||||
--num-examples 749 --no-think
|
||||
|
||||
# LiveVQA — see "LiveVQA Separate Pipeline" section below
|
||||
```
|
||||
|
||||
### Text Retrieval — Trafilatura (Text → Text)
|
||||
|
||||
Requires: text search API on port 30889 with Trafilatura-parsed text chunks.
|
||||
|
||||
```bash
|
||||
# SimpleQA — Trafilatura text retrieval
|
||||
python run_bench.py \
|
||||
--task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
|
||||
--text-api --text-api-url http://localhost:30889/search \
|
||||
--retrieval-top-k 3 --num-examples 1000 --no-think
|
||||
|
||||
# NQ — Trafilatura text retrieval
|
||||
python run_bench.py \
|
||||
--task nq --model Qwen/Qwen3.5-4B-Instruct \
|
||||
--text-api --text-api-url http://localhost:30889/search \
|
||||
--retrieval-top-k 3 --num-examples 1000 --no-think
|
||||
|
||||
# NQ-Tables
|
||||
python run_bench.py \
|
||||
--task nq_tables --model Qwen/Qwen3.5-4B-Instruct \
|
||||
--text-api --text-api-url http://localhost:30889/search \
|
||||
--retrieval-top-k 3 --num-examples 1000 --no-think
|
||||
|
||||
# MMSearch (multimodal query: text + image → text index)
|
||||
python run_bench.py \
|
||||
--task mmsearch --model Qwen/Qwen3.5-4B-Instruct \
|
||||
--text-api --text-api-url http://localhost:30889/search \
|
||||
--retrieval-top-k 3 --num-examples 300 --no-think
|
||||
|
||||
# EVQA
|
||||
python run_bench.py \
|
||||
--task encyclopedic_vqa --model Qwen/Qwen3.5-4B-Instruct \
|
||||
--text-api --text-api-url http://localhost:30889/search \
|
||||
--evqa-dataset-filter landmarks --evqa-question-type-filter automatic \
|
||||
--retrieval-top-k 3 --num-examples 749 --no-think
|
||||
|
||||
# LiveVQA — see "LiveVQA Separate Pipeline" section below
|
||||
```
|
||||
|
||||
### Text Retrieval — mwparserfromhell
|
||||
|
||||
Same as Trafilatura but requires a separate text index built with mwparserfromhell parser.
|
||||
The text API must be started pointing to that index.
|
||||
|
||||
```bash
|
||||
# Same commands as Trafilatura above, but --text-api-url points to
|
||||
# the mwparserfromhell text index API (different port or index-dir).
|
||||
# The parser choice is baked into the index at build time, not a runtime flag.
|
||||
```
|
||||
|
||||
### PixelRAG (base) — Screenshot → Screenshot
|
||||
|
||||
Requires: screenshot search API on port 30888 with base (pretrained) embedding index.
|
||||
|
||||
```bash
|
||||
# SimpleQA — pixel retrieval (base)
|
||||
python run_bench.py \
|
||||
--task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
|
||||
--local-api --local-api-url http://localhost:30888/search \
|
||||
--retrieval-top-k 3 --num-examples 1000 --no-think
|
||||
|
||||
# NQ — pixel retrieval (base)
|
||||
python run_bench.py \
|
||||
--task nq --model Qwen/Qwen3.5-4B-Instruct \
|
||||
--local-api --local-api-url http://localhost:30888/search \
|
||||
--retrieval-top-k 3 --num-examples 1000 --no-think
|
||||
|
||||
# NQ-Tables
|
||||
python run_bench.py \
|
||||
--task nq_tables --model Qwen/Qwen3.5-4B-Instruct \
|
||||
--local-api --local-api-url http://localhost:30888/search \
|
||||
--retrieval-top-k 3 --num-examples 1000 --no-think
|
||||
|
||||
# MMSearch (multimodal: query image sent alongside text)
|
||||
python run_bench.py \
|
||||
--task mmsearch --model Qwen/Qwen3.5-4B-Instruct \
|
||||
--local-api --local-api-url http://localhost:30888/search \
|
||||
--retrieval-top-k 3 --num-examples 300 --no-think
|
||||
|
||||
# EVQA (multimodal: landmark photo + question text)
|
||||
python run_bench.py \
|
||||
--task encyclopedic_vqa --model Qwen/Qwen3.5-4B-Instruct \
|
||||
--local-api --local-api-url http://localhost:30888/search \
|
||||
--evqa-dataset-filter landmarks --evqa-question-type-filter automatic \
|
||||
--retrieval-top-k 3 --num-examples 749 --no-think
|
||||
|
||||
# LiveVQA — see "LiveVQA Separate Pipeline" section below
|
||||
```
|
||||
|
||||
### PixelRAG (fine-tuned) — Screenshot → Screenshot with LoRA embedding
|
||||
|
||||
Same commands as PixelRAG (base), but the search API must be started with the fine-tuned index:
|
||||
|
||||
```bash
|
||||
# Start search API with fine-tuned index
|
||||
pixelrag-serve \
|
||||
--index-dir pixelrag-data/search_index_lora_vit_ckpt200_v2 \
|
||||
--tiles-dir /path/to/wikipedia_tiles \
|
||||
--articles-json pixelrag-data/articles.json \
|
||||
--model Qwen/Qwen3-VL-Embedding-2B \
|
||||
--peft-adapter /path/to/lora_checkpoint_200 \
|
||||
--device cuda --port 30888
|
||||
```
|
||||
|
||||
Then run the same `--local-api` commands above.
|
||||
|
||||
### Grading
|
||||
|
||||
```bash
|
||||
cd ~/pixelrag/eval
|
||||
|
||||
# Grade with GPT-4.1 judge (Wikipedia QA tasks)
|
||||
python grade.py simpleqa eval_output/simpleqa_*.jsonl
|
||||
python grade.py encyclopedic_vqa eval_output/encyclopedic_vqa_*.jsonl
|
||||
python grade.py mmsearch eval_output/mmsearch_*.jsonl
|
||||
|
||||
# For NQ/NQ-Tables (with LLM judge for paper numbers)
|
||||
python grade.py nq eval_output/nq_*.jsonl --llm-judge
|
||||
python grade.py nq_tables eval_output/nq_tables_*.jsonl --llm-judge
|
||||
|
||||
# For LiveVQA (exact letter match — handled by the LiveVQA pipeline scripts)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Table 3: Retrieval–Reader Modality Ablation
|
||||
|
||||
**Task**: SimpleQA (1000) + LiveVQA (6632), **Reader**: Qwen3.5-4B, **k=3**,
|
||||
**Embedding**: Qwen3-VL-Embedding-2B (base, no LoRA)
|
||||
|
||||
| Row | Retrieval | Reader Input | Flags |
|
||||
|-----|-----------|-------------|-------|
|
||||
| Screenshot → Screenshot | Pixel index | Raw tile images | `--local-api` |
|
||||
| Screenshot → OCR text | Pixel index | OCR'd text from tiles | `--local-api --read-as-text-ocr` |
|
||||
| Text → Rendered image | Text index | Text chunks rendered as PNG | `--text-api --render-as-image` |
|
||||
| Text → Text | Text index | Raw text chunks | `--text-api` |
|
||||
| Text → HTML | Text index | Raw HTML from kiwix | `--text-api --html-dom-lookup` |
|
||||
|
||||
```bash
|
||||
# Screenshot → Screenshot (same as main results PixelRAG base)
|
||||
python run_bench.py \
|
||||
--task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
|
||||
--local-api --local-api-url http://localhost:30888/search \
|
||||
--retrieval-top-k 3 --num-examples 1000 --no-think
|
||||
|
||||
# Screenshot → OCR text
|
||||
python run_bench.py \
|
||||
--task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
|
||||
--local-api --local-api-url http://localhost:30888/search \
|
||||
--read-as-text-ocr --ocr-url http://localhost:8202/v1 \
|
||||
--retrieval-top-k 3 --num-examples 1000 --no-think
|
||||
|
||||
# Text → Rendered image
|
||||
python run_bench.py \
|
||||
--task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
|
||||
--text-api --text-api-url http://localhost:30889/search \
|
||||
--render-as-image \
|
||||
--retrieval-top-k 3 --num-examples 1000 --no-think
|
||||
|
||||
# Text → Text (same as main results Trafilatura)
|
||||
python run_bench.py \
|
||||
--task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
|
||||
--text-api --text-api-url http://localhost:30889/search \
|
||||
--retrieval-top-k 3 --num-examples 1000 --no-think
|
||||
|
||||
# Text → HTML (DOM lookup)
|
||||
python run_bench.py \
|
||||
--task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
|
||||
--text-api --text-api-url http://localhost:30889/search \
|
||||
--html-dom-lookup \
|
||||
--retrieval-top-k 3 --num-examples 1000 --no-think
|
||||
```
|
||||
|
||||
For LiveVQA, use the separate pipeline (see "LiveVQA Separate Pipeline" section) with the corresponding ablation scripts.
|
||||
|
||||
---
|
||||
|
||||
## Table 4: Embedding Training Recipe Ablation
|
||||
|
||||
**Evaluated on mini-datastore** (400 queries, 7426 tiles).
|
||||
|
||||
This ablation uses `--prebuilt-tiles-dir` pointing to the pre-built mini-datastore, with different embedding checkpoints. Each row corresponds to a different embedding training recipe:
|
||||
|
||||
```bash
|
||||
# Base model (no fine-tuning)
|
||||
python run_bench.py \
|
||||
--task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
|
||||
--use-tiled-retrieval --use-qwen3vl-embedding \
|
||||
--qwen3vl-model Qwen/Qwen3-VL-Embedding-2B \
|
||||
--embedding-backend hf \
|
||||
--prebuilt-tiles-dir tiles-hard-mini/ \
|
||||
--retrieval-top-k 3 --num-examples 400 --no-think
|
||||
|
||||
# With LoRA checkpoint (dynamic hard negatives + ViT unfrozen)
|
||||
python run_bench.py \
|
||||
--task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
|
||||
--use-tiled-retrieval --use-qwen3vl-embedding \
|
||||
--qwen3vl-model Qwen/Qwen3-VL-Embedding-2B \
|
||||
--embedding-backend biqwen3 \
|
||||
--peft-adapter /path/to/checkpoint-200 \
|
||||
--prebuilt-tiles-dir tiles-hard-mini/ \
|
||||
--retrieval-top-k 3 --num-examples 400 --no-think
|
||||
```
|
||||
|
||||
The intermediate checkpoints (in-batch negatives, naive hard negatives, dynamic hard negatives frozen) each have their own PEFT adapter path.
|
||||
|
||||
---
|
||||
|
||||
## Figure 2: Token Efficiency (SimpleQA, k=1,2,3, 4 readers)
|
||||
|
||||
**Task**: SimpleQA (1000), **Readers**: Qwen3.5-4B, Qwen3.5-9B, Qwen3.5-27B, Qwen3.6-35B-A3B
|
||||
|
||||
For each reader × k × retrieval method, run:
|
||||
|
||||
```bash
|
||||
# Example: Qwen3.5-4B, k=1, PixelRAG (fine-tuned)
|
||||
python run_bench.py \
|
||||
--task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
|
||||
--local-api --local-api-url http://localhost:30888/search \
|
||||
--retrieval-top-k 3 --reader-top-k 1 \
|
||||
--num-examples 1000 --no-think
|
||||
|
||||
# Example: Qwen3.5-4B, k=2, PixelRAG (fine-tuned)
|
||||
python run_bench.py \
|
||||
--task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
|
||||
--local-api --local-api-url http://localhost:30888/search \
|
||||
--retrieval-top-k 3 --reader-top-k 2 \
|
||||
--num-examples 1000 --no-think
|
||||
|
||||
# Example: Qwen3.5-4B, k=3, PixelRAG (fine-tuned)
|
||||
python run_bench.py \
|
||||
--task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
|
||||
--local-api --local-api-url http://localhost:30888/search \
|
||||
--retrieval-top-k 3 \
|
||||
--num-examples 1000 --no-think
|
||||
```
|
||||
|
||||
> **Optimization**: Use `--retrieval-top-k 3 --reader-top-k N` to retrieve once at k=3 and evaluate at k=1,2,3 from the same JSONL (the full retrieved set is stored in `retrieved_images`).
|
||||
|
||||
For each reader, change `--model` and start the appropriate vLLM server.
|
||||
Repeat for text retrieval (Trafilatura: `--text-api`) and PixelRAG base (base index).
|
||||
|
||||
The plot script is at `arxiv/figures/plot_token_efficiency.py`.
|
||||
|
||||
---
|
||||
|
||||
## Figure 3: Agentic Multi-Hop QA (MoNaCo)
|
||||
|
||||
**Task**: MoNaCo (1315 questions), **Agent**: GPT-5 ReAct, **k=5 per search**
|
||||
|
||||
Uses `eval/run_monaco.py` — a ReAct agent that issues search tool calls.
|
||||
|
||||
```bash
|
||||
cd ~/pixelrag/eval
|
||||
|
||||
# PixelRAG backend
|
||||
python run_monaco.py \
|
||||
--reader gpt-5 \
|
||||
--retrieval pixel \
|
||||
--pixel-api http://localhost:30888/search \
|
||||
--default-top-k 5
|
||||
|
||||
# Text retrieval backend (Trafilatura)
|
||||
python run_monaco.py \
|
||||
--reader gpt-5 \
|
||||
--retrieval text \
|
||||
--text-api http://localhost:30889/search \
|
||||
--default-top-k 5
|
||||
|
||||
# Grade (token F1 computed inline; add --judge for LLM judge F1)
|
||||
python run_monaco.py \
|
||||
--reader gpt-5 \
|
||||
--retrieval pixel \
|
||||
--judge --judge-model gpt-4.1-2025-04-14
|
||||
|
||||
# Or grade existing predictions:
|
||||
python grade.py monaco eval_output/monaco/<run_tag>
|
||||
```
|
||||
|
||||
The dataset (`monaco_version_1_release.jsonl`) should be placed at
|
||||
`eval/data/monaco/` or passed via `--data-path`.
|
||||
|
||||
---
|
||||
|
||||
## Figure 4: Image Compression Curve
|
||||
|
||||
**Task**: SimpleQA (1000), **Reader**: Qwen3.5-4B (base + SFT), k=1..5, compression c=1×/2×/3×
|
||||
|
||||
```bash
|
||||
# No compression (c=1×), k=3
|
||||
python run_bench.py \
|
||||
--task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
|
||||
--local-api --local-api-url http://localhost:30888/search \
|
||||
--retrieval-top-k 5 --reader-top-k 3 \
|
||||
--num-examples 1000 --no-think
|
||||
|
||||
# 2× compression, k=3
|
||||
python run_bench.py \
|
||||
--task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
|
||||
--local-api --local-api-url http://localhost:30888/search \
|
||||
--retrieval-top-k 5 --reader-top-k 3 \
|
||||
--pixel-compress-ratio 2.0 \
|
||||
--num-examples 1000 --no-think
|
||||
|
||||
# 3× compression, k=3
|
||||
python run_bench.py \
|
||||
--task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
|
||||
--local-api --local-api-url http://localhost:30888/search \
|
||||
--retrieval-top-k 5 --reader-top-k 3 \
|
||||
--pixel-compress-ratio 3.0 \
|
||||
--num-examples 1000 --no-think
|
||||
```
|
||||
|
||||
For the SFT reader, replace `--model` with the SFT checkpoint path and serve it via vLLM.
|
||||
|
||||
The plot script is at `arxiv/figures/plot_sft_compression_curve.py`.
|
||||
|
||||
---
|
||||
|
||||
## Table 8: Full Reader-Model Sweep (31 VLMs)
|
||||
|
||||
**Task**: SimpleQA (1000), **k=3**, pixel retrieval (base) vs text retrieval (Trafilatura)
|
||||
|
||||
For each of the 31 reader models, run two jobs:
|
||||
|
||||
```bash
|
||||
# Pixel retrieval
|
||||
python run_bench.py \
|
||||
--task simpleqa --model <MODEL_NAME> \
|
||||
--local-api --local-api-url http://localhost:30888/search \
|
||||
--retrieval-top-k 3 --num-examples 1000 --no-think
|
||||
|
||||
# Text retrieval
|
||||
python run_bench.py \
|
||||
--task simpleqa --model <MODEL_NAME> \
|
||||
--text-api --text-api-url http://localhost:30889/search \
|
||||
--retrieval-top-k 3 --num-examples 1000 --no-think
|
||||
```
|
||||
|
||||
where `<MODEL_NAME>` is one of:
|
||||
- `liuhaotian/llava-v1.5-7b`
|
||||
- `meta-llama/Llama-3.2-11B-Vision-Instruct` (k=1 for pixel due to architecture limit)
|
||||
- `meta-llama/Llama-3.2-90B-Vision-Instruct` (k=1 for pixel)
|
||||
- `meta-llama/Llama-4-Scout-17B-16E-Instruct`
|
||||
- `meta-llama/Llama-4-Maverick-17B-128E-Instruct`
|
||||
- `Qwen/Qwen2-VL-2B-Instruct` through `Qwen/Qwen2-VL-72B-Instruct`
|
||||
- `Qwen/Qwen2.5-VL-3B-Instruct` through `Qwen/Qwen2.5-VL-72B-Instruct`
|
||||
- `Qwen/Qwen3-VL-2B` through `Qwen/Qwen3-VL-235B-A22B`
|
||||
- `Qwen/Qwen3.5-0.8B` through `Qwen/Qwen3.5-35B-A3B`
|
||||
- `Qwen/Qwen3.6-27B`, `Qwen/Qwen3.6-35B-A3B`
|
||||
|
||||
For reasoning-mode models, omit `--no-think`.
|
||||
|
||||
Each model requires its own vLLM instance (or OpenRouter/Commonstack for API models).
|
||||
|
||||
---
|
||||
|
||||
## LiveVQA (Table 1 + Table 3)
|
||||
|
||||
LiveVQA uses `eval/run_livevqa.py` — a dedicated script for the news corpus.
|
||||
|
||||
**Requires**: News pixel search API (port 30890), news text search API (port 30892),
|
||||
LiveVQA v4 JSON dataset, vLLM reader.
|
||||
|
||||
```bash
|
||||
cd ~/pixelrag/eval
|
||||
|
||||
# No retrieval
|
||||
python run_livevqa.py --mode naive \
|
||||
--model Qwen/Qwen3.5-4B-Instruct \
|
||||
--output eval_output/livevqa_naive.jsonl
|
||||
|
||||
# PixelRAG (screenshot → screenshot)
|
||||
python run_livevqa.py --mode pixel \
|
||||
--pixel-api http://localhost:30890/search \
|
||||
--model Qwen/Qwen3.5-4B-Instruct \
|
||||
--output eval_output/livevqa_pixel.jsonl
|
||||
|
||||
# Text retrieval (Trafilatura)
|
||||
python run_livevqa.py --mode text \
|
||||
--text-api http://localhost:30892/search \
|
||||
--model Qwen/Qwen3.5-4B-Instruct \
|
||||
--output eval_output/livevqa_text.jsonl
|
||||
|
||||
# Hybrid (pixel + text)
|
||||
python run_livevqa.py --mode hybrid \
|
||||
--pixel-api http://localhost:30890/search \
|
||||
--text-api http://localhost:30892/search \
|
||||
--model Qwen/Qwen3.5-4B-Instruct \
|
||||
--output eval_output/livevqa_hybrid.jsonl
|
||||
```
|
||||
|
||||
Grading is automatic (5-option MC exact letter match) — printed at the end of each run.
|
||||
|
||||
---
|
||||
|
||||
## Known Issues (Blockers for Reproduction)
|
||||
|
||||
### ~~0. Missing simpleqa modules~~ (FIXED)
|
||||
|
||||
`screenshot.py` and `pixel_query.py` have been copied into `eval/lib/`.
|
||||
Selenium import is deferred so it doesn't block `--local-api` users.
|
||||
|
||||
### ~~1. `dr_agent` not importable~~ (FIXED)
|
||||
|
||||
Dataset loaders extracted into `eval/lib/benchmarks.py`. The `run_bench.py`
|
||||
import now reads from `simpleqa.datasets_loader` instead of `dr_agent`.
|
||||
|
||||
### ~~2. Grading script not in this repo~~ (FIXED)
|
||||
|
||||
`eval/grade.py` implements GPT-4.1 3-way grading (CORRECT/INCORRECT/NOT_ATTEMPTED) using
|
||||
the same prompt template as the paper. No dependency on the old repo's evaluation framework.
|
||||
|
||||
For the legacy full evaluation framework (per-example HTML reports, etc.), the original
|
||||
is still at `~/pixelrag-src/Vis-RAG/agent/scripts/evaluate.py`.
|
||||
|
||||
### 3. Hardcoded paths in retrieval.py
|
||||
|
||||
`eval/lib/retrieval.py` lines 84–88 have placeholder paths (`/path/to/project`, `/path/to/data`) for the local kiwix tile store. These are only used by `LocalWikiTiledScreenshotRetriever` (ground-truth screenshot mode), not by the production `--local-api` mode.
|
||||
|
||||
### ~~4. LiveVQA uses separate pipeline~~ (FIXED)
|
||||
|
||||
`eval/run_livevqa.py` handles all LiveVQA modes (naive, pixel, text, hybrid).
|
||||
|
||||
### ~~5. MoNaCo runs from old repo~~ (FIXED)
|
||||
|
||||
`eval/run_monaco.py` implements the full ReAct agent loop with pixel/text retrieval backends.
|
||||
|
||||
### 6. mwparserfromhell text index
|
||||
|
||||
The paper's second text baseline uses mwparserfromhell parser. The text index must be built separately with this parser — the parser choice is embedded at index build time, not at query time. The build pipeline for this variant needs to be documented.
|
||||
|
||||
### 7. News corpus indexes
|
||||
|
||||
LiveVQA requires separate tile and text indexes built over the news corpus (BBC/AP/CNN). These indexes are on a different machine/path and need their own `pixelrag-serve` instances.
|
||||
|
||||
---
|
||||
|
||||
## Grading Protocol Summary
|
||||
|
||||
| Benchmark | Metric | Grader |
|
||||
|-----------|--------|--------|
|
||||
| SimpleQA | CORRECT/INCORRECT/NOT_ATTEMPTED → accuracy | GPT-4.1 (temp=0, seed=42) |
|
||||
| NQ | Same 3-way judge | GPT-4.1 (temp=0, seed=42) |
|
||||
| NQ-Tables | Same 3-way judge (up to 10 gold aliases joined with OR) | GPT-4.1 |
|
||||
| MMSearch | Same 3-way judge | GPT-4.1 |
|
||||
| EVQA | Same 3-way judge (reference_list → "Any of: ref1 \| ref2") | GPT-4.1 |
|
||||
| LiveVQA | 5-option multiple-choice exact letter match | No LLM |
|
||||
| MoNaCo | Token-level F1 (primary), LLM judge F1 (secondary) | GPT-4.1 |
|
||||
|
||||
---
|
||||
|
||||
## Quick Smoke Test (Verify Pipeline Works)
|
||||
|
||||
Run a single example end-to-end before committing to full runs:
|
||||
|
||||
```bash
|
||||
# 1. Verify search API is responding
|
||||
curl -s http://localhost:30888/status | python -m json.tool
|
||||
|
||||
# 2. Run 5 examples, no retrieval
|
||||
python run_bench.py \
|
||||
--task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
|
||||
--num-examples 5 --no-think --force
|
||||
|
||||
# 3. Run 5 examples, pixel retrieval
|
||||
python run_bench.py \
|
||||
--task simpleqa --model Qwen/Qwen3.5-4B-Instruct \
|
||||
--local-api --local-api-url http://localhost:30888/search \
|
||||
--retrieval-top-k 3 --num-examples 5 --no-think --force
|
||||
|
||||
# 4. Grade
|
||||
cd ~/pixelrag-src/Vis-RAG/agent
|
||||
python scripts/evaluate.py simpleqa ~/pixelrag/eval/eval_output/<output>.jsonl
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Output File Convention
|
||||
|
||||
All outputs go to `eval_output/` with auto-generated filenames:
|
||||
|
||||
```
|
||||
eval_output/{task}_{mode}_{model_safe}_{n}.jsonl
|
||||
```
|
||||
|
||||
Examples:
|
||||
- `eval_output/simpleqa_naive_qwen_qwen3.5_4b_instruct_1000.jsonl`
|
||||
- `eval_output/simpleqa_local_api_qwen_qwen3.5_4b_instruct_1000.jsonl`
|
||||
- `eval_output/nq_text_api_qwen_qwen3.5_4b_instruct_1000.jsonl`
|
||||
|
||||
Grading results are saved alongside as `*_eval_results.json`.
|
||||
@@ -141,6 +141,14 @@ sub-frames). None achieved 100% correct at 48 workers.
|
||||
|
||||
## Reproducing
|
||||
|
||||
**Measure with the bench harness, not a hand-rolled loop.** `tiles_per_s` here is
|
||||
**capture-only**: the bench's timer starts *after* `strategy.setup()` brings the Chrome
|
||||
workers up. At 48 workers, that startup is ~49s of serial Chrome launches — it is setup
|
||||
cost, not throughput, so it must not be counted. A naive end-to-end loop that includes the
|
||||
48-worker startup reports ~13 t/s (the startup tax), which is not the capture rate.
|
||||
Re-measured on the reference box (EPYC 7763, 128c) with the harness below: **130 t/s
|
||||
capture-only** (200 maxi-ZIM pages, 48 workers, `fmt="raw"`).
|
||||
|
||||
```python
|
||||
from pixelrag_render.strategies.cdp_phased import CDPPhasedStrategy
|
||||
from pixelrag_render.bench import Bench
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -1,438 +0,0 @@
|
||||
# Chromium Build on Centralia Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Build a patched Chromium (v150.0.7844.0) on Centralia (SSH: `CentraliaB200`, user `yichuan_wang`) with our two custom CDP features: `rawFilePath` parameter for `Page.captureScreenshot` and `directClip` parameter for parallel tile capture.
|
||||
|
||||
**Architecture:** All work happens under `/work/yichuan_wang/chromium-build/` on Centralia (NFS-mounted /work has 12TB free). depot_tools is cloned alongside the chromium checkout. The patch is generated locally from `~/chromium/src` (HEAD~1 diff) and transferred via scp. Build uses a release/official/no-debug/no-PGO args.gn for a fast, deployable binary.
|
||||
|
||||
**Tech Stack:** Chromium source (~30GB no-history), depot_tools, gn, autoninja (ninja), Python 3.12 (already on Centralia), Ubuntu 24.04, 224 cores for parallel build.
|
||||
|
||||
---
|
||||
|
||||
## Environment Facts (verified pre-plan)
|
||||
|
||||
- **Centralia SSH alias:** `CentraliaB200` (user `yichuan_wang`)
|
||||
- **Workspace:** `/work/yichuan_wang/chromium-build/` (NFS, ~12TB free — plenty of room)
|
||||
- **Local disk on Centralia:** `/dev/md0` 209GB free — avoid storing large files there
|
||||
- **Local patch source:** `~/chromium/src` on local machine, `HEAD~1` diff covers 6 files, 166 insertions
|
||||
- **Chromium version:** 150.0.7844.0 (MAJOR=150, BUILD=7844)
|
||||
- **OS on Centralia:** Ubuntu 24.04.4 LTS (Noble)
|
||||
- **Python:** `/usr/bin/python3` (3.12.3) — already present, no install needed
|
||||
- **ninja/autoninja:** NOT present on Centralia — comes from depot_tools, added to PATH
|
||||
|
||||
---
|
||||
|
||||
## File Map
|
||||
|
||||
| Location | Purpose |
|
||||
|---|---|
|
||||
| `/work/yichuan_wang/chromium-build/depot_tools/` | Google's build tools (gn, fetch, autoninja, gclient) |
|
||||
| `/work/yichuan_wang/chromium-build/chromium/` | gclient checkout root (contains `.gclient`) |
|
||||
| `/work/yichuan_wang/chromium-build/chromium/src/` | Chromium source tree |
|
||||
| `/work/yichuan_wang/chromium-build/chromium/src/out/Release/` | Build output dir |
|
||||
| `/work/yichuan_wang/chromium-build/chromium/src/out/Release/args.gn` | Build configuration |
|
||||
| `/work/yichuan_wang/chromium-build/chromium_patches.diff` | Our custom patch (transferred from local) |
|
||||
| `/work/yichuan_wang/chromium-build/build.log` | autoninja build log (stream with `tail -f`) |
|
||||
|
||||
---
|
||||
|
||||
## Task 1: Verify workspace and install depot_tools
|
||||
|
||||
**Files:**
|
||||
- Create: `/work/yichuan_wang/chromium-build/` (directory)
|
||||
- Create: `/work/yichuan_wang/chromium-build/depot_tools/` (git clone)
|
||||
|
||||
- [ ] **Step 1.1: Create workspace directory on Centralia**
|
||||
|
||||
```bash
|
||||
ssh CentraliaB200 "mkdir -p /work/yichuan_wang/chromium-build && echo 'workspace ready'"
|
||||
```
|
||||
|
||||
Expected output: `workspace ready`
|
||||
|
||||
- [ ] **Step 1.2: Clone depot_tools into workspace**
|
||||
|
||||
Note: the correct URL is `chromium/tools/depot_tools` (not `chromium/depot_tools`).
|
||||
|
||||
```bash
|
||||
ssh CentraliaB200 "git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git /work/yichuan_wang/chromium-build/depot_tools"
|
||||
```
|
||||
|
||||
Expected: clone completes, last line something like `Resolving deltas: 100%`.
|
||||
|
||||
- [ ] **Step 1.3: Verify depot_tools tools exist**
|
||||
|
||||
```bash
|
||||
ssh CentraliaB200 "ls /work/yichuan_wang/chromium-build/depot_tools/fetch /work/yichuan_wang/chromium-build/depot_tools/gclient /work/yichuan_wang/chromium-build/depot_tools/autoninja"
|
||||
```
|
||||
|
||||
Expected: three file paths printed (no errors).
|
||||
|
||||
---
|
||||
|
||||
## Task 2: Fetch Chromium source (no history)
|
||||
|
||||
This is the longest step — `fetch --no-history chromium` downloads ~30GB and runs `gclient sync`. With a fast connection it takes 30–90 minutes.
|
||||
|
||||
**Files:**
|
||||
- Create: `/work/yichuan_wang/chromium-build/chromium/` (gclient root)
|
||||
- Create: `/work/yichuan_wang/chromium-build/chromium/src/` (source tree, ~30GB)
|
||||
|
||||
- [ ] **Step 2.1: Create the chromium checkout directory**
|
||||
|
||||
```bash
|
||||
ssh CentraliaB200 "mkdir -p /work/yichuan_wang/chromium-build/chromium"
|
||||
```
|
||||
|
||||
- [ ] **Step 2.2: Start the fetch in a detached screen session**
|
||||
|
||||
`fetch` must run from the chromium checkout root dir. We use `screen` so SSH disconnection doesn't kill it. The output is redirected to a log file for monitoring.
|
||||
|
||||
IMPORTANT: The home dir (`/home/eecs/yichuan_wang`) is full (10GB NFS, 0 bytes free). Set XDG dirs to /work to prevent depot_tools from failing on `~/.config/depot_tools`. Also put both `depot_tools/.cipd_bin` and `depot_tools` in PATH — vpython3 needs cipd in PATH.
|
||||
|
||||
```bash
|
||||
ssh CentraliaB200 "screen -dmS chromium_fetch bash -c '
|
||||
export XDG_CONFIG_HOME=/work/yichuan_wang/chromium-build/xdg/config
|
||||
export XDG_CACHE_HOME=/work/yichuan_wang/chromium-build/xdg/cache
|
||||
export XDG_DATA_HOME=/work/yichuan_wang/chromium-build/xdg/data
|
||||
export XDG_STATE_HOME=/work/yichuan_wang/chromium-build/xdg/state
|
||||
export PATH=/work/yichuan_wang/chromium-build/depot_tools/.cipd_bin:/work/yichuan_wang/chromium-build/depot_tools:\$PATH
|
||||
export DEPOT_TOOLS_DIR=/work/yichuan_wang/chromium-build/depot_tools
|
||||
cd /work/yichuan_wang/chromium-build/chromium
|
||||
echo \"FETCH_START \$(date)\" > /work/yichuan_wang/chromium-build/fetch.log
|
||||
fetch --no-history chromium >> /work/yichuan_wang/chromium-build/fetch.log 2>&1
|
||||
echo \"FETCH_DONE exit=\$? at \$(date)\" >> /work/yichuan_wang/chromium-build/fetch.log
|
||||
'"
|
||||
```
|
||||
|
||||
- [ ] **Step 2.3: Verify screen session started**
|
||||
|
||||
```bash
|
||||
ssh CentraliaB200 "screen -ls | grep chromium_fetch"
|
||||
```
|
||||
|
||||
Expected: a line like `12345.chromium_fetch (Detached)`.
|
||||
|
||||
- [ ] **Step 2.4: Monitor fetch progress (check periodically)**
|
||||
|
||||
```bash
|
||||
ssh CentraliaB200 "tail -20 /work/yichuan_wang/chromium-build/fetch.log"
|
||||
```
|
||||
|
||||
Re-run this command to watch progress. Fetch is done when the log contains `FETCH_DONE exit=0`.
|
||||
|
||||
- [ ] **Step 2.5: Verify source tree exists after fetch completes**
|
||||
|
||||
```bash
|
||||
ssh CentraliaB200 "ls /work/yichuan_wang/chromium-build/chromium/src/chrome/VERSION"
|
||||
```
|
||||
|
||||
Expected: file path printed (no error). If the file doesn't exist, fetch failed — check `fetch.log` for errors.
|
||||
|
||||
- [ ] **Step 2.6: Verify Chromium version matches local**
|
||||
|
||||
```bash
|
||||
ssh CentraliaB200 "cat /work/yichuan_wang/chromium-build/chromium/src/chrome/VERSION"
|
||||
```
|
||||
|
||||
Expected:
|
||||
```
|
||||
MAJOR=150
|
||||
MINOR=0
|
||||
BUILD=7844
|
||||
PATCH=0
|
||||
```
|
||||
|
||||
If the version differs, the patch may not apply cleanly. Record the actual version for the patch step.
|
||||
|
||||
---
|
||||
|
||||
## Task 3: Transfer and apply our patches
|
||||
|
||||
Our patch adds two CDP features to `Page.captureScreenshot`:
|
||||
- `rawFilePath`: write screenshot directly to a file path (bypassing base64 encoding)
|
||||
- `directClip`: clip parameter for parallel tile capture
|
||||
|
||||
**Files:**
|
||||
- Modify: `content/browser/devtools/protocol/page_handler.cc` (primary patch target)
|
||||
- Modify: `content/browser/devtools/protocol/page_handler.h`
|
||||
- Modify: `content/renderer/render_widget_host/render_widget_host_impl.cc`
|
||||
- Modify: `content/renderer/render_widget_host/render_widget_host_impl.h`
|
||||
- Modify: `third_party/blink/public/devtools_protocol/domains/Page.pdl`
|
||||
- Modify: `third_party/blink/renderer/platform/widget/widget_base.cc`
|
||||
- Transfer: `/work/yichuan_wang/chromium-build/chromium_patches.diff`
|
||||
|
||||
- [ ] **Step 3.1: Generate the patch from local machine**
|
||||
|
||||
Run this on the LOCAL machine:
|
||||
|
||||
```bash
|
||||
git -C ~/chromium/src diff HEAD~1 > /tmp/chromium_patches.diff
|
||||
wc -l /tmp/chromium_patches.diff
|
||||
```
|
||||
|
||||
Expected: file is non-empty (~300+ lines).
|
||||
|
||||
- [ ] **Step 3.2: Transfer patch to Centralia**
|
||||
|
||||
Run this on the LOCAL machine:
|
||||
|
||||
```bash
|
||||
scp /tmp/chromium_patches.diff CentraliaB200:/work/yichuan_wang/chromium-build/chromium_patches.diff
|
||||
```
|
||||
|
||||
- [ ] **Step 3.3: Verify patch arrived on Centralia**
|
||||
|
||||
```bash
|
||||
ssh CentraliaB200 "wc -l /work/yichuan_wang/chromium-build/chromium_patches.diff"
|
||||
```
|
||||
|
||||
Expected: same line count as local.
|
||||
|
||||
- [ ] **Step 3.4: Apply the patch**
|
||||
|
||||
```bash
|
||||
ssh CentraliaB200 "cd /work/yichuan_wang/chromium-build/chromium/src && git apply /work/yichuan_wang/chromium-build/chromium_patches.diff"
|
||||
```
|
||||
|
||||
Expected: no output (silent success). If you see errors like "patch does not apply", see Step 3.5.
|
||||
|
||||
- [ ] **Step 3.5: (If patch fails) Try with --3way or check fuzz**
|
||||
|
||||
If Step 3.4 fails with "patch does not apply":
|
||||
|
||||
```bash
|
||||
ssh CentraliaB200 "cd /work/yichuan_wang/chromium-build/chromium/src && git apply --3way /work/yichuan_wang/chromium-build/chromium_patches.diff"
|
||||
```
|
||||
|
||||
If that also fails, the Chromium version on Centralia differs from the local checkout. Check `cat /work/yichuan_wang/chromium-build/chromium/src/chrome/VERSION` and compare to local (`cat ~/chromium/src/chrome/VERSION`). If versions differ significantly, you may need to regenerate the patch from the correct base commit — fetch the local HEAD's commit hash with `git -C ~/chromium/src rev-parse HEAD~1` and use that.
|
||||
|
||||
- [ ] **Step 3.6: Verify patch was applied**
|
||||
|
||||
```bash
|
||||
ssh CentraliaB200 "cd /work/yichuan_wang/chromium-build/chromium/src && git diff --stat"
|
||||
```
|
||||
|
||||
Expected output:
|
||||
```
|
||||
content/browser/devtools/protocol/page_handler.cc | 129 +++++...
|
||||
content/browser/devtools/protocol/page_handler.h | 14 +++
|
||||
content/renderer/render_widget_host/render_widget_host_impl.cc | 9 ++
|
||||
content/renderer/render_widget_host/render_widget_host_impl.h | 5 +
|
||||
third_party/blink/public/devtools_protocol/domains/Page.pdl | 6 +
|
||||
third_party/blink/renderer/platform/widget/widget_base.cc | 12 +-
|
||||
6 files changed, 166 insertions(+), 9 deletions(-)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 4: Configure the build
|
||||
|
||||
**Files:**
|
||||
- Create: `/work/yichuan_wang/chromium-build/chromium/src/out/Release/` (directory)
|
||||
- Create: `/work/yichuan_wang/chromium-build/chromium/src/out/Release/args.gn`
|
||||
|
||||
- [ ] **Step 4.1: Create the build output directory**
|
||||
|
||||
```bash
|
||||
ssh CentraliaB200 "mkdir -p /work/yichuan_wang/chromium-build/chromium/src/out/Release"
|
||||
```
|
||||
|
||||
- [ ] **Step 4.2: Write args.gn**
|
||||
|
||||
```bash
|
||||
ssh CentraliaB200 "cat > /work/yichuan_wang/chromium-build/chromium/src/out/Release/args.gn << 'EOF'
|
||||
is_debug = false
|
||||
is_official_build = true
|
||||
is_component_build = false
|
||||
symbol_level = 0
|
||||
blink_symbol_level = 0
|
||||
chrome_pgo_phase = 0
|
||||
EOF"
|
||||
```
|
||||
|
||||
- [ ] **Step 4.3: Verify args.gn content**
|
||||
|
||||
```bash
|
||||
ssh CentraliaB200 "cat /work/yichuan_wang/chromium-build/chromium/src/out/Release/args.gn"
|
||||
```
|
||||
|
||||
Expected exact output:
|
||||
```
|
||||
is_debug = false
|
||||
is_official_build = true
|
||||
is_component_build = false
|
||||
symbol_level = 0
|
||||
blink_symbol_level = 0
|
||||
chrome_pgo_phase = 0
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 5: Run gn gen
|
||||
|
||||
`gn gen` reads `args.gn` and generates all the ninja build files. This takes 2–5 minutes on 224 cores.
|
||||
|
||||
**Files:**
|
||||
- Create: `/work/yichuan_wang/chromium-build/chromium/src/out/Release/build.ninja` (generated)
|
||||
- Create: `/work/yichuan_wang/chromium-build/gn_gen.log`
|
||||
|
||||
- [ ] **Step 5.1: Run gn gen in a screen session**
|
||||
|
||||
```bash
|
||||
ssh CentraliaB200 "screen -dmS chromium_gn bash -c '
|
||||
export PATH=/work/yichuan_wang/chromium-build/depot_tools:\$PATH
|
||||
cd /work/yichuan_wang/chromium-build/chromium/src
|
||||
gn gen out/Release > /work/yichuan_wang/chromium-build/gn_gen.log 2>&1
|
||||
echo \"GN_DONE exit=\$?\" >> /work/yichuan_wang/chromium-build/gn_gen.log
|
||||
'"
|
||||
```
|
||||
|
||||
- [ ] **Step 5.2: Wait for gn gen to finish**
|
||||
|
||||
```bash
|
||||
ssh CentraliaB200 "tail -5 /work/yichuan_wang/chromium-build/gn_gen.log"
|
||||
```
|
||||
|
||||
Re-run until you see `GN_DONE exit=0`. If exit is non-zero, check the full log:
|
||||
|
||||
```bash
|
||||
ssh CentraliaB200 "cat /work/yichuan_wang/chromium-build/gn_gen.log"
|
||||
```
|
||||
|
||||
Common gn errors and fixes:
|
||||
- `Python not found`: verify `which python3` works on Centralia (it does per our check)
|
||||
- `No targets match`: args.gn typo — re-check Step 4.2
|
||||
|
||||
- [ ] **Step 5.3: Verify build.ninja was generated**
|
||||
|
||||
```bash
|
||||
ssh CentraliaB200 "ls -lh /work/yichuan_wang/chromium-build/chromium/src/out/Release/build.ninja"
|
||||
```
|
||||
|
||||
Expected: file exists, non-zero size.
|
||||
|
||||
---
|
||||
|
||||
## Task 6: Build chrome with autoninja
|
||||
|
||||
This is the main build step. With 224 cores and no debug symbols, expect 60–120 minutes for a full build. The output is the `chrome` binary.
|
||||
|
||||
**Files:**
|
||||
- Create: `/work/yichuan_wang/chromium-build/chromium/src/out/Release/chrome` (built binary)
|
||||
- Create: `/work/yichuan_wang/chromium-build/build.log`
|
||||
|
||||
- [ ] **Step 6.1: Start autoninja build in screen session**
|
||||
|
||||
`autoninja` automatically sets `-j` based on CPU count (will use ~224 jobs). It reads the `NINJA_SUMMARIZE_BUILD` env var to show progress.
|
||||
|
||||
```bash
|
||||
ssh CentraliaB200 "screen -dmS chromium_build bash -c '
|
||||
export PATH=/work/yichuan_wang/chromium-build/depot_tools:\$PATH
|
||||
cd /work/yichuan_wang/chromium-build/chromium/src
|
||||
autoninja -C out/Release chrome > /work/yichuan_wang/chromium-build/build.log 2>&1
|
||||
echo \"BUILD_DONE exit=\$?\" >> /work/yichuan_wang/chromium-build/build.log
|
||||
'"
|
||||
```
|
||||
|
||||
- [ ] **Step 6.2: Verify screen session started**
|
||||
|
||||
```bash
|
||||
ssh CentraliaB200 "screen -ls | grep chromium_build"
|
||||
```
|
||||
|
||||
Expected: a line like `12345.chromium_build (Detached)`.
|
||||
|
||||
- [ ] **Step 6.3: Monitor build progress**
|
||||
|
||||
```bash
|
||||
ssh CentraliaB200 "tail -5 /work/yichuan_wang/chromium-build/build.log"
|
||||
```
|
||||
|
||||
You'll see ninja progress lines like `[1234/89000] CXX obj/content/...`. Re-run every few minutes to watch progress. Build is done when you see `BUILD_DONE exit=0`.
|
||||
|
||||
To watch CPU utilization:
|
||||
```bash
|
||||
ssh CentraliaB200 "uptime"
|
||||
```
|
||||
|
||||
If the build is running, load average should be ~200+.
|
||||
|
||||
- [ ] **Step 6.4: Check for build errors (if BUILD_DONE shows non-zero exit)**
|
||||
|
||||
```bash
|
||||
ssh CentraliaB200 "grep -i 'error:' /work/yichuan_wang/chromium-build/build.log | tail -20"
|
||||
```
|
||||
|
||||
Common errors:
|
||||
- `undefined reference`: usually means a `.h` change wasn't matched with a `.cc` change in the patch. Check that the patch applied fully (Task 3).
|
||||
- `ninja: build stopped`: check the lines above for the actual C++ error.
|
||||
- Disk full: run `df -h /work` — if /work is at 100%, free space or use a different path.
|
||||
|
||||
- [ ] **Step 6.5: Verify chrome binary exists and is executable**
|
||||
|
||||
```bash
|
||||
ssh CentraliaB200 "ls -lh /work/yichuan_wang/chromium-build/chromium/src/out/Release/chrome"
|
||||
```
|
||||
|
||||
Expected: file ~200–300MB, executable bit set (permissions like `-rwxr-xr-x`).
|
||||
|
||||
- [ ] **Step 6.6: Smoke-test the binary**
|
||||
|
||||
```bash
|
||||
ssh CentraliaB200 "/work/yichuan_wang/chromium-build/chromium/src/out/Release/chrome --version"
|
||||
```
|
||||
|
||||
Expected: `Chromium 150.0.7844.0` (or similar version line).
|
||||
|
||||
Note: Chrome may print warnings about display/GPU on a headless server — that's normal. We care only that the binary runs and prints its version.
|
||||
|
||||
---
|
||||
|
||||
## Task 7: Verify our custom CDP features are compiled in
|
||||
|
||||
Our patch adds `rawFilePath` and `directClip` parameters to `Page.captureScreenshot`. We verify they made it into the compiled protocol.
|
||||
|
||||
**Files:**
|
||||
- Read: `/work/yichuan_wang/chromium-build/chromium/src/out/Release/gen/third_party/blink/public/devtools_protocol/protocol/page.json` (generated protocol JSON)
|
||||
|
||||
- [ ] **Step 7.1: Check the generated protocol JSON for our parameters**
|
||||
|
||||
```bash
|
||||
ssh CentraliaB200 "grep -n 'rawFilePath\|directClip' /work/yichuan_wang/chromium-build/chromium/src/out/Release/gen/third_party/blink/public/devtools_protocol/protocol/page.json"
|
||||
```
|
||||
|
||||
Expected: at least 2 lines mentioning `rawFilePath` and `directClip`.
|
||||
|
||||
- [ ] **Step 7.2: Check the compiled binary for our parameter strings**
|
||||
|
||||
```bash
|
||||
ssh CentraliaB200 "strings /work/yichuan_wang/chromium-build/chromium/src/out/Release/chrome | grep -c 'rawFilePath'"
|
||||
```
|
||||
|
||||
Expected: at least 1 (the string is embedded in the binary). If 0, the patch didn't compile into the binary — recheck that `git diff --stat` in Task 3 Step 6 was correct.
|
||||
|
||||
---
|
||||
|
||||
## Timing Estimates
|
||||
|
||||
| Task | Estimated Duration |
|
||||
|---|---|
|
||||
| Task 1: depot_tools clone | 2–5 min |
|
||||
| Task 2: `fetch --no-history chromium` | 30–90 min (network-dependent) |
|
||||
| Task 3: patch transfer + apply | 2 min |
|
||||
| Task 4: args.gn setup | 1 min |
|
||||
| Task 5: `gn gen` | 3–7 min |
|
||||
| Task 6: `autoninja` build | 60–120 min (224 cores, no debug) |
|
||||
| Task 7: verification | 2 min |
|
||||
| **Total** | **~2–4 hours** |
|
||||
|
||||
---
|
||||
|
||||
## Recovery Notes
|
||||
|
||||
- **If screen session dies unexpectedly:** Re-attach with `screen -r chromium_build` to see any final error, then restart from the last completed step.
|
||||
- **If fetch is interrupted:** Re-run `fetch --no-history chromium` from the same directory — gclient will resume.
|
||||
- **If build is interrupted:** Re-run `autoninja -C out/Release chrome` — ninja tracks completed targets and resumes from where it left off.
|
||||
- **If /work fills up:** `du -sh /work/yichuan_wang/chromium-build/chromium/src/out/Release/obj/` is usually the largest dir. Consider deleting `.o` files after build if you only need the final binary: `find out/Release/obj -name '*.o' -delete`.
|
||||
@@ -1,359 +0,0 @@
|
||||
# PixelRAG Unirepo Restructure — Design Spec
|
||||
|
||||
**Date:** 2026-05-11
|
||||
**Status:** Draft
|
||||
|
||||
## Context
|
||||
|
||||
PixelRAG is a visual document retrieval framework: any document (web page, PDF, image) → visual rendering → embedding → FAISS index → search API. Three private repos (yichuan-w/Vis-RAG, andylizf/wiki-screenshot, andylizf/wiki-screenshot-training) are being merged into a single public repo with a clean architecture.
|
||||
|
||||
The current repo at ~/pixelrag/ has a messy first-pass merge. This spec defines the target architecture.
|
||||
|
||||
## Users
|
||||
|
||||
**A — Framework user (primary):** "I have documents (web pages, PDFs, local files) and want to build a visual retrieval system." Needs the full pipeline. Cares about generality — not a Wikipedia-specific tool.
|
||||
|
||||
**B — Paper reproducer:** "I want to reproduce PixelRAG results." Downloads pre-built indexes, starts the search API, runs eval. Should exist but not the design focus.
|
||||
|
||||
**C — Agent developer:** Uses screenshot capture and visual search as agent skills/tools. Needs callable APIs: give URL → get screenshot, give query → get results. Will demo agent integration.
|
||||
|
||||
**D — Model trainer:** Trains visual embedding models. Contrastive learning + hard negative mining via search API. Should exist but not the design focus.
|
||||
|
||||
## Package Architecture
|
||||
|
||||
Five packages, single-direction dependencies:
|
||||
|
||||
```
|
||||
ingest ←── index ──→ embed
|
||||
|
||||
serve (independent)
|
||||
|
||||
train → serve (API calls for mining)
|
||||
```
|
||||
|
||||
### Package 1: pixelrag-render
|
||||
|
||||
**"Document → image tiles."** Standalone rendering tool. Agents call it directly; index calls it for batch jobs.
|
||||
|
||||
```
|
||||
src/pixelrag_render/
|
||||
├── render.py # Public API:
|
||||
│ # render_url(url, output_dir, backend="cdp") → list[Path]
|
||||
│ # render_pdf(path, output_dir) → list[Path]
|
||||
│ # render_file(path, output_dir) → list[Path] (auto-detect)
|
||||
├── backends/
|
||||
│ ├── cdp.py # Lean CDP capture — default, fastest
|
||||
│ │ # Direct Page.captureScreenshot, multi-browser workers
|
||||
│ │ # JPEG q85, DPR 1, fromSurface=False, optimizeForSpeed=True
|
||||
│ │ # Based on render_news_pages.py (23.9s/50 articles benchmark)
|
||||
│ ├── playwright.py # Full Playwright — more options, experimental/compat
|
||||
│ │ # Stripped to production-useful config only
|
||||
│ │ # Keeps: CDP screenshot mode, segmented tiles, GPU rasterization
|
||||
│ │ # Removes: unused experimental options
|
||||
│ └── pdf.py # PDF → page images (pdf2image or PyMuPDF)
|
||||
└── bench/ # Rendering benchmarks
|
||||
├── benchmark.py # Config sweep (workers, batch size, concurrency)
|
||||
├── benchmark_optimizations.py # GPU accel, PNG compression, tile sizes
|
||||
├── benchmark_fullpage.py # Screenshot strategy comparison
|
||||
└── benchmark_longtail_matrix.py # Long pages × tile size × concurrency
|
||||
```
|
||||
|
||||
**Dependencies:** playwright, pillow, aiohttp (lightweight — no torch)
|
||||
|
||||
**CLI entry points:**
|
||||
- `pixelrag-render` → `pixelrag_render.render:main` (render URLs/files to tiles)
|
||||
|
||||
**Source of code:**
|
||||
- `cdp.py` ← `scripts/render_news_pages.py` capture_article + worker + multi-browser setup (generalized, news-specific parts removed)
|
||||
- `playwright.py` ← `tools/playwright_tool.py` (stripped from 2388L to production-relevant config)
|
||||
- `bench/` ← `bench/` directory (kept as-is)
|
||||
- `render.py` — new, thin API layer that dispatches to backends
|
||||
|
||||
### Package 2: pixelrag-embed
|
||||
|
||||
**"Image tiles → vectors → FAISS index."** Three independent CLI tools, orchestrator-free. Each has its own `main()`, no imports between them.
|
||||
|
||||
```
|
||||
src/pixelrag_embed/
|
||||
├── chunk.py # Large image → 1024px strips
|
||||
│ # Input: tile directory. Output: chunk PNGs + chunks.json
|
||||
│ # Pure PIL, no torch. ~380 lines.
|
||||
├── embed.py # Images → embedding vectors
|
||||
│ # Input: chunk directory. Output: shard_NNN.npz
|
||||
│ # vLLM/sglang backend, multi-GPU. ~2400 lines.
|
||||
└── index.py # Vectors → FAISS IVFFlat index
|
||||
# Input: embedding .npz shards. Output: index.faiss + metadata.npz
|
||||
# ~330 lines.
|
||||
```
|
||||
|
||||
**Dependencies:** torch, transformers, faiss-cpu, pillow, numpy, tqdm
|
||||
|
||||
**CLI entry points:**
|
||||
- `pixelrag-chunk` → `pixelrag_embed.chunk:main`
|
||||
- `pixelrag-embed` → `pixelrag_embed.embed:main`
|
||||
- `pixelrag-build-index` → `pixelrag_embed.index:main`
|
||||
|
||||
**Source of code:**
|
||||
- `chunk.py` ← `embedding/chunk_tiles.py`
|
||||
- `embed.py` ← `embedding/embed_tiles.py`
|
||||
- `index.py` ← `indexing/build_index.py`
|
||||
|
||||
### Package 3: pixelrag-index
|
||||
|
||||
**"Data source → complete searchable index."** Orchestration layer. Knows how to chain ingest + embed for different data sources. Two modes: single-machine (default, no S3) and distributed (S3 coordination for multi-machine).
|
||||
|
||||
```
|
||||
src/pixelrag_index/
|
||||
├── config.py # pixelrag.yaml parser
|
||||
│ # Defines: source type, paths, embed model, output location
|
||||
├── sources/ # Data source iterators (yield items for ingest to render)
|
||||
│ ├── kiwix.py # Wikipedia ZIM → iterate articles → call ingest per article
|
||||
│ ├── web.py # URL list/sitemap → download HTML+assets → call ingest
|
||||
│ │ # Download + SQLite state are internal to this source
|
||||
│ │ # Includes presets (e.g. "news") with per-domain rate limits,
|
||||
│ │ # cookie banner CSS, source-specific HTML handling (BBC/CNN/AP)
|
||||
│ │ # Usage: --source web --preset news
|
||||
│ ├── pdf.py # PDF directory → iterate files → call ingest per file
|
||||
│ └── local.py # Scan directory → auto-detect file types → route to above
|
||||
├── pipelines.py # End-to-end: source → ingest → chunk → embed → build
|
||||
│ # Chains the stages, handles checkpointing between stages
|
||||
├── distributed.py # S3ShardCoordinator + claim-loop worker (optional)
|
||||
│ # Only used with --distributed flag
|
||||
│ # Used by both capture and embedding distributed runs
|
||||
└── monitor.py # Cross-machine progress dashboard (reads S3 claims)
|
||||
│ # Only relevant in distributed mode
|
||||
```
|
||||
|
||||
**Two orchestration modes:**
|
||||
- `pixelrag-index build --source ./my_docs` — single machine, iterate locally, no S3
|
||||
- `pixelrag-index build --source kiwix --distributed --bucket my-bucket` — multi-machine, S3 coordination
|
||||
|
||||
**Dependencies:** pixelrag-render, pixelrag-embed, boto3 (optional, only for distributed), tqdm
|
||||
|
||||
**CLI entry points:**
|
||||
- `pixelrag-index` → `pixelrag_index.pipelines:main` (build index from source)
|
||||
- `pixelrag-monitor` → `pixelrag_index.monitor:main` (progress dashboard)
|
||||
|
||||
**pixelrag.yaml — parameter forwarding pattern:**
|
||||
|
||||
Each section's parameters are forwarded directly to the corresponding package. Index only manages orchestration order, not parameter details.
|
||||
|
||||
```python
|
||||
# index/config.py — forwarding logic
|
||||
source_type = config["source"].pop("type")
|
||||
source = SOURCES[source_type](**config["source"]) # forward all source params
|
||||
# ingest params forwarded to render calls
|
||||
# embed params forwarded to chunk/embed/build calls
|
||||
```
|
||||
|
||||
```yaml
|
||||
# Example: local files (User A)
|
||||
source:
|
||||
type: local
|
||||
path: ./my_docs
|
||||
|
||||
ingest:
|
||||
backend: cdp
|
||||
quality: 85
|
||||
|
||||
embed:
|
||||
model: Qwen/Qwen3-VL-Embedding-2B
|
||||
device: cuda
|
||||
gpu_ids: [0, 1, 2, 3]
|
||||
batch_size: 128
|
||||
|
||||
output: ./my_index
|
||||
```
|
||||
|
||||
```yaml
|
||||
# Example: web URLs with news preset
|
||||
source:
|
||||
type: web
|
||||
urls: ./urls.txt
|
||||
preset: news
|
||||
concurrency: 200
|
||||
|
||||
ingest:
|
||||
backend: cdp
|
||||
|
||||
embed:
|
||||
model: Qwen/Qwen3-VL-Embedding-2B
|
||||
gpu_ids: [0, 1]
|
||||
|
||||
output: ./news_index
|
||||
```
|
||||
|
||||
```yaml
|
||||
# Example: PDF collection
|
||||
source:
|
||||
type: pdf
|
||||
path: ./papers/
|
||||
dpi: 300
|
||||
pages: "1-10"
|
||||
|
||||
embed:
|
||||
model: Qwen/Qwen3-VL-Embedding-2B
|
||||
device: cpu
|
||||
|
||||
output: ./paper_index
|
||||
```
|
||||
|
||||
```yaml
|
||||
# Example: Wikipedia (distributed)
|
||||
source:
|
||||
type: kiwix
|
||||
zim: ./wikipedia.zim
|
||||
serve_url: http://localhost:9454
|
||||
|
||||
distributed:
|
||||
bucket: my-bucket
|
||||
prefix: kiwix
|
||||
|
||||
embed:
|
||||
model: Qwen/Qwen3-VL-Embedding-2B
|
||||
gpu_ids: [0, 1, 2, 3, 4, 5, 6, 7]
|
||||
backend: sglang
|
||||
|
||||
output: s3://my-bucket/index
|
||||
```
|
||||
|
||||
**Source of code:**
|
||||
- `distributed.py` ← `coordinator.py` (S3ShardCoordinator) + claim loop from `coordinator_worker.py` and `embedding_worker.py`
|
||||
- `sources/kiwix.py` ← `datasources/kiwix.py` (article iteration logic)
|
||||
- `sources/web.py` ← `datasources/news.py` + `news/download.py` + `news/db.py` (generalized, news-specific naming removed; download + SQLite state are internal implementation details of this source)
|
||||
- `sources/local.py` — new
|
||||
- `sources/pdf.py` — new (thin, delegates rendering to ingest)
|
||||
- `pipelines.py` ← new, chains stages
|
||||
- `monitor.py` ← `scripts/monitor_global.py`
|
||||
- `config.py` — new
|
||||
|
||||
### Package 4: pixelrag-serve
|
||||
|
||||
**"FAISS index → search API."** One unified FastAPI server that serves any index.
|
||||
|
||||
```
|
||||
src/pixelrag_serve/
|
||||
└── api.py # Unified search API
|
||||
# POST /search — text/image/embedding queries → top-k results
|
||||
# GET /health, GET /status
|
||||
# Configurable via CLI args or env vars
|
||||
# Supports CPU and CUDA for query embedding
|
||||
```
|
||||
|
||||
**Dependencies:** fastapi, uvicorn, faiss-cpu, torch, transformers, pillow, numpy
|
||||
|
||||
**CLI entry points:**
|
||||
- `pixelrag-serve` → `pixelrag_serve.api:main`
|
||||
|
||||
**Source of code:**
|
||||
- `api.py` ← merge of `search_api.py` + `text_search_api.py` + `news_search_api.py` into one unified API. Hex ID mapping handled at index build time (in pixelrag-embed), not at serve time.
|
||||
|
||||
### Package 5: pixelrag-train
|
||||
|
||||
**"Train visual embedding models."**
|
||||
|
||||
```
|
||||
src/pixelrag_train/
|
||||
├── models/
|
||||
│ └── biqwen3.py # BiQwen3: Qwen3VLModel + last-token pooling + L2 norm
|
||||
├── contrastive.py # GradCache contrastive training with LoRA/DoRA
|
||||
└── mine.py # Hard negative mining (calls serve API)
|
||||
# Unified: image mining (:30888) + text mining (:30889)
|
||||
```
|
||||
|
||||
**Dependencies:** torch, transformers, peft, accelerate, wandb, faiss-cpu
|
||||
|
||||
**CLI entry points:**
|
||||
- `pixelrag-train` → `pixelrag_train.contrastive:main`
|
||||
- `pixelrag-mine` → `pixelrag_train.mine:main`
|
||||
|
||||
**Source of code:**
|
||||
- `biqwen3.py` ← `models/biqwen3.py` (unchanged)
|
||||
- `contrastive.py` ← `train_contrastors.py` (renamed)
|
||||
- `mine.py` ← merge of `mine_hard_negatives.py` + `mine_text_hard_negatives.py`
|
||||
|
||||
### eval/
|
||||
|
||||
Not a package. Script directory for paper reproduction (User B).
|
||||
|
||||
```
|
||||
eval/
|
||||
├── run_naive_simpleqa.py # Main eval runner
|
||||
└── simpleqa/ # Support library (data, llm, retrieval, etc.)
|
||||
```
|
||||
|
||||
Source: kept from current repo, unchanged.
|
||||
|
||||
## Dependency Graph
|
||||
|
||||
```
|
||||
pixelrag-index
|
||||
├── pixelrag-render (calls render_url/render_pdf for capture stage)
|
||||
├── pixelrag-embed (calls chunk/embed/index tools)
|
||||
└── boto3 (S3 coordination)
|
||||
|
||||
pixelrag-serve (independent — no deps on other pixelrag packages)
|
||||
|
||||
pixelrag-train
|
||||
└── calls pixelrag-serve API over HTTP (not a Python dependency)
|
||||
|
||||
pixelrag-render (independent)
|
||||
pixelrag-embed (independent)
|
||||
```
|
||||
|
||||
## Data Flow
|
||||
|
||||
```
|
||||
User A: "Build me a visual search index"
|
||||
|
||||
pixelrag-index build --source ./my_docs
|
||||
│
|
||||
├─ sources/local.py scans directory, classifies files
|
||||
│
|
||||
├─ For each document:
|
||||
│ pixelrag-render.render_url() or render_pdf()
|
||||
│ → tiles/{doc_id}.tiles/tile_0000.jpg, tile_0001.jpg, ...
|
||||
│
|
||||
├─ pixelrag-embed.chunk
|
||||
│ → chunks/{doc_id}.tiles/chunk_0000_00.png, ...
|
||||
│
|
||||
├─ pixelrag-embed.embed (GPU)
|
||||
│ → embeddings/shard_NNN.npz
|
||||
│
|
||||
└─ pixelrag-embed.index
|
||||
→ output/index.faiss + metadata.npz
|
||||
|
||||
pixelrag-serve --index-dir ./output --port 30888
|
||||
→ POST /search {"queries": [{"text": "..."}]} → top-k results
|
||||
```
|
||||
|
||||
## What Gets Cut
|
||||
|
||||
From the current ~/pixelrag/ repo:
|
||||
- `packages/capture/` → replaced by `packages/render/` (new structure)
|
||||
- `packages/serving/` → replaced by `packages/serve/` (unified API)
|
||||
- `packages/training/` → replaced by `packages/train/` (renamed files)
|
||||
- `packages/embed/` — new package (from loose scripts)
|
||||
- `packages/index/` — new package (from loose scripts + new code)
|
||||
- `eval/` — kept
|
||||
|
||||
From source repos (~/pixelrag-src/), code NOT carried forward:
|
||||
- `executors/base.py`, `executors/skypilot.py` — executor ABC and cloud-specific code
|
||||
- `proxy/` — proxy rotation (not needed for offline rendering)
|
||||
- `lead_images/` — lead image extraction (hardcoded paths)
|
||||
- `datasources/enterprise.py`, `datasources/wikimedia.py` — paid API / superseded
|
||||
- `tools/streaming_capture.py` — superseded by lean CDP backend
|
||||
- `tools/raw_pixels.py`, `tools/temp_dirs.py` — helpers for old PlaywrightTool
|
||||
- Most of PlaywrightTool's 2388 lines — stripped to production config
|
||||
- `run.py`, `monitor.py` (top-level) — replaced by index CLI
|
||||
- `scripts/run_embeddings.py` — thin wrapper, redundant with embed CLI
|
||||
- `scripts/status.py` — replaced by monitor
|
||||
|
||||
## Migration Strategy
|
||||
|
||||
1. Create new package directories under ~/pixelrag/packages/
|
||||
2. Copy + transform code from ~/pixelrag-src/ (source repos are read-only)
|
||||
3. For each package: create pyproject.toml, rename imports, add CLI entry points
|
||||
4. Verify `uv sync --package <name>` works for each
|
||||
5. Verify existing endpoint (port 30001) still works with new pixelrag-serve
|
||||
6. Commit as clean restructure
|
||||
@@ -1,290 +0,0 @@
|
||||
# PixelRAG Frontend Design Spec
|
||||
|
||||
## Overview
|
||||
|
||||
A modern web frontend for the PixelRAG visual retrieval engine, serving as both an academic paper companion demo and a functional API service. Built as a standalone Next.js application alongside the existing FastAPI backend.
|
||||
|
||||
## Goals
|
||||
|
||||
- Showcase visual retrieval quality with rich tile image display
|
||||
- Provide interactive search (text + image queries) over the FAISS index
|
||||
- Document the API with live try-it-out capability
|
||||
- Look professional enough for paper/conference demos — not generic
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- User authentication or multi-tenancy
|
||||
- Index management or data ingestion UI
|
||||
- Mobile-first design (desktop-first, responsive is fine)
|
||||
|
||||
## Architecture
|
||||
|
||||
### Stack
|
||||
|
||||
| Layer | Technology |
|
||||
|-------|-----------|
|
||||
| Framework | Next.js 15 (App Router) |
|
||||
| Styling | Tailwind CSS 4 |
|
||||
| Components | shadcn/ui |
|
||||
| Animation | Framer Motion |
|
||||
| Language | TypeScript |
|
||||
| Backend | FastAPI (existing, unchanged except CORS) |
|
||||
|
||||
### Project Structure
|
||||
|
||||
```
|
||||
web/ ← new Next.js app
|
||||
src/
|
||||
app/
|
||||
page.tsx ← search home
|
||||
docs/page.tsx ← API reference
|
||||
status/page.tsx ← index dashboard
|
||||
layout.tsx ← shell (nav, theme provider)
|
||||
components/
|
||||
SearchBar.tsx ← text input + image upload/drag-drop
|
||||
ResultGroup.tsx ← article group with horizontal tile row
|
||||
TileCard.tsx ← single tile result card
|
||||
Lightbox.tsx ← fullscreen tile viewer with pan/zoom
|
||||
ComparePanel.tsx ← side-by-side tile comparison
|
||||
ApiPlayground.tsx ← try-it-live widget for /docs
|
||||
StatusCard.tsx ← metric card for /status dashboard
|
||||
lib/
|
||||
api.ts ← typed fetch wrapper for all API endpoints
|
||||
types.ts ← shared TypeScript types matching Pydantic models
|
||||
next.config.ts ← rewrites /api/* → FastAPI
|
||||
tailwind.config.ts
|
||||
package.json
|
||||
|
||||
serve/ ← existing (minimal changes)
|
||||
src/pixelrag_serve/api.py ← add CORSMiddleware
|
||||
```
|
||||
|
||||
### API Proxy
|
||||
|
||||
In development, `next.config.ts` rewrites `/api/*` to `http://localhost:30001/*` so the frontend can call the FastAPI backend without CORS issues. In production, CORS middleware on FastAPI allows the Next.js origin.
|
||||
|
||||
## Visual Design
|
||||
|
||||
### Color Palette
|
||||
|
||||
| Role | Value | Usage |
|
||||
|------|-------|-------|
|
||||
| Background | `#0c0c0c` | Page background |
|
||||
| Surface | `#1a1a1a` | Cards, inputs, panels |
|
||||
| Border | `#222222` | Card borders, dividers |
|
||||
| Text primary | `#ffffff` | Headings, important text |
|
||||
| Text secondary | `#888888` | Descriptions, metadata |
|
||||
| Text muted | `#555555` | Labels, placeholders |
|
||||
| Accent | `#6366f1` | Links, scores, CTAs, active states |
|
||||
| Accent gradient | `#6366f1 → #8b5cf6` | Primary buttons |
|
||||
|
||||
### Typography
|
||||
|
||||
- **Inter** — UI text (body, labels, metadata)
|
||||
- **Crimson Pro** — Branding headings (logo, page titles)
|
||||
- **JetBrains Mono** — Code blocks, API examples, monospace data
|
||||
|
||||
### Design Principles
|
||||
|
||||
- Dark theme only (matches academic demo context, highlights tile images)
|
||||
- Generous whitespace, no visual clutter
|
||||
- Images are the hero — UI chrome stays minimal
|
||||
- Subtle borders over drop shadows
|
||||
- Micro-animations for state transitions (loading, lightbox open/close)
|
||||
|
||||
## Pages
|
||||
|
||||
### 1. Search Home (`/`)
|
||||
|
||||
The landing page and primary interface.
|
||||
|
||||
**Layout:**
|
||||
- Centered logo + tagline at top: "PixelRAG — Visual retrieval over 15.7M Wikipedia tiles"
|
||||
- Search bar below: text input with search button. Supports drag-and-drop or click-to-upload for image queries. Image preview shown inline when an image is attached.
|
||||
- Mode chips below search bar: "Text query", "Image upload", "Drag & drop"
|
||||
- Results appear below after search
|
||||
|
||||
**Search Controls (collapsible):**
|
||||
- `n_docs` — number of results (default 10)
|
||||
- `nprobe` — FAISS nprobe override
|
||||
- `min_tile_height` — filter small/blank tiles
|
||||
- `instruction` — custom embedding instruction
|
||||
- Defaults are hidden; expand via "Advanced" toggle
|
||||
|
||||
**Results Display:**
|
||||
|
||||
Results are **grouped by article**. The API returns a flat ranked list of hits; the frontend groups them by `article_id`.
|
||||
|
||||
Each article group shows:
|
||||
- Article title (derived from `url` field, decode the Wikipedia slug)
|
||||
- External link to the Wikipedia article
|
||||
- Tile count badge
|
||||
- Horizontal scrollable row of tile cards
|
||||
|
||||
Each tile card shows:
|
||||
- Tile image (loaded via `GET /tile?path=...`)
|
||||
- Global rank badge (top-left corner, e.g. "#1")
|
||||
- Cosine similarity score
|
||||
- Tile height in pixels
|
||||
- Tile position identifier (e.g. "tile 2:1" = tile_index 2, chunk_index 1)
|
||||
|
||||
**Status bar** between search bar and results:
|
||||
- Result count
|
||||
- Total latency
|
||||
- Latency breakdown: measure client-side round-trip time (no backend changes needed; server-side encode/search breakdown is logged to stdout already)
|
||||
|
||||
### 2. API Documentation (`/docs`)
|
||||
|
||||
Custom-built API reference (not Swagger/ReDoc — those are functional but ugly and break visual consistency).
|
||||
|
||||
**Layout:**
|
||||
- Left sidebar: endpoint list with HTTP method badges (POST green, GET blue)
|
||||
- Guides section below endpoints: "Quick Start", "Python Client"
|
||||
- Main content area: endpoint detail
|
||||
|
||||
**Each endpoint section:**
|
||||
- Method + path + description
|
||||
- Request body schema with syntax-highlighted JSON
|
||||
- Response schema
|
||||
- "Try It" playground: editable JSON input + Send button + response preview
|
||||
- curl example
|
||||
|
||||
**Endpoints documented:**
|
||||
- `POST /search` — primary search (text, image, or embedding queries)
|
||||
- `GET /status` — index metadata and stats
|
||||
- `GET /tile?path=...` — serve tile image by path
|
||||
- `GET /health` — health check
|
||||
- `POST /reconstruct` — reconstruct stored embeddings by vector_id
|
||||
|
||||
### 3. Index Dashboard (`/status`)
|
||||
|
||||
Displays data from `GET /status` in a visual dashboard.
|
||||
|
||||
**Metric cards (2×2 grid):**
|
||||
- Total vectors (formatted: "15.7M")
|
||||
- Embedding dimension
|
||||
- Model name
|
||||
- Index size (human-readable bytes)
|
||||
|
||||
**Additional info:**
|
||||
- Index build timestamp
|
||||
- Metadata size
|
||||
- nlist / nprobe configuration
|
||||
- Index and tiles directory paths
|
||||
|
||||
Auto-refreshes on page load. No polling needed (index stats are static during a session).
|
||||
|
||||
## Interactions
|
||||
|
||||
### Tile Lightbox
|
||||
|
||||
Click any tile card → full-screen overlay:
|
||||
- Full-resolution tile image with pan and zoom (mouse wheel / pinch)
|
||||
- Metadata sidebar: score, article title + link, tile position, tile height, y_offset
|
||||
- Arrow keys or swipe to navigate between results (respects global rank order)
|
||||
- Esc or click backdrop to close
|
||||
- Animated open/close with Framer Motion
|
||||
|
||||
### Image Query
|
||||
|
||||
- Click the image upload area or drag-and-drop onto the search bar
|
||||
- Shows image preview thumbnail inline in the search bar
|
||||
- Sends base64-encoded image in the `queries[].image` field
|
||||
- Can combine with text for multimodal query (text + image simultaneously)
|
||||
|
||||
### Side-by-Side Compare
|
||||
|
||||
- Checkbox or shift-click on tile cards to select 2+ tiles
|
||||
- "Compare" button appears in a floating action bar
|
||||
- Opens a comparison panel: selected tiles shown at equal width with scores overlaid
|
||||
- Useful for evaluating retrieval quality on similar-looking results
|
||||
|
||||
### Search Controls
|
||||
|
||||
- Hidden by default behind an "Advanced" toggle
|
||||
- Collapsible panel with labeled inputs for n_docs, nprobe, min_tile_height, instruction
|
||||
- Changes take effect on next search
|
||||
- URL query params reflect current settings (shareable search URLs)
|
||||
|
||||
## Backend Changes
|
||||
|
||||
Minimal changes to `serve/src/pixelrag_serve/api.py`:
|
||||
|
||||
1. **Add CORS middleware** — allow requests from Next.js dev server (`localhost:3000`) and production origin
|
||||
2. **No other changes** — all existing endpoints remain as-is
|
||||
|
||||
```python
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
|
||||
app.add_middleware(
|
||||
CORSMiddleware,
|
||||
allow_origins=["http://localhost:3000"],
|
||||
allow_credentials=True,
|
||||
allow_methods=["*"],
|
||||
allow_headers=["*"],
|
||||
)
|
||||
```
|
||||
|
||||
## Dev & Deploy
|
||||
|
||||
### Development
|
||||
|
||||
```bash
|
||||
# Terminal 1: FastAPI backend
|
||||
pixelrag-serve --index-dir ./index --tiles-dir ./tiles --articles-json ./articles.json --device cuda
|
||||
|
||||
# Terminal 2: Next.js frontend
|
||||
cd web && npm run dev
|
||||
# Runs on localhost:3000, proxies /api/* → localhost:30001
|
||||
```
|
||||
|
||||
### Production
|
||||
|
||||
```bash
|
||||
# Build frontend
|
||||
cd web && npm run build
|
||||
|
||||
# Run both
|
||||
pixelrag-serve --device cuda --port 30001 &
|
||||
cd web && npm start -- -p 3000
|
||||
```
|
||||
|
||||
### next.config.ts Rewrites
|
||||
|
||||
```typescript
|
||||
async rewrites() {
|
||||
return [
|
||||
{
|
||||
source: '/api/:path*',
|
||||
destination: 'http://localhost:30001/:path*',
|
||||
},
|
||||
];
|
||||
},
|
||||
```
|
||||
|
||||
## Scope & Milestones
|
||||
|
||||
### Phase 1: Core Search (MVP)
|
||||
|
||||
- Project scaffolding (Next.js + Tailwind + shadcn/ui)
|
||||
- Search page with text query
|
||||
- Result display with article grouping and tile images
|
||||
- Tile lightbox with zoom
|
||||
- CORS on FastAPI
|
||||
- Navigation shell
|
||||
|
||||
### Phase 2: Full Features
|
||||
|
||||
- Image upload / drag-and-drop query
|
||||
- Side-by-side comparison panel
|
||||
- Advanced search controls
|
||||
- API documentation page with try-it playground
|
||||
- Index status dashboard
|
||||
|
||||
### Phase 3: Polish
|
||||
|
||||
- Loading states and skeleton screens
|
||||
- Error handling and empty states
|
||||
- Shareable search URLs (query params)
|
||||
- Keyboard navigation (arrow keys in lightbox, Cmd+K for search focus)
|
||||
- Performance optimization (image lazy loading, virtualized lists for large result sets)
|
||||
@@ -0,0 +1,6 @@
|
||||
# Eval run artifacts — local only
|
||||
*.jsonl
|
||||
*.json
|
||||
*.db
|
||||
*_out/
|
||||
eval_output/
|
||||
@@ -1,129 +0,0 @@
|
||||
# Paper Experiment Map
|
||||
Maps paper results → source experiments in `~/pixelrag-src/Vis-RAG/agent/experiments/`.
|
||||
|
||||
## Shared Config (all paper experiments unless noted)
|
||||
- **think**: enabled (no `--no-think` flag)
|
||||
- **max_tokens**: 16384
|
||||
- **retrieval_top_k**: 5
|
||||
- **reader_top_k**: 3
|
||||
- **query_instruction (pixel)**: "Retrieve images or text relevant to the user's query."
|
||||
- **query_instruction (text)**: "Retrieve text relevant to the user's query."
|
||||
- **Readers**: Qwen3-VL-4B-Instruct (VL-4B) and Qwen3.5-4B (Q3.5)
|
||||
|
||||
## Table 1: Text-centric Wikipedia QA
|
||||
|
||||
### SimpleQA → `simpleqa_paper_top3_v1`
|
||||
- Script: `experiments/simpleqa_paper_top3_v1/run.sh`
|
||||
- Grader: GPT-4o judge (`scripts/evaluate.py simpleqa`)
|
||||
- Ports: base=30888, LoRA=30893, DoRA=30895, Traf=30889, NeuML=30896
|
||||
- n=1000
|
||||
- summary.tsv has graded_count, not accuracy (accuracy was in evaluate.py stdout)
|
||||
- Outputs: `$EXP_DIR/outputs/sqa_*.jsonl` (cleaned/deleted)
|
||||
|
||||
### NQ → `nq_paper_top3_v1`
|
||||
- Script: `experiments/nq_paper_top3_v1/run.sh`
|
||||
- Grader: exact match
|
||||
- n=1000
|
||||
- summary.tsv has EM and F1
|
||||
|
||||
### NQ-Tables → `nqt_paper_top3_v1`
|
||||
- Script: `experiments/nqt_paper_top3_v1/run.sh`
|
||||
- Grader: exact match
|
||||
- n=1068
|
||||
- summary.tsv has EM and F1
|
||||
|
||||
### TriviaQA → `triviaqa_paper_top3_v1`
|
||||
- Script: `experiments/triviaqa_paper_top3_v1/run.sh`
|
||||
- Grader: exact match
|
||||
- n=1000
|
||||
|
||||
## Table 1: Multimodal QA
|
||||
|
||||
### MMSearch → `mmsearch_paper_top3_v1`
|
||||
- Script: `experiments/mmsearch_paper_top3_v1/run.sh`
|
||||
- n=300
|
||||
- summary.tsv has scores
|
||||
|
||||
### EVQA → `evqa_paper_top3_v1`
|
||||
- Script: `experiments/evqa_paper_top3_v1/run.sh`
|
||||
- Grader: GPT-4.1 judge
|
||||
- n=1000 per subset (landmarks, inaturalist)
|
||||
- NOTE: Q3.5 cells originally ran with `--no-think`, later backfilled in `q35_think_backfill_v1`
|
||||
|
||||
### LiveVQA → `livevqa_v3_qa_v1`
|
||||
- Script: `experiments/livevqa_v3_qa_v1/run.sh` (if exists)
|
||||
- Also backfilled in `q35_think_backfill_v1`
|
||||
|
||||
## Figure 2: Token Efficiency (SimpleQA)
|
||||
|
||||
### No-think version → `token_efficiency_q35_nothink_v1`
|
||||
- Script: `experiments/token_efficiency_q35_nothink_v1/run.sh`
|
||||
- max_tokens=200, --no-think
|
||||
- summary.tsv has actual accuracy numbers:
|
||||
- base top1=0.575, top2=0.677, top3=0.722
|
||||
- LoRA top1=0.629, top2=0.719, top3=0.750
|
||||
- These are NO-THINK numbers; paper Figure 2 likely uses think numbers
|
||||
|
||||
### Bug-fixed text version → `token_efficiency_v2`
|
||||
- Fixed text retrieval bug (retrieval_top_k used instead of reader_top_k)
|
||||
- Adds top-2 cells
|
||||
|
||||
## Table 3: Modality Ablation → `ablation_modality_v1`
|
||||
- Script: `experiments/ablation_modality_v1/run.sh`
|
||||
|
||||
## Think vs No-Think
|
||||
|
||||
### `q35_nothink_full_v1`
|
||||
- Full benchmark sweep with Q3.5 no-think (max_tokens=200)
|
||||
- Intended as comparison to VL-4B paper runs
|
||||
|
||||
### `q35_think_backfill_v1`
|
||||
- Re-runs Q3.5 cells with think enabled (max_tokens=16384)
|
||||
- Matches VL-4B paper config exactly
|
||||
- Backfills EVQA, NeuML text, LiveVQA
|
||||
|
||||
### `q35_matrix_completion_v1`
|
||||
- Fills missing cells in think/no-think × retriever × k matrix
|
||||
- Expected values noted in README:
|
||||
- no-think base top3: ~72.2%
|
||||
- think LoRA top3: ~77.9%
|
||||
- think Traf top3: ~70.2%
|
||||
|
||||
## Reference Numbers from Experiment Summaries
|
||||
|
||||
### NQ (EM, from nq_paper_top3_v1/summary.tsv)
|
||||
q35: base=0.338, lora=0.328, dora=0.334, traf=0.280
|
||||
vl4b: base=0.317, lora=0.311, dora=0.311, traf=0.294
|
||||
|
||||
### NQ-Tables (EM, from nqt_paper_top3_v1/summary.tsv)
|
||||
q35: base=0.258, lora=0.275, dora=0.274, traf=0.227 (n=497!)
|
||||
vl4b: base=0.241, lora=0.266, dora=0.271, traf=0.219
|
||||
|
||||
### MMSearch (score, from mmsearch_paper_top3_v1/summary.tsv)
|
||||
q35: base=0.287, lora=0.277, dora=0.283, traf=0.253, naive=0.147
|
||||
vl4b: base=0.240, lora=0.247, dora=0.240, traf=0.203, naive=0.130
|
||||
|
||||
### TriviaQA (EM, from triviaqa_paper_top3_v1/summary.tsv)
|
||||
q35: base=0.718, lora=0.718, dora=0.710, traf=0.714 (n=248!)
|
||||
vl4b: base=0.696, lora=0.713, dora=0.702, traf=0.731
|
||||
|
||||
### SimpleQA no-think (accuracy, from token_efficiency_q35_nothink_v1/summary.tsv)
|
||||
base: top1=0.575, top2=0.677, top3=0.722
|
||||
LoRA: top1=0.629, top2=0.719, top3=0.750
|
||||
|
||||
### SimpleQA think (expected, from q35_matrix_completion_v1/README.md)
|
||||
base top3: ~72.2% (no-think ~72.2% — think doesn't help base much)
|
||||
LoRA top3: ~77.9% (no-think 75.0% — think adds ~3%)
|
||||
Traf top3: ~70.2% (no-think ~68.5% est — think adds ~2%)
|
||||
|
||||
## Key Findings for Reproduction
|
||||
|
||||
1. **All paper Q3.5 numbers use think mode** (max_tokens=16384), not no-think
|
||||
2. Our no-think runs are ~3-6% lower than paper think numbers (SimpleQA LoRA/Traf)
|
||||
3. Base pixel is insensitive to think (72.2% think vs 72.2% no-think)
|
||||
4. NQ/NQ-Tables use exact match grading, less sensitive to think/no-think
|
||||
5. SimpleQA uses LLM judge (GPT-4o in paper, GPT-4.1 in ours)
|
||||
6. The LoRA index needs the merged LoRA encoder model for query encoding
|
||||
- Adapter: `/opt/dlami/nvme/adapters/lora_vit_ckpt200/lora_vit/ckpt200`
|
||||
- Merged model: created at runtime via `PeftModel.from_pretrained()` + `merge_and_unload()`
|
||||
- See `embedding/embed_tiles.py:558-582`
|
||||
+173
@@ -0,0 +1,173 @@
|
||||
# Reproducing PixelRAG paper Table 1 (Qwen3.5-4B, k=3)
|
||||
|
||||
Everything needed lives in this directory: the benchmark driver (`run_bench.py`), the
|
||||
dataset loaders (`lib/`), and the LLM-judge grader (`lib/grader.py`). No external checkout
|
||||
is required.
|
||||
|
||||
The reproduction script just runs the pipeline and prints a score. Run the reader on an
|
||||
**H100** to match the paper's greedy decode.
|
||||
|
||||
## 1. Environment (locked)
|
||||
|
||||
```bash
|
||||
cd eval
|
||||
uv sync --frozen # base client (retrieval + reader + grader over HTTP); Python 3.12
|
||||
|
||||
# Optional — to self-host the Qwen3.5-4B reader from this package (needs a CUDA GPU):
|
||||
uv sync --frozen --extra reader # also installs vLLM 0.19.0 into eval/.venv
|
||||
```
|
||||
|
||||
The base install is a pure HTTP client — no torch/vllm. The `reader` extra adds vLLM so you
|
||||
can serve the reader from the same venv; vLLM needs `numpy<2.3`, hence the numpy bound.
|
||||
|
||||
Grader needs an OpenAI key with access to `gpt-4.1-2025-04-14`. `reproduce.sh` auto-loads
|
||||
`OPENAI_API_KEY` / `OPENAI_BASE_URL` from `../.env`.
|
||||
|
||||
## 2. Serve topology (must be running before `reproduce.sh`)
|
||||
|
||||
| role | default port | index / model | notes |
|
||||
|------|------|------|------|
|
||||
| **reader** | `READER_URL` :8010 | `Qwen/Qwen3.5-4B`, **vLLM 0.19.0**, **H100** | install via `uv sync --extra reader`, then `CUDA_VISIBLE_DEVICES=0 HF_HOME=… .venv/bin/vllm serve Qwen/Qwen3.5-4B --port 8010` on an H100 (tunnel to :8010 if remote) |
|
||||
| base pixel | :30088 | `search_index_normed_v2` (wiki, 28.2M), base encoder, direct_gpu | multimodal query |
|
||||
| lora pixel | :30096 | wiki lora-vit-ckpt200 index (26.3M) | multimodal query |
|
||||
| traf text | :30097 | `text_search_index_1024_normed` (wiki, 15.7M, nprobe 128) | text query |
|
||||
| news pixel | :30095 | `news_image_search_index` (3.63M, nprobe 128), base, direct_gpu | LiveVQA only |
|
||||
|
||||
All pixel/text serves are **direct_gpu** (the reader sends the raw query; the serve encodes
|
||||
it — do NOT POST precomputed embeddings). Local tiles for the reader live at
|
||||
`TILES_DIR=/mnt/data/yichuan/kiwix_tiles` (wiki) and `/mnt/data/yichuan/news_tiles` (news);
|
||||
EVQA query images at `/mnt/data/yichuan/{landmark,inat}_images/`. The HF datasets
|
||||
(`CaraJ/MMSearch`, encyclopedic_vqa csv) are read from `~/.cache`. LiveVQA reads its QA
|
||||
dataset (questions/options/GT/img_path) from `LIVEVQA_V4_PATH`
|
||||
(default `/mnt/data/yichuan/livevqa_v4_multimodal.json`; retrieval is re-done live).
|
||||
|
||||
These data dirs are large external inputs (not vendored in the repo), same as the tile
|
||||
stores and HF caches.
|
||||
|
||||
## Data sources (where each input comes from)
|
||||
|
||||
| input | size | source |
|
||||
|-------|------|--------|
|
||||
| FAISS indexes (base/lora pixel, text, news) | ~570G | HF dataset `StarTrail-org/pixelrag-faiss-indexes` (4 subdirs; `serve_up.sh` downloads them) |
|
||||
| reader Qwen3.5-4B / LoRA encoder / training data / QA datasets | — | HF (`Qwen/Qwen3.5-4B`, `Chrisyichuan/*`, `CaraJ/MMSearch`, encyclopedic_vqa csv) |
|
||||
| **wiki + news tiles** (reader's image evidence) | ~4T | HF dataset `StarTrail-org/pixelrag-tiles` (or regenerate from the public kiwix ZIM via the `render` stage) |
|
||||
| EVQA/LiveVQA query images (landmark/inat/editorial photo) | ~6G | small; landmark=GLDv2, inat=iNaturalist, livevqa=editorial photos (note: editorial photos are copyrighted — redistribute with care) |
|
||||
|
||||
So: indexes, tiles, models, and QA all come straight from HF; the tile corpus can also be
|
||||
regenerated from the public Wikipedia ZIM via the render pipeline.
|
||||
|
||||
### Three ways to run retrieval + supply the tile images
|
||||
|
||||
The reader always asks the serve for images (`include_images`), so the retrieved tiles can
|
||||
reach it three ways — pick one:
|
||||
|
||||
1. **Self-hosted serve, index + tiles.** `serve_up.sh` downloads the FAISS index and the tile
|
||||
corpus (`StarTrail-org/pixelrag-tiles`, or rendered from the ZIM). The serve returns each
|
||||
retrieved tile inline as base64; or set `TILES_DIR` to read the tiles from the reader's
|
||||
local disk instead. Full self-host.
|
||||
2. **Public API (no self-hosting).** Point the retrieval URL at the public endpoint
|
||||
(e.g. `http://api.pixelrag.ai:30001/search`) instead of a local serve. It returns base64
|
||||
tiles, so you only run the reader + grader — no index, no tile corpus. Note: `:30001`
|
||||
serves the **un-normed** base index (`search_index`), which does **not** match the paper's
|
||||
`base`/`lora` cells (those use `search_index_normed_v2`) — handy for exercising the pipeline,
|
||||
but self-host the normed index to match the paper numbers. Command in §3.
|
||||
3. **Self-hosted serve, index + on-demand render.** Run the serve with the index but **no** tile
|
||||
corpus, started with an on-demand renderer; it renders each retrieved page to tiles at query
|
||||
time and returns them as base64. Needs the kiwix ZIM, not the ~4T corpus.
|
||||
|
||||
In modes 2 and 3 the reader needs no local tiles, so leave `TILES_DIR` empty.
|
||||
|
||||
**Self-hosting the search serve (modes 1 & 3) — what you need:**
|
||||
- `pip install -e '..[serve]'` (faiss + torch/torchvision + transformers + the query encoder
|
||||
`Qwen/Qwen3-VL-Embedding-2B`). This is **separate from the eval client and the reader** — a
|
||||
CUDA GPU box with plenty of RAM.
|
||||
- The FAISS index: `search_index_normed_v2` (~217G download from `StarTrail-org/pixelrag-faiss-indexes`,
|
||||
~220G RAM to load — `serve_up.sh` fetches it).
|
||||
- `articles.json` (the article-id → wiki-slug map the serve needs to resolve hit URLs) lives in
|
||||
the **`StarTrail-org/pixelrag-tiles`** dataset, not the faiss-indexes one.
|
||||
- Tiles for the reader: either the full corpus at `TILES_DIR` (mode 1), or start the serve with
|
||||
`--render-on-demand --kiwix-url <kiwix-serve>` so it renders only the retrieved pages from a
|
||||
kiwix ZIM (mode 3) — no ~4T corpus.
|
||||
- On-demand render (mode 3) is slow (one page rendered per retrieved tile). Raise the retrieval
|
||||
client timeout so a batch doesn't time out into an empty (closed-book) result:
|
||||
`PIXELRAG_RETRIEVAL_TIMEOUT=7200`.
|
||||
|
||||
## 3. Run a cell
|
||||
|
||||
```bash
|
||||
bash reproduce.sh <bench> <retrieval>
|
||||
# bench = nq | nqt | sqa | mms | evqa | livevqa
|
||||
# retrieval = naive | traf | base | lora
|
||||
# e.g.
|
||||
bash reproduce.sh evqa base # -> prints Score: 0.4xx
|
||||
bash reproduce.sh mms lora
|
||||
NUM=20 bash reproduce.sh nq traf # NUM overrides the example count for a quick smoke
|
||||
```
|
||||
|
||||
`reproduce.sh` targets **self-hosted serves on localhost** (modes 1/3 above). To run a cell
|
||||
against the **public API** (mode 2), drive `run_bench.py` directly — `reproduce.sh` hardcodes
|
||||
`localhost`, so it can't point at a remote serve:
|
||||
|
||||
```bash
|
||||
# NQ via the public base endpoint (then grade — see §5; the paper used --llm-judge):
|
||||
.venv/bin/python run_bench.py --task nq --model Qwen/Qwen3.5-4B \
|
||||
--api-base "$READER_URL" --api-key dummy --no-think \
|
||||
--retrieval-top-k 5 --reader-top-k 3 --num-examples 1000 --max-tokens 200 \
|
||||
--local-api --local-api-url http://api.pixelrag.ai:30001/search \
|
||||
--query-instruction "Retrieve images or text relevant to the user's query."
|
||||
```
|
||||
|
||||
Before running, `reproduce.sh` runs a **preflight**: it curls the reader and the retrieval
|
||||
serve(s) that *this* cell needs and checks each is up with the expected index (`/status`
|
||||
`total_vectors`). If a serve is down / on the wrong port / wrong index, it prints the exact
|
||||
`pixelrag serve --index-dir … --port …` command to launch it and exits (no silent empty run).
|
||||
|
||||
Per-cell config is locked inside `reproduce.sh`:
|
||||
|
||||
| bench | think | max_tokens | n | grader | notes |
|
||||
|-------|-------|-----------|---|--------|-------|
|
||||
| nq / nqt | no-think | 200 | 1000 / all | LLM-judge¹ | ¹paper numbers used the gpt-4.1 judge; `reproduce.sh` passes `--llm-judge` (needs the OpenAI key) |
|
||||
| sqa | no-think | 200 | 1000 | SimpleQA judge | nprobe 2000 |
|
||||
| mms (base/lora/traf) | **think** | 16384 | all | WorldVQA judge | pixel instr = V1 "Retrieve images or text relevant to the user's query." |
|
||||
| mms (naive) | no-think | 200 | all | WorldVQA judge | |
|
||||
| evqa | no-think | 16384 | all | WorldVQA judge | **landmarks + question_type=automatic only**; iNaturalist & templated/multi_answer excluded |
|
||||
| livevqa (naive/base) | no-think | 16 | all | MCQ exact-match | news pipeline `run_livevqa.py` |
|
||||
|
||||
## 4. Published numbers (for your own comparison — NOT used by the script)
|
||||
|
||||
Paper Table 1 (Qwen3.5-4B, k=3):
|
||||
|
||||
| | naive | Trafilatura | base | LoRA |
|
||||
|---|---|---|---|---|
|
||||
| NQ | 30.4 | 55.9 | 57.9 | 58.7 |
|
||||
| NQ-Tables | 24.5 | 42.5 | 47.0 | 48.8 |
|
||||
| SimpleQA | 7.0 | 71.6 | 73.8 | 78.8 |
|
||||
| LiveVQA | 63.6 | 59.0 | 70.3 | 70.0 |
|
||||
| MMSearch | 12.7 | 24.7 | 28.3 | 28.3 |
|
||||
| EVQA (lm/auto) | 27.2 | 29.6 | 40.7 | 45.1 |
|
||||
|
||||
On H100, this harness reproduces the pixel cells (LiveVQA/MMS/EVQA base+lora) within ~1pp.
|
||||
|
||||
NOTE on NQ/NQ-Tables grading: the paper's published numbers use the **gpt-4.1 LLM judge**
|
||||
(semantic match), not strict exact-match. The reader answers short but paraphrases the gold
|
||||
span, so strict exact-match scores ~20pp lower (≈ the naive number) even when the answer is
|
||||
right. Grade these cells with `--llm-judge` (what `reproduce.sh` does) to match the paper.
|
||||
|
||||
NOTE on traf (text retrieval): `reproduce.sh` passes `--no-query-image` to match the paper's
|
||||
text-only text retrieval.
|
||||
|
||||
## 5. Grader
|
||||
|
||||
`eval/lib/grader.py` (faithful to the paper's grading procedure):
|
||||
- WorldVQA judge (mmsearch / encyclopedic_vqa): prompt verbatim, GT for EVQA =
|
||||
`"Any of: " + " | ".join(reference_list)` (any reference matches → correct), `<think>` stripped,
|
||||
judge gpt-4.1 temp 0 + `system="You are a helpful assistant."` + `seed=42` + `max_tokens=1000`.
|
||||
- nq / nq_tables: **default** strict exact-match (SQuAD normalize + equality vs the gold list,
|
||||
no API key). Pass `--llm-judge` to grade with the gpt-4.1 judge instead — that is what the
|
||||
paper used for its published NQ/NQT numbers (strict exact-match runs ~20pp lower because the
|
||||
reader paraphrases). `reproduce.sh` uses `--llm-judge` for these cells.
|
||||
- SimpleQA judge (simpleqa): the SimpleQA `GRADER_TEMPLATE` → A/B/C.
|
||||
|
||||
```bash
|
||||
PYTHONPATH=. .venv/bin/python -m lib.grader <task> <responses.jsonl>
|
||||
```
|
||||
@@ -1,118 +0,0 @@
|
||||
# Reproducing PixelRAG paper Table 1 (Qwen3.5-4B, k=3)
|
||||
|
||||
Self-contained in this repo (`eval/run_bench.py` + `eval/lib/` + `eval/lib/grader.py`).
|
||||
**No dependency on the old `Vis-RAG` / `dr-agent` repo.** The driver and grader were
|
||||
migrated from it (provenance noted in the file headers); the old repo can be deleted.
|
||||
|
||||
The reproduction script just runs the pipeline and prints a score. It does **not** compare
|
||||
to the paper and does **not** branch on hardware. Run the reader on an **H100** and the
|
||||
numbers land within ~1pp of the paper (B200 systematically diverges ~0.6–1.6pp on the
|
||||
greedy decode; see `gpu-hardware-reproduction`).
|
||||
|
||||
## 1. Environment (locked)
|
||||
|
||||
```bash
|
||||
cd eval
|
||||
uv sync --frozen # creates eval/.venv from pyproject.toml + uv.lock (Python 3.12)
|
||||
```
|
||||
|
||||
Grader needs an OpenAI key with access to `gpt-4.1-2025-04-14`. `reproduce.sh` auto-loads
|
||||
`OPENAI_API_KEY` / `OPENAI_BASE_URL` from `../.env`.
|
||||
|
||||
## 2. Serve topology (must be running before `reproduce.sh`)
|
||||
|
||||
| role | default port | index / model | notes |
|
||||
|------|------|------|------|
|
||||
| **reader** | `READER_URL` :8010 | `Qwen/Qwen3.5-4B`, **vLLM 0.19.0**, **H100** | `CUDA_VISIBLE_DEVICES=0 HF_HOME=… vllm serve Qwen/Qwen3.5-4B --port 8010` on an H100; tunnel it to :8010 |
|
||||
| base pixel | :30088 | `search_index_normed_v2` (wiki, 28.2M), base encoder, direct_gpu | multimodal query |
|
||||
| lora pixel | :30096 | wiki lora-vit-ckpt200 index (26.3M) | multimodal query |
|
||||
| traf text | :30097 | `text_search_index_1024_normed` (wiki, 15.7M, nprobe 128) | text query |
|
||||
| news pixel | :30095 | `news_image_search_index` (3.63M, nprobe 128), base, direct_gpu | LiveVQA only |
|
||||
|
||||
All pixel/text serves are **direct_gpu** (the reader sends the raw query; the serve encodes
|
||||
it — do NOT POST precomputed embeddings). Local tiles for the reader live at
|
||||
`TILES_DIR=/mnt/data/yichuan/kiwix_tiles` (wiki) and `/mnt/data/yichuan/news_tiles` (news);
|
||||
EVQA query images at `/mnt/data/yichuan/{landmark,inat}_images/`. The HF datasets
|
||||
(`CaraJ/MMSearch`, encyclopedic_vqa csv) are read from `~/.cache`. LiveVQA reads its QA
|
||||
dataset (questions/options/GT/img_path) from `LIVEVQA_V4_PATH`
|
||||
(default `/mnt/data/yichuan/livevqa_v4_multimodal.json`; retrieval is re-done live).
|
||||
|
||||
These data dirs are large external inputs (not vendored in the repo), same as the tile
|
||||
stores and HF caches.
|
||||
|
||||
## Data sources (where each input comes from)
|
||||
|
||||
| input | size | source |
|
||||
|-------|------|--------|
|
||||
| FAISS indexes (base/lora pixel, text, news) | ~570G | HF dataset `StarTrail-org/pixelrag-faiss-indexes` (4 subdirs; `serve_up.sh` downloads them) |
|
||||
| reader Qwen3.5-4B / LoRA encoder / training data / QA datasets | — | HF (`Qwen/Qwen3.5-4B`, `Chrisyichuan/*`, `CaraJ/MMSearch`, encyclopedic_vqa csv) |
|
||||
| **wiki + news tiles** (reader's image evidence) | **~13T** (12T wiki + 838G news) | **NOT on HF** — render from the public kiwix ZIM via the `render` stage (render→embed→index→serve), or render on-demand for the retrieved pages. Too large to publish. |
|
||||
| EVQA/LiveVQA query images (landmark/inat/editorial photo) | ~6G | small; landmark=GLDv2, inat=iNaturalist, livevqa=editorial photos (note: editorial photos are copyrighted — redistribute with care) |
|
||||
|
||||
So: indexes + models + QA come straight from HF; the 13T tile corpus is regenerated from the
|
||||
public Wikipedia ZIM (not downloaded), which is the only piece that needs the render pipeline.
|
||||
|
||||
## 3. Run a cell
|
||||
|
||||
```bash
|
||||
bash reproduce.sh <bench> <retrieval>
|
||||
# bench = nq | nqt | sqa | mms | evqa | livevqa
|
||||
# retrieval = naive | traf | base | lora
|
||||
# e.g.
|
||||
bash reproduce.sh evqa base # -> prints Score: 0.4xx
|
||||
bash reproduce.sh mms lora
|
||||
NUM=20 bash reproduce.sh nq traf # NUM overrides the example count for a quick smoke
|
||||
```
|
||||
|
||||
Before running, `reproduce.sh` runs a **preflight**: it curls the reader and the retrieval
|
||||
serve(s) that *this* cell needs and checks each is up with the expected index (`/status`
|
||||
`total_vectors`). If a serve is down / on the wrong port / wrong index, it prints the exact
|
||||
`pixelrag serve --index-dir … --port …` command to launch it and exits (no silent empty run).
|
||||
|
||||
Per-cell config is locked inside `reproduce.sh` (verified against the paper's saved
|
||||
response metadata, not the experiment scripts):
|
||||
|
||||
| bench | think | max_tokens | n | grader | notes |
|
||||
|-------|-------|-----------|---|--------|-------|
|
||||
| nq / nqt | no-think | 200 | 1000 / 1068 | exact-match | |
|
||||
| sqa | no-think | 200 | 1000 | SimpleQA judge | nprobe 2000 |
|
||||
| mms (base/lora/traf) | **think** | 16384 | 300 | WorldVQA judge | pixel instr = V1 "Retrieve images or text relevant to the user's query." (NOT promptG) |
|
||||
| mms (naive) | no-think | 200 | 300 | WorldVQA judge | |
|
||||
| evqa | no-think | 16384 | 749 | WorldVQA judge | **landmarks + question_type=automatic only**; iNaturalist & templated/multi_answer excluded |
|
||||
| livevqa (naive/base) | no-think | 16 | 26888 | MCQ exact-match | news pipeline `run_livevqa.py` |
|
||||
|
||||
## 4. Published numbers (for your own comparison — NOT used by the script)
|
||||
|
||||
Paper Table 1 (Qwen3.5-4B, k=3):
|
||||
|
||||
| | naive | Trafilatura | base | LoRA |
|
||||
|---|---|---|---|---|
|
||||
| NQ | 30.4 | 55.9 | 57.9 | 58.7 |
|
||||
| NQ-Tables | 24.5 | 42.5 | 47.0 | 48.8 |
|
||||
| SimpleQA | 7.0 | 71.6 | 73.8 | 78.8 |
|
||||
| LiveVQA | 63.6 | 59.0 | 70.3 | 70.0 |
|
||||
| MMSearch | 12.7 | 24.7 | 28.3 | 28.3 |
|
||||
| EVQA (lm/auto) | 27.2 | 29.6 | 40.7 | 45.1 |
|
||||
|
||||
On H100, this harness reproduces every pixel cell (LiveVQA/MMS/EVQA base+lora) within ~1pp.
|
||||
The MMS/EVQA grader (`gpt-4.1-2025-04-14`, temp 0) has ~2–6pp run-to-run noise, so re-grading
|
||||
even the paper's own responses wanders by that much.
|
||||
|
||||
NOTE on traf (text retrieval): the paper kept text retrieval **text-only** (it did NOT send the
|
||||
query image to the text serve — the "add query image to text retrieval" change existed but was
|
||||
not used in the paper). `reproduce.sh` therefore passes `--no-query-image` for traf. An earlier
|
||||
run WITHOUT it sent the landmark photo to the text serve, ~2x'd EVQA-traf retrieval recall
|
||||
(9.1% vs 4.8%) and read ~+4pp high — that was a config bug on our side, not "better retrieval".
|
||||
|
||||
## 5. Grader
|
||||
|
||||
`eval/lib/grader.py` (migrated, byte-faithful to the paper's `evaluate.py` + `worldvqa_eval`):
|
||||
- WorldVQA judge (mmsearch / encyclopedic_vqa): prompt verbatim, GT for EVQA =
|
||||
`"Any of: " + " | ".join(reference_list)` (any reference matches → correct), `<think>` stripped,
|
||||
judge gpt-4.1 temp 0 + `system="You are a helpful assistant."` + `seed=42` + `max_tokens=1000`.
|
||||
- exact-match (nq / nq_tables): SQuAD-style normalize + match against the gold answer list.
|
||||
- SimpleQA judge (simpleqa): the SimpleQA `GRADER_TEMPLATE` → A/B/C.
|
||||
|
||||
```bash
|
||||
PYTHONPATH=. .venv/bin/python -m lib.grader <task> <responses.jsonl>
|
||||
```
|
||||
@@ -1,366 +0,0 @@
|
||||
# ============ TODO -- OPEN WORK FOR A CLEAN, NO-FREEZING REPRODUCTION ============
|
||||
# Principle (per user): reproduction must run the FULL pipeline live. Freezing the paper's
|
||||
# retrieval JSON is a SHORTCUT and does NOT count -- others can't reproduce it. Evaluate
|
||||
# retrieval by RECALL@k (gold gt_hex_id/gt-tile in top-k) + final ACC, NOT byte-exact tiles.
|
||||
#
|
||||
# [x] 1. LiveVQA retrieval -- DONE, REPRODUCES LIVE (no freezing). eval/repro_livevqa_live_retrieval.py
|
||||
# sends raw {image,text} to news serve :30095 (= paper's news_image_search_index,
|
||||
# 3,626,535 vec, nprobe=128, base Qwen3-VL-Embedding-2B) and lets the serve's OWN
|
||||
# direct_gpu encoder embed -> recall matches the frozen :30890 EXACTLY on 100 ex:
|
||||
# @1 29/29, @3 42/42, @5 50/50, @10 57/56. Two earlier myths busted: (a) "paper 57%"
|
||||
# was recall@10 mislabeled as @3 (LIVE @10 = 57%); (b) our "live 31%" came from POSTing
|
||||
# PRECOMPUTED bf16-SDPA embeddings (eval/embed_query_gpu.py) which DON'T align with the
|
||||
# serve's torch.compile direct_gpu encode -- DON'T POST embeddings, send raw queries.
|
||||
# The 30095 index was paper's all along; no rebuild needed.
|
||||
# [x] 2. LiveVQA reader on LIVE tiles -- DONE, end-to-end no-freeze ACC measured (full n=26888,
|
||||
# same :8000 Qwen3.5-4B reader, no-think, top_k=3): LIVE retrieval 70.31% vs FROZEN
|
||||
# retrieval 70.34% vs paper 70.3% -- live==frozen within 0.03pp (9/26888), both hit paper.
|
||||
# Note: top-3 tiles differ on ~23% of examples (FAISS approx + cross-instance encode), but
|
||||
# the final answer is unchanged on those, so end-to-end ACC is identical. Pipeline:
|
||||
# eval/repro_livevqa_retrieve.py (:30095) -> eval/repro_livevqa_reader.py (:8000). Fully
|
||||
# live, zero freezing. Artifacts: eval/live_pixel_full.json, eval/{live,frozen}_reader_full.json.
|
||||
# [x] 3. MMS -- DONE, no-freeze no-POST-hack live, via the PAPER'S OWN driver
|
||||
# (run_naive_simpleqa.py) hitting our GPU pixel serves (:30088 base, :30096 lora) live,
|
||||
# reader :8000 Qwen3.5-4B THINKING-ON max_tokens=16384 (NOT no-think -- MMS run.sh keeps
|
||||
# thinking), retrieval_top_k=5 reader_top_k=3, instruction V1. WorldVQA same-grader (n=300,
|
||||
# ours-live vs paper-responses): base 27.7 vs 29.7 (-2.0) | lora 28.3 vs 27.7 (+0.6) |
|
||||
# naive 17.0 vs 12.7 (+4.3). All within MMS's known high variance (grader API
|
||||
# non-determinism + n=300 + reader GPU arch). Pipeline: eval/repro_mms_driver.py (thin
|
||||
# wrapper that runs the paper driver + monkeypatches LocalAPIRetriever._hits_to_result to
|
||||
# glob-resolve the serve's '?' subshard placeholder in tile paths; the serve's FAISS
|
||||
# metadata lost the subshard, pure path fix, no semantic change). Deps: eval/.venv-agent.
|
||||
# Tiles via /opt/dlami/nvme symlinks -> /mnt/data/yichuan. GRADER KEY: use .env key, the
|
||||
# shell OPENAI_API_KEY is an archived/dead project (else all-incorrect 0/300).
|
||||
# Artifacts: eval/mms_{base,lora,naive}_live.jsonl.
|
||||
# [x] 4. EVQA-pixel -- DONE, no-freeze live, MATCHES PUBLISHED within 1pp. Two corrections were
|
||||
# needed (found by reproducing the paper's grader, per user -- do NOT dismiss gaps as noise):
|
||||
# (A) GRADER: use the PAPER'S OWN scripts/evaluate.py (task encyclopedic_vqa -> WorldVQAEval
|
||||
# with GT = "Any of: " + reference_list joined, ANY-match counts correct, strips <think>).
|
||||
# Our hand-rolled eval/grade_evqa_worldvqa.py used only a single answer -> systematically
|
||||
# ~5pp LOW. PROOF evaluate.py is the right grader: re-grading PAPER's responses recovers
|
||||
# published within noise (base 42.5 vs pub 40.7, lora 43.8 vs 45.1).
|
||||
# (B) READER CONFIG: paper EVQA used reader_no_think=TRUE (run_metadata confirms; paper
|
||||
# responses median 537 chars, 0 <think>). We first wrongly used THINKING (q35_think_backfill
|
||||
# is an ABLATION, not the published config) -> rambling 9499-char responses that never
|
||||
# commit a clean "Exact Answer" -> grader marks more incorrect -> -4pp.
|
||||
# After both fixes (--no-think + evaluate.py): base lm 39.4/inat 44.3 = COMBINED 41.8 (pub
|
||||
# 40.7, +1.1); lora lm 42.9/inat 46.6 = COMBINED 44.8 (pub 45.1, -0.3). Retrieval recall
|
||||
# ours >= paper (base lm R@1 17.0 vs 13.8) -> gap was purely reader-config, not retrieval.
|
||||
# Config: paper driver encyclopedic_vqa, :30088 base/:30096 lora live, Qwen3.5-4B --NO-THINK
|
||||
# max_tokens=16384 rtk5/rk3 instruction V1, --tiles-dir /mnt/data/yichuan/tiles_evqa.
|
||||
# Artifacts: eval/evqa_{base,lora}_{landmarks,inat}_nothink.jsonl. EVQA-traf already live (text).
|
||||
# NOTE: MMS naive also needed no-think + max_tokens=200 (paper config); ours 14.0 vs pub 12.7.
|
||||
# [ ] 4. Reader-side residual: rerun the reproduced cells on flowmatic H100 (paper's reader GPU
|
||||
# type) to remove the B200-vs-H100 greedy-decode divergence (proven 24%->43% byte-match).
|
||||
# [ ] 5. Grader: always grade with the paper WorldVQA judge for MMS/EVQA (eval/grade_*worldvqa.py)
|
||||
# and compare same-grader (ours vs paper-responses-regraded), never to published numbers.
|
||||
# [ ] 6. Remove all /tmp dependencies from the eval scripts; pin everything under eval/ + uv.lock
|
||||
# so the whole pipeline is rerunnable from scratch.
|
||||
# ================================================================================
|
||||
#
|
||||
# ============ PER-CELL CONFIG QUICK REFERENCE (how to reproduce each cell) ============
|
||||
# Shared reader: Qwen/Qwen3.5-4B, no-think (enable_thinking=False), temp=0, vLLM 0.19.0.
|
||||
# Shared retrieval (rtk=5, rk=3): base pixel = port 30088 normed_v2 (28.2M); lora pixel =
|
||||
# 30096 LoRA index + pre-merged encoder; traf text = 30097 wiki text. Pixel/multimodal
|
||||
# query instruction = "Retrieve images or text relevant to the user's query." (NOT promptG);
|
||||
# text instruction = "Retrieve text relevant to the user's query.".
|
||||
# *** Pixel/multimodal (image-in-query) retrieval: the CLEANEST reproduction is to run the
|
||||
# index serve with the direct_gpu backend on a GPU and send RAW {image,text} queries so
|
||||
# the serve encodes natively (cosine=1.0 with the bf16-built index). PROVEN on LiveVQA:
|
||||
# raw->serve recall matches frozen :30890 EXACTLY (@3 42/42). Do NOT POST precomputed
|
||||
# embeddings: our bf16-SDPA encode (eval/embed_query_gpu.py) does not match the serve's
|
||||
# torch.compile max-autotune encode and misaligns (LiveVQA 31% vs 42%). Precomputed-POST
|
||||
# is only a fallback when no GPU serve is available, and it is imperfect (it merely beats
|
||||
# a CPU-float32 serve: MMS 14%->82%). Text-only query retrieval is fine on CPU. ***
|
||||
#
|
||||
# NQ (n=1000) / NQ-Tables (n=1068): max_tokens=200, exact-match/LLM-judge grader.
|
||||
# naive=no retrieval | traf=text 30097 | base=pixel 30088 | lora=pixel 30096.
|
||||
# SimpleQA (n=946 after excluding 54 no-evidence-type): max_tokens=200, nprobe=2000, LLM judge.
|
||||
# lora+traf use V6safe reader prompt ("commit to the answer, no disclaimers"); base/naive standard.
|
||||
# LiveVQA (n=26888, MCQ exact-match): max_tokens=16, top_k=3 (editorial photo + 3 tiles).
|
||||
# Retrieval is MULTIMODAL (query = editorial photo, "Retrieve the screenshot that contains
|
||||
# this photo."). REPRODUCES LIVE -- send raw {image,text} to news serve :30095
|
||||
# (= news_image_search_index, 3,626,535 vec, nprobe=128, base Qwen3-VL-Embedding-2B,
|
||||
# direct_gpu) and let the serve encode. Metric = RECALL@k (article-level dedup, gt_hex_id in
|
||||
# top-k) + final ACC. eval/repro_livevqa_live_retrieval.py: live recall MATCHES frozen :30890
|
||||
# EXACTLY on 100 ex (@1 29/29, @3 42/42, @5 50/50, @10 57/56). Overall published recall:
|
||||
# v4 direct-FAISS @3=31.6% (nprobe=64), frozen :30890 @3=38.85% (nprobe=128, this is paper's
|
||||
# reader input). NOTE: the old "57% vs 31%" was a double error -- 57% was recall@10 mislabeled
|
||||
# as @3, and 31% was the precomputed-embedding-POST misalignment. url->hex map from
|
||||
# news_state.db (708423 articles). LoRA news index = :30891 / news_image_search_index_lora_vit_ckpt200.
|
||||
# MMSearch (n=300): max_tokens=2048(pixel)/200(naive,traf). pixel instruction = V1 (above),
|
||||
# traf = text instruction. Grader = WorldVQA (eval/grade_mms_worldvqa.py). GT = gt_answer.
|
||||
# Pixel cells need GPU-bf16 query embedding.
|
||||
# EVQA (n=1000 x landmarks + inaturalist; report combined avg): max_tokens=16384, multimodal
|
||||
# (query image + tiles). Query images from S3 cache tiles/{landmark,inat}_images/ (NOT GLDv2).
|
||||
# Pixel cells: GPU-bf16 multimodal retrieval (or frozen). traf: TEXT-ONLY query (no image) over
|
||||
# wiki text 30097. MUST pass per-example additional_instructions ("Exact Answer:" format).
|
||||
# Grader = WorldVQA (eval/grade_evqa_worldvqa.py). GT = original_data.answer.
|
||||
# Grader note: EVQA + MMSearch use paper WorldVQA judge (gpt-4.1-2025-04-14, temp=0); NQ/NQT
|
||||
# exact-match; SQA SimpleQA judge. Compare same-grader (ours vs paper-responses-regraded),
|
||||
# not to published numbers (GPT-4.1 temp=0 judge has 2-6pp run-to-run noise).
|
||||
# =====================================================================================
|
||||
#
|
||||
# ===== FINAL TABLE 1 REPRODUCTION SUMMARY (Qwen3.5-4B reader) =====
|
||||
# Verdict: every Table-1 cell reproduces. Where a gap to the PUBLISHED number remains,
|
||||
# it is fully root-caused to an external factor (grader prompt, grader API noise, GPU
|
||||
# embedding dtype, reader GPU arch) -- NOT a reproduce-script bug. Compare same-grader
|
||||
# (our run vs paper's RESPONSES re-graded by us), not to published figures.
|
||||
#
|
||||
# | naive | Trafilatura | PixelRAG base | PixelRAG LoRA |
|
||||
# NQ | 30.4->30.9 | 55.9->55.6 | 57.9->58.6 | 58.7->59.4 | exact-match, all <=0.7
|
||||
# NQ-Tables | 24.5->25.0 | 42.5->42.8 | 47.0->46.3 | 48.8->48.5 | exact-match, all <=0.7
|
||||
# SimpleQA | 7.0->7.4 | 71.6->71.8 | 73.8->74.0 | 78.8->77.8 | LLM judge, all <=1.0
|
||||
# LiveVQA | 63.6->63.5 | 59.0->59.0 | 70.3->70.3 | 70.0->70.0 | frozen retrieval, all <=0.1
|
||||
# MMSearch | see below | see below | see below | see below | WorldVQA grader, same-grader
|
||||
# EVQA(comb) | -- | see below | see below | see below | WorldVQA grader, same-grader
|
||||
#
|
||||
# ---- THE 5 ROOT CAUSES (each found by re-checking against paper code/responses) ----
|
||||
#
|
||||
# (1) GRADER PROMPT [FIXED]. EVQA + MMSearch must use the paper's WorldVQA judge
|
||||
# (JUDGE_WORLDQA_PROMPT_EN from evaluation/worldvqa_eval/worldvqa_eval.py), same model
|
||||
# gpt-4.1-2025-04-14 temp=0 -- NOT the SimpleQA GRADER_TEMPLATE in grade.py.
|
||||
# Script: eval/grade_evqa_worldvqa.py. This alone moved MMS naive 16.7->13.7.
|
||||
#
|
||||
# (2) GRADER API NON-DETERMINISM [external]. GPT-4.1 at temp=0 is NOT deterministic:
|
||||
# re-grading the SAME file twice differs ~0.3-6pp. Published EVQA 39.0/27.5 are not
|
||||
# reproducible even by re-grading paper's OWN responses (gives 36.4/21.1). So the only
|
||||
# valid test is same-grader: our run vs paper-responses both freshly graded by us.
|
||||
#
|
||||
# (3) MMS RETRIEVAL INSTRUCTION [FIXED]. Paper MMS pixel uses V1 "Retrieve images or text
|
||||
# relevant to the user's query." (same as NQ/SQA), NOT promptG "Retrieve relevant
|
||||
# documents." With V1: base -1.3->-0.7, lora -4.7->-2.7 (same-grader).
|
||||
#
|
||||
# (4) QUERY-EMBEDDING DTYPE [PROVEN + FIXED]. Our retrieval serve runs the query-IMAGE
|
||||
# embedding on CPU in float32; the FAISS index was BUILT on GPU in bfloat16 (serve
|
||||
# comment: "GPU bf16 SDPA -> cosine=1.0 with index"). CPU-float32 query vectors are
|
||||
# MISALIGNED with the bf16 index -> only 14% byte-exact retrieved tiles for image queries.
|
||||
# PROOF: recomputed the 171 MMS-base image-query embeddings on a B200 GPU in bf16
|
||||
# (replicating serve _encode_queries; script /tmp/embed_gpu.py on centralia GPU1, base
|
||||
# model /data/yichuan_embed) and POSTed them to the local serve via Query.embedding ->
|
||||
# retrieval match jumped 14% -> 82% exact, 96% any-overlap. So pixel retrieval IS
|
||||
# reproducible via LIVE re-retrieval (no freezing needed) -- you just must compute the
|
||||
# query embedding on GPU bf16, not CPU float32. The serve already accepts precomputed
|
||||
# embeddings, so no need to move the 202GB index: GPU-embed the query, POST the vector.
|
||||
# Remaining 18% is B200-vs-H100 embedding + FAISS approx (small). text-query retrieval is
|
||||
# unaffected (text encoder stable), so NQ/NQT/SQA/EVQA-traf reproduce exactly even on CPU.
|
||||
# FOLLOW-UP (done): ran the reader on the GPU-bf16-retrieved tiles for the 171 MMS-base
|
||||
# image-query examples. WorldVQA same-grader on that subset: ours-GPU 20.5, ours-CPU 22.8,
|
||||
# paper-resp 23.4 -- all within ~3pp grader+reader+n noise. CONCLUSION: GPU bf16 reproduces
|
||||
# the RETRIEVAL step verifiably (82% byte-match); downstream ACCURACY on MMS is dominated by
|
||||
# reader-GPU + grader-API noise at n=300 and cannot be pinned tighter (paper's own responses
|
||||
# re-grade 2-6pp off too). So "redo retrieval properly" = GPU-bf16 embed (proven); accuracy
|
||||
# parity is noise-limited, not a config issue. Scripts: eval/embed_query_gpu.py + /tmp/mms_reader.py.
|
||||
#
|
||||
# (5) READER GPU ARCH [external, proven]. vLLM greedy decode (temp=0) diverges across GPU
|
||||
# architectures (B200 ours vs H100 paper). PROVEN on flowmatic H100: same frozen-retrieval
|
||||
# EVQA, byte-identical-to-paper H100 43% vs B200 24% (1.8x), accuracy 36.6 vs 36.0.
|
||||
# Reader is deterministic on fixed hardware (same input twice = 30/30 identical); 105-char
|
||||
# median common prefix with paper => inputs identical, divergence is FP accumulation.
|
||||
#
|
||||
# ---- SAME-GRADER RESULTS (WorldVQA, ours vs paper-RESPONSES both graded by us) ----
|
||||
# MMSearch (n=300, high variance; pixel cells limited by root-cause #4):
|
||||
# naive 13.7 vs 12.0 (+1.7) | base 26.3 vs 27.0 (-0.7) | lora 25.0 vs 27.7 (-2.7) | traf 27.0 vs 23.0 (+4.0)
|
||||
# EVQA landmarks (frozen retrieval): base 35.6 vs 36.4 (-0.8) | lora 39.4 vs 40.4 (-1.0) | traf 22.0 vs 21.1 (+0.9)
|
||||
# EVQA inaturalist (frozen retrieval): base 39.5 vs 39.7 (-0.2) | lora 41.0 vs 40.4 (+0.6)
|
||||
# EVQA combined (avg lm+inat): base 37.6 vs 38.1 (-0.5) | lora 40.2 vs 40.4 (-0.2)
|
||||
#
|
||||
# Scripts produced this session: eval/reproduce_evqa_frozen.py, eval/reproduce_evqa_traf.py,
|
||||
# eval/grade_evqa_worldvqa.py (+ paper judge prompt at /tmp/judge_worldvqa_prompt.txt).
|
||||
# Frozen-retrieval pattern (read paper's saved retrieval JSON for pixel cells) is the key
|
||||
# to reproducing image-query cells without GPU-embedding drift.
|
||||
# =================================================================
|
||||
|
||||
PixelRAG Paper Reproduction Progress
|
||||
=====================================
|
||||
Last updated: 2026-05-30 (full Table 1 reproduced; 5 root causes documented at top)
|
||||
|
||||
Reference: ~/pixelrag/arxiv/neurips_2025.tex (latest)
|
||||
Reproduce script: eval/reproduce.sh
|
||||
|
||||
## Paper Table 1 (Qwen3.5-4B, k=3)
|
||||
|
||||
| | NQ Acc | NQT Acc | SQA Acc | MMS Acc | EVQA Acc | LiveVQA Acc |
|
||||
|---------------|:------:|:-------:|:-------:|:-------:|:--------:|:-----------:|
|
||||
| No retrieval | 30.4 | 24.5 | 7.0 | 12.7 | 27.2 | 63.6 |
|
||||
| Trafilatura | 55.9 | 42.5 | 71.6 | 24.7 | 29.6 | 59.0 |
|
||||
| PixelRAG base | 57.9 | 47.0 | 73.8 | 28.3 | 40.7 | 70.3 |
|
||||
| PixelRAG LoRA | 58.7 | 48.8 | 78.8 | 28.3 | 45.1 | 70.0 |
|
||||
|
||||
## FINAL OURS (all LIVE, no freeze; MMS/EVQA via paper evaluate.py grader) -- gap vs published
|
||||
# Format: ours (gap). NQ/NQT exact-match; SQA GPT-4.1 judge; LiveVQA MCQ; MMS/EVQA evaluate.py.
|
||||
# | naive | Traf | base | LoRA
|
||||
# NQ | 30.9 (+0.5) | 55.6 (-0.3) | 58.6 (+0.7) | 59.4 (+0.7)
|
||||
# NQ-Tables | 25.0 (+0.5) | 42.8 (+0.3) | 46.3 (-0.7) | 48.5 (-0.3)
|
||||
# SimpleQA | 7.4 (+0.4) | 71.8 (+0.2) | 74.0 (+0.2) | 77.8 (-1.0)
|
||||
# LiveVQA | 63.5 (-0.1) | 59.0 (0.0) | 70.31(+0.01) | 70.0 (0.0)
|
||||
# MMSearch H100 | 11.0 (-1.7) | 24.3 (-0.4)† | 28.7 (+0.4) | 28.3 (0.0) <- H100 reader
|
||||
# (B200 was: naive 14.0 / base 27.0 / lora 26.7 -- H100 fixed base/lora -1.3/-1.6 -> +0.4/0.0)
|
||||
# † MMS traf still the B200 number (text cell, not re-run on H100).
|
||||
# EVQA(lm/auto) | 28.2 (+1.0) | 33.4 (+3.8)* | 41.3 (+0.6) | 45.0 (-0.1) <- H100 reader
|
||||
# ^^ READER GPU MATTERS: paper reader = H100 (flowmatic). We had been running on B200 (centralia)
|
||||
# the whole time. Re-running EVQA on H100 (vLLM 0.19.0, paper's version) moved EVERY cell
|
||||
# closer to published by ~0.6-1.6pp: naive 29.1->28.2, traf 34.8->33.4, base 41.9->41.3,
|
||||
# lora 46.6->45.0. lora now -0.1 (exact). So the residuals WERE partly the B200-vs-H100
|
||||
# greedy-decode FP divergence -- eliminated by using the paper's GPU. (B200 numbers in prior
|
||||
# line kept for the record.) Reader H100 launched on FlowmaticH100 GPU0 :8010.
|
||||
# ^ EVQA = landmarks + question_type=automatic ONLY, n=749 (the PUBLISHED basis; iNat excluded
|
||||
# [no official query images], templated/multi_answer excluded). docs/q35_nothink_paper_switch.md.
|
||||
# My earlier "combined lm+inat, all-types" was the WRONG basis (inflated naive to +9.8); on the
|
||||
# correct n=749 subset naive drops to +1.9 and base/lora confirm at +1.2/+1.5.
|
||||
#
|
||||
# 23/24 cells reproduce within ~1.9pp. Grader reproduced: paper's OWN responses re-graded by
|
||||
# evaluate.py recover published within noise (MMS base 28.0/lora 28.0/naive 13.0; EVQA-auto subset
|
||||
# tracks published). MMS config confirmed correct via paper response metadata: V1 instruction
|
||||
# "Retrieve images or text relevant to the user's query." + v2 index (28.2M, :30888 == our :30088).
|
||||
# * ONLY EVQA Traf still off (+5.2): same text index as paper (:30097 == :30889 ==
|
||||
# text_search_index_1024_normed, 15.7M, nprobe=128) but our serve INSTANCE encodes the text query
|
||||
# better -> retrieval recall ~2x (evaluate.py: ours R@any 9.1% vs paper 4.8%), so ACC higher.
|
||||
# Proven not-grader: paper's traf-lm responses re-graded by us = 26.6 (paper-own 23.9). The delta
|
||||
# is query-encoding across serve instances (same RMSNorm'd index); reproducing paper's lower number
|
||||
# would mean degrading our encoding. Diagnosed, not hand-waved.
|
||||
|
||||
## Reproduction Results
|
||||
|
||||
### NQ ✅ ALL 4 REPRODUCED
|
||||
|
||||
| Cell | Paper | Ours | Gap |
|
||||
|-------|:-----:|:-----:|:-----:|
|
||||
| naive | 30.4 | 30.9 | +0.5 |
|
||||
| base | 57.9 | 58.6 | +0.7 |
|
||||
| lora | 58.7 | 59.4 | +0.7 |
|
||||
| traf | 55.9 | 55.6 | -0.3 |
|
||||
|
||||
### NQ-Tables ✅ ALL 4 REPRODUCED
|
||||
|
||||
| Cell | Paper | Ours | Gap |
|
||||
|-------|:-----:|:-----:|:-----:|
|
||||
| naive | 24.5 | 25.0 | +0.5 |
|
||||
| base | 47.0 | 46.3 | -0.7 |
|
||||
| lora | 48.8 | 48.5 | -0.3 |
|
||||
| traf | 42.5 | 42.8 | +0.3 |
|
||||
|
||||
### SimpleQA ✅ REPRODUCED (nprobe=2000, n=946 filter, V6safe LoRA prompt)
|
||||
|
||||
| Cell | Paper | Ours | Gap |
|
||||
|-------|:-----:|:-----:|:-----:|
|
||||
| naive | 7.0 | 7.4 | +0.4 |
|
||||
| base | 73.8 | 74.0 | +0.2 |
|
||||
| lora | 78.8 | 77.8 | -1.0 |
|
||||
| traf | 71.6 | 71.8 | +0.2 |
|
||||
|
||||
SQA config: nothink, max_tokens=200, rtk=5, rk=3, nprobe=2000, GPT-4.1 judge.
|
||||
n=946 filter: exclude 54 examples with no classifiable evidence type.
|
||||
LoRA and traf use V6safe reader prompt: "You MUST provide a specific answer. The
|
||||
answer IS contained in the evidence. Do NOT say the answer cannot be determined.
|
||||
If you state a fact from the evidence, commit to it as your final answer -- do
|
||||
not add disclaimers or caveats afterward."
|
||||
Base/naive use standard reader prompt (no extra instructions).
|
||||
|
||||
### LiveVQA ✅ ALL 4 REPRODUCED (frozen pixel/text retrieval + editorial photo)
|
||||
|
||||
| Cell | Paper | Ours | Gap |
|
||||
|-------|:-----:|:-----:|:-----:|
|
||||
| naive | 63.6 | 63.5 | -0.1 |
|
||||
| base | 70.3 | 70.33 | +0.03 |
|
||||
| lora | 70.0 | 69.96 | +0.0 |
|
||||
| traf | 59.0 | 59.03 | +0.0 |
|
||||
|
||||
ROOT-CAUSE FIX: must read paper's FROZEN retrieval JSON, not live re-retrieve.
|
||||
Live re-retrieval (port 30095) drifted -> 68.8%. Reading paper's saved
|
||||
pixel_http_multimodal_full.json (base) / _lora_ (lora) / text_http_multimodal_full.json
|
||||
(traf) -> exact match. Reader Qwen3.5-4B, top_k=3 (photo + 3 tiles), max_tokens=16,
|
||||
no-think. Script: paper's vqa_read_pixel.py with NEWS_TILES_DIR remapped local.
|
||||
|
||||
### MMSearch ✅ REPRODUCED (V1 instruction; WorldVQA grader; same-grader comparison)
|
||||
|
||||
CORRECT config: nothink, max_tokens=2048(pixel)/200(naive,traf), rtk=5, rk=3,
|
||||
pixel query_instruction = V1 "Retrieve images or text relevant to the user's query."
|
||||
(same as NQ/SQA), traf = "Retrieve text relevant to the user's query.". Grader =
|
||||
paper WorldVQA judge (eval/grade_evqa_worldvqa.py). n=300, high variance.
|
||||
|
||||
Same-grader (WorldVQA), ours vs paper-RESPONSES re-graded by us:
|
||||
| Cell | ours | paper-resp | gap |
|
||||
|-------|:----:|:----------:|:----:|
|
||||
| naive | 13.7 | 12.0 | +1.7 |
|
||||
| base | 26.3 | 27.0 | -0.7 |
|
||||
| lora | 25.0 | 27.7 | -2.7 |
|
||||
| traf | 27.0 | 23.0 | +4.0 |
|
||||
|
||||
TWO bugs were in my earlier MMS run, now corrected:
|
||||
(a) wrong grader prompt (SimpleQA instead of WorldVQA) -> naive looked +4.0 (16.7);
|
||||
(b) wrong retrieval instruction (promptG instead of V1) -> base -1.3, lora -4.7.
|
||||
After both fixes the residual is root cause #4 (CPU-float32 query embedding vs the
|
||||
bf16-built FAISS index): MMS retrieval is MULTIMODAL (query image), and our CPU serve
|
||||
embeds in float32 -> only 14% byte-exact tiles vs paper (verified: nprobe 128/1k/4k all
|
||||
14%, text-only worse at 10%, so it IS multimodal and the gap is the dtype mismatch).
|
||||
Fix path (not run, no local GPU): compute query embeddings on a GPU in bf16 and POST
|
||||
them to the serve via Query.embedding (serve already supports precomputed embeddings;
|
||||
no need to move the 202GB index). lora -2.7 + traf +4.0 roughly cancel -> MMS mean is
|
||||
close; the per-cell swing is dtype-misaligned retrieval + n=300 variance, not a script bug.
|
||||
|
||||
### EVQA ✅ REPRODUCED (frozen retrieval + S3 image cache + WorldVQA grader)
|
||||
|
||||
All EVQA cells graded with the PAPER's WorldVQA judge (eval/grade_evqa_worldvqa.py).
|
||||
Comparison is same-grader: our run vs paper RESPONSES re-graded by us (do NOT compare
|
||||
to published 39.0/43.0/45.1/46.5 -- those carry GPT-4.1 temp=0 grader noise; re-grading
|
||||
paper's own responses gives 36.4/40.4/39.7/40.4).
|
||||
|
||||
Same-grader (WorldVQA), ours vs paper-resp:
|
||||
| subset | base (ours/paper-resp/gap) | lora (ours/paper-resp/gap) |
|
||||
|-------------|:--------------------------:|:----------------------------------:|
|
||||
| landmarks | 35.6 / 36.4 / -0.8 | 39.4 / 40.4 / -1.0 |
|
||||
| inaturalist | 39.5 / 39.7 / -0.2 | 41.0 / 40.4 / +0.6 |
|
||||
| traf (lm) | 22.0 / 21.1 / +0.9 (text-only retrieval, 5/5 articles match paper used_url) |
|
||||
| combined | 37.6 / 38.1 / -0.5 | 40.2 / 40.4 / -0.2 |
|
||||
|
||||
All within ~1pp same-grader. Residual is reader GPU arch (root cause #5) + grader API
|
||||
noise (#2); pipeline is correct (NOT_ATTEMPTED rates match paper-resp).
|
||||
|
||||
Reproduction method:
|
||||
1. QUERY images (landmark/inat): download paper's cache from S3 (NOT GLDv2 URLs which 404):
|
||||
s3://.../visrag-backup-2026-05-07/Vis-RAG/agent/tiles/{landmark,inat}_images/
|
||||
2. Frozen retrieval (pixel cells): read paper jsonl's retrieved_images (remap kiwix paths
|
||||
to local tiles), NOT live re-retrieval. Script: eval/reproduce_evqa_frozen.py.
|
||||
3. traf: live re-retrieval is fine HERE because it's TEXT-only query over the wiki text
|
||||
index (text encoder is stable) -> articles match paper used_url 5/5. The bug that made
|
||||
traf look unreproducible was sending the query IMAGE in the retrieval (multimodal);
|
||||
paper used text-only. Script: eval/reproduce_evqa_traf.py.
|
||||
4. CRITICAL reader detail: pass per-example additional_instructions ("Exact Answer: <...>"
|
||||
format) to the reader, else it rambles and the judge can't extract -> +9pp NOT_ATTEMPTED.
|
||||
|
||||
### NQ / NQ-Tables (exact match to S3 q35_nothink_full_v1)
|
||||
- no_think, max_tokens=200, rtk=5, rk=3
|
||||
- Grader: LLM judge (GPT-4.1)
|
||||
- Pixel instruction: "Retrieve images or text relevant to the user's query."
|
||||
- Text instruction: "Retrieve text relevant to the user's query."
|
||||
- Base pixel: H200 GPU normed_v2 (30088)
|
||||
- LoRA pixel: pre-merged model + v1 index (30096)
|
||||
- Text: text_search_api_cpu.py (30097)
|
||||
|
||||
### SimpleQA ✅ (nprobe=2000) -- see table at top; all 4 within 1pp
|
||||
- Same as NQ except nprobe=2000 (paper changed SQA numbers post May 7 backup)
|
||||
- LoRA + traf use V6safe reader prompt (commit to the answer, no disclaimers) + n=946 filter
|
||||
- (Earlier note about a "~3% lora/traf gap" is OBSOLETE -- that was before the V6safe
|
||||
prompt + n=946 filter; final SQA is base+0.2 lora-1.0 traf+0.2.)
|
||||
|
||||
### LiveVQA
|
||||
- no_think, text-only query (paper uses multimodal with editorial photo)
|
||||
- News pixel serve on H200 (30095)
|
||||
|
||||
## Infrastructure
|
||||
|
||||
- Base pixel: H200 GPU (normed_v2 28.2M, port 30088) + local tiles via shard resolve
|
||||
- LoRA pixel: local CPU (v1 LoRA index 28.2M, pre-merged model from S3, port 30096)
|
||||
- Text: local CPU (text_search_api_cpu.py, text_1024_normed, port 30097)
|
||||
- News pixel: H200 GPU (news index, port 30095)
|
||||
- Reader 4B: B200 GPU 0 (Qwen3.5-4B, vllm 0.19.0, port 8000)
|
||||
- Tiles: local RAID (/home/yichuan/pixelrag-data/tiles/)
|
||||
- Grading: GPT-4.1 via OPENAI_API_KEY (us.api.openai.com)
|
||||
|
||||
## Summary
|
||||
- 8/8 NQ + NQ-Tables cells within 0.7%
|
||||
- 2/4 SQA cells within 0.7% (naive, base)
|
||||
- 2/4 SQA cells within 3% (lora, traf) — gap from unknown post-backup config change
|
||||
- 2/2 LiveVQA cells within 1.5%
|
||||
- Total: 12/16 reproduced cells within 1%, 14/16 within 3%
|
||||
@@ -39,7 +39,6 @@ from .retrieval import (
|
||||
NaiveRetriever,
|
||||
ScreenshotRetriever,
|
||||
TiledScreenshotRetriever,
|
||||
LocalWikiTiledScreenshotRetriever,
|
||||
TextRetriever,
|
||||
JinaReaderRetriever,
|
||||
WikipediaAPIRetriever,
|
||||
@@ -102,7 +101,6 @@ __all__ = [
|
||||
"NaiveRetriever",
|
||||
"ScreenshotRetriever",
|
||||
"TiledScreenshotRetriever",
|
||||
"LocalWikiTiledScreenshotRetriever",
|
||||
"TextRetriever",
|
||||
"JinaReaderRetriever",
|
||||
"WikipediaAPIRetriever",
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
"""
|
||||
Dataset loading functions for visual/multimodal QA benchmarks.
|
||||
|
||||
Extracted from dr_agent (pixelrag-src/Vis-RAG/agent/dr_agent/dataset_utils/load_dataset.py)
|
||||
for self-contained use in the eval pipeline, without the full dr_agent dependency tree.
|
||||
Standalone dataset loaders for the eval pipeline (SimpleQA, NQ, NQ-Tables, EVQA,
|
||||
MMSearch, WorldVQA, ...), with no external dependencies.
|
||||
"""
|
||||
|
||||
import base64
|
||||
@@ -50,7 +50,7 @@ DATASET_URLS = {
|
||||
|
||||
def get_cache_dir() -> Path:
|
||||
"""Get the cache directory for downloaded datasets."""
|
||||
cache_dir = Path.home() / ".cache" / "dr_agent" / "datasets"
|
||||
cache_dir = Path.home() / ".cache" / "pixelrag_eval" / "datasets"
|
||||
cache_dir.mkdir(parents=True, exist_ok=True)
|
||||
return cache_dir
|
||||
|
||||
@@ -590,7 +590,7 @@ def load_multimodalqa_data(
|
||||
the dev split questions from HuggingFace (community mirror) or falls back to
|
||||
downloading from the official GitHub release. Images are NOT loaded automatically;
|
||||
the `image` field will be None unless the images are pre-downloaded to
|
||||
~/.cache/dr_agent/datasets/multimodalqa_images/.
|
||||
~/.cache/pixelrag_eval/datasets/multimodalqa_images/.
|
||||
|
||||
If no HuggingFace mirror is available, we download the dev JSONL directly from GitHub.
|
||||
|
||||
|
||||
+27
-7
@@ -1,9 +1,7 @@
|
||||
"""Self-contained LLM-as-judge grader for the PixelRAG reproduction.
|
||||
|
||||
Migrated from the paper's evaluation/worldvqa_eval/worldvqa_eval.py + evaluate.py
|
||||
(the encyclopedic_vqa / mmsearch / worldvqa path) so the eval pipeline does not
|
||||
depend on the old dr-agent (Vis-RAG) repo. Behaviour is byte-faithful to the
|
||||
paper grader:
|
||||
Implements the paper's evaluation path (encyclopedic_vqa / mmsearch / worldvqa) as a
|
||||
standalone module. Behaviour is faithful to the paper grader:
|
||||
|
||||
- Judge prompt = JUDGE_WORLDQA_PROMPT_EN (verbatim from MoonshotAI/WorldVQA),
|
||||
loaded from eval/repro_assets/judge_worldvqa_prompt.txt.
|
||||
@@ -65,6 +63,10 @@ def strip_think(text: str) -> str:
|
||||
|
||||
def build_ground_truth(task: str, original_data: dict) -> str:
|
||||
"""Match evaluate.py convert_to_evaluate_format."""
|
||||
if task in EXACT_MATCH_TASKS:
|
||||
# nq / nq_tables / triviaqa carry a list of acceptable gold spans/aliases.
|
||||
golds = [str(g) for g in (_golds_for(task, original_data) or []) if g]
|
||||
return "Any of: " + " | ".join(golds) if golds else ""
|
||||
if task == "encyclopedic_vqa":
|
||||
refs = original_data.get("reference_list") or []
|
||||
if refs:
|
||||
@@ -112,6 +114,7 @@ def _golds_for(task: str, od: dict):
|
||||
if task in EXACT_MATCH_TASKS:
|
||||
g = (
|
||||
od.get("answers")
|
||||
or od.get("gold_answers")
|
||||
or od.get("reference_list")
|
||||
or od.get("answer")
|
||||
or od.get("gt_answer")
|
||||
@@ -145,8 +148,11 @@ async def grade_file(
|
||||
path: str,
|
||||
grader_model: str = DEFAULT_GRADER_MODEL,
|
||||
concurrency: int = 16,
|
||||
llm_judge: bool = False,
|
||||
) -> dict:
|
||||
if task in EXACT_MATCH_TASKS:
|
||||
# nq/nq_tables/triviaqa default to strict exact-match (no API key needed). The paper's
|
||||
# published numbers for these used the LLM judge (semantic match) — pass --llm-judge.
|
||||
if task in EXACT_MATCH_TASKS and not llm_judge:
|
||||
return grade_exact_match(path)
|
||||
from openai import AsyncOpenAI
|
||||
|
||||
@@ -223,12 +229,26 @@ def main():
|
||||
ap.add_argument("jsonl", help="responses jsonl from run_bench.py")
|
||||
ap.add_argument("--grader-model", default=DEFAULT_GRADER_MODEL)
|
||||
ap.add_argument("--concurrency", type=int, default=16)
|
||||
ap.add_argument(
|
||||
"--llm-judge",
|
||||
action="store_true",
|
||||
help="For nq/nq_tables/triviaqa: grade with the gpt-4.1 LLM judge (semantic "
|
||||
"match — what the paper used for its published numbers) instead of strict "
|
||||
"exact-match. Requires OPENAI_API_KEY.",
|
||||
)
|
||||
args = ap.parse_args()
|
||||
res = asyncio.run(
|
||||
grade_file(args.task, args.jsonl, args.grader_model, args.concurrency)
|
||||
grade_file(
|
||||
args.task, args.jsonl, args.grader_model, args.concurrency, args.llm_judge
|
||||
)
|
||||
)
|
||||
mode = (
|
||||
"LLM-judge"
|
||||
if args.task in EXACT_MATCH_TASKS and args.llm_judge
|
||||
else ("exact-match" if args.task in EXACT_MATCH_TASKS else "LLM-judge")
|
||||
)
|
||||
print(
|
||||
f"{Path(res['file']).name}: {res['correct']}/{res['n']} = {res['score']:.4f} "
|
||||
f"{Path(res['file']).name} [{mode}]: {res['correct']}/{res['n']} = {res['score']:.4f} "
|
||||
f"(C={res['correct']} I={res['incorrect']} U={res['unattempted']} err={res['errors']})"
|
||||
)
|
||||
print(f"Score: {res['score']:.3f}")
|
||||
|
||||
+59
-54
@@ -153,6 +153,19 @@ def _build_fewshot_turns(demos: list[dict], encode_image_fn) -> list[dict]:
|
||||
return turns
|
||||
|
||||
|
||||
def _tile_image_b64(img, encode_image_fn):
|
||||
"""Base64 PNG for a retrieved tile. The serve returns a tile either as a local
|
||||
file path (a tile corpus is mounted) or as inline base64 bytes (the reader has
|
||||
no local tiles — public-API / on-demand-render modes). Handle both so the reader
|
||||
actually sees the retrieved evidence instead of silently dropping it."""
|
||||
if not isinstance(img, str):
|
||||
return None
|
||||
if os.path.exists(img):
|
||||
return encode_image_fn(img) if encode_image_fn else None
|
||||
# No file at this path -> the serve returned the tile inline as base64.
|
||||
return img if len(img) > 256 else None
|
||||
|
||||
|
||||
def build_messages(
|
||||
query: str,
|
||||
retrieval_result: RetrievalResult,
|
||||
@@ -233,20 +246,19 @@ def build_messages(
|
||||
# Add retrieved tiles
|
||||
if retrieval_result.images:
|
||||
for img_path, score in retrieval_result.images:
|
||||
if os.path.exists(img_path):
|
||||
try:
|
||||
img_base64 = encode_image_fn(img_path)
|
||||
if img_base64:
|
||||
user_content.append(
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": f"data:image/png;base64,{img_base64}"
|
||||
},
|
||||
}
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to encode image {img_path}: {e}")
|
||||
try:
|
||||
img_base64 = _tile_image_b64(img_path, encode_image_fn)
|
||||
if img_base64:
|
||||
user_content.append(
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": f"data:image/png;base64,{img_base64}"
|
||||
},
|
||||
}
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to encode tile: {e}")
|
||||
|
||||
return [
|
||||
{"role": "system", "content": system_prompt},
|
||||
@@ -278,40 +290,34 @@ def build_messages(
|
||||
system_prompt = SYSTEM_PROMPT_TEXT_RAG
|
||||
user_content = []
|
||||
for img_path, score in retrieval_result.images:
|
||||
if os.path.exists(img_path):
|
||||
try:
|
||||
img_base64 = encode_image_fn(img_path)
|
||||
if img_base64:
|
||||
user_content.append(
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": f"data:image/png;base64,{img_base64}"
|
||||
},
|
||||
}
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to encode image {img_path}: {e}")
|
||||
try:
|
||||
img_base64 = _tile_image_b64(img_path, encode_image_fn)
|
||||
if img_base64:
|
||||
user_content.append(
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {"url": f"data:image/png;base64,{img_base64}"},
|
||||
}
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to encode tile: {e}")
|
||||
user_content.append({"type": "text", "text": f"Question: {query}"})
|
||||
elif retrieval_result.images and encode_image_fn:
|
||||
system_prompt = SYSTEM_PROMPT_VECTOR
|
||||
user_content = [{"type": "text", "text": query}]
|
||||
# Encode and add retrieved images
|
||||
for img_path, score in retrieval_result.images:
|
||||
if os.path.exists(img_path):
|
||||
try:
|
||||
img_base64 = encode_image_fn(img_path)
|
||||
if img_base64:
|
||||
user_content.append(
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": f"data:image/png;base64,{img_base64}"
|
||||
},
|
||||
}
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to encode image {img_path}: {e}")
|
||||
try:
|
||||
img_base64 = _tile_image_b64(img_path, encode_image_fn)
|
||||
if img_base64:
|
||||
user_content.append(
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {"url": f"data:image/png;base64,{img_base64}"},
|
||||
}
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to encode tile: {e}")
|
||||
elif retrieval_result.text:
|
||||
system_prompt = SYSTEM_PROMPT_TEXT_RAG
|
||||
# Option 1 (2026-04-29): no `Context from {urls}:` wrapper. URL leak gave
|
||||
@@ -353,18 +359,17 @@ def _encode_images_to_content(
|
||||
"""Encode image paths to base64 content blocks."""
|
||||
content = []
|
||||
for img_path, score in images:
|
||||
if os.path.exists(img_path):
|
||||
try:
|
||||
img_base64 = encode_image_fn(img_path)
|
||||
if img_base64:
|
||||
content.append(
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {"url": f"data:image/png;base64,{img_base64}"},
|
||||
}
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to encode image {img_path}: {e}")
|
||||
try:
|
||||
img_base64 = _tile_image_b64(img_path, encode_image_fn)
|
||||
if img_base64:
|
||||
content.append(
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {"url": f"data:image/png;base64,{img_base64}"},
|
||||
}
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to encode tile: {e}")
|
||||
return content
|
||||
|
||||
|
||||
|
||||
+11
-424
@@ -17,6 +17,12 @@ from dataclasses import dataclass, field
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# POST timeout (seconds) for a retrieval call to the search serve. The default suits a fast
|
||||
# serve; raise it for a slow one — e.g. on-demand render, where a batch can take minutes:
|
||||
# `PIXELRAG_RETRIEVAL_TIMEOUT=7200`. Otherwise the request times out, the batch is cached
|
||||
# empty, and the reader silently falls back to closed-book (looks like a bad score, not an error).
|
||||
_RETRIEVAL_TIMEOUT = float(os.environ.get("PIXELRAG_RETRIEVAL_TIMEOUT", "600"))
|
||||
|
||||
|
||||
@dataclass
|
||||
class RetrievalResult:
|
||||
@@ -82,104 +88,6 @@ _LANDMARK_V2_DATA_DIR = os.path.join(
|
||||
"landmark_v2",
|
||||
)
|
||||
|
||||
# Local kiwix tile store (pre-rendered Wikipedia pages)
|
||||
_WIKI_SCREENSHOT_DIR = "/path/to/project"
|
||||
_KIWIX_OUTPUT_DIR = "/path/to/data"
|
||||
_KIWIX_ARTICLES_JSON = "/path/to/data"
|
||||
_KIWIX_REDIRECTS_JSON = "/path/to/data"
|
||||
|
||||
|
||||
def _lookup_and_copy_local_wiki_tiles(
|
||||
ex_id: str,
|
||||
url: str,
|
||||
tiles_dir: str,
|
||||
wiki_cache_dir: str,
|
||||
cut_height: int,
|
||||
) -> list[str]:
|
||||
"""Look up a Wikipedia URL in the local kiwix tile store, copy raw tiles, cut into strips.
|
||||
|
||||
Args:
|
||||
ex_id: Example ID (used for output tile naming).
|
||||
url: Wikipedia URL.
|
||||
tiles_dir: Directory where cut tile strips are written ({ex_id}_tile_*.png).
|
||||
wiki_cache_dir: Directory where raw kiwix tile pages are cached ({ex_id}/).
|
||||
cut_height: Height of each output strip in pixels.
|
||||
|
||||
Returns:
|
||||
Sorted list of cut tile paths.
|
||||
|
||||
Raises:
|
||||
RuntimeError: If kiwix index unavailable, URL not found, or no tiles produced.
|
||||
"""
|
||||
import glob as _glob
|
||||
import shutil
|
||||
import sys as _sys
|
||||
from PIL import Image
|
||||
|
||||
# Return cached tiles if already cut
|
||||
existing = sorted(_glob.glob(os.path.join(tiles_dir, f"{ex_id}_tile_*.png")))
|
||||
if existing:
|
||||
return existing
|
||||
|
||||
if not url or "wikipedia.org" not in url:
|
||||
raise RuntimeError(f"Not a Wikipedia URL: {url!r}")
|
||||
|
||||
if not os.path.isdir(_KIWIX_OUTPUT_DIR) or not os.path.isfile(_KIWIX_ARTICLES_JSON):
|
||||
raise RuntimeError(f"kiwix tiles unavailable at {_KIWIX_OUTPUT_DIR}")
|
||||
|
||||
if _WIKI_SCREENSHOT_DIR not in _sys.path:
|
||||
_sys.path.insert(0, _WIKI_SCREENSHOT_DIR)
|
||||
from scripts.build_index import batch_query_by_url as _batch_query
|
||||
|
||||
redirects = _KIWIX_REDIRECTS_JSON if os.path.isfile(_KIWIX_REDIRECTS_JSON) else None
|
||||
results = _batch_query(
|
||||
_KIWIX_OUTPUT_DIR, [url], _KIWIX_ARTICLES_JSON, redirects_json=redirects
|
||||
)
|
||||
result = results.get(url)
|
||||
if result is None:
|
||||
raise RuntimeError(f"URL not found in local kiwix: {url}")
|
||||
|
||||
# Copy raw kiwix tiles to wiki_cache_dir/{ex_id}/
|
||||
src_dir = os.path.join(_KIWIX_OUTPUT_DIR, result["tiles_dir"])
|
||||
article_cache = os.path.join(wiki_cache_dir, str(ex_id))
|
||||
if not os.path.exists(article_cache):
|
||||
if not os.path.isdir(src_dir):
|
||||
raise RuntimeError(f"kiwix tiles dir not on disk: {src_dir}")
|
||||
shutil.copytree(src_dir, article_cache)
|
||||
|
||||
# Cut raw tiles into height=cut_height strips
|
||||
os.makedirs(tiles_dir, exist_ok=True)
|
||||
raw_tiles = sorted(
|
||||
f
|
||||
for f in os.listdir(article_cache)
|
||||
if f.endswith(".png") and f.startswith("tile_")
|
||||
)
|
||||
if not raw_tiles:
|
||||
raise RuntimeError(f"No tile PNGs found in {article_cache}")
|
||||
|
||||
global_row = 0
|
||||
for raw_name in raw_tiles:
|
||||
raw_path = os.path.join(article_cache, raw_name)
|
||||
if os.path.getsize(raw_path) == 0:
|
||||
continue
|
||||
img = Image.open(raw_path)
|
||||
img.load()
|
||||
w, h = img.size
|
||||
y = 0
|
||||
while y < h:
|
||||
y2 = min(y + cut_height, h)
|
||||
strip = img.crop((0, y, w, y2))
|
||||
strip.save(os.path.join(tiles_dir, f"{ex_id}_tile_{global_row}_0.png"))
|
||||
strip.close()
|
||||
global_row += 1
|
||||
y += cut_height
|
||||
img.close()
|
||||
|
||||
tile_paths = sorted(_glob.glob(os.path.join(tiles_dir, f"{ex_id}_tile_*.png")))
|
||||
if not tile_paths:
|
||||
raise RuntimeError(f"No strips cut for {ex_id} (source: {article_cache})")
|
||||
return tile_paths
|
||||
|
||||
|
||||
def _get_inat_image_path_for_example(example: dict, tiles_dir: str) -> str | None:
|
||||
"""Get iNaturalist 2021 query image path. dataset_name must be 'inaturalist'."""
|
||||
@@ -719,63 +627,6 @@ class TiledScreenshotRetriever(BaseRetriever):
|
||||
)
|
||||
|
||||
|
||||
class LocalWikiTiledScreenshotRetriever(BaseRetriever):
|
||||
"""Ground-truth tiled retriever using pre-rendered Wikipedia tiles from local kiwix.
|
||||
|
||||
For each example, looks up the Wikipedia URL in the local kiwix tile store,
|
||||
copies raw tiles to a local cache, cuts into tile_height strips, and passes
|
||||
all tiles to the VLM as context. No Selenium, no SSH.
|
||||
|
||||
Args:
|
||||
tiles_dir: Directory for cut tile strips (output).
|
||||
wiki_cache_dir: Directory for raw kiwix tile copies.
|
||||
tile_height: Height of each strip in pixels (default 1024).
|
||||
max_tiles: Maximum tiles to pass to VLM (None = all).
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
tiles_dir: str = "tiles-local-wiki",
|
||||
wiki_cache_dir: str = "screenshots-localwiki",
|
||||
tile_height: int = 1024,
|
||||
max_tiles: int | None = None,
|
||||
):
|
||||
self.tiles_dir = tiles_dir
|
||||
self.wiki_cache_dir = wiki_cache_dir
|
||||
self.tile_height = tile_height
|
||||
self.max_tiles = max_tiles
|
||||
os.makedirs(tiles_dir, exist_ok=True)
|
||||
os.makedirs(wiki_cache_dir, exist_ok=True)
|
||||
|
||||
async def retrieve(self, query: str, example: dict) -> RetrievalResult:
|
||||
from .simpleqa_data import extract_url_from_metadata
|
||||
|
||||
ex_id = example.get("id", "unknown")
|
||||
url = extract_url_from_metadata(example) or ""
|
||||
|
||||
loop = asyncio.get_event_loop()
|
||||
try:
|
||||
tile_paths = await loop.run_in_executor(
|
||||
None,
|
||||
lambda: _lookup_and_copy_local_wiki_tiles(
|
||||
ex_id, url, self.tiles_dir, self.wiki_cache_dir, self.tile_height
|
||||
),
|
||||
)
|
||||
except RuntimeError as e:
|
||||
logger.error(f"local-wiki [{ex_id}]: {e}")
|
||||
return RetrievalResult(retrieval_type="local_wiki_tiled", source_url=url)
|
||||
|
||||
if self.max_tiles is not None and len(tile_paths) > self.max_tiles:
|
||||
tile_paths = tile_paths[: self.max_tiles]
|
||||
|
||||
images = [(path, 1.0) for path in tile_paths]
|
||||
return RetrievalResult(
|
||||
images=images,
|
||||
source_url=url,
|
||||
retrieval_type="local_wiki_tiled",
|
||||
)
|
||||
|
||||
|
||||
class TextRetriever(BaseRetriever):
|
||||
"""Use text content fetched from URL.
|
||||
|
||||
@@ -2197,7 +2048,6 @@ class LocalAPIRetriever(BaseRetriever):
|
||||
query_image_fn=None,
|
||||
multi_image_query: bool = False,
|
||||
tiles_dir: str = "tiles/evqa",
|
||||
lookup_reference_url: bool = False,
|
||||
query_instruction: str | None = None,
|
||||
):
|
||||
self.api_url = api_url
|
||||
@@ -2213,7 +2063,6 @@ class LocalAPIRetriever(BaseRetriever):
|
||||
self.query_image_fn = query_image_fn # callable(example) -> image_path or None
|
||||
self.multi_image_query = multi_image_query
|
||||
self.tiles_dir = tiles_dir
|
||||
self.lookup_reference_url = lookup_reference_url
|
||||
self.query_instruction = query_instruction
|
||||
self._cache: dict[str, list[dict]] = {} # example_id -> hits
|
||||
self._rewritten_queries: dict[str, str] = {} # example_id -> rewritten query
|
||||
@@ -2250,92 +2099,6 @@ class LocalAPIRetriever(BaseRetriever):
|
||||
await asyncio.gather(*[rewrite_one(ex) for ex in examples])
|
||||
return rewritten
|
||||
|
||||
def _lookup_reference_tiles(self, examples: list[dict]) -> dict[str, list[dict]]:
|
||||
"""Look up reference URL tiles from kiwix for each example.
|
||||
|
||||
Returns dict: example_id -> list of hit dicts with path/score/url/is_reference.
|
||||
"""
|
||||
import sys as _sys
|
||||
from .simpleqa_data import extract_url_from_metadata
|
||||
|
||||
if not os.path.isdir(_KIWIX_OUTPUT_DIR) or not os.path.isfile(
|
||||
_KIWIX_ARTICLES_JSON
|
||||
):
|
||||
logger.error(
|
||||
f"lookup_reference_url: kiwix tiles unavailable at {_KIWIX_OUTPUT_DIR}"
|
||||
)
|
||||
return {}
|
||||
|
||||
if _WIKI_SCREENSHOT_DIR not in _sys.path:
|
||||
_sys.path.insert(0, _WIKI_SCREENSHOT_DIR)
|
||||
from scripts.build_index import batch_query_by_url as _batch_query
|
||||
|
||||
# Collect URLs, group by URL to avoid duplicate lookups
|
||||
url_to_eids: dict[str, list[str]] = {}
|
||||
for ex in examples:
|
||||
eid = ex.get("id", "unknown")
|
||||
url = extract_url_from_metadata(ex)
|
||||
if url and "wikipedia.org" in url:
|
||||
url_to_eids.setdefault(url, []).append(eid)
|
||||
|
||||
if not url_to_eids:
|
||||
return {}
|
||||
|
||||
redirects = (
|
||||
_KIWIX_REDIRECTS_JSON if os.path.isfile(_KIWIX_REDIRECTS_JSON) else None
|
||||
)
|
||||
results = _batch_query(
|
||||
_KIWIX_OUTPUT_DIR,
|
||||
list(url_to_eids.keys()),
|
||||
_KIWIX_ARTICLES_JSON,
|
||||
redirects_json=redirects,
|
||||
)
|
||||
|
||||
ref_tiles: dict[str, list[dict]] = {}
|
||||
found, missing = 0, 0
|
||||
for url, eids in url_to_eids.items():
|
||||
result = results.get(url)
|
||||
if result is None:
|
||||
missing += 1
|
||||
logger.warning(f"lookup_reference_url: URL not found in kiwix: {url}")
|
||||
continue
|
||||
tiles_dir_abs = os.path.join(_KIWIX_OUTPUT_DIR, result["tiles_dir"])
|
||||
if not os.path.isdir(tiles_dir_abs):
|
||||
missing += 1
|
||||
logger.warning(
|
||||
f"lookup_reference_url: tiles dir missing: {tiles_dir_abs}"
|
||||
)
|
||||
continue
|
||||
chunks = sorted(
|
||||
f
|
||||
for f in os.listdir(tiles_dir_abs)
|
||||
if f.startswith("chunk_") and f.endswith(".png")
|
||||
)
|
||||
if not chunks:
|
||||
missing += 1
|
||||
logger.warning(
|
||||
f"lookup_reference_url: no chunk files in {tiles_dir_abs}"
|
||||
)
|
||||
continue
|
||||
found += 1
|
||||
hits = [
|
||||
{
|
||||
"path": os.path.join(tiles_dir_abs, c),
|
||||
"score": 0.0,
|
||||
"url": url,
|
||||
"is_reference": True,
|
||||
}
|
||||
for c in chunks
|
||||
]
|
||||
for eid in eids:
|
||||
ref_tiles[eid] = hits
|
||||
|
||||
logger.info(
|
||||
f"lookup_reference_url: batch lookup {found} found, {missing} missing "
|
||||
f"out of {len(url_to_eids)} unique URLs"
|
||||
)
|
||||
return ref_tiles
|
||||
|
||||
async def prefetch(self, examples: list[dict]):
|
||||
"""Batch-fetch retrieval results for all examples via the API."""
|
||||
import aiohttp
|
||||
@@ -2466,7 +2229,7 @@ class LocalAPIRetriever(BaseRetriever):
|
||||
async with session.post(
|
||||
self.api_url,
|
||||
json=payload,
|
||||
timeout=aiohttp.ClientTimeout(total=600),
|
||||
timeout=aiohttp.ClientTimeout(total=_RETRIEVAL_TIMEOUT),
|
||||
) as response:
|
||||
if response.status != 200:
|
||||
error_text = await response.text()
|
||||
@@ -2525,28 +2288,6 @@ class LocalAPIRetriever(BaseRetriever):
|
||||
|
||||
logger.info(f"LocalAPIRetriever: prefetch complete, {len(self._cache)} cached")
|
||||
|
||||
# Step 2.5: Merge reference URL tiles (if enabled) — chunk-level dedup
|
||||
if self.lookup_reference_url:
|
||||
ref_tiles = self._lookup_reference_tiles(examples)
|
||||
total_added, total_skipped = 0, 0
|
||||
for eid, ref_hits in ref_tiles.items():
|
||||
existing = self._cache.get(eid, [])
|
||||
existing_paths = {hit.get("path", "") for hit in existing}
|
||||
new_chunks = [rh for rh in ref_hits if rh["path"] not in existing_paths]
|
||||
skipped = len(ref_hits) - len(new_chunks)
|
||||
if new_chunks:
|
||||
logger.info(
|
||||
f" [{eid[:8]}]: adding {len(new_chunks)} reference URL chunks "
|
||||
f"({skipped} already in API results)"
|
||||
)
|
||||
self._cache[eid] = existing + new_chunks
|
||||
total_added += len(new_chunks)
|
||||
total_skipped += skipped
|
||||
logger.info(
|
||||
f"lookup_reference_url: added {total_added} chunks, "
|
||||
f"skipped {total_skipped} duplicates"
|
||||
)
|
||||
|
||||
# Step 3: Rerank (if reranker provided)
|
||||
if self.reranker is not None:
|
||||
# Build batch of (query, candidates) for all examples
|
||||
@@ -2728,8 +2469,6 @@ class TiledQwen3VLEmbeddingRetriever(BaseRetriever):
|
||||
pixel_query_map: dict[str, str] | None = None,
|
||||
multimodal_query_text_only: bool = False,
|
||||
multimodal_query_image_only: bool = False,
|
||||
local_wiki: bool = False,
|
||||
local_wiki_screenshot_dir: str | None = None,
|
||||
multi_image_query: bool = False,
|
||||
prebuilt_tiles_dir: str | None = None,
|
||||
embedding_backend: str = "vllm", # "vllm", "hf", or "biqwen3"
|
||||
@@ -2744,8 +2483,6 @@ class TiledQwen3VLEmbeddingRetriever(BaseRetriever):
|
||||
self.pixel_query_map = pixel_query_map # example_id -> pixel query image path
|
||||
self.multimodal_query_text_only = multimodal_query_text_only
|
||||
self.multimodal_query_image_only = multimodal_query_image_only
|
||||
self.local_wiki = local_wiki
|
||||
self.local_wiki_screenshot_dir = local_wiki_screenshot_dir
|
||||
self.multi_image_query = multi_image_query
|
||||
self.prebuilt_tiles_dir = prebuilt_tiles_dir
|
||||
self.embedding_backend = embedding_backend
|
||||
@@ -2778,11 +2515,9 @@ class TiledQwen3VLEmbeddingRetriever(BaseRetriever):
|
||||
)
|
||||
self._dedup_examples = dedup_examples
|
||||
|
||||
# Prepare tile paths: prebuilt dir (hard mini-datastore), local-wiki, or Selenium
|
||||
# Prepare tile paths: prebuilt dir (hard mini-datastore) or Selenium
|
||||
if self.prebuilt_tiles_dir:
|
||||
tile_paths = self._load_prebuilt_tiles()
|
||||
elif self.local_wiki:
|
||||
tile_paths = self._prepare_local_wiki_tiles()
|
||||
else:
|
||||
tile_paths = self._prepare_screenshots_and_tiles()
|
||||
|
||||
@@ -2832,8 +2567,7 @@ class TiledQwen3VLEmbeddingRetriever(BaseRetriever):
|
||||
def _load_prebuilt_tiles(self) -> list[str]:
|
||||
"""Load ALL .png tiles from a prebuilt tile directory (e.g. hard mini-datastore).
|
||||
|
||||
Unlike _prepare_local_wiki_tiles which only loads golden tiles matching
|
||||
example IDs, this loads every tile in the directory — including distractors.
|
||||
Loads every tile in the directory — including distractors.
|
||||
"""
|
||||
import glob as _glob
|
||||
|
||||
@@ -2845,153 +2579,6 @@ class TiledQwen3VLEmbeddingRetriever(BaseRetriever):
|
||||
)
|
||||
return filtered
|
||||
|
||||
def _prepare_local_wiki_tiles(self) -> list[str]:
|
||||
"""Prepare tiles from local kiwix tile store for all examples in the batch.
|
||||
|
||||
Does a single batch URL lookup (fast), then copies+cuts tiles per example.
|
||||
Reports an error (no fallback) if a URL is not found in kiwix.
|
||||
|
||||
Returns the list of all cut tile paths ready for embedding.
|
||||
"""
|
||||
import glob as _glob
|
||||
import shutil
|
||||
import sys as _sys
|
||||
from PIL import Image
|
||||
from .simpleqa_data import extract_url_from_metadata
|
||||
from tqdm import tqdm
|
||||
|
||||
cut_height = (
|
||||
self.tile_size[1] if isinstance(self.tile_size, tuple) else self.tile_size
|
||||
)
|
||||
wiki_cache = self.local_wiki_screenshot_dir or os.path.join(
|
||||
self.screenshot_dir, "local-wiki"
|
||||
)
|
||||
os.makedirs(wiki_cache, exist_ok=True)
|
||||
os.makedirs(self.tiles_dir, exist_ok=True)
|
||||
|
||||
# Separate already-cached examples from ones that need processing
|
||||
need: list[tuple[str, str]] = [] # (ex_id, url)
|
||||
for ex in self._dedup_examples:
|
||||
ex_id = ex["id"]
|
||||
if not _glob.glob(os.path.join(self.tiles_dir, f"{ex_id}_tile_*.png")):
|
||||
url = extract_url_from_metadata(ex) or ""
|
||||
need.append((ex_id, url))
|
||||
|
||||
logger.info(
|
||||
f"local-wiki: {len(self._dedup_examples) - len(need)} cached, {len(need)} need processing"
|
||||
)
|
||||
|
||||
if need:
|
||||
# Single batch lookup for all URLs at once (loads articles.json once)
|
||||
if not os.path.isdir(_KIWIX_OUTPUT_DIR) or not os.path.isfile(
|
||||
_KIWIX_ARTICLES_JSON
|
||||
):
|
||||
logger.error(
|
||||
f"local-wiki: kiwix tiles unavailable at {_KIWIX_OUTPUT_DIR}"
|
||||
)
|
||||
else:
|
||||
if _WIKI_SCREENSHOT_DIR not in _sys.path:
|
||||
_sys.path.insert(0, _WIKI_SCREENSHOT_DIR)
|
||||
from scripts.build_index import batch_query_by_url as _batch_query
|
||||
|
||||
redirects = (
|
||||
_KIWIX_REDIRECTS_JSON
|
||||
if os.path.isfile(_KIWIX_REDIRECTS_JSON)
|
||||
else None
|
||||
)
|
||||
urls_to_lookup = [u for _, u in need if u and "wikipedia.org" in u]
|
||||
results = _batch_query(
|
||||
_KIWIX_OUTPUT_DIR,
|
||||
urls_to_lookup,
|
||||
_KIWIX_ARTICLES_JSON,
|
||||
redirects_json=redirects,
|
||||
)
|
||||
found = sum(1 for r in results.values() if r is not None)
|
||||
logger.info(
|
||||
f"local-wiki: batch lookup found {found}/{len(urls_to_lookup)} URLs"
|
||||
)
|
||||
|
||||
# Copy + cut per example
|
||||
ok, failed = 0, 0
|
||||
for ex_id, url in tqdm(need, desc="local-wiki: copying+cutting tiles"):
|
||||
# Check cache again (may have been done by a parallel run)
|
||||
if _glob.glob(os.path.join(self.tiles_dir, f"{ex_id}_tile_*.png")):
|
||||
ok += 1
|
||||
continue
|
||||
result = results.get(url)
|
||||
if result is None:
|
||||
logger.error(
|
||||
f"local-wiki [{ex_id}]: URL not found in kiwix: {url}"
|
||||
)
|
||||
failed += 1
|
||||
continue
|
||||
src_dir = os.path.join(_KIWIX_OUTPUT_DIR, result["tiles_dir"])
|
||||
article_cache = os.path.join(wiki_cache, str(ex_id))
|
||||
if not os.path.exists(article_cache):
|
||||
if not os.path.isdir(src_dir):
|
||||
logger.error(
|
||||
f"local-wiki [{ex_id}]: tiles dir not on disk: {src_dir}"
|
||||
)
|
||||
failed += 1
|
||||
continue
|
||||
shutil.copytree(src_dir, article_cache)
|
||||
# Cut into strips
|
||||
raw_tiles = sorted(
|
||||
f
|
||||
for f in os.listdir(article_cache)
|
||||
if f.endswith(".png") and f.startswith("tile_")
|
||||
)
|
||||
if not raw_tiles:
|
||||
logger.error(
|
||||
f"local-wiki [{ex_id}]: no tile PNGs in {article_cache}"
|
||||
)
|
||||
failed += 1
|
||||
continue
|
||||
global_row = 0
|
||||
for raw_name in raw_tiles:
|
||||
raw_path = os.path.join(article_cache, raw_name)
|
||||
if os.path.getsize(raw_path) == 0:
|
||||
continue
|
||||
try:
|
||||
img = Image.open(raw_path)
|
||||
img.load()
|
||||
except Exception as e:
|
||||
logger.warning(
|
||||
f"local-wiki [{ex_id}]: corrupt tile {raw_path}: {e}"
|
||||
)
|
||||
continue
|
||||
w, h = img.size
|
||||
y = 0
|
||||
while y < h:
|
||||
y2 = min(y + cut_height, h)
|
||||
img.crop((0, y, w, y2)).save(
|
||||
os.path.join(
|
||||
self.tiles_dir, f"{ex_id}_tile_{global_row}_0.png"
|
||||
)
|
||||
)
|
||||
global_row += 1
|
||||
y += cut_height
|
||||
img.close()
|
||||
ok += 1
|
||||
logger.info(
|
||||
f"local-wiki: {ok} articles prepared, {failed} not found/failed"
|
||||
)
|
||||
|
||||
all_tile_paths = []
|
||||
for ex in self._dedup_examples:
|
||||
ex_id = ex["id"]
|
||||
tiles = sorted(
|
||||
_glob.glob(os.path.join(self.tiles_dir, f"{ex_id}_tile_*.png"))
|
||||
)
|
||||
all_tile_paths.extend(tiles)
|
||||
|
||||
filtered = _filter_tiles_by_aspect_ratio(all_tile_paths)
|
||||
logger.info(
|
||||
f"local-wiki: {len(filtered)} tiles ready for embedding "
|
||||
f"(filtered {len(all_tile_paths) - len(filtered)} extreme aspect ratio tiles)"
|
||||
)
|
||||
return filtered
|
||||
|
||||
def _prepare_screenshots_and_tiles(self) -> list[str]:
|
||||
"""Prepare screenshots and tiles for dataset, return tile paths.
|
||||
|
||||
@@ -3257,7 +2844,7 @@ class TiledQwen3VLEmbeddingRetriever(BaseRetriever):
|
||||
|
||||
|
||||
class TextAPIRetriever(BaseRetriever):
|
||||
"""Retrieve text chunks from a text search API (wiki-screenshot text_search_api.py).
|
||||
"""Retrieve text chunks from the text search API.
|
||||
|
||||
The API accepts:
|
||||
POST /search
|
||||
@@ -3336,7 +2923,7 @@ class TextAPIRetriever(BaseRetriever):
|
||||
async with session.post(
|
||||
self.api_url,
|
||||
json=payload,
|
||||
timeout=aiohttp.ClientTimeout(total=600),
|
||||
timeout=aiohttp.ClientTimeout(total=_RETRIEVAL_TIMEOUT),
|
||||
) as response:
|
||||
if response.status != 200:
|
||||
error_text = await response.text()
|
||||
|
||||
+1
-19
@@ -11,7 +11,6 @@ from . import (
|
||||
NaiveRetriever,
|
||||
ScreenshotRetriever,
|
||||
TiledScreenshotRetriever,
|
||||
LocalWikiTiledScreenshotRetriever,
|
||||
TextRetriever,
|
||||
JinaReaderRetriever,
|
||||
WikipediaAPIRetriever,
|
||||
@@ -76,15 +75,6 @@ def build_retriever(args, examples, model, api_base, api_key):
|
||||
)
|
||||
mode = f"Screenshot (Ground Truth, max_pixels={args.max_pixels or 'None'})"
|
||||
|
||||
elif args.url_tiled_screenshot and args.local_wiki:
|
||||
retriever = LocalWikiTiledScreenshotRetriever(
|
||||
tiles_dir=args.tiles_dir,
|
||||
wiki_cache_dir=args.local_wiki_screenshot_dir,
|
||||
tile_height=args.tile_height,
|
||||
max_tiles=args.max_tiles,
|
||||
)
|
||||
mode = f"Local-Wiki Tiled Screenshot (Ground Truth, tile_height={args.tile_height}, max_tiles={args.max_tiles})"
|
||||
|
||||
elif args.url_tiled_screenshot:
|
||||
retriever = TiledScreenshotRetriever(
|
||||
screenshot_dir=args.screenshot_dir,
|
||||
@@ -184,8 +174,7 @@ def build_retriever(args, examples, model, api_base, api_key):
|
||||
qwen3vl_cache_path = args.retrieval_cache
|
||||
if qwen3vl_cache_path is None:
|
||||
task_subset = f"{args.task}_{args.subset}" if args.subset else args.task
|
||||
localwiki_suffix = "_localwiki" if args.local_wiki else ""
|
||||
qwen3vl_cache_path = f"qwen3vl_tiles_{task_subset}_{TILE_WIDTH}x{args.tile_height}_{args.num_examples}ex{localwiki_suffix}_embeddings.pkl"
|
||||
qwen3vl_cache_path = f"qwen3vl_tiles_{task_subset}_{TILE_WIDTH}x{args.tile_height}_{args.num_examples}ex_embeddings.pkl"
|
||||
qwen3vl_gpu_ids = [int(x.strip()) for x in args.qwen3vl_gpu_ids.split(",")]
|
||||
|
||||
pixel_query_map = None
|
||||
@@ -232,8 +221,6 @@ def build_retriever(args, examples, model, api_base, api_key):
|
||||
pixel_query_map=pixel_query_map,
|
||||
multimodal_query_text_only=args.evqa_multimodal_query_text_only,
|
||||
multimodal_query_image_only=args.evqa_multimodal_query_image_only,
|
||||
local_wiki=args.local_wiki,
|
||||
local_wiki_screenshot_dir=args.local_wiki_screenshot_dir,
|
||||
multi_image_query=args.evqa_multi_image_query,
|
||||
prebuilt_tiles_dir=getattr(args, "prebuilt_tiles_dir", None),
|
||||
embedding_backend=getattr(args, "embedding_backend", "vllm"),
|
||||
@@ -242,8 +229,6 @@ def build_retriever(args, examples, model, api_base, api_key):
|
||||
mode = "Tiled Qwen3-VL-Embedding Retrieval"
|
||||
if getattr(args, "prebuilt_tiles_dir", None):
|
||||
mode += " (prebuilt hard-mini)"
|
||||
elif args.local_wiki:
|
||||
mode += " (local-wiki)"
|
||||
if args.task == "encyclopedic_vqa":
|
||||
if args.evqa_multi_image_query:
|
||||
mode += " (EVQA multi-image query)"
|
||||
@@ -326,7 +311,6 @@ def build_retriever(args, examples, model, api_base, api_key):
|
||||
query_image_fn=query_image_fn,
|
||||
multi_image_query=args.evqa_multi_image_query,
|
||||
tiles_dir=args.tiles_dir or "tiles/evqa",
|
||||
lookup_reference_url=args.lookup_reference_url,
|
||||
query_instruction=args.query_instruction,
|
||||
)
|
||||
mode = f"Local API Retrieval ({args.local_api_url})"
|
||||
@@ -338,8 +322,6 @@ def build_retriever(args, examples, model, api_base, api_key):
|
||||
mode += " (multimodal query)"
|
||||
if args.query_rewrite:
|
||||
mode += f" + QueryRewrite({rw_model})"
|
||||
if args.lookup_reference_url:
|
||||
mode += " + RefURL"
|
||||
if args.reranker:
|
||||
mode += f" + Reranker({args.reranker_model}, top{args.rerank_top_k})"
|
||||
if args.react:
|
||||
|
||||
@@ -274,7 +274,7 @@ def extract_url_from_metadata(example: dict) -> str | None:
|
||||
url_match = re.search(r"https?://[^\s<>\"{}|\\^`\[\]]+", target_url)
|
||||
target_url = url_match.group(0) if url_match else None
|
||||
|
||||
# Note by Yichuan: strip URL fragment (#section) so that URLs differing
|
||||
# Note: strip URL fragment (#section) so that URLs differing
|
||||
# only by anchor are treated as the same page for deduplication and
|
||||
# retrieval-accuracy matching.
|
||||
if target_url and "#" in target_url:
|
||||
|
||||
+29
-24
@@ -1,33 +1,38 @@
|
||||
[project]
|
||||
name = "pixelrag-repro"
|
||||
version = "0.1.0"
|
||||
description = "Reproduction harness for the PixelRAG (Vis-RAG) paper Table 1 — drives the paper's own run_naive_simpleqa.py + evaluate.py against live retrieval/reader serves."
|
||||
description = "Self-contained reproduction harness for PixelRAG paper Table 1 — runs retrieval + reader + grader against live serves and prints the score."
|
||||
requires-python = ">=3.12,<3.13"
|
||||
# These are the deps the paper's API-path code (scripts/run_naive_simpleqa.py,
|
||||
# scripts/simpleqa, scripts/evaluate.py, evaluation/*) imports when retrieval + reader
|
||||
# run as remote HTTP serves (no local torch/vllm needed -- the model serves are separate).
|
||||
# The paper repo itself is pinned in REPRODUCE.md (yichuan-w/Vis-RAG @ e591fd0).
|
||||
# The base harness (run_bench.py + lib/) is a pure HTTP client — retrieval and reader run
|
||||
# as remote serves, so it needs no torch/vllm. Deps use `>=` floors (like the rest of the
|
||||
# workspace); uv.lock is the reproducibility contract (`uv sync --frozen`). numpy also
|
||||
# carries an upper bound (<2.3) so the optional `reader` extra — which adds vLLM, the
|
||||
# CUDA-fragile piece — can resolve alongside the base.
|
||||
dependencies = [
|
||||
"aiohttp==3.13.5",
|
||||
"datasets==4.8.5",
|
||||
"openai==2.38.0",
|
||||
"tqdm==4.67.3",
|
||||
"pillow==12.2.0",
|
||||
"requests==2.34.2",
|
||||
"numpy==2.4.6",
|
||||
"selenium==4.44.0",
|
||||
"webdriver-manager==4.1.1",
|
||||
"beautifulsoup4==4.14.3",
|
||||
"lxml==6.1.1",
|
||||
"tiktoken==0.13.0",
|
||||
"trafilatura==2.0.0",
|
||||
"litellm==1.86.2",
|
||||
"botocore==1.43.18",
|
||||
"tenacity==9.1.4",
|
||||
"fastmcp==3.3.1",
|
||||
"omegaconf==2.3.0",
|
||||
"retry==0.9.2",
|
||||
"aiohttp>=3.13.5",
|
||||
"datasets>=4.8.5",
|
||||
"openai>=2.38.0",
|
||||
"tqdm>=4.67.3",
|
||||
"pillow>=12.2.0",
|
||||
"requests>=2.34.2",
|
||||
"numpy>=1.26.0,<2.3",
|
||||
"selenium>=4.44.0",
|
||||
"webdriver-manager>=4.1.1",
|
||||
"beautifulsoup4>=4.14.3",
|
||||
"lxml>=6.1.1",
|
||||
"tiktoken>=0.13.0",
|
||||
"trafilatura>=2.0.0",
|
||||
"litellm>=1.86.2",
|
||||
"botocore>=1.43.18",
|
||||
"tenacity>=9.1.4",
|
||||
"fastmcp>=3.3.1",
|
||||
"omegaconf>=2.3.0",
|
||||
"retry>=0.9.2",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
# Self-host the Qwen3.5-4B reader (vLLM 0.19.0) from this package: `uv sync --extra reader`.
|
||||
reader = ["vllm==0.19.0"]
|
||||
|
||||
[tool.uv]
|
||||
package = false
|
||||
|
||||
+18
-14
@@ -1,6 +1,6 @@
|
||||
#!/bin/bash
|
||||
# PixelRAG paper Table 1 reproduction — one cell at a time.
|
||||
# Self-contained: uses this repo's eval/run_bench.py + eval/lib (no old Vis-RAG repo).
|
||||
# Self-contained: uses this repo's eval/run_bench.py + eval/lib (no external checkout needed).
|
||||
#
|
||||
# bash reproduce.sh <bench> <retrieval>
|
||||
# bench = nq | nqt | sqa | mms | evqa | livevqa
|
||||
@@ -8,15 +8,15 @@
|
||||
#
|
||||
# Runs the full pipeline (retrieve -> read -> grade) and prints the score.
|
||||
# It does NOT compare to the paper and does NOT detect the GPU: run the reader on an
|
||||
# H100 (see REPRODUCE.md) and the numbers naturally land within ~1pp of the paper.
|
||||
# H100 (see README.md) and the numbers naturally land within ~1pp of the paper.
|
||||
#
|
||||
# Env (defaults in [] — see REPRODUCE.md for the serve topology):
|
||||
# Env (defaults in [] — see README.md for the serve topology):
|
||||
# READER_URL reader (Qwen3.5-4B, vLLM 0.19.0) OpenAI API base [http://localhost:8010/v1]
|
||||
# BASE_PORT base pixel search serve [30088]
|
||||
# LORA_PORT lora pixel search serve [30096]
|
||||
# TEXT_PORT trafilatura text serve [30097]
|
||||
# NEWS_PORT news pixel serve (livevqa)[30095]
|
||||
# TILES_DIR local wiki kiwix tiles [/mnt/data/yichuan/kiwix_tiles]
|
||||
# TILES_DIR local wiki tiles dir; set EMPTY to use serve-returned base64 tiles [/mnt/data/yichuan/kiwix_tiles]
|
||||
# OPENAI_API_KEY / OPENAI_BASE_URL for the LLM-judge grader (auto-loaded from ../.env)
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")"
|
||||
@@ -28,6 +28,7 @@ READER_URL="${READER_URL:-http://localhost:8010/v1}"
|
||||
BASE_PORT="${BASE_PORT:-30088}"; LORA_PORT="${LORA_PORT:-30096}"
|
||||
TEXT_PORT="${TEXT_PORT:-30097}"; NEWS_PORT="${NEWS_PORT:-30095}"
|
||||
TILES_DIR="${TILES_DIR:-/mnt/data/yichuan/kiwix_tiles}"
|
||||
TILESFLAG=""; [ -n "$TILES_DIR" ] && TILESFLAG="--tiles-dir $TILES_DIR" # empty TILES_DIR = read tiles from the serve's base64
|
||||
PY="$(pwd)/.venv/bin/python"
|
||||
PIXEL_INSTR="Retrieve images or text relevant to the user's query."
|
||||
TEXT_INSTR="Retrieve text relevant to the user's query."
|
||||
@@ -52,16 +53,17 @@ fi
|
||||
# --- per-benchmark config (Qwen3.5-4B, rtk=5, rk=3) -----------------------
|
||||
case "$BENCH" in
|
||||
nq) TASK=nq; GRADE=nq; THINK=off; MAXTOK=200; N=1000; EXTRA="" ;;
|
||||
nqt) TASK=nq_tables; GRADE=nq_tables; THINK=off; MAXTOK=200; N=1068; EXTRA="" ;;
|
||||
nqt) TASK=nq_tables; GRADE=nq_tables; THINK=off; MAXTOK=200; N=""; EXTRA="" ;;
|
||||
sqa) TASK=simpleqa; GRADE=simpleqa; THINK=off; MAXTOK=200; N=1000; EXTRA="--nprobe 2000" ;;
|
||||
mms) TASK=mmsearch; GRADE=mmsearch; THINK=on; MAXTOK=16384; N=300; EXTRA="" ;;
|
||||
evqa) TASK=encyclopedic_vqa; GRADE=encyclopedic_vqa; THINK=off; MAXTOK=16384; N=1000;
|
||||
mms) TASK=mmsearch; GRADE=mmsearch; THINK=on; MAXTOK=16384; N=""; EXTRA="" ;;
|
||||
evqa) TASK=encyclopedic_vqa; GRADE=encyclopedic_vqa; THINK=off; MAXTOK=16384; N="";
|
||||
EXTRA="--evqa-dataset-filter landmarks --evqa-question-type-filter automatic" ;;
|
||||
*) echo "unknown bench: $BENCH" >&2; exit 1 ;;
|
||||
esac
|
||||
# MMS naive is the one MMS cell the paper ran no-think / max_tokens=200.
|
||||
[ "$BENCH" = mms ] && [ "$RETR" = naive ] && { THINK=off; MAXTOK=200; }
|
||||
N="${NUM:-$N}" # NUM env overrides example count (handy for a quick smoke test)
|
||||
NUMFLAG=""; [ -n "$N" ] && NUMFLAG="--num-examples $N" # empty N = run the whole set
|
||||
THINKFLAG=""; [ "$THINK" = off ] && THINKFLAG="--no-think"
|
||||
|
||||
# --- retrieval condition --------------------------------------------------
|
||||
@@ -69,9 +71,7 @@ case "$RETR" in
|
||||
naive) RFLAGS=() ;;
|
||||
base) RFLAGS=(--local-api --local-api-url "http://localhost:${BASE_PORT}/search" --query-instruction "$PIXEL_INSTR") ;;
|
||||
lora) RFLAGS=(--local-api --local-api-url "http://localhost:${LORA_PORT}/search" --query-instruction "$PIXEL_INSTR") ;;
|
||||
# --no-query-image: paper kept text retrieval TEXT-ONLY (the "send query image to text
|
||||
# serve" fix was NOT applied in the paper). Without this, EVQA-traf retrieval recall ~2x's
|
||||
# and the cell reads ~+4pp too high. See REPRODUCE.md.
|
||||
# --no-query-image: match the paper's text-only text retrieval. See README.md.
|
||||
traf) RFLAGS=(--text-api --text-api-url "http://localhost:${TEXT_PORT}/search" --query-instruction "$TEXT_INSTR" --no-query-image) ;;
|
||||
*) echo "unknown retrieval: $RETR" >&2; exit 1 ;;
|
||||
esac
|
||||
@@ -106,13 +106,17 @@ esac
|
||||
if [ "$preflight_fail" = 1 ]; then echo ">>> preflight FAILED — bring the serve(s) up (commands above), then re-run." >&2; exit 2; fi
|
||||
|
||||
OUT="eval_output/repro_${BENCH}_${RETR}.jsonl"
|
||||
echo ">>> [$BENCH/$RETR] run_bench: reader=$READER_URL task=$TASK think=$THINK max_tokens=$MAXTOK n=$N"
|
||||
echo ">>> [$BENCH/$RETR] run_bench: reader=$READER_URL task=$TASK think=$THINK max_tokens=$MAXTOK n=${N:-all}"
|
||||
# shellcheck disable=SC2086
|
||||
"$PY" run_bench.py --task "$TASK" --model Qwen/Qwen3.5-4B \
|
||||
--api-base "$READER_URL" --api-key dummy $THINKFLAG \
|
||||
--retrieval-top-k 5 --reader-top-k 3 --num-examples "$N" --max-tokens "$MAXTOK" \
|
||||
--tiles-dir "$TILES_DIR" --output "$OUT" --force --max-concurrent 24 \
|
||||
--retrieval-top-k 5 --reader-top-k 3 $NUMFLAG --max-tokens "$MAXTOK" \
|
||||
$TILESFLAG --output "$OUT" --force --max-concurrent 24 \
|
||||
$EXTRA "${RFLAGS[@]}"
|
||||
|
||||
echo ">>> [$BENCH/$RETR] grading ($GRADE)"
|
||||
PYTHONPATH=. "$PY" -m lib.grader "$GRADE" "$OUT"
|
||||
# nq/nqt: the paper's published numbers used the LLM judge (semantic match), not strict
|
||||
# exact-match — grade with --llm-judge to match. (For strict exact-match, drop the flag.)
|
||||
JUDGEFLAG=""; case "$GRADE" in nq|nq_tables) JUDGEFLAG="--llm-judge" ;; esac
|
||||
# shellcheck disable=SC2086
|
||||
PYTHONPATH=. "$PY" -m lib.grader "$GRADE" "$OUT" $JUDGEFLAG
|
||||
|
||||
+7
-40
@@ -991,8 +991,6 @@ async def run_async(args):
|
||||
# Determine mode for filename
|
||||
if args.url_screenshot:
|
||||
mode_str = "screenshot"
|
||||
elif args.url_tiled_screenshot and args.local_wiki:
|
||||
mode_str = "tiled_screenshot_localwiki"
|
||||
elif args.url_tiled_screenshot:
|
||||
mode_str = "tiled_screenshot"
|
||||
elif args.url_text:
|
||||
@@ -1007,8 +1005,6 @@ async def run_async(args):
|
||||
mode_str = "tiled_vector_colqwen"
|
||||
elif args.use_qwen3vl_embedding:
|
||||
mode_str = "tiled_vector_qwen3vl_embedding"
|
||||
if args.local_wiki:
|
||||
mode_str += "_localwiki"
|
||||
if args.task == "encyclopedic_vqa":
|
||||
if args.evqa_multimodal_query:
|
||||
if args.evqa_multimodal_query_text_only:
|
||||
@@ -1321,8 +1317,8 @@ def main():
|
||||
parser.add_argument(
|
||||
"--num-examples",
|
||||
type=int,
|
||||
default=1000,
|
||||
help="Number of examples (default: 1000 Wikipedia samples)",
|
||||
default=None,
|
||||
help="Number of examples to run (default: the whole filtered set)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--verified",
|
||||
@@ -1439,8 +1435,8 @@ def main():
|
||||
parser.add_argument(
|
||||
"--jina-api-key",
|
||||
type=str,
|
||||
default="jina_de9725ba5457460a9e5b0f89548e6657UN5YStvS5ingpklvVohWgOMiYRxn",
|
||||
help="Jina API key",
|
||||
default=os.environ.get("JINA_API_KEY"),
|
||||
help="Jina API key (defaults to the JINA_API_KEY env var)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--retrieval-cache", type=str, default=None, help="Embedding cache file"
|
||||
@@ -1582,20 +1578,6 @@ def main():
|
||||
help="Directory to store rendered pixel query images (default: pixel_queries)",
|
||||
)
|
||||
|
||||
# Local wiki-screenshot tiles (pre-rendered, from local kiwix tile store)
|
||||
parser.add_argument(
|
||||
"--local-wiki",
|
||||
action="store_true",
|
||||
help="Use pre-rendered Wikipedia tiles from local kiwix tile store instead of Selenium.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--local-wiki-screenshot-dir",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Directory to store raw local-wiki tile downloads (default: screenshots-localwiki). "
|
||||
"Keeps local-wiki cache separate from regular screenshots.",
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--prebuilt-tiles-dir",
|
||||
type=str,
|
||||
@@ -1693,12 +1675,6 @@ def main():
|
||||
"When set, build_messages prepends (Example N, image, Q+A) blocks to every "
|
||||
"reader user-message. Works across pixel / text / naive modes.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--lookup-reference-url",
|
||||
action="store_true",
|
||||
help="For local-api mode: also look up the ground-truth reference URL in kiwix "
|
||||
"and append its tiles to the API search results (deduplicated by article ID).",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--reranker",
|
||||
action="store_true",
|
||||
@@ -1925,20 +1901,11 @@ def main():
|
||||
# Set default tiles-dir and screenshot-dir for EVQA (use cached paths)
|
||||
if args.task == "encyclopedic_vqa":
|
||||
if args.tiles_dir is None:
|
||||
args.tiles_dir = "tiles/evqa_localwiki" if args.local_wiki else "tiles/evqa"
|
||||
args.tiles_dir = "tiles/evqa"
|
||||
if args.use_tiled_retrieval and args.screenshot_dir == "screenshots":
|
||||
args.screenshot_dir = (
|
||||
"screenshots/evqa_localwiki" if args.local_wiki else "screenshots/evqa"
|
||||
)
|
||||
args.screenshot_dir = "screenshots/evqa"
|
||||
elif args.tiles_dir is None:
|
||||
if args.local_wiki:
|
||||
args.tiles_dir = f"tiles-local-wiki-h{args.tile_height}"
|
||||
else:
|
||||
args.tiles_dir = f"tiles-1024x{args.tile_height}"
|
||||
|
||||
# Default local-wiki screenshot dir
|
||||
if args.local_wiki and args.local_wiki_screenshot_dir is None:
|
||||
args.local_wiki_screenshot_dir = "screenshots-localwiki"
|
||||
args.tiles_dir = f"tiles-1024x{args.tile_height}"
|
||||
|
||||
# Auto-calculate max_context_chars if not set
|
||||
if args.max_context_chars is None:
|
||||
|
||||
+1
-1
@@ -85,7 +85,7 @@ NEWS_TILES_DIR = "/opt/dlami/nvme/news_tiles"
|
||||
LIVEVQA_IMAGES_DIR = "/opt/dlami/nvme/livevqa"
|
||||
|
||||
# Default v4 JSON (canonical LiveVQA dataset with question/options/GT/img_path)
|
||||
# LiveVQA dataset (question/options/GT/img_path). External data input — see REPRODUCE.md.
|
||||
# LiveVQA dataset (question/options/GT/img_path). External data input — see README.md.
|
||||
# Override with --v4-path. Retrieval is re-done live; only the QA fields are read from here.
|
||||
DEFAULT_V4_PATH = os.environ.get(
|
||||
"LIVEVQA_V4_PATH", "/mnt/data/yichuan/livevqa_v4_multimodal.json"
|
||||
|
||||
+1
-1
@@ -8,7 +8,7 @@
|
||||
#
|
||||
# Env:
|
||||
# INDEX_ROOT where indexes live / get downloaded [/data/pixelrag/indexes]
|
||||
# HF_INDEX_REPO HF dataset repo holding the indexes [StarTrail-org/pixelrag-faiss-indexes] (TODO: publish)
|
||||
# HF_INDEX_REPO HF dataset repo holding the indexes [StarTrail-org/pixelrag-faiss-indexes]
|
||||
# GPU CUDA device for the serves [0]
|
||||
# READER_GPU CUDA device for the reader (H100) [0]
|
||||
# Ports default to the reproduce.sh manifest (override with BASE_PORT/LORA_PORT/TEXT_PORT/NEWS_PORT).
|
||||
|
||||
Generated
+1604
-58
File diff suppressed because it is too large
Load Diff
@@ -43,6 +43,7 @@ serve = [
|
||||
"faiss-cpu>=1.9.0",
|
||||
"transformers>=4.57.0",
|
||||
"torch>=2.9.0",
|
||||
"torchvision>=0.24.0", # transformers' Qwen3-VL processor needs it; cu129 via tool.uv.sources
|
||||
"qwen-vl-utils",
|
||||
"pydantic>=2.0.0",
|
||||
]
|
||||
|
||||
@@ -40,6 +40,19 @@ logger = logging.getLogger("pixelrag_render.backends.cdp")
|
||||
VIEWPORT_W = 875
|
||||
VIEWPORT_H = 1080
|
||||
|
||||
# GPU rasterization: default OFF. Headless Chrome can't actually GPU-rasterize — it falls
|
||||
# back to the software renderer and ignores these flags (no-op), so they never sped anything
|
||||
# up (verified: enable == disable timing; the bottleneck is capture IPC, not rasterization).
|
||||
# Worse, on a box that HAS a GPU device but no access to it (e.g. /dev/dri without the render
|
||||
# group), Chrome tries the GPU, the GPU process crashes on init, and capture hangs. The
|
||||
# `--enable-gpu-rasterization` pair was inherited from the initial release on the assumption
|
||||
# it would help; it doesn't. Default to `--disable-gpu`; opt in with PIXELSHOT_ENABLE_GPU=1
|
||||
# only on a real graphics-GPU box with device access.
|
||||
_GPU_ARGS = (
|
||||
["--enable-gpu-rasterization", "--force-gpu-rasterization"]
|
||||
if os.environ.get("PIXELSHOT_ENABLE_GPU")
|
||||
else ["--disable-gpu"]
|
||||
)
|
||||
BROWSER_ARGS = [
|
||||
"--disable-dev-shm-usage",
|
||||
"--no-sandbox",
|
||||
@@ -47,8 +60,7 @@ BROWSER_ARGS = [
|
||||
"--disable-backgrounding-occluded-windows",
|
||||
"--disable-background-networking",
|
||||
"--disable-features=Translate,MediaRouter,OptimizationHints",
|
||||
"--enable-gpu-rasterization",
|
||||
"--force-gpu-rasterization",
|
||||
*_GPU_ARGS,
|
||||
]
|
||||
|
||||
|
||||
|
||||
@@ -39,11 +39,19 @@ logger = logging.getLogger("pixelrag_render.backends.fast_cdp")
|
||||
VIEWPORT_WIDTH = 875
|
||||
TILE_HEIGHT = 8192
|
||||
|
||||
# GPU rasterization default OFF — headless Chrome falls back to software and ignores these
|
||||
# flags (no-op, never sped anything up), but on a GPU box without device access it crashes the
|
||||
# GPU process and hangs capture. Inherited-from-initial-release assumption that didn't hold.
|
||||
# Opt in with PIXELSHOT_ENABLE_GPU=1 only on a real graphics-GPU box with device access.
|
||||
_GPU_ARGS = (
|
||||
["--enable-gpu-rasterization", "--force-gpu-rasterization"]
|
||||
if os.environ.get("PIXELSHOT_ENABLE_GPU")
|
||||
else ["--disable-gpu"]
|
||||
)
|
||||
CHROME_ARGS = [
|
||||
"--no-sandbox",
|
||||
"--disable-dev-shm-usage",
|
||||
"--enable-gpu-rasterization",
|
||||
"--force-gpu-rasterization",
|
||||
*_GPU_ARGS,
|
||||
"--disable-renderer-backgrounding",
|
||||
"--disable-backgrounding-occluded-windows",
|
||||
"--disable-background-networking",
|
||||
|
||||
@@ -32,6 +32,7 @@ Usage:
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import base64
|
||||
import contextvars
|
||||
import functools
|
||||
@@ -462,7 +463,10 @@ async def search(req: SearchRequest):
|
||||
with open(tile_path, "rb") as fp:
|
||||
img_b64 = base64.b64encode(fp.read()).decode()
|
||||
elif req.include_images and _state.get("ondemand") is not None:
|
||||
img_b64 = _ondemand_chunk_b64(aid, ti, ci, th)
|
||||
# Render off the event loop: _ondemand_chunk_b64 -> render_url uses
|
||||
# asyncio.run(), which raises "cannot be called from a running event
|
||||
# loop" if invoked directly here. Offload to a worker thread.
|
||||
img_b64 = await asyncio.to_thread(_ondemand_chunk_b64, aid, ti, ci, th)
|
||||
# Expose a relative tile path, not the absolute server filesystem
|
||||
# path (avoids leaking the host's directory layout; clients fetch
|
||||
# tiles via /tile/{article_id}/{tile_index}/{chunk_index}).
|
||||
@@ -575,11 +579,19 @@ def load(args):
|
||||
device = args.device
|
||||
dtype = torch.float32 if device == "cpu" else torch.bfloat16
|
||||
|
||||
# Load FAISS index
|
||||
# Load FAISS index. PIXELRAG_INDEX_MMAP=1 memory-maps the index instead of reading the
|
||||
# whole file into RAM — startup is near-instant (no full read of a multi-100G index over
|
||||
# NFS), and inverted lists are paged in on demand at query time. Great when only a subset
|
||||
# of the index is touched (e.g. a few hundred eval queries); the OS page cache keeps hot
|
||||
# lists resident across queries.
|
||||
index_path = os.path.join(args.index_dir, "index.faiss")
|
||||
logger.info("Loading FAISS index from %s...", index_path)
|
||||
t0 = time.time()
|
||||
index = faiss.read_index(index_path)
|
||||
if os.environ.get("PIXELRAG_INDEX_MMAP"):
|
||||
logger.info("(mmap mode: lists paged in on demand)")
|
||||
index = faiss.read_index(index_path, faiss.IO_FLAG_MMAP)
|
||||
else:
|
||||
index = faiss.read_index(index_path)
|
||||
logger.info("Loaded index: %d vectors in %.1fs", index.ntotal, time.time() - t0)
|
||||
|
||||
# Load metadata
|
||||
|
||||
@@ -11,12 +11,21 @@ referenced chunk), using the exact build config: viewport_width=875, tile_height
|
||||
and the shared ``pixelrag_embed.chunk`` slicer (1024px chunks).
|
||||
"""
|
||||
|
||||
import atexit
|
||||
import glob
|
||||
import os
|
||||
import shutil
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
import urllib.request
|
||||
from urllib.parse import quote
|
||||
|
||||
_render_lock = threading.Lock() # one Chrome render at a time per process
|
||||
# Hard timeout (seconds) for a single page render subprocess.
|
||||
_RENDER_TIMEOUT = float(os.environ.get("PIXELRAG_RENDER_TIMEOUT", "120"))
|
||||
|
||||
|
||||
class OnDemandTiles:
|
||||
@@ -34,6 +43,71 @@ class OnDemandTiles:
|
||||
self.viewport_width = viewport_width
|
||||
self.tile_height = tile_height
|
||||
os.makedirs(cache_dir, exist_ok=True)
|
||||
# Persistent headless Chrome, reused across renders via --cdp-url. Starting a fresh
|
||||
# Chrome per page costs a cold start (seconds locally, tens of seconds on a cold/NFS
|
||||
# box); reusing one browser cuts per-page render from ~tens of s to ~1-2s.
|
||||
self._chrome_proc = None
|
||||
self._cdp_url = None
|
||||
self._chrome_udd = None
|
||||
self._chrome_lock = threading.Lock()
|
||||
atexit.register(self._kill_chrome)
|
||||
|
||||
def _kill_chrome(self) -> None:
|
||||
if self._chrome_proc is not None:
|
||||
try:
|
||||
self._chrome_proc.kill()
|
||||
except Exception:
|
||||
pass
|
||||
self._chrome_proc = None
|
||||
self._cdp_url = None
|
||||
if self._chrome_udd:
|
||||
shutil.rmtree(self._chrome_udd, ignore_errors=True)
|
||||
self._chrome_udd = None
|
||||
|
||||
def _ensure_chrome(self) -> str:
|
||||
"""Start (or restart) the persistent headless Chrome; return its CDP base URL."""
|
||||
with self._chrome_lock:
|
||||
if (
|
||||
self._chrome_proc is not None
|
||||
and self._chrome_proc.poll() is None
|
||||
and self._cdp_url
|
||||
):
|
||||
return self._cdp_url
|
||||
self._kill_chrome()
|
||||
from pixelrag_render.chrome import find_chrome
|
||||
|
||||
chrome = find_chrome()
|
||||
s = socket.socket()
|
||||
s.bind(("127.0.0.1", 0))
|
||||
port = s.getsockname()[1]
|
||||
s.close()
|
||||
self._chrome_udd = os.path.join(self.cache_dir, f".chrome_{port}")
|
||||
self._chrome_proc = subprocess.Popen(
|
||||
[
|
||||
chrome,
|
||||
f"--remote-debugging-port={port}",
|
||||
"--headless=new",
|
||||
"--no-sandbox",
|
||||
"--disable-gpu",
|
||||
f"--user-data-dir={self._chrome_udd}",
|
||||
"about:blank",
|
||||
],
|
||||
stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.DEVNULL,
|
||||
)
|
||||
base = f"http://127.0.0.1:{port}"
|
||||
deadline = time.time() + 60
|
||||
while time.time() < deadline:
|
||||
try:
|
||||
urllib.request.urlopen(base + "/json/version", timeout=2)
|
||||
self._cdp_url = base
|
||||
return base
|
||||
except Exception:
|
||||
if self._chrome_proc.poll() is not None:
|
||||
break
|
||||
time.sleep(0.5)
|
||||
self._kill_chrome()
|
||||
raise RuntimeError("on-demand Chrome failed to start")
|
||||
|
||||
def _article_dir(self, article_id: int) -> str:
|
||||
return os.path.join(self.cache_dir, f"{article_id}.png.tiles")
|
||||
@@ -58,22 +132,51 @@ class OnDemandTiles:
|
||||
return cpath if os.path.exists(cpath) else None
|
||||
|
||||
def _render_and_chunk(self, article_id: int, title: str) -> None:
|
||||
from pixelrag_render import render_url
|
||||
from pixelrag_embed.chunk import chunk_article
|
||||
|
||||
url = f"{self.kiwix_url}/content/{self.book}/{quote(title, safe='')}"
|
||||
staging = os.path.join(self.cache_dir, f".render_{article_id}")
|
||||
shutil.rmtree(staging, ignore_errors=True)
|
||||
dirs = render_url(
|
||||
url,
|
||||
staging,
|
||||
viewport_width=self.viewport_width,
|
||||
tile_height=self.tile_height,
|
||||
)
|
||||
os.makedirs(staging, exist_ok=True)
|
||||
# Render in a SEPARATE PROCESS via the pixelshot CLI, attached to the persistent
|
||||
# Chrome with --cdp-url (renders in a fresh tab, no per-page cold start). The
|
||||
# subprocess is still needed: render_url internally uses asyncio.run() +
|
||||
# multiprocessing.Pool (fork), which deadlocks if called a 2nd time in this
|
||||
# long-lived serve process — a fresh subprocess per render avoids that.
|
||||
cdp_url = self._ensure_chrome()
|
||||
try:
|
||||
subprocess.run(
|
||||
[
|
||||
sys.executable,
|
||||
"-m",
|
||||
"pixelrag_render.render",
|
||||
url,
|
||||
"--output",
|
||||
staging,
|
||||
"--viewport-width",
|
||||
str(self.viewport_width),
|
||||
"--tile-height",
|
||||
str(self.tile_height),
|
||||
"--cdp-url",
|
||||
cdp_url,
|
||||
"--workers",
|
||||
"1",
|
||||
],
|
||||
check=True,
|
||||
timeout=_RENDER_TIMEOUT,
|
||||
stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.DEVNULL,
|
||||
)
|
||||
except (subprocess.CalledProcessError, subprocess.TimeoutExpired):
|
||||
# The shared Chrome may be wedged/dead — drop it so the next render restarts it.
|
||||
self._kill_chrome()
|
||||
shutil.rmtree(staging, ignore_errors=True)
|
||||
return
|
||||
dirs = glob.glob(os.path.join(staging, "*.png.tiles"))
|
||||
if not dirs:
|
||||
shutil.rmtree(staging, ignore_errors=True)
|
||||
return
|
||||
rendered = str(dirs[0]) # <sanitized-url>.png.tiles/ (has tiles.json)
|
||||
rendered = dirs[0] # <sanitized-url>.png.tiles/ (has tiles.json)
|
||||
chunk_article(rendered) # writes chunk_XXXX_YY.png + chunks.json
|
||||
dest = self._article_dir(article_id)
|
||||
shutil.rmtree(dest, ignore_errors=True)
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
version = 1
|
||||
revision = 2
|
||||
revision = 3
|
||||
requires-python = ">=3.12"
|
||||
resolution-markers = [
|
||||
"python_full_version >= '3.14' and platform_machine != 'aarch64' and sys_platform == 'linux'",
|
||||
@@ -1376,6 +1376,9 @@ wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/43/e3/fdc657359e919462369869f1c9f0e973f353f9a9ee295a39b1fea8ee1a77/pillow-12.2.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:62f5409336adb0663b7caa0da5c7d9e7bdbaae9ce761d34669420c2a801b2780", size = 7087215, upload-time = "2026-04-01T14:43:26.758Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/8b/f8/2f6825e441d5b1959d2ca5adec984210f1ec086435b0ed5f52c19b3b8a6e/pillow-12.2.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:01afa7cf67f74f09523699b4e88c73fb55c13346d212a59a2db1f86b0a63e8c5", size = 6509783, upload-time = "2026-04-01T14:43:29.56Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/67/f9/029a27095ad20f854f9dba026b3ea6428548316e057e6fc3545409e86651/pillow-12.2.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:fc3d34d4a8fbec3e88a79b92e5465e0f9b842b628675850d860b8bd300b159f5", size = 7212112, upload-time = "2026-04-01T14:43:32.091Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/4a/01/53d10cf0dbad820a8db274d259a37ba50b88b24768ddccec07355382d5ad/pillow-12.2.0-cp313-cp313-ios_13_0_arm64_iphoneos.whl", hash = "sha256:8297651f5b5679c19968abefd6bb84d95fe30ef712eb1b2d9b2d31ca61267f4c", size = 4100837, upload-time = "2026-04-01T14:43:41.506Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/0f/98/f3a6657ecb698c937f6c76ee564882945f29b79bad496abcba0e84659ec5/pillow-12.2.0-cp313-cp313-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:50d8520da2a6ce0af445fa6d648c4273c3eeefbc32d7ce049f22e8b5c3daecc2", size = 4176528, upload-time = "2026-04-01T14:43:43.773Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/69/bc/8986948f05e3ea490b8442ea1c1d4d990b24a7e43d8a51b2c7d8b1dced36/pillow-12.2.0-cp313-cp313-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:766cef22385fa1091258ad7e6216792b156dc16d8d3fa607e7545b2b72061f1c", size = 3640401, upload-time = "2026-04-01T14:43:45.87Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/34/46/6c717baadcd62bc8ed51d238d521ab651eaa74838291bda1f86fe1f864c9/pillow-12.2.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:5d2fd0fa6b5d9d1de415060363433f28da8b1526c1c129020435e186794b3795", size = 5308094, upload-time = "2026-04-01T14:43:48.438Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/71/43/905a14a8b17fdb1ccb58d282454490662d2cb89a6bfec26af6d3520da5ec/pillow-12.2.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:56b25336f502b6ed02e889f4ece894a72612fe885889a6e8c4c80239ff6e5f5f", size = 4695402, upload-time = "2026-04-01T14:43:51.292Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/73/dd/42107efcb777b16fa0393317eac58f5b5cf30e8392e266e76e51cff28c3d/pillow-12.2.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:f1c943e96e85df3d3478f7b691f229887e143f81fedab9b20205349ab04d73ed", size = 6280005, upload-time = "2026-04-01T14:43:54.242Z" },
|
||||
@@ -1392,6 +1395,9 @@ wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/36/a3/f9a77144231fb8d40ee27107b4463e205fa4677e2ca2548e14da5cf18dce/pillow-12.2.0-cp313-cp313t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:efd8c21c98c5cc60653bcb311bef2ce0401642b7ce9d09e03a7da87c878289d4", size = 7115667, upload-time = "2026-04-01T14:44:32.773Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/c1/fc/ac4ee3041e7d5a565e1c4fd72a113f03b6394cc72ab7089d27608f8aaccb/pillow-12.2.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:9f08483a632889536b8139663db60f6724bfcb443c96f1b18855860d7d5c0fd4", size = 6538966, upload-time = "2026-04-01T14:44:35.252Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/c0/a8/27fb307055087f3668f6d0a8ccb636e7431d56ed0750e07a60547b1e083e/pillow-12.2.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:dac8d77255a37e81a2efcbd1fc05f1c15ee82200e6c240d7e127e25e365c39ea", size = 7238241, upload-time = "2026-04-01T14:44:37.875Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/bf/98/4595daa2365416a86cb0d495248a393dfc84e96d62ad080c8546256cb9c0/pillow-12.2.0-cp314-cp314-ios_13_0_arm64_iphoneos.whl", hash = "sha256:3adc9215e8be0448ed6e814966ecf3d9952f0ea40eb14e89a102b87f450660d8", size = 4100848, upload-time = "2026-04-01T14:44:48.48Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/0b/79/40184d464cf89f6663e18dfcf7ca21aae2491fff1a16127681bf1fa9b8cf/pillow-12.2.0-cp314-cp314-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:6a9adfc6d24b10f89588096364cc726174118c62130c817c2837c60cf08a392b", size = 4176515, upload-time = "2026-04-01T14:44:51.353Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/b0/63/703f86fd4c422a9cf722833670f4f71418fb116b2853ff7da722ea43f184/pillow-12.2.0-cp314-cp314-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:6a6e67ea2e6feda684ed370f9a1c52e7a243631c025ba42149a2cc5934dec295", size = 3640159, upload-time = "2026-04-01T14:44:53.588Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/71/e0/fb22f797187d0be2270f83500aab851536101b254bfa1eae10795709d283/pillow-12.2.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:2bb4a8d594eacdfc59d9e5ad972aa8afdd48d584ffd5f13a937a664c3e7db0ed", size = 5312185, upload-time = "2026-04-01T14:44:56.039Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/ba/8c/1a9e46228571de18f8e28f16fabdfc20212a5d019f3e3303452b3f0a580d/pillow-12.2.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:80b2da48193b2f33ed0c32c38140f9d3186583ce7d516526d462645fd98660ae", size = 4695386, upload-time = "2026-04-01T14:44:58.663Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/70/62/98f6b7f0c88b9addd0e87c217ded307b36be024d4ff8869a812b241d1345/pillow-12.2.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:22db17c68434de69d8ecfc2fe821569195c0c373b25cccb9cbdacf2c6e53c601", size = 6280384, upload-time = "2026-04-01T14:45:01.5Z" },
|
||||
@@ -1494,6 +1500,9 @@ serve = [
|
||||
{ name = "qwen-vl-utils", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
||||
{ name = "torch", version = "2.9.1+cu129", source = { registry = "https://download.pytorch.org/whl/cu129" }, marker = "sys_platform == 'linux'" },
|
||||
{ name = "torch", version = "2.12.1", source = { registry = "https://pypi.org/simple" }, marker = "sys_platform == 'darwin'" },
|
||||
{ name = "torchvision", version = "0.24.1", source = { registry = "https://download.pytorch.org/whl/cu129" }, marker = "platform_machine == 'aarch64' and sys_platform == 'linux'" },
|
||||
{ name = "torchvision", version = "0.24.1+cu129", source = { registry = "https://download.pytorch.org/whl/cu129" }, marker = "platform_machine != 'aarch64' and sys_platform == 'linux'" },
|
||||
{ name = "torchvision", version = "0.27.1", source = { registry = "https://pypi.org/simple" }, marker = "sys_platform == 'darwin'" },
|
||||
{ name = "transformers", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
||||
{ name = "uvicorn", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
||||
]
|
||||
@@ -1532,7 +1541,9 @@ requires-dist = [
|
||||
{ name = "torch", marker = "sys_platform != 'linux' and extra == 'embed'", specifier = ">=2.9.0" },
|
||||
{ name = "torch", marker = "sys_platform != 'linux' and extra == 'serve'", specifier = ">=2.9.0" },
|
||||
{ name = "torchvision", marker = "sys_platform == 'linux' and extra == 'embed'", specifier = ">=0.24.0", index = "https://download.pytorch.org/whl/cu129" },
|
||||
{ name = "torchvision", marker = "sys_platform == 'linux' and extra == 'serve'", specifier = ">=0.24.0", index = "https://download.pytorch.org/whl/cu129" },
|
||||
{ name = "torchvision", marker = "sys_platform != 'linux' and extra == 'embed'", specifier = ">=0.24.0" },
|
||||
{ name = "torchvision", marker = "sys_platform != 'linux' and extra == 'serve'", specifier = ">=0.24.0" },
|
||||
{ name = "tqdm", specifier = ">=4.60.0" },
|
||||
{ name = "trafilatura", marker = "extra == 'eval'", specifier = ">=1.6" },
|
||||
{ name = "transformers", marker = "extra == 'embed'", specifier = ">=4.57.0" },
|
||||
@@ -2426,7 +2437,10 @@ wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/8a/85/237e446c25abced71e9c53d269f2cef5bab8a82b3f88a12e00c5368e7368/xxhash-3.7.0-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:44fba4a5f1d179b7ddc7b3dc40f56f9209046421679b57025d4d8821b376fd8d", size = 275345, upload-time = "2026-04-25T11:06:42.525Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/62/34/c2c26c0a6a9cc739bc2a5f0ae03ba8b87deb12b8bce35f7ac495e790dc6d/xxhash-3.7.0-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:31e3516a0f829d06ded4a2c0f3c7c5561993256bfa1c493975fb9dc7bfa828a1", size = 414056, upload-time = "2026-04-25T11:06:44.343Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/a0/aa/5c58e9bc8071b8afd8dcf297ff362f723c4892168faba149f19904132bf4/xxhash-3.7.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:b59ee2ac81de57771a09ecad09191e840a1d2fae1ef684208320591055768f83", size = 191485, upload-time = "2026-04-25T11:06:46.262Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/41/d0/abc6c9d347ba1f1e1e1d98125d0881a0452c7f9a76a9dd03a7b5d2197f23/xxhash-3.7.0-cp313-cp313-android_21_x86_64.whl", hash = "sha256:845d347df254d6c619f616afa921331bada8614b8d373d58725c663ba97c3605", size = 35121, upload-time = "2026-04-25T11:06:53.048Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/c1/ca/d5174b4c36d10f64d4ca7050563138c5a599efb01a765858ddefc9c1202a/xxhash-3.7.0-cp313-cp313-android_21_arm64_v8a.whl", hash = "sha256:4b6d6b33f141158692bd4eafbb96edbc5aa0dabdb593a962db01a91983d4f8fa", size = 36813, upload-time = "2026-04-25T11:06:51.73Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/bf/11/4cc834eb3d79f2f2b3a6ef7324195208bcdfbdcf7534d2b17267aa5f3a8f/xxhash-3.7.0-cp313-cp313-ios_13_0_arm64_iphoneos.whl", hash = "sha256:fddbbb69a6fff4f421e7a0d1fa28f894b20112e9e3fab306af451e2dfd0e459b", size = 29624, upload-time = "2026-04-25T11:06:54.311Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/23/83/e97d3e7b635fe73a1dfb1e91f805324dd6d930bb42041cbf18f183bc0b6d/xxhash-3.7.0-cp313-cp313-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:54876a4e45101cec2bf8f31a973cda073a23e2e108538dad224ba07f85f22487", size = 30638, upload-time = "2026-04-25T11:06:55.864Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/f4/40/d84951d80c35db1f4c40a29a64a8520eea5d56e764c603906b4fe763580f/xxhash-3.7.0-cp313-cp313-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:0c72fe9c7e3d6dfd7f1e21e224a877917fa09c465694ba4e06464b9511b65544", size = 33323, upload-time = "2026-04-25T11:06:57.336Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/89/cc/c7dc6558d97e9ab023f663d69ab28b340ed9bf4d2d94f2c259cf896bb354/xxhash-3.7.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:a6d73a830b17ef49bc04e00182bd839164c1b3c59c127cd7c54fcb10c7ed8ee8", size = 33362, upload-time = "2026-04-25T11:06:58.656Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/2a/6e/46b84017b1301d54091430353d4ad5901654a3e0871649877a416f7f1644/xxhash-3.7.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:91c3b07cf3362086d8f126c6aecd8e5e9396ad8b2f2219ea7e49a8250c318acd", size = 30874, upload-time = "2026-04-25T11:06:59.834Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/df/5e/8f9158e3ab906ad3fec51e09b5ea0093e769f12207bfa42a368ca204e7ab/xxhash-3.7.0-cp313-cp313-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:50e879ebbac351c81565ca108db766d7832f5b8b6a5b14b8c0151f7190028e3d", size = 194185, upload-time = "2026-04-25T11:07:01.658Z" },
|
||||
@@ -2459,7 +2473,10 @@ wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/a1/7e/106d4067130c59f1e18a55ffadcd876d8c68534883a1e02685b29d3d8153/xxhash-3.7.0-cp313-cp313t-musllinux_1_2_riscv64.whl", hash = "sha256:1910df4756a5ab58cfad8744fc2d0f23926e3efcc346ee76e87b974abab922f4", size = 277600, upload-time = "2026-04-25T11:07:51.745Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/c5/86/a081dd30da71d720b2612a792bfd55e45fa9a07ac76a0507f60487473c25/xxhash-3.7.0-cp313-cp313t-musllinux_1_2_s390x.whl", hash = "sha256:d006faf3b491957efcb433489be3c149efe4787b7063d5cddb8ddaefdc60e0c1", size = 416980, upload-time = "2026-04-25T11:07:53.504Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/35/29/1a95221a029a3c1293773869e1ab47b07cbbdd82444a42809e8c60156626/xxhash-3.7.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:abb65b4e947e958f7b3b0d71db3ce447d1bc5f37f5eab871ce7223bda8768a04", size = 193840, upload-time = "2026-04-25T11:07:55.103Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/bc/01/255ec513e0a705d1f9a61413e78dfce4e3235203f0ed525a24c2b4b56345/xxhash-3.7.0-cp314-cp314-android_24_x86_64.whl", hash = "sha256:506a0b488f190f0a06769575e30caf71615c898ed93ab18b0dbcb6dec5c3713c", size = 35003, upload-time = "2026-04-25T11:08:02.338Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/0e/cc/431db584f6fbb9312e40a173af027644e5580d39df1f73603cbb9dca4d6b/xxhash-3.7.0-cp314-cp314-android_24_arm64_v8a.whl", hash = "sha256:8c5fcfd806c335bfa2adf1cd0b3110a44fc7b6995c3a648c27489bae85801465", size = 36644, upload-time = "2026-04-25T11:08:00.658Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/68/70/c55fc33c93445b44d8fc5a17b41ed99e3cebe92bcf8396809e63fc9a1165/xxhash-3.7.0-cp314-cp314-ios_13_0_arm64_iphoneos.whl", hash = "sha256:ec68dbba21532c0173a9872298e65c89749f7c9d21538c3a78b5bb6105871568", size = 29655, upload-time = "2026-04-25T11:08:03.701Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/c2/72/ff8de73df000d74467d12a59ce6d6e2b2a368b978d41ab7b1fba5ed442be/xxhash-3.7.0-cp314-cp314-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:fa77e7ec1450d415d20129961814787c9abd9a07f98872f070b1fe96c5084611", size = 30664, upload-time = "2026-04-25T11:08:05.011Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/b6/91/08416d9bd9bc3bf39d831abe8a5631ac2db5141dfd6fe81c3fe59a1f9264/xxhash-3.7.0-cp314-cp314-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:fe32736295ea38e43e7d9424053c8c47c9f64fecfc7c895fb3da9b30b131c9ee", size = 33317, upload-time = "2026-04-25T11:08:06.413Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/0e/3b/86b1caa4dee10a99f4bf9521e623359341c5e50d05158fa10c275b2bd079/xxhash-3.7.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:ab9dd2c83c4bbd63e422181a76f13502d049d3ddcac9a1bdc29196263d692bb8", size = 33457, upload-time = "2026-04-25T11:08:08.099Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/ed/38/98ea14ad1517e1461292a65906951458d520689782bfbae111050145bdba/xxhash-3.7.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:3afec3a336a2286601a437cb07562ab0227685e6fbb9ec17e8c18457ff348ecf", size = 30894, upload-time = "2026-04-25T11:08:09.429Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/61/a2/074654d0b893606541199993c7db70067d9fc63b748e0d60020a52a1bd36/xxhash-3.7.0-cp314-cp314-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:565df64437a9390f84465dcca33e7377114c7ede8d05cd2cf20081f831ea788e", size = 194409, upload-time = "2026-04-25T11:08:10.91Z" },
|
||||
|
||||
Reference in New Issue
Block a user