* feat(rag): parallelize document indexing via index_documents_parallel (#5118)
Adds LibraryRAGService.index_documents_parallel — a bounded ThreadPoolExecutor
fan-out over index_document — and rewires all six serial
"for doc: index_document(...)" loops to use it:
- index_collection (SSE) — also drives a 5s heartbeat to keep proxies alive
- index_all (SSE)
- _background_index_worker
- _auto_index_documents_worker
- _reconcile_unindexed_documents (background scheduler, both branches)
New setting rag.indexing_max_parallel_docs (default 4, range 1-16).
rag.indexing_batch_size description clarified to reflect its actual legacy
SSE batch-pacing role.
Concurrency invariants preserved: per-doc index_document already opens a
fresh DB session per call, FAISS mutations remain gated by the existing
per-(user,index_path) _get_faiss_write_lock inside
_merge_and_persist_locked, and LocalEmbeddingManager uses double-checked
locking for first-load.
Cancellation is polled between completions inside the helper; queued-but-
not-started futures are cancelled on shutdown (ThreadPoolExecutor.shutdown
with cancel_futures=True) so a cancelled background indexer doesn't block
on stuck embedding sockets.
Tests:
- New tests/research_library/services/test_index_documents_parallel.py (10
tests covering dispatch, dedup, cancellation, progress callback, worker
bounds, empty input, result aggregation).
- New tests/research_library/routes/test_rag_routes_parallel_indexing.py
(signature + settings-default pinning).
- Updated 12 existing tests in the 4 rag_routes coverage suites to assert
against the new index_documents_parallel entry point instead of the
legacy index_document direct call.
Closes#5118.
* style: fix missing newline at end of changelog fragment (pre-commit autofix)
* fix(rag): drain SSE workers on cancel, serialise first FAISS init, honour indexing_max_parallel_docs in reconciler (#5119)
Followup to #5118. Three runtime regressions uncovered after the parallel
indexing helper landed:
* index_documents_parallel no longer races on the lazy faiss_index is None
check plus the unique index_hash INSERT inside _get_or_create_rag_index.
LibraryRAGService now serialises the first init via a per-instance lock
and _ensure_faiss_index helper, and _get_or_create_rag_index recovers
from a concurrent IntegrityError by re-querying the winning row.
* Cancellation no longer lets in-flight workers outlive the caller's
service context. The helper always shuts its pool down with wait=True
(cancel_futures still skips queued ones). Both SSE generators in
rag_routes.py wire a threading.Event via is_cancelled; index_collection
additionally joins its background worker thread before safe_close so
no worker observes the closed embedding manager.
* The scheduled reconciler in scheduler/background.py now reads
rag.indexing_max_parallel_docs instead of hard-coding max_workers=4,
so a configured value of 1 truly restores sequential scheduled indexing.
Also: regenerate_golden_master.py writes LF instead of CRLF so the
pre-commit hook doesn't false-positive on Windows; golden master
regenerated for the new rag.indexing_max_parallel_docs key.
* docs: add newline to changelog fragment and reformat test file
Co-authored-by: kiloconnect[bot] <240665456+kiloconnect[bot]@users.noreply.github.com>
* fix(rag): close post-cancel submission leak in index_documents_parallel
The bounded-submission loop in `index_documents_parallel` now polls
`is_cancelled` BEFORE admitting any new work into the pool, instead of
submitting all docs up front. This closes the post-cancel leak identified
in the latest PR #5119 review: queued futures could previously be picked
up by idle workers between the caller's cancel signal
(`_sse_cancel.set()` in `rag_routes.index_collection`) and the helper's
`pool.shutdown(cancel_futures=True)`.
In-flight futures still drain naturally via `wait=True` — only NEW
submissions are gated on the cancel poll. Once cancel is observed the
gate stays closed until the loop exits, so post-cancel starts are zero
by construction.
Also migrates `tests/news/test_library_sweep.py` from the legacy serial
`index_document` mocks to the parallel entrypoint (12 tests previously
failing on stale mocks now green) and adds two deterministic regression
tests in `test_index_documents_parallel.py`:
- `test_no_new_index_document_calls_after_cancel_observed`: with
`max_workers=2` + 20 docs and a cancel that flips True after 2
polls, bounded submission admits ≤4 calls (vs. 20 under the old
model).
- `test_zero_index_document_calls_when_cancel_polled_first`: when
`is_cancelled` returns True on the first poll, ZERO calls are
admitted (vs. all of them under the old model).
* Fix Windows-specific os.fsync bad file descriptor in FAISS persist and update gaps coverage mocks
* Skip chmod permission test on Windows in test_faiss_store.py
* style: add missing newline to changelog fragment
* fix(ci): remove stale migration head assertion
* test(db): keep migration head checks generic
* Address review comments on PR #5119
* style: run pre-commit to fix formatting in library_rag_service.py and test_index_documents_parallel.py
* fix(rag): check cancellation in workers and restore LF in test_auto_indexing.py
* fix(rag): address cancellation/submission gap and docstring invariants
- Update index_documents_parallel docstring: replace non-existent
self.faiss_index / _merge_and_persist_locked references with the real
serialization path (VectorIndex.apply() under _get_faiss_write_lock,
call sites L422/L879/L1097/L2025).
- Add deterministic regression test for the false-poll-to-submit gap:
a separate cancel-setter thread sets the cancel event AFTER the main
thread's is_cancelled() returns False but BEFORE the worker checks.
Asserts that no index_document call is admitted after the signal.
- Add regression test for cancel preservation during the last
in-flight worker: if the cancel event is set while the final worker
is running and that worker completes successfully, the aggregate
must still report cancelled=True (covered by the finally-block poll).
* fix(rag): refine cancellation mechanics and task status state transitions
S
7a8b85162b