Bumps the Python backend dependency set, holding back the ones that are
resolver-blocked or that regress behaviour this repo depends on.
Notable upgrades:
cryptography 46.0.7 -> 50.0.0 (requires msal 1.37.0, which relaxes its cap)
protobuf 6.33.6 -> 7.35.1 (floats opentelemetry-* to 1.44.0)
openai 2.32.0 -> 2.53.0, anthropic 0.88.0 -> 0.121.0
google-genai 1.73.1 -> 2.17.0 (the 2.0 break is scoped to the Interactions
API, which this repo does not use)
fastmcp 3.2.4 -> 3.4.6, gunicorn 25.3.0 -> 26.0.0
starlette 1.0.0 -> 1.6.0, uvicorn 0.42.0 -> 0.52.1
sentence-transformers 5.3.0 -> 5.7.0, numpy 2.4.4 -> 2.5.1
faiss-cpu 1.13.2 -> 1.15.0, pillow -> 12.3.0 (pinned; security release)
pypdf 6.9.2 -> 6.15.0, lxml 6.0.2 -> 6.1.1 (CVE-2026-41066)
Code changes needed by the bumps:
- openai >= 2.53 rejects a falsy api_key at construction. Keyless
OpenAI-compatible backends (Ollama, llama.cpp, vLLM) legitimately have
none, and pydantic-settings yields "" for a bare `API_KEY=` in .env, so
both call sites now fall back to a placeholder.
- Flask >= 3.1.2 tears a stream_with_context request down twice, and a
ContextVar token may only be reset once. The log-context teardown hook
is now idempotent.
- sentence-transformers renamed get_sentence_embedding_dimension to
get_embedding_dimension in 5.4; use the new name with a fallback.
Held back deliberately:
torch 2.11.0 - 2.13.0 drags torchvision 0.26 -> 0.28 and the whole
docling stack; torch is only probed for cuda.empty_cache()
tokenizers 0.22.2 - transformers pins <=0.23.0 and no stable 0.23.0 exists
transformers - capped <5.9.0: 5.9+ breaks docling's PDF layout model on
Apple Silicon (MPS float64), matching docling-core's own
darwin pin
docling 2.84.0 - 2.118.1 needs rapidocr >=3.9.1 and conflicts with
transformers on macOS; wants its own PR with a
golden-corpus diff
redis 7.4.0 - 8.x defaults socket_timeout to 5s, silently capping the
blocking reads in the device broker and SSE tail
websockets 16.0 - google-genai caps <17.0
marshmallow - dataclasses-json hard-caps <4; spec tightened to match
langchain block - langchain-community 0.4.2 deletes the Qdrant vectorstore
this repo imports; langchain 1.3.x needs websockets <16
Source access control
---------------------
`active_docs` is client-supplied and reached the retriever unchecked, and the
retriever queries `WHERE source_id = <id>` with no owner predicate — so any
caller could pass any source id to /stream or /api/answer and have another
tenant's documents quoted back, while /api/sources/<id>/search correctly
refused the same id. Gate it through `can_access`, the helper the guarded
endpoints already use, and filter `self.source` down to the authorized set.
Fails closed: no principal, or a check that errors, drops the source.
Three sibling paths had the same gap:
- workflow agent nodes: `AgentNodeConfig.sources` is written verbatim from
client JSON at save time and nothing validated it, so a node could name any
tenant's source. Gate against the workflow owner, so shared workflows keep
reading their owner's sources like shared agents do.
- /api/share: `_resolve_source_pg_id` resolved any id with no ownership
predicate and baked it into the agent the share creates; /api/search then
searched it. Authorize before attaching.
- search_service: re-resolve the ids stored on an agent row instead of
trusting them, so a row written by any future path with the same gap cannot
be read back.
Team grantees previously lost their source's retrieval config: the post-check
read was still owner-scoped, so it missed and fell back to defaults (an
`agentic_tool` source was bulk-prefetched for every grantee). Read unscoped
after `can_access` passes.
Retrieval
---------
`PGVectorStore._ensure_table_exists` created an IVFFlat index on the empty
table it had just created. IVFFlat computes centroids at build time, so those
centroids were random, and combined with the `source_id` post-filter a source
with hundreds of embedded chunks returned zero rows — retrieval reported no
documents, the model answered from memory, and nothing was logged. Stop
creating the index (exact search is correct and fast well past the sizes most
deployments reach); raise `ivfflat.probes` to sqrt(lists) where an index still
exists; and re-run a short indexed search exactly, since post-filtering means
no index setting can guarantee a full result. `graphrag` had the same
empty-table index with no fallback at all.
Also: bound `chunks` to 0-500 on both the request and agent paths (0 still
means "skip retrieval"), let a source's configured `retrieval.chunks` outrank
the request body, and cap ClassicRAG's per-source floor at
max(top_k, n_sources) so attaching sources cannot inflate the result set.
Silent failures
---------------
An empty retrieval was invisible to both the model and the client: the `source`
event was suppressed when the list was empty, so "searched and found nothing"
looked identical to "no source attached", and the prompt said nothing at all.
Emit the event always, and tell the model when a search ran and returned
nothing. A file that parses to nothing now fails ingest with a message naming
the cause instead of storing an embedding of the empty string. `score_threshold`
returns warnings when the active store or retriever cannot honour it.
Prompt structure
----------------
Retrieved documents move from the system prompt into the user turn, with the
injection guard restated next to them: they change every turn (defeating prefix
caching), they are third-party text that should not carry system authority, and
routing them through the query budget makes them truncatable rather than
silently crowding it out. Documents are shed lowest-ranked-first before the
question is touched.
The six chat presets (3 tones x 2 retrieval modes) differed only in their
Answering section; they are now composed from single-source fragments at load
time, not through Jinja inheritance, which would have opened a file-read
surface in the template sandbox and broken the tool-prefetch parser. Per-tool
guidance moves out of the prompt into tool schemas, so it travels with the tool
and cannot render when the tool is absent. A plain-text custom prompt is staged
as a persona value inside the skeleton instead of replacing it wholesale — it
used to silently lose the injection guard, platform block, memory and
attachments, and its braces are now inert.
Other fixes
-----------
- agents/base: an oversized system prompt drove the query budget negative and
dispatched a full-price request with an empty question; raise instead.
- llm/anthropic: migrate off the retired Text Completions API. It flattened
history to first+last message and ignored tools entirely. Adds the missing
Anthropic handler, without which every tool call was silently dropped.
- sources/upload: `sitemap` had no branch, so every sitemap ingest died on a
TypeError; `validate_url` now rejects a falsy URL cleanly.
- workflow nodes: retrieved documents never reached the node agent, so a
classic node with a source and an ordinary prompt answered "I have no
documents" while the run reported completed.
- parser/bulk: copy the metadata dict, or every chunk reports the last chunk's
token_count.
- crawler_loader: carry the page title, or citations render the whole chunk
body as the label.
Migration 0023_wiki_pages (source-scoped; version/content_hash/embed_status; FK source_id->sources ON DELETE CASCADE; UNIQUE(source_id,path) + prefix index). WikiPagesRepository: path-keyed CRUD with content-hash short-circuit, version bump, and move-with-reject. BaseVectorStore.delete_chunks_by_source_path (loop default) + parameterized pgvector override (DELETE WHERE source_id=%s AND metadata->>'source'=%s). Unit 1 of F-Wiki.
Introduces a per-source config contract that makes RAG behavior strategy-dispatched instead of a single hardcoded path. Every source gains a validated JSONB config; an empty/absent config reproduces current behavior byte-for-byte, and the whole path is gated by PER_SOURCE_RETRIEVAL_ENABLED.
Foundation: sources.config JSONB column + migration 0022_source_config; SourceConfig/ChunkingConfig/RetrievalConfig pydantic models (strict on write, lenient on read); ChunkerCreator and RetrieverCreator.register registries; config threaded through the upload routes, ingest/remote/connector workers, and reingest.
Retrieval: a Dispatcher groups sources by retriever key (all-classic collapses to today's single ClassicRAG under one shared token budget; non-classic retrievers get their own instance), removing the previous single-global-retriever collapse in stream_processor. Per-source chunks, score_threshold (honored for pgvector/mongodb, safely ignored elsewhere), and rephrase_query toggle. New PATCH /api/sources/<id>/config with team-aware (effective_write_owner) authz and a requires_reingest signal.
Chunking strategies: recursive, markdown, parent_child (selectable per source; re-ingest to apply). Search exposure: per-source prefetch vs agentic_tool for agentic/research agents. Map-reduce prescreen: optional LLM relevance pre-filter implemented as a composable post-retrieval stage that wraps any retriever.
Backend and frontend (shared Retrieval options panel + edit modal) with tests; backend suite and frontend vitest green. Excludes the wiki and GraphRAG flagships.
Alex