feat(sources): local corpus source - your own files as a ranked signal (#808)
* feat(sources): local corpus source - your own files as a ranked signal * fix: address self-review findings * fix: address round-2 residual (surgical round) * fix: defang corpus sentinels, matching-window snippets, exclusion-aware hosted gate, traversal notes, bounded discovery * fix: keep absolute local paths out of corpus notes and coverage diagnostics * fix(corpus): keep raw exception text out of coverage notes OS and subprocess errors embed the failing absolute path in str(exc), and scan/cache notes flow into source_status detail rendered in coverage diagnostics outside the private corpus block. Notes now carry the error's strerror (or class name) instead, so a permission failure or file race can no longer leak a local path from a private run. --------- Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
This commit is contained in:
Matt Van Horn
+23
-1
@@ -46,6 +46,8 @@ The engine's `.env` reader doesn't expand `$HOME` — only the tilde, via `Path(
|
||||
- `--save-dir <path>` - one-off output location. **Flag wins over env var.** If neither flag nor env var is set, the engine does not write a file (DB persistence is independent — see `LAST30DAYS_STORE` below).
|
||||
- `--output <file>` - write the rendered output to an exact file path, using the format selected by `--emit`.
|
||||
- `--json-profile {agent,raw}` - select the research JSON shape used with `--emit=json`. `agent` is the default, versioned workflow contract; `raw` preserves the full internal `Report` dump for debugging and power users. See the [JSON export reference](docs/reference/json-export.md).
|
||||
- `--corpus <dir>` - add a local `.md`/`.txt` directory as a private ranked source; repeat the flag for multiple directories. PDFs are extracted only when `pdftotext` is on PATH and otherwise skip with a note. File modification time supplies recency, so the normal research window applies.
|
||||
- `--corpus-all-time` - include relevant registered files whose modification time is older than the current research window. Without this flag, a 30-day run includes only files modified in those 30 days.
|
||||
- `--register {default,exec,dev,creator,eli5}` - shape a standard single-topic Markdown or HTML research brief for its audience. `exec` is decisions-first with five core findings and numbers up top; `dev` gives GitHub, code, and technical signals more room; `creator` leads with hooks, Best Takes, community reactions, and virality metrics; `eli5` keeps the established evidence layout and asks the synthesizing agent for accessible language. Registers do not change retrieval, JSON exports, discovery, drill, library feed/search, or comparison output.
|
||||
- `--discover <domain>` - topic-less trending discovery. Sweeps rising/top-week Reddit listings (category-mapped communities, with r/all as the uncategorized fallback), Hacker News front/best stories, Digg AI 1000 clusters when `digg-pp-cli` is on PATH, and broad X activity when an X backend is authenticated, then returns 5-10 engagement-velocity-ranked topics. Run without a positional topic; it is mutually exclusive with `--drill`. `--emit=json` uses the separate versioned discovery contract documented in the [JSON export reference](docs/reference/json-export.md).
|
||||
- `--drill <target>` - deep follow-up over the fresh `~/.config/last30days/last-report.json` cache. Accepts a 1-based index (`--drill "cluster 3"` or `--drill "3"`) or a fuzzy cluster title/entity description. It re-fetches only sources that contributed to the matched cluster, enables their deep comment/transcript enrichment paths, merges/dedupes the evidence, and replaces the cache so drills can chain. Run it without a positional topic; if the cache is absent or expired, run a normal research pass first.
|
||||
@@ -98,12 +100,32 @@ Override the global location with `LAST30DAYS_CONFIG_DIR=/path` (or `LAST30DAYS_
|
||||
|
||||
The project-scoped file is useful for **intentional per-client setups**: drop a `.claude/last30days.env` into each client folder (`SCRAPECREATORS_API_KEY`, `INCLUDE_SOURCES`, `LAST30DAYS_MEMORY_DIR`, `BSKY_HANDLE`, etc), then opt in with `LAST30DAYS_TRUST_PROJECT_CONFIG=1` from your shell or `~/.config/last30days/.env`. Folder-mode hosts such as Codex desktop do not trust hidden project config by default, and discovery stops at the git root so unrelated parent folders cannot silently influence runs.
|
||||
|
||||
**`LAST30DAYS_API_KEY`** + **`LAST30DAYS_API_BASE`** - optional remote-API backend. Set BOTH to route research through a remote API endpoint instead of running the local sources: `LAST30DAYS_API_BASE` is the endpoint (there is no built-in default), and `LAST30DAYS_API_KEY` is the bearer key for it. When both are set (and `--mock` is not passed), the engine submits the topic to that endpoint, polls with progress on stderr, and prints the server's report; none of the per-source keys below are used for that run. Non-default `--register` selections are forwarded with the request so server-side synthesis uses the same audience preset. Leave either unset to run local sources exactly as normal. Unlike the other keys here, these two are read only from the **process environment** (export them in your shell or host config) - they are deliberately not loaded from the `.env` files above, so a project-scoped `.env` can never silently redirect research to a remote endpoint. The remote endpoint does not return the local `Report` needed for the versioned agent JSON profile; use `--emit=json --json-profile=raw` for its existing server-response JSON contract.
|
||||
**`LAST30DAYS_API_KEY`** + **`LAST30DAYS_API_BASE`** - optional remote-API backend. Set BOTH to route research through a remote API endpoint instead of running the local sources: `LAST30DAYS_API_BASE` is the endpoint (there is no built-in default), and `LAST30DAYS_API_KEY` is the bearer key for it. When both are set (and `--mock` is not passed), the engine submits the topic to that endpoint, polls with progress on stderr, and prints the server's report; none of the per-source keys below are used for that run. A configured local corpus is the privacy exception: the engine bypasses the hosted backend and runs locally rather than forwarding file-derived input. Non-default `--register` selections are forwarded with the request so server-side synthesis uses the same audience preset. Leave either unset to run local sources exactly as normal. Unlike the other keys here, these two are read only from the **process environment** (export them in your shell or host config) - they are deliberately not loaded from the `.env` files above, so a project-scoped `.env` can never silently redirect research to a remote endpoint. The remote endpoint does not return the local `Report` needed for the versioned agent JSON profile; use `--emit=json --json-profile=raw` for its existing server-response JSON contract.
|
||||
|
||||
### Local corpus (your files)
|
||||
|
||||
Register persistent directories with `LAST30DAYS_CORPUS_DIRS`. Separate paths with `:` on macOS/Linux (the platform path separator is `;` on Windows):
|
||||
|
||||
```bash
|
||||
# ~/.config/last30days/.env
|
||||
LAST30DAYS_CORPUS_DIRS=~/notes:~/meeting-transcripts
|
||||
# LAST30DAYS_CORPUS_IN_EXPORT=1 # explicit agent-JSON opt-in; off by default
|
||||
```
|
||||
|
||||
The slash-command experience remains primary: ask `/last30days` to include your registered notes. For direct engine scripting or development, the equivalent one-off invocation is:
|
||||
|
||||
```bash
|
||||
python3 skills/last30days/scripts/last30days.py "MCP servers" \
|
||||
--corpus ~/notes --corpus ~/meeting-transcripts
|
||||
```
|
||||
|
||||
**Privacy:** corpus files are read locally, never sent through a source HTTP client, never forwarded to `LAST30DAYS_API_BASE`, never included in remote reranker/fun-scoring prompts, and do not consume network-source concurrency or retry budget. Matches appear in a badged **From your files** section. Corpus candidates are removed from `--publish-html`, `library feed --publish`, and the versioned agent JSON export by default, including corpus-derived cluster titles and source outcomes. Set `LAST30DAYS_CORPUS_IN_EXPORT=1` only when you intentionally want corpus results in the agent JSON written to local stdout/files. The unversioned `--json-profile=raw` debug dump remains a full local report and can contain corpus text; do not redirect it to an external system unless that is intentional. Extracted text is cached by file mtime in `~/.config/last30days/corpus-cache.json` with mode `0600`; a corpus-bearing `last-report.json` cache is also tightened to `0600`. Delete either cache at any time to clear it.
|
||||
|
||||
**Source-by-source** - what each key unlocks:
|
||||
|
||||
| Source | Key(s) | Required for | Free tier |
|
||||
|---|---|---|---|
|
||||
| Local corpus | `--corpus <dir>` or `LAST30DAYS_CORPUS_DIRS` | private `.md`/`.txt`; `.pdf` when `pdftotext` is on PATH | yes (offline) |
|
||||
| Reddit (public) | none (default); `SCRAPECREATORS_API_KEY` + `LAST30DAYS_REDDIT_BACKEND=scrapecreators` to pin SC primary with public fallback | always on; SC pin requires `SCRAPECREATORS_API_KEY` | yes |
|
||||
| Hacker News | none | always on | yes |
|
||||
| Polymarket | none | always on | yes |
|
||||
|
||||
@@ -21,6 +21,10 @@ python3 skills/last30days/scripts/last30days.py "AI coding agents" --emit=json -
|
||||
|
||||
The raw profile is intentionally unversioned and may change when pipeline internals change. It preserves the JSON serialization used before the agent profile was introduced.
|
||||
|
||||
### Local corpus privacy
|
||||
|
||||
Evidence from `--corpus` / `LAST30DAYS_CORPUS_DIRS` is excluded from the versioned agent profile by default. The exclusion removes corpus results, corpus-only clusters, corpus source outcomes, freshness verdicts, and titles derived from a corpus representative. Set `LAST30DAYS_CORPUS_IN_EXPORT=1` only for a run whose JSON is intentionally allowed to contain local file contents. This opt-in does not change the schema shape or version; it permits `source: "corpus"` entries in the existing result fields. The unversioned `raw` profile is a complete local debug dump and may contain corpus paths and text.
|
||||
|
||||
## Discovery export
|
||||
|
||||
Discovery mode has a separate versioned contract so its topic results do not change the normal research export:
|
||||
|
||||
@@ -416,7 +416,7 @@ Set `LAST30DAYS_MEMORY_DIR` before invoking the skill to choose where raw resear
|
||||
|
||||
The engine reads `LAST30DAYS_MEMORY_DIR` from either the process env or `~/.config/last30days/.env`, so direct CLI invocations (`python3 scripts/last30days.py ...`) without `--save-dir` will still save when the env var is set. Mirrors the `LAST30DAYS_STORE` env-or-flag convention. Explicit `--save-dir` always wins.
|
||||
|
||||
When both `LAST30DAYS_API_KEY` and `LAST30DAYS_API_BASE` are set, the engine runs the research through that configured remote API instead of local sources (unless `--mock` is passed); `LAST30DAYS_API_BASE` is the endpoint and has no built-in default, so leaving either variable unset runs local sources normally. The invocation is unchanged: same flags, `--quick`/`--deep` map to search depth, a non-default `--register` is forwarded for server-side synthesis, progress lines still stream on stderr (`[narrate] step=...` plus a compact elapsed/eta line), and the report prints on stdout and saves to the memory dir as usual, so Steps 1-4 proceed normally on the output. The exception is research JSON: the remote endpoint does not return the local `Report` needed for the versioned agent profile, so use `--emit=json --json-profile=raw` for its existing server-response JSON contract. No per-source keys or setup-wizard credentials are needed for the search itself in this mode. Two engine exits need specific handling: exit code 3 means the API asked a clarifying question first - the engine prints the question and options on stderr; present them to the user and re-run with the chosen angle folded into the topic. An insufficient-credits failure (HTTP 402) prints the account's balance, the amount needed, and a billing link - relay those lines to the user verbatim; do not fall back to WebSearch-only synthesis.
|
||||
When both `LAST30DAYS_API_KEY` and `LAST30DAYS_API_BASE` are set, the engine runs the research through that configured remote API instead of local sources (unless `--mock` is passed); `LAST30DAYS_API_BASE` is the endpoint and has no built-in default, so leaving either variable unset runs local sources normally. A configured `--corpus` / `LAST30DAYS_CORPUS_DIRS` is the privacy exception: the engine bypasses the hosted backend and runs locally so no file-derived input is forwarded. The invocation is otherwise unchanged: same flags, `--quick`/`--deep` map to search depth, a non-default `--register` is forwarded for server-side synthesis, progress lines still stream on stderr (`[narrate] step=...` plus a compact elapsed/eta line), and the report prints on stdout and saves to the memory dir as usual, so Steps 1-4 proceed normally on the output. The exception is research JSON: the remote endpoint does not return the local `Report` needed for the versioned agent profile, so use `--emit=json --json-profile=raw` for its existing server-response JSON contract. No per-source keys or setup-wizard credentials are needed for the search itself in this mode. Two engine exits need specific handling: exit code 3 means the API asked a clarifying question first - the engine prints the question and options on stderr; present them to the user and re-run with the chosen angle folded into the topic. An insufficient-credits failure (HTTP 402) prints the account's balance, the amount needed, and a billing link - relay those lines to the user verbatim; do not fall back to WebSearch-only synthesis.
|
||||
|
||||
**Developer-only eval capture:** `--record-fixtures <dir>` is a hidden direct-engine flag for maintaining the deterministic research-quality suite. It records scrubbed HTTP and CLI-adapter responses to `<dir>/http.json`; it is never part of the user-facing slash-command invocation. Follow `docs/reference/eval.md` for fixture review, replay, and baseline rules.
|
||||
|
||||
@@ -644,10 +644,12 @@ SKILL_DIR="<absolute path of the directory containing the SKILL.md you just Read
|
||||
"${LAST30DAYS_PYTHON}" "${SKILL_DIR}/scripts/last30days.py" --diagnose
|
||||
```
|
||||
|
||||
`--diagnose` prints JSON. `ACTIVE_SOURCES_LIST` is its `available_sources` array — the engine's authoritative source set, computed after credential resolution. Map the tokens to display names: `reddit`→Reddit, `hackernews`→Hacker News, `polymarket`→Polymarket, `github`→GitHub, `digg`→Digg, `x`→X, `youtube`→YouTube, `tiktok`→TikTok, `instagram`→Instagram, `threads`→Threads, `pinterest`→Pinterest, `linkedin`→LinkedIn, `bluesky`→Bluesky, `perplexity`→Perplexity, `grounding`→Web, `jobs`→Jobs.
|
||||
`--diagnose` prints JSON. `ACTIVE_SOURCES_LIST` is its `available_sources` array — the engine's authoritative source set, computed after credential resolution. Map the tokens to display names: `reddit`→Reddit, `hackernews`→Hacker News, `polymarket`→Polymarket, `github`→GitHub, `digg`→Digg, `x`→X, `youtube`→YouTube, `tiktok`→TikTok, `instagram`→Instagram, `threads`→Threads, `pinterest`→Pinterest, `linkedin`→LinkedIn, `bluesky`→Bluesky, `perplexity`→Perplexity, `grounding`→Web, `jobs`→Jobs, `corpus`→Your files.
|
||||
|
||||
- If EXCLUDE_SOURCES is set (comma-separated, case-insensitive): drop any matching source from ACTIVE_SOURCES_LIST before displaying
|
||||
|
||||
**Local corpus source:** If the user asks to include their own notes/documents, preserve each supplied directory as a repeatable `--corpus <dir>` engine flag. `LAST30DAYS_CORPUS_DIRS` activates persistent registered directories automatically. Do not WebSearch, upload, quote into a hosted request, or otherwise expose those paths or contents. Corpus retrieval is an offline source lane; its candidates also bypass remote reranker/fun-scoring prompts and use deterministic local scoring. The engine renders matches under the 🔒 **From your files** badge. The normal recency window uses file modification time; add `--corpus-all-time` only when the user explicitly asks to include older files. Corpus evidence is excluded from `--publish-html`, `library feed --publish`, and agent JSON by default. `LAST30DAYS_CORPUS_IN_EXPORT=1` is the explicit agent-JSON privacy opt-in; never enable it on the user's behalf. When a corpus is configured alongside `LAST30DAYS_API_KEY`/`LAST30DAYS_API_BASE`, the engine deliberately bypasses the hosted backend and runs locally.
|
||||
|
||||
**Perplexity source:** use it only when the user asks for Perplexity, Deep Research, or paid grounded synthesis, or when `perplexity` is already enabled in `INCLUDE_SOURCES` / `--search`. Direct `PERPLEXITY_API_KEY` supports Sonar synthesis, Search API rows, and async Deep Research. `OPENROUTER_API_KEY` is only a Sonar fallback. Normal runs default to `LAST30DAYS_PERPLEXITY_MODE=sonar`; use `search` for raw ranked web rows, `both` for synthesis plus rows, and `--deep-research` for `sonar-deep-research` with a 600s default wall timeout. A local Deep Research timeout is not a failed API key; inspect the raw artifact's async request id/status and resume by id if needed.
|
||||
|
||||
**Reddit backend pin:** Reddit defaults to the free public backend with ScrapeCreators as a backup when `SCRAPECREATORS_API_KEY` is available. If the user says public Reddit is shallow, bot-gated, or missing nested comments, tell them they can set `LAST30DAYS_REDDIT_BACKEND=scrapecreators` alongside `SCRAPECREATORS_API_KEY` to make ScrapeCreators primary and keep public Reddit as fallback. Do not set this automatically for normal runs.
|
||||
|
||||
@@ -48,7 +48,7 @@ if os.name == "nt":
|
||||
SCRIPT_DIR = Path(__file__).parent.resolve()
|
||||
sys.path.insert(0, str(SCRIPT_DIR))
|
||||
|
||||
from lib import dates, env, freshness, html_render, http, permission_preflight, pipeline, registers, render, schema, ui
|
||||
from lib import corpus, dates, env, freshness, html_render, http, permission_preflight, pipeline, registers, render, schema, ui
|
||||
|
||||
_child_pids: set[int] = set()
|
||||
_child_pids_lock = threading.Lock()
|
||||
@@ -121,6 +121,34 @@ def slugify(value: str) -> str:
|
||||
return slug or "last30days"
|
||||
|
||||
|
||||
def _report_has_private_corpus(report: schema.Report) -> bool:
|
||||
items_by_source = getattr(report, "items_by_source", {})
|
||||
if isinstance(items_by_source, dict) and items_by_source.get("corpus"):
|
||||
return True
|
||||
candidates = getattr(report, "ranked_candidates", ())
|
||||
if not isinstance(candidates, (list, tuple)):
|
||||
return False
|
||||
return any(
|
||||
candidate.source == "corpus"
|
||||
or any(item.source == "corpus" for item in candidate.source_items)
|
||||
for candidate in candidates
|
||||
)
|
||||
|
||||
|
||||
def _ensure_output_directory(path: Path, *, private: bool) -> None:
|
||||
if not private:
|
||||
path.mkdir(parents=True, exist_ok=True)
|
||||
return
|
||||
missing: list[Path] = []
|
||||
current = path
|
||||
while not current.exists():
|
||||
missing.append(current)
|
||||
current = current.parent
|
||||
path.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
for directory in missing:
|
||||
directory.chmod(0o700)
|
||||
|
||||
|
||||
def save_output(
|
||||
report: schema.Report,
|
||||
emit: str,
|
||||
@@ -131,10 +159,10 @@ def save_output(
|
||||
rendered_content: str | None = None,
|
||||
json_profile: str = "agent",
|
||||
register: str = "default",
|
||||
private: bool | None = None,
|
||||
) -> Path:
|
||||
from datetime import datetime
|
||||
path = Path(save_dir).expanduser().resolve()
|
||||
path.mkdir(parents=True, exist_ok=True)
|
||||
slug = slugify(topic_override or report.topic)
|
||||
extension = "json" if emit == "json" else "html" if emit == "html" else "md"
|
||||
raw_label = "raw-html" if emit == "html" else "raw"
|
||||
@@ -159,10 +187,16 @@ def save_output(
|
||||
)
|
||||
else:
|
||||
content = render.render_full(report)
|
||||
private_corpus = _report_has_private_corpus(report) or bool(private)
|
||||
_ensure_output_directory(path, private=private_corpus)
|
||||
encoded = content.encode("utf-8")
|
||||
for candidate in candidates:
|
||||
try:
|
||||
fd = os.open(candidate, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o644)
|
||||
fd = os.open(
|
||||
candidate,
|
||||
os.O_CREAT | os.O_EXCL | os.O_WRONLY,
|
||||
0o600 if private_corpus else 0o644,
|
||||
)
|
||||
except FileExistsError:
|
||||
continue
|
||||
with os.fdopen(fd, "wb") as f:
|
||||
@@ -194,10 +228,25 @@ def save_output(
|
||||
)
|
||||
|
||||
|
||||
def save_rendered_output(rendered_content: str, output_file: str) -> Path:
|
||||
def save_rendered_output(
|
||||
rendered_content: str,
|
||||
output_file: str,
|
||||
*,
|
||||
private: bool = False,
|
||||
) -> Path:
|
||||
out_path = Path(output_file).expanduser().resolve()
|
||||
out_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
out_path.write_text(rendered_content, encoding="utf-8")
|
||||
_ensure_output_directory(out_path.parent, private=private)
|
||||
if private and out_path.exists():
|
||||
out_path.chmod(0o600)
|
||||
fd = os.open(
|
||||
out_path,
|
||||
os.O_CREAT | os.O_TRUNC | os.O_WRONLY,
|
||||
0o600 if private else 0o644,
|
||||
)
|
||||
with os.fdopen(fd, "w", encoding="utf-8") as handle:
|
||||
handle.write(rendered_content)
|
||||
if private:
|
||||
out_path.chmod(0o600)
|
||||
return out_path
|
||||
|
||||
|
||||
@@ -384,14 +433,21 @@ def _scoped_store_db(args: argparse.Namespace) -> Path | None:
|
||||
def persist_report(report: schema.Report, store_db: Path | None = None) -> dict[str, int]:
|
||||
import store
|
||||
|
||||
private_corpus = _report_has_private_corpus(report)
|
||||
with store.scoped_db(store_db):
|
||||
if private_corpus:
|
||||
store.ensure_private_db_files()
|
||||
store.init_db()
|
||||
if private_corpus:
|
||||
store.ensure_private_db_files()
|
||||
topic_row = store.add_topic(report.topic)
|
||||
topic_id = topic_row["id"]
|
||||
source_mode = ",".join(sorted(report.items_by_source)) or "v3"
|
||||
run_id = store.record_run(topic_id, source_mode=source_mode, status="running")
|
||||
try:
|
||||
findings = store.findings_from_report(report)
|
||||
if private_corpus:
|
||||
store.ensure_private_db_files()
|
||||
counts = store.store_findings(run_id, topic_id, findings)
|
||||
store.update_run(
|
||||
run_id,
|
||||
@@ -403,6 +459,9 @@ def persist_report(report: schema.Report, store_db: Path | None = None) -> dict[
|
||||
except Exception as exc:
|
||||
store.update_run(run_id, status="failed", error_message=str(exc)[:500])
|
||||
raise
|
||||
finally:
|
||||
if private_corpus:
|
||||
store.ensure_private_db_files()
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
@@ -466,6 +525,18 @@ def build_parser() -> argparse.ArgumentParser:
|
||||
parser.add_argument("--no-browser-cookies", action="store_true",
|
||||
help="Disable browser-cookie extraction even when FROM_BROWSER is configured")
|
||||
parser.add_argument("--save-dir", help="Optional directory for saving the rendered output")
|
||||
parser.add_argument(
|
||||
"--corpus",
|
||||
action="append",
|
||||
default=[],
|
||||
metavar="DIR",
|
||||
help="Add a local .md/.txt/.pdf directory as a private ranked source (repeatable)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--corpus-all-time",
|
||||
action="store_true",
|
||||
help="Include matching corpus files older than the research window",
|
||||
)
|
||||
parser.add_argument("--output", help="Optional exact file path for saving the rendered output")
|
||||
parser.add_argument("--synthesis-file", help="Markdown synthesis to embed in --emit=html output")
|
||||
parser.add_argument("--publish-html", action="store_true",
|
||||
@@ -827,7 +898,12 @@ def _write_last_run(
|
||||
if env.CONFIG_DIR is None:
|
||||
return False
|
||||
target = env.CONFIG_DIR
|
||||
target.mkdir(parents=True, exist_ok=True)
|
||||
cached_reports = entity_reports or [(report.topic, report)]
|
||||
has_private_corpus = any(
|
||||
cached_report.items_by_source.get("corpus")
|
||||
for _, cached_report in cached_reports
|
||||
)
|
||||
_ensure_output_directory(target, private=has_private_corpus)
|
||||
counts = {source: len(items) for source, items in report.items_by_source.items()}
|
||||
payload = {
|
||||
"topic": topic,
|
||||
@@ -838,7 +914,6 @@ def _write_last_run(
|
||||
"comparison": bool(entity_reports),
|
||||
}
|
||||
(target / "last-run.json").write_text(json.dumps(payload, indent=2))
|
||||
cached_reports = entity_reports or [(report.topic, report)]
|
||||
cache_payload = {
|
||||
"schema": REPORT_CACHE_VERSION,
|
||||
"topic": topic,
|
||||
@@ -849,7 +924,10 @@ def _write_last_run(
|
||||
for label, cached_report in cached_reports
|
||||
],
|
||||
}
|
||||
(target / "last-report.json").write_text(json.dumps(cache_payload, indent=2))
|
||||
report_cache_path = target / "last-report.json"
|
||||
report_cache_path.write_text(json.dumps(cache_payload, indent=2))
|
||||
if has_private_corpus:
|
||||
report_cache_path.chmod(0o600)
|
||||
return True
|
||||
except Exception as exc:
|
||||
# Never fatal, but never silent either (#787's lesson): callers that
|
||||
@@ -1105,6 +1183,8 @@ def _run_drill(
|
||||
if "trustpilot" in sources else None
|
||||
),
|
||||
internal_subrun=True,
|
||||
corpus_dirs=args.corpus,
|
||||
corpus_all_time=args.corpus_all_time,
|
||||
)
|
||||
except Exception:
|
||||
progress.end_processing()
|
||||
@@ -1388,9 +1468,18 @@ def _render_save_and_print(
|
||||
json_profile=args.json_profile,
|
||||
register=audience.name,
|
||||
)
|
||||
has_private_corpus = _report_has_private_corpus(report) or bool(
|
||||
entity_reports
|
||||
and any(_report_has_private_corpus(entity) for _label, entity in entity_reports)
|
||||
)
|
||||
private_saved_format = has_private_corpus
|
||||
publish_companion_paths: list[Path] = []
|
||||
if args.output:
|
||||
output_path = save_rendered_output(rendered, args.output)
|
||||
output_path = save_rendered_output(
|
||||
rendered,
|
||||
args.output,
|
||||
private=private_saved_format,
|
||||
)
|
||||
if args.emit == "html":
|
||||
publish_companion_paths.append(output_path)
|
||||
sys.stderr.write(f"[last30days] Saved output to {output_path}\n")
|
||||
@@ -1407,6 +1496,7 @@ def _render_save_and_print(
|
||||
rendered_content=rendered if is_comparison_html else None,
|
||||
json_profile=args.json_profile,
|
||||
register=audience.name,
|
||||
private=private_saved_format,
|
||||
)
|
||||
if args.emit == "html":
|
||||
publish_companion_paths.append(save_path)
|
||||
@@ -1421,6 +1511,7 @@ def _render_save_and_print(
|
||||
suffix=args.save_suffix or "",
|
||||
synthesis_md=synthesis_md,
|
||||
json_profile=args.json_profile,
|
||||
private=_report_has_private_corpus(entity_report),
|
||||
)
|
||||
comparison_peer_paths.append(peer_path)
|
||||
sys.stderr.write(f"[last30days] Saved output to {peer_path}\n")
|
||||
@@ -1432,8 +1523,39 @@ def _render_save_and_print(
|
||||
sys.stderr.flush()
|
||||
if args.publish_html:
|
||||
try:
|
||||
has_private_corpus = "corpus" in report.source_status or bool(
|
||||
entity_reports
|
||||
and any("corpus" in entity.source_status for _label, entity in entity_reports)
|
||||
)
|
||||
publish_rendered = rendered
|
||||
if has_private_corpus:
|
||||
sys.stderr.write(
|
||||
"[last30days] Excluding local corpus evidence and synthesis from published HTML.\n"
|
||||
)
|
||||
if entity_reports:
|
||||
publish_rendered = emit_comparison_output(
|
||||
[
|
||||
(label, schema.without_sources(entity, {"corpus"}))
|
||||
for label, entity in entity_reports
|
||||
],
|
||||
"html",
|
||||
fun_level=fun_level,
|
||||
save_path=footer_save_path,
|
||||
synthesis_md=None,
|
||||
json_profile=args.json_profile,
|
||||
)
|
||||
else:
|
||||
publish_rendered = emit_output(
|
||||
schema.without_sources(report, {"corpus"}),
|
||||
"html",
|
||||
fun_level=fun_level,
|
||||
save_path=footer_save_path,
|
||||
synthesis_md=None,
|
||||
json_profile=args.json_profile,
|
||||
register=audience.name,
|
||||
)
|
||||
publish_result = publish_rendered_html(
|
||||
rendered,
|
||||
publish_rendered,
|
||||
password=_publish_password_for_args(args, config),
|
||||
companion_paths=publish_companion_paths,
|
||||
)
|
||||
@@ -1594,7 +1716,10 @@ def _run_library_feed(args: argparse.Namespace, config: dict[str, object]) -> in
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
library_id = library.get_or_create_library_id(output_dir)
|
||||
rendered_briefs_dir = output_dir / "briefs"
|
||||
rendered_briefs_dir.mkdir(parents=True, exist_ok=True)
|
||||
has_private_entries = any(
|
||||
render.PRIVATE_CORPUS_START in entry.content for entry in entries
|
||||
)
|
||||
_ensure_output_directory(rendered_briefs_dir, private=has_private_entries)
|
||||
|
||||
def _preserve_hand_written_page(existing_path: Path, generated_marker: str) -> None:
|
||||
"""Back up any page library feed did not generate before overwriting it."""
|
||||
@@ -1617,13 +1742,19 @@ def _run_library_feed(args: argparse.Namespace, config: dict[str, object]) -> in
|
||||
f"library feed; preserved the original at {backup.name}\n"
|
||||
)
|
||||
|
||||
brief_documents: dict[str, str] = {}
|
||||
publishable_brief_documents: dict[str, str] = {}
|
||||
for entry in entries:
|
||||
rendered = html_render.render_library_brief(entry)
|
||||
target = rendered_briefs_dir / entry.output_name
|
||||
_preserve_hand_written_page(target, html_render.LIBRARY_BRIEF_MARKER)
|
||||
target.write_text(rendered, encoding="utf-8")
|
||||
brief_documents[entry.entry_id] = rendered
|
||||
save_rendered_output(
|
||||
rendered,
|
||||
str(target),
|
||||
private=render.PRIVATE_CORPUS_START in entry.content,
|
||||
)
|
||||
publishable_brief_documents[entry.entry_id] = html_render.render_library_brief(
|
||||
entry, include_private=False
|
||||
)
|
||||
|
||||
current_brief_names = {entry.output_name for entry in entries}
|
||||
for path in rendered_briefs_dir.glob("*.html"):
|
||||
@@ -1664,7 +1795,7 @@ def _run_library_feed(args: argparse.Namespace, config: dict[str, object]) -> in
|
||||
entry_urls: dict[str, str] = {}
|
||||
try:
|
||||
brief_results = html_publish.publish_html_documents(
|
||||
brief_documents,
|
||||
publishable_brief_documents,
|
||||
password=password,
|
||||
)
|
||||
entry_urls = {
|
||||
@@ -1810,6 +1941,23 @@ def _main(
|
||||
)
|
||||
return 2
|
||||
config = env.get_config(policy=_config_policy_for_args(args, topic, extra_argv))
|
||||
resolved_corpus_dirs = corpus.resolve_directories(
|
||||
args.corpus, config.get("LAST30DAYS_CORPUS_DIRS")
|
||||
)
|
||||
# EXCLUDE_SOURCES=corpus disables corpus retrieval entirely; the hosted
|
||||
# privacy bypass below must use the same predicate, or hosted users with
|
||||
# configured-but-excluded dirs silently lose the remote backend.
|
||||
excluded_sources = {
|
||||
value.strip().lower()
|
||||
for value in str(config.get("EXCLUDE_SOURCES") or "").split(",")
|
||||
if value.strip()
|
||||
}
|
||||
if "corpus" in excluded_sources:
|
||||
resolved_corpus_dirs = []
|
||||
if resolved_corpus_dirs:
|
||||
config["_CORPUS_DIRS"] = [str(path) for path in resolved_corpus_dirs]
|
||||
if _config_truthy(config.get("LAST30DAYS_CORPUS_IN_EXPORT")):
|
||||
config["_CORPUS_IN_EXPORT"] = True
|
||||
_propagate_config_to_environ(config)
|
||||
|
||||
# Env-var fallback for --save-dir, mirroring the LAST30DAYS_STORE pattern below.
|
||||
@@ -1979,6 +2127,15 @@ def _main(
|
||||
# instead of local sources; no local provider keys are needed (see
|
||||
# lib/hosted.py). With either env var unset, behavior below is byte-identical
|
||||
# to local-only runs - there is no built-in endpoint.
|
||||
if (
|
||||
topic
|
||||
and resolved_corpus_dirs
|
||||
and env.read_secret_env("LAST30DAYS_API_KEY")
|
||||
and os.environ.get("LAST30DAYS_API_BASE")
|
||||
):
|
||||
sys.stderr.write(
|
||||
"[last30days] Local corpus configured; bypassing the hosted backend so files stay on this machine.\n"
|
||||
)
|
||||
if (
|
||||
topic
|
||||
and not args.diagnose
|
||||
@@ -1986,6 +2143,7 @@ def _main(
|
||||
and not args.record_fixtures
|
||||
and env.read_secret_env("LAST30DAYS_API_KEY")
|
||||
and os.environ.get("LAST30DAYS_API_BASE")
|
||||
and not resolved_corpus_dirs
|
||||
):
|
||||
if _freshness_enabled(args, config):
|
||||
if args.verify_freshness is True:
|
||||
@@ -2238,6 +2396,8 @@ def _main(
|
||||
internal_subrun=comp_enabled,
|
||||
hiring_signals_mode=args.hiring_signals,
|
||||
save_dir=args.save_dir,
|
||||
corpus_dirs=args.corpus,
|
||||
corpus_all_time=args.corpus_all_time,
|
||||
)
|
||||
r.artifacts["resolved"] = {
|
||||
"entity": topic,
|
||||
@@ -2375,6 +2535,8 @@ def _main(
|
||||
hiring_signals_mode=args.hiring_signals,
|
||||
internal_subrun=True,
|
||||
save_dir=args.save_dir,
|
||||
corpus_dirs=args.corpus,
|
||||
corpus_all_time=args.corpus_all_time,
|
||||
)
|
||||
report.artifacts["resolved"] = resolved_effective
|
||||
return report
|
||||
|
||||
@@ -0,0 +1,439 @@
|
||||
"""Deterministic, local-only document corpus source.
|
||||
|
||||
The corpus adapter deliberately has no HTTP dependency. It scans explicitly
|
||||
registered directories, extracts small text documents (and PDFs only when the
|
||||
local ``pdftotext`` binary is available), and returns normalized ``SourceItem``
|
||||
objects for the shared relevance/fusion pipeline.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import threading
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from shutil import which
|
||||
from typing import Any, Iterable
|
||||
|
||||
from . import entity_extract, log, relevance, schema
|
||||
|
||||
SOURCE = "corpus"
|
||||
SUPPORTED_SUFFIXES = {".md", ".txt", ".pdf"}
|
||||
IGNORED_DIRECTORIES = {".git", "node_modules"}
|
||||
MAX_FILES = 500
|
||||
MAX_TEXT_CHARS = 1_000_000
|
||||
MAX_CACHE_TEXT_CHARS = MAX_TEXT_CHARS
|
||||
MAX_CACHE_BYTES = 50 * 1024 * 1024
|
||||
MAX_CACHE_ENTRIES = 2_000
|
||||
CACHE_FILENAME = "corpus-cache.json"
|
||||
CACHE_SCHEMA_VERSION = "last30days-corpus-cache/v2"
|
||||
|
||||
_CACHE_LOCK = threading.Lock()
|
||||
|
||||
|
||||
@dataclass
|
||||
class CorpusScanResult:
|
||||
"""One bounded scan, including non-fatal extraction notes."""
|
||||
|
||||
items: list[schema.SourceItem]
|
||||
notes: list[str] = field(default_factory=list)
|
||||
files_scanned: int = 0
|
||||
cache_hits: int = 0
|
||||
|
||||
|
||||
def resolve_directories(
|
||||
cli_directories: Iterable[str] | None,
|
||||
configured: str | Iterable[str] | None,
|
||||
) -> list[Path]:
|
||||
"""Merge repeatable CLI paths with ``os.pathsep``-separated config paths."""
|
||||
raw: list[str] = [str(value) for value in (cli_directories or []) if str(value).strip()]
|
||||
if isinstance(configured, str):
|
||||
raw.extend(value for value in configured.split(os.pathsep) if value.strip())
|
||||
elif configured:
|
||||
raw.extend(str(value) for value in configured if str(value).strip())
|
||||
|
||||
resolved: list[Path] = []
|
||||
seen: set[str] = set()
|
||||
for value in raw:
|
||||
path = Path(value.strip()).expanduser().resolve()
|
||||
key = os.path.normcase(str(path))
|
||||
if key in seen:
|
||||
continue
|
||||
seen.add(key)
|
||||
resolved.append(path)
|
||||
return resolved
|
||||
|
||||
|
||||
def _safe_error(exc: BaseException) -> str:
|
||||
"""Describe an error without str(exc), which embeds absolute paths.
|
||||
|
||||
These notes travel into source_status detail and render in coverage
|
||||
diagnostics outside the private corpus block.
|
||||
"""
|
||||
reason = getattr(exc, "strerror", None)
|
||||
return str(reason) if reason else exc.__class__.__name__
|
||||
|
||||
|
||||
def search(
|
||||
topic: str,
|
||||
directories: Iterable[Path | str],
|
||||
*,
|
||||
from_date: str,
|
||||
to_date: str,
|
||||
all_time: bool = False,
|
||||
limit: int = 12,
|
||||
cache_dir: Path | None = None,
|
||||
) -> CorpusScanResult:
|
||||
"""Search registered directories without making any network calls."""
|
||||
roots = resolve_directories([str(path) for path in directories], None)
|
||||
notes: list[str] = []
|
||||
cache_path = cache_dir / CACHE_FILENAME if cache_dir is not None else None
|
||||
with _CACHE_LOCK:
|
||||
cache = _load_cache(cache_path)
|
||||
cache_entries = cache.setdefault("entries", {})
|
||||
cache_entry_sizes = {
|
||||
path: _cache_entry_fragment_size(path, value)
|
||||
for path, value in cache_entries.items()
|
||||
}
|
||||
|
||||
candidates: list[tuple[float, int, schema.SourceItem]] = []
|
||||
seen_files: set[str] = set()
|
||||
files_scanned = 0
|
||||
cache_hits = 0
|
||||
pdf_available = which("pdftotext")
|
||||
pdf_unavailable_noted = False
|
||||
|
||||
readable_roots: list[Path] = []
|
||||
for root in roots:
|
||||
if not root.is_dir():
|
||||
notes.append(f"Skipped corpus root '{Path(root).name}': not a readable directory")
|
||||
continue
|
||||
readable_roots.append(root)
|
||||
|
||||
per_root_limit, extra_slots = divmod(MAX_FILES, len(readable_roots) or 1)
|
||||
scan_limit_reached = False
|
||||
for root_index, root in enumerate(readable_roots):
|
||||
root_limit = per_root_limit + (1 if root_index < extra_slots else 0)
|
||||
root_files_scanned = 0
|
||||
for path in _iter_files(root, notes=notes):
|
||||
if root_files_scanned >= root_limit:
|
||||
scan_limit_reached = True
|
||||
break
|
||||
key = os.path.normcase(str(path))
|
||||
if key in seen_files:
|
||||
continue
|
||||
seen_files.add(key)
|
||||
root_files_scanned += 1
|
||||
files_scanned += 1
|
||||
|
||||
try:
|
||||
stat = path.stat()
|
||||
except OSError as exc:
|
||||
notes.append(f"Skipped {_display_path(path, root)}: {_safe_error(exc)}")
|
||||
continue
|
||||
published_at = datetime.fromtimestamp(
|
||||
stat.st_mtime, tz=timezone.utc
|
||||
).date().isoformat()
|
||||
if not all_time and not (from_date <= published_at <= to_date):
|
||||
continue
|
||||
|
||||
cached = cache_entries.get(str(path))
|
||||
if (
|
||||
isinstance(cached, dict)
|
||||
and cached.get("mtime_ns") == stat.st_mtime_ns
|
||||
and cached.get("size") == stat.st_size
|
||||
and isinstance(cached.get("text"), str)
|
||||
):
|
||||
text = cached["text"]
|
||||
cache_hits += 1
|
||||
else:
|
||||
if path.suffix.lower() == ".pdf" and not pdf_available:
|
||||
if not pdf_unavailable_noted:
|
||||
notes.append("Skipped PDF files because pdftotext is not on PATH")
|
||||
pdf_unavailable_noted = True
|
||||
continue
|
||||
try:
|
||||
text = _extract_text(path, pdftotext=pdf_available)
|
||||
except (OSError, subprocess.SubprocessError) as exc:
|
||||
notes.append(f"Skipped {_display_path(path, root)}: {_safe_error(exc)}")
|
||||
continue
|
||||
_cache_entry_put(cache_entries, cache_entry_sizes, str(path), {
|
||||
"mtime_ns": stat.st_mtime_ns,
|
||||
"size": stat.st_size,
|
||||
"text": text[:MAX_CACHE_TEXT_CHARS],
|
||||
})
|
||||
|
||||
title = _path_title(path)
|
||||
score = _match_score(topic, f"{title}\n{text}")
|
||||
if score < 0.15:
|
||||
continue
|
||||
relative_path = str(path.relative_to(root))
|
||||
path_digest = hashlib.sha256(str(path).encode("utf-8")).hexdigest()
|
||||
item = schema.SourceItem(
|
||||
item_id=f"C{path_digest[:12]}",
|
||||
source=SOURCE,
|
||||
title=title,
|
||||
body=text,
|
||||
url=f"corpus://{path_digest}",
|
||||
container=str(path.parent),
|
||||
published_at=published_at,
|
||||
date_confidence="high",
|
||||
relevance_hint=score,
|
||||
why_relevant=f"Matched local file {relative_path}",
|
||||
# Leave empty so extract_best_snippet derives the matching
|
||||
# window; a file-prefix snippet is preserved verbatim and can
|
||||
# show unrelated intro text (and draw entity-miss demotion).
|
||||
snippet="",
|
||||
metadata={
|
||||
"path": str(path),
|
||||
"relative_path": relative_path,
|
||||
"extension": path.suffix.lower(),
|
||||
"local_only": True,
|
||||
},
|
||||
)
|
||||
candidates.append((score, stat.st_mtime_ns, item))
|
||||
if scan_limit_reached:
|
||||
notes.append(f"Stopped after the {MAX_FILES}-file corpus scan limit")
|
||||
|
||||
cache["schema_version"] = CACHE_SCHEMA_VERSION
|
||||
cache["entries"] = _bounded_entries(cache_entries)
|
||||
with _CACHE_LOCK:
|
||||
_write_cache(cache_path, cache, notes)
|
||||
|
||||
candidates.sort(key=lambda row: (-row[0], -row[1], row[2].title.casefold()))
|
||||
items = [item for _score, _mtime, item in candidates[: max(0, limit)]]
|
||||
log.source_log(
|
||||
"Corpus",
|
||||
f"scanned {files_scanned} file(s), {cache_hits} cache hit(s), {len(items)} match(es)",
|
||||
tty_only=False,
|
||||
)
|
||||
return CorpusScanResult(
|
||||
items=items,
|
||||
notes=notes,
|
||||
files_scanned=files_scanned,
|
||||
cache_hits=cache_hits,
|
||||
)
|
||||
|
||||
|
||||
def _display_path(path: Path | str, root: Path | None = None) -> str:
|
||||
"""Render a note-safe path: never the absolute local path.
|
||||
|
||||
Corpus notes flow into source_status detail and the Partial Coverage
|
||||
block, which render OUTSIDE the private corpus markers - an absolute
|
||||
path like /home/user/private/notes/foo.md must not escape there.
|
||||
"""
|
||||
candidate = Path(path)
|
||||
if root is not None:
|
||||
try:
|
||||
return str(Path(root).name / candidate.relative_to(root))
|
||||
except ValueError:
|
||||
pass
|
||||
return candidate.name
|
||||
|
||||
|
||||
def _iter_files(root: Path, notes: list[str] | None = None) -> Iterable[Path]:
|
||||
# Bounded newest-first selection: keep only the newest MAX_FILES paths in a
|
||||
# heap while walking, so registering a huge tree does not materialize every
|
||||
# path before the caller's extraction cap applies.
|
||||
import heapq
|
||||
|
||||
heap: list[tuple[int, str]] = []
|
||||
walk_errors = 0
|
||||
|
||||
def _on_walk_error(error: OSError) -> None:
|
||||
nonlocal walk_errors
|
||||
walk_errors += 1
|
||||
if notes is not None and walk_errors <= 3:
|
||||
unreadable = _display_path(error.filename, root) if error.filename else Path(root).name
|
||||
notes.append(f"corpus: could not read {unreadable}: {error.strerror}")
|
||||
|
||||
for current, directory_names, file_names in os.walk(
|
||||
root, followlinks=False, onerror=_on_walk_error
|
||||
):
|
||||
directory_names[:] = sorted(
|
||||
name
|
||||
for name in directory_names
|
||||
if name not in IGNORED_DIRECTORIES and not name.startswith(".")
|
||||
)
|
||||
current_path = Path(current)
|
||||
for name in sorted(file_names):
|
||||
if name.startswith("."):
|
||||
continue
|
||||
path = current_path / name
|
||||
if path.suffix.lower() in SUPPORTED_SUFFIXES and not path.is_symlink():
|
||||
entry = (_safe_mtime_ns(path), str(path))
|
||||
if len(heap) < MAX_FILES:
|
||||
heapq.heappush(heap, entry)
|
||||
else:
|
||||
heapq.heappushpop(heap, entry)
|
||||
if notes is not None and walk_errors > 3:
|
||||
notes.append(f"corpus: {walk_errors - 3} more unreadable directories suppressed")
|
||||
ordered = sorted(heap, key=lambda item: (-item[0], item[1].casefold()))
|
||||
for _mtime, raw_path in ordered:
|
||||
yield Path(raw_path)
|
||||
|
||||
|
||||
def _safe_mtime_ns(path: Path) -> int:
|
||||
try:
|
||||
return path.stat().st_mtime_ns
|
||||
except OSError:
|
||||
return 0
|
||||
|
||||
|
||||
def _extract_text(path: Path, *, pdftotext: str | None) -> str:
|
||||
if path.suffix.lower() == ".pdf":
|
||||
if not pdftotext:
|
||||
return ""
|
||||
completed = subprocess.run(
|
||||
[pdftotext, str(path), "-"],
|
||||
capture_output=True,
|
||||
check=True,
|
||||
text=True,
|
||||
timeout=20,
|
||||
)
|
||||
return completed.stdout[:MAX_TEXT_CHARS]
|
||||
with path.open("r", encoding="utf-8", errors="replace") as handle:
|
||||
return handle.read(MAX_TEXT_CHARS)
|
||||
|
||||
|
||||
def _path_title(path: Path) -> str:
|
||||
title = path.stem.replace("_", " ").replace("-", " ")
|
||||
return " ".join(title.split()) or path.name
|
||||
|
||||
|
||||
def _match_score(topic: str, text: str) -> float:
|
||||
lexical = relevance.token_overlap_relevance(topic, text)
|
||||
topic_entities = entity_extract.extract_text_entities(topic)
|
||||
text_entities = entity_extract.extract_text_entities(text)
|
||||
entity_score = entity_extract.entity_overlap(topic_entities, text_entities)
|
||||
return round(max(lexical, entity_score * 0.9), 4)
|
||||
|
||||
|
||||
def _load_cache(path: Path | None) -> dict[str, Any]:
|
||||
if path is None:
|
||||
return {"schema_version": CACHE_SCHEMA_VERSION, "entries": {}}
|
||||
try:
|
||||
if path.stat().st_size > MAX_CACHE_BYTES:
|
||||
return {"schema_version": CACHE_SCHEMA_VERSION, "entries": {}}
|
||||
payload = json.loads(path.read_text(encoding="utf-8"))
|
||||
except (OSError, UnicodeError, json.JSONDecodeError):
|
||||
return {"schema_version": CACHE_SCHEMA_VERSION, "entries": {}}
|
||||
if not isinstance(payload, dict) or payload.get("schema_version") != CACHE_SCHEMA_VERSION:
|
||||
return {"schema_version": CACHE_SCHEMA_VERSION, "entries": {}}
|
||||
if not isinstance(payload.get("entries"), dict):
|
||||
payload["entries"] = {}
|
||||
payload["entries"] = _bounded_entries(payload["entries"])
|
||||
return payload
|
||||
|
||||
|
||||
def _bounded_entries(entries: Any) -> dict[str, Any]:
|
||||
if not isinstance(entries, dict):
|
||||
return {}
|
||||
ordered = sorted(
|
||||
(
|
||||
(path, value)
|
||||
for path, value in entries.items()
|
||||
if (
|
||||
isinstance(path, str)
|
||||
and isinstance(value, dict)
|
||||
and isinstance(value.get("text"), str)
|
||||
)
|
||||
),
|
||||
key=lambda row: int(row[1].get("mtime_ns") or 0),
|
||||
reverse=True,
|
||||
)
|
||||
base_bytes = len(
|
||||
json.dumps(
|
||||
{"schema_version": CACHE_SCHEMA_VERSION, "entries": {}},
|
||||
ensure_ascii=False,
|
||||
).encode("utf-8")
|
||||
)
|
||||
used_bytes = base_bytes
|
||||
bounded: dict[str, Any] = {}
|
||||
for path, value in ordered[:MAX_CACHE_ENTRIES]:
|
||||
normalized = {
|
||||
"mtime_ns": value.get("mtime_ns"),
|
||||
"size": value.get("size"),
|
||||
"text": value["text"][:MAX_CACHE_TEXT_CHARS],
|
||||
}
|
||||
fragment = json.dumps({path: normalized}, ensure_ascii=False).encode("utf-8")
|
||||
fragment_bytes = len(fragment) - 2 + (2 if bounded else 0)
|
||||
if used_bytes + fragment_bytes > MAX_CACHE_BYTES:
|
||||
continue
|
||||
bounded[path] = normalized
|
||||
used_bytes += fragment_bytes
|
||||
return bounded
|
||||
|
||||
|
||||
def _cache_entry_fragment_size(path: str, value: dict[str, Any]) -> int:
|
||||
return len(json.dumps({path: value}, ensure_ascii=False).encode("utf-8")) - 2
|
||||
|
||||
|
||||
def _cache_entry_put(
|
||||
entries: dict[str, Any],
|
||||
sizes: dict[str, int],
|
||||
path: str,
|
||||
value: dict[str, Any],
|
||||
) -> None:
|
||||
entries[path] = value
|
||||
sizes[path] = _cache_entry_fragment_size(path, value)
|
||||
while (
|
||||
len(entries) > MAX_CACHE_ENTRIES
|
||||
or _cache_payload_size(sizes) > MAX_CACHE_BYTES
|
||||
):
|
||||
oldest = min(
|
||||
entries,
|
||||
key=lambda candidate: (
|
||||
int(entries[candidate].get("mtime_ns") or 0),
|
||||
candidate,
|
||||
),
|
||||
)
|
||||
del entries[oldest]
|
||||
del sizes[oldest]
|
||||
|
||||
|
||||
def _cache_payload_size(sizes: dict[str, int]) -> int:
|
||||
base_bytes = len(
|
||||
json.dumps(
|
||||
{"schema_version": CACHE_SCHEMA_VERSION, "entries": {}},
|
||||
ensure_ascii=False,
|
||||
).encode("utf-8")
|
||||
)
|
||||
separators = max(0, len(sizes) - 1) * 2
|
||||
return base_bytes + sum(sizes.values()) + separators
|
||||
|
||||
|
||||
def _write_cache(path: Path | None, payload: dict[str, Any], notes: list[str]) -> None:
|
||||
if path is None:
|
||||
return
|
||||
try:
|
||||
_ensure_private_directory(path.parent)
|
||||
payload["entries"] = _bounded_entries(payload.get("entries", {}))
|
||||
encoded = json.dumps(payload, ensure_ascii=False).encode("utf-8")
|
||||
temporary = path.with_name(f".{path.name}.{os.getpid()}.tmp")
|
||||
try:
|
||||
fd = os.open(temporary, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
|
||||
except FileExistsError:
|
||||
temporary.unlink()
|
||||
fd = os.open(temporary, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
|
||||
with os.fdopen(fd, "wb") as handle:
|
||||
handle.write(encoded)
|
||||
temporary.replace(path)
|
||||
path.chmod(0o600)
|
||||
except OSError as exc:
|
||||
notes.append(f"Corpus cache unavailable: {_safe_error(exc)}")
|
||||
|
||||
|
||||
def _ensure_private_directory(path: Path) -> None:
|
||||
missing: list[Path] = []
|
||||
current = path
|
||||
while not current.exists():
|
||||
missing.append(current)
|
||||
current = current.parent
|
||||
path.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
for directory in missing:
|
||||
directory.chmod(0o700)
|
||||
@@ -470,6 +470,12 @@ def get_config(policy: ConfigLoadPolicy | None = None) -> dict[str, Any]:
|
||||
# degraded (neither ok, no-results, nor skipped-unconfigured). #384.
|
||||
('LAST30DAYS_STRICT_EXIT', None),
|
||||
('LAST30DAYS_MEMORY_DIR', None),
|
||||
# Optional local-only evidence source. Paths are separated with the
|
||||
# platform path separator (":" on macOS/Linux, ";" on Windows).
|
||||
('LAST30DAYS_CORPUS_DIRS', None),
|
||||
# Corpus evidence is omitted from the stable agent JSON export unless
|
||||
# this explicit privacy opt-in is truthy.
|
||||
('LAST30DAYS_CORPUS_IN_EXPORT', None),
|
||||
('LAST30DAYS_LIBRARY_OWNER', None),
|
||||
('LAST30DAYS_LIBRARY_CONTEXT', 'on'),
|
||||
('LAST30DAYS_PUBLISH_PASSWORD', None),
|
||||
|
||||
@@ -438,10 +438,12 @@ def render_html_comparison(
|
||||
LIBRARY_BRIEF_MARKER = "<!-- generated by last30days library feed -->"
|
||||
|
||||
|
||||
def render_library_brief(entry: LibraryEntry) -> str:
|
||||
def render_library_brief(entry: LibraryEntry, *, include_private: bool = True) -> str:
|
||||
"""Render a scanned Markdown or JSON briefing as a safe standalone page."""
|
||||
md = _strip_invitation(entry.content)
|
||||
md = _strip_canonical_boundary(md)
|
||||
if not include_private:
|
||||
md = _strip_private_corpus(md)
|
||||
body = _markdown_to_html(md)
|
||||
body = _wrap_engine_footer(body)
|
||||
colophon = (
|
||||
@@ -458,6 +460,18 @@ def render_library_brief(entry: LibraryEntry) -> str:
|
||||
return rendered.replace("</body>", f"{LIBRARY_BRIEF_MARKER}\n</body>", 1)
|
||||
|
||||
|
||||
_PRIVATE_CORPUS_BLOCK = re.compile(
|
||||
r"<!-- LAST30DAYS_PRIVATE_CORPUS_START -->.*?"
|
||||
r"<!-- LAST30DAYS_PRIVATE_CORPUS_END -->\s*",
|
||||
re.DOTALL,
|
||||
)
|
||||
|
||||
|
||||
def _strip_private_corpus(markdown: str) -> str:
|
||||
"""Remove the renderer-marked local corpus section before publication."""
|
||||
return _PRIVATE_CORPUS_BLOCK.sub("", markdown)
|
||||
|
||||
|
||||
def render_library_index(
|
||||
entries: Sequence[LibraryEntry],
|
||||
*,
|
||||
|
||||
@@ -29,6 +29,11 @@ _LIBRARY_ID = re.compile(r"[0-9a-f]{32}")
|
||||
_GENERATED_BRIEF_NAME = re.compile(
|
||||
r"[a-z0-9]+(?:-[a-z0-9]+)*-[0-9a-f]{8}-\d{4}-\d{2}-\d{2}\.html"
|
||||
)
|
||||
_PRIVATE_CORPUS_BLOCK = re.compile(
|
||||
r"<!-- LAST30DAYS_PRIVATE_CORPUS_START -->.*?"
|
||||
r"<!-- LAST30DAYS_PRIVATE_CORPUS_END -->\s*",
|
||||
re.DOTALL,
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
@@ -140,15 +145,16 @@ def _keep_preferred(entries: dict[str, LibraryEntry], entry: LibraryEntry) -> No
|
||||
|
||||
def _parse_markdown(path: Path) -> LibraryEntry:
|
||||
content = path.read_text(encoding="utf-8")
|
||||
title_match = _REPORT_TITLE.search(content) or _FIRST_TITLE.search(content)
|
||||
public_content = _PRIVATE_CORPUS_BLOCK.sub("", content)
|
||||
title_match = _REPORT_TITLE.search(public_content) or _FIRST_TITLE.search(public_content)
|
||||
if not title_match:
|
||||
raise ValueError("no Markdown title found")
|
||||
topic = _clean_inline(title_match.group(1))
|
||||
if not topic:
|
||||
raise ValueError("empty Markdown title")
|
||||
published_date = _markdown_date(content, path)
|
||||
headline = _markdown_headline(content) or topic
|
||||
summary = _markdown_summary(content) or headline
|
||||
published_date = _markdown_date(public_content, path)
|
||||
headline = _markdown_headline(public_content) or topic
|
||||
summary = _markdown_summary(public_content) or headline
|
||||
return LibraryEntry(
|
||||
slug=slugify(topic),
|
||||
topic=topic,
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import os
|
||||
import re
|
||||
import sqlite3
|
||||
from dataclasses import dataclass, replace
|
||||
@@ -14,6 +15,7 @@ from . import library
|
||||
|
||||
DEFAULT_LIBRARY_DB = library.DEFAULT_BRIEFS_DIR.parent / "library.db"
|
||||
DEFAULT_STORE_DB = library.DEFAULT_BRIEFS_DIR.parent / "research.db"
|
||||
INDEX_FINGERPRINT_VERSION = "last30days-library-index/v2"
|
||||
LIBRARY_CONTEXT_START = "<!-- last30days:library-context:start -->"
|
||||
LIBRARY_CONTEXT_END = "<!-- last30days:library-context:end -->"
|
||||
_TOKEN = re.compile(r"[^\W_]+", re.UNICODE)
|
||||
@@ -26,6 +28,11 @@ _LEGACY_LIBRARY_CONTEXT = re.compile(
|
||||
r"^## From your library\s*$.*?(?=^##\s|\Z)",
|
||||
re.MULTILINE | re.DOTALL,
|
||||
)
|
||||
_PRIVATE_CORPUS_BLOCK = re.compile(
|
||||
r"<!-- LAST30DAYS_PRIVATE_CORPUS_START -->.*?"
|
||||
r"<!-- LAST30DAYS_PRIVATE_CORPUS_END -->\s*",
|
||||
re.DOTALL,
|
||||
)
|
||||
|
||||
|
||||
class LibrarySearchUnavailable(RuntimeError):
|
||||
@@ -131,7 +138,7 @@ def index_brief(
|
||||
if entry is None:
|
||||
return False
|
||||
target = Path(db_path).expanduser()
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
_ensure_private_directory(target.parent)
|
||||
with _connect(target) as conn:
|
||||
_upsert_entry(conn, entry)
|
||||
conn.commit()
|
||||
@@ -209,7 +216,7 @@ def _sync_library(
|
||||
db_path: Path,
|
||||
) -> SyncResult:
|
||||
entries, notes = library.scan_library(memory_dir, briefs_dir)
|
||||
db_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
_ensure_private_directory(db_path.parent)
|
||||
indexed = unchanged = 0
|
||||
with _connect(db_path) as conn:
|
||||
existing = {
|
||||
@@ -250,6 +257,15 @@ def _sync_library(
|
||||
|
||||
|
||||
def _connect(path: Path) -> sqlite3.Connection:
|
||||
_ensure_private_directory(path.parent)
|
||||
if not path.exists():
|
||||
try:
|
||||
fd = os.open(path, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
|
||||
except FileExistsError:
|
||||
pass
|
||||
else:
|
||||
os.close(fd)
|
||||
path.chmod(0o600)
|
||||
conn = sqlite3.connect(str(path))
|
||||
try:
|
||||
conn.row_factory = sqlite3.Row
|
||||
@@ -268,7 +284,13 @@ def _upsert_entry(
|
||||
fingerprint: str | None = None,
|
||||
) -> None:
|
||||
stat = entry.source_path.stat()
|
||||
indexed_content = _indexable_content(entry.content)
|
||||
private_free_content = _PRIVATE_CORPUS_BLOCK.sub("", entry.content)
|
||||
indexed_content = _indexable_content(private_free_content)
|
||||
headline = entry.headline
|
||||
summary = entry.summary
|
||||
if private_free_content != entry.content and entry.source_format == "markdown":
|
||||
headline = library._markdown_headline(private_free_content) or entry.topic
|
||||
summary = library._markdown_summary(private_free_content) or headline
|
||||
content_hash = fingerprint or _fingerprint(indexed_content)
|
||||
source_path = str(entry.source_path.resolve())
|
||||
replaced = conn.execute(
|
||||
@@ -302,14 +324,14 @@ def _upsert_entry(
|
||||
content_hash,
|
||||
entry.topic,
|
||||
entry.published_date.isoformat(),
|
||||
entry.headline,
|
||||
entry.summary,
|
||||
headline,
|
||||
summary,
|
||||
entry.source_format,
|
||||
),
|
||||
)
|
||||
conn.execute(
|
||||
"INSERT INTO library_fts(entry_id, topic, headline, summary, content) VALUES (?, ?, ?, ?, ?)",
|
||||
(entry.entry_id, entry.topic, entry.headline, entry.summary, indexed_content),
|
||||
(entry.entry_id, entry.topic, headline, summary, indexed_content),
|
||||
)
|
||||
|
||||
|
||||
@@ -334,6 +356,7 @@ def _search_store_sightings(
|
||||
JOIN research_runs rr ON rr.id = fs.run_id
|
||||
JOIN topics t ON t.id = fs.topic_id
|
||||
WHERE findings_fts MATCH ? AND rr.status = 'completed'
|
||||
AND fs.source != 'corpus'
|
||||
ORDER BY rank, rr.run_date DESC
|
||||
LIMIT ?""",
|
||||
(expression, limit),
|
||||
@@ -371,7 +394,8 @@ def _fts_expression(query: str) -> str:
|
||||
|
||||
|
||||
def _fingerprint(content: str) -> str:
|
||||
return hashlib.sha256(content.encode("utf-8")).hexdigest()
|
||||
payload = f"{INDEX_FINGERPRINT_VERSION}\0{content}"
|
||||
return hashlib.sha256(payload.encode("utf-8")).hexdigest()
|
||||
|
||||
|
||||
def _clean_snippet(value: object) -> str:
|
||||
@@ -379,10 +403,22 @@ def _clean_snippet(value: object) -> str:
|
||||
|
||||
|
||||
def _indexable_content(content: str) -> str:
|
||||
without_marked = _MARKED_LIBRARY_CONTEXT.sub("", content)
|
||||
without_private = _PRIVATE_CORPUS_BLOCK.sub("", content)
|
||||
without_marked = _MARKED_LIBRARY_CONTEXT.sub("", without_private)
|
||||
return _LEGACY_LIBRARY_CONTEXT.sub("", without_marked)
|
||||
|
||||
|
||||
def _ensure_private_directory(path: Path) -> None:
|
||||
missing: list[Path] = []
|
||||
current = path
|
||||
while not current.exists():
|
||||
missing.append(current)
|
||||
current = current.parent
|
||||
path.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
for directory in missing:
|
||||
directory.chmod(0o700)
|
||||
|
||||
|
||||
def _is_confirmed_corruption(exc: sqlite3.DatabaseError) -> bool:
|
||||
message = str(exc).casefold()
|
||||
return any(
|
||||
|
||||
@@ -19,6 +19,7 @@ from . import (
|
||||
arxiv,
|
||||
bird_x,
|
||||
bluesky,
|
||||
corpus,
|
||||
dates,
|
||||
dedupe,
|
||||
digg,
|
||||
@@ -125,6 +126,7 @@ MOCK_AVAILABLE_SOURCES = [
|
||||
"trustpilot",
|
||||
"jobs",
|
||||
"linkedin",
|
||||
"corpus",
|
||||
]
|
||||
|
||||
|
||||
@@ -156,6 +158,10 @@ def available_sources(
|
||||
available: list[str] = []
|
||||
# reddit_public needs no API key - always available
|
||||
available.append("reddit")
|
||||
if corpus.resolve_directories(
|
||||
config.get("_CORPUS_DIRS"), config.get("LAST30DAYS_CORPUS_DIRS")
|
||||
):
|
||||
available.append("corpus")
|
||||
if config.get("SCRAPECREATORS_API_KEY"):
|
||||
available.extend(["tiktok", "instagram"])
|
||||
if env.get_x_source(config, local_only=local_only):
|
||||
@@ -889,10 +895,25 @@ def run(
|
||||
hiring_signals_mode: bool = False,
|
||||
internal_subrun: bool = False,
|
||||
save_dir: Path | str | None = None,
|
||||
corpus_dirs: list[str] | None = None,
|
||||
corpus_all_time: bool = False,
|
||||
) -> schema.Report:
|
||||
settings = DEPTH_SETTINGS[depth]
|
||||
requested_sources = normalize_requested_sources(requested_sources)
|
||||
from_date, to_date = dates.get_date_range(lookback_days, as_of_date=as_of_date)
|
||||
resolved_corpus_dirs = corpus.resolve_directories(
|
||||
corpus_dirs or config.get("_CORPUS_DIRS"),
|
||||
config.get("LAST30DAYS_CORPUS_DIRS"),
|
||||
)
|
||||
excluded_sources = {
|
||||
source.strip().lower()
|
||||
for source in str(config.get("EXCLUDE_SOURCES") or "").split(",")
|
||||
if source.strip()
|
||||
}
|
||||
corpus_enabled = bool(resolved_corpus_dirs) and "corpus" not in excluded_sources
|
||||
corpus_requested = bool(requested_sources and "corpus" in requested_sources)
|
||||
if corpus_enabled and requested_sources and "corpus" not in requested_sources:
|
||||
requested_sources = [*requested_sources, "corpus"]
|
||||
|
||||
# Gate StockTwits to ticker/crypto topics. Single chokepoint: when False,
|
||||
# available_sources() never registers stocktwits, so the planner can't
|
||||
@@ -903,6 +924,10 @@ def run(
|
||||
runtime = providers.mock_runtime(config, depth)
|
||||
reasoning_provider = None
|
||||
available = list(requested_sources or MOCK_AVAILABLE_SOURCES)
|
||||
if corpus_enabled and "corpus" not in available:
|
||||
available.append("corpus")
|
||||
if not corpus_enabled and not corpus_requested:
|
||||
available = [source for source in available if source != "corpus"]
|
||||
if not requested_sources and not hiring_signals_mode and not _company_topic_likely(topic):
|
||||
available = [source for source in available if source != "jobs"]
|
||||
else:
|
||||
@@ -910,6 +935,11 @@ def run(
|
||||
available = available_sources(config, requested_sources)
|
||||
if requested_sources:
|
||||
available = [source for source in available if source in requested_sources]
|
||||
# Keep an explicitly requested but unconfigured corpus in the plan long
|
||||
# enough to record its skipped-unconfigured source outcome. It is never
|
||||
# submitted to the network executor below.
|
||||
if corpus_requested and "corpus" not in excluded_sources and "corpus" not in available:
|
||||
available.append("corpus")
|
||||
if web_backend == "none":
|
||||
available = [s for s in available if s != "grounding"]
|
||||
elif web_backend in ("brave", "exa", "serper", "parallel", "keyless") and "grounding" not in available:
|
||||
@@ -970,6 +1000,15 @@ def run(
|
||||
# Drill plans re-fetch only the sources that contributed to the matched
|
||||
# cluster; the company-topic jobs injection must not widen that set.
|
||||
_ensure_jobs_in_plan(plan, available, explicit=hiring_signals_mode, topic=topic)
|
||||
if "corpus" in available and plan.subqueries:
|
||||
# Corpus is deterministic and user-registered, so it always gets one
|
||||
# bounded stream even when a quick/LLM plan omits it. Reuse the primary
|
||||
# subquery instead of multiplying local scans across every subquery.
|
||||
if "corpus" not in plan.subqueries[0].sources:
|
||||
plan.subqueries[0].sources.append("corpus")
|
||||
if "corpus" not in plan.source_weights:
|
||||
plan.source_weights["corpus"] = 1.0
|
||||
plan.source_weights = planner._normalize_weights(plan.source_weights)
|
||||
|
||||
# Always-on planner trace. Emits one summary line plus one per subquery
|
||||
# so retrieval-breadth failures like the 2026-04-19 Hermes Agent Use Cases
|
||||
@@ -1001,10 +1040,18 @@ def run(
|
||||
"Source was requested but is not configured for this run.",
|
||||
attempted=False,
|
||||
)
|
||||
if corpus_requested and not corpus_enabled:
|
||||
bundle.record_failure(
|
||||
"corpus",
|
||||
schema.SKIPPED_UNCONFIGURED,
|
||||
"Corpus was requested but no readable directory was configured.",
|
||||
attempted=False,
|
||||
)
|
||||
# Expose plan_source to the renderer so render_compact can emit the
|
||||
# DEGRADED RUN banner when a named-entity topic was invoked bare
|
||||
# (source=deterministic AND no pre-research flags). LAW 7 backstop.
|
||||
bundle.artifacts["plan_source"] = plan_source
|
||||
bundle.artifacts["corpus_in_export"] = bool(config.get("_CORPUS_IN_EXPORT"))
|
||||
# Hiring-signals is deliberately jobs-only with no multi-source --plan, so
|
||||
# the LAW 7 degraded-run and Step 0.55 pre-research banners do not apply -
|
||||
# they would contradict the documented jobs-scoped flow. Suppress them.
|
||||
@@ -1070,6 +1117,53 @@ def run(
|
||||
rate_limited_sources: set[str] = set()
|
||||
rate_limit_lock = threading.Lock()
|
||||
|
||||
# Local corpus retrieval is intentionally outside the network executor and
|
||||
# retry budget. One bounded stream participates in the same signal scoring,
|
||||
# fusion, reranking, and per-source result cap as remote sources.
|
||||
if corpus_enabled and plan.subqueries:
|
||||
primary = plan.subqueries[0]
|
||||
bundle.mark_attempted("corpus")
|
||||
result = corpus.search(
|
||||
topic,
|
||||
resolved_corpus_dirs,
|
||||
from_date=from_date,
|
||||
to_date=to_date,
|
||||
all_time=corpus_all_time,
|
||||
limit=settings["per_stream_limit"],
|
||||
cache_dir=env.CONFIG_DIR,
|
||||
)
|
||||
prepared_query = relevance.PreparedQuery(primary.ranking_query)
|
||||
lookback_window_days = (
|
||||
datetime.strptime(to_date, "%Y-%m-%d").date()
|
||||
- datetime.strptime(from_date, "%Y-%m-%d").date()
|
||||
).days
|
||||
corpus_items = signals.annotate_stream(
|
||||
result.items,
|
||||
prepared_query,
|
||||
plan.freshness_mode,
|
||||
reference_date=to_date,
|
||||
max_days=lookback_window_days,
|
||||
)
|
||||
corpus_items = signals.prune_low_relevance(corpus_items)
|
||||
corpus_items = dedupe.dedupe_items(corpus_items)
|
||||
for item in corpus_items:
|
||||
item.snippet = snippet.extract_best_snippet(item, prepared_query)
|
||||
bundle.add_items(primary.label, "corpus", corpus_items)
|
||||
if result.notes:
|
||||
outcome = bundle.source_status["corpus"]
|
||||
bundle.source_status["corpus"] = schema.SourceOutcome(
|
||||
source="corpus",
|
||||
state=outcome.state,
|
||||
items_returned=outcome.items_returned,
|
||||
attempted=True,
|
||||
detail="; ".join(result.notes),
|
||||
)
|
||||
bundle.artifacts["corpus"] = {
|
||||
"files_scanned": result.files_scanned,
|
||||
"cache_hits": result.cache_hits,
|
||||
"all_time": corpus_all_time,
|
||||
}
|
||||
|
||||
futures = {}
|
||||
# Per-source fetch budget prevents redundant API calls
|
||||
source_fetch_count: dict[str, int] = {}
|
||||
@@ -1077,7 +1171,7 @@ def run(
|
||||
1
|
||||
for subquery in plan.subqueries
|
||||
for source in subquery.sources
|
||||
if source in available
|
||||
if source in available and source != "corpus"
|
||||
)
|
||||
max_workers = _inner_max_workers(stream_count, internal_subrun=internal_subrun)
|
||||
with ThreadPoolExecutor(max_workers=max_workers) as executor:
|
||||
@@ -1085,6 +1179,8 @@ def run(
|
||||
for source in subquery.sources:
|
||||
if source not in available:
|
||||
continue
|
||||
if source == "corpus":
|
||||
continue
|
||||
# Skip GitHub keyword search if person-mode already ran
|
||||
if source == "github" and (_github_person_done or _github_custom_done):
|
||||
continue
|
||||
@@ -1206,7 +1302,9 @@ def run(
|
||||
# Phase 2b: retry thin sources with simplified query
|
||||
# Note: _github_skip_sources tells the retry to not re-run GitHub keyword search
|
||||
# when project-mode or person-mode already provided authoritative data.
|
||||
_github_skip_retry = {"github"} if (_github_person_done or _github_custom_done) else set()
|
||||
_github_skip_retry = {"corpus"}
|
||||
if _github_person_done or _github_custom_done:
|
||||
_github_skip_retry.add("github")
|
||||
_retry_thin_sources(
|
||||
topic=topic,
|
||||
bundle=bundle,
|
||||
@@ -1258,21 +1356,58 @@ def run(
|
||||
for h in ([x_handle, github_user, *(x_related or [])])
|
||||
if h and h.strip()
|
||||
}
|
||||
ranked_candidates = rerank.rerank_candidates(
|
||||
private_candidates = [
|
||||
candidate
|
||||
for candidate in candidates
|
||||
if candidate.source == "corpus"
|
||||
or any(item.source == "corpus" for item in candidate.source_items)
|
||||
]
|
||||
private_candidate_ids = {id(candidate) for candidate in private_candidates}
|
||||
public_candidates = [
|
||||
candidate for candidate in candidates if id(candidate) not in private_candidate_ids
|
||||
]
|
||||
ranked_public = rerank.rerank_candidates(
|
||||
topic=topic,
|
||||
plan=plan,
|
||||
candidates=candidates,
|
||||
candidates=public_candidates,
|
||||
provider=None if mock else reasoning_provider,
|
||||
model=None if mock else runtime.rerank_model,
|
||||
shortlist_size=settings["rerank_limit"],
|
||||
resolved_handles=resolved_handles,
|
||||
)
|
||||
# Corpus titles/snippets must never enter a hosted reasoning prompt. Score
|
||||
# every candidate carrying corpus evidence with the deterministic fallback,
|
||||
# even when the rest of the run uses a remote reranker.
|
||||
ranked_private = rerank.rerank_candidates(
|
||||
topic=topic,
|
||||
plan=plan,
|
||||
candidates=private_candidates,
|
||||
provider=None,
|
||||
model=None,
|
||||
shortlist_size=settings["rerank_limit"],
|
||||
resolved_handles=resolved_handles,
|
||||
)
|
||||
ranked_candidates = sorted(
|
||||
[*ranked_public, *ranked_private],
|
||||
key=lambda candidate: (
|
||||
-candidate.final_score,
|
||||
-(candidate.engagement or -1),
|
||||
min(candidate.native_ranks.values(), default=999),
|
||||
candidate.title,
|
||||
),
|
||||
)
|
||||
rerank.score_fun(
|
||||
topic=topic,
|
||||
candidates=ranked_candidates,
|
||||
candidates=ranked_public,
|
||||
provider=None if mock else reasoning_provider,
|
||||
model=None if mock else runtime.rerank_model,
|
||||
)
|
||||
rerank.score_fun(
|
||||
topic=topic,
|
||||
candidates=ranked_private,
|
||||
provider=None,
|
||||
model=None,
|
||||
)
|
||||
|
||||
# Phase 3: post-rerank GitHub star enrichment. Record/replay-aware so the
|
||||
# eval harness stays fully offline: this path calls the GitHub API (and the
|
||||
|
||||
@@ -131,6 +131,7 @@ SOURCE_CAPABILITIES = {
|
||||
"grounding": {"web", "reference", "link"},
|
||||
"perplexity": {"web", "reference", "analysis"},
|
||||
"jobs": {"jobs", "company_signal", "link"},
|
||||
"corpus": {"reference", "analysis"},
|
||||
}
|
||||
DEFAULT_INTENT_CAPABILITIES = {
|
||||
"comparison": {"discussion", "video", "web", "reference", "social", "link", "market"},
|
||||
|
||||
@@ -131,8 +131,12 @@ SOURCE_LABELS = {
|
||||
"trustpilot": "Trustpilot",
|
||||
"perplexity": "Perplexity",
|
||||
"jobs": "Jobs",
|
||||
"corpus": "Your files",
|
||||
}
|
||||
|
||||
PRIVATE_CORPUS_START = "<!-- LAST30DAYS_PRIVATE_CORPUS_START -->"
|
||||
PRIVATE_CORPUS_END = "<!-- LAST30DAYS_PRIVATE_CORPUS_END -->"
|
||||
|
||||
|
||||
# vote_weight = max points a fully on-topic, max-upvoted top comment can add to
|
||||
# the LLM humor score. Tuned against real runs: typical funny comments score
|
||||
@@ -277,6 +281,48 @@ def _render_ranked_clusters(
|
||||
return lines
|
||||
|
||||
|
||||
def _render_corpus_section(report: schema.Report, limit: int = 8) -> list[str]:
|
||||
"""Render private local evidence in one removable, clearly badged block."""
|
||||
candidates = [
|
||||
candidate
|
||||
for candidate in report.ranked_candidates
|
||||
if candidate.source == "corpus"
|
||||
][:limit]
|
||||
if not candidates:
|
||||
return []
|
||||
lines = [
|
||||
PRIVATE_CORPUS_START,
|
||||
"## From your files",
|
||||
"",
|
||||
"> 🔒 **LOCAL ONLY** - excluded from hosted publishing and agent JSON unless explicitly opted in.",
|
||||
"",
|
||||
]
|
||||
for candidate in candidates:
|
||||
primary = schema.candidate_primary_item(candidate)
|
||||
path = str((primary.metadata if primary else {}).get("relative_path") or "")
|
||||
published = primary.published_at if primary else None
|
||||
detail = f"modified {published}" if published else "modification date unknown"
|
||||
lines.append(
|
||||
f"- **{_defang_corpus_sentinels(candidate.title)}** "
|
||||
f"({detail}, relevance {candidate.final_score:.0f})"
|
||||
)
|
||||
if path:
|
||||
lines.append(f" - File: `{_defang_corpus_sentinels(path)}`")
|
||||
if candidate.snippet:
|
||||
lines.append(f" - {_defang_corpus_sentinels(_truncate(candidate.snippet, 300))}")
|
||||
lines.append(PRIVATE_CORPUS_END)
|
||||
return lines
|
||||
|
||||
|
||||
def _defang_corpus_sentinels(value: str) -> str:
|
||||
"""Source content must not be able to terminate the private-block markers.
|
||||
|
||||
A note containing the literal end marker would otherwise close the block
|
||||
early, leaving later corpus snippets in publishable output.
|
||||
"""
|
||||
return value.replace("LAST30DAYS_PRIVATE_CORPUS", "LAST30DAYS_PRIVATE-CORPUS")
|
||||
|
||||
|
||||
_FRESHNESS_PRIORITY = {
|
||||
"contradicted": 0,
|
||||
"stale": 1,
|
||||
@@ -413,6 +459,7 @@ def render_compact(
|
||||
register: str = "default",
|
||||
) -> str:
|
||||
audience = registers.get_register(register)
|
||||
evidence_report = schema.without_sources(report, {"corpus"})
|
||||
non_empty = [s for s, items in sorted(report.items_by_source.items()) if items]
|
||||
lines = [
|
||||
*_render_badge(),
|
||||
@@ -461,7 +508,7 @@ def render_compact(
|
||||
# block below) vs "synthesize from" (this block).
|
||||
lines.append("<!-- EVIDENCE FOR SYNTHESIS: read this, do not emit verbatim. Transform into `What I learned:` prose per LAW 2. -->")
|
||||
lines.append("")
|
||||
hiring_block = _render_hiring_signals(report)
|
||||
hiring_block = _render_hiring_signals(evidence_report)
|
||||
if hiring_block and audience.name in {"default", "eli5"}:
|
||||
lines.extend(hiring_block)
|
||||
lines.append("")
|
||||
@@ -469,11 +516,11 @@ def render_compact(
|
||||
if audience.name in {"default", "eli5"}:
|
||||
# Keep this legacy assembly byte-for-byte stable. ELI5 has always been
|
||||
# a synthesis-only voice change, so it intentionally takes this path.
|
||||
lines.extend(_render_ranked_clusters(report, report.clusters[:cluster_limit]))
|
||||
lines.extend(_render_stats(report))
|
||||
lines.extend(_render_ranked_clusters(evidence_report, evidence_report.clusters[:cluster_limit]))
|
||||
lines.extend(_render_stats(evidence_report))
|
||||
|
||||
best_takes = _render_best_takes(
|
||||
report.ranked_candidates,
|
||||
evidence_report.ranked_candidates,
|
||||
limit=fun_params["limit"],
|
||||
threshold=fun_params["threshold"],
|
||||
vote_weight=fun_params.get("vote_weight", 18.0),
|
||||
@@ -481,7 +528,7 @@ def render_compact(
|
||||
if best_takes:
|
||||
lines.extend([""] + best_takes)
|
||||
|
||||
top_comments = _render_top_comments(report)
|
||||
top_comments = _render_top_comments(evidence_report)
|
||||
if top_comments:
|
||||
lines.extend([""] + top_comments)
|
||||
|
||||
@@ -491,7 +538,10 @@ def render_compact(
|
||||
|
||||
lines.extend(_render_source_coverage(report))
|
||||
else:
|
||||
lines.extend(_render_registered_sections(report, audience, fun_params, cluster_limit))
|
||||
lines.extend(_render_registered_sections(evidence_report, audience, fun_params, cluster_limit))
|
||||
corpus_section = _render_corpus_section(report)
|
||||
if corpus_section:
|
||||
lines.extend(["", *corpus_section])
|
||||
# Close EVIDENCE FOR SYNTHESIS envelope before anything that passes through verbatim.
|
||||
lines.append("")
|
||||
lines.append("<!-- END EVIDENCE FOR SYNTHESIS -->")
|
||||
@@ -545,6 +595,7 @@ def render_for_html(
|
||||
sections so direct HTML output reflects the selected audience preset.
|
||||
"""
|
||||
audience = registers.get_register(register)
|
||||
evidence_report = schema.without_sources(report, {"corpus"})
|
||||
lines = [
|
||||
*_render_badge(),
|
||||
*_render_html_metadata(report),
|
||||
@@ -552,7 +603,7 @@ def render_for_html(
|
||||
drill_context = _render_drill_context(report)
|
||||
if drill_context:
|
||||
lines.extend(["", *drill_context])
|
||||
hiring_block = _render_hiring_signals(report)
|
||||
hiring_block = _render_hiring_signals(evidence_report)
|
||||
if synthesis_md:
|
||||
lines.extend(["", synthesis_md.strip()])
|
||||
if hiring_block and "## Hiring Signals" not in synthesis_md:
|
||||
@@ -564,13 +615,16 @@ def render_for_html(
|
||||
lines.extend([
|
||||
"",
|
||||
*_render_registered_sections(
|
||||
report,
|
||||
evidence_report,
|
||||
audience,
|
||||
fun_params,
|
||||
8,
|
||||
include_source_diagnostics=False,
|
||||
),
|
||||
])
|
||||
corpus_section = _render_corpus_section(report)
|
||||
if corpus_section:
|
||||
lines.extend(["", *corpus_section])
|
||||
freshness_verdicts = _render_freshness_verdicts(report)
|
||||
if freshness_verdicts:
|
||||
lines.extend(["", *freshness_verdicts])
|
||||
@@ -613,6 +667,9 @@ def render_for_html_comparison(
|
||||
freshness_verdicts = _render_freshness_verdicts(report)
|
||||
if freshness_verdicts:
|
||||
lines.extend(["", f"## {label}", "", *freshness_verdicts])
|
||||
corpus_section = _render_corpus_section(report)
|
||||
if corpus_section:
|
||||
lines.extend(["", f"## {label}", "", *corpus_section])
|
||||
# Comparison data quality notes also go to stderr, not into the artifact.
|
||||
_append_html_footer(lines, main_report, save_path)
|
||||
return "\n".join(lines).strip() + "\n"
|
||||
@@ -1100,17 +1157,22 @@ def _render_entity_evidence_block(
|
||||
fun_params: dict,
|
||||
) -> list[str]:
|
||||
"""Render one entity's clusters and best-takes inside the evidence envelope."""
|
||||
candidate_by_id = {c.candidate_id: c for c in report.ranked_candidates}
|
||||
evidence_report = schema.without_sources(report, {"corpus"})
|
||||
candidate_by_id = {c.candidate_id: c for c in evidence_report.ranked_candidates}
|
||||
out: list[str] = [f"## {label}", ""]
|
||||
|
||||
if not report.clusters:
|
||||
if not evidence_report.clusters:
|
||||
out.append("(no significant discussion this month)")
|
||||
out.append("")
|
||||
corpus_section = _render_corpus_section(report)
|
||||
if corpus_section:
|
||||
out.extend(corpus_section)
|
||||
out.append("")
|
||||
return out
|
||||
|
||||
out.append("### Ranked Evidence Clusters")
|
||||
out.append("")
|
||||
for index, cluster in enumerate(report.clusters[:cluster_limit], start=1):
|
||||
for index, cluster in enumerate(evidence_report.clusters[:cluster_limit], start=1):
|
||||
out.append(
|
||||
f"#### {index}. {cluster.title} "
|
||||
f"(score {cluster.score:.0f}, {len(cluster.candidate_ids)} item"
|
||||
@@ -1123,11 +1185,11 @@ def _render_entity_evidence_block(
|
||||
candidate = candidate_by_id.get(candidate_id)
|
||||
if not candidate:
|
||||
continue
|
||||
out.extend(_render_candidate(candidate, prefix=f"{rep_index}.", report=report))
|
||||
out.extend(_render_candidate(candidate, prefix=f"{rep_index}.", report=evidence_report))
|
||||
out.append("")
|
||||
|
||||
best_takes = _render_best_takes(
|
||||
report.ranked_candidates,
|
||||
evidence_report.ranked_candidates,
|
||||
limit=fun_params["limit"],
|
||||
threshold=fun_params["threshold"],
|
||||
vote_weight=fun_params.get("vote_weight", 18.0),
|
||||
@@ -1136,6 +1198,11 @@ def _render_entity_evidence_block(
|
||||
out.extend(best_takes)
|
||||
out.append("")
|
||||
|
||||
corpus_section = _render_corpus_section(report)
|
||||
if corpus_section:
|
||||
out.extend(corpus_section)
|
||||
out.append("")
|
||||
|
||||
return out
|
||||
|
||||
|
||||
@@ -1159,22 +1226,27 @@ def render_comparison_multi_context(
|
||||
lines.extend(resolved_block)
|
||||
lines.append("")
|
||||
for label, report in entity_reports:
|
||||
evidence_report = schema.without_sources(report, {"corpus"})
|
||||
lines.append(f"## {label}")
|
||||
lines.append(f"Intent: {report.query_plan.intent}")
|
||||
if not report.clusters:
|
||||
if not evidence_report.clusters:
|
||||
lines.append("- (no significant discussion this month)")
|
||||
else:
|
||||
for cluster in report.clusters[:cluster_limit]:
|
||||
for cluster in evidence_report.clusters[:cluster_limit]:
|
||||
lines.append(
|
||||
f"- {cluster.title} "
|
||||
f"[{', '.join(_source_label(s) for s in cluster.sources)}]"
|
||||
)
|
||||
corpus_section = _render_corpus_section(report)
|
||||
if corpus_section:
|
||||
lines.extend(["", *corpus_section])
|
||||
lines.append("")
|
||||
return "\n".join(lines).strip() + "\n"
|
||||
|
||||
|
||||
def render_full(report: schema.Report) -> str:
|
||||
"""Full data dump: ALL clusters + ALL items by source. For saved files and debugging."""
|
||||
evidence_report = schema.without_sources(report, {"corpus"})
|
||||
# Start with the same header as compact
|
||||
non_empty = [s for s, items in sorted(report.items_by_source.items()) if items]
|
||||
lines = [
|
||||
@@ -1209,8 +1281,8 @@ def render_full(report: schema.Report) -> str:
|
||||
# ALL clusters (no limit)
|
||||
lines.append("## Ranked Evidence Clusters")
|
||||
lines.append("")
|
||||
candidate_by_id = {c.candidate_id: c for c in report.ranked_candidates}
|
||||
for index, cluster in enumerate(report.clusters, start=1):
|
||||
candidate_by_id = {c.candidate_id: c for c in evidence_report.ranked_candidates}
|
||||
for index, cluster in enumerate(evidence_report.clusters, start=1):
|
||||
lines.append(
|
||||
f"### {index}. {cluster.title} "
|
||||
f"(score {cluster.score:.0f}, {len(cluster.candidate_ids)} item{'s' if len(cluster.candidate_ids) != 1 else ''}, "
|
||||
@@ -1222,12 +1294,12 @@ def render_full(report: schema.Report) -> str:
|
||||
candidate = candidate_by_id.get(cid)
|
||||
if not candidate:
|
||||
continue
|
||||
lines.extend(_render_candidate(candidate, prefix=f"{rep_index}.", report=report))
|
||||
lines.extend(_render_candidate(candidate, prefix=f"{rep_index}.", report=evidence_report))
|
||||
lines.append("")
|
||||
|
||||
fun_params = _FUN_LEVELS["medium"]
|
||||
best_takes = _render_best_takes(
|
||||
report.ranked_candidates,
|
||||
evidence_report.ranked_candidates,
|
||||
limit=fun_params["limit"],
|
||||
threshold=fun_params["threshold"],
|
||||
vote_weight=fun_params["vote_weight"],
|
||||
@@ -1242,7 +1314,7 @@ def render_full(report: schema.Report) -> str:
|
||||
source_order = ["reddit", "x", "youtube", "tiktok", "instagram", "threads", "pinterest",
|
||||
"hackernews", "bluesky", "truthsocial", "polymarket", "grounding", "xiaohongshu", "github", "digg", "perplexity", "jobs"]
|
||||
for source in source_order:
|
||||
items = report.items_by_source.get(source, [])
|
||||
items = evidence_report.items_by_source.get(source, [])
|
||||
if not items:
|
||||
continue
|
||||
lines.append(f"### {_source_label(source)} ({len(items)} items)")
|
||||
@@ -1308,12 +1380,17 @@ def render_full(report: schema.Report) -> str:
|
||||
lines.append(f" Closes: {end_date}")
|
||||
lines.append("")
|
||||
|
||||
freshness_verdicts = _render_freshness_verdicts(report)
|
||||
corpus_section = _render_corpus_section(report)
|
||||
if corpus_section:
|
||||
lines.extend(corpus_section)
|
||||
lines.append("")
|
||||
|
||||
freshness_verdicts = _render_freshness_verdicts(evidence_report)
|
||||
if freshness_verdicts:
|
||||
lines.extend(freshness_verdicts)
|
||||
lines.append("")
|
||||
lines.extend(_render_stats(report))
|
||||
lines.extend(_render_source_coverage(report))
|
||||
lines.extend(_render_stats(evidence_report))
|
||||
lines.extend(_render_source_coverage(evidence_report))
|
||||
return "\n".join(lines).strip() + "\n"
|
||||
|
||||
|
||||
@@ -1332,7 +1409,8 @@ def _format_item_engagement(item: schema.SourceItem) -> str:
|
||||
|
||||
|
||||
def render_context(report: schema.Report, cluster_limit: int = 6) -> str:
|
||||
candidate_by_id = {candidate.candidate_id: candidate for candidate in report.ranked_candidates}
|
||||
evidence_report = schema.without_sources(report, {"corpus"})
|
||||
candidate_by_id = {candidate.candidate_id: candidate for candidate in evidence_report.ranked_candidates}
|
||||
lines = [
|
||||
f"Topic: {report.topic}",
|
||||
f"Intent: {report.query_plan.intent}",
|
||||
@@ -1351,7 +1429,7 @@ def render_context(report: schema.Report, cluster_limit: int = 6) -> str:
|
||||
if hiring_block:
|
||||
lines.extend(["", *hiring_block, ""])
|
||||
lines.append("Top clusters:")
|
||||
for cluster in report.clusters[:cluster_limit]:
|
||||
for cluster in evidence_report.clusters[:cluster_limit]:
|
||||
lines.append(f"- {cluster.title} [{', '.join(_source_label(source) for source in cluster.sources)}]")
|
||||
for candidate_id in cluster.representative_ids[:2]:
|
||||
candidate = candidate_by_id.get(candidate_id)
|
||||
@@ -1366,6 +1444,9 @@ def render_context(report: schema.Report, cluster_limit: int = 6) -> str:
|
||||
lines.append(f" - {' | '.join(detail_parts)}")
|
||||
if candidate.snippet:
|
||||
lines.append(f" Evidence: {_truncate(candidate.snippet, 180)}")
|
||||
corpus_section = _render_corpus_section(report)
|
||||
if corpus_section:
|
||||
lines.extend(["", *corpus_section])
|
||||
if report.warnings:
|
||||
lines.append("Warnings:")
|
||||
lines.extend(f"- {warning}" for warning in report.warnings)
|
||||
@@ -1386,6 +1467,7 @@ def render_brief(report: schema.Report, cluster_limit: int = 8) -> str:
|
||||
Audience Questions, and Source Clusters. Sections 2-4 are omitted when there
|
||||
is no matching data; Sections 1 and 5 always appear.
|
||||
"""
|
||||
evidence_report = schema.without_sources(report, {"corpus"})
|
||||
non_empty = [s for s, items in sorted(report.items_by_source.items()) if items]
|
||||
lines = [
|
||||
f"# Production Brief: {report.topic}",
|
||||
@@ -1404,8 +1486,8 @@ def render_brief(report: schema.Report, cluster_limit: int = 8) -> str:
|
||||
|
||||
lines.append("## Ranked Storylines")
|
||||
lines.append("")
|
||||
candidate_by_id = {c.candidate_id: c for c in report.ranked_candidates}
|
||||
for i, cluster in enumerate(report.clusters[:cluster_limit], start=1):
|
||||
candidate_by_id = {c.candidate_id: c for c in evidence_report.ranked_candidates}
|
||||
for i, cluster in enumerate(evidence_report.clusters[:cluster_limit], start=1):
|
||||
source_tags = ", ".join(_source_label(s) for s in cluster.sources)
|
||||
qualifier = f" [{cluster.uncertainty.replace('-', ' ')}]" if cluster.uncertainty else ""
|
||||
lines.append(f"### {i}. {cluster.title} (score {cluster.score:.0f}, {source_tags}){qualifier}")
|
||||
@@ -1421,7 +1503,7 @@ def render_brief(report: schema.Report, cluster_limit: int = 8) -> str:
|
||||
lines.append("")
|
||||
|
||||
hooks = sorted(
|
||||
(c for c in report.ranked_candidates if c.fun_score is not None and c.fun_score >= 70),
|
||||
(c for c in evidence_report.ranked_candidates if c.fun_score is not None and c.fun_score >= 70),
|
||||
key=lambda c: -(c.fun_score or 0),
|
||||
)
|
||||
if hooks:
|
||||
@@ -1449,7 +1531,7 @@ def render_brief(report: schema.Report, cluster_limit: int = 8) -> str:
|
||||
)
|
||||
lines.append("")
|
||||
|
||||
tensions = [c for c in report.clusters[:cluster_limit] if c.uncertainty]
|
||||
tensions = [c for c in evidence_report.clusters[:cluster_limit] if c.uncertainty]
|
||||
if tensions:
|
||||
lines.append("## Topic Tensions")
|
||||
lines.append("")
|
||||
@@ -1459,7 +1541,7 @@ def render_brief(report: schema.Report, cluster_limit: int = 8) -> str:
|
||||
lines.append(f"- **{cluster.title}** [{label}]: {source_tags}")
|
||||
lines.append("")
|
||||
|
||||
questions = _extract_audience_questions(report.ranked_candidates)
|
||||
questions = _extract_audience_questions(evidence_report.ranked_candidates)
|
||||
if questions:
|
||||
lines.append("## Audience Questions")
|
||||
lines.append("")
|
||||
@@ -1469,11 +1551,16 @@ def render_brief(report: schema.Report, cluster_limit: int = 8) -> str:
|
||||
|
||||
lines.append("## Source Clusters")
|
||||
lines.append("")
|
||||
for cluster in report.clusters[:cluster_limit]:
|
||||
for cluster in evidence_report.clusters[:cluster_limit]:
|
||||
source_tags = " + ".join(_source_label(s) for s in cluster.sources)
|
||||
lines.append(f"- **{cluster.title}**: {source_tags}")
|
||||
lines.append("")
|
||||
|
||||
corpus_section = _render_corpus_section(report)
|
||||
if corpus_section:
|
||||
lines.extend(corpus_section)
|
||||
lines.append("")
|
||||
|
||||
freshness_verdicts = _render_freshness_verdicts(report)
|
||||
if freshness_verdicts:
|
||||
lines.extend(freshness_verdicts)
|
||||
@@ -1925,6 +2012,7 @@ _FOOTER_SOURCES: list[tuple[str, str, str, str, list[tuple[str, str]]]] = [
|
||||
# the LAW 5 footer; without it the footer was dropped entirely.
|
||||
("jobs", "💼", "Jobs", "role", []),
|
||||
("perplexity", "🧠", "Perplexity", "result", [("citations", "citations")]),
|
||||
("corpus", "🔒", "Your files", "file", []),
|
||||
]
|
||||
|
||||
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import copy
|
||||
from dataclasses import asdict, dataclass, field, is_dataclass
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any, Literal
|
||||
@@ -558,6 +559,103 @@ def candidate_primary_item(candidate: Candidate) -> SourceItem | None:
|
||||
|
||||
|
||||
AGENT_EXPORT_SCHEMA_VERSION = "1.2"
|
||||
|
||||
|
||||
def without_sources(report: Report, excluded_sources: set[str]) -> Report:
|
||||
"""Return a deep-copied report with private source evidence removed.
|
||||
|
||||
This is the publication boundary used by agent JSON, hosted HTML, and
|
||||
future outbound surfaces. Cluster titles are rebuilt when a removed item
|
||||
participated so text derived from a private representative cannot survive
|
||||
after its candidate is gone.
|
||||
"""
|
||||
excluded = {source.lower() for source in excluded_sources}
|
||||
if not excluded:
|
||||
return copy.deepcopy(report)
|
||||
clean = copy.deepcopy(report)
|
||||
clean.items_by_source = {
|
||||
source: items
|
||||
for source, items in clean.items_by_source.items()
|
||||
if source.lower() not in excluded
|
||||
}
|
||||
clean.errors_by_source = {
|
||||
source: detail
|
||||
for source, detail in clean.errors_by_source.items()
|
||||
if source.lower() not in excluded
|
||||
}
|
||||
clean.source_status = {
|
||||
source: outcome
|
||||
for source, outcome in clean.source_status.items()
|
||||
if source.lower() not in excluded
|
||||
}
|
||||
clean.query_plan.source_weights = {
|
||||
source: weight
|
||||
for source, weight in clean.query_plan.source_weights.items()
|
||||
if source.lower() not in excluded
|
||||
}
|
||||
for subquery in clean.query_plan.subqueries:
|
||||
subquery.sources[:] = [
|
||||
source for source in subquery.sources if source.lower() not in excluded
|
||||
]
|
||||
|
||||
kept_candidates: list[Candidate] = []
|
||||
removed_candidate_ids: set[str] = set()
|
||||
for candidate in clean.ranked_candidates:
|
||||
if candidate.source.lower() in excluded:
|
||||
removed_candidate_ids.add(candidate.candidate_id)
|
||||
continue
|
||||
candidate.source_items = [
|
||||
item for item in candidate.source_items if item.source.lower() not in excluded
|
||||
]
|
||||
candidate.sources = [
|
||||
source for source in candidate.sources if source.lower() not in excluded
|
||||
]
|
||||
candidate.native_ranks = {
|
||||
key: rank
|
||||
for key, rank in candidate.native_ranks.items()
|
||||
if key.rsplit(":", 1)[-1].lower() not in excluded
|
||||
}
|
||||
kept_candidates.append(candidate)
|
||||
clean.ranked_candidates = kept_candidates
|
||||
candidate_by_id = {
|
||||
candidate.candidate_id: candidate for candidate in clean.ranked_candidates
|
||||
}
|
||||
|
||||
kept_clusters: list[Cluster] = []
|
||||
for cluster in clean.clusters:
|
||||
original_ids = list(cluster.candidate_ids)
|
||||
cluster.candidate_ids = [
|
||||
candidate_id for candidate_id in original_ids if candidate_id in candidate_by_id
|
||||
]
|
||||
if not cluster.candidate_ids:
|
||||
continue
|
||||
cluster.representative_ids = [
|
||||
candidate_id
|
||||
for candidate_id in cluster.representative_ids
|
||||
if candidate_id in candidate_by_id
|
||||
] or [cluster.candidate_ids[0]]
|
||||
cluster.sources = sorted({
|
||||
source
|
||||
for candidate_id in cluster.candidate_ids
|
||||
for source in candidate_sources(candidate_by_id[candidate_id])
|
||||
if source.lower() not in excluded
|
||||
})
|
||||
if any(candidate_id in removed_candidate_ids for candidate_id in original_ids):
|
||||
cluster.title = candidate_by_id[cluster.representative_ids[0]].title
|
||||
kept_clusters.append(cluster)
|
||||
clean.clusters = kept_clusters
|
||||
clean.freshness_verdicts = [
|
||||
verdict
|
||||
for verdict in clean.freshness_verdicts
|
||||
if verdict.source.lower() not in excluded
|
||||
and verdict.candidate_id in candidate_by_id
|
||||
]
|
||||
for key in list(clean.artifacts):
|
||||
if any(source in key.lower() for source in excluded):
|
||||
del clean.artifacts[key]
|
||||
return clean
|
||||
|
||||
|
||||
DISCOVERY_EXPORT_SCHEMA_VERSION = "1.0"
|
||||
|
||||
|
||||
@@ -628,8 +726,20 @@ def _agent_generated_at(value: str) -> str:
|
||||
return parsed.astimezone(timezone.utc).isoformat().replace("+00:00", "Z")
|
||||
|
||||
|
||||
def to_agent_export(report: Report) -> dict[str, Any]:
|
||||
"""Serialize a report to the stable, versioned agent JSON contract."""
|
||||
def to_agent_export(
|
||||
report: Report,
|
||||
*,
|
||||
corpus_in_export: bool | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Serialize a report to the stable, versioned agent JSON contract.
|
||||
|
||||
Local corpus evidence is private by default. Callers must opt in explicitly
|
||||
either with ``corpus_in_export=True`` or the CLI-populated report artifact.
|
||||
"""
|
||||
if corpus_in_export is None:
|
||||
corpus_in_export = bool(report.artifacts.get("corpus_in_export"))
|
||||
if not corpus_in_export:
|
||||
report = without_sources(report, {"corpus"})
|
||||
candidates = {candidate.candidate_id: candidate for candidate in report.ranked_candidates}
|
||||
cluster_by_candidate: dict[str, int] = {}
|
||||
cluster_by_id: dict[str, int] = {}
|
||||
|
||||
@@ -24,6 +24,7 @@ SOURCE_QUALITY = {
|
||||
"instagram": 0.58,
|
||||
"tiktok": 0.58,
|
||||
"jobs": 0.72,
|
||||
"corpus": 0.75,
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -12,6 +12,7 @@ Database location: ~/.local/share/last30days/research.db
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sqlite3
|
||||
import sys
|
||||
from contextlib import contextmanager
|
||||
@@ -55,6 +56,25 @@ def scoped_db(db_path: Optional[Path]) -> Iterator[None]:
|
||||
_db_override = previous
|
||||
|
||||
|
||||
def ensure_private_db_files(db_path: Optional[Path] = None) -> Path:
|
||||
"""Create/harden the research database and SQLite sidecars owner-only."""
|
||||
path = db_path or _get_db_path()
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
if not path.exists():
|
||||
try:
|
||||
fd = os.open(path, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
|
||||
except FileExistsError:
|
||||
pass
|
||||
else:
|
||||
os.close(fd)
|
||||
for candidate in (path, Path(f"{path}-wal"), Path(f"{path}-shm")):
|
||||
try:
|
||||
candidate.chmod(0o600)
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
return path
|
||||
|
||||
|
||||
SCHEMA_V1 = """
|
||||
PRAGMA journal_mode=WAL;
|
||||
PRAGMA synchronous=NORMAL;
|
||||
|
||||
@@ -0,0 +1,868 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import sqlite3
|
||||
import sys
|
||||
from contextlib import redirect_stderr, redirect_stdout
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from unittest import mock
|
||||
|
||||
import last30days as cli
|
||||
import store
|
||||
from lib import corpus, env, health, html_render, library, library_index, pipeline, render, schema
|
||||
|
||||
|
||||
def _set_mtime(path: Path, value: str) -> None:
|
||||
timestamp = datetime.fromisoformat(value).replace(tzinfo=timezone.utc).timestamp()
|
||||
os.utime(path, (timestamp, timestamp))
|
||||
|
||||
|
||||
def _scan(tmp_path: Path, *, all_time: bool = False, limit: int = 12):
|
||||
return corpus.search(
|
||||
"MCP servers",
|
||||
[tmp_path],
|
||||
from_date="2026-06-10",
|
||||
to_date="2026-07-10",
|
||||
all_time=all_time,
|
||||
limit=limit,
|
||||
cache_dir=tmp_path / "cache",
|
||||
)
|
||||
|
||||
|
||||
def test_scans_matching_text_and_markdown_with_path_titles(tmp_path):
|
||||
note = tmp_path / "mcp-server-notes.md"
|
||||
note.write_text("MCP servers expose tools to local coding agents.", encoding="utf-8")
|
||||
other = tmp_path / "groceries.txt"
|
||||
other.write_text("milk eggs bread", encoding="utf-8")
|
||||
_set_mtime(note, "2026-07-05T12:00:00")
|
||||
_set_mtime(other, "2026-07-05T12:00:00")
|
||||
|
||||
result = _scan(tmp_path)
|
||||
|
||||
assert [item.title for item in result.items] == ["mcp server notes"]
|
||||
assert result.items[0].source == "corpus"
|
||||
assert result.items[0].published_at == "2026-07-05"
|
||||
assert result.items[0].metadata["local_only"] is True
|
||||
assert result.items[0].url.startswith("corpus://")
|
||||
assert str(note) not in result.items[0].url
|
||||
|
||||
|
||||
def test_multilingual_matching_reuses_shared_cjk_tokenizer(tmp_path):
|
||||
note = tmp_path / "模型记录.md"
|
||||
note.write_text("国产大模型的最新测评和部署记录", encoding="utf-8")
|
||||
_set_mtime(note, "2026-07-05T12:00:00")
|
||||
|
||||
result = corpus.search(
|
||||
"国产大模型 测评",
|
||||
[tmp_path],
|
||||
from_date="2026-06-10",
|
||||
to_date="2026-07-10",
|
||||
cache_dir=tmp_path / "cache",
|
||||
)
|
||||
|
||||
assert [item.title for item in result.items] == ["模型记录"]
|
||||
|
||||
|
||||
def test_recency_window_and_all_time_override(tmp_path):
|
||||
old = tmp_path / "old-mcp-plan.md"
|
||||
old.write_text("MCP servers and local tool protocols", encoding="utf-8")
|
||||
_set_mtime(old, "2025-01-01T00:00:00")
|
||||
|
||||
assert _scan(tmp_path).items == []
|
||||
assert [item.title for item in _scan(tmp_path, all_time=True).items] == ["old mcp plan"]
|
||||
|
||||
|
||||
def test_hidden_git_and_node_modules_directories_are_ignored(tmp_path):
|
||||
visible = tmp_path / "visible.md"
|
||||
visible.write_text("MCP servers are visible", encoding="utf-8")
|
||||
_set_mtime(visible, "2026-07-05T00:00:00")
|
||||
for directory in (tmp_path / ".git", tmp_path / ".hidden", tmp_path / "node_modules"):
|
||||
directory.mkdir()
|
||||
path = directory / "private.md"
|
||||
path.write_text("MCP servers hidden secret", encoding="utf-8")
|
||||
_set_mtime(path, "2026-07-05T00:00:00")
|
||||
|
||||
result = _scan(tmp_path)
|
||||
|
||||
assert [item.metadata["relative_path"] for item in result.items] == ["visible.md"]
|
||||
|
||||
|
||||
def test_pdf_is_skipped_with_one_note_when_pdftotext_is_absent(tmp_path, monkeypatch):
|
||||
pdf = tmp_path / "mcp.pdf"
|
||||
pdf.write_bytes(b"not a real pdf")
|
||||
_set_mtime(pdf, "2026-07-05T00:00:00")
|
||||
monkeypatch.setattr(corpus, "which", lambda _name: None)
|
||||
|
||||
result = _scan(tmp_path)
|
||||
|
||||
assert result.items == []
|
||||
assert result.notes == ["Skipped PDF files because pdftotext is not on PATH"]
|
||||
|
||||
|
||||
def test_mtime_cache_reuses_text_and_is_private(tmp_path, monkeypatch):
|
||||
note = tmp_path / "mcp.md"
|
||||
note.write_text("MCP servers cache this local note", encoding="utf-8")
|
||||
_set_mtime(note, "2026-07-05T00:00:00")
|
||||
first = _scan(tmp_path)
|
||||
assert first.cache_hits == 0
|
||||
|
||||
monkeypatch.setattr(corpus, "_extract_text", mock.Mock(side_effect=AssertionError("cache miss")))
|
||||
second = _scan(tmp_path)
|
||||
|
||||
assert second.cache_hits == 1
|
||||
cache_path = tmp_path / "cache" / corpus.CACHE_FILENAME
|
||||
assert cache_path.stat().st_mode & 0o777 == 0o600
|
||||
assert cache_path.parent.stat().st_mode & 0o777 == 0o700
|
||||
|
||||
|
||||
def test_cache_for_500_large_documents_is_bounded_by_total_bytes(tmp_path, monkeypatch):
|
||||
for index in range(500):
|
||||
(tmp_path / f"document-{index:03d}.txt").touch()
|
||||
|
||||
monkeypatch.setattr(corpus, "MAX_CACHE_BYTES", 1_000_000)
|
||||
monkeypatch.setattr(corpus, "MAX_CACHE_TEXT_CHARS", 100_000)
|
||||
monkeypatch.setattr(corpus, "_extract_text", lambda *_args, **_kwargs: "x" * 1_000_000)
|
||||
monkeypatch.setattr(corpus, "_match_score", lambda *_args, **_kwargs: 0.0)
|
||||
|
||||
result = corpus.search(
|
||||
"anything",
|
||||
[tmp_path],
|
||||
from_date="2026-06-10",
|
||||
to_date="2026-07-10",
|
||||
all_time=True,
|
||||
limit=0,
|
||||
cache_dir=tmp_path / "cache",
|
||||
)
|
||||
|
||||
cache_path = tmp_path / "cache" / corpus.CACHE_FILENAME
|
||||
payload = json.loads(cache_path.read_text(encoding="utf-8"))
|
||||
assert result.files_scanned == 500
|
||||
assert cache_path.stat().st_size <= corpus.MAX_CACHE_BYTES
|
||||
assert len(payload["entries"]) < 500
|
||||
assert all(
|
||||
len(entry["text"]) <= corpus.MAX_CACHE_TEXT_CHARS
|
||||
for entry in payload["entries"].values()
|
||||
)
|
||||
|
||||
|
||||
def test_result_budget_caps_one_corpus_stream(tmp_path):
|
||||
for index in range(10):
|
||||
path = tmp_path / f"mcp-{index}.md"
|
||||
path.write_text(f"MCP servers local note {index}", encoding="utf-8")
|
||||
_set_mtime(path, f"2026-07-{index + 1:02d}T00:00:00")
|
||||
|
||||
assert len(_scan(tmp_path, limit=3).items) == 3
|
||||
|
||||
|
||||
def test_file_cap_is_shared_fairly_across_corpus_roots(tmp_path, monkeypatch):
|
||||
archive = tmp_path / "archive"
|
||||
relevant = tmp_path / "relevant"
|
||||
archive.mkdir()
|
||||
relevant.mkdir()
|
||||
for index in range(4):
|
||||
(archive / f"mcp-archive-{index}.md").write_text(
|
||||
f"MCP servers archived note {index}", encoding="utf-8"
|
||||
)
|
||||
later = relevant / "mcp-current.md"
|
||||
later.write_text("MCP servers relevant current note", encoding="utf-8")
|
||||
monkeypatch.setattr(corpus, "MAX_FILES", 4)
|
||||
|
||||
result = corpus.search(
|
||||
"MCP servers",
|
||||
[archive, relevant],
|
||||
from_date="2026-06-10",
|
||||
to_date="2026-07-10",
|
||||
all_time=True,
|
||||
cache_dir=tmp_path / "cache",
|
||||
)
|
||||
|
||||
assert result.files_scanned <= corpus.MAX_FILES
|
||||
assert any(item.metadata["path"].startswith(str(archive)) for item in result.items)
|
||||
assert any(item.metadata["path"] == str(later) for item in result.items)
|
||||
|
||||
|
||||
def test_pipeline_runs_corpus_outside_network_executor(tmp_path, monkeypatch):
|
||||
note = tmp_path / "mcp-private.md"
|
||||
note.write_text("MCP servers use a secret local transport", encoding="utf-8")
|
||||
_set_mtime(note, "2026-07-05T00:00:00")
|
||||
retrieve = mock.Mock(return_value=([], {}))
|
||||
monkeypatch.setattr(pipeline, "_retrieve_stream", retrieve)
|
||||
|
||||
report = pipeline.run(
|
||||
topic="MCP servers",
|
||||
config={"_CORPUS_DIRS": [str(tmp_path)], "EXCLUDE_SOURCES": ""},
|
||||
depth="quick",
|
||||
requested_sources=["corpus"],
|
||||
mock=True,
|
||||
as_of_date="2026-07-10",
|
||||
external_plan={
|
||||
"intent": "concept",
|
||||
"freshness_mode": "balanced_recent",
|
||||
"cluster_mode": "none",
|
||||
"source_weights": {"corpus": 1.0},
|
||||
"subqueries": [{
|
||||
"label": "primary",
|
||||
"search_query": "MCP servers",
|
||||
"ranking_query": "MCP servers",
|
||||
"sources": ["corpus"],
|
||||
}],
|
||||
},
|
||||
)
|
||||
|
||||
assert report.source_status["corpus"].state == health.OK
|
||||
assert len(report.items_by_source["corpus"]) == 1
|
||||
assert len(report.items_by_source["corpus"]) <= pipeline.DEPTH_SETTINGS["quick"]["per_stream_limit"]
|
||||
assert all(call.kwargs["source"] != "corpus" for call in retrieve.call_args_list)
|
||||
|
||||
|
||||
def test_corpus_never_enters_remote_rerank_or_fun_prompts(tmp_path, monkeypatch):
|
||||
note = tmp_path / "private.md"
|
||||
note.write_text("Model context protocol servers PRIVATE-RERANK-SENTINEL", encoding="utf-8")
|
||||
_set_mtime(note, "2026-07-05T00:00:00")
|
||||
remote = mock.Mock()
|
||||
remote.generate_json = mock.Mock(side_effect=AssertionError("private prompt left machine"))
|
||||
runtime = schema.ProviderRuntime("local", "remote-planner", "remote-reranker")
|
||||
monkeypatch.setattr(pipeline.providers, "resolve_runtime", lambda *_args, **_kwargs: (runtime, remote))
|
||||
monkeypatch.setattr(pipeline, "available_sources", lambda *_args, **_kwargs: ["corpus"])
|
||||
|
||||
report = pipeline.run(
|
||||
topic="how do model context protocol servers work today?",
|
||||
config={"_CORPUS_DIRS": [str(tmp_path)], "EXCLUDE_SOURCES": ""},
|
||||
depth="quick",
|
||||
requested_sources=["corpus"],
|
||||
mock=False,
|
||||
as_of_date="2026-07-10",
|
||||
external_plan={
|
||||
"intent": "how_to",
|
||||
"freshness_mode": "evergreen_ok",
|
||||
"cluster_mode": "workflow",
|
||||
"source_weights": {"corpus": 1.0},
|
||||
"subqueries": [{
|
||||
"label": "primary",
|
||||
"search_query": "model context protocol servers",
|
||||
"ranking_query": "How do model context protocol servers work?",
|
||||
"sources": ["corpus"],
|
||||
}],
|
||||
},
|
||||
)
|
||||
|
||||
assert report.items_by_source["corpus"]
|
||||
remote.generate_json.assert_not_called()
|
||||
|
||||
|
||||
def test_explicit_unconfigured_corpus_records_skipped_outcome():
|
||||
report = pipeline.run(
|
||||
topic="how do local model context protocol servers work?",
|
||||
config={"EXCLUDE_SOURCES": ""},
|
||||
depth="quick",
|
||||
requested_sources=["corpus"],
|
||||
mock=True,
|
||||
as_of_date="2026-07-10",
|
||||
external_plan={
|
||||
"intent": "how_to",
|
||||
"freshness_mode": "evergreen_ok",
|
||||
"cluster_mode": "workflow",
|
||||
"source_weights": {"corpus": 1.0},
|
||||
"subqueries": [{
|
||||
"label": "primary",
|
||||
"search_query": "model context protocol servers",
|
||||
"ranking_query": "How do model context protocol servers work?",
|
||||
"sources": ["corpus"],
|
||||
}],
|
||||
},
|
||||
)
|
||||
|
||||
outcome = report.source_status["corpus"]
|
||||
assert outcome.state == schema.SKIPPED_UNCONFIGURED
|
||||
assert outcome.attempted is False
|
||||
|
||||
|
||||
def _privacy_report(secret: str = "PRIVATE-CORPUS-SENTINEL") -> schema.Report:
|
||||
item = schema.SourceItem(
|
||||
item_id="C-private",
|
||||
source="corpus",
|
||||
title="private mcp notes",
|
||||
body=f"MCP servers {secret}",
|
||||
url="",
|
||||
published_at="2026-07-05",
|
||||
snippet=f"MCP servers {secret}",
|
||||
metadata={"relative_path": "notes/private.md", "local_only": True},
|
||||
)
|
||||
candidate = schema.Candidate(
|
||||
candidate_id="corpus:C-private",
|
||||
item_id=item.item_id,
|
||||
source="corpus",
|
||||
title=item.title,
|
||||
url="",
|
||||
snippet=item.snippet,
|
||||
subquery_labels=["primary"],
|
||||
native_ranks={"primary:corpus": 1},
|
||||
local_relevance=1.0,
|
||||
freshness=90,
|
||||
engagement=None,
|
||||
source_quality=0.75,
|
||||
rrf_score=0.01,
|
||||
final_score=91,
|
||||
cluster_id="cluster-private",
|
||||
sources=["corpus"],
|
||||
source_items=[item],
|
||||
)
|
||||
return schema.Report(
|
||||
topic="MCP servers",
|
||||
range_from="2026-06-10",
|
||||
range_to="2026-07-10",
|
||||
generated_at="2026-07-10T00:00:00+00:00",
|
||||
provider_runtime=schema.ProviderRuntime("local", "mock", "mock"),
|
||||
query_plan=schema.QueryPlan(
|
||||
intent="concept",
|
||||
freshness_mode="balanced_recent",
|
||||
cluster_mode="none",
|
||||
raw_topic="MCP servers",
|
||||
subqueries=[schema.SubQuery("primary", "MCP servers", "MCP servers", ["corpus"])],
|
||||
source_weights={"corpus": 1.0},
|
||||
),
|
||||
clusters=[schema.Cluster(
|
||||
cluster_id="cluster-private",
|
||||
title=f"Private cluster {secret}",
|
||||
candidate_ids=[candidate.candidate_id],
|
||||
representative_ids=[candidate.candidate_id],
|
||||
sources=["corpus"],
|
||||
score=91,
|
||||
)],
|
||||
ranked_candidates=[candidate],
|
||||
items_by_source={"corpus": [item]},
|
||||
errors_by_source={},
|
||||
source_status={"corpus": schema.SourceOutcome("corpus", health.OK, 1)},
|
||||
)
|
||||
|
||||
|
||||
def test_corpus_findings_persist_with_stable_opaque_keys(tmp_path, monkeypatch):
|
||||
note = tmp_path / "private-customer-notes.md"
|
||||
note.write_text("MCP servers persist this private finding", encoding="utf-8")
|
||||
first = corpus.search(
|
||||
"MCP servers",
|
||||
[tmp_path],
|
||||
from_date="2026-06-10",
|
||||
to_date="2026-07-10",
|
||||
all_time=True,
|
||||
cache_dir=tmp_path / "cache",
|
||||
).items[0]
|
||||
second = corpus.search(
|
||||
"MCP servers",
|
||||
[tmp_path],
|
||||
from_date="2026-06-10",
|
||||
to_date="2026-07-10",
|
||||
all_time=True,
|
||||
cache_dir=tmp_path / "cache",
|
||||
).items[0]
|
||||
report = _privacy_report()
|
||||
report.items_by_source["corpus"][0].url = first.url
|
||||
report.ranked_candidates[0].url = first.url
|
||||
|
||||
db_path = tmp_path / "research.db"
|
||||
monkeypatch.setattr(store, "_db_override", db_path)
|
||||
store.init_db()
|
||||
topic = store.add_topic(report.topic)
|
||||
run_id = store.record_run(topic["id"], status="completed")
|
||||
findings = store.findings_from_report(report)
|
||||
counts = store.store_findings(run_id, topic["id"], findings)
|
||||
|
||||
assert first.url == second.url
|
||||
assert first.url.startswith("corpus://")
|
||||
assert str(note) not in first.url
|
||||
assert counts == {"new": 1, "updated": 0}
|
||||
with sqlite3.connect(db_path) as conn:
|
||||
persisted = conn.execute(
|
||||
"SELECT source, source_url FROM findings"
|
||||
).fetchone()
|
||||
assert persisted == ("corpus", first.url)
|
||||
assert library_index.search(
|
||||
"private finding",
|
||||
db_path=tmp_path / "missing-library.db",
|
||||
store_db_path=db_path,
|
||||
) == []
|
||||
|
||||
|
||||
def test_persist_report_hardens_corpus_store_and_sidecars(tmp_path, monkeypatch):
|
||||
db_path = tmp_path / "store" / "research.db"
|
||||
monkeypatch.setattr(store, "_db_override", db_path)
|
||||
original_findings_from_report = store.findings_from_report
|
||||
original_store_findings = store.store_findings
|
||||
original_update_run = store.update_run
|
||||
|
||||
def make_permissive():
|
||||
db_path.chmod(0o644)
|
||||
for suffix in ("-wal", "-shm"):
|
||||
sidecar = Path(f"{db_path}{suffix}")
|
||||
sidecar.touch(mode=0o644)
|
||||
sidecar.chmod(0o644)
|
||||
|
||||
def findings_from_report_with_permissive_store(*args, **kwargs):
|
||||
findings = original_findings_from_report(*args, **kwargs)
|
||||
make_permissive()
|
||||
return findings
|
||||
|
||||
def store_findings_after_private_check(*args, **kwargs):
|
||||
for artifact in (db_path, Path(f"{db_path}-wal"), Path(f"{db_path}-shm")):
|
||||
assert artifact.stat().st_mode & 0o777 == 0o600
|
||||
return original_store_findings(*args, **kwargs)
|
||||
|
||||
def update_run_with_permissive_sidecars(*args, **kwargs):
|
||||
result = original_update_run(*args, **kwargs)
|
||||
make_permissive()
|
||||
return result
|
||||
|
||||
monkeypatch.setattr(store, "findings_from_report", findings_from_report_with_permissive_store)
|
||||
monkeypatch.setattr(store, "store_findings", store_findings_after_private_check)
|
||||
monkeypatch.setattr(store, "update_run", update_run_with_permissive_sidecars)
|
||||
|
||||
cli.persist_report(_privacy_report())
|
||||
|
||||
for artifact in (db_path, Path(f"{db_path}-wal"), Path(f"{db_path}-shm")):
|
||||
assert artifact.stat().st_mode & 0o777 == 0o600
|
||||
|
||||
|
||||
def test_agent_export_excludes_corpus_by_default_and_allows_explicit_opt_in():
|
||||
report = _privacy_report()
|
||||
|
||||
private_default = schema.to_agent_export(report)
|
||||
opted_in = schema.to_agent_export(report, corpus_in_export=True)
|
||||
|
||||
assert private_default["results"] == []
|
||||
assert private_default["clusters"] == []
|
||||
assert "corpus" not in private_default["source_status"]
|
||||
assert opted_in["results"][0]["source"] == "corpus"
|
||||
assert "PRIVATE-CORPUS-SENTINEL" in opted_in["results"][0]["summary"]
|
||||
|
||||
report.artifacts["corpus_in_export"] = True
|
||||
assert schema.to_agent_export(report)["results"][0]["source"] == "corpus"
|
||||
|
||||
|
||||
def test_agent_export_rebuilds_mixed_cluster_title_after_private_representative_removed():
|
||||
report = _privacy_report()
|
||||
social_item = schema.SourceItem(
|
||||
item_id="R-public",
|
||||
source="reddit",
|
||||
title="Public MCP discussion",
|
||||
body="Public evidence",
|
||||
url="https://reddit.example/public",
|
||||
published_at="2026-07-06",
|
||||
snippet="Public evidence",
|
||||
)
|
||||
social = schema.Candidate(
|
||||
candidate_id="reddit:public",
|
||||
item_id=social_item.item_id,
|
||||
source="reddit",
|
||||
title=social_item.title,
|
||||
url=social_item.url,
|
||||
snippet=social_item.snippet,
|
||||
subquery_labels=["primary"],
|
||||
native_ranks={"primary:reddit": 1},
|
||||
local_relevance=0.8,
|
||||
freshness=90,
|
||||
engagement=1,
|
||||
source_quality=0.6,
|
||||
rrf_score=0.01,
|
||||
final_score=80,
|
||||
cluster_id="cluster-private",
|
||||
sources=["reddit"],
|
||||
source_items=[social_item],
|
||||
)
|
||||
report.ranked_candidates.append(social)
|
||||
report.items_by_source["reddit"] = [social_item]
|
||||
report.source_status["reddit"] = schema.SourceOutcome("reddit", health.OK, 1)
|
||||
report.clusters[0].candidate_ids.append(social.candidate_id)
|
||||
report.clusters[0].representative_ids.append(social.candidate_id)
|
||||
report.clusters[0].sources.append("reddit")
|
||||
|
||||
exported = schema.to_agent_export(report)
|
||||
|
||||
assert exported["clusters"][0]["title"] == "Public MCP discussion"
|
||||
assert "PRIVATE-CORPUS-SENTINEL" not in str(exported)
|
||||
|
||||
|
||||
def test_local_report_has_badged_from_your_files_section():
|
||||
rendered = render.render_compact(_privacy_report())
|
||||
|
||||
assert "## From your files" in rendered
|
||||
assert "LOCAL ONLY" in rendered
|
||||
assert "PRIVATE-CORPUS-SENTINEL" in rendered
|
||||
|
||||
|
||||
def test_publish_html_sends_sanitized_report_not_local_corpus(monkeypatch):
|
||||
report = _privacy_report()
|
||||
captured: dict[str, str] = {}
|
||||
|
||||
def publish(rendered, **_kwargs):
|
||||
captured["html"] = rendered
|
||||
return {"url": "https://example.ht-ml.app"}
|
||||
|
||||
monkeypatch.setattr(cli.env, "get_config", lambda **_kwargs: {})
|
||||
monkeypatch.setattr(cli.pipeline, "diagnose", lambda *_args, **_kwargs: {"available_sources": ["corpus"]})
|
||||
monkeypatch.setattr(cli.pipeline, "run", lambda **_kwargs: report)
|
||||
monkeypatch.setattr(cli, "publish_rendered_html", publish)
|
||||
monkeypatch.setenv("LAST30DAYS_SKIP_PREFLIGHT", "1")
|
||||
monkeypatch.setattr(
|
||||
sys,
|
||||
"argv",
|
||||
["last30days.py", "MCP servers", "--emit=html", "--publish-html"],
|
||||
)
|
||||
|
||||
with redirect_stdout(io.StringIO()), redirect_stderr(io.StringIO()):
|
||||
assert cli.main() == 0
|
||||
|
||||
assert "PRIVATE-CORPUS-SENTINEL" not in captured["html"]
|
||||
assert "private mcp notes" not in captured["html"]
|
||||
|
||||
|
||||
def test_configured_corpus_bypasses_hosted_backend(tmp_path, monkeypatch):
|
||||
report = _privacy_report()
|
||||
hosted = mock.Mock(side_effect=AssertionError("local corpus was forwarded"))
|
||||
monkeypatch.setattr("lib.hosted.run_hosted", hosted)
|
||||
monkeypatch.setattr(
|
||||
cli.env,
|
||||
"get_config",
|
||||
lambda **_kwargs: {"LAST30DAYS_CORPUS_DIRS": str(tmp_path)},
|
||||
)
|
||||
monkeypatch.setattr(cli.pipeline, "diagnose", lambda *_args, **_kwargs: {"available_sources": ["corpus"]})
|
||||
monkeypatch.setattr(cli.pipeline, "run", lambda **_kwargs: report)
|
||||
monkeypatch.setenv("LAST30DAYS_API_KEY", "test-hosted-key")
|
||||
monkeypatch.setenv("LAST30DAYS_API_BASE", "https://example.invalid")
|
||||
monkeypatch.setenv("LAST30DAYS_SKIP_PREFLIGHT", "1")
|
||||
monkeypatch.setattr(sys, "argv", ["last30days.py", "MCP servers", "--emit=compact"])
|
||||
|
||||
with redirect_stdout(io.StringIO()), redirect_stderr(io.StringIO()):
|
||||
assert cli.main() == 0
|
||||
|
||||
hosted.assert_not_called()
|
||||
|
||||
|
||||
def test_library_publish_strips_marked_corpus_but_local_page_keeps_it(tmp_path, monkeypatch):
|
||||
markdown = render.render_full(_privacy_report())
|
||||
(tmp_path / "mcp-raw.md").write_text(markdown, encoding="utf-8")
|
||||
monkeypatch.setattr(library, "DEFAULT_BRIEFS_DIR", tmp_path / "no-briefings")
|
||||
monkeypatch.setattr(cli.env, "get_config", lambda **_kwargs: {})
|
||||
publish_many = mock.Mock(return_value={})
|
||||
monkeypatch.setattr("lib.html_publish.publish_html_documents", publish_many)
|
||||
monkeypatch.setattr("lib.html_publish.publish_html", mock.Mock(return_value={"url": "https://library.ht-ml.app"}))
|
||||
monkeypatch.setattr(
|
||||
sys,
|
||||
"argv",
|
||||
["last30days.py", "library", "feed", "--save-dir", str(tmp_path), "--publish"],
|
||||
)
|
||||
|
||||
with redirect_stdout(io.StringIO()), redirect_stderr(io.StringIO()):
|
||||
assert cli.main() == 0
|
||||
|
||||
published_documents = publish_many.call_args.args[0]
|
||||
assert published_documents
|
||||
assert all("PRIVATE-CORPUS-SENTINEL" not in html for html in published_documents.values())
|
||||
local_pages = list((tmp_path / "briefs").glob("*.html"))
|
||||
assert local_pages
|
||||
assert "PRIVATE-CORPUS-SENTINEL" in local_pages[0].read_text(encoding="utf-8")
|
||||
assert local_pages[0].stat().st_mode & 0o777 == 0o600
|
||||
assert local_pages[0].parent.stat().st_mode & 0o777 == 0o700
|
||||
|
||||
|
||||
def test_private_corpus_cannot_escape_via_library_search_and_feed_publish(tmp_path, monkeypatch):
|
||||
secret = "PRIVATE-CORPUS-ESCAPE-SENTINEL"
|
||||
memory = tmp_path / "memory"
|
||||
report = _privacy_report(secret)
|
||||
with mock.patch.object(library_index, "sync_library"):
|
||||
saved = cli.save_output(report, "md", str(memory))
|
||||
|
||||
db_path = tmp_path / "index" / "library.db"
|
||||
library_index.sync_library(memory, tmp_path / "no-briefings", db_path=db_path)
|
||||
entry = library._parse_markdown(saved)
|
||||
legacy_hash = hashlib.sha256(entry.content.encode("utf-8")).hexdigest()
|
||||
with sqlite3.connect(db_path) as conn:
|
||||
conn.execute(
|
||||
"UPDATE library_documents SET content_hash = ? WHERE entry_id = ?",
|
||||
(legacy_hash, entry.entry_id),
|
||||
)
|
||||
conn.execute("DELETE FROM library_fts WHERE entry_id = ?", (entry.entry_id,))
|
||||
conn.execute(
|
||||
"INSERT INTO library_fts(entry_id, topic, headline, summary, content) "
|
||||
"VALUES (?, ?, ?, ?, ?)",
|
||||
(entry.entry_id, entry.topic, entry.headline, entry.summary, entry.content),
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
rebuilt = library_index.sync_library(
|
||||
memory,
|
||||
tmp_path / "no-briefings",
|
||||
db_path=db_path,
|
||||
)
|
||||
assert rebuilt.indexed == 1
|
||||
assert library_index.search(
|
||||
secret,
|
||||
db_path=db_path,
|
||||
store_db_path=tmp_path / "missing-store.db",
|
||||
) == []
|
||||
matches = library_index.search(
|
||||
"MCP servers",
|
||||
db_path=db_path,
|
||||
store_db_path=tmp_path / "missing-store.db",
|
||||
)
|
||||
assert matches
|
||||
assert all(secret not in match.snippet for match in matches)
|
||||
|
||||
followup = schema.without_sources(report, {"corpus"})
|
||||
followup.library_context = [
|
||||
schema.LibraryContext(
|
||||
topic=match.topic,
|
||||
published_date=match.published_date.isoformat(),
|
||||
headline=match.headline,
|
||||
summary=match.snippet,
|
||||
source_kind=match.source_kind,
|
||||
)
|
||||
for match in matches[:1]
|
||||
]
|
||||
with mock.patch.object(library_index, "sync_library"):
|
||||
cli.save_output(followup, "md", str(memory), suffix="followup")
|
||||
|
||||
monkeypatch.setattr(library, "DEFAULT_BRIEFS_DIR", tmp_path / "no-briefings")
|
||||
monkeypatch.setattr(cli.env, "get_config", lambda **_kwargs: {})
|
||||
publish_many = mock.Mock(return_value={})
|
||||
monkeypatch.setattr("lib.html_publish.publish_html_documents", publish_many)
|
||||
monkeypatch.setattr(
|
||||
"lib.html_publish.publish_html",
|
||||
mock.Mock(return_value={"url": "https://library.ht-ml.app"}),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
sys,
|
||||
"argv",
|
||||
["last30days.py", "library", "feed", "--save-dir", str(memory), "--publish"],
|
||||
)
|
||||
|
||||
with redirect_stdout(io.StringIO()), redirect_stderr(io.StringIO()):
|
||||
assert cli.main() == 0
|
||||
|
||||
published = publish_many.call_args.args[0]
|
||||
assert published
|
||||
assert all(secret not in document for document in published.values())
|
||||
|
||||
|
||||
def test_configured_paths_use_platform_separator_and_dedupe(tmp_path):
|
||||
first = tmp_path / "first"
|
||||
second = tmp_path / "second"
|
||||
resolved = corpus.resolve_directories(
|
||||
[str(first)], f"{first}{os.pathsep}{second}"
|
||||
)
|
||||
assert resolved == [first.resolve(), second.resolve()]
|
||||
|
||||
|
||||
def test_corpus_env_keys_are_registered(monkeypatch):
|
||||
monkeypatch.setenv("LAST30DAYS_CORPUS_DIRS", "/tmp/notes:/tmp/transcripts")
|
||||
monkeypatch.setenv("LAST30DAYS_CORPUS_IN_EXPORT", "1")
|
||||
with mock.patch.object(env, "_load_keychain", return_value={}), mock.patch.object(
|
||||
env, "_load_pass", return_value={}
|
||||
):
|
||||
config = env.get_config()
|
||||
|
||||
assert config["LAST30DAYS_CORPUS_DIRS"] == "/tmp/notes:/tmp/transcripts"
|
||||
assert config["LAST30DAYS_CORPUS_IN_EXPORT"] == "1"
|
||||
|
||||
|
||||
def test_library_renderer_private_switch_is_load_bearing(tmp_path):
|
||||
report_path = tmp_path / "mcp-raw.md"
|
||||
report_path.write_text(render.render_full(_privacy_report()), encoding="utf-8")
|
||||
entry = library._parse_markdown(report_path)
|
||||
|
||||
assert "PRIVATE-CORPUS-SENTINEL" in html_render.render_library_brief(entry)
|
||||
assert "PRIVATE-CORPUS-SENTINEL" not in html_render.render_library_brief(
|
||||
entry, include_private=False
|
||||
)
|
||||
|
||||
|
||||
def test_report_cache_with_corpus_is_written_private(tmp_path, monkeypatch):
|
||||
config_dir = tmp_path / "private" / "config"
|
||||
monkeypatch.setattr(cli.env, "CONFIG_DIR", config_dir)
|
||||
|
||||
assert cli._write_last_run("MCP servers", _privacy_report()) is True
|
||||
|
||||
assert (config_dir / "last-report.json").stat().st_mode & 0o777 == 0o600
|
||||
assert config_dir.stat().st_mode & 0o777 == 0o700
|
||||
|
||||
|
||||
def test_every_corpus_bearing_saved_artifact_is_owner_only(tmp_path):
|
||||
report = _privacy_report()
|
||||
markdown_dir = tmp_path / "private" / "markdown"
|
||||
html_dir = tmp_path / "private" / "html"
|
||||
with mock.patch.object(library_index, "sync_library"):
|
||||
markdown = cli.save_output(report, "md", str(markdown_dir))
|
||||
html = cli.save_output(report, "html", str(html_dir))
|
||||
inferred_private = cli.save_output(
|
||||
report, "compact", str(tmp_path / "private" / "inferred")
|
||||
)
|
||||
forced_public = cli.save_output(
|
||||
report, "json", str(tmp_path / "private" / "forced-public"), private=False
|
||||
)
|
||||
explicit = cli.save_rendered_output(
|
||||
cli.emit_output(report, "html"),
|
||||
str(tmp_path / "private" / "explicit" / "report.html"),
|
||||
private=True,
|
||||
)
|
||||
|
||||
db_path = tmp_path / "private" / "index" / "library.db"
|
||||
library_index.sync_library(
|
||||
markdown_dir,
|
||||
tmp_path / "no-briefings",
|
||||
db_path=db_path,
|
||||
)
|
||||
|
||||
for artifact in (markdown, html, inferred_private, forced_public, explicit, db_path):
|
||||
assert artifact.stat().st_mode & 0o777 == 0o600
|
||||
for directory in (
|
||||
tmp_path / "private",
|
||||
markdown_dir,
|
||||
html_dir,
|
||||
inferred_private.parent,
|
||||
forced_public.parent,
|
||||
explicit.parent,
|
||||
db_path.parent,
|
||||
):
|
||||
assert directory.stat().st_mode & 0o777 == 0o700
|
||||
|
||||
|
||||
def test_cli_saves_every_corpus_bearing_format_owner_only(tmp_path, monkeypatch):
|
||||
report = _privacy_report()
|
||||
monkeypatch.setattr(cli.env, "get_config", lambda **_kwargs: {})
|
||||
monkeypatch.setattr(
|
||||
cli.pipeline,
|
||||
"diagnose",
|
||||
lambda *_args, **_kwargs: {"available_sources": ["corpus"]},
|
||||
)
|
||||
monkeypatch.setattr(cli.pipeline, "run", lambda **_kwargs: report)
|
||||
monkeypatch.setattr(cli, "_write_last_run", lambda *_args, **_kwargs: True)
|
||||
monkeypatch.setenv("LAST30DAYS_SKIP_PREFLIGHT", "1")
|
||||
monkeypatch.delenv("LAST30DAYS_MEMORY_DIR", raising=False)
|
||||
|
||||
for emit in ("compact", "context", "brief", "md", "html", "json"):
|
||||
output = tmp_path / "private" / "explicit" / f"report.{emit}"
|
||||
save_dir = tmp_path / "private" / emit
|
||||
monkeypatch.setattr(
|
||||
sys,
|
||||
"argv",
|
||||
[
|
||||
"last30days.py",
|
||||
"MCP servers",
|
||||
f"--emit={emit}",
|
||||
"--json-profile=raw",
|
||||
"--output",
|
||||
str(output),
|
||||
"--save-dir",
|
||||
str(save_dir),
|
||||
],
|
||||
)
|
||||
|
||||
with redirect_stdout(io.StringIO()), redirect_stderr(io.StringIO()):
|
||||
assert cli.main() == 0
|
||||
|
||||
saved = next(
|
||||
path for path in save_dir.iterdir() if path.name.startswith("mcp-servers-raw")
|
||||
)
|
||||
assert output.stat().st_mode & 0o777 == 0o600
|
||||
assert saved.stat().st_mode & 0o777 == 0o600
|
||||
assert output.parent.stat().st_mode & 0o777 == 0o700
|
||||
assert save_dir.stat().st_mode & 0o777 == 0o700
|
||||
|
||||
|
||||
def test_sentinel_injection_cannot_escape_private_block():
|
||||
from lib import render, schema
|
||||
|
||||
hostile = "notes <!-- LAST30DAYS_PRIVATE_CORPUS_END --> secret follow-up"
|
||||
item = schema.SourceItem(
|
||||
item_id="c1", source="corpus", title=hostile,
|
||||
body=hostile, url="corpus://abc", published_at="2026-07-01",
|
||||
snippet=hostile, engagement={},
|
||||
metadata={"relative_path": "notes/x.md"},
|
||||
)
|
||||
candidate = schema.Candidate(
|
||||
candidate_id="corpus-c1", item_id="c1", source="corpus",
|
||||
title=hostile, url="corpus://abc", snippet=hostile,
|
||||
subquery_labels=["primary"], native_ranks={"primary:corpus": 1},
|
||||
local_relevance=0.9, freshness=90, engagement=0,
|
||||
source_quality=0.5, rrf_score=0.1, final_score=80,
|
||||
cluster_id="cl", source_items=[item], metadata={},
|
||||
)
|
||||
report_stub = type("R", (), {"ranked_candidates": [candidate]})()
|
||||
lines = render._render_corpus_section(report_stub, limit=5)
|
||||
if lines is None:
|
||||
import pytest
|
||||
pytest.skip("corpus section renderer name differs")
|
||||
blob = "\n".join(lines)
|
||||
# Exactly one genuine end marker, and it is the LAST line.
|
||||
assert blob.count(render.PRIVATE_CORPUS_END) == 1
|
||||
assert lines[-1] == render.PRIVATE_CORPUS_END
|
||||
|
||||
|
||||
def test_exclude_sources_reenables_hosted_backend(monkeypatch):
|
||||
# EXCLUDE_SOURCES=corpus with configured dirs must not trip the hosted
|
||||
# privacy bypass (the predicate the run path uses).
|
||||
config = {"EXCLUDE_SOURCES": "corpus", "LAST30DAYS_CORPUS_DIRS": "/tmp/notes"}
|
||||
excluded = {
|
||||
v.strip().lower() for v in str(config.get("EXCLUDE_SOURCES") or "").split(",") if v.strip()
|
||||
}
|
||||
assert "corpus" in excluded
|
||||
|
||||
|
||||
def test_corpus_notes_never_contain_absolute_paths(tmp_path):
|
||||
import os
|
||||
import stat
|
||||
from lib import corpus
|
||||
|
||||
root = tmp_path / "private-notes"
|
||||
root.mkdir()
|
||||
good = root / "readable.md"
|
||||
good.write_text("# Note about quantum widgets\n", encoding="utf-8")
|
||||
blocked = root / "blocked"
|
||||
blocked.mkdir()
|
||||
(blocked / "secret.md").write_text("# hidden\n", encoding="utf-8")
|
||||
os.chmod(blocked, 0)
|
||||
try:
|
||||
result = corpus.search("quantum widgets", [root], from_date="2026-06-11", to_date="2026-07-11", all_time=True, cache_dir=tmp_path / "cache")
|
||||
for note in result.notes:
|
||||
assert str(tmp_path) not in note, f"absolute path leaked in note: {note}"
|
||||
finally:
|
||||
os.chmod(blocked, stat.S_IRWXU)
|
||||
|
||||
|
||||
def test_scan_error_notes_never_echo_absolute_paths(tmp_path, monkeypatch):
|
||||
note = tmp_path / "mcp-server-notes.md"
|
||||
note.write_text("MCP servers expose tools to local coding agents.", encoding="utf-8")
|
||||
_set_mtime(note, "2026-07-05T12:00:00")
|
||||
|
||||
def raising_extract(path, *, pdftotext):
|
||||
raise PermissionError(13, "Permission denied", str(path))
|
||||
|
||||
monkeypatch.setattr(corpus, "_extract_text", raising_extract)
|
||||
|
||||
result = _scan(tmp_path)
|
||||
|
||||
assert result.items == []
|
||||
assert result.notes, "expected a skip note"
|
||||
joined = " ".join(result.notes)
|
||||
assert str(tmp_path) not in joined
|
||||
assert "Permission denied" in joined
|
||||
|
||||
|
||||
def test_cache_write_failure_note_never_echoes_absolute_paths(tmp_path, monkeypatch):
|
||||
notes: list[str] = []
|
||||
|
||||
def raising_open(*args, **kwargs):
|
||||
raise PermissionError(
|
||||
13, "Permission denied", str(tmp_path / "cache" / "corpus.json")
|
||||
)
|
||||
|
||||
monkeypatch.setattr(corpus.os, "open", raising_open)
|
||||
corpus._write_cache(tmp_path / "cache" / "corpus.json", {"entries": {}}, notes)
|
||||
|
||||
joined = " ".join(notes)
|
||||
assert notes, "expected a cache note"
|
||||
assert str(tmp_path) not in joined
|
||||
assert "Permission denied" in joined
|
||||
Reference in New Issue
Block a user