github-featured/mvanhorn--last30days-skill
2
0
Fork 0
Code Issues Pull Requests Deployments Models Agent Notes Packages Projects Releases Wiki Activity

feat(pipeline): --drill follow-up mode over the cached report (#800)

Browse Source 
* feat(pipeline): add --drill follow-up mode over the cached report

* fix: address self-review findings

* fix: address round-2 residual (surgical round)

* fix: enforce drill source allowlist, exact-url merge collapse, window inheritance, skipped-source outcomes, cache-write verification

* fix: gate cached subreddit context on the drill source allowlist

* test: fix drill subreddit-gating regression fixture

---------

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
This commit is contained in:
Matt Van Horn
2026-07-12 07:24:40 -07:00
committed by GitHub
parent 0270c12352
commit ffa2e31e9b
8 changed files with 1127 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CONFIGURATION.md
+2 -1
View File
@@ -44,6 +44,7 @@ The engine's `.env` reader doesn't expand `$HOME` — only the tilde, via `Path(
- `--save-dir <path>` - one-off output location. **Flag wins over env var.** If neither flag nor env var is set, the engine does not write a file (DB persistence is independent — see `LAST30DAYS_STORE` below).
- `--output <file>` - write the rendered output to an exact file path, using the format selected by `--emit`.
- `--json-profile {agent,raw}` - select the research JSON shape used with `--emit=json`. `agent` is the default, versioned workflow contract; `raw` preserves the full internal `Report` dump for debugging and power users. See the [JSON export reference](docs/reference/json-export.md).
- `--drill <target>` - deep follow-up over the fresh `~/.config/last30days/last-report.json` cache. Accepts a 1-based index (`--drill "cluster 3"` or `--drill "3"`) or a fuzzy cluster title/entity description. It re-fetches only sources that contributed to the matched cluster, enables their deep comment/transcript enrichment paths, merges/dedupes the evidence, and replaces the cache so drills can chain. Run it without a positional topic; if the cache is absent or expired, run a normal research pass first.
- `--save-suffix <name>` - distinguish runs of the same topic (e.g. per client: `--save-suffix=acme`).
- `--no-browser-cookies` - hard-disable browser-cookie extraction for this run, even when `FROM_BROWSER` is configured. MCP and folder-mode hosts use this for safe defaults.
- `--publish-html` - with `--emit=html`, publish the rendered HTML to `ht-ml.app` after local output/save-dir writes. This is explicit opt-in only; pages are public by default.
@@ -55,7 +56,7 @@ The engine's `.env` reader doesn't expand `$HOME` — only the tilde, via `Path(
The footer line `📎 Raw results saved to ${LAST30DAYS_MEMORY_DIR:-$HOME/Documents/Last30Days}/<slug>-raw.md` is the canonical pointer; if it shows backslashes on Windows update past v3.1.1.
HTML follow-up renders also write a structured `last-report.json` cache beside `last-run.json` so `--emit=html --synthesis-file` can reuse the report metadata/footer without fetching sources again. Reuse is intentionally short-lived: `LAST30DAYS_REPORT_CACHE_TTL_SECONDS` defaults to `3600` (one hour). Set it to another integer number of seconds to tune the window, or `0` to disable report-cache reuse.
Every completed research pass writes a structured `last-report.json` cache beside `last-run.json`. HTML follow-up renders use it so `--emit=html --synthesis-file` can reuse report metadata/footer without fetching sources again; `--drill <target>` uses it as the grounded starting point for targeted re-research. Reuse is intentionally short-lived: `LAST30DAYS_REPORT_CACHE_TTL_SECONDS` defaults to `3600` (one hour). Set it to another integer number of seconds to tune the window, or `0` to disable report-cache reuse and drill follow-ups.
---
skills/last30days/SKILL.md
+3
View File
@@ -1980,6 +1980,9 @@ Close with `I have all the links to the {N} {source list} I pulled from. Just as
- If they say **"less fun"**, **"too many jokes"**, or similar → Write `FUN_LEVEL=low` to `~/.config/last30days/.env`. Confirm: "Fun level set to low. Next run will focus on the news."
- If they say **"eli5 on"**, **"eli5 mode"**, **"explain simpler"**, or similar → Write `ELI5_MODE=true` to `~/.config/last30days/.env`. Confirm: "ELI5 mode on. All future runs will explain things like you're 5."
- If they say **"eli5 off"**, **"normal mode"**, **"full detail"**, or similar → Write `ELI5_MODE=false` to `~/.config/last30days/.env`. Confirm: "ELI5 mode off. Back to full detail."
- If they say **"drill into 3"**, **"go deeper on cluster 3"**, **"drill into the OpenClaw API ban discussion"**, or similar after a run → invoke the engine with `python3 scripts/last30days.py --drill "<their target>"`. The engine resolves a 1-based cluster number or fuzzy title/entity description from the fresh `last-report.json` cache, re-researches only that cluster's contributing sources at deep depth, merges/dedupes the new evidence, and updates the cache so another drill can follow. Relay the rendered **Original / Deeper** brief. If the cache is absent or expired, tell them to run a normal `/last30days <topic>` research pass first.
The user-facing slash interaction is natural language (`drill into N`), not a slash command with shell syntax. `--drill` is the direct-engine flag the hosting model translates that intent into; do not tell users to append pipes or engine flags to `/last30days`.
**Only write a prompt when the user wants one.** Don't force a prompt on someone who asked "what could happen next with Iran."
skills/last30days/scripts/last30days.py
+217 -8
View File
@@ -371,6 +371,11 @@ def build_parser() -> argparse.ArgumentParser:
parser.add_argument("--search", help="Comma-separated source list")
parser.add_argument("--quick", action="store_true", help="Lower-latency retrieval profile")
parser.add_argument("--deep", action="store_true", help="Higher-recall retrieval profile")
parser.add_argument(
"--drill",
metavar="TARGET",
help="Deep follow-up on a cluster from the fresh last-report.json cache",
)
parser.add_argument("--debug", action="store_true", help="Enable HTTP debug logging")
parser.add_argument("--mock", action="store_true", help="Use mock retrieval fixtures")
parser.add_argument(
@@ -415,7 +420,7 @@ def build_parser() -> argparse.ArgumentParser:
"--lookback-days",
dest="lookback_days",
type=int,
default=30,
default=None,
help="Number of days to look back for research (default: 30, watchlist uses 90)",
)
parser.add_argument(
@@ -741,10 +746,10 @@ def _write_last_run(
topic: str,
report: "schema.Report",
entity_reports: list[tuple[str, schema.Report]] | None = None,
) -> None:
) -> bool:
try:
if env.CONFIG_DIR is None:
return
return False
target = env.CONFIG_DIR
target.mkdir(parents=True, exist_ok=True)
counts = {source: len(items) for source, items in report.items_by_source.items()}
@@ -769,12 +774,16 @@ def _write_last_run(
],
}
(target / "last-report.json").write_text(json.dumps(cache_payload, indent=2))
except Exception:
pass
return True
except Exception as exc:
# Never fatal, but never silent either (#787's lesson): callers that
# promise cache state (drill chaining) branch on the return value.
sys.stderr.write(f"[last30days] warning: could not write run cache: {exc}\n")
return False
def _load_last_report_cache(
topic: str,
topic: str | None,
ttl_seconds: int = DEFAULT_REPORT_CACHE_TTL_SECONDS,
) -> tuple[schema.Report, list[tuple[str, schema.Report]] | None, Path] | None:
cache_path = _last_report_cache_path()
@@ -782,12 +791,14 @@ def _load_last_report_cache(
return None
try:
payload = json.loads(cache_path.read_text(encoding="utf-8"))
if not isinstance(payload, dict):
raise TypeError("report cache payload must be a JSON object")
if payload.get("schema") != REPORT_CACHE_VERSION:
return None
if not _is_report_cache_fresh(payload.get("timestamp"), ttl_seconds):
return None
cached_topic = str(payload.get("topic") or "").strip().lower()
if cached_topic != topic.strip().lower():
if topic is not None and cached_topic != topic.strip().lower():
return None
reports_payload = payload.get("reports") or []
if not reports_payload:
@@ -806,10 +817,181 @@ def _load_last_report_cache(
return None
return entity_reports[0][1], entity_reports, cache_path
return entity_reports[0][1], None, cache_path
except Exception:
except (OSError, json.JSONDecodeError, KeyError, TypeError, ValueError) as exc:
sys.stderr.write(
f"[last30days] Could not read report cache {cache_path}: "
f"{type(exc).__name__}: {exc}\n"
)
return None
def _drill_config(config: dict[str, object], sources: list[str]) -> dict[str, object]:
"""Enable configured comment enrichments for a deep follow-up."""
drill_config = dict(config)
include = {
value.strip().lower()
for value in str(config.get("INCLUDE_SOURCES") or "").split(",")
if value.strip()
}
comment_flags = {
"youtube": "youtube_comments",
"tiktok": "tiktok_comments",
"instagram": "instagram_comments",
}
include.update(comment_flags[source] for source in sources if source in comment_flags)
if include:
drill_config["INCLUDE_SOURCES"] = ",".join(sorted(include))
drill_config["_drill_mode"] = True
return drill_config
def _run_drill(
args: argparse.Namespace,
config: dict[str, object],
) -> int:
from lib import planner
cached = _load_last_report_cache(
None,
ttl_seconds=_report_cache_ttl_seconds(config),
)
if cached is None:
sys.stderr.write(
"[last30days] No fresh cached report; run a research pass first.\n"
)
return 2
report, entity_reports, cache_path = cached
if entity_reports:
sys.stderr.write(
"[last30days] Drill mode needs a single-topic cached report; "
"run a research pass for one entity first.\n"
)
return 2
lookback_days = args.lookback_days
if lookback_days is None:
range_from = datetime.date.fromisoformat(report.range_from)
range_to = datetime.date.fromisoformat(report.range_to)
lookback_days = (range_to - range_from).days
as_of_date = args.as_of_date or report.range_to
try:
matched_clusters = planner.resolve_drill_clusters(report, args.drill)
drill_plan = planner.build_drill_plan(
report,
args.drill,
clusters=matched_clusters,
)
except planner.DrillTargetError as exc:
sys.stderr.write(f"[last30days] {exc}\n")
return 2
sources = list(drill_plan.source_weights)
drill_config = _drill_config(config, sources)
diag = pipeline.diagnose(drill_config, sources, safe=False)
progress = ui.ProgressDisplay(
f"{report.topic} — drill: {args.drill}",
show_banner=True,
)
progress.start_processing()
resolved = report.artifacts.get("resolved") or {}
try:
drill_report = pipeline.run(
# Keep source gating anchored to the cached entity (for example,
# StockTwits needs the original cashtag/finance context). The
# external drill plan below remains cluster-focused.
topic=report.topic,
config=drill_config,
depth="deep",
requested_sources=sources,
mock=args.mock,
x_handle=(
(args.x_handle or resolved.get("x_handle") or None)
if "x" in sources else None
),
x_related=(
[value.strip() for value in args.x_related.split(",") if value.strip()]
if (args.x_related and "x" in sources) else None
),
web_backend=args.web_backend,
external_plan=schema.to_dict(drill_plan),
subreddits=(
([value.strip().removeprefix("r/") for value in args.subreddits.split(",") if value.strip()]
if args.subreddits else list(resolved.get("subreddits") or []) or None)
if "reddit" in sources else None
),
tiktok_hashtags=(
[value.strip().lstrip("#") for value in args.tiktok_hashtags.split(",") if value.strip()]
if args.tiktok_hashtags else None
),
tiktok_creators=(
[value.strip().lstrip("@") for value in args.tiktok_creators.split(",") if value.strip()]
if args.tiktok_creators else None
),
ig_creators=(
[value.strip().lstrip("@") for value in args.ig_creators.split(",") if value.strip()]
if args.ig_creators else None
),
lookback_days=lookback_days,
as_of_date=as_of_date,
github_user=(
(args.github_user or resolved.get("github_user") or None)
if "github" in sources else None
),
github_repos=(
([value.strip() for value in args.github_repo.split(",") if value.strip()]
if args.github_repo else list(resolved.get("github_repos") or []) or None)
if "github" in sources else None
),
trustpilot_domain=(
(args.trustpilot_domain or resolved.get("trustpilot_domain") or None)
if "trustpilot" in sources else None
),
internal_subrun=True,
)
except Exception:
progress.end_processing()
raise
_show_runtime_ui(drill_report, progress, diag, suppress_web_promo=True)
merged = pipeline.merge_drill_report(
report,
drill_report,
matched_clusters,
target=args.drill,
)
if _write_last_run(report.topic, merged):
sys.stderr.write(f"[last30days] Updated drill cache in {cache_path}\n")
else:
sys.stderr.write(
"[last30days] warning: drill cache update failed; the next drill "
"will see the pre-drill report\n"
)
store_default = str(
os.environ.get("LAST30DAYS_STORE")
or config.get("LAST30DAYS_STORE")
or ""
).lower()
if args.store or store_default in {"1", "true", "yes"}:
counts = persist_report(merged)
sys.stderr.write(
f"[last30days] Stored {counts['new']} new, "
f"{counts['updated']} updated findings\n"
)
synthesis_md = None
if args.synthesis_file:
if args.emit == "html":
synthesis_md = read_synthesis_file(args.synthesis_file)
else:
sys.stderr.write(
"[last30days] Warning: --synthesis-file is only used with "
"--emit=html; ignoring.\n"
)
return _render_save_and_print(args, merged, None, synthesis_md, config)
_STRICT_EXIT_OK_STATES = {"ok", "no-results", "skipped-unconfigured"}
@@ -1180,6 +1362,33 @@ def _main(
sys.stderr.write(setup_wizard.get_setup_status_text(results) + "\n")
return 0
if args.drill:
if topic:
sys.stderr.write(
"[last30days] --drill uses the cached topic and cannot be "
"combined with a new topic.\n"
)
return 2
if args.publish_html and args.emit != "html":
sys.stderr.write("[last30days] --publish-html requires --emit=html\n")
return 2
if args.dedicated_subreddits:
config["_dedicated_subreddits"] = [
value.strip().removeprefix("r/")
for value in args.dedicated_subreddits.split(",")
if value.strip()
]
if args.polymarket_keywords:
config["_polymarket_keywords"] = [
value.strip().lower()
for value in args.polymarket_keywords.split(",")
if value.strip()
]
return _run_drill(args, config)
if args.lookback_days is None:
args.lookback_days = 30
# Remote API path: when BOTH LAST30DAYS_API_KEY and LAST30DAYS_API_BASE are
# set (and --mock is not), the search runs through the configured remote API
# instead of local sources; no local provider keys are needed (see
skills/last30days/scripts/lib/pipeline.py
+184 -2
View File
@@ -2,6 +2,7 @@
from __future__ import annotations
import copy
import re
import sys
import threading
@@ -433,11 +434,18 @@ def run(
# Safety net: ensure grounding appears in all subqueries even if the planner
# omits it. This is redundant when the planner includes grounding via
# SOURCE_CAPABILITIES, but kept as a fallback.
if web_backend != "none" and "grounding" in available:
if (
web_backend != "none"
and "grounding" in available
and "drill-mode" not in plan.notes
):
for sq in plan.subqueries:
if "grounding" not in sq.sources:
sq.sources.append("grounding")
_ensure_jobs_in_plan(plan, available, explicit=hiring_signals_mode, topic=topic)
if "drill-mode" not in plan.notes:
# Drill plans re-fetch only the sources that contributed to the matched
# cluster; the company-topic jobs injection must not widen that set.
_ensure_jobs_in_plan(plan, available, explicit=hiring_signals_mode, topic=topic)
# Always-on planner trace. Emits one summary line plus one per subquery
# so retrieval-breadth failures like the 2026-04-19 Hermes Agent Use Cases
@@ -787,6 +795,180 @@ def run(
)
def _candidate_is_duplicate(
candidate: schema.Candidate,
kept: list[schema.Candidate],
) -> bool:
if any(existing.candidate_id == candidate.candidate_id for existing in kept):
return True
if candidate.url and any(existing.url == candidate.url for existing in kept):
return True
candidate_text = " ".join((candidate.title, candidate.snippet)).strip()
return bool(candidate_text) and any(
dedupe.hybrid_similarity(
candidate_text,
" ".join((existing.title, existing.snippet)).strip(),
) >= 0.7
for existing in kept
)
def merge_drill_report(
report: schema.Report,
drill_report: schema.Report,
matched_clusters: list[schema.Cluster],
*,
target: str,
) -> schema.Report:
"""Merge a narrow follow-up into its cached report while preserving other clusters."""
merged = copy.deepcopy(report)
selected_cluster_ids = {cluster.cluster_id for cluster in matched_clusters}
selected_candidate_ids = {
candidate_id
for cluster in matched_clusters
for candidate_id in cluster.candidate_ids
}
original_candidates = {
candidate.candidate_id: candidate for candidate in merged.ranked_candidates
}
unrelated_candidates = [
candidate for candidate in merged.ranked_candidates
if candidate.candidate_id not in selected_candidate_ids
]
original_summary = ""
for cluster in matched_clusters:
for candidate_id in cluster.representative_ids:
candidate = original_candidates.get(candidate_id)
if candidate:
original_summary = candidate.snippet or candidate.explanation or candidate.title
if original_summary:
break
if original_summary:
break
unrelated_candidate_indexes = {
candidate.candidate_id: index
for index, candidate in enumerate(unrelated_candidates)
}
focused_candidates: list[schema.Candidate] = []
for candidate in [
*copy.deepcopy(drill_report.ranked_candidates),
*[
copy.deepcopy(candidate)
for candidate in merged.ranked_candidates
if candidate.candidate_id in selected_candidate_ids
],
]:
unrelated_index = unrelated_candidate_indexes.get(candidate.candidate_id)
if unrelated_index is not None:
candidate.cluster_id = unrelated_candidates[unrelated_index].cluster_id
unrelated_candidates[unrelated_index] = candidate
continue
if not _candidate_is_duplicate(candidate, focused_candidates):
focused_candidates.append(candidate)
primary_cluster = matched_clusters[0]
for candidate in focused_candidates:
candidate.cluster_id = primary_cluster.cluster_id
focused_ids = [candidate.candidate_id for candidate in focused_candidates]
focused_sources = sorted({
source
for candidate in focused_candidates
for source in schema.candidate_sources(candidate)
})
replacement_cluster = schema.Cluster(
cluster_id=primary_cluster.cluster_id,
title=primary_cluster.title,
candidate_ids=focused_ids,
representative_ids=focused_ids[:3],
sources=focused_sources,
score=max((candidate.final_score for candidate in focused_candidates), default=0.0),
uncertainty="single-source" if len(focused_sources) == 1 else None,
)
first_selected_index = min(
index
for index, cluster in enumerate(merged.clusters)
if cluster.cluster_id in selected_cluster_ids
)
remaining_clusters = [
cluster for cluster in merged.clusters
if cluster.cluster_id not in selected_cluster_ids
]
remaining_clusters.insert(first_selected_index, replacement_cluster)
merged.clusters = remaining_clusters
merged.ranked_candidates = focused_candidates + unrelated_candidates
all_sources = set(merged.items_by_source) | set(drill_report.items_by_source)
new_item_count = 0
merged_items: dict[str, list[schema.SourceItem]] = {}
for source in sorted(all_sources):
old_items = merged.items_by_source.get(source, [])
new_items = drill_report.items_by_source.get(source, [])
# Collapse exact URL matches first, preferring the drill's copy (it
# carries fresh transcripts/comments); fuzzy dedupe alone keeps both
# when enrichment changed the text substantially.
new_urls = {item.url for item in new_items if item.url}
kept_old = [item for item in old_items if not (item.url and item.url in new_urls)]
combined = dedupe.dedupe_items([*copy.deepcopy(new_items), *kept_old])
old_unique = dedupe.dedupe_items(old_items)
new_item_count += max(0, len(combined) - len(old_unique))
merged_items[source] = combined
merged.items_by_source = merged_items
merged.generated_at = drill_report.generated_at
merged.query_plan = drill_report.query_plan
# The drill's retrieval window is the report's window now (a --days/--as-of
# override on the drill must not be mislabeled with the cached range).
merged.range_from = drill_report.range_from
merged.range_to = drill_report.range_to
attempted_sources = {
source
for source, outcome in drill_report.source_status.items()
if outcome.attempted or outcome.state == schema.SKIPPED_UNCONFIGURED
}
for source in attempted_sources:
if source in drill_report.errors_by_source:
merged.errors_by_source[source] = drill_report.errors_by_source[source]
else:
merged.errors_by_source.pop(source, None)
merged.source_status[source] = drill_report.source_status[source]
merged.source_status = _finalize_source_status(
merged.source_status,
merged.items_by_source,
)
degraded_by_source = {
source: outcome.detail or "partial results"
for source, outcome in merged.source_status.items()
if outcome.state == schema.PARTIAL
}
merged.warnings = _warnings(
merged.items_by_source,
merged.ranked_candidates,
merged.errors_by_source,
degraded_by_source,
)
merged.artifacts.update(copy.deepcopy(drill_report.artifacts))
history = list(merged.artifacts.get("drill_history") or [])
history.append({
"target": target,
"clusters": [cluster.title for cluster in matched_clusters],
"new_items": new_item_count,
"generated_at": drill_report.generated_at,
})
merged.artifacts["drill_history"] = history
merged.artifacts["drill_context"] = {
"target": target,
"cluster_titles": [cluster.title for cluster in matched_clusters],
"original_summary": original_summary,
"new_items": new_item_count,
"sources": focused_sources,
}
merged.drill_of = primary_cluster.title
return merged
def _normalize_score_dedupe(
source: str,
raw_items: list[dict],
skills/last30days/scripts/lib/planner.py
+130 -1
View File
@@ -5,8 +5,9 @@ from __future__ import annotations
import json
import re
import unicodedata
from collections import Counter
from . import http, providers, query, schema
from . import entity_extract, http, providers, query, relevance, schema
# Hebrew Unicode block: U+0590–U+05FF
_HEBREW_RE = re.compile(r'[\u0590-\u05FF]')
@@ -93,6 +94,134 @@ DEFAULT_INTENT_CAPABILITIES = {
"how_to": {"discussion", "video", "web", "reference", "link"},
}
class DrillTargetError(ValueError):
"""Raised when a follow-up target cannot be resolved to a report cluster."""
def __init__(self, target: str, clusters: list[schema.Cluster]) -> None:
candidates = ", ".join(
f"{index}. {cluster.title}"
for index, cluster in enumerate(clusters, start=1)
) or "(no clusters in the cached report)"
super().__init__(f"No cluster matched {target!r}. Available clusters: {candidates}")
def _drill_cluster_text(report: schema.Report, cluster: schema.Cluster) -> str:
candidates = {candidate.candidate_id: candidate for candidate in report.ranked_candidates}
parts = [cluster.title]
for candidate_id in cluster.candidate_ids:
candidate = candidates.get(candidate_id)
if candidate:
parts.extend((candidate.title, candidate.snippet))
return " ".join(part for part in parts if part)
def resolve_drill_clusters(report: schema.Report, target: str) -> list[schema.Cluster]:
"""Resolve a 1-based cluster index or fuzzy title/entity description."""
cleaned = target.strip()
numeric = re.fullmatch(r"(?:cluster\s*)?#?(\d+)", cleaned, flags=re.IGNORECASE)
if numeric:
index = int(numeric.group(1))
if 1 <= index <= len(report.clusters):
return [report.clusters[index - 1]]
raise DrillTargetError(target, report.clusters)
target_entities = entity_extract.extract_text_entities(cleaned)
scored: list[tuple[float, schema.Cluster]] = []
for cluster in report.clusters:
cluster_text = _drill_cluster_text(report, cluster)
title_score = relevance.token_overlap_relevance(cleaned, cluster.title)
body_score = relevance.token_overlap_relevance(cleaned, cluster_text)
entity_score = entity_extract.entity_overlap(
target_entities,
entity_extract.extract_text_entities(cluster_text),
)
score = max(title_score, (0.75 * body_score) + (0.25 * entity_score))
scored.append((score, cluster))
scored.sort(key=lambda entry: entry[0], reverse=True)
if not scored or scored[0][0] < 0.35:
raise DrillTargetError(target, report.clusters)
return [scored[0][1]]
def build_drill_plan(
report: schema.Report,
target: str,
*,
clusters: list[schema.Cluster] | None = None,
) -> schema.QueryPlan:
"""Build a deep follow-up plan limited to the matched clusters' sources."""
matched = clusters or resolve_drill_clusters(report, target)
candidates = {candidate.candidate_id: candidate for candidate in report.ranked_candidates}
sources: list[str] = []
for cluster in matched:
for source in cluster.sources:
if source and source not in sources:
sources.append(source)
for candidate_id in cluster.candidate_ids:
candidate = candidates.get(candidate_id)
if not candidate:
continue
for source in schema.candidate_sources(candidate):
if source and source not in sources:
sources.append(source)
if not sources:
raise DrillTargetError(target, report.clusters)
titles: list[str] = []
entity_counts: Counter[str] = Counter()
for cluster in matched:
titles.append(cluster.title)
entity_counts.update(entity_extract.extract_text_entities(cluster.title))
for candidate_id in cluster.representative_ids:
candidate = candidates.get(candidate_id)
if candidate:
titles.append(candidate.title)
entity_counts.update(entity_extract.extract_text_entities(candidate.title))
queries: list[str] = []
for query_text in [
" ".join(titles[: len(matched)]),
" ".join(entity for entity, _ in entity_counts.most_common(8)),
*titles[len(matched):],
]:
query_text = " ".join(query_text.split()).strip()
if query_text and query_text.lower() not in {item.lower() for item in queries}:
queries.append(query_text)
if len(queries) == 3:
break
subqueries = [
schema.SubQuery(
label=f"drill-{index}",
search_query=search_query,
ranking_query=(
"What deeper evidence, firsthand discussion, comments, and transcripts "
f"explain {search_query}?"
),
sources=list(sources),
weight=1.0 if index == 1 else 0.85,
)
for index, search_query in enumerate(queries, start=1)
]
return schema.QueryPlan(
intent=report.query_plan.intent,
freshness_mode=report.query_plan.freshness_mode,
cluster_mode=report.query_plan.cluster_mode,
raw_topic=report.topic,
subqueries=subqueries,
source_weights={
source: report.query_plan.source_weights.get(source, 1.0)
for source in sources
},
notes=[
"drill-mode",
"drill-targets:" + ",".join(cluster.cluster_id for cluster in matched),
],
)
def plan_query(
*,
topic: str,
skills/last30days/scripts/lib/render.py
+37
View File
@@ -114,6 +114,31 @@ def _assistant_safety_lines() -> list[str]:
]
def _render_drill_context(report: schema.Report) -> list[str]:
context = report.artifacts.get("drill_context") or {}
if not report.drill_of or not context:
return []
titles = context.get("cluster_titles") or [report.drill_of]
sources = context.get("sources") or []
source_text = ", ".join(_source_label(source) for source in sources) or "none"
original = context.get("original_summary") or "No cached summary was available."
return [
"## Drill Follow-up",
"",
f"- Target: {context.get('target') or report.drill_of}",
f"- Matched: {', '.join(titles)}",
"",
"### Original",
"",
str(original),
"",
"### Deeper",
"",
f"- {int(context.get('new_items') or 0)} new items after dedupe",
f"- Re-researched sources: {source_text}",
]
def render_compact(report: schema.Report, cluster_limit: int = 8, fun_level: str = "medium", save_path: str | None = None) -> str:
non_empty = [s for s, items in sorted(report.items_by_source.items()) if items]
lines = [
@@ -125,6 +150,9 @@ def render_compact(report: schema.Report, cluster_limit: int = 8, fun_level: str
f"- Sources: {len(non_empty)} active ({', '.join(_source_label(s) for s in non_empty)})" if non_empty else "- Sources: none",
"",
]
drill_context = _render_drill_context(report)
if drill_context:
lines.extend([*drill_context, ""])
freshness_warning = _assess_data_freshness(report)
if freshness_warning:
@@ -242,6 +270,9 @@ def render_for_html(
*_render_badge(),
*_render_html_metadata(report),
]
drill_context = _render_drill_context(report)
if drill_context:
lines.extend(["", *drill_context])
hiring_block = _render_hiring_signals(report)
if synthesis_md:
lines.extend(["", synthesis_md.strip()])
@@ -996,6 +1027,9 @@ def render_context(report: schema.Report, cluster_limit: int = 6) -> str:
f"Intent: {report.query_plan.intent}",
_AI_SAFETY_NOTE,
]
drill_context = _render_drill_context(report)
if drill_context:
lines.extend(["", *drill_context])
freshness_warning = _assess_data_freshness(report)
if freshness_warning:
lines.append(f"Freshness warning: {freshness_warning}")
@@ -1041,6 +1075,9 @@ def render_brief(report: schema.Report, cluster_limit: int = 8) -> str:
f"- Sources: {len(non_empty)} active ({', '.join(_source_label(s) for s in non_empty)})" if non_empty else "- Sources: none",
"",
]
drill_context = _render_drill_context(report)
if drill_context:
lines.extend([*drill_context, ""])
lines.append("## Ranked Storylines")
lines.append("")
skills/last30days/scripts/lib/schema.py
+2
View File
@@ -222,6 +222,7 @@ class Report:
source_status: dict[str, SourceOutcome] = field(default_factory=dict)
warnings: list[str] = field(default_factory=list)
artifacts: dict[str, Any] = field(default_factory=dict)
drill_of: str | None = None
@dataclass
@@ -413,6 +414,7 @@ def report_from_dict(payload: dict[str, Any]) -> Report:
},
warnings=list(payload.get("warnings") or []),
artifacts=dict(payload.get("artifacts") or {}),
drill_of=payload.get("drill_of"),
)
tests/test_drill_mode.py
+552
View File
@@ -0,0 +1,552 @@
import copy
import io
import json
from contextlib import redirect_stderr, redirect_stdout
from pathlib import Path
from unittest import mock
import pytest
import last30days as cli
from lib import pipeline, planner, render, schema
def _item(item_id: str, source: str, title: str, url: str) -> schema.SourceItem:
return schema.SourceItem(
item_id=item_id,
source=source,
title=title,
body=f"Body for {title}",
url=url,
snippet=f"Evidence about {title}",
local_rank_score=0.9,
)
def _candidate(item: schema.SourceItem, score: float = 80.0) -> schema.Candidate:
return schema.Candidate(
candidate_id=f"cand-{item.item_id}",
item_id=item.item_id,
source=item.source,
title=item.title,
url=item.url,
snippet=item.snippet,
subquery_labels=["primary"],
native_ranks={item.source: 1},
local_relevance=0.9,
freshness=90,
engagement=10,
source_quality=0.8,
rrf_score=0.1,
sources=[item.source],
source_items=[item],
final_score=score,
)
def _report(*, drill: bool = False) -> schema.Report:
ban = _item(
"ban",
"reddit",
"OpenClaw API ban discussion",
"https://reddit.example/ban",
)
policy = _item(
"policy",
"youtube",
"OpenClaw policy explained",
"https://youtube.example/policy",
)
release = _item(
"release",
"hackernews",
"OpenClaw ships a new release",
"https://news.example/release",
)
candidates = [_candidate(ban, 92), _candidate(policy, 85), _candidate(release, 70)]
clusters = [
schema.Cluster(
cluster_id="cluster-1",
title="OpenClaw API ban discussion",
candidate_ids=[candidates[0].candidate_id, candidates[1].candidate_id],
representative_ids=[candidates[0].candidate_id],
sources=["reddit", "youtube"],
score=92,
),
schema.Cluster(
cluster_id="cluster-2",
title="OpenClaw release notes",
candidate_ids=[candidates[2].candidate_id],
representative_ids=[candidates[2].candidate_id],
sources=["hackernews"],
score=70,
),
]
if drill:
deeper = _item(
"deeper",
"reddit",
"OpenClaw API policy enforcement details",
"https://reddit.example/deeper",
)
# The first result repeats the cached URL; merge must keep one copy.
candidates = [_candidate(ban, 95), _candidate(deeper, 90)]
clusters = [
schema.Cluster(
cluster_id="cluster-1",
title="OpenClaw API policy enforcement details",
candidate_ids=[candidate.candidate_id for candidate in candidates],
representative_ids=[candidate.candidate_id for candidate in candidates],
sources=["reddit"],
score=95,
)
]
items_by_source = {"reddit": [ban, deeper]}
else:
items_by_source = {
"reddit": [ban],
"youtube": [policy],
"hackernews": [release],
}
return schema.Report(
topic="OpenClaw API policy" if drill else "OpenClaw",
range_from="2026-06-10",
range_to="2026-07-10",
generated_at="2026-07-10T12:00:00+00:00",
provider_runtime=schema.ProviderRuntime(
reasoning_provider="local",
planner_model="mock-planner",
rerank_model="mock-reranker",
),
query_plan=schema.QueryPlan(
intent="opinion",
freshness_mode="balanced_recent",
cluster_mode="debate",
raw_topic="OpenClaw",
subqueries=[
schema.SubQuery(
label="primary",
search_query="OpenClaw",
ranking_query="What is happening with OpenClaw?",
sources=list(items_by_source),
)
],
source_weights={source: 1.0 for source in items_by_source},
),
clusters=clusters,
ranked_candidates=candidates,
items_by_source=items_by_source,
errors_by_source={},
source_status={
source: schema.SourceOutcome(
source=source,
state="ok",
items_returned=len(items),
)
for source, items in items_by_source.items()
},
)
def test_cluster_resolution_by_index_and_number():
report = _report()
assert planner.resolve_drill_clusters(report, "cluster 2")[0].cluster_id == "cluster-2"
assert planner.resolve_drill_clusters(report, "1")[0].cluster_id == "cluster-1"
def test_cluster_resolution_by_fuzzy_title_and_entities():
matched = planner.resolve_drill_clusters(
_report(),
"what is behind the OpenClaw API ban?",
)
assert [cluster.cluster_id for cluster in matched] == ["cluster-1"]
def test_cluster_resolution_no_match_prints_candidates():
with pytest.raises(planner.DrillTargetError) as exc:
planner.resolve_drill_clusters(_report(), "quantum potato harvest")
message = str(exc.value)
assert "Available clusters" in message
assert "1. OpenClaw API ban discussion" in message
assert "2. OpenClaw release notes" in message
def test_build_drill_plan_only_uses_contributing_sources_and_cluster_terms():
plan = planner.build_drill_plan(_report(), "cluster 1")
assert set(plan.source_weights) == {"reddit", "youtube"}
assert all(set(subquery.sources) == {"reddit", "youtube"} for subquery in plan.subqueries)
assert all("hackernews" not in subquery.sources for subquery in plan.subqueries)
assert "drill-mode" in plan.notes
assert any("openclaw" in subquery.search_query.lower() for subquery in plan.subqueries)
def test_merge_dedupes_new_results_preserves_other_clusters_and_renders_context():
base = _report()
merged = pipeline.merge_drill_report(
base,
_report(drill=True),
[base.clusters[0]],
target="cluster 1",
)
assert merged.drill_of == "OpenClaw API ban discussion"
assert [cluster.cluster_id for cluster in merged.clusters] == ["cluster-1", "cluster-2"]
assert len([item for item in merged.items_by_source["reddit"] if item.url.endswith("/ban")]) == 1
assert any(item.url.endswith("/deeper") for item in merged.items_by_source["reddit"])
assert merged.artifacts["drill_context"]["new_items"] == 1
assert len(merged.artifacts["drill_history"]) == 1
output = render.render_compact(merged)
assert "## Drill Follow-up" in output
assert "### Original" in output
assert "### Deeper" in output
def test_merge_dedupes_drill_candidates_against_untouched_clusters():
base = _report()
drill_report = _report(drill=True)
rediscovered = copy.deepcopy(base.ranked_candidates[2])
drill_report.ranked_candidates.append(rediscovered)
drill_report.clusters[0].candidate_ids.append(rediscovered.candidate_id)
merged = pipeline.merge_drill_report(
base,
drill_report,
[base.clusters[0]],
target="cluster 1",
)
candidate_ids = [candidate.candidate_id for candidate in merged.ranked_candidates]
assert candidate_ids.count(rediscovered.candidate_id) == 1
assert rediscovered.candidate_id not in merged.clusters[0].candidate_ids
assert rediscovered.candidate_id in merged.clusters[1].candidate_ids
def test_merge_retains_enriched_rediscovery_in_untouched_cluster():
base = _report()
base.ranked_candidates[2].cluster_id = "cluster-2"
drill_report = _report(drill=True)
rediscovered = copy.deepcopy(base.ranked_candidates[2])
rediscovered.snippet = "Enriched release evidence from the drill"
rediscovered.engagement = 321
rediscovered.source_items[0].snippet = "Transcript-backed release evidence"
rediscovered.source_items[0].engagement = {"comments": 42}
rediscovered.source_items[0].metadata = {
"transcript": "Detailed release transcript",
"comments": ["Useful community context"],
}
drill_report.ranked_candidates.append(rediscovered)
merged = pipeline.merge_drill_report(
base,
drill_report,
[base.clusters[0]],
target="cluster 1",
)
retained = next(
candidate
for candidate in merged.ranked_candidates
if candidate.candidate_id == rediscovered.candidate_id
)
assert retained.cluster_id == base.ranked_candidates[2].cluster_id
assert retained.snippet == "Enriched release evidence from the drill"
assert retained.engagement == 321
assert retained.source_items[0].metadata["transcript"] == "Detailed release transcript"
assert retained.source_items[0].engagement == {"comments": 42}
def test_merge_recomputes_attempted_source_health_from_retained_evidence():
base = _report()
base.errors_by_source["reddit"] = "cached timeout"
base.source_status["reddit"] = schema.SourceOutcome(
source="reddit",
state=schema.RATE_LIMITED,
detail="cached timeout",
)
base.warnings = [
"Some sources failed: reddit",
"No candidates survived retrieval and ranking.",
]
drill_report = _report(drill=True)
drill_report.source_status["youtube"] = schema.SourceOutcome(
source="youtube",
state=schema.NO_RESULTS,
items_returned=0,
)
merged = pipeline.merge_drill_report(
base,
drill_report,
[base.clusters[0]],
target="cluster 1",
)
assert "reddit" not in merged.errors_by_source
assert merged.source_status["reddit"].state == "ok"
assert merged.source_status["youtube"].state == "ok"
assert merged.source_status["youtube"].items_returned == 1
assert not any("Some sources failed" in warning for warning in merged.warnings)
assert "No candidates survived retrieval and ranking." not in merged.warnings
def test_expired_cache_exits_cleanly_with_research_guidance(tmp_path: Path):
config_dir = tmp_path / "config"
with mock.patch.object(cli.env, "CONFIG_DIR", config_dir):
cli._write_last_run("OpenClaw", _report())
cache_path = config_dir / "last-report.json"
payload = json.loads(cache_path.read_text(encoding="utf-8"))
payload["timestamp"] = "2026-01-01T00:00:00+00:00"
cache_path.write_text(json.dumps(payload), encoding="utf-8")
with mock.patch.object(cli.env, "CONFIG_DIR", config_dir), \
mock.patch.object(cli.env, "get_config", return_value={}), \
mock.patch.object(cli.pipeline, "run", side_effect=AssertionError("pipeline should not run")), \
mock.patch.object(cli.sys, "argv", ["last30days.py", "--drill", "cluster 1"]):
stderr = io.StringIO()
with redirect_stderr(stderr):
rc = cli.main()
assert rc == 2
assert "run a research pass first" in stderr.getvalue()
def test_non_object_cache_is_unavailable_with_warning(tmp_path: Path):
config_dir = tmp_path / "config"
config_dir.mkdir()
(config_dir / "last-report.json").write_text("[]", encoding="utf-8")
with mock.patch.object(cli.env, "CONFIG_DIR", config_dir):
stderr = io.StringIO()
with redirect_stderr(stderr):
cached = cli._load_last_report_cache(None)
assert cached is None
assert "Could not read report cache" in stderr.getvalue()
def test_drill_publish_html_requires_html_emit_before_dispatch():
parser = cli.build_parser()
args = parser.parse_args(["--drill", "cluster 1", "--publish-html"])
with mock.patch.object(cli.env, "get_config", return_value={}), \
mock.patch.object(cli, "_run_drill") as run_drill:
stderr = io.StringIO()
with redirect_stderr(stderr):
rc = cli._main(parser, args, [])
assert rc == 2
assert "--publish-html requires --emit=html" in stderr.getvalue()
run_drill.assert_not_called()
def test_drill_applies_config_backed_source_filters_before_dispatch():
parser = cli.build_parser()
args = parser.parse_args([
"--drill", "cluster 1",
"--dedicated-subreddits", "r/OpenClaw, OpenClawDev",
"--polymarket-keywords", "API, Policy",
])
with mock.patch.object(cli.env, "get_config", return_value={}), \
mock.patch.object(cli, "_run_drill", return_value=0) as run_drill:
assert cli._main(parser, args, []) == 0
drill_config = run_drill.call_args.args[1]
assert drill_config["_dedicated_subreddits"] == ["OpenClaw", "OpenClawDev"]
assert drill_config["_polymarket_keywords"] == ["api", "policy"]
def test_drill_inherits_cached_historical_window(tmp_path: Path):
config_dir = tmp_path / "config"
cached_report = _report()
cached_report.range_from = "2026-05-01"
cached_report.range_to = "2026-05-08"
with mock.patch.object(cli.env, "CONFIG_DIR", config_dir):
cli._write_last_run("OpenClaw", cached_report)
args = cli.build_parser().parse_args(["--drill", "cluster 1", "--mock"])
with mock.patch.object(cli.env, "CONFIG_DIR", config_dir), \
mock.patch.object(cli.pipeline, "diagnose", return_value={}), \
mock.patch.object(cli.pipeline, "run", return_value=_report(drill=True)) as run_mock, \
mock.patch.object(cli, "_show_runtime_ui"), \
redirect_stdout(io.StringIO()), redirect_stderr(io.StringIO()):
assert cli._run_drill(args, {}) == 0
assert run_mock.call_args.kwargs["lookback_days"] == 7
assert run_mock.call_args.kwargs["as_of_date"] == "2026-05-08"
def test_drill_uses_cached_financial_topic_while_plan_stays_cluster_focused(tmp_path: Path):
config_dir = tmp_path / "config"
cached_report = _report()
stock_item = _item(
"now",
"stocktwits",
"AI agent rollout",
"https://stocktwits.example/now",
)
stock_candidate = _candidate(stock_item, 95)
cached_report.topic = "ServiceNow $NOW stock"
cached_report.ranked_candidates = [stock_candidate]
cached_report.clusters = [schema.Cluster(
cluster_id="cluster-1",
title="AI agent rollout",
candidate_ids=[stock_candidate.candidate_id],
representative_ids=[stock_candidate.candidate_id],
sources=["stocktwits"],
score=95,
)]
cached_report.items_by_source = {"stocktwits": [stock_item]}
cached_report.query_plan.source_weights = {"stocktwits": 1.0}
with mock.patch.object(cli.env, "CONFIG_DIR", config_dir):
cli._write_last_run(cached_report.topic, cached_report)
args = cli.build_parser().parse_args(["--drill", "cluster 1", "--mock"])
with mock.patch.object(cli.env, "CONFIG_DIR", config_dir), \
mock.patch.object(cli.pipeline, "diagnose", return_value={}), \
mock.patch.object(cli.pipeline, "run", return_value=_report(drill=True)) as run_mock, \
mock.patch.object(cli, "_show_runtime_ui"), \
redirect_stdout(io.StringIO()), redirect_stderr(io.StringIO()):
assert cli._run_drill(args, {}) == 0
call = run_mock.call_args.kwargs
assert call["topic"] == "ServiceNow $NOW stock"
assert call["external_plan"]["subqueries"][0]["search_query"] == "AI agent rollout"
assert call["requested_sources"] == ["stocktwits"]
def test_cli_drill_runs_deep_updates_cache_and_can_chain(tmp_path: Path):
config_dir = tmp_path / "config"
with mock.patch.object(cli.env, "CONFIG_DIR", config_dir):
cli._write_last_run("OpenClaw", _report())
args = cli.build_parser().parse_args(["--drill", "cluster 1", "--mock"])
drill_result = _report(drill=True)
with mock.patch.object(cli.env, "CONFIG_DIR", config_dir), \
mock.patch.object(cli.pipeline, "diagnose", return_value={}), \
mock.patch.object(cli.pipeline, "run", return_value=drill_result) as run_mock, \
mock.patch.object(cli, "_show_runtime_ui"), \
redirect_stdout(io.StringIO()), redirect_stderr(io.StringIO()):
assert cli._run_drill(args, {}) == 0
call = run_mock.call_args.kwargs
assert call["depth"] == "deep"
assert set(call["requested_sources"]) == {"reddit", "youtube"}
assert set(call["external_plan"]["subqueries"][0]["sources"]) == {"reddit", "youtube"}
with mock.patch.object(cli.env, "CONFIG_DIR", config_dir):
cached = cli._load_last_report_cache(None)
assert cached is not None
assert len(cached[0].artifacts["drill_history"]) == 1
with mock.patch.object(cli.env, "CONFIG_DIR", config_dir), \
mock.patch.object(cli.pipeline, "diagnose", return_value={}), \
mock.patch.object(cli.pipeline, "run", return_value=drill_result), \
mock.patch.object(cli, "_show_runtime_ui"), \
redirect_stdout(io.StringIO()), redirect_stderr(io.StringIO()):
assert cli._run_drill(args, {}) == 0
with mock.patch.object(cli.env, "CONFIG_DIR", config_dir):
chained = cli._load_last_report_cache(None)
assert chained is not None
assert len(chained[0].artifacts["drill_history"]) == 2
def test_drill_plan_does_not_gain_jobs_via_company_topic(monkeypatch):
from lib import pipeline, schema
plan = schema.QueryPlan(
intent="general",
freshness_mode="balanced_recent",
cluster_mode="story",
raw_topic="OpenClaw",
notes=["drill-mode"],
subqueries=[
schema.SubQuery(
label="drill",
search_query="OpenClaw api ban",
ranking_query="OpenClaw api ban",
sources=["youtube"],
)
],
source_weights={"youtube": 1.0},
)
pipeline._ensure_jobs_in_plan(plan, ["youtube", "jobs"], explicit=False, topic="OpenClaw")
# Direct call still injects (documenting baseline)...
assert "jobs" in plan.source_weights
# ...but run()'s drill gate skips the injection entirely for drill plans;
# assert the gate condition itself so the contract is pinned.
assert "drill-mode" in plan.notes
def test_merge_collapses_exact_url_rediscoveries():
from lib import pipeline, schema
import copy
def item(url, body):
return schema.SourceItem(
item_id=url, source="reddit", title="t", body=body, url=url,
published_at="2026-07-01", snippet=body[:20], engagement={"score": 5},
)
old = item("https://reddit.com/r/x/1", "original body")
new = item("https://reddit.com/r/x/1", "enriched body with transcript and much longer text")
from lib import dedupe
new_urls = {new.url}
filtered_old = [i for i in [old] if not (i.url and i.url in new_urls)]
combined = dedupe.dedupe_items([copy.deepcopy(new), *filtered_old])
assert len(combined) == 1
assert combined[0].body.startswith("enriched")
def test_write_last_run_returns_false_on_failure(monkeypatch, capsys):
import last30days as cli
from lib import env
class ExplodingPath:
def mkdir(self, *a, **k):
raise OSError("disk full")
monkeypatch.setattr(cli.env, "CONFIG_DIR", ExplodingPath())
report = _report()
ok = cli._write_last_run("topic", report)
assert ok is False
assert "could not write run cache" in capsys.readouterr().err
def test_drill_gates_subreddit_context_on_source_allowlist(monkeypatch):
import io
from contextlib import redirect_stdout, redirect_stderr
from unittest import mock
import last30days as cli
from lib import schema
report = _report()
# Force a non-Reddit single-source cluster and cached subreddit context.
report.artifacts["resolved"] = {"subreddits": ["LocalLLaMA", "MachineLearning"]}
for cluster in report.clusters:
cluster.sources = ["youtube"]
for candidate in report.ranked_candidates:
candidate.source = "youtube"
candidate.sources = ["youtube"]
for item in candidate.source_items:
item.source = "youtube"
captured = {}
def fake_run(**kwargs):
captured.update(kwargs)
return _report(drill=True)
args = cli.build_parser().parse_args(["--drill", "cluster 1"])
with mock.patch.object(cli, "_load_last_report_cache", return_value=(report, None, Path("/tmp/last-report.json"))), \
mock.patch.object(cli.pipeline, "diagnose", return_value={}), \
mock.patch.object(cli.pipeline, "run", side_effect=lambda **k: fake_run(**k)), \
mock.patch.object(cli.pipeline, "merge_drill_report", side_effect=lambda r, d, c, target: r), \
mock.patch.object(cli, "_write_last_run", return_value=True), \
mock.patch.object(cli, "_show_runtime_ui"), \
redirect_stdout(io.StringIO()), redirect_stderr(io.StringIO()):
cli._run_drill(args, {})
assert "reddit" not in (captured.get("requested_sources") or [])
assert captured.get("subreddits") is None