feat(youtube): fetch comments free via yt-dlp, drop the ScrapeCreators requirement (#827)
YouTube comments previously required a paid ScrapeCreators key plus a youtube_comments opt-in in INCLUDE_SOURCES. yt-dlp already backs YouTube search and transcripts here and can fetch comments too, so the comment lane no longer needs a credential or an opt-in. - youtube_yt: new _ytdlp_comments_result() returns (comments, ran_cleanly) so a clean "video has zero comments" run never falls back to a paid SC call; the list-returning _fetch_video_comments_ytdlp() wraps it. yt-dlp is tried first; SC stays as the backstop only on genuine failure and only when a token is configured. Command requests top-sorted comments, bounded by _COMMENT_TIMEOUT=20s per video (3 videos, parallel). - env.is_youtube_comments_available: True whenever yt-dlp is installed; legacy SC path still applies when yt-dlp is absent; EXCLUDE_SOURCES= youtube_comments remains a hard off-switch that wins over both. - doctor: stop prescribing a paid SC key for comments when yt-dlp is present (was selling a fix for a non-problem); caveat now names yt-dlp (free) as the first way out. - CONFIGURATION.md: YouTube comments row corrected to free/keyless. - Tests: new tests/test_youtube_comments_ytdlp.py locks the command flags, the yt-dlp-first preference, the no-SC-on-clean-empty behavior, and the no-key availability gate; existing doctor/gating tests updated to the new contract and pinned hermetic (is_ytdlp_available derived from probe). Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
micmicalpha
+1
-1
@@ -134,7 +134,7 @@ python3 skills/last30days/scripts/last30days.py "MCP servers" \
|
||||
| DripStack | none | opt-in only: per run with `--search dripstack`, or persistently with `INCLUDE_SOURCES=dripstack` in `.env`. Searches premium financial newsletters and analyst writeups via a free, public search API — no key needed. Never active without the opt-in. | yes when opted in (public API, no auth) |
|
||||
| GitHub | `gh` CLI installed (uses your GitHub auth) | always on if `gh` present | yes |
|
||||
| YouTube | `yt-dlp` CLI installed; `SCRAPECREATORS_API_KEY` adds a server-side transcript fallback used only when yt-dlp fails (429 / bot-gate) | always on if `yt-dlp` present; SC transcript fallback default-on when key set (no credit spent unless yt-dlp fails) | yes |
|
||||
| YouTube comments | `SCRAPECREATORS_API_KEY` + `INCLUDE_SOURCES` contains `youtube_comments` (**on by default** — written by the Step 5 Recommended tier) | top comments (by likes) on the top ~3 videos by engagement | ~3 calls/run; 10K free calls |
|
||||
| YouTube comments | `yt-dlp` CLI installed — **free and keyless, no API key and no opt-in needed**. Falls back to `SCRAPECREATORS_API_KEY` + `INCLUDE_SOURCES` containing `youtube_comments` only when yt-dlp is absent. Suppress with `EXCLUDE_SOURCES=youtube_comments`. | top comments (by likes) on the top ~3 videos by engagement | yes — free via yt-dlp (no credits spent) |
|
||||
| TikTok comments | `SCRAPECREATORS_API_KEY` + `INCLUDE_SOURCES` contains `tiktok_comments` (**on by default** — Step 5 Recommended tier) | top comments (by `digg_count`) on the top ~3 TikTok posts | ~3 calls/run; 10K free calls |
|
||||
| Instagram comments | `SCRAPECREATORS_API_KEY` + `INCLUDE_SOURCES` contains `instagram_comments` (**on by default** — Step 5 Recommended tier) | top comments (by `comment_like_count`) on the top ~3 Instagram posts, via `/v2/instagram/post/comments` | ~3 calls/run; 10K free calls |
|
||||
| Digg | `digg-pp-cli` on PATH (auto-installed during first-run setup via `npx -y @mvanhorn/printing-press-library@0.1.16 install digg --cli-only`; binary defaults to `$HOME/.local/bin` — Hermes/OpenClaw agent subprocesses must inherit that dir on PATH for Digg to activate; prior pp-digg installs use the same path) | always on if `digg-pp-cli` on PATH | yes (free, keyless, read-only) |
|
||||
|
||||
@@ -449,12 +449,13 @@ def _youtube_record(config):
|
||||
"captions for caption-free videos"
|
||||
)
|
||||
record["fix"] = _fix_text(entry)
|
||||
# Comment *text* comes from ScrapeCreators, never yt-dlp (yt-dlp yields
|
||||
# search, transcripts, and a comment count only). Say so accurately so a
|
||||
# user does not expect yt-dlp to surface comment text.
|
||||
# Comment *text* is free via yt-dlp, so this caveat only fires when yt-dlp
|
||||
# is absent and the legacy ScrapeCreators path is the only one left. Never
|
||||
# prescribe a paid key for something the installed toolchain already does.
|
||||
if not env.is_youtube_comments_available(config):
|
||||
notes.append(
|
||||
"comment text needs a ScrapeCreators key + youtube_comments opt-in"
|
||||
"comment text needs yt-dlp (free) or a ScrapeCreators key "
|
||||
"+ youtube_comments opt-in"
|
||||
)
|
||||
# Actionable fix, matching the transcription branch. The transcription
|
||||
# fix takes precedence when both caveats fire (one fix line per record).
|
||||
|
||||
@@ -887,14 +887,20 @@ def is_ytdlp_available() -> bool:
|
||||
def is_youtube_comments_available(config: dict[str, Any]) -> bool:
|
||||
"""Check if YouTube comment enrichment is available.
|
||||
|
||||
Requires SCRAPECREATORS_API_KEY AND ``youtube_comments`` in
|
||||
``INCLUDE_SOURCES`` (mirrors ``is_tiktok_comments_available``). Cost is
|
||||
bounded by ``enrich_with_comments(max_videos=3)`` (~3 credits per run).
|
||||
yt-dlp fetches YouTube comments free and keyless, so when it is installed
|
||||
comments need no credential and no ``INCLUDE_SOURCES`` opt-in — the opt-in
|
||||
only ever existed to gate ScrapeCreators credit spend, and there is none to
|
||||
gate. ``EXCLUDE_SOURCES=youtube_comments`` remains the off-switch.
|
||||
|
||||
In the default onboarding tier: the Recommended tier now enables comments
|
||||
(posts on -> comments on for TikTok/Instagram/YouTube), writing
|
||||
``youtube_comments`` into INCLUDE_SOURCES.
|
||||
Without yt-dlp, the legacy ScrapeCreators path still applies: it requires
|
||||
SCRAPECREATORS_API_KEY AND ``youtube_comments`` in ``INCLUDE_SOURCES``
|
||||
(mirroring ``is_tiktok_comments_available``), bounded by
|
||||
``enrich_with_comments(max_videos=3)`` at ~3 credits per run.
|
||||
"""
|
||||
if 'youtube_comments' in _parse_exclude_sources(config):
|
||||
return False
|
||||
if is_ytdlp_available():
|
||||
return True
|
||||
if not config.get('SCRAPECREATORS_API_KEY'):
|
||||
return False
|
||||
return 'youtube_comments' in _parse_include_sources(config)
|
||||
|
||||
@@ -71,6 +71,9 @@ _TRANSCRIPT_MAX_RETRIES = 2
|
||||
_TRANSCRIPT_BACKOFF_BASE = 2.0 # seconds; multiplied by (attempt + 1)
|
||||
_TRANSCRIPT_TIMEOUT = 30 # seconds per yt-dlp attempt (keyless: no fallback to fail over to)
|
||||
_TRANSCRIPT_FAST_TIMEOUT = 12 # seconds per attempt when a ScrapeCreators fallback exists
|
||||
# Comments are enrichment, not core evidence: keep the budget tight so a slow
|
||||
# comment API can never dominate a run's wall clock (bounded to 3 videos).
|
||||
_COMMENT_TIMEOUT = 20
|
||||
_SC_LOW_CREDIT_THRESHOLD = 50 # warn once ScrapeCreators credits drop below this
|
||||
# Transient = worth retrying (and definitely not "no captions").
|
||||
_TRANSIENT_RE = re.compile(
|
||||
@@ -1073,7 +1076,10 @@ def enrich_with_comments(
|
||||
Returns:
|
||||
Items list (mutated in place) with top_comments added to enriched items.
|
||||
"""
|
||||
if not items or not token or max_videos <= 0:
|
||||
if not items or max_videos <= 0:
|
||||
return items
|
||||
# yt-dlp needs no key, so an empty token is only fatal when it is absent too.
|
||||
if not token and not is_ytdlp_installed():
|
||||
return items
|
||||
|
||||
ranked = sorted(items, key=_total_engagement, reverse=True)
|
||||
@@ -1106,21 +1112,108 @@ def enrich_with_comments(
|
||||
return items
|
||||
|
||||
|
||||
def _ytdlp_comments_result(
|
||||
video_id: str,
|
||||
max_comments: int = 5,
|
||||
) -> tuple[List[Dict[str, Any]], bool]:
|
||||
"""Fetch top comments via yt-dlp, returning ``(comments, ran_cleanly)``.
|
||||
|
||||
The bool distinguishes "yt-dlp succeeded, this video simply has no
|
||||
comments" (True, []) from "yt-dlp was absent or errored" (False, []), so
|
||||
the caller only spends a ScrapeCreators credit on a genuine failure — not
|
||||
on a video that legitimately has zero comments. Mirrors the transcript
|
||||
path, which is likewise careful not to bill SC for a caption-less video.
|
||||
|
||||
Comments are sorted by top so a low ``max_comments`` still returns the
|
||||
highest-voted ones rather than an arbitrary slice.
|
||||
"""
|
||||
if not is_ytdlp_installed():
|
||||
return [], False
|
||||
|
||||
cmd = _wrap_ytdlp_cmd([
|
||||
"yt-dlp",
|
||||
"--write-comments",
|
||||
"--skip-download",
|
||||
"--dump-single-json",
|
||||
"--no-warnings",
|
||||
"--ignore-config",
|
||||
"--extractor-args",
|
||||
f"youtube:comment_sort=top;max_comments={max_comments},all,{max_comments}",
|
||||
f"https://www.youtube.com/watch?v={video_id}",
|
||||
])
|
||||
|
||||
try:
|
||||
result = subproc.run_with_timeout(cmd, timeout=_COMMENT_TIMEOUT)
|
||||
except Exception as exc:
|
||||
_log(f"yt-dlp comment fetch failed for {video_id}: {exc}")
|
||||
return [], False
|
||||
|
||||
if result.returncode != 0 or not result.stdout:
|
||||
_log(f"yt-dlp comment fetch failed for {video_id} (exit {result.returncode})")
|
||||
return [], False
|
||||
|
||||
try:
|
||||
payload = json.loads(result.stdout)
|
||||
except (ValueError, TypeError) as exc:
|
||||
_log(f"yt-dlp comment JSON parse failed for {video_id}: {exc}")
|
||||
return [], False
|
||||
|
||||
comments = []
|
||||
for c in (payload.get("comments") or [])[:max_comments]:
|
||||
text = c.get("text") or ""
|
||||
if not text:
|
||||
continue
|
||||
comments.append({
|
||||
"author": c.get("author") or "",
|
||||
"text": text[:400],
|
||||
"likes": c.get("like_count") or 0,
|
||||
"date": c.get("_time_text") or "",
|
||||
})
|
||||
return comments, True
|
||||
|
||||
|
||||
def _fetch_video_comments_ytdlp(
|
||||
video_id: str,
|
||||
max_comments: int = 5,
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""Comments for a video via yt-dlp (free, keyless), or [] on any failure.
|
||||
|
||||
Thin list-returning wrapper over ``_ytdlp_comments_result`` for callers
|
||||
that don't need to tell a clean empty result from a failure.
|
||||
"""
|
||||
return _ytdlp_comments_result(video_id, max_comments)[0]
|
||||
|
||||
|
||||
def _fetch_video_comments(
|
||||
video_id: str,
|
||||
token: str,
|
||||
max_comments: int = 5,
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""Fetch comments for a single YouTube video via ScrapeCreators.
|
||||
"""Fetch comments for one video, preferring the free yt-dlp path.
|
||||
|
||||
yt-dlp is tried first because it is keyless and costs nothing.
|
||||
ScrapeCreators stays as the backstop for when yt-dlp is absent or gets
|
||||
throttled, and is only called when a token is actually configured.
|
||||
|
||||
Args:
|
||||
video_id: YouTube video ID
|
||||
token: ScrapeCreators API key
|
||||
token: ScrapeCreators API key (may be empty — yt-dlp needs none)
|
||||
max_comments: Maximum comments to return
|
||||
|
||||
Returns:
|
||||
List of comment dicts with author, text, likes, date.
|
||||
"""
|
||||
ytdlp_comments, ran_cleanly = _ytdlp_comments_result(video_id, max_comments)
|
||||
if ytdlp_comments:
|
||||
return ytdlp_comments
|
||||
# Clean run with no comments -> the video simply has none. Don't spend an
|
||||
# SC credit chasing comments that aren't there; only fall back on failure.
|
||||
if ran_cleanly:
|
||||
return []
|
||||
|
||||
if not token:
|
||||
return []
|
||||
|
||||
video_url = f"https://www.youtube.com/watch?v={video_id}"
|
||||
try:
|
||||
data = http.get(
|
||||
|
||||
+29
-13
@@ -92,8 +92,14 @@ class _Hermetic:
|
||||
"""Context manager stack making doctor runs machine-independent."""
|
||||
|
||||
def __init__(self, probe_map=None, default_status=health.MISSING):
|
||||
# yt-dlp now backs YouTube comments (free, keyless), so the comment
|
||||
# gate reads env.is_ytdlp_available() -> shutil.which on the real host.
|
||||
# Pin it to the same yt-dlp the probe_map declares, or doctor's comment
|
||||
# branch would silently depend on whether the dev box has yt-dlp.
|
||||
ytdlp_ok = (probe_map or {}).get("yt-dlp", default_status) == health.OK
|
||||
self._patches = [
|
||||
mock.patch("lib.health.probe_dependency", _probe_dep(probe_map, default_status)),
|
||||
mock.patch("lib.env.is_ytdlp_available", return_value=ytdlp_ok),
|
||||
mock.patch("lib.bird_x.is_bird_installed", return_value=False),
|
||||
mock.patch("lib.bird_x.set_credentials", lambda *a, **k: None),
|
||||
mock.patch("lib.bird_x.get_bird_status", return_value=dict(BIRD_STATUS_OFF)),
|
||||
@@ -560,11 +566,13 @@ class YoutubeTranscriptionNote(unittest.TestCase):
|
||||
self.assertIn(self.entry.fix_nl, record["fix"])
|
||||
self.assertIn(self.entry.fix_cli, record["fix"])
|
||||
|
||||
def test_comment_text_attributed_to_scrapecreators_not_ytdlp(self):
|
||||
# config has no ScrapeCreators key -> comment note names ScrapeCreators,
|
||||
# never claims comment text comes from yt-dlp.
|
||||
def test_no_paid_comment_prescription_when_ytdlp_is_installed(self):
|
||||
# yt-dlp fetches comment text free. With it installed, doctor must NOT
|
||||
# tell the user to buy a ScrapeCreators key for comments — that would
|
||||
# be selling a fix for a problem they do not have.
|
||||
note = self.report["sources"]["youtube"]["note"].lower()
|
||||
self.assertIn("comment text needs a scrapecreators key", note)
|
||||
self.assertNotIn("comment text needs", note)
|
||||
self.assertNotIn("scrapecreators", note)
|
||||
|
||||
def test_text_line_includes_the_fix_on_the_ok_line(self):
|
||||
text = doctor.render_text(self.report)
|
||||
@@ -585,28 +593,36 @@ class YoutubeCommentsFixLine(unittest.TestCase):
|
||||
"""Greptile P2: when only the comment-text caveat fires (transcription key
|
||||
present), the record still carries an actionable fix line."""
|
||||
|
||||
def test_comments_fix_names_scrapecreators_when_no_key(self):
|
||||
def test_no_comment_caveat_when_ytdlp_present(self):
|
||||
"""yt-dlp installed -> comments are free -> nothing to prescribe."""
|
||||
record = _build(
|
||||
{"GROQ_API_KEY": "dummy-groq-secret-000"},
|
||||
probe_map={"yt-dlp": health.OK},
|
||||
)["sources"]["youtube"]
|
||||
self.assertEqual("ok", record["status"])
|
||||
note = record["note"].lower()
|
||||
self.assertIn("comment text needs", note)
|
||||
self.assertNotIn("caption-free", note) # transcription caveat absent
|
||||
self.assertTrue(record["fix"], "comment-text caveat must carry a fix")
|
||||
self.assertNotIn("comment text needs", note)
|
||||
|
||||
def test_comments_fix_names_optin_when_key_present(self):
|
||||
def test_comment_caveat_fires_when_ytdlp_absent_but_sc_backs_youtube(self):
|
||||
"""No yt-dlp, but an SC key keeps YouTube itself alive. Comments then
|
||||
still need the youtube_comments opt-in, so the caveat must surface —
|
||||
and must name yt-dlp as the free way out, not only the paid one.
|
||||
|
||||
(With neither yt-dlp nor a key, YouTube has no backend at all and the
|
||||
record short-circuits to 'no backend configured' — no video, no
|
||||
comments to caveat.)
|
||||
"""
|
||||
record = _build(
|
||||
{
|
||||
"GROQ_API_KEY": "dummy-groq-secret-000",
|
||||
"SCRAPECREATORS_API_KEY": "dummy-sc-secret-000",
|
||||
},
|
||||
probe_map={"yt-dlp": health.OK},
|
||||
probe_map={"yt-dlp": health.MISSING},
|
||||
)["sources"]["youtube"]
|
||||
self.assertEqual("ok", record["status"])
|
||||
self.assertIn("youtube_comments", record["fix"])
|
||||
self.assertIn("INCLUDE_SOURCES", record["fix"])
|
||||
note = record["note"].lower()
|
||||
self.assertIn("comment text needs", note)
|
||||
self.assertIn("yt-dlp (free)", note)
|
||||
self.assertTrue(record["fix"], "comment-text caveat must carry a fix")
|
||||
|
||||
def test_transcription_fix_takes_precedence_when_both_fire(self):
|
||||
record = _build({}, probe_map={"yt-dlp": health.OK})["sources"]["youtube"]
|
||||
|
||||
@@ -0,0 +1,198 @@
|
||||
"""YouTube comments via yt-dlp: the free, keyless path.
|
||||
|
||||
ScrapeCreators used to be the only way to get YouTube comments. yt-dlp already
|
||||
powers YouTube search and transcripts here and can fetch comments too, so the
|
||||
comment lane no longer needs a paid key. These tests lock in that yt-dlp is
|
||||
preferred, that ScrapeCreators still works as a fallback, and that a missing
|
||||
key is no longer fatal.
|
||||
"""
|
||||
|
||||
import json
|
||||
import unittest
|
||||
from unittest import mock
|
||||
|
||||
from lib import env, youtube_yt
|
||||
from lib.subproc import SubprocResult
|
||||
|
||||
|
||||
def _ytdlp_payload(comments):
|
||||
"""A yt-dlp --dump-single-json blob carrying `comments`."""
|
||||
return json.dumps({"id": "abc123", "title": "vid", "comments": comments})
|
||||
|
||||
|
||||
# yt-dlp's real comment shape, as emitted by --write-comments.
|
||||
_RAW = [
|
||||
{
|
||||
"author": "@BestFlorin",
|
||||
"text": "Trump said the Hormuz strait is open",
|
||||
"like_count": 11,
|
||||
"_time_text": "2 days ago",
|
||||
},
|
||||
{
|
||||
"author": "@princem4006",
|
||||
"text": "The U.S. cannot be trusted here",
|
||||
"like_count": 7,
|
||||
"_time_text": "1 day ago",
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
class TestFetchViaYtdlp(unittest.TestCase):
|
||||
def test_parses_ytdlp_comments_into_canonical_shape(self):
|
||||
"""yt-dlp's like_count/_time_text map onto the engine's likes/date."""
|
||||
result = SubprocResult(returncode=0, stdout=_ytdlp_payload(_RAW), stderr="")
|
||||
with mock.patch.object(youtube_yt, "is_ytdlp_installed", return_value=True), \
|
||||
mock.patch.object(youtube_yt.subproc, "run_with_timeout", return_value=result):
|
||||
got = youtube_yt._fetch_video_comments_ytdlp("abc123", max_comments=5)
|
||||
|
||||
self.assertEqual(2, len(got))
|
||||
self.assertEqual(
|
||||
{
|
||||
"author": "@BestFlorin",
|
||||
"text": "Trump said the Hormuz strait is open",
|
||||
"likes": 11,
|
||||
"date": "2 days ago",
|
||||
},
|
||||
got[0],
|
||||
)
|
||||
|
||||
def test_honors_max_comments(self):
|
||||
result = SubprocResult(returncode=0, stdout=_ytdlp_payload(_RAW), stderr="")
|
||||
with mock.patch.object(youtube_yt, "is_ytdlp_installed", return_value=True), \
|
||||
mock.patch.object(youtube_yt.subproc, "run_with_timeout", return_value=result):
|
||||
got = youtube_yt._fetch_video_comments_ytdlp("abc123", max_comments=1)
|
||||
|
||||
self.assertEqual(1, len(got))
|
||||
|
||||
def test_returns_empty_when_ytdlp_not_installed(self):
|
||||
with mock.patch.object(youtube_yt, "is_ytdlp_installed", return_value=False):
|
||||
self.assertEqual([], youtube_yt._fetch_video_comments_ytdlp("abc123"))
|
||||
|
||||
def test_returns_empty_on_ytdlp_failure(self):
|
||||
"""A non-zero exit is a fetch error, not an empty comment section."""
|
||||
result = SubprocResult(returncode=1, stdout="", stderr="boom")
|
||||
with mock.patch.object(youtube_yt, "is_ytdlp_installed", return_value=True), \
|
||||
mock.patch.object(youtube_yt.subproc, "run_with_timeout", return_value=result):
|
||||
self.assertEqual([], youtube_yt._fetch_video_comments_ytdlp("abc123"))
|
||||
|
||||
def test_command_requests_top_sorted_comments(self):
|
||||
"""Lock the command: top-sort and the max_comments cap must be present,
|
||||
or a refactor could silently return arbitrary (newest) comments."""
|
||||
result = SubprocResult(returncode=0, stdout=_ytdlp_payload(_RAW), stderr="")
|
||||
with mock.patch.object(youtube_yt, "is_ytdlp_installed", return_value=True), \
|
||||
mock.patch.object(youtube_yt.subproc, "run_with_timeout", return_value=result) as run:
|
||||
youtube_yt._fetch_video_comments_ytdlp("abc123", max_comments=4)
|
||||
|
||||
cmd = run.call_args.args[0]
|
||||
joined = " ".join(cmd)
|
||||
self.assertIn("--write-comments", cmd)
|
||||
self.assertIn("comment_sort=top", joined)
|
||||
self.assertIn("max_comments=4", joined)
|
||||
self.assertTrue(any("watch?v=abc123" in a for a in cmd))
|
||||
|
||||
|
||||
class TestBackendPreference(unittest.TestCase):
|
||||
def test_prefers_ytdlp_and_never_calls_scrapecreators(self):
|
||||
"""The free path wins: no SC credit is spent when yt-dlp delivers."""
|
||||
with mock.patch.object(
|
||||
youtube_yt,
|
||||
"_ytdlp_comments_result",
|
||||
return_value=([{"author": "a", "text": "t", "likes": 1, "date": ""}], True),
|
||||
), mock.patch.object(youtube_yt.http, "get") as sc_get:
|
||||
got = youtube_yt._fetch_video_comments("abc123", token="sk-live", max_comments=5)
|
||||
|
||||
self.assertEqual(1, len(got))
|
||||
sc_get.assert_not_called()
|
||||
|
||||
def test_falls_back_to_scrapecreators_when_ytdlp_fails(self):
|
||||
"""SC remains the backstop when yt-dlp is missing or throttled."""
|
||||
sc_payload = {"comments": [{"text": "from SC", "author": {"name": "@x"}, "likes": 3}]}
|
||||
with mock.patch.object(youtube_yt, "_ytdlp_comments_result", return_value=([], False)), \
|
||||
mock.patch.object(youtube_yt.http, "get", return_value=sc_payload) as sc_get:
|
||||
got = youtube_yt._fetch_video_comments("abc123", token="sk-live", max_comments=5)
|
||||
|
||||
sc_get.assert_called_once()
|
||||
self.assertEqual("from SC", got[0]["text"])
|
||||
|
||||
def test_no_token_and_ytdlp_failure_yields_no_comments_without_calling_sc(self):
|
||||
with mock.patch.object(youtube_yt, "_ytdlp_comments_result", return_value=([], False)), \
|
||||
mock.patch.object(youtube_yt.http, "get") as sc_get:
|
||||
got = youtube_yt._fetch_video_comments("abc123", token="", max_comments=5)
|
||||
|
||||
self.assertEqual([], got)
|
||||
sc_get.assert_not_called()
|
||||
|
||||
def test_no_sc_fallback_when_ytdlp_succeeds_with_zero_comments(self):
|
||||
"""A video that genuinely has no comments must not burn an SC credit.
|
||||
yt-dlp exit 0 + empty comments is success, not a throttle to retry."""
|
||||
ok_empty = SubprocResult(returncode=0, stdout=_ytdlp_payload([]), stderr="")
|
||||
with mock.patch.object(youtube_yt, "is_ytdlp_installed", return_value=True), \
|
||||
mock.patch.object(youtube_yt.subproc, "run_with_timeout", return_value=ok_empty), \
|
||||
mock.patch.object(youtube_yt.http, "get") as sc_get:
|
||||
got = youtube_yt._fetch_video_comments("abc123", token="sk-live", max_comments=5)
|
||||
|
||||
self.assertEqual([], got)
|
||||
sc_get.assert_not_called()
|
||||
|
||||
def test_sc_fallback_fires_when_ytdlp_actually_fails(self):
|
||||
"""A non-zero exit is a real failure -> SC backstop should still fire."""
|
||||
failed = SubprocResult(returncode=1, stdout="", stderr="throttled")
|
||||
sc_payload = {"comments": [{"text": "from SC", "author": {"name": "@x"}, "likes": 3}]}
|
||||
with mock.patch.object(youtube_yt, "is_ytdlp_installed", return_value=True), \
|
||||
mock.patch.object(youtube_yt.subproc, "run_with_timeout", return_value=failed), \
|
||||
mock.patch.object(youtube_yt.http, "get", return_value=sc_payload) as sc_get:
|
||||
got = youtube_yt._fetch_video_comments("abc123", token="sk-live", max_comments=5)
|
||||
|
||||
sc_get.assert_called_once()
|
||||
self.assertEqual("from SC", got[0]["text"])
|
||||
|
||||
|
||||
class TestEnrichWithoutKey(unittest.TestCase):
|
||||
def test_enriches_with_empty_token_when_ytdlp_available(self):
|
||||
"""A missing ScrapeCreators key must no longer disable comments."""
|
||||
items = [{"video_id": "abc123", "engagement": {"views": 100}}]
|
||||
with mock.patch.object(youtube_yt, "is_ytdlp_installed", return_value=True), \
|
||||
mock.patch.object(
|
||||
youtube_yt,
|
||||
"_fetch_video_comments",
|
||||
return_value=[{"author": "@a", "text": "hi", "likes": 2, "date": ""}],
|
||||
):
|
||||
youtube_yt.enrich_with_comments(items, token="")
|
||||
|
||||
self.assertEqual("hi", items[0]["top_comments"][0]["text"])
|
||||
|
||||
def test_noop_with_no_token_and_no_ytdlp(self):
|
||||
items = [{"video_id": "abc123", "engagement": {"views": 100}}]
|
||||
with mock.patch.object(youtube_yt, "is_ytdlp_installed", return_value=False):
|
||||
youtube_yt.enrich_with_comments(items, token="")
|
||||
|
||||
self.assertNotIn("top_comments", items[0])
|
||||
|
||||
|
||||
class TestAvailabilityGate(unittest.TestCase):
|
||||
def test_available_without_sc_key_when_ytdlp_installed(self):
|
||||
"""Comments are free now, so no key and no opt-in should be required."""
|
||||
with mock.patch.object(env, "is_ytdlp_available", return_value=True):
|
||||
self.assertTrue(env.is_youtube_comments_available({}))
|
||||
|
||||
def test_sc_path_still_available_when_ytdlp_missing(self):
|
||||
cfg = {
|
||||
"SCRAPECREATORS_API_KEY": "sk-live",
|
||||
"INCLUDE_SOURCES": "youtube_comments",
|
||||
}
|
||||
with mock.patch.object(env, "is_ytdlp_available", return_value=False):
|
||||
self.assertTrue(env.is_youtube_comments_available(cfg))
|
||||
|
||||
def test_unavailable_with_no_ytdlp_and_no_key(self):
|
||||
with mock.patch.object(env, "is_ytdlp_available", return_value=False):
|
||||
self.assertFalse(env.is_youtube_comments_available({}))
|
||||
|
||||
def test_exclude_sources_still_suppresses_the_free_path(self):
|
||||
"""Comments going default-on must not defeat the documented off-switch."""
|
||||
cfg = {"EXCLUDE_SOURCES": "youtube_comments"}
|
||||
with mock.patch.object(env, "is_ytdlp_available", return_value=True):
|
||||
self.assertFalse(env.is_youtube_comments_available(cfg))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -1042,21 +1042,27 @@ class TestScTranscriptParsing(unittest.TestCase):
|
||||
|
||||
|
||||
class TestYoutubeCommentsGating(unittest.TestCase):
|
||||
"""YouTube comments are opt-in via INCLUDE_SOURCES (Everything tier)."""
|
||||
"""The legacy ScrapeCreators comment path, which applies only when yt-dlp
|
||||
is absent. With yt-dlp installed, comments are free and need no opt-in —
|
||||
see tests/test_youtube_comments_ytdlp.py."""
|
||||
|
||||
def test_off_with_key_and_no_include_sources(self):
|
||||
"""Recommended tier (key, no INCLUDE_SOURCES) does NOT fetch comments."""
|
||||
"""SC path: key without INCLUDE_SOURCES does NOT fetch comments."""
|
||||
from lib import env
|
||||
self.assertFalse(env.is_youtube_comments_available({"SCRAPECREATORS_API_KEY": "k"}))
|
||||
with mock.patch.object(env, "is_ytdlp_available", return_value=False):
|
||||
self.assertFalse(env.is_youtube_comments_available({"SCRAPECREATORS_API_KEY": "k"}))
|
||||
|
||||
def test_on_with_include_sources(self):
|
||||
from lib import env
|
||||
cfg = {"SCRAPECREATORS_API_KEY": "k", "INCLUDE_SOURCES": "youtube_comments"}
|
||||
self.assertTrue(env.is_youtube_comments_available(cfg))
|
||||
with mock.patch.object(env, "is_ytdlp_available", return_value=False):
|
||||
self.assertTrue(env.is_youtube_comments_available(cfg))
|
||||
|
||||
def test_unavailable_without_key(self):
|
||||
"""SC path: no key and no yt-dlp means no comments at all."""
|
||||
from lib import env
|
||||
self.assertFalse(env.is_youtube_comments_available({"INCLUDE_SOURCES": "youtube_comments"}))
|
||||
with mock.patch.object(env, "is_ytdlp_available", return_value=False):
|
||||
self.assertFalse(env.is_youtube_comments_available({"INCLUDE_SOURCES": "youtube_comments"}))
|
||||
|
||||
def test_tiktok_comments_still_opt_in(self):
|
||||
"""Regression: TikTok comments must STILL require INCLUDE_SOURCES."""
|
||||
|
||||
Reference in New Issue
Block a user