## Summary Follows #1269. The daily description backfill already fetches every channel video's snippet, status and content details, then throws them away. This saves them to Firestore as a lasting record of the channel, so we can investigate YouTube video activity later. It uses no extra YouTube quota. ## Data model - `youtube_videos/{video_id}`, one doc per video: - from YouTube: title, description, tags, published_at, privacy_status, upload_status, category_id, duration, default_language, made_for_kids, license, embeddable; - parsed from the title: artist, song_title, brand_code, kind, backfill_target; - bookkeeping: first_seen_at, last_seen_at, last_changed_at, in_uploads_playlist, deleted, deleted_at. - `youtube_videos/{id}/history/{auto}`: `{at, event, changes: {field: {old, new}}, source}`. - `event`: first_seen, changed, deleted or restored. - `source`: `backfill` for this job's own rewrites, otherwise `observed`. - If a Studio edit and a backfill rewrite land in the same run, they're logged as two separate entries. - **Soft-delete only.** A known video YouTube stops returning gets `deleted=True` and a `deleted_at`. It stays in the known-ID list and keeps being checked, so if it comes back that's logged as `restored`. - **Writes only on change.** `last_seen_at` is refreshed at most once every 20 hours. A video's doc and its history entries always go in one Firestore batch. - **Best effort:** if the catalogue sync fails, the run logs a WARNING and still succeeds. A dry run skips it. ## Changes - New `backend/services/youtube_video_catalog.py` (100% covered). - `fetch_snippets` also requests `contentDetails`, at no extra quota cost. - The worker passes the metadata from before each rewrite (for attribution), keeps gone IDs in `known_video_ids`, and adds `catalog` stats to the run summary. - README `scripts/youtube-descriptions` documents the catalogue. Version 0.297.3. ## Testing - `pytest backend/tests -k "youtube_comment or backfill or video_catalog"`: 129 passed. - 12 new tests use an in-memory Firestore fake. They cover first seen, unchanged same day and next day, Studio edits, backfill attribution, both kinds of change in one run, soft-delete once, a stub for a video never catalogued, restore, playlist-only changes, and batches never splitting a doc from its history. - After deploy: one manual run on prod to fill the catalogue (expect about 1.9k new docs), then check it. Local CodeRabbit: 2 rounds, 3 minor issues fixed (source attribution, doc and history in one batch, a video first seen and rewritten in the same run). @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVVbErmEFhmiW48uuFK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
YouTube description tooling
Keep every video description on the Nomad Karaoke YouTube channel consistent with the current template — the same one newly-published videos get.
There are two ways to drive this, both sharing one implementation
(backend/services/youtube_backfill.py for classify/target/render/update-body,
and backend/services/youtube_description.py for the template itself):
-
Automated daily drain (production). A scheduled backend worker (
backend/workers/youtube_description_backfill_worker.py) triggered by Cloud Scheduler →POST /api/internal/youtube-backfill/run. It rewrites a quota-capped batch each day until the whole channel matches the template, then emails a completion note. Whenever the template changes it automatically re-drains the whole channel (a template-fingerprint in Firestore starts a fresh cycle) — this is the reusable, hands-off path. See the plan doc:docs/archive/2026-09-03-youtube-descriptions-plan.md.The uploads playlist is not a complete index of the channel. Its pages overlap (duplicate IDs) and videos silently drop out of it for weeks (2026-11: five 2023 videos missed the whole Oct 22 template cycle, so "complete" was reported early). The worker therefore keeps every video ID it has ever seen in Firestore
youtube_backfill/known_video_ids(seeded once from the quota ledger'sdesc-backfill:<id>entries) and looks those up directly each run. The progress email and the run summary (recovered_from_known) say how many videos the playlist left out.Video catalogue. Each run also mirrors every video's metadata to Firestore
youtube_videos/{video_id}(title, description, tags, publish date, privacy, duration, parsed artist/song/brand code,first_seen_at,last_seen_at,last_changed_at,in_uploads_playlist). Every change is logged inyoutube_videos/{id}/historyas{at, event, changes: {field: {old, new}}, source}(event= first_seen / changed / deleted / restored;source=backfillfor this job's own rewrites, elseobserved). Videos YouTube stops returning are soft-deleted (deleted=True,deleted_at), never removed, and keep being checked. No extra YouTube quota; writes only on change. Code:backend/services/youtube_video_catalog.py. Example: what changed on a video?db.collection("youtube_videos").document(vid).collection("history").order_by("at").stream(). -
This CLI (manual / local one-offs). For analysis, dry-runs, reviewing diffs, or updating specific videos by hand.
CLI setup
Run from this directory using the karaoke-gen backend environment (has
google-api-python-client + Secret Manager access). Auth reuses the production
youtube-oauth-credentials secret (full read/write scope). If your ADC can't
read the secret, export it locally and pass --credentials-file:
cd scripts/youtube-descriptions
export GOOGLE_CLOUD_PROJECT=nomadkaraoke
gcloud auth login
gcloud secrets versions access latest --secret=youtube-oauth-credentials \
--project=nomadkaraoke > creds.json # gitignored
CLI workflow
# 1) Read-only: enumerate + categorize the whole channel; write report + diffs.
python youtube_descriptions.py --credentials-file creds.json analyze
# -> .state/report.md (counts, review buckets)
# -> .state/review-diffs.txt (old -> new for every video that would change)
# -> .state/analysis.json (machine-readable, consumed by `apply`)
# 2) Review report.md + diffs. Curate the two lists in backend/data/youtube_backfill/:
# skip_ids.txt - never rewrite these
# include_ids.txt - force-rewrite these (optionally with a title override)
# 3) First live test on ONE video (bypasses resume-state; --update-tags enriches tags):
python youtube_descriptions.py --credentials-file creds.json apply --video-id VIDEOID --update-tags
# 4) Manual batch (resumable, quota-capped, shared 10k/day pool):
python youtube_descriptions.py --credentials-file creds.json apply --dry-run
python youtube_descriptions.py --credentials-file creds.json apply --daily-quota 6000 --update-tags
python youtube_descriptions.py status
Curation lists (shared by CLI + worker)
Both live in the backend package so they ship in the Cloud Run image:
backend/data/youtube_backfill/skip_ids.txt— one video ID per line (#comments).backend/data/youtube_backfill/include_ids.txt— bare<id>, or<id> | <Artist> | <Title>to override the parsed artist/title (used for genuine tracks whose(Karaoke)suffix was truncated by YouTube's title limit).
Safety properties
analyzewrites nothing to YouTube.- High-confidence only + auto-skip. Only titles ending in
(Karaoke)are auto-eligible; demos/tutorials/lyrics-with-vocals land in review buckets and are never touched unless explicitly force-included. - Description-only by default.
update()re-sends the existing title/tags/category/language; only--update-tagsrefreshes tags. - Idempotent + resumable. Only videos whose rendered description actually differs are updated.
- Per-video variants must be recoverable from the live title. The worker
re-checks the whole channel every day (even after a cycle completes), so any
description variant the upload pipeline renders — tempo notice, translated-lyrics
notice — has to be re-derived from the YouTube title or the worker will
"correct" it away.
classify()strips the(With Translation into <Language>)label from the parsed song title and passes the language to the renderer; add the same handling here for any new per-video variant. - Quota-capped.
videos.update= 50 units; the worker also reserves headroom for uploads viaYouTubeQuotaService.
Quota math
~1,600 videos × 50 units = ~80k units vs. 10,000/day (shared with uploads). The
worker paces itself; the CLI uses --daily-quota.
.state/ and creds.json are gitignored.