Files
Andrew Beveridge ☄️andClaude Opus 5.5 1840645faa feat(youtube): Firestore catalogue of every channel video with change history (v0.297.3) (#1270)
## Summary
Follows #1269. The daily description backfill already fetches every
channel video's snippet, status and content details, then throws them
away. This saves them to Firestore as a lasting record of the channel,
so we can investigate YouTube video activity later. It uses no extra
YouTube quota.

## Data model
- `youtube_videos/{video_id}`, one doc per video:
- from YouTube: title, description, tags, published_at, privacy_status,
upload_status, category_id, duration, default_language, made_for_kids,
license, embeddable;
- parsed from the title: artist, song_title, brand_code, kind,
backfill_target;
- bookkeeping: first_seen_at, last_seen_at, last_changed_at,
in_uploads_playlist, deleted, deleted_at.
- `youtube_videos/{id}/history/{auto}`: `{at, event, changes: {field:
{old, new}}, source}`.
  - `event`: first_seen, changed, deleted or restored.
- `source`: `backfill` for this job's own rewrites, otherwise
`observed`.
- If a Studio edit and a backfill rewrite land in the same run, they're
logged as two separate entries.
- **Soft-delete only.** A known video YouTube stops returning gets
`deleted=True` and a `deleted_at`. It stays in the known-ID list and
keeps being checked, so if it comes back that's logged as `restored`.
- **Writes only on change.** `last_seen_at` is refreshed at most once
every 20 hours. A video's doc and its history entries always go in one
Firestore batch.
- **Best effort:** if the catalogue sync fails, the run logs a WARNING
and still succeeds. A dry run skips it.

## Changes
- New `backend/services/youtube_video_catalog.py` (100% covered).
- `fetch_snippets` also requests `contentDetails`, at no extra quota
cost.
- The worker passes the metadata from before each rewrite (for
attribution), keeps gone IDs in `known_video_ids`, and adds `catalog`
stats to the run summary.
- README `scripts/youtube-descriptions` documents the catalogue. Version
0.297.3.

## Testing
- `pytest backend/tests -k "youtube_comment or backfill or
video_catalog"`: 129 passed.
- 12 new tests use an in-memory Firestore fake. They cover first seen,
unchanged same day and next day, Studio edits, backfill attribution,
both kinds of change in one run, soft-delete once, a stub for a video
never catalogued, restore, playlist-only changes, and batches never
splitting a doc from its history.
- After deploy: one manual run on prod to fill the catalogue (expect
about 1.9k new docs), then check it.

Local CodeRabbit: 2 rounds, 3 minor issues fixed (source attribution,
doc and history in one batch, a video first seen and rewritten in the
same run).

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVVbErmEFhmiW48uuFK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 18:41:04 -05:00
..

YouTube description tooling

Keep every video description on the Nomad Karaoke YouTube channel consistent with the current template — the same one newly-published videos get.

There are two ways to drive this, both sharing one implementation (backend/services/youtube_backfill.py for classify/target/render/update-body, and backend/services/youtube_description.py for the template itself):

  1. Automated daily drain (production). A scheduled backend worker (backend/workers/youtube_description_backfill_worker.py) triggered by Cloud Scheduler → POST /api/internal/youtube-backfill/run. It rewrites a quota-capped batch each day until the whole channel matches the template, then emails a completion note. Whenever the template changes it automatically re-drains the whole channel (a template-fingerprint in Firestore starts a fresh cycle) — this is the reusable, hands-off path. See the plan doc: docs/archive/2026-09-03-youtube-descriptions-plan.md.

    The uploads playlist is not a complete index of the channel. Its pages overlap (duplicate IDs) and videos silently drop out of it for weeks (2026-11: five 2023 videos missed the whole Oct 22 template cycle, so "complete" was reported early). The worker therefore keeps every video ID it has ever seen in Firestore youtube_backfill/known_video_ids (seeded once from the quota ledger's desc-backfill:<id> entries) and looks those up directly each run. The progress email and the run summary (recovered_from_known) say how many videos the playlist left out.

    Video catalogue. Each run also mirrors every video's metadata to Firestore youtube_videos/{video_id} (title, description, tags, publish date, privacy, duration, parsed artist/song/brand code, first_seen_at, last_seen_at, last_changed_at, in_uploads_playlist). Every change is logged in youtube_videos/{id}/history as {at, event, changes: {field: {old, new}}, source} (event = first_seen / changed / deleted / restored; source = backfill for this job's own rewrites, else observed). Videos YouTube stops returning are soft-deleted (deleted=True, deleted_at), never removed, and keep being checked. No extra YouTube quota; writes only on change. Code: backend/services/youtube_video_catalog.py. Example: what changed on a video? db.collection("youtube_videos").document(vid).collection("history").order_by("at").stream().

  2. This CLI (manual / local one-offs). For analysis, dry-runs, reviewing diffs, or updating specific videos by hand.

CLI setup

Run from this directory using the karaoke-gen backend environment (has google-api-python-client + Secret Manager access). Auth reuses the production youtube-oauth-credentials secret (full read/write scope). If your ADC can't read the secret, export it locally and pass --credentials-file:

cd scripts/youtube-descriptions
export GOOGLE_CLOUD_PROJECT=nomadkaraoke
gcloud auth login
gcloud secrets versions access latest --secret=youtube-oauth-credentials \
  --project=nomadkaraoke > creds.json      # gitignored

CLI workflow

# 1) Read-only: enumerate + categorize the whole channel; write report + diffs.
python youtube_descriptions.py --credentials-file creds.json analyze
#    -> .state/report.md         (counts, review buckets)
#    -> .state/review-diffs.txt  (old -> new for every video that would change)
#    -> .state/analysis.json     (machine-readable, consumed by `apply`)

# 2) Review report.md + diffs. Curate the two lists in backend/data/youtube_backfill/:
#      skip_ids.txt     - never rewrite these
#      include_ids.txt  - force-rewrite these (optionally with a title override)

# 3) First live test on ONE video (bypasses resume-state; --update-tags enriches tags):
python youtube_descriptions.py --credentials-file creds.json apply --video-id VIDEOID --update-tags

# 4) Manual batch (resumable, quota-capped, shared 10k/day pool):
python youtube_descriptions.py --credentials-file creds.json apply --dry-run
python youtube_descriptions.py --credentials-file creds.json apply --daily-quota 6000 --update-tags

python youtube_descriptions.py status

Curation lists (shared by CLI + worker)

Both live in the backend package so they ship in the Cloud Run image:

  • backend/data/youtube_backfill/skip_ids.txt — one video ID per line (# comments).
  • backend/data/youtube_backfill/include_ids.txt — bare <id>, or <id> | <Artist> | <Title> to override the parsed artist/title (used for genuine tracks whose (Karaoke) suffix was truncated by YouTube's title limit).

Safety properties

  • analyze writes nothing to YouTube.
  • High-confidence only + auto-skip. Only titles ending in (Karaoke) are auto-eligible; demos/tutorials/lyrics-with-vocals land in review buckets and are never touched unless explicitly force-included.
  • Description-only by default. update() re-sends the existing title/tags/category/language; only --update-tags refreshes tags.
  • Idempotent + resumable. Only videos whose rendered description actually differs are updated.
  • Per-video variants must be recoverable from the live title. The worker re-checks the whole channel every day (even after a cycle completes), so any description variant the upload pipeline renders — tempo notice, translated-lyrics notice — has to be re-derived from the YouTube title or the worker will "correct" it away. classify() strips the (With Translation into <Language>) label from the parsed song title and passes the language to the renderer; add the same handling here for any new per-video variant.
  • Quota-capped. videos.update = 50 units; the worker also reserves headroom for uploads via YouTubeQuotaService.

Quota math

~1,600 videos × 50 units = ~80k units vs. 10,000/day (shared with uploads). The worker paces itself; the CLI uses --daily-quota.

.state/ and creds.json are gitignored.