mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary Follows #1269. The daily description backfill already fetches every channel video's snippet, status and content details, then throws them away. This saves them to Firestore as a lasting record of the channel, so we can investigate YouTube video activity later. It uses no extra YouTube quota. ## Data model - `youtube_videos/{video_id}`, one doc per video: - from YouTube: title, description, tags, published_at, privacy_status, upload_status, category_id, duration, default_language, made_for_kids, license, embeddable; - parsed from the title: artist, song_title, brand_code, kind, backfill_target; - bookkeeping: first_seen_at, last_seen_at, last_changed_at, in_uploads_playlist, deleted, deleted_at. - `youtube_videos/{id}/history/{auto}`: `{at, event, changes: {field: {old, new}}, source}`. - `event`: first_seen, changed, deleted or restored. - `source`: `backfill` for this job's own rewrites, otherwise `observed`. - If a Studio edit and a backfill rewrite land in the same run, they're logged as two separate entries. - **Soft-delete only.** A known video YouTube stops returning gets `deleted=True` and a `deleted_at`. It stays in the known-ID list and keeps being checked, so if it comes back that's logged as `restored`. - **Writes only on change.** `last_seen_at` is refreshed at most once every 20 hours. A video's doc and its history entries always go in one Firestore batch. - **Best effort:** if the catalogue sync fails, the run logs a WARNING and still succeeds. A dry run skips it. ## Changes - New `backend/services/youtube_video_catalog.py` (100% covered). - `fetch_snippets` also requests `contentDetails`, at no extra quota cost. - The worker passes the metadata from before each rewrite (for attribution), keeps gone IDs in `known_video_ids`, and adds `catalog` stats to the run summary. - README `scripts/youtube-descriptions` documents the catalogue. Version 0.297.3. ## Testing - `pytest backend/tests -k "youtube_comment or backfill or video_catalog"`: 129 passed. - 12 new tests use an in-memory Firestore fake. They cover first seen, unchanged same day and next day, Studio edits, backfill attribution, both kinds of change in one run, soft-delete once, a stub for a video never catalogued, restore, playlist-only changes, and batches never splitting a doc from its history. - After deploy: one manual run on prod to fill the catalogue (expect about 1.9k new docs), then check it. Local CodeRabbit: 2 rounds, 3 minor issues fixed (source attribution, doc and history in one batch, a video first seen and rewritten in the same run). @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVVbErmEFhmiW48uuFK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>