Files
Andrew Beveridge ☄️andClaude Opus 5.5 1840645faa feat(youtube): Firestore catalogue of every channel video with change history (v0.297.3) (#1270)
## Summary
Follows #1269. The daily description backfill already fetches every
channel video's snippet, status and content details, then throws them
away. This saves them to Firestore as a lasting record of the channel,
so we can investigate YouTube video activity later. It uses no extra
YouTube quota.

## Data model
- `youtube_videos/{video_id}`, one doc per video:
- from YouTube: title, description, tags, published_at, privacy_status,
upload_status, category_id, duration, default_language, made_for_kids,
license, embeddable;
- parsed from the title: artist, song_title, brand_code, kind,
backfill_target;
- bookkeeping: first_seen_at, last_seen_at, last_changed_at,
in_uploads_playlist, deleted, deleted_at.
- `youtube_videos/{id}/history/{auto}`: `{at, event, changes: {field:
{old, new}}, source}`.
  - `event`: first_seen, changed, deleted or restored.
- `source`: `backfill` for this job's own rewrites, otherwise
`observed`.
- If a Studio edit and a backfill rewrite land in the same run, they're
logged as two separate entries.
- **Soft-delete only.** A known video YouTube stops returning gets
`deleted=True` and a `deleted_at`. It stays in the known-ID list and
keeps being checked, so if it comes back that's logged as `restored`.
- **Writes only on change.** `last_seen_at` is refreshed at most once
every 20 hours. A video's doc and its history entries always go in one
Firestore batch.
- **Best effort:** if the catalogue sync fails, the run logs a WARNING
and still succeeds. A dry run skips it.

## Changes
- New `backend/services/youtube_video_catalog.py` (100% covered).
- `fetch_snippets` also requests `contentDetails`, at no extra quota
cost.
- The worker passes the metadata from before each rewrite (for
attribution), keeps gone IDs in `known_video_ids`, and adds `catalog`
stats to the run summary.
- README `scripts/youtube-descriptions` documents the catalogue. Version
0.297.3.

## Testing
- `pytest backend/tests -k "youtube_comment or backfill or
video_catalog"`: 129 passed.
- 12 new tests use an in-memory Firestore fake. They cover first seen,
unchanged same day and next day, Studio edits, backfill attribution,
both kinds of change in one run, soft-delete once, a stub for a video
never catalogued, restore, playlist-only changes, and batches never
splitting a doc from its history.
- After deploy: one manual run on prod to fill the catalogue (expect
about 1.9k new docs), then check it.

Local CodeRabbit: 2 rounds, 3 minor issues fixed (source attribution,
doc and history in one batch, a video first seen and rewritten in the
same run).

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVVbErmEFhmiW48uuFK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 18:41:04 -05:00
..
2026-01-05 01:33:17 +00:00