mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary Andrew got "[YouTube Descriptions] 5 updated, 0 remaining" on 2026-11-14. The cause: the channel's uploads playlist (`playlistItems.list` on `UU…`) isn't a complete list of the channel. - Its pages overlap: 1914 items, but 11 are duplicates. - It leaves out public videos for weeks at a time. Five 2023 This Is Me Smiling videos were missing for the whole Oct 22 template cycle, so the backfill reported "complete" while they still had the old description. They came back on 11-14 and were rewritten. - `hSPiNfrqf5w` (Alive in the Chase) is still on the pre-10-22 template, and `QP2lKxkfeqE` is missing from the playlist again right now. ## Changes - `youtube_backfill.iter_all_video_ids` removes duplicate IDs. - New `youtube_backfill.enumerate_channel(youtube, known_ids)`: lists the playlist, then looks up known IDs it left out directly with `videos.list` (about 1 unit per 50 IDs). Returns `recovered_ids` and `gone_ids`. - The worker keeps every ID it has seen in Firestore `youtube_backfill/known_video_ids`. - The first run seeds the set from the quota ledger's `desc-backfill:<id>` entries (1800 IDs on prod, read-only check). - If seeding fails, nothing is saved, so the next run retries. - Deleted videos are dropped from the set. - A dry run saves nothing. - The run summary and progress email report `recovered_from_known`. A WARNING log names the IDs the playlist omitted. - Docs: `scripts/youtube-descriptions/README.md`, `docs/LESSONS-LEARNED.md`. Version 0.297.2. ## Testing - `pytest backend/tests -k "youtube_comment or backfill"`: 114 passed. - New tests cover duplicate removal, recovering videos the playlist omits, gone IDs, seeding from the ledger, a failed seed being retried, no saves on a dry run, and the email count. - Seeding checked against prod Firestore (read-only): 1800 IDs, including all 5 affected videos and Alive in the Chase. - The first prod run after deploy should log "uploads playlist omitted N known videos" and rewrite Alive in the Chase. Local CodeRabbit: 2 rounds. Fixed 1 major (failed seed was never retried) and 1 minor (loose test assertion). @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVVbErmEFhmiW48uuFK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Backend Development Setup
One-Time Setup
1. Create Virtual Environment
cd /Users/andrew/Projects/karaoke-gen
python3 -m venv backend/venv
2. Activate Virtual Environment
source backend/venv/bin/activate
3. Install Dependencies
IMPORTANT: Use Python 3.12 (not 3.13/3.14 - pydantic has compatibility issues)
# Backend dependencies
pip install -r backend/requirements.txt
# Install karaoke_gen in development mode
pip install -e .
# Install lyrics_transcriber submodule
pip install -e lyrics_transcriber_local/
# Install Google API client (for YouTube upload)
pip install google-api-python-client google-auth-httplib2 google-auth-oauthlib
4. Add Shell Alias (Optional)
Add to ~/.zshrc:
alias backend='cd /Users/andrew/Projects/karaoke-gen && source backend/venv/bin/activate'
Then you can just run:
backend
# Now you're in the project directory with venv activated!
Daily Workflow
Before Making Changes
# Activate venv
source backend/venv/bin/activate
# Or if you set up the alias:
backend
After Making Changes
# 1. Run full validation (catches import errors!)
python3 backend/validate.py
# 2. If validation passes, deploy
./scripts/deploy.sh
Validation Options
Full Validation (Recommended - Requires venv)
source backend/venv/bin/activate
python3 backend/validate.py
Catches:
- ✅ Syntax errors
- ✅ Import errors (like the processing_service issue!)
- ✅ Configuration issues
- ✅ FastAPI app creation problems
- ✅ Missing dependencies
This would have caught the ModuleNotFoundError!
Quick Check (No venv needed)
./backend/quick-check.sh
Catches:
- ✅ Syntax errors
- ✅ Missing files
- ✅ Some obvious import issues
Good for quick checks, but won't catch all import errors
Example: Complete Development Session
# 1. Activate environment
backend # or: source backend/venv/bin/activate
# 2. Make your changes
vim backend/api/routes/jobs.py
# 3. Validate (catches import errors!)
python3 backend/validate.py
# ✅ All validations passed!
# 4. Deploy
./scripts/deploy.sh
# 5. Test
export BACKEND_URL="https://karaoke-backend-ipzqd2k4yq-uc.a.run.app"
export AUTH_TOKEN=$(gcloud auth print-identity-token)
curl -H "Authorization: Bearer $AUTH_TOKEN" $BACKEND_URL/api/health
What About karaoke_gen Imports?
The backend imports from karaoke_gen (the CLI package):
from karaoke_gen.karaoke_gen import KaraokePrep
from karaoke_gen.audio_processor import AudioProcessor
Options:
Option 1: Install in development mode (Recommended)
source backend/venv/bin/activate
pip install -e .
This makes karaoke_gen available in the venv without copying files.
Option 2: Use system Python
Your system Python already has karaoke_gen installed, so validation will work even if the venv doesn't have it (Python will fall back to system packages).
Running the Backend Locally
# Activate venv
source backend/venv/bin/activate
# Set required environment variables
export GOOGLE_CLOUD_PROJECT="nomadkaraoke"
export GCS_BUCKET_NAME="karaoke-gen-storage-nomadkaraoke"
# Optional: Point to local credentials
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account-key.json"
# Run the server
cd backend
uvicorn main:app --reload --port 8080
Visit: http://localhost:8080/api/health
Troubleshooting
"No module named 'fastapi'"
Fix:
source backend/venv/bin/activate
pip install -r backend/requirements.txt
"No module named 'karaoke_gen'"
Fix:
source backend/venv/bin/activate
cd /Users/andrew/Projects/karaoke-gen
pip install -e .
Validation passes locally but fails in Cloud Run
Check:
- Is
requirements.txtup to date? - Did you add a new dependency without adding it to
requirements.txt? - Run:
pip freeze | grep <package-name>
Updating Dependencies
When you add new packages:
# 1. Install in venv
source backend/venv/bin/activate
pip install new-package
# 2. Update requirements.txt
pip freeze > backend/requirements.txt
# 3. Or manually add just the new package:
echo "new-package==1.2.3" >> backend/requirements.txt
# 4. Validate
python3 backend/validate.py
# 5. Deploy
./scripts/deploy.sh
Summary
Best practice workflow:
- One-time setup: Create venv and install dependencies
- Daily: Activate venv when working on backend
- Before deploy: Run
python3 backend/validate.py(catches import errors!) - Deploy: Use
./scripts/deploy.sh
The import error would have been caught if you ran validation in a venv! ✅