mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary Job `dd892383` (NOMAD-1800) finished while the business Dropbox was at its 6 TiB quota. Both Dropbox upload passes failed with `insufficient_space`, but the job still went COMPLETE and emailed the customer a delivery email whose link was the literal `[Dropbox URL not available]`. The only #alerts signal was a raw traceback. Alert: https://discord.com/channels/1198691101943541770/1265517528608669707/1562874010939949110 ## Changes - **New `backend/services/delivery_guard.py`** - `completion_email_hold_reason(job)`: if the job's *effective* distribution settings deliver to Dropbox (`dropbox_path` and `brand_prefix`, the same condition the workers use, so private/NonPublished jobs count too) and `state_data.dropbox_link` is missing, the email is **held**. - `hold_completion_email(...)`: records `state_data.completion_email_held = {reason_code, reason, recipient, at}`, logs a WARNING, and sends **one actionable #alerts message per job** (`send_throttled_ops_alert`, keyed per job). The message covers the job, brand code, the masked recipient whose email was held, the reason (`insufficient_space` is classified as "Dropbox is FULL"), the fix steps and the runbook. The Pushbullet push is unchanged. - `check_dropbox_space_usage()`: calls `users_get_space_usage()`, using team allocation for the business account. It posts one #alerts message when usage reaches 95% and records its state in Firestore (`ops_failure_alerts/dropbox-space-usage`), so it stays quiet while usage stays high and re-arms once usage drops below 95%. It runs on the existing hourly `process-stale-reviews` cron, so no infra change is needed. - **`JobManager._schedule_completion_email`**: checks the hold before sending. Made-for-you ownership still transfers to the customer. - **Admin `POST /api/admin/jobs/{id}/send-completion-email`**: returns 409 when the link is missing, unless `allow_missing_dropbox_link: true` is passed. A successful send clears `completion_email_held`. - **`TemplateService.render_job_completion`**: no more `[Dropbox URL not available]` / `[YouTube URL not available]` placeholders. When a link is missing, that whole section is dropped (English default and GCS wording, plus the localized template). Jobs with no Dropbox delivery at all (tenant, no Dropbox configured) still get their email, just without a Dropbox paragraph. The YouTube-queued and private paths are unchanged. - Dropbox upload failure logs (orchestrator and native pass) go from ERROR to WARNING, so one root cause raises one alert (the held-delivery alert) instead of an error-monitor traceback. - Docs: `TROUBLESHOOTING.md` § "Held completion email (Dropbox delivery failed)" with the manual recovery steps, and `API.md` for the new 409 and override. - Version 0.282.0. Not done: item 5 of the handoff (an admin "redistribute Dropbox only" action). The existing admin re-render also re-uploads YouTube, so it isn't a clean fix for public jobs. The manual recovery steps are in TROUBLESHOOTING.md instead. ## Testing - New `backend/tests/test_delivery_guard.py` (31 tests): failure classification, hold/no-hold matrix (no link, link present, no Dropbox, tenant, private defaults, settings error), held marker and alert content, quota threshold, alert-once and re-arm, failed send does not latch, API error never raises. - `test_made_for_you.py`: email held when the link is missing (ownership still transfers, marker recorded, alert sent); email sent when the link is present; email sent for a job with no Dropbox delivery. - `test_admin_email_endpoints.py`: 409 without the link; override sends and clears the marker; fixed link sends and clears the marker; job with no Dropbox delivery sends normally. - `test_template_service.py`: missing links never render placeholders (en, de, unknown GCS template). - Backend unit suite (`backend/tests`, excluding emulator tests): 6637 passed, 7 failed, 25 skipped. All 7 failures are `test_internal_api.py::*requires_auth` / `*requires_admin`, which fail the same way on unmodified `origin/main` locally (they get 200 instead of 401, apparently from the local environment), and this PR doesn't touch them. - The local CodeRabbit CLI failed ("Invalid token"), so this PR is left open for the bot review. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_016F41fLoBotUw2wUjGCuAok Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Backend Development Setup
One-Time Setup
1. Create Virtual Environment
cd /Users/andrew/Projects/karaoke-gen
python3 -m venv backend/venv
2. Activate Virtual Environment
source backend/venv/bin/activate
3. Install Dependencies
IMPORTANT: Use Python 3.12 (not 3.13/3.14 - pydantic has compatibility issues)
# Backend dependencies
pip install -r backend/requirements.txt
# Install karaoke_gen in development mode
pip install -e .
# Install lyrics_transcriber submodule
pip install -e lyrics_transcriber_local/
# Install Google API client (for YouTube upload)
pip install google-api-python-client google-auth-httplib2 google-auth-oauthlib
4. Add Shell Alias (Optional)
Add to ~/.zshrc:
alias backend='cd /Users/andrew/Projects/karaoke-gen && source backend/venv/bin/activate'
Then you can just run:
backend
# Now you're in the project directory with venv activated!
Daily Workflow
Before Making Changes
# Activate venv
source backend/venv/bin/activate
# Or if you set up the alias:
backend
After Making Changes
# 1. Run full validation (catches import errors!)
python3 backend/validate.py
# 2. If validation passes, deploy
./scripts/deploy.sh
Validation Options
Full Validation (Recommended - Requires venv)
source backend/venv/bin/activate
python3 backend/validate.py
Catches:
- ✅ Syntax errors
- ✅ Import errors (like the processing_service issue!)
- ✅ Configuration issues
- ✅ FastAPI app creation problems
- ✅ Missing dependencies
This would have caught the ModuleNotFoundError!
Quick Check (No venv needed)
./backend/quick-check.sh
Catches:
- ✅ Syntax errors
- ✅ Missing files
- ✅ Some obvious import issues
Good for quick checks, but won't catch all import errors
Example: Complete Development Session
# 1. Activate environment
backend # or: source backend/venv/bin/activate
# 2. Make your changes
vim backend/api/routes/jobs.py
# 3. Validate (catches import errors!)
python3 backend/validate.py
# ✅ All validations passed!
# 4. Deploy
./scripts/deploy.sh
# 5. Test
export BACKEND_URL="https://karaoke-backend-ipzqd2k4yq-uc.a.run.app"
export AUTH_TOKEN=$(gcloud auth print-identity-token)
curl -H "Authorization: Bearer $AUTH_TOKEN" $BACKEND_URL/api/health
What About karaoke_gen Imports?
The backend imports from karaoke_gen (the CLI package):
from karaoke_gen.karaoke_gen import KaraokePrep
from karaoke_gen.audio_processor import AudioProcessor
Options:
Option 1: Install in development mode (Recommended)
source backend/venv/bin/activate
pip install -e .
This makes karaoke_gen available in the venv without copying files.
Option 2: Use system Python
Your system Python already has karaoke_gen installed, so validation will work even if the venv doesn't have it (Python will fall back to system packages).
Running the Backend Locally
# Activate venv
source backend/venv/bin/activate
# Set required environment variables
export GOOGLE_CLOUD_PROJECT="nomadkaraoke"
export GCS_BUCKET_NAME="karaoke-gen-storage-nomadkaraoke"
# Optional: Point to local credentials
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account-key.json"
# Run the server
cd backend
uvicorn main:app --reload --port 8080
Visit: http://localhost:8080/api/health
Troubleshooting
"No module named 'fastapi'"
Fix:
source backend/venv/bin/activate
pip install -r backend/requirements.txt
"No module named 'karaoke_gen'"
Fix:
source backend/venv/bin/activate
cd /Users/andrew/Projects/karaoke-gen
pip install -e .
Validation passes locally but fails in Cloud Run
Check:
- Is
requirements.txtup to date? - Did you add a new dependency without adding it to
requirements.txt? - Run:
pip freeze | grep <package-name>
Updating Dependencies
When you add new packages:
# 1. Install in venv
source backend/venv/bin/activate
pip install new-package
# 2. Update requirements.txt
pip freeze > backend/requirements.txt
# 3. Or manually add just the new package:
echo "new-package==1.2.3" >> backend/requirements.txt
# 4. Validate
python3 backend/validate.py
# 5. Deploy
./scripts/deploy.sh
Summary
Best practice workflow:
- One-time setup: Create venv and install dependencies
- Daily: Activate venv when working on backend
- Before deploy: Run
python3 backend/validate.py(catches import errors!) - Deploy: Use
./scripts/deploy.sh
The import error would have been caught if you ran validation in a venv! ✅