Files
karaoke-gen/backend
Andrew Beveridge ☄️andClaude Opus 5.5 3bbf397dc4 fix: keep auth, tenant lookup and job-log writes off the API event loop (v0.278.14) (#1194)
@coderabbitai ignore

## Why
The residual event-loop freezes BACKLOG item. Seven days of
`EVENT_LOOP_STALL in progress` stacks (2026-10-12→19, 355 samples),
grouped by their outermost app frame, ranked the app-level loop
blockers:

| Source | Samples | Max |
|---|---|---|
| in-process screens worker → job-log handler → sync Firestore write per
log line | 30 | 2.2s |
| `require_auth` → `validate_token_full` (session read on **every**
authed request) | 19 | 1.9s |
| preview encodes
(`_run_preview_encode`/`_encode_preview_in_background`) → job-log write
| 15 | 2.4s |
| `change_visibility` → sync Dropbox/GCS deletes | 6 | 1.2s |
| tenant middleware → `get_tenant_config` (GCS on cache miss) | 5 | 1.2s
|
| `require_review_auth` → `get_job` | 5 | 2.2s |
| `auto_correct_worker._run_suggest` → `get_job` (the only stall since
#1192) | 1 | 1.2s |

(The render/video in-process stalls stopped with #1192.)

## What
- **`backend/services/log_write_offload.py`**: when a job-log write is
made on a running event loop in the API process, it goes to one
background writer thread. The queue is bounded at 5000 and a drop is
logged at WARNING, rate-limited; `atexit` drains the queue. Writes stay
inline in threads and Cloud Run Jobs, so the worker code paths behave
exactly as before. Logs are read back ordered by their own `timestamp`,
so the hand-off doesn't reorder them.
- **`asyncio.to_thread`** for
`require_auth`/`require_auth_legacy`/`require_review_auth`, the tenant
middleware, `rerender_with_current_theme`'s tenant lookup,
`auto_correct._run_suggest`, and `change_visibility`'s `get_job`.
- **Visibility change:** `_delete_*_outputs` are now plain `def`, and
the sync steps run in one worker thread. Moving them off the loop
removed the implicit serialisation, so both directions now **claim
`visibility_change_in_progress` atomically** before touching anything.
They use the new `FirestoreService.try_set_job_state_flag`, a
transactional test-and-set. A lost claim returns 409. The
regenerate→private continuation passes `holds_claim=True`, because its
regenerate already set the flag. The claim is released if the reset
fails, and a raising screens trigger rolls back like a `False` one.
- **Guard (`test_no_blocking_async_routes.py`):** `BLOCKING_CALLS` gains
`validate_token(_full)`, `validate_session`, `get_tenant_config`,
`append_log_to_subcollection` and `append_worker_log`. Every `async def`
in `api/dependencies.py` and `middleware/` is now checked, not just
decorated routes. Mutation-checked: reverting `require_auth` fails the
guard.
- `docs/LESSONS-LEARNED.md` entry. About 70 direct
`get_job`/`update_job` calls in async routes remain. They're single
round-trips that rarely show in the watchdog, and the doc says to move
them when one does.

## Testing
- New: `test_log_write_offload.py` (inline vs offloaded, Cloud Run Job
inline, ordering, full-queue drop, Firestore append off the loop
thread), claim tests in `test_change_visibility.py` (lost claim, release
on failure, chained continuation, trigger exception, route 409), and
emulator `test_try_set_job_state_flag.py` (once-only, already set,
missing job, one winner among 4 concurrent callers).
- Backend: 6478 passed. The 8 failures are
`test_emulator_integration.py`, caused by my local environment: another
session's emulator setup has an `ssh` forward on :4443 instead of
fake-GCS, and they fail identically on untouched main. CI's emulator job
covers them and the new emulator test.
- Local CodeRabbit: 3 cycles. I fixed the claim race (two findings),
trigger-exception rollback, and the guarded claim release. I skipped two
minor nitpicks on the pre-existing trigger-failure rollback.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Jw21syo4HizJJjYw2mg2Vp

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 14:41:34 -04:00
..

Backend Development Setup

One-Time Setup

1. Create Virtual Environment

cd /Users/andrew/Projects/karaoke-gen
python3 -m venv backend/venv

2. Activate Virtual Environment

source backend/venv/bin/activate

3. Install Dependencies

IMPORTANT: Use Python 3.12 (not 3.13/3.14 - pydantic has compatibility issues)

# Backend dependencies
pip install -r backend/requirements.txt

# Install karaoke_gen in development mode
pip install -e .

# Install lyrics_transcriber submodule
pip install -e lyrics_transcriber_local/

# Install Google API client (for YouTube upload)
pip install google-api-python-client google-auth-httplib2 google-auth-oauthlib

4. Add Shell Alias (Optional)

Add to ~/.zshrc:

alias backend='cd /Users/andrew/Projects/karaoke-gen && source backend/venv/bin/activate'

Then you can just run:

backend
# Now you're in the project directory with venv activated!

Daily Workflow

Before Making Changes

# Activate venv
source backend/venv/bin/activate

# Or if you set up the alias:
backend

After Making Changes

# 1. Run full validation (catches import errors!)
python3 backend/validate.py

# 2. If validation passes, deploy
./scripts/deploy.sh

Validation Options

source backend/venv/bin/activate
python3 backend/validate.py

Catches:

  • ✅ Syntax errors
  • ✅ Import errors (like the processing_service issue!)
  • ✅ Configuration issues
  • ✅ FastAPI app creation problems
  • ✅ Missing dependencies

This would have caught the ModuleNotFoundError!

Quick Check (No venv needed)

./backend/quick-check.sh

Catches:

  • ✅ Syntax errors
  • ✅ Missing files
  • ✅ Some obvious import issues

Good for quick checks, but won't catch all import errors


Example: Complete Development Session

# 1. Activate environment
backend  # or: source backend/venv/bin/activate

# 2. Make your changes
vim backend/api/routes/jobs.py

# 3. Validate (catches import errors!)
python3 backend/validate.py
# ✅ All validations passed!

# 4. Deploy
./scripts/deploy.sh

# 5. Test
export BACKEND_URL="https://karaoke-backend-ipzqd2k4yq-uc.a.run.app"
export AUTH_TOKEN=$(gcloud auth print-identity-token)
curl -H "Authorization: Bearer $AUTH_TOKEN" $BACKEND_URL/api/health

What About karaoke_gen Imports?

The backend imports from karaoke_gen (the CLI package):

from karaoke_gen.karaoke_gen import KaraokePrep
from karaoke_gen.audio_processor import AudioProcessor

Options:

source backend/venv/bin/activate
pip install -e .

This makes karaoke_gen available in the venv without copying files.

Option 2: Use system Python

Your system Python already has karaoke_gen installed, so validation will work even if the venv doesn't have it (Python will fall back to system packages).


Running the Backend Locally

# Activate venv
source backend/venv/bin/activate

# Set required environment variables
export GOOGLE_CLOUD_PROJECT="nomadkaraoke"
export GCS_BUCKET_NAME="karaoke-gen-storage-nomadkaraoke"

# Optional: Point to local credentials
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account-key.json"

# Run the server
cd backend
uvicorn main:app --reload --port 8080

Visit: http://localhost:8080/api/health


Troubleshooting

"No module named 'fastapi'"

Fix:

source backend/venv/bin/activate
pip install -r backend/requirements.txt

"No module named 'karaoke_gen'"

Fix:

source backend/venv/bin/activate
cd /Users/andrew/Projects/karaoke-gen
pip install -e .

Validation passes locally but fails in Cloud Run

Check:

  1. Is requirements.txt up to date?
  2. Did you add a new dependency without adding it to requirements.txt?
  3. Run: pip freeze | grep <package-name>

Updating Dependencies

When you add new packages:

# 1. Install in venv
source backend/venv/bin/activate
pip install new-package

# 2. Update requirements.txt
pip freeze > backend/requirements.txt

# 3. Or manually add just the new package:
echo "new-package==1.2.3" >> backend/requirements.txt

# 4. Validate
python3 backend/validate.py

# 5. Deploy
./scripts/deploy.sh

Summary

Best practice workflow:

  1. One-time setup: Create venv and install dependencies
  2. Daily: Activate venv when working on backend
  3. Before deploy: Run python3 backend/validate.py (catches import errors!)
  4. Deploy: Use ./scripts/deploy.sh

The import error would have been caught if you ran validation in a venv! ✅