Files
karaoke-gen/scripts/run-replay-local.sh
Andrew Beveridge ☄️andClaude Fable 5 f01213f5cb Fully-auto review for confident jobs: review_mode framework + auto-approval scorer/executor + replay tool (v0.202.0) (#947)
## Summary

Ships the **fully-automated review framework**: jobs now default to
`review_mode="auto"` — when the auto-approval scorer is fully confident
on BOTH axes, the combined review screens are skipped entirely and the
job renders straight through. Everything else falls back to human review
exactly as before.

**The confident class (deliberately narrow to start, ~5% of jobs
expected):**
- **Lyrics**: synced reference present, ≥90-99% anchor coverage, and
every gap word covered by the pre-generated AI auto-correct suggestions
(which the executor applies server-side — a faithful port of the review
UI's on-load auto-apply, including conflict-group resolution) — OR zero
gaps outright.
- **Backing vocals**: non-subjective only — no audible backing content →
clean instrumental.
- **Never-auto gates**: vocalization sections (da-da-dun/woo-woo — P3),
phantom/absurd-duration lines (P8), too-few-words, no/unsynced
reference, errored backing analysis.

**Calibrated against a 20-job human-review corpus** (all recent admin
reviews replayed + annotated): safety 100% (every job that truly needed
a human stays in review), lyrics-auto 3/20, overall-auto 1/20.

## How it works

- `screens_worker` (pre-AWAITING_REVIEW) and `audio_worker` (post-stems,
for the audio-lags-lyrics ordering) both call
`maybe_auto_complete_review()`. Every job gets a verdict recorded in
`processing_metadata.auto_approval` (shadow data keeps accumulating for
calibration); enforcement additionally requires: flag on +
`review_mode="auto"` + not made-for-you + no existing/custom
instrumental + audio complete.
- Enforce path mirrors a human "Complete Review": apply cached AI
suggestions → sanity checks → `corrections_updated.json` →
`instrumental_selection="clean"` → clear progress keys →
`GENERATING_SCREENS→REVIEW_COMPLETE` (new legal edge; no review token or
"ready for review" notification is ever issued) → trigger render worker.
- **Fail-safe everywhere**: stale suggestions, duplicate-word artifacts
(P1 self-conflict signature), state changed mid-enforce (human opened
editor), any exception → normal human review. Auto-approval can never
fail a job.
- Kill switch: `AUTO_APPROVAL_ENFORCE_ENABLED=false` (scoring
continues). Per-job: `review_mode="always_review"` (admin PATCH,
validated).
- Excluded from enforcement (still shadow-scored): made-for-you orders
(human QC is part of that product), custom-instrumental jobs.

## Also in this PR (sessions 1-3 of the initiative)

- **Review replay tool** (admin-only, prod-inert): `?replay=1` reopens
the real lyrics/instrumental review UIs read-only for completed jobs —
queue nav, Post-AI/Final toggle, edit-log action panel, dev-only audio
byte-proxy (env-gated `REVIEW_AUDIO_PROXY`, path-anchored). This is what
produced the calibration corpus.
- `auto_approval` service package: scorer (anchor/gap/AI-coverage
signals + P3/P8 gate detectors), apply engine, executor, lyrics-diff,
offline capture script.

## Testing

- 100+ new unit tests across scorer/apply/executor/replay routes; full
backend suite 4000+ green locally (8 pre-existing emulator failures
reproduce identically on pristine `main` — local-env only, green in CI).
Frontend: 1196 Jest tests green.
- Scorer validated read-only against the 20 real prod jobs in the
private calibration corpus.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-27 03:07:33 -04:00

50 lines
2.1 KiB
Bash
Executable File

#!/bin/bash
# Run the backend locally in REPLAY mode against REAL production data (read-only),
# so you can re-open the lyrics-review + instrumental-selection UIs for COMPLETED
# jobs with full audio/backing-vocals playback.
#
# Uses your own gcloud ADC (must be `gcloud auth application-default login` as an
# account that can READ Firestore `jobs` + the GCS bucket). No writes are made.
# REVIEW_AUDIO_PROXY=1 makes the backend stream review audio bytes instead of GCS
# signed URLs (local user ADC can't sign) — nothing customer-facing changes.
#
# Usage:
# ./scripts/run-replay-local.sh
# then in another terminal: (cd frontend && npm run dev)
#
# See docs/REPLAY.md for the full walkthrough + the job list.
set -e
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
cd "$PROJECT_ROOT"
export GOOGLE_CLOUD_PROJECT="${GOOGLE_CLOUD_PROJECT:-nomadkaraoke}"
export GCS_BUCKET_NAME="${GCS_BUCKET_NAME:-karaoke-gen-storage-nomadkaraoke}"
export FIRESTORE_COLLECTION="${FIRESTORE_COLLECTION:-jobs}" # REAL prod jobs (not jobs-dev)
export ENVIRONMENT="development"
export EDGE_AUTH_MODE="off"
export REVIEW_AUDIO_PROXY="1" # stream bytes, no signing
export ADMIN_TOKENS="${ADMIN_TOKENS:-replay-local-token}" # matches the frontend token below
export PORT="${PORT:-8000}"
cat <<EOF
🎬 Replay backend starting on http://127.0.0.1:$PORT (REAL prod data, read-only)
Next steps:
1. In another terminal: cd frontend && npm run dev # http://localhost:3000
2. Open http://localhost:3000 in your browser, then in DevTools console run:
localStorage.setItem('karaoke_access_token', '$ADMIN_TOKENS')
(then reload — you'll be signed in as admin@nomadkaraoke.com)
3. Open a job in replay (LYRICS review):
http://localhost:3000/en/app/jobs?baseApiUrl=http://127.0.0.1:$PORT&replay=1#/<JOB_ID>/review
…or the INSTRUMENTAL screen:
http://localhost:3000/en/app/jobs?baseApiUrl=http://127.0.0.1:$PORT&replay=1#/<JOB_ID>/instrumental
Job list + walkthrough: docs/REPLAY.md
EOF
exec poetry run uvicorn backend.main:app --host 127.0.0.1 --port "$PORT"