Files
karaoke-gen/scripts
Andrew Beveridge ☄️andClaude Fable 5 f01213f5cb Fully-auto review for confident jobs: review_mode framework + auto-approval scorer/executor + replay tool (v0.202.0) (#947)
## Summary

Ships the **fully-automated review framework**: jobs now default to
`review_mode="auto"` — when the auto-approval scorer is fully confident
on BOTH axes, the combined review screens are skipped entirely and the
job renders straight through. Everything else falls back to human review
exactly as before.

**The confident class (deliberately narrow to start, ~5% of jobs
expected):**
- **Lyrics**: synced reference present, ≥90-99% anchor coverage, and
every gap word covered by the pre-generated AI auto-correct suggestions
(which the executor applies server-side — a faithful port of the review
UI's on-load auto-apply, including conflict-group resolution) — OR zero
gaps outright.
- **Backing vocals**: non-subjective only — no audible backing content →
clean instrumental.
- **Never-auto gates**: vocalization sections (da-da-dun/woo-woo — P3),
phantom/absurd-duration lines (P8), too-few-words, no/unsynced
reference, errored backing analysis.

**Calibrated against a 20-job human-review corpus** (all recent admin
reviews replayed + annotated): safety 100% (every job that truly needed
a human stays in review), lyrics-auto 3/20, overall-auto 1/20.

## How it works

- `screens_worker` (pre-AWAITING_REVIEW) and `audio_worker` (post-stems,
for the audio-lags-lyrics ordering) both call
`maybe_auto_complete_review()`. Every job gets a verdict recorded in
`processing_metadata.auto_approval` (shadow data keeps accumulating for
calibration); enforcement additionally requires: flag on +
`review_mode="auto"` + not made-for-you + no existing/custom
instrumental + audio complete.
- Enforce path mirrors a human "Complete Review": apply cached AI
suggestions → sanity checks → `corrections_updated.json` →
`instrumental_selection="clean"` → clear progress keys →
`GENERATING_SCREENS→REVIEW_COMPLETE` (new legal edge; no review token or
"ready for review" notification is ever issued) → trigger render worker.
- **Fail-safe everywhere**: stale suggestions, duplicate-word artifacts
(P1 self-conflict signature), state changed mid-enforce (human opened
editor), any exception → normal human review. Auto-approval can never
fail a job.
- Kill switch: `AUTO_APPROVAL_ENFORCE_ENABLED=false` (scoring
continues). Per-job: `review_mode="always_review"` (admin PATCH,
validated).
- Excluded from enforcement (still shadow-scored): made-for-you orders
(human QC is part of that product), custom-instrumental jobs.

## Also in this PR (sessions 1-3 of the initiative)

- **Review replay tool** (admin-only, prod-inert): `?replay=1` reopens
the real lyrics/instrumental review UIs read-only for completed jobs —
queue nav, Post-AI/Final toggle, edit-log action panel, dev-only audio
byte-proxy (env-gated `REVIEW_AUDIO_PROXY`, path-anchored). This is what
produced the calibration corpus.
- `auto_approval` service package: scorer (anchor/gap/AI-coverage
signals + P3/P8 gate detectors), apply engine, executor, lyrics-diff,
offline capture script.

## Testing

- 100+ new unit tests across scorer/apply/executor/replay routes; full
backend suite 4000+ green locally (8 pre-existing emulator failures
reproduce identically on pristine `main` — local-env only, green in CI).
Frontend: 1196 Jest tests green.
- Scorer validated read-only against the 20 real prod jobs in the
private calibration corpus.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-27 03:07:33 -04:00
..
2026-01-05 01:33:17 +00:00