113 Commits
Author SHA1 Message Date
Andrew Beveridge ☄️andClaude Opus 5.5 1840645faa feat(youtube): Firestore catalogue of every channel video with change history (v0.297.3) (#1270)
## Summary
Follows #1269. The daily description backfill already fetches every
channel video's snippet, status and content details, then throws them
away. This saves them to Firestore as a lasting record of the channel,
so we can investigate YouTube video activity later. It uses no extra
YouTube quota.

## Data model
- `youtube_videos/{video_id}`, one doc per video:
- from YouTube: title, description, tags, published_at, privacy_status,
upload_status, category_id, duration, default_language, made_for_kids,
license, embeddable;
- parsed from the title: artist, song_title, brand_code, kind,
backfill_target;
- bookkeeping: first_seen_at, last_seen_at, last_changed_at,
in_uploads_playlist, deleted, deleted_at.
- `youtube_videos/{id}/history/{auto}`: `{at, event, changes: {field:
{old, new}}, source}`.
  - `event`: first_seen, changed, deleted or restored.
- `source`: `backfill` for this job's own rewrites, otherwise
`observed`.
- If a Studio edit and a backfill rewrite land in the same run, they're
logged as two separate entries.
- **Soft-delete only.** A known video YouTube stops returning gets
`deleted=True` and a `deleted_at`. It stays in the known-ID list and
keeps being checked, so if it comes back that's logged as `restored`.
- **Writes only on change.** `last_seen_at` is refreshed at most once
every 20 hours. A video's doc and its history entries always go in one
Firestore batch.
- **Best effort:** if the catalogue sync fails, the run logs a WARNING
and still succeeds. A dry run skips it.

## Changes
- New `backend/services/youtube_video_catalog.py` (100% covered).
- `fetch_snippets` also requests `contentDetails`, at no extra quota
cost.
- The worker passes the metadata from before each rewrite (for
attribution), keeps gone IDs in `known_video_ids`, and adds `catalog`
stats to the run summary.
- README `scripts/youtube-descriptions` documents the catalogue. Version
0.297.3.

## Testing
- `pytest backend/tests -k "youtube_comment or backfill or
video_catalog"`: 129 passed.
- 12 new tests use an in-memory Firestore fake. They cover first seen,
unchanged same day and next day, Studio edits, backfill attribution,
both kinds of change in one run, soft-delete once, a stub for a video
never catalogued, restore, playlist-only changes, and batches never
splitting a doc from its history.
- After deploy: one manual run on prod to fill the catalogue (expect
about 1.9k new docs), then check it.

Local CodeRabbit: 2 rounds, 3 minor issues fixed (source attribution,
doc and history in one batch, a video first seen and rewritten in the
same run).

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVVbErmEFhmiW48uuFK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 18:41:04 -05:00
Andrew Beveridge ☄️andClaude Opus 5.5 4a23398c04 fix(youtube-backfill): check known video IDs the uploads playlist drops (v0.297.2) (#1269)
## Summary
Andrew got "[YouTube Descriptions] 5 updated, 0 remaining" on
2026-11-14. The cause: the channel's uploads playlist
(`playlistItems.list` on `UU…`) isn't a complete list of the channel.
- Its pages overlap: 1914 items, but 11 are duplicates.
- It leaves out public videos for weeks at a time. Five 2023 This Is Me
Smiling videos were missing for the whole Oct 22 template cycle, so the
backfill reported "complete" while they still had the old description.
They came back on 11-14 and were rewritten.
- `hSPiNfrqf5w` (Alive in the Chase) is still on the pre-10-22 template,
and `QP2lKxkfeqE` is missing from the playlist again right now.

## Changes
- `youtube_backfill.iter_all_video_ids` removes duplicate IDs.
- New `youtube_backfill.enumerate_channel(youtube, known_ids)`: lists
the playlist, then looks up known IDs it left out directly with
`videos.list` (about 1 unit per 50 IDs). Returns `recovered_ids` and
`gone_ids`.
- The worker keeps every ID it has seen in Firestore
`youtube_backfill/known_video_ids`.
- The first run seeds the set from the quota ledger's
`desc-backfill:<id>` entries (1800 IDs on prod, read-only check).
  - If seeding fails, nothing is saved, so the next run retries.
  - Deleted videos are dropped from the set.
  - A dry run saves nothing.
- The run summary and progress email report `recovered_from_known`. A
WARNING log names the IDs the playlist omitted.
- Docs: `scripts/youtube-descriptions/README.md`,
`docs/LESSONS-LEARNED.md`. Version 0.297.2.

## Testing
- `pytest backend/tests -k "youtube_comment or backfill"`: 114 passed.
- New tests cover duplicate removal, recovering videos the playlist
omits, gone IDs, seeding from the ledger, a failed seed being retried,
no saves on a dry run, and the email count.
- Seeding checked against prod Firestore (read-only): 1800 IDs,
including all 5 affected videos and Alive in the Chase.
- The first prod run after deploy should log "uploads playlist omitted N
known videos" and rewrite Alive in the Chase.

Local CodeRabbit: 2 rounds. Fixed 1 major (failed seed was never
retried) and 1 minor (loose test assertion).

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVVbErmEFhmiW48uuFK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 14:03:13 -05:00
Andrew Beveridge ☄️andClaude Opus 5.5 2db5407a5d feat: archive deleted jobs (doc + logs) instead of losing them (v0.295.0) (#1261)
## Why
When NOMAD-1875 was deleted (2026-11-09), its Firestore record and logs
were gone for good, so we couldn't use them to debug the delete-outputs
result afterwards. Andrew's call: job records are **system data**. Keep
them forever, and on account deletion redact the user's identity rather
than deleting the record.

## What
- `FirestoreService.archive_job()` copies `jobs/{id}` to
`jobs_archive/{id}`, adding `archived_at`, `archived_by` and
`archive_reason`. It also copies the `logs` subcollection to
`archived_logs` without `ttl_expiry` (the TTL policy is on the `logs`
collection group, so otherwise they'd expire after 30 days), and the
`audio_edit_sessions` subcollection.
- `JobManager.delete_job()` archives first and **aborts the delete if
the archive fails**.
- Callers pass who deleted the job and why: `admin_delete` /
`owner_delete` from `DELETE /api/jobs/{id}`, and
`cleanup_distribution_delete`.
- **Account deletion** now also redacts archived jobs: owner/customer
email becomes `deleted:<uuid>`, and IPs, notes and contact fields are
removed. These are single-field queries, so no new composite index is
needed. Archived jobs are never deleted.
- Bulk test cleanup (`delete_jobs_by_filter`) still hard-deletes.
- `scripts/get_job.py` falls back to the archive. Documented in
`docs/ARCHITECTURE.md`.

No infra change: Firestore creates the collection on first write, and no
TTL policy is set because archived jobs are kept forever.

## Testing
- Unit: archive happens before delete; the delete is aborted when the
archive fails; archived jobs are redacted on account deletion (including
made-for-you customer jobs).
- Emulator: `backend/tests/emulator/test_job_archive_emulator.py` checks
the doc, logs (no `ttl_expiry`) and audio-edit sessions are copied, and
that a missing job returns False.
- `test_account_deletion`, `test_job_manager`, `test_jobs_api`,
`test_services`, `test_api_integration`: 195 passed.
- Local CodeRabbit: no findings.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVUwKapcjF8czVQrPD5

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 12:06:51 -05:00
Andrew Beveridge ☄️andClaude Opus 5.5 ce57b4b598 fix(review): Tap To Sync partial pass keeps word order; local stack never reaches prod (v0.288.1) (#1237)
## Summary

### 1. Tap To Sync: a pass that stops before the last word
(customer-facing)
If a pass stopped before the last word (auto-stop about a second after
the line end, or the user pressing Stop), the untapped words after the
last tapped one kept their **old** timings, and those could now start
*before* the last tapped word. On gen's fake fixture line ("Nothing here
was ever transcribed", words 15.5 to 19.5, next line at 20.0), taps at
16.0, 17.2, 18.4 and 19.6 followed by auto-stop left "ever" at 19.6 and
"transcribed" at a stale 18.74 to 19.5. The two words were out of order,
and the video rendered them out of order.

- `frontend/hooks/useManualSync.ts`: new pure
`commitPartialSyncPass(words, synced)` runs in `cleanupManualSync`,
which is both the auto-stop path and the Stop toggle. It clears (sets
start/end to null) every untapped word after the last tapped one that
starts before it, then caps the last tapped word's end before the next
still-timed word in the line. The existing untimed-lyrics guard then
blocks finishing and points the user at those words. This is the same
rule as nomad-mobile's review-core `commitTapSync`
(`packages/review-core/src/tapSync.ts`, branch
`feat/sess-20261030-1930-e2e-health`). It uses the hook's existing
`syncedIndicesRef`.
- Backend backstop, `backend/services/timing_sanitizer.py`
`sanitize_corrections`: within a line, a word that starts before the
previous word's **start** is now treated like any other invalid start.
Small overlaps (a start a few ms before the previous word's end, common
in AI transcriptions) are left exactly as on main, so they get no move,
no WARNING and no edit-modal toast. It is **moved** to the previous
word's end, not cleared. I chose moving because that is how the module
already repairs bad or missing starts, because a cleared word would just
be refilled at prev_end on the next sanitize, and because nulls would
crash the render (`find_missing_timings` only guards the preview). The
module logs one aggregated `WARNING` per submission (never ERROR, since
this is an expected and repaired condition).
- `frontend/lib/lyrics-review/sanitizeWordTimings.ts` (the frontend
mirror, run when the edit modal opens) applies the same ordering rule,
with the same exemption for small overlaps.

### 2. The local stack must not reach production
nomad-mobile's E2E runs gen web inside an Android emulator's Chrome.
- `components/google-analytics.tsx`: GA (`G-D0E63LB9FY`) is no longer
loaded when the page host is localhost, 127.0.0.1 or 10.0.2.2. The check
runs at runtime after mount, so it also covers CI, and the server render
and first client render match.
- `lib/api/autoCorrect.ts` and `lib/api/customLyrics.ts` now use
`API_BASE_URL` from `lib/api.ts` instead of `NEXT_PUBLIC_API_BASE ??
'https://api.nomadkaraoke.com'`. That env var is set nowhere. A local
page now calls its own backend through relative URLs, so it never sends
its bearer token to production.
- New `lib/local-host.ts` is the single list of local hostnames, shared
by `lib/api.ts` and GA. `lib/api.ts` now also treats 10.0.2.2 as local.
- `scripts/local-stack.sh` backend_env: blank `FLACFETCH_API_URL=` and
`FLACFETCH_API_KEY=`, so direnv can't point the local stack at the real
flacfetch.

Version is bumped to 0.288.1.

## Testing
Each change had a failing test written first.
- Frontend jest (hook, sanitizer, edit modals, GA, API clients,
api/local-mode, auto-correct, custom lyrics): 13 suites, **150 passed**.
- New tests: 3 partial-pass hook tests (auto-stop, Stop, and the
keep-and-cap case), 2 sanitizer tests (the stale-word move, and a 20 ms
overlap left untouched), 4 GA tests, and 8 API-base tests (local hosts
use relative URLs, the production host uses the production API).
- Backend pytest: `test_timing_sanitizer.py` +
`test_jobs_corrections.py` **20 passed** (3 new, including a 20 ms
overlap that is left untouched), and `test_routes_review.py` **114
passed**.
- `customLyrics.test.ts`'s AbortSignal test now asserts that the
caller's abort reaches fetch, not that the signal object is identical. A
relative URL makes `apiFetch` manage the request and wrap the signal.

## Notes
- The web hook has no view of the next line, so the cap only uses words
in the same line. The backend sanitizer and segment bounds handle the
rest.
- The render-time `SegmentResizer._sanitize_segment_timings` is
unchanged (it clamps to the window only).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVSLS3eVDqPctfYBMTt



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved lyric timing cleanup when recording or editing stops before
the final word. Out-of-order timings are corrected while small overlaps
are preserved, helping keep lyrics synchronized without changing valid
timings.
* Corrected out-of-order word timings during lyric cleanup and editing.
* **Improvements**
* Local app use now works more consistently across supported addresses,
including Android emulators, with API requests routed appropriately.
  * Analytics scripts are now omitted on local pages.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-30 20:28:19 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 65940c6f57 fix(local-stack): decide partner calls go to local decide, never production (#1235)
The local stack's backend env had no `KARAOKE_DECIDE_API_URL`, so gen's
decide partner client (account deletion) used the production default and
sent `DELETE /api/partner/gen/users` to decide.nomadkaraoke.com.
Production answered 401, because the local secret differs, so nothing
was touched. It surfaced in nomad-mobile's E2E account-1 flow. The fix
defaults it to decide's local stack on :8030, and callers can still
override it. Script only, no runtime code.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Configuration**
* The local environment can now use a custom karaoke decision service
address, with a default address provided when none is configured.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 10:34:54 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 437b34ee64 fix(ci): wheel over PyPI's 100 MiB limit — drop unused RSC payloads + size guard (v0.287.2) (#1233)
## Summary
Every `Deploy - Publish to PyPI` since v0.279.4 (about 10-22) has failed
with `400 File too large. Limit for project 'karaoke-gen' is 100 MB`.
The wheel had grown to 110 MB (105.1 MiB), so PyPI is still on 0.279.4.
Backend deploys were not affected. I found this while watching deploys
during an alerts-triage session.

In the wheel, 66 MB of the bundled Next.js export is RSC `.txt` payloads
(about 3 MB per locale × 33). The only thing that uses the export is the
CLI local-review server
(`karaoke_gen/lyrics_transcriber/review/server.py`). It mounts only
`/_next` and `/static`, and every other route returns an HTML entry
point. Next's RSC fetch paths (`/{locale}/app/.../index.txt`) already
get HTML from those routes, so the `.txt` files are never served.

- `make build-frontend` deletes `*.txt` from the copied export. On the
real CI wheel from run 40826404322 this takes it from 105.1 MiB to 41.1
MiB.
- `scripts/verify_wheel_frontend.py`, which already runs in both
package-build jobs, now fails above 95 MiB. Growth will show up on the
PR instead of as a failed publish after merge.

## Testing
- New `tests/unit/test_verify_wheel_frontend.py`: 5 tests covering the
pass case, missing entrypoint, too few files, the size guard, and
headroom below 100 MiB.
- Re-zipped the real 0.286.0 CI wheel without the `.txt` files: 41.1
MiB, and the verifier passes. The verifier rejects the original wheel
(105.1 MiB) with the new error.
- CodeRabbit: 1 finding ("deleting RSC payloads can break client
navigation"). Skipped, because the local server never serves those files
(see above), so it can't change runtime behaviour.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVTiUPKxn3sQPyp3Teh

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 03:52:39 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 1715a07010 local stack: env overrides for parallel E2E lanes (defaults unchanged) (#1222)
Lets several local stacks run on one machine for nomad-mobile's parallel
Maestro lanes (the spare MacBook Pro runner): \`LOCAL_STACK_PROJECT\`
(per-lane Firestore project on the shared emulator, so global
fake-config does not leak between lanes) and \`LOCAL_STACK_STATE_DIR\`,
plus \`LOCAL_STACK_KEEP_EMULATORS=1\` for \`down\`. Every default is
unchanged, so existing local and CI runs behave exactly as before. Docs:
docs/TESTING.md § Local stack.

Context: nomadkaraoke/nomad-mobile#21 (E2E off Andrew's Mac).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Local development stacks can now use configurable state directories
and Firestore project IDs, making it easier to run isolated stacks side
by side.
* You can keep shared emulators running when stopping a stack, and
backend log locations reflect the selected state directory.
* **Documentation**
* Updated local setup guidance with recommendations for ports, projects,
state directories, and sharing emulators across stacks.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-26 21:55:52 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 d76a136ee6 feat: mobile Phase 6 — RevenueCat IAP webhook, Expo push, self-serve account deletion (flagged off), App Review demo login (v0.284.0) (#1220)
Phase 6 (mobile release readiness) — gen side. Spec: nomad-mobile
`docs/superpowers/specs/2026-10-25-mobile-phase6-release-readiness-design.md`
(§4 payments, §5 push, §6 deletion, §9 demo account).

## What
- **`/me` app fields:** opaque `app_user_id` (UUID, used for
RevenueCat/Sentry — no email to third parties),
`capabilities.account_deletion`, `push_prefs`.
- **In-app purchases:** `POST /api/users/webhooks/revenuecat` (shared
Authorization header). Exactly-once grants keyed by
`{store}:{transaction_id}` inside the same Firestore transaction as the
balance change; refunds (REFUND / CANCELLATION with CUSTOMER_SUPPORT or
negative price) debit the granted account, refund tombstones block late
grants; other cancellations ignored. `store_payments` for accounting +
revenue sums; referral earnings on **net** proceeds; sandbox grants but
excluded from revenue/referrals. Credit packages gain `store_product_id`
+ server-driven `web_link` (iOS US only for now).
- **Push:** `POST/DELETE /api/push/expo-token` (token moves between
users atomically), `PATCH /api/users/me/push-prefs` (ready / review /
problems, gates Expo **and** Web Push), Expo sender (batched,
DeviceNotRegistered cleanup), new FAILED / AWAITING_AUDIO_SELECTION
triggers.
- **Account deletion** (`ACCOUNT_DELETION_ENABLED`, **off in prod**
until Andrew's manual pass): `GET /api/users/me/deletion-preview`,
`DELETE /api/users/me`. Refuses admin / unpaid referral balance / Stripe
Connect / made-for-you in progress (409). Cancels active jobs, deletes
decide data (partner call, twice), RevenueCat subscriber, revokes Sign
in with Apple (when a code is given), queues kjbox
(`/api/kjbox/account-deletions` pull + ack), deletes **inputs only**
(uploads, input/, audio_edit/, style/, staged instrumentals) — **never
published outputs** — strips personal fields, anonymises
jobs/referrals/song requests/show-credit records, keeps payment records.
Every step idempotent; failures → 503 retry; sessions revoked last.
- **Web:** `/account/delete` page (Google Play web deletion path),
hidden while the flag is off.
- **App Review demo login:** fixed code for one configured email — inert
until `APP_REVIEW_DEMO_EMAIL` is set (not set in prod).
- **CI:** mounts `REVENUECAT_WEBHOOK_AUTH`, `REVENUECAT_SECRET_API_KEY`,
`APP_REVIEW_DEMO_CODE` (secrets exist; runtime SA already has accessor).

## Tests
6838 backend unit tests pass (emulator suite needs emulators;
iap/deletion emulator tests pass locally), frontend jest 1785/1785.
Per-task reviews + local CodeRabbit review (5 minor findings fixed).

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01CZf1cM9vefnNw5djARRhyA

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 17:31:01 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 53bed95e76 feat(local-stack): kjbox partner secret + fake quick version (v0.280.1) (#1206)
## Summary
Local-stack only (no prod impact). For mobile Phase 5 "Tonight": the
kjbox local stack points at gen's local stack for make-it.
- `scripts/local-stack.sh`: `KJBOX_PARTNER_SECRET` and
`DECIDE_GEN_PARTNER_SECRET` default to `local-stack-secret`, external
values win.
- Fake workers emit a quick version (`state_data.quick_version` ready +
`file_urls.quick.video_mp4`) for kjbox-sourced jobs, using prod's
`should_build_quick_version` gate and quick-version helpers.
`FAKE_WORKERS` is still refused in production.
- docs/TESTING.md § Local stack updated.

## Testing
`test_kjbox_partner_api.py`, `test_fake_workers.py`,
`test_kjbox_resolve_partner.py`, `test_fake_match_judge.py`: all pass.
Reviewed by a task reviewer (1 fix round).

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_013Bkasn6trBadg1Zb39vSVR

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 02:06:01 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 6c53f0471a feat(youtube): auto-reply to song requests in YouTube comments with requests-board links (v0.280.0) (#1205)
## Summary

Backlog Frog #6 (YouTube channel → gen funnel). Andrew's spec: *"we
could probably also solve that by building some automation to detect
youtube comments asking for songs and auto-reply with a link to the
requests board"*.

An hourly bot reads new top-level comments on the Nomad Karaoke channel
and, when someone asks for a song:
- **Song already on the channel:** replies with that video's link.
- **Otherwise:** adds the song to the requests board (or finds the
existing request) and replies with a link straight to it, e.g. `Added
"Tiny Hitman" by Bear Ghost to our requests board 🥳 upvote it here and
we'll email you when it's made: <link>`.
- **Vague asks** ("more Johnnie please") get a generic board link.
- **Non-English comments** get a translated reply, which falls back to
English if a URL doesn't survive translation.

The reply promotes only the board, not gen (Andrew's call). The goal is
turning anonymous YouTubers into board users with an email address.

**Andrew signed this off on a 6-month local replay**
(`scripts/youtube-comments/replay.py`, read-only):
- 265 threads, 33 song requests.
- It flagged all 29 requests Andrew had answered by hand, and all 7
vague asks.
- It posted no replies to non-requests.
- His calls on borderline cases became prompt rules: yes to Playa Fly,
no to "Joe Cuervo Karaoke 🎤" on an unrelated video.

## Changes

- **Decision core:** `backend/services/youtube_comment_bot.py`, a pure
module with injectable ports (Gemini classifier/translator, board,
channel index), shared by the replay and the worker.
- **Worker:** `backend/workers/youtube_comment_bot_worker.py`, called
through `POST /api/internal/youtube-comments/run` (`?dry_run=true`).
- Creates a create-only action doc per comment **before** posting, so
nothing gets a double reply.
- Keeps a high-water mark with a 6h overlap. A Gemini error causes a
retry, not a permanent skip.
  - Per-run cap and quota coordination.
  - Posts to #noise per reply; failures go to #alerts, throttled.
- **Kill switch:** `YOUTUBE_COMMENT_AUTOREPLY_ENABLED` defaults to
**false**, and this PR leaves it off. A follow-up PR switches it on
after a prod dry run. `YOUTUBE_COMMENT_AUTOREPLY_START` enables a
one-time backfill of unanswered requests.
- **Requests board:**
- New `source="youtube_comment"`; bot-added requests are owned by the
placeholder `youtube-comment@nomadkaraoke.com` with no votes.
- The **first real up-voter claims the request**, inside the vote
transaction. The daily picker skips unclaimed requests, so a free credit
never goes to the placeholder.
- Frontend `?request=<id>` deep link: highlight, scroll, banner (33
locales).
- Signing in from the link uses purpose `requests_board:<id>`, so the
magic link returns to the request with `&upvote=1` and the upvote is
cast once.
- **OAuth:** the bot uses its **own** Desktop OAuth client and secret
`youtube-comment-bot-credentials` (`youtube` + `youtube.force-ssl`),
because Google refuses `force-ssl` in the device flow. The upload token
is untouched.
- **Infra (Pulumi, already applied locally):** secret container + Cloud
Scheduler `youtube-comment-autoreply-hourly`.
- **Description template:** a make-your-own line moves above YouTube's
"...more" fold (`utm_campaign=make_cta_top`). The daily backfill
re-drains the channel over about 11 days.
- **Docs:** `docs/REQUESTS-BOARD.md` § YouTube comment auto-reply,
API.md, README.md, plus the plan in
`docs/archive/2026-10-22-youtube-comment-autoreply-plan.md`.

## Testing

- `backend/tests/test_youtube_comment_bot.py` (33): skips, routing,
vague asks, translation fallback, malformed classifier output, API
helpers.
- `backend/tests/test_youtube_comment_bot_worker.py` (12): kill switch,
dry run, idempotency (second run and concurrent claim), go-live start,
cap, quota, missing scope, post failure, classifier-error retry.
- `backend/tests/emulator/test_song_request_service.py`: bot add/dedupe,
claim on first upvote, downvote doesn't claim, picker skips unclaimed,
human resubmit claims, `find_published_by_key`.
- `backend/tests/test_admin_login_link.py`: `requests_board:<id>`
redirect and purpose helper.
- `frontend/e2e/regression/requests-board-deep-link.spec.ts` (3):
highlight + sign-in purpose, upvote-once on return + param dropped, no
toggle-off when already upvoted.
- Full `make test`: backend 6584 passed; the only failures are the
local-environment `test_emulator_integration` cases, which fail
identically on `origin/main` (port 8080 is held by another session's
emulator). Frontend Jest: 1760 passed.
- Behavioural acceptance: the 6-month replay above.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_016E3ibN2EcKxnGQvEf6T4V8

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 01:51:15 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 9d6aa3df7d feat(medley): per-section backing_vocals_db for the karaoke track (v0.278.6) (#1184)
## Summary
The Fiverr medley customer said the backing singers in two songs were
too loud to sing over in tune, and asked for them as a faint background.

- **New option:** a per-section `backing_vocals_db` in the medley spec.
For the karaoke track only, the section is rebuilt from the clean
instrumental plus the backing-vocal stem at that level, e.g. `-14` for a
faint background. The rest of the karaoke track, the clean instrumental
and the guide are unchanged.
- **Cached mix:** the mix is cached per job/level, written to a
`.partial.flac` and renamed, so an interrupted run never leaves a
truncated mix to be reused. If the job has no backing stem, it stops
with a clear error.
- **Docs:** `docs/MEDLEYS.md` spec table.

## Testing
- `test_make_medley.py`: parsing, the ffmpeg graph, only the karaoke
variant changing, the atomic write and the cache. 20 + chord-chart tests
pass.
- Real data: the customer's medley with `-14` on Remember When and On
the Other Hand.
- Only those two sections change; the others differ by ≤ -43 dB, which
is crossfade overlap.
  - Overall section loudness is unchanged.
- A local CodeRabbit review found 1 issue (stale or partial cache). It
is fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 15:04:43 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 03ee693f28 feat(medley): customer chord charts + static-gain master (v0.278.5) (#1183)
## Summary
Two follow-ups on the medley tooling, both from the Fiverr medley
customer's feedback.

- **Fixes the robotic artifact at the start of the karaoke audio.** The
final single-pass `loudnorm` ran in dynamic mode. On a song that opens a
cappella (HSLHT sings from 0.4s, band at 3s), the instrumental's first
3s is near-silent separation residue at -70 dB, and `loudnorm` boosted
it by about 50 dB into a loud warble. The master is now one static gain
to -14 LUFS plus a 4×-oversampled limiter at -1 dB (`master_filter`).
After the fix, the first 3s measure -61 dB (were -15 dB). Sync is
sample-identical and true peak is -0.9 dB.
- **Imports the customer's own chord chart.**
`scripts/medley/chord_chart.py` plus a new `make_medley.py chart` step
read a chords-over-lyrics chart (.txt or .docx in Courier, capo shapes,
`Capo N` / `Key change Capo 1` lines). They align it word-by-word to the
reviewed lyrics, so chords come out timed on the words the chart puts
them on. Detected chords fill intros, breaks and spoken verses,
transposed into the capo shapes.
- **Shows the capo in the chord box.** New `ChordChange.capo` field,
shown as "capo N" under the chord.
- **Docs:** `docs/MEDLEYS.md` gets a new section on importing a customer
chart, and the mastering guidance now says never to use single-pass
`loudnorm`. `docs/README.md` has an entry.

## Testing
- Unit tests:
  - New: `tests/unit/test_chord_chart.py` (9).
- Added: capo tests in `test_guitar_chords.py`, plus master and capo
tests in `test_make_medley.py`.
- Full `tests/unit` suite: 2100 passed, 65 skipped (run before the last
CodeRabbit fix; only the 9 chord-chart tests were re-run after it).
- Real data: the customer's 4-song chart against the medley. The
lyrics-with-chords output matched the chart line by line, and 4K frames
rendered with capo labels.
- A local CodeRabbit review found 1 issue (two chord groups after the
same word could come out in the wrong order). It is fixed and has a
test.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01QV2TQwyC7pwVkGPCk98Bqg

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 14:24:45 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 7cf5beede8 feat: guitar chords in karaoke videos + ChordMini ensemble chord detection (v0.277.0) (#1175)
## Summary
Adds optional guitar chords to karaoke videos, first used for a Fiverr
medley customer who wants chords on their video.

- **Rendering (gen library):**
`OutputGenerator.generate_outputs(chords=[ChordChange(start, end,
label)])`.
- Each chord change during a sung line is drawn above the word it lands
on, in a row reserved above every line (the same slot mechanism as
translated lyrics).
- Chords light up when played, using the same `\k` highlight as lyrics:
gold while upcoming (theme `chord_color`), sung-lyric colour once
played. Quick changes on one word never overlap.
- A top-right "now / next" chord box runs for the whole song, covering
intros and instrumental breaks.
- With no chords supplied, output is unchanged. A test asserts the ASS
is byte-identical.
- **Detection (`scripts/medley/detect_chords.py`):**
- Ensembles both ChordMini checkpoints (BTC + 2E1D, MIT; about 80%
majmin/MIREX, the best open weights we found) over the full mix, gen's
clean instrumental, and an htdemucs_6s harmonic mix (guitar + piano +
bass + other).
- Weighted per-frame vote, then merges short blips, snaps changes to
beats and spells chords for the key (Ab/Db/Eb7, F#m, borrowed
bIII/bVI/bVII as flats).
- **Medley script:** new `chords` step plus a `"chords": true` spec
option. Chords are clipped per section and hand over mid-crossfade, then
shifted for the countdown.
- **Docs:** `docs/MEDLEYS.md` § Guitar chords (setup, method, accuracy
caveats, theme keys) and a docs/README.md entry.

There's no job or UI option yet. The medley pipeline uses this for now;
a product version would need a chords toggle plus detection in a worker.

## Testing
- New: `test_guitar_chords.py` (12: chord timeline helpers, word
placement measured against the font, layout geometry, highlight timing,
now/next box, theme colour, no-chords path byte-identical),
`test_detect_chords.py` (7), and 2 medley chord-shifting tests.
- `tests/unit`: 2087 passed, 65 skipped.
- End to end on the 5 real medley songs: models/inputs agree on 91–99%
of major/minor labels, and the output matches the expected I–IV–V
progressions. Rendered 1080p and 4K medley videos with 146 chord
changes; spot-checked frames (verse, instrumental screen, chorus).
- Local CodeRabbit: 1 finding (doc section reference), fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01QV2TQwyC7pwVkGPCk98Bqg

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 00:24:11 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 85390fd12b docs: medley-making guide + spec-driven make_medley script (#1174)
## Summary
Documents the multi-song medley process used for the first
classic-country medley, and turns it into a reusable script so the next
one only needs a spec file.

- `docs/MEDLEYS.md`: the full process. One gen job per original song,
then identical cuts on every audio variant, and the reviewed lyrics
shifted onto the medley timeline and rendered with gen's own
`OutputGenerator`. Also covers how to choose cut points (crossfades,
pot-out/pot-in, keys/tempos, checking the brief against the recording)
and gotchas.
- `scripts/medley/make_medley.py`: `fetch` (Firestore + gcloud CLI) →
`audio` (per-section LUFS gain, crossfade/gap/pot fades, `loudnorm` to
-14 LUFS / -1 dBTP, three variants) → `video` (shifted reviewed lyrics +
countdown + themed title/end screens + karaoke and guide MP4s).
- `scripts/medley/example-spec.json`: the country medley as a worked
example.
- Linked from `CLAUDE.md` Quick Reference and `docs/README.md`.

Ops tooling only. No backend or frontend change, so no version bump.

## Testing
- `tests/unit/test_make_medley.py`: 12 unit tests covering spec
validation, the timeline offset maths, lyric-segment shifting (including
the same song used twice) and the ffmpeg filtergraph.
- End-to-end run against the real prod jobs: `fetch` → `audio`
reproduces the hand-built v1 medley **bit-for-bit** (max sample diff 0.0
on both the instrumental and guide WAVs). `video --resolution 360p`
renders both MP4s and the LRC.
- Local CodeRabbit review: 1 finding (unchecked ffmpeg exit code in
`measure_lufs`), fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01QV2TQwyC7pwVkGPCk98Bqg

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 19:44:27 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 c277bc8ee3 feat: local E2E stack (fake workers + test API) + email-code sign-in (v0.275.0) (#1165)
## Summary
- **Local stack:** `scripts/local-stack.sh up --frontend` runs the
**real** backend on the Firestore + GCS emulators with `FAKE_WORKERS=1`.
In-process fake workers move jobs through their real states in seconds,
using tiny real media, so web Playwright and mobile Maestro full-journey
tests can run offline against the same endpoints.
- **Email-code sign-in** (nomad-mobile spec §4.1): `POST
/api/users/auth/send-code` + `verify-code`. The code logic is shared
with kjbox via `LoginCodeStore`, and sends are throttled per email and
per IP.
- **Bug fix found by the new suite:** `confirm-duration` moved jobs to
`separating_stage1`, from which they can never reach review. It now
resumes at `downloading`. No prod job had taken this path yet.

## Changes
- `backend/services/fake_workers/`: dispatch seam in
`WorkerService.trigger_*`, plus fake search, download, checkout and
outbox, scenarios, and per-user config. The real stem upload, transcode,
backing-vocals analysis and preview render are reused. Every check is
strict (`is True`), so a MagicMock settings object never enables the
fakes, and startup **refuses FAKE_WORKERS in production**.
- `/api/test/*` router (mounted only in that mode): `status`,
`scenario`, `fake-config`, `emails`, `reset`, plus the fake Stripe
checkout page. Paying it runs the real webhook handler.
- `frontend/e2e/local-stack/` (9 specs, ~50s):
  - sign-up → make → review → instrumental → download
  - buy and cancel credits
  - duration confirm, failed, complete and review states
  - code sign-in and lockout
- New required CI job `local-stack-e2e`.
- Docs: TESTING.md § Local stack, DEVELOPMENT.md, API.md, README,
LESSONS-LEARNED.
- `MAX_SIGNUPS_PER_IP` can now be set by env. Only the local stack
raises it; production keeps the default of 2.

## Testing
- [x] Backend unit: 6175 passed
- [x] Emulator: 152 passed
- [x] Frontend Jest: 1706 passed
- [x] Local-stack Playwright: 9/9 passed (CI mode)

## Review
- [x] CodeRabbit CLI review completed locally. 1 minor finding declined:
it suggested swallowing infrastructure errors in send-code as fake
"sent" responses, which would hide real outages from users.

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01AGuxtgoscF6RJGtpjHPg3W

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 16:00:24 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 39dcf2598e fix: PyPI prune snippet + force marker only in PR title (v0.268.10) (#1146)
## Summary
Two follow-ups to #1145, both found while doing today's prune.

1. **Prune console snippet silently did nothing.** PyPI's delete-version
form now requires five `acknowledge_*` checkboxes. A submit without them
gets a **200 re-render**, which the old snippet counted as "deleted".
The snippet now reads each release's real form (CSRF plus checkboxes),
submits it, and counts a version deleted only when its manage page 404s.
This is the exact approach used for today's prune (88 releases, 10.73 →
~4.1 GB).
2. **The force marker matched the commit body.** A squash commit's body
is the PR description. #1145's description just *mentioned* the marker,
which forced a publish on merge. That publish ran before the prune, so
it 400'd and uploaded nothing. The marker now only counts in the commit
subject (the PR title).

Docs: the runbook covers the new form, the per-version JSON cache lag,
and a prune log.

## Testing
- Unit tests: 30 pass across both scripts, with new tests for
checkbox/404 handling and for a marker in the body not forcing a publish
- `node --check` on the generated snippet
- Replayed #1145's real merge message through the gate: no publish
- Local CodeRabbit review: 0 findings
- Full `make test` not re-run (no app code touched); PR CI covers the
package suites

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01MSW9yjwGnzz1S8MgCGm6t5

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 13:34:57 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 5005d3cb80 ci: publish to PyPI only when CLI code changed (v0.268.9) (#1145)
## Summary
PyPI publishing failed on every deploy since v0.268.4. karaoke-gen hit
PyPI's 10 GB project cap for the third time (June, Sept 29, Oct 12).
Each wheel is ~62 MiB and we published on every merge (~100/month), so
each prune only bought a couple of weeks.

PyPI is only the public `pip install` channel for the CLIs. Cloud Run
builds from source, and the encoding worker pulls the wheel from GCS. So
we now publish only when something CLI users get has changed.

## Changes
- `scripts/pypi_publish_gate.py` (stdlib): diffs
`scripts/pypi_publish_paths.txt` (`karaoke_gen/` plus the `backend/`
modules it imports) and `pyproject.toml` (ignoring the package `version
=` line only) against **the latest version on PyPI**. A skipped or
failed publish therefore never loses a CLI change. Fails open.
- `ci.yml` `package-publish`: new "Decide PyPI publish" step gates
`poetry publish` (now `--skip-existing`). **Tags and GitHub releases
still happen on every version**, because the encoding-worker gate diffs
against tags. Override with `[publish-pypi]` in the PR title/commit, or
the new workflow_dispatch input "Publish to PyPI".
- Not gated on purpose: `frontend/**` (bundled review UI) and
`backend/config.py` (cloud settings). A test keeps the backend import
list in sync, with `config.py` as the one documented exemption.
- Docs: PYPI-STORAGE-PRUNE.md, TROUBLESHOOTING.md, prune script and
reminder workflow comments.

## Impact
Replayed over v0.200 to v0.268: **24 of 136** releases would have
published (~0.7 GB/month instead of ~4 GB/month). Run against current
main, it decides "no CLI-affecting changes since v0.268.3", which is
correct for v0.268.4 to v0.268.8.

## Testing
- 15 new unit tests in `tests/unit/test_pypi_publish_gate.py`
- `make test` all green
- actionlint clean on the new step
- Local CodeRabbit review: 1 finding (doc recovery path), fixed

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01MSW9yjwGnzz1S8MgCGm6t5

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 12:56:24 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 0329d47e38 fix: YouTube description backfill reverted translated-lyrics notices (v0.268.9) (#1144)
## Summary
The daily YouTube description backfill worker was undoing the
translated-lyrics notice on videos with translated lyrics.

**Root cause:** `classify()` parsed `Artist - Title (With Translation
into Spanish) (Karaoke)` with the translation label left inside the song
title, and `render_for()` never passed a `translation_language_name`.
Every translated upload therefore looked "pending". The worker re-checks
the whole channel every day, even after a cycle completes, so it rewrote
each new translated video. The rewrite dropped the `This karaoke video
includes translated lyrics (…)` notice, put the label into `{title}`,
and added it to the tags as well.

## Changes
- `backend/services/youtube_backfill.py`: `classify()` strips the `(With
Translation into X)` label from the parsed song title and records
`translation_language_name`. `render_for()` passes it to
`render_youtube_description`, which is the same thing the upload
pipeline does. Tags now use the clean song title.
- On its next run the worker restores the notice on the translated
videos it already reverted, because their titles still carry the label.
- README: added a safety note that any per-video description variant has
to be recoverable from the live title.
- Version 0.268.9

## Testing
- 5 new tests in `backend/tests/test_youtube_backfill.py`: parsing,
untranslated case, matching upload left alone, reverted description
restored, title preserved + clean tags.
- `make test-backend-unit`: 5961 passed.
- Local CodeRabbit review: no findings.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_013RzGSdPUXLn1nK5eMEZZTW

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 12:32:23 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 3ecea6f2e7 feat(storage): enable real purge with deletion audit log and orphan cleanup mode (v0.266.0) (#1127)
## Summary
Andrew approved this follow-up to #1125 and #1126. It turns on real
deletion, adds an audit log of every deleted object, and adds an
orphan-folder cleanup mode.

### Real deletion on
`STORAGE_RETENTION_DRY_RUN=false` and
`STORAGE_RETENTION_MAX_JOBS_PER_RUN=50` are added to the backend
deploy's `--set-env-vars`. The API service is the only thing that reads
them: it passes `--dry-run` to `storage-retention-job`. That keeps the
setting in one place, so a later deploy can't silently revert it.

### Deletion audit log
- `storage-retention/deletion-logs/<ts>-<mode>.jsonl` has one line per
deleted or failed object: timestamp, run_id, mode, job_id, path,
size_bytes, generation, category and result.
- The log is rewritten after every job or folder, so a crash still
leaves a complete record. Each run also writes
`<ts>-<mode>.summary.json`.
- Deletes are pinned to the logged generation (`if_generation_match`).
- `assert_deletable` refuses any path outside `jobs/{id}/`, so
`storage-retention/` is never touched. No lifecycle rule matches live
objects there.
- The Firestore manifest on each job doc now also records generations
and the log path.

### Orphan cleanup
- `POST /api/internal/storage-retention/orphans` runs on the same Cloud
Run job with `--mode orphans`.
- It targets `jobs/{id}/` folders with no doc in `jobs`, `jobs-dev` or
`youtube_upload_queue`, re-checked live per folder against a fresh
listing.
- It skips any folder with an object modified in the last 30 days, keeps
`input/`, and deletes the rest.
- `encoding_worker_jobs` doesn't count as a record: it's the encoder's
expiring result cache, which deleted jobs leave behind.
- It is a dry-run unless called with
`dry_run=false&confirm=delete-orphan-job-folders`, and the scheduler
never runs it.

### Docs
- `docs/STORAGE-RETENTION.md`: run modes, log locations, jq review
commands and restore one-liners (noncurrent `cp #gen` and soft-delete
`restore`).
- `scripts/storage_retention_report.py`: list runs, summarise a run, or
find which run deleted a path or job.

## Testing
New tests cover the log contents, generation-pinned deletes, per-job
flushing across a crash, the path guard, orphan planning (input kept,
30-day skip, live record re-check, batch limit), and the endpoint and
worker modes. Full backend suite: 5774 passed.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added admin-triggered orphan-folder cleanup, with dry-run enabled by
default, optional folder limits, and explicit confirmation required for
deletion.
* Added deletion reports and audit logs covering removed objects,
reclaimed space, and failures, plus a read-only tool to review runs and
search records.
* **Bug Fixes**
* Added safeguards to prevent deleting changed objects or files outside
their intended job folders.
* **Documentation**
* Expanded storage-retention guidance with recovery details, review
commands, and cleanup instructions.
* **Chores**
* Set the production cleanup limit to 50 jobs per run and updated the
package version.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 11:54:05 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 4e43449559 fix(backend): stop API event-loop freezes + identify users in degradation telemetry (v0.264.0) (#1120)
## Summary
The "Reconnecting" pill and the "servers unavailable" banner (164 / 49
events, 2026-09-19 → 10-02) were mostly caused by the single API
instance's **event loop freezing for 20–45s**. Every request on the
instance, `/api/health` included, completed at the same moment. The full
evidence and plan are in
`docs/archive/2026-10-03-backend-loop-freezes-telemetry-plan.md`.

### What froze the loop, and the fix for each
- **Screens worker:** it ran as a post-response `async def`
BackgroundTask doing 4K PIL renders, GCS transfers and ffmpeg transcodes
on the loop. Under `--cpu-throttling` it was also CPU-starved. **Fix:**
it now runs inline in its Cloud Tasks request (CPU stays allocated,
failures still return 200 so Cloud Tasks doesn't double-run it), with
the heavy steps in `asyncio.to_thread`. Also offloaded: style load,
timing signals, pre-apply.
- **Matplotlib font cache:** it was rebuilt on every new instance (17/28
episodes). **Fix:** it's pre-built in `Dockerfile.base`
(`MPLCONFIGDIR=/opt/mplconfig`), and pyplot is imported lazily.
- **Hourly stale-review cron:** an `async def` with zero awaits doing
≤1500 reads and SMTP sends. **Fix:** it runs in a thread.
- **166 `async def` routes that never awaited** (sync
Firestore/GCS/email/Gemini on the loop, e.g. magic-link verify).
**Fix:** they're now plain `def`, so they run in FastAPI's threadpool.
The new AST guard `test_no_blocking_async_routes.py` fails CI on new
ones. It also flags sync routes that call
`asyncio.create_task`/`get_running_loop`; that caught `confirm_duration`
and `trigger_youtube_queue_processing`, which stay async.

### Safety net
- `backend/services/loop_watchdog.py`: a heartbeat plus a daemon thread.
Any stall over 1s logs `EVENT_LOOP_STALL` with the loop thread's stack
and a `culprit=file:line`. Stalls of 5s or more are written to
`client_events` as `server_loop_stall`.
- Pulumi: the log metric `backend/event_loop_stalls` plus an alert on ≥2
stalls of ≥10s in 30 min. **This still needs a local `pulumi up`.**
Pulumi is never applied by CI, and the admin gcloud login has expired.

### Telemetry gaps fixed
- **Identity:** `user_email` used to be hard-coded to null. The frontend
now sends its session token, and the server resolves `user_email`,
`is_admin`, `is_internal` and `is_test`. A client-sent email is ignored.
- **New fields:** `device_fingerprint` (the existing FingerprintJS
visitorId), `tab_id`, `episode_id`, and `tenant`.
- **New event:** `banner_recovered`, carrying `duration_ms` and
`peak_status`.
- **Report:** `scripts/client_events_report.py` summarises the data,
excluding admin/internal/test accounts by default.

## Testing
- `make test` passes: backend 5529, plus the emulator suite, karaoke_gen
unit tests, and frontend Jest 1623.
- New tests:
- Watchdog: a deterministic state machine plus detection of a real
blocking call.
- Event-loop responsiveness: a ticker keeps ticking while the heavy
steps run.
  - Screens inline and failure semantics.
  - `client_events` identity and fields.
  - The AST guard.
  - The report summary.
  - Frontend reporter and banner episode tests.
- Local CodeRabbit CLI review: 0 findings.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

@coderabbitai ignore

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 14:17:51 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 8490ab75eb chore(gemini): move every Gemini call from Vertex AI to the Gemini Developer API (#1108)
## Summary
Moves **every** Gemini call in gen off **Vertex AI** (billed to the
`nomadkaraoke` GCP project, ~$85/mo) onto the **Gemini Developer API**
using the `gemini-api-key` Secret Manager secret (an AI Studio key
billed outside this project). Once deployed and verified,
`aiplatform.googleapis.com` / `generativelanguage.googleapis.com` get
disabled in nomadkaraoke.

### Key injection
- `backend/services/gemini_client.get_genai_client(timeout_ms)` — key
from `GEMINI_API_KEY`, else Secret Manager `gemini-api-key` via
`settings.get_secret` (runtime SA already has project-level
`secretAccessor`). No `project`/`location`.
- Service: CI `--set-secrets ... GEMINI_API_KEY=gemini-api-key:latest`.
- Jobs: Pulumi secret env on `lyrics-transcription-job`,
`video-encoding-job` (renders + translations), `nomad-error-monitor` —
**already applied** (targeted `pulumi up`, only that env added). Other
jobs fall back to Secret Manager.
- Local scripts (`frontend/scripts/translate.py`,
`scripts/measure_autocorrect_cost.py`,
`scripts/test_agentic_correction.py`): env, else `gcloud secrets
versions access` (in-process only).
- **Rotation:** add a new secret version; new instances / job executions
pick it up. No code change.

### Degradation (quota / billing / invalid key; per-minute 429s stay
retryable)
| Caller | Behaviour |
|---|---|
| auto-correct | non-retryable 503 → compare mode drops the Gemini leg,
keeps Opus |
| match-judge / free-text / parse-titles / tenant bulk | existing
fallbacks (no suggestion / blanks / regex) |
| credit eval | `pending_review` (fail-closed, unchanged) |
| lyrics translation | no retry; render proceeds without translation,
status recorded |
| custom lyrics | clean 503 "temporarily unavailable…" (+ 5-min timeout
so it can't hang) |
| error-monitor LLM | no retry; alerts sent without AI summary |
| translate.py | exit 2 "Gemini quota/credit exhausted — top up AI
Studio or rotate gemini-api-key" |

`note_gemini_failure()` also sends one Discord ops alert per 6h
(Firestore-throttled across instances via new
`ops_alerts.send_throttled_ops_alert`).

### Infra
- Pulumi: removes `roles/aiplatform.user` from `karaoke-backend` —
**will be applied after this deploys and Vertex traffic is confirmed at
zero**.
- Agentic correction default model `gemini/gemini-3.8-flash`
(`vertexai/` prefix still accepted, routed to the Developer API).
- Version 0.261.1 → 0.262.0. Docs: CLAUDE.md, LESSONS-LEARNED, README,
ARCHITECTURE, API, etc.

## Testing
- New: `backend/tests/services/test_backend_gemini_client.py`
(classifier, key resolution, no-Vertex client kwargs, alert throttle,
per-caller degradation),
`tests/unit/lyrics_transcriber/test_agentic_gemini_provider.py`,
`frontend/scripts/test_gemini_client.py`, lyrics-translation quota test.
- `pytest backend/tests --ignore=backend/tests/emulator` — all pass;
`pytest tests/unit` — 2021 passed.
- Verified `gemini-3.8-flash` is listed on the Developer API and
answered a 5-token test call with the key.
- `translate.py --dry-run` on `frontend/messages` and
`backend/translations` OK.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Updates**
* AI-powered lyric translation, custom lyrics, auto-correction, title
parsing, match judging, and related features now use the Gemini
Developer API.
* AI features use the configured Gemini API key. Quota, billing, or key
issues may show an unavailable message or use an existing fallback
instead of repeated attempts.
* **Documentation**
* Updated product, API, setup, and recovery guidance to reflect Gemini
API key configuration and usage.
* Updated guidance for handling AI service availability and quota
issues.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 21:58:53 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 f8d0febcf5 Translated lyrics beneath the karaoke lyrics for language learners (v0.260.0) (#1101)
## Summary
New job-submission option: **Add translated lyrics** → pick a language
(33 UI locales) → the karaoke videos show an AI translation of each
lyric line in smaller text beneath the synced lyrics, so language
learners can sing songs in another language while learning what the
words mean.

- **Job creation**: toggle + language picker (default = UI language) on
the Customize step, with a **live preview** of a real rendered frame
(`GET /api/themes/translation-preview?language=xx`; pre-generated sample
translations, no LLM call at preview time). Any job, public or private,
free for now. `translation_language` accepted by every creation route;
admin can edit it.
- **Post-review translation**: right after the job enters
RENDERING_VIDEO, the render worker translates the *final reviewed*
lyrics line by line with Gemini
(`backend/services/lyrics_translation.py`, `LYRICS_TRANSLATION_MODEL`,
GCS-cached so theme/admin re-renders are free) into
`jobs/{id}/lyrics/translations.json`. Not editable. A failure never
blocks the render (renders without translations, recorded in
`state_data.lyrics_translation`); a song already in the target language
gets no row.
- **Rendering** (landscape + portrait, GCE worker + local fallback):
`LyricsSegment.translation`; when present, slots get taller and screens
show 3 lines; a `Karaoke.Translation` style (unsung colour, ~½ size)
draws one static event beneath each line, sanitized, wrapped/scaled to
fit. Jobs without translations render identically.
- **RTL + LTR together**: the translation is its own ASS event with auto
paragraph direction, so e.g. a Hebrew song keeps its right-to-left
highlight with an English translation beneath (and vice versa). Long
lines split by the resizer share their translation across the pieces
(grapheme-safe for Thai/CJK).

Design: `docs/archive/2026-10-01-translated-lyrics-plan.md`.

## Testing
- `make test` all green (root 2014, backend 5416, emulator 150, frontend
1583).
- New: `tests/unit/lyrics_transcriber/output/test_translated_lyrics.py`
(layout/positions/style/sanitizing/fit/mixed RTL-LTR/Thai/portrait),
`test_translated_lyrics_render.py` (libass pixel tests incl.
Hebrew/Arabic lyrics with English/Arabic translations and CJK),
`backend/tests/test_lyrics_translation.py`,
`backend/tests/test_render_video_worker_translations.py`, GCE endpoint
test, frontend CustomizeStep/GuidedJobFlow/api tests, prod E2E
`e2e/production/translated-lyrics.spec.ts`.
- `scripts/run-render-tests-linux.sh`: 135 passed on Debian ffmpeg AND
prod's static ffmpeg (no skips).
- Real-data check: job 5710831e (Hebrew) translated to English by Gemini
and rendered with the nomad theme — RTL highlight intact, English
beneath, split line shares translation.

Local CodeRabbit CLI unavailable in this workspace (no seat), so leaving
this PR for the CodeRabbit bot; an independent local review was done and
its findings fixed in the second commit.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added an opt-in translated-lyrics option to job setup, with language
selection and a preview.
* Translations are generated after lyric review and appear beneath
lyrics in landscape and portrait videos, with text layout for multiple
languages and writing directions.
  * Translation settings can be changed or cleared by admins.
* **Bug Fixes**
* If translation fails or is unavailable, video rendering continues
without translated lyrics.
* **Documentation**
* Updated API and product documentation with translated-lyrics setup and
behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 13:41:54 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 777daaf25f Missing-lyrics gate: block auto-approval when singing has no lyrics and the reference has lines there (v0.258.0) (#1097)
## Summary
Phase 2a of the missing-lyrics work (phase 1: #1095, shadow detector +
backfill).

**Audit** (262 completed jobs, 60 days, server-side
`scripts/audit_vocal_gaps.py`):
- Longest unlyricked vocal run per job: p50 0.7s, p90 5.1s, p99 22.4s.
47 jobs (18%) have a run of 3s or more, far too many to gate on audio
alone.
- Cross-checked against **synced** reference lyrics (LRCLIB line
timestamps): of 57 suspect gaps that have a synced reference, only 15
contain a reference line. Audio-only is ~25% precise. The rest are vocal
chops, samples, ad-libs and synth bleed, mostly in drum & bass tracks.
- **Evidence-based rule:** a gap is reliable when the reference lyrics
place lines in it.

**Changes**
- `vocal_gaps` 0.2.0: records the synced reference lines inside each
gap. `evidenced` = suspect vocal run AND reference lines, either synced
or between the neighbouring anchors.
- Scorer 0.5.0: new never-auto `missing-lyrics-gate` with a readable
reason, e.g. *"possible missing lyrics: ~10s of singing at 0:20-0:32 has
no transcribed words, and the reference lyrics have 3 line(s) there"*.
Audio-only gaps don't gate.
- The executor analyzes vocal gaps *before* scoring (off the event loop,
fingerprinted cache) and passes them to both scoring passes. It fails
open on pending audio or analysis errors.
- `GET /api/review/{id}/correction-data` now includes `vocal_gaps`, for
the upcoming review-UI marker (phase 2b).

On job 5710831e, the synced LRCLIB lines inside the gap are exactly the
three dropped lines. The genuine instrumentals have none.

## Testing
- `test_vocal_gaps.py`: synced lines with edge tolerance; section
headers ignored; evidence requires both singing and reference
(instrumental with misplaced reference lines doesn't gate; singing
without reference doesn't gate).
- `test_scorer.py`: reasons only for evidenced gaps; the gate overrides
confident lyrics; non-evidenced gaps don't gate.
- `test_executor.py`: evidenced gaps block auto-approval with the reason
and skip the timing check; cached evidenced gaps still gate.
- `test_combined_review_flow.py`: `vocal_gaps` is included in
correction-data.
- Full backend suite: 5211 passed.

Versioned 0.258.0 to follow #1096 (0.257.0).

No local CodeRabbit review (the CLI has no seat for this org), so this
is left open for the CodeRabbit bot.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 23:56:48 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 e2f7dbd13c Detect sung stretches with no transcribed lyrics (shadow) + backfill endpoint (v0.256.0) (#1095)
## Summary

Transcription can silently **drop whole sung lines**. On job 5710831e
(Hebrew), the words stop at 20.66s and resume at 32.40s while the lead
vocal is ~96% active, and both reference sources have three pre-chorus
lines there. `SectionDetector` (gap ≥ 10s, no audio check) then rendered
"♪ INSTRUMENTAL ♪" over the singing. Nothing caught it:
- the scorer's anchor coverage measures how much of the *transcribed*
text matches the reference, so omissions don't lower it (98.8% here);
- `timing_check` G3 (`max_unclaimed_run_s`) does fire on this job but is
shadow-only and only computed for AUTO-bound jobs;
- the review UI shows nothing at the gap.

This PR is **phase 1: a shadow signal plus server-side backfill for
calibration**. It changes no verdicts or rendering.

- `backend/services/auto_approval/vocal_gaps.py`: for every
transcription gap ≥ 3s, records the lead-vocal active fraction, the
longest breath-bridged vocal run (with a 1.5s allowance for a held note
carried over from the previous word), and the reference lines between
the neighbouring words' reference positions. Section headers are dropped
and ±1-word anchor drift at line boundaries is snapped. `suspect` = run
≥ 3s.
- The executor analyzes every job once stems exist (any verdict), off
the event loop, and stores `state_data.vocal_gaps` keyed by an
`input_key` fingerprint so resets/edits recompute. A summary goes in
`processing_metadata.auto_approval.vocal_gaps`.
- `POST /api/internal/jobs/{id}/vocal-gaps[?dry_run=true]` (admin) plus
`scripts/audit_vocal_gaps.py` for backfill/calibration. Cloud Run does
the stem work.

**On the real job:** one suspect gap, 20.66–32.40s (10.26s run after the
allowance), whose reference lines are exactly the three missing Genius
lines. The three genuine instrumentals have ≤ 0.16s runs.

## Testing
- `test_vocal_gaps.py` (25 tests, synthesized audio + synthetic
correction data):
  - dropped lines vs instrumental vs ad-lib vs held note
  - a short dropped line diluted by a long gap
  - breath bridging
  - silent stem = error
  - malformed reference data
- reference snapping (off-by-one, single-word line, headers, reversed
chorus mapping)
- Executor tests: stored without changing the outcome; analyzed for
review-bound jobs; cache hit only for the same `input_key`, stale
recompute; pending audio / no stem / error / exception all non-fatal;
backfill uses the reviewed lyrics + original references; dry run.
- Internal endpoint tests (store / dry_run / admin).
- Full backend suite: 5199 passed.
- A local code review's findings were addressed in the second commit:
event loop, stale cache, silent stem, held notes, reference robustness,
audit ordering.

## Next (after deploy)
Run the audit → hand-check suspects → choose thresholds → phase 2:
scorer REVIEW gate with an explicit reason, no fake INSTRUMENTAL over
vocals, review-UI marker with one-click insert of the reference lines.
Also noted: anchor reference ids look shifted by one word at `[Section]`
header boundaries. Separate issue, documented in the plan doc.

No local CodeRabbit review (the CLI has no seat for this org), so this
is left open for the CodeRabbit bot.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added vocal-gap analysis to identify stretches of missing lyrics that
overlap with lead vocals. Results are saved with job data and can be
requested through an admin-only internal API.
* Analysis also runs automatically during auto-approval when vocal stems
are available. It does not change job status or approval decisions.
* **Documentation**
* Documented the internal API, its dry-run option, and the analysis
result statuses.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 22:10:47 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 354e43bab2 Render Hebrew/Arabic karaoke right-to-left; fix title-card tofu (v0.255.0) (#1094)
## Summary

A Hebrew KaraokeHunt job (5710831e, Omer Adam – "שני משוגעים") shipped
with its lyrics **laid out and highlighted left-to-right** and a **title
card of "?" boxes**. Diagnosed by measuring rendered pixels (not by
eye), reproduced locally and with prod's static ffmpeg 7.0.2 in Docker.

**Root causes**
- ASS style `Encoding=0` makes libass force LTR paragraphs; the `{\kf}`
before each word splits the line into runs, so each word is shaped
correctly but the words go left→right.
- `\kf` always wipes left→right within a word
([libass#406](https://github.com/libass/libass/issues/406)).
- PIL (title/end cards) has no per-glyph font fallback; the theme font
(Avenir Next) has no Hebrew glyphs.

**Fixes**
- `build_karaoke_styles` forces `Encoding=-1` (libass auto paragraph
direction; LTR unchanged).
- RTL lines get `{\frz(180+angle)\frx180\fry180}` — a visual no-op that
makes libass ≥0.15 sweep fills right→left without splitting words
(Arabic still joins). Theme angle preserved; re-emitted after duet
`{\r}` resets.
- Lead-in indicator mirrored for RTL lines.
- Text measurement mirrors libass: per-glyph fontconfig fallback, size
scaled from each font's metrics (old fixed 0.70 was 20%+ off for
fallback fonts).
- New `karaoke_gen/utils/font_fallback.py` (fontTools cmap +
fc-match/fc-list) used by title/end cards, portrait headers and ASS
measurement.
- Found by the new tests: inter-word gaps ≤0.1s were dropped, so the
highlight ran up to ~0.2s early by line end (all languages). Karaoke
tags now derive from one absolute centisecond timeline.
- Fixed pre-existing `test_render_video_worker_gce` failure on main
(MagicMock generation → `_g<mock>` id).

## Testing
- **Pixel-level render tests** (`test_ass_render_highlight.py` +
`ass_render_harness.py`): real `SubtitlesGenerator` output rendered with
ffmpeg/libass in a pure-colour palette, asserting on frames for English,
Hebrew, Arabic, Chinese, Japanese, Korean + an RTL duet line: highlight
starts at the reading-start edge, sung pixels always precede unsung in
reading order (word order + in-word fill direction), monotonic and in
time with the words, lead-in on the correct side, no missing glyphs, RTL
tags don't move the line (incl. rotated themes).
- 32 fail on pre-fix main; mutation checks ("Encoding only", "no re-emit
after `\r`", "ignore theme angle") each fail exactly the expected cases.
- 123/123 pass on Linux against **both** Debian ffmpeg (CI-equivalent)
and **prod's johnvansickle static ffmpeg** via
`scripts/run-render-tests-linux.sh`.
- CI now installs `fonts-noto-core fonts-noto-cjk`; under `CI=true`
these tests fail rather than skip on missing preconditions. (macOS
libass uses CoreText → font-sensitive checks skip locally.)
- Title-card tests (`test_font_fallback_render.py`): no `.notdef` tofu
for Hebrew/Arabic/CJK titles; Hebrew title reads right-to-left.
- Unit tests: `test_rtl_ass_output.py`, `test_font_fallback.py`.
- `make test`: unit 1972 passed; emulator 150 passed; frontend 1491
passed.
- Full local re-render of job 5710831e reviewed by Andrew: RTL highlight
smooth and correct, title card correct.

## Follow-ups (not in this PR)
- CDG output still renders RTL left-to-right / tofu (separate renderer).
- Separate investigation: transcription dropped 3 sung lines in this
job, which the section detector then labelled INSTRUMENTAL.

No local CodeRabbit review (CLI has no seat for this org), so leaving
this open for the CodeRabbit bot. A local code review ran; its findings
are addressed in the second commit.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Karaoke lyrics now render more reliably for Hebrew, Arabic, Chinese,
Japanese, and Korean, with improved reading-direction highlighting,
timing, and lead-in placement.
* Titles and portrait text can use fallback fonts when the selected font
lacks the necessary characters.

* **Bug Fixes**
* Improved lyric centering, word highlighting, and glyph rendering
across supported scripts.

* **Documentation**
  * Added guidance on lyric rendering and multilingual test coverage.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 21:02:53 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 badd845a1d perf(jobs): move video-encoding-job (video + render) to us-east4 (v0.249.1) (#1083)
## Summary
Follow-up to #1077. `video-encoding-job` runs both the post-review
render (`render_video_worker`) and the final video (`video_worker`). In
us-central1 each run waited **2–5 min** for Cloud Run Jobs to start it,
so every customer job paid that twice. This moves it to us-east4
alongside the download, lyrics and bulk-search jobs.

## Cost (measured, not estimated)
- Over the last 30 days the job read **156 GB** from Google services,
mostly the us-central1 bucket, across 494 runs (Cloud Monitoring
`container/network/received_bytes_count`, kind=google). That's about
0.32 GB per run.
- Pricing from the Billing Catalog: GCS inter-region within North
America is **free for the first 100 GB/month, then $0.02/GB**. Cloud Run
writes to Google APIs are **$0**.
- Internet egress (224 GB/month, Dropbox etc.) and Cloud Run CPU cost
the same in either region.
- Existing cross-region reads are about 24 GB/month (GPU job and
lyrics), which the billing export shows at about $0.
- **Net: about $1.60/month**, or about $3/month worst case. Time saved:
roughly 4–8 min per customer job.

## Changes
- **backend**: `trigger_video_worker` (via `_trigger_cloud_run_job`) and
`trigger_render_video_worker` now use `cpu_jobs_region`. `GCP_REGION`
stays us-central1; encoding VM and zone logic are unchanged.
- **infra (Pulumi, already applied)**: new us-east4
`video-encoding-job`, with the legacy us-central1 copy kept for rollback
and its deletion protection turned off. A preview after applying showed
290 unchanged.
- **ci**: the video job image update now targets us-east4, using the
us-east4 CPU image copy.
- **Also updated**: rollback.sh, tests (video and render routing),
TROUBLESHOOTING/ARCHITECTURE/LESSONS-LEARNED.

## Testing
- `make test` passes: backend tests plus 1443 frontend tests.
- Prod smoke test of the new us-east4 job: it imports `video_worker` and
`render_video_worker` and calls the encoding VM's `/health` on its
public IP (port 8080 is open to 0.0.0.0/0 with API-key auth). **Started
in 25 s, exit 0.**

🤖 Generated with [Claude Code](https://claude.com/claude-code)


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Updates**
- Video encoding and render jobs now run in the `us-east4` region,
alongside other CPU jobs.
- Video encoding continues to access storage and encoding resources in
`us-central1`; cross-region transfer may incur costs.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 16:33:23 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 cb5ce7f162 chore(pypi): prune keep-window default 60→30 days (#1082)
Ran the PyPI prune runbook today (2026-09-29): publishing was blocked
(`Project size too large`). The 60-day policy would only have freed 1.08
GB (→9.64 GB), refilling within ~a week, so pruned with `--keep-days 30`
instead (69 releases deleted, 0.188.5–0.207.0; newest release of each
older month retained) → ~6.8 GB. Re-ran the failed publish; 0.249.0 is
live on PyPI.

This PR makes 30 days the default in the planner script, the monthly
reminder workflow, and the runbook. Unit tests pass (13).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 16:17:43 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 3e855ff65d perf(jobs): run download/lyrics/bulk-search Cloud Run Jobs in us-east4 — cut 2-5 min job start latency (v0.247.1) (#1077)
## Summary
Every gen job waited **2–5 min** before `audio-download-job` (and then
`lyrics-transcription-job`) even started. That blocked kjbox's goal of
singing within about 4 minutes of making a song.

**Root cause: Cloud Run Jobs in us-central1 are slow to start on
Google's side.** Our image, CPU and code aren't the cause.
- The `Started` condition fires about 10 s before the first Python log,
so our process startup is fine.
- decide's 203 MB image on 4 CPU is just as slow in us-central1.
- A throwaway job running Google's tiny sample image, 4 runs per region:
**us-central1 4m–5m07s**, **us-east4 8–19 s**, **us-east1 8–25 s**. All
four us-central1 runs started in the same second, as if released from a
queue. Quotas are far from their limits.

## Changes
- **backend**: new `CPU_JOBS_REGION` setting (default `us-east4`) for
triggering `audio-download-job`, `lyrics-transcription-job` and
`bulk-search-job`. `GCP_REGION` is unchanged. `video-encoding-job` stays
in us-central1, because it pulls multi-GB outputs from the us-central1
bucket.
- **infra (Pulumi, already applied)**: the three jobs are created in
us-east4 as new resources. The legacy us-central1 copies are kept as the
rollback path, with deletion protection turned off so a follow-up can
remove them. The preview after applying showed 289 unchanged, so there's
no drift.
- **ci**: the CPU app image is also pushed to
`us-east4-docker.pkg.dev/nomadkaraoke/karaoke-backend-gpu/karaoke-backend-cpu`,
so jobs pull from their own region. The lyrics job image update now
targets us-east4. After each deploy, CI starts a no-op
`audio-download-job` execution so the one-time ~2.5 min image import
doesn't land on the first customer.
- `scripts/rollback.sh`: lyrics job moves to the us-east4 image, and
jobs whose target tag is missing are skipped instead of aborting the
loop.
- Docs: TROUBLESHOOTING runbook (measure + rollback), ARCHITECTURE,
LESSONS-LEARNED.

## Testing
- `make test`: all backend tests pass, plus 1422 frontend tests.
- New tests in `backend/tests/test_worker_service_cpu_jobs_region.py`:
routing of each moved job, video job stays in `GCP_REGION`, env override
for rollback, and the Pulumi/backend region constants must match.
- Prod smoke test of the new us-east4 `audio-download-job` with the real
image (`import backend.workers.audio_download_worker`): **Started in 7.0
s, 8.6 s, 10.0 s** (first run of a new digest adds a one-time ~2.5 min
import, which the CI step covers).
- Local review via a reviewer agent: no must-fix findings; the
rollback-path suggestions are addressed.

## Rollout
One-time image seed (Cloud Build gcrane copy) and `pulumi up` are done.
After merge, CI deploys the backend, which starts triggering the
us-east4 jobs.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Performance**
* Audio downloads, bulk searches, and lyric transcriptions now run in a
region with faster job startup times, reducing delays before processing
begins. Video encoding continues to run in its existing region.
* **Reliability**
* Rollbacks now skip jobs when the target image is unavailable and
continue with the remaining jobs. Other image lookup errors still stop
the rollback.
* **Documentation**
* Updated architecture and troubleshooting guides with regional job
details, startup-delay guidance, and rollback instructions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 01:02:19 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 e9465c73d4 chore: gemini-3.8-flash everywhere + auto-correct on Opus 5.5 (cost) (#1054)
## Why
GCP credits expired 2026-09-19; Vertex Gemini 3.1 Pro spend had grown $9
→ $37 → $105/30d. `gemini-3.8-flash` is $0.75/$3.75 per 1M (vs $2/$12
for 3.1 Pro; rises to $1.50/$7.50 on 2027-01-01). Opus 5.5 is $4/$20 (vs
$5/$25 for 4.8).

## Changes
- All Gemini defaults → `gemini-3.8-flash`: credit eval, custom lyrics,
auto-correct (default + compare leg), match judge, parse titles, tenant
bulk analyze, agentic AI router, error-monitor LLM analysis (also
`us-central1` → `global`, required for Gemini 3.x), frontend
`translate.py`.
- Auto-correct Claude leg `claude-opus-4-8` → `claude-opus-5-5` (ci.yml
env), with `effort: "high"` pinned (5.5 defaults to medium). Pricing
table gains both new models.
- Docs updated; version 0.241.1.

## Evidence (12 recent prod jobs, pre-review lyrics scored against the
human-reviewed final)
| Model | Precision | Word-level recall | $/job |
|---|---|---|---|
| gemini-3.1-pro-preview | 91% | 74% | $0.067 |
| gemini-3.8-flash | 98% | 38–41% | $0.020 |
| gemini-3.8-flash @ high thinking | 97% | 60% (4/12 timed out, 504) |
$0.041 |
| claude-opus-4-8 | 94% | 33% | $0.080 |
| claude-opus-5-5 (effort high) | 93% | 58% | $0.048 |

Flash for the Gemini leg is a deliberate cost/recall trade-off (Andrew
chose it); Opus 5.5 more than compensates. Also verified:
`gemini-3.8-flash` works on Vertex `global` (rejects
`thinking_level=minimal`; nothing here uses it), match-judge latency
median 3.3s / max 6.2s (< 12s timeout), Japanese translation smoke test
preserves ICU placeholders. Translation cache is keyed on English text +
locale, so no re-translation churn.

Tests: auto_correct, credit eval, custom lyrics, parse_titles,
error_monitor, agentic router suites pass locally.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* AI-powered correction, translation, matching, title parsing, and
evaluation now use updated models, including Gemini 3.8 Flash and Claude
Opus 5.5.
* **Updates**
* AI service pricing reflects scheduled rate changes, including a Gemini
3.8 Flash rate change beginning January 1, 2027.
  * Application version updated to 0.241.1.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-26 14:53:04 -04:00
Andrew Beveridge ☄️andClaude Fable 5 30cf41e7d6 feat: concurrent-reliability phases 2-5 — Cloud Run shape, degradation telemetry, banner tuning, load-test guard (#1028)
## Summary

Phases 2–5 of the concurrent-load reliability initiative (Phase 1 =
#1024, verified live: the 10-tab meltdown went from 9/30 failed requests
to 30/30 OK). Verbatim spec:
`docs/archive/2026-09-17-concurrent-review-reliability-verbatim-record.md`;
plan:
`docs/archive/2026-09-18-concurrent-review-reliability-investigation-and-plan.md`.

## P2 — Cloud Run shape
- **`--concurrency 80 → 16`** for karaoke-backend: the service is one
uvicorn process per 2-vCPU instance, and review loads are heavy; at 80 a
burst never triggers scale-out and all lands on the single min-instance.
16 makes bursts fan out while staying well above normal per-instance
load. (Deploy flags live in ci.yml — the service is not Pulumi-managed.)
- **Local preview ffmpeg fallback bounded**
(`LOCAL_PREVIEW_MAX_CONCURRENCY=1`, 900s queue timeout, fast
RuntimeError after that): the GCE-outage fallback runs ffmpeg ON the API
instance; unbounded, a few concurrent Preview clicks during an encoder
outage pegged both vCPUs, stalled the health probe, and showed everyone
the orange banner (Andrew's banner-during-preview screenshot).

## P3 — degradation telemetry (the explicit ask)
- New **`POST /api/client-events`** (unauthenticated so review-token
users report too; 30/min/IP rate limit; bot-filtered; detail capped;
URLs query-stripped). Each event → Firestore `client_events` collection
**and** a structured `client_event type=…` log line, so "how often are
users seeing this?" is reviewable via Firestore aggregation or Cloud
Logging correlation.
- Frontend reporter `lib/degradation-events.ts` (60s per-type throttle,
localhost no-op, never throws), wired into:
- `BackendStatusBanner` → `banner_reconnecting` / `banner_unavailable`
with stall age, in-flight count, probe state
- review lyrics-load failure → `lyrics_load_failed` with message/status
- `VocalsAudioDataLoader` → `waveform_slow` (>20s to first strips) and
`waveform_failed` (gave up) with poll/retry counts

## P4 — banner tuning
- Escalating to the full "We're having trouble reaching our servers"
card now requires **two consecutive failed health probes**
(`UNAVAILABLE_PROBE_FAILURES=2`). One failed 4s probe against a
briefly-pegged instance shows only the calm "Reconnecting…" pill.
Single-failure behavior for the pill is unchanged, as is instant
clearing on recovery.

## P5 — regression guard
- **`scripts/load-test-review.py`** checked in: replays the 10-tab
incident (Firestore-ADC review tokens, curl request trios, pass/fail
thresholds, exit code). Documented in `docs/TESTING.md`. Validated
against prod post-Phase-1: 9/9 requests OK, tabs < 3s.

## Testing
- Backend: 6 new tests for client-events
(persist/validate/rate-limit/bot/fail-soft/detail-cap) + local-preview
bound test — full suite **4517 passed**.
- Frontend: new degradation-events suite, two-probe escalation +
debug-snapshot tests; one existing api-fetch test updated to the new
escalation contract — full suite **1298 passed**.
- Live validation of the load-test tool against prod (small 3-tab run):
PASS.

## Notes for review
- Cloud Run concurrency change means bursts now spin up extra instances
(billable during requests only; max-instances unchanged at 20).
- Firestore `client_events` has no TTL policy yet (needs a
gcloud/console action our read-only ADC can't do); volume is tiny
(throttled per client) — can add later if it ever matters.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Added automatic reporting for degraded experiences, including
reconnecting or unavailable backend status, failed lyrics loads, and
slow or failed waveform loads.
- Added concurrent review-page load testing to help verify performance
under multiple open tabs.

- **Improvements**
- Backend status now remains in “reconnecting” until multiple health
checks fail, reducing premature unavailable messages.
- Local preview processing is limited during heavy demand to improve
stability.

- **Documentation**
- Added API documentation for client telemetry and guidance for running
concurrent-load regression tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-19 12:48:21 -04:00
Andrew Beveridge ☄️andClaude Opus 4.8 912ca37a01 Lyrics review: Waveforms mode + always-show backing vocals in preview (#1010)
## Summary

Two review-UI improvements, both driven from real reviewing pain:

### 1. New "Waveforms" mode in Synced Lyrics
A third view (alongside Simple/Advanced) that renders a **compact inline
copy of the Edit Segment timeline for every segment** — so you can spot
mis-timed words (e.g. an over-long trailing word whose vocal energy died
earlier) at a glance without opening a modal per line.

- Reuses the modal's `TimelineEditor`: resizable/draggable word bars
over each segment's vocal waveform, with the 1s buffer padding,
start/end boundary lines, and greyed read-only neighbour words.
- **Drag** a bar body moves timing; **drag** an edge resizes; both
commit on release (a 4px threshold distinguishes drag from click). A
plain **click** on a word — or the index/✎ — opens the Edit modal.
**Trash** deletes the segment (matches Advanced).
- Bars colour-code with the **exact `HIGHLIGHT_CLASSES` tints** the
Advanced pills use (anchor/gap/corrected/AI/edited) and show the
struck-through **AI-correction ghost text** above corrected bars.
- Tight vertical density (~39px/segment: 20px bars + 14px waveform + 5px
gap) to stay close to Advanced.
- View mode persists to `localStorage` (`lyricsReviewViewMode`,
migrating the old `lyricsReviewAdvancedMode` boolean).
- Advanced + Waveforms now size the **Reference column to its longest
line** so Synced gets the freed width (no more ~25% wasted space).

### 2. Preview modal always shows the backing-vocals waveform
Previously the backing-vocals waveform only appeared *after* switching
away from the recommended "Clean instrumental", making it easy to accept
clean without ever seeing/hearing the backing vocals you'd lose. Now
it's **always shown** whenever a playable backing stem exists, with
selection-aware wording ("kept" vs "removed — click to hear what you'd
be leaving out"). Clicking still auditions the backing track without
changing the final choice.

### Also
- Fixes a pre-existing `Header` `FORMATTING_ERROR` (`autoCorrectedDesc`
rendered without its `count` param on jobs with ≥1 auto-correction).
- `scripts/list-review-jobs.py` — lists in-review jobs with
ready-to-open local review URLs.

## Testing
- New unit tests: `contextWords`, `segmentTiming` (recompute + view-mode
migration), `wordDecorations` (classifier + palette),
`WaveformSegmentRow` (control wiring + untimed fallback).
- Updated `ReviewChangesModal` tests for the always-show behavior +
reworded copy.
- Full frontend Jest suite green (104 suites / 1283 tests).
- i18n: all 33 locales translated; stale key pruned; parity validated.
- Verified live against a real in-review prod job (Playwright):
coloring, ghost text, click-to-open, compact layout, dynamic columns.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **New Features**
- Added a Waveforms view to Synced Lyrics for inline word-timing
adjustments.
- Added controls to edit segments, play timed segments, delete segments,
and handle untimed lyrics.
- Waveform bars now visually distinguish corrections, edits, gaps, and
related word states.
- Backing-vocal waveforms remain available for review even when the
clean instrumental is selected.

- **Improvements**
- Auto-correction messaging now displays the number of applied
corrections.
- Clarified backing-vocal and clean-instrumental guidance across
supported languages.
  - Updated the application version to 0.227.0.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-09-17 02:19:14 -04:00
Andrew Beveridge ☄️andClaude Opus 4.8 462f246768 internal: admin endpoint to re-run auto-approval eval on parked review jobs (#1005)
## Summary
Adds `POST /api/internal/jobs/{job_id}/auto-approval-eval` (admin-only):
re-runs the auto-approval evaluator on a job parked at AWAITING_REVIEW.
The KaraokeHunt outreach batch runs with `made_for_you: true`
(stale-review exemption), which is also an enforcement blocker at
scoring time — this endpoint lets us lift the flag and re-evaluate so
confident jobs full-auto through review.

## Testing
- `test_internal_api.py::TestAutoApprovalEval` — outcome passthrough +
auth required; file green in full-collection run.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Administrators can manually re-run auto-approval evaluation for jobs
awaiting review.
- Jobs remain in review unless the evaluation is fully confident,
providing a fail-safe review process.

- **Bug Fixes**
- Improved batch job processing so an issue updating an optional job
setting no longer stops the remaining workflow.
- Prevented missing cached results from causing failures during
recanonicalization tasks.

- **Tests**
- Added coverage for authorized evaluation requests and unauthorized
access.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-09-15 09:22:39 +00:00
Andrew Beveridge ☄️andClaude Opus 4.8 6800c14e57 admin: make made_for_you editable on jobs (stale-review exemption) + KaraokeHunt outreach tooling (#1004)
## Summary
- Add `made_for_you` to `EDITABLE_JOB_FIELDS` (with a boolean guard) so
admin batch flows can exempt jobs from the 48h stale-review auto-expiry
(`stale_review_processor` skips MFY jobs). Needed for the KaraokeHunt
outreach batch: ~90 jobs held at review under a holding account until
coordinated emails go out.
- KaraokeHunt outreach tooling under `scripts/karaokehunt_outreach/`
(phase1b/1c analysis, phase2 executor, batch generator, torrent review)
— standalone scripts, no runtime impact.
- Session handoff docs in `docs/archive/`.

## Testing
- `backend/tests/test_admin_job_update.py`: new
`TestUpdateJobMadeForYou` (editable + non-bool rejected) — 20/20 pass.
- Full backend suite: 4417 passed (unit) + emulator suite green via
`make test-e2e`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Administrators can mark jobs as “made for you,” keeping them available
for review indefinitely instead of automatically expiring after 48
hours.
- Invalid values for this setting are rejected with a clear validation
error.

- **Documentation**
- Added operational documentation for the KaraokeHunt outreach and
migration workflow, including request review, song matching, account
credits, and generation stages.

- **Chores**
  - Updated the application version to 0.225.4.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-09-15 06:12:06 +00:00
Andrew Beveridge ☄️andClaude Opus 4.8 5b5264dc54 Incident hardening: deploy/CI/runner resilience (R1/R2/R3/R4/C1/C2) (#987)
Part 3 of 3 of the **Incident hardening (post-NOMAD-1632)** initiative
(plan: `docs/archive/2026-09-12-incident-hardening-plan.md`). Closes the
deploy/CI/runner gaps: no functional gate, no codified rollback, bad
revision serves 100% immediately, fire-and-forget runner stranding, and
a flaky smoke. **No backend app code changed → no version bump.**

## ⚠️ Reviewer note — read first
**R3/R4 rewrites the live `deploy-backend` step.** It's designed to be
*auto-revert-safe* (a failed candidate smoke means we simply never
migrate traffic, so the prior revision keeps serving 100% — a bad
revision serves **0%**, not an outage). But it changes how **every**
backend deploy works and **can't be fully exercised in CI**. This is the
one to watch on first merge. If you'd rather de-risk, R3/R4 is the only
item here that's safe to split out — the rest (R1/R2/C1/C2) are
independent and lower-risk.

## R2 — codified fast rollback (`scripts/rollback.sh` + runbook)
Dry-run-able, explicit-target-required rollback for the Cloud Run
**service** (traffic-shift to a prior revision — instant, no rebuild)
and the three **jobs** (image re-pin, correct per-job region/repo —
verified against ci.yml). `--list` shows available revisions/tags;
`--dry-run` prints every gcloud command. `docs/TROUBLESHOOTING.md` "Fast
rollback" runbook added as the first section.

## R3/R4 — service traffic-canary
`deploy-backend` now deploys `--no-traffic --tag candidate`, smokes the
tagged candidate URL (health + expected version, with retries), then
`update-traffic --to-latest` **only on pass**. Cloud Run Jobs can't
traffic-split (noted inline) → they rely on R1 + R2.

## R1 — post-deploy functional canary, version-change-gated
New `deploy-verify` job runs the real production happy-path
(`happy-path-real-user.spec.ts`, admin-credit variant — reuses
e2e-daily's impersonate token refresh, secret names verified identical)
after a backend deploy, **only when the backend version actually
changed** (new "Detect backend version change" step compares `poetry
version` to the currently-served `/api/health/detailed`).
Frontend/docs/infra-only merges short-circuit. On failure → Discord
alert + loud fail printing the exact `scripts/rollback.sh` invocation.
Per the plan's caveat, a fresh happy-path won't reproduce every incident
class (e.g. Failure B's stale-cache) — an admin reset+retry variant is a
noted follow-up.

## C1 — ephemeral-runner stranding fix
`infrastructure/functions/runner_manager/ephemeral.py` now confirms the
VM insert **operation** succeeded instead of fire-and-forget: a late
stockout/quota/transient error raises → existing zone-fallback / webhook
redelivery kicks in instead of silently stranding a queued job for ~15
min. An insert-confirm *timeout* is treated as "still provisioning" (not
a failure); guarded for older clients. **+4 tests (46 pass).**

## C2 — de-flake the E2E smoke
`networkidle` → `domcontentloaded` + explicit element waits (the
marketing page's background requests made `networkidle` time out and
redden real deploys).

## Validation (local only — no deploy)
- `ci.yml` + `e2e-daily.yml` parse; **actionlint clean** on new code.
- `rollback.sh`: `bash -n` + `shellcheck` clean; constants cross-checked
against ci.yml.
- `test_ephemeral.py`: **46 passed**.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-09-12 16:58:12 -04:00
Andrew Beveridge ☄️andClaude Opus 4.8 a344783376 gen: unify YouTube descriptions + automated back-catalogue rewrite (#978)
## Summary

Unifies the descriptions across all ~1.6k videos on
youtube.com/@nomadkaraoke to one consistent, conversion-focused
template, makes new uploads use it automatically, and adds a
**reusable** automated system to re-apply any future template change
across the whole channel.

Old descriptions were inconsistent: pre-2024 ones linked a dead $10
Fiverr gig; newer ones led with "AI-powered" (bad rep among viewers) and
linked nowhere useful.

## New template
Single source of truth: `Settings.default_youtube_description` rendered
by `backend/services/youtube_description.py` (placeholders `{title}
{artist} {artist_hashtag} {brand_code}`). Drops all "AI" framing; leads
with the craft; CTA to the requests board then the referral link
`nomadkaraoke.com/r/youtube` (50% off, tracked via referrals); Nomad
Discord + decide + KaraokeNerds; fair-use line; preserves the brand
code.

## Live pipeline (new uploads)
Both upload sites (`video_worker_orchestrator.py`,
`youtube_queue_processor.py`) now render via the shared module +
enriched tags — new uploads use the new template automatically.

## Bulk rewrite of the back catalogue
- Shared logic `backend/services/youtube_backfill.py` (classify / target
/ render / update-body / enumeration). **High-confidence-only**
eligibility (title ends in `(Karaoke)`) + curated `skip_ids.txt` /
`include_ids.txt` lists shipped in the package.
- **Manual CLI** `scripts/youtube-descriptions/` (`analyze` / `apply` /
`status`) — read-only report + full old→new diffs before any write.
- **Automated daily drain**
`backend/workers/youtube_description_backfill_worker.py` + `POST
/api/internal/youtube-backfill/run` + Cloud Scheduler (Pulumi). Progress
in Firestore keyed by a **template fingerprint**, so any future template
change auto-re-drains the whole channel with no manual reset.
Quota-coordinated with uploads (reserves headroom via
`YouTubeQuotaService`); Postmark progress + completion emails.

## Verified
- Analyzed all 1,631 videos → **1,614 to rewrite** (1,610 clean + 4
rescued truncated titles), **17 skipped** (lyrics-with-vocals, demos,
tutorials), zero false positives.
- One video (`PmwRXIZoCpg`) updated live and verified on YouTube (new
description + enriched tags; title/category preserved).

## Tests
42 new unit tests (renderer, classifier/targeting/update-body, worker:
full drain, budget cap, quota reserve, dry-run, completion/idempotency,
template-change cycle, quotaExceeded stop). Local review done via the
local code-review fallback (CodeRabbit CLI was rate-limited) — 3
findings fixed, 2 documented trade-offs.

## Deploy notes
- Backend auto-deploys on merge (worker + endpoint + config + data
files).
- **`pulumi up` required** to create the
`youtube-description-backfill-daily` Cloud Scheduler job.
- New config env (all defaulted): `YOUTUBE_BACKFILL_ENABLED`,
`..._DAILY_MAX_UPDATES`, `..._QUOTA_RESERVE`, `..._ENRICH_TAGS`,
`..._REPORT_EMAIL`.

Plan/architecture:
`docs/archive/2026-09-03-youtube-descriptions-plan.md`

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- YouTube uploads now use standardized descriptions with artist, title,
brand-code, and hashtag details.
- Tags are generated consistently from karaoke, artist, and title
information.
- Existing channel videos can be analyzed and updated in quota-limited
batches, with dry-run support and progress reporting.
- Added scheduled daily backfills and administrative controls for update
limits, exclusions, and overrides.
- Added safeguards for low-confidence titles and explicitly skipped
videos.

- **Documentation**
- Added guidance for automated and manual YouTube description management
workflows.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-09-03 02:20:19 -04:00
Andrew Beveridge ☄️andClaude Opus 4.8 37b9e6088b ci(pypi): recurring storage-cap prune reminder (fix failing PyPI publish) (#960)
## Problem

The `Deploy - Publish to PyPI` step on `main` has been failing:

```
HTTP Error 400: Project size too large. Limit for project 'karaoke-gen'
total size is 10 GB.
```

This is a recurrence of the known 10 GB PyPI project cap. The project is
at
**10.71 GB / 479 releases**; each release is a ~62 MiB universal wheel
(bundled
33-locale frontend) auto-published on every merge (~3 GB/month), so it
refills
the cap every few months. **Only the public `pip install` publish step
fails** —
Cloud Run and the GCE encoding worker (pulls the wheel from GCS) are
unaffected.

The prior fix (v0.187.5) was wheel-only publishing + a one-time manual
deletion,
which only bought ~2 months. This makes the cleanup **recurring and
durable**.

## Why not just automate deletion in CI?

PyPI has **no deletion API** (upload tokens / Trusted Publishing are
upload-only,
and 2FA is mandatory). Deletion is only possible from a logged-in
browser
session, which can't be safely/durably stored in CI. So we automate the
*decision + reminder* and keep the *deletion* a ~1-minute human step.

## What this adds

- **`scripts/prune_pypi_releases.py`** (stdlib-only) — reads the public
JSON API
and computes a retention plan: **keep last 60 days + newest release per
older
calendar month + latest overall**. Renders a table / JSON / Markdown
report and
a **paste-into-browser-console** deletion snippet. Never deletes
anything
  itself. Today's plan: keep 82, delete 397, **10.71 GB → 4.17 GB**.
- **`.github/workflows/pypi-prune-reminder.yml`** — monthly cron (+
manual
dispatch) that runs the planner and, once the project crosses a 7 GB
watermark,
opens/updates a single tracking issue with the plan + snippet. No
secrets.
- **`docs/PYPI-STORAGE-PRUNE.md`** runbook + a `TROUBLESHOOTING.md`
entry.
- **`tests/unit/test_prune_pypi_releases.py`** — 13 tests covering the
policy
(recency window, per-month keeper, latest-overall, same-day timestamp
ranking,
  empty/undated releases, sizes, rendering).

## Testing

- `pytest tests/unit/test_prune_pypi_releases.py` → 13 passed.
- Ran the planner against live PyPI data; generated JS validated with
`node --check`.
- `black` clean; workflow YAML validated; `should_nag` gate verified.

## Notes

- **No version bump** — tooling/CI/docs only, nothing in the wheel
changes.
Merging therefore **skips** the publish step (tag `v0.209.1` already
exists),
so this PR does not itself add another fat wheel while we're over the
cap.
- **To unblock publishing now:** after merge, run the prune (locally via
the
script or trigger the workflow with `workflow_dispatch`), paste the
snippet on
the logged-in PyPI releases page, then re-run the publish. The monthly
cron
  keeps it under the cap going forward.
- Related: filed
[pypi/support#12050](https://github.com/pypi/support/issues/12050)
requesting a 50 GB limit as a one-off buffer; the recurring prune is the
  durable fix.

@coderabbitai ignore

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-29 21:46:10 -04:00
Andrew Beveridge ☄️andClaude Opus 4.8 8ce0d39faa fix(lyrics-review): Tap To Sync timing corruption + padded edit-segment timeline (#956)
## Summary

Fixes **Tap To Sync timing corruption** in the Edit Segment modal and
reworks the segment timeline for easier re-timing of the first/last
words.

### Tap To Sync fixes (`useManualSync`)
- **Negative durations on re-sync** — the overlap-correction effect
pulled a just-tapped word's end back to the *next* word's **stale**
start (it hadn't been re-tapped yet), producing `end_time < start_time`.
The sanitizer then reported *"Fixed N word timing(s) that fell outside
this segment"* on reopen. Now guarded + clamped so it never inverts.
- **First word squished / segment start ratcheting** — `clampSyncTime`
forced every tap into the tight `[segStart, segEnd]`, so a legit tap in
the 3s playback lead-in got shoved up to `segStart` and the start could
never be re-synced earlier. Now allows taps in `[segStart − leadIn,
segEnd + leadOut]` (floored at 0).
- **Last word cut off** — symmetric lead-out + auto-stop no longer fires
while the spacebar is held, so the last word can extend the segment
(hold past the end / default-duration overshoot) instead of being
truncated.
- Both keyboard **and touch** sync paths use the widened window.

### Edit Segment timeline UX
- ±1s **greyed context padding** on each side (dashed boundary), **faded
waveform** outside the segment, and **neighbouring segments' words** as
read-only context blocks. Word bars, drag, ruler, cursor and
click-to-scrub all map over the padded view; words can be dragged/synced
into the padding to extend the segment.
- Modal widened `960px → 95vw / max 1400px`.
- Header **play button** starts 1s before and auto-stops 1s after the
segment.

### Dev tooling (dev-gated, no prod impact)
- `&edit=1` makes a replay session editable locally (guarded by
`NODE_ENV !== 'production'`) so the real review UI can be exercised
against real job data + audio. Documented in `docs/REPLAY.md`.
- `review_test_fixtures.py` now serves a synthetic **click-track** so
Tap To Sync is exercisable offline.

## Testing
- New/expanded unit tests in `hooks/__tests__/useManualSync*.ts(x)`:
negative-duration regression (fails on old code, passes on fix),
lead-in/lead-out clamp cases, and segment-end extension. All 13 pass;
178 lyrics-review + hook tests green.
- Typecheck clean on touched files.
- Verified end-to-end in the browser against a real completed job
(replay + real vocals): tap-sync produces valid monotonic timings, no
sanitizer banner, first/last words extend correctly, timeline padding +
fade + neighbour words render as intended.

## Notes
- CodeRabbit reviewed locally; its one finding (touch path missing the
lead window) is fixed.

@coderabbitai ignore

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-29 17:17:10 -04:00
Andrew Beveridge ☄️andClaude Fable 5 9b6c4f236e fix: Cloud Tasks queue region from GPU workers + REVIEW_COMPLETE lost-trigger recovery (v0.202.1) (#948)
## Summary

Follow-up to #947, found by live-testing auto-approval in prod (job
`b8bda9c2` — the first real fully-auto-approved job).

**Bug:** the GPU `audio-separation-job` sets `GCP_REGION` to its GPU
region, but every Cloud Tasks queue lives in `us-central1`. Any queue
enqueue from inside that worker fails with `404 Queue does not exist` —
which silently dropped the auto-approval render trigger (job
auto-approved perfectly, then sat at `review_complete` forever). This is
also a latent pre-existing bug for any other enqueue attempted from the
GPU job.

**Fixes:**
1. New `settings.cloud_tasks_region` (default `us-central1`, overridable
via `CLOUD_TASKS_REGION`) used for all Cloud Tasks queue paths.
`gcp_region` is still used for Cloud Run Job locations (correct).
2. `recover-stuck-jobs` cron gains a sweep for `REVIEW_COMPLETE` jobs
stalled >10 min with no `render_progress` — re-triggers the render
worker (idempotent via worker-generation fence). This closes the
lost-trigger orphan gap for BOTH the human complete-review
fire-and-forget task and the auto-approval path, and will auto-rescue
`b8bda9c2` on the first tick after deploy.

## Testing
- New unit tests: queue path uses `us-central1` even when
`GCP_REGION=us-east4`; stall detector (stalled / fresh /
render-underway).
- Note: `test_internal_api.py` auth tests fail identically on pristine
`main` in this local env (green in CI) — pre-existing, unrelated.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-27 05:07:17 -04:00
Andrew Beveridge ☄️andClaude Fable 5 f01213f5cb Fully-auto review for confident jobs: review_mode framework + auto-approval scorer/executor + replay tool (v0.202.0) (#947)
## Summary

Ships the **fully-automated review framework**: jobs now default to
`review_mode="auto"` — when the auto-approval scorer is fully confident
on BOTH axes, the combined review screens are skipped entirely and the
job renders straight through. Everything else falls back to human review
exactly as before.

**The confident class (deliberately narrow to start, ~5% of jobs
expected):**
- **Lyrics**: synced reference present, ≥90-99% anchor coverage, and
every gap word covered by the pre-generated AI auto-correct suggestions
(which the executor applies server-side — a faithful port of the review
UI's on-load auto-apply, including conflict-group resolution) — OR zero
gaps outright.
- **Backing vocals**: non-subjective only — no audible backing content →
clean instrumental.
- **Never-auto gates**: vocalization sections (da-da-dun/woo-woo — P3),
phantom/absurd-duration lines (P8), too-few-words, no/unsynced
reference, errored backing analysis.

**Calibrated against a 20-job human-review corpus** (all recent admin
reviews replayed + annotated): safety 100% (every job that truly needed
a human stays in review), lyrics-auto 3/20, overall-auto 1/20.

## How it works

- `screens_worker` (pre-AWAITING_REVIEW) and `audio_worker` (post-stems,
for the audio-lags-lyrics ordering) both call
`maybe_auto_complete_review()`. Every job gets a verdict recorded in
`processing_metadata.auto_approval` (shadow data keeps accumulating for
calibration); enforcement additionally requires: flag on +
`review_mode="auto"` + not made-for-you + no existing/custom
instrumental + audio complete.
- Enforce path mirrors a human "Complete Review": apply cached AI
suggestions → sanity checks → `corrections_updated.json` →
`instrumental_selection="clean"` → clear progress keys →
`GENERATING_SCREENS→REVIEW_COMPLETE` (new legal edge; no review token or
"ready for review" notification is ever issued) → trigger render worker.
- **Fail-safe everywhere**: stale suggestions, duplicate-word artifacts
(P1 self-conflict signature), state changed mid-enforce (human opened
editor), any exception → normal human review. Auto-approval can never
fail a job.
- Kill switch: `AUTO_APPROVAL_ENFORCE_ENABLED=false` (scoring
continues). Per-job: `review_mode="always_review"` (admin PATCH,
validated).
- Excluded from enforcement (still shadow-scored): made-for-you orders
(human QC is part of that product), custom-instrumental jobs.

## Also in this PR (sessions 1-3 of the initiative)

- **Review replay tool** (admin-only, prod-inert): `?replay=1` reopens
the real lyrics/instrumental review UIs read-only for completed jobs —
queue nav, Post-AI/Final toggle, edit-log action panel, dev-only audio
byte-proxy (env-gated `REVIEW_AUDIO_PROXY`, path-anchored). This is what
produced the calibration corpus.
- `auto_approval` service package: scorer (anchor/gap/AI-coverage
signals + P3/P8 gate detectors), apply engine, executor, lyrics-diff,
offline capture script.

## Testing

- 100+ new unit tests across scorer/apply/executor/replay routes; full
backend suite 4000+ green locally (8 pre-existing emulator failures
reproduce identically on pristine `main` — local-env only, green in CI).
Frontend: 1196 Jest tests green.
- Scorer validated read-only against the 20 real prod jobs in the
private calibration corpus.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-27 03:07:33 -04:00
Andrew Beveridge ☄️andClaude Fable 5 70ea964ca9 feat(dev): full local backend for external contributors (no GCP creds needed) (#946)
## Summary

Motivated by external contributor Martín (netux) asking "is there an
easy way to run the full backend without GCS?" — he couldn't test PR
#895's backend changes locally. This PR makes the whole backend runnable
locally against emulators with **zero GCP credentials and zero paid API
keys**.

### What was broken for contributors
- Workers migrated to Cloud Run Jobs for prod robustness
(audio-download, audio-separation, lyrics-transcription, bulk-search)
silently bypassed the direct-HTTP dev mode → local jobs stalled at
PENDING forever
- Signed URL generation called `google.auth.default()` → crashed with no
credentials (review audio, downloads, theme previews, upload URLs)
- Magic-link login 503'd locally (console email provider counted as "not
configured")
- Fresh emulator had no default theme → every job create returned 422
- `docs/DEVELOPMENT.md` told people to `cp .env.example .env` but the
file didn't exist — and nothing loaded `.env` anyway
- `run-backend-local.sh` defaulted to the production GCP project

### Changes
- **worker_service**: CRJ-based workers run as local subprocesses
(`python -m backend.workers.X --job-id Y`, exact CRJ parity) when
`ENABLE_CLOUD_TASKS=false` **and** not `is_production()`
(regression-tested both ways)
- **storage_service**: signed URLs → plain fake-gcs-server URLs when
`STORAGE_EMULATOR_HOST` set (GET via JSON API download path; PUT via
unvalidated-signature path, enabled by `-public-host` on the emulator)
- **users route**: magic-link 503 only in production; dev prints the
link to the console
- **file_upload bugfix**: a theme with style params but no asset files
never got `style_params_gcs_path` written (empty-dict falsiness in `elif
theme_style_assets:`) — surfaced by the minimal seed theme, latent in
prod
- **scripts/seed-local-data.py** (new): creates bucket + default theme;
wired into `run-backend-local.sh --with-emulators`, which now uses
`test-project`
- **config**: `load_dotenv()` so `.env` actually works; new root
`.env.example`
- **CONTRIBUTING.md** (new): no-credentials quickstart; README +
DEVELOPMENT.md link to it

### Verified end-to-end locally (no GCP credentials)
Upload job → local audio separation (real models) → transcription →
screens → `awaiting_review`; review endpoints all working including
`/api/review/{job}/audio/vocals` (the PR #895 endpoint) and 302 →
emulator audio URLs. Magic-link login round-trip via console link.

### Tests
- 224 tests across affected files pass; full backend unit suite: 3969
passed
- `backend/tests/test_emulator_integration.py` has 8 pre-existing
failures that reproduce identically on unmodified `main` locally (broken
theme-service mock in that file) — unrelated to this PR
- New tests: dev-mode subprocess dispatch (audio + bulk-search),
prod-mode never-Popen guard, emulator URL generation (GET + PUT)

@coderabbitai ignore

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-27 06:48:00 +00:00
Andrew Beveridge ☄️ 7729d30937 build: build frontend into the wheel in CI instead of committing out/ to git (supersedes #892) (#926)
Build the Next.js frontend into the wheel in CI instead of committing the generated out/ to git (fixes #876; supersedes #892). Validated: package-build-test green (npm ci → make build-frontend → poetry build → verify_wheel_frontend guard → install). Thanks @netux for surfacing it.
2026-08-23 01:19:11 -04:00
Andrew Beveridge ☄️andClaude Opus 4.8 6f86b3d901 test(e2e): rewrite tenant daily E2E as a faithful frontend journey (#873)
## Why

The daily tenant E2E (`scripts/e2e/tenant_e2e.py`) drove the whole flow
via **direct backend API calls**, bypassing the frontend. Two problems:
1. **Not representative** — it never exercised the tenant portal UI a
real operator uses.
2. **Intermittent stuck-render orphans** (Run #31 vocalstar, #32 singa)
— because it never loaded the lyrics-review page, it never fired
`encoding-worker/warmup`, so the encoder was cold when the render
started and the job orphaned in `rendering_video` (no recovery).
Root-caused this session; the fix is to make the test faithful, which
fires the warmup a real user fires.

## What

Replaces `tenant_e2e.py` with
**`frontend/e2e/production/tenant-happy-path.spec.ts`** — a Playwright
journey mirroring `happy-path-real-user.spec.ts`, run per-tenant via the
existing `e2e-tenant-daily.yml` matrix (vocalstar, singa; `max-parallel:
1`):

login (real magic-link UI) → submit track (tenant form + real audio
upload) → wait for transcription → **open review UI (fires warmup)** →
approve → wait for render → assert tenant distribution (Dropbox present,
no YouTube, downloads, `is_private`) → cleanup.

- **Tenant auth**: added `inbox.testmail.app` to both tenants'
`allowed_email_domains` (setup scripts + live GCS config via
`scripts/e2e/update_tenant_allowlist.py`) so the test signs in through
the real magic-link UI.
- **Test audio**: short 60s clips fetched from private GCS in CI
(`scripts/e2e/fetch_test_audio.sh`) — fast, reliable uploads and quick
renders.
- Workflow reworked to Node/Playwright; `tenant_e2e.py` deleted; version
`0.190.3 → 0.190.4`.

## Findings surfaced (validating the exercise)

- **Browser upload was CORS-blocked in Playwright** by the shared
config's global `x-playwright-test` header (it made the GCS signed-URL
PUT preflight request a header the bucket's CORS doesn't allow).
Verified the same PUT succeeds in a real browser — a **test-harness
artifact, not a product bug**. Worked around by stripping the header for
`storage.googleapis.com` requests. (Note: any prod E2E doing a browser
signed-URL upload is affected by this global header.)

## Verification

Both tenants run **green end-to-end against prod** locally (login →
submit → review → render → distribution; render completed without
orphaning). Reworked workflow dispatched on this branch to confirm CI
wiring (Node setup + private-audio fetch + secrets).

Test-only + CI + tenant-config change. No product code.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

@coderabbitai ignore

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 02:33:20 -04:00
Andrew Beveridge ☄️andClaude Opus 4.8 21e8a05c61 test(tenant): retry daily E2E once on transient encoder failure (#834)
@coderabbitai ignore

The shared GCE encoder intermittently drops a job mid-encode ("lost
contact with worker ... not found") — a pre-existing,
**non-tenant-specific** transient (it hit the daily E2E twice but not
the manual runs; both tenants complete when the encoder cooperates). The
daily health check now detects that specific error and retries the whole
job once, so it doesn't false-alarm on encoder flakiness. Other failures
still fail immediately. Test-script only.

Separately worth a look (out of scope here): the encoder "lost contact /
job not found" reliability affects consumer renders too.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 10:28:58 -04:00
Andrew Beveridge ☄️andClaude Opus 4.8 1218337346 test(tenant): daily E2E asserts portal downloads via file_urls (#831)
@coderabbitai ignore

The daily tenant E2E asserted on `job.download_urls` (backed by
`output_files`, which the orchestrator pipeline never populates), so it
failed even on fully-successful jobs. The `OutputLinks` UI reads
`job.file_urls` (finals/packages) — assert that instead.

Confirmed in prod: **both vocalstar and singa pass the full E2E** — job
completes, Dropbox link present, no YouTube, and all downloads available
(4K / 720p / with-vocals MP4 + CDG + TXT). Doc updated with the final
outcome.

Test/docs only — no production code change.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 09:44:40 -04:00
Andrew Beveridge ☄️andClaude Opus 4.8 66dcfd7d19 fix(tenant): stage instrumental in the orchestrator encode path (#829)
@coderabbitai ignore

Follow-up to #828. That PR staged the instrumental in `_encode_via_gce`
(legacy `video_worker.py`), but production jobs run the **orchestrator**
pipeline (`video_worker_orchestrator.py`), whose `_run_encoding` builds
its own `EncodingInput` and never made the user-supplied instrumental
available under `jobs/{job_id}/`. Tenant jobs still failed at the GCE
encoder with "No instrumental audio found" (confirmed — the "Staged
existing instrumental" log never appeared).

**Fix:** in the orchestrator's `_run_encoding`, when using the GCE
backend, upload the local instrumental (existing/custom — identified by
the `(Instrumental User)` / `(Instrumental Custom)` filename) into
`jobs/{job_id}/` so the encoder's recursive `find_file` locates it.
Consumer `clean`/`backing` jobs are untouched (their stems already live
under `jobs/{job_id}/`).

Also: the daily E2E uses a tenant-specific title so the two matrix
tenants don't share a `base_name` (the GCE encoder caches by base_name).

Validated by re-running the full prod E2E to completion after deploy.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 07:44:17 -04:00
Andrew Beveridge ☄️andClaude Opus 4.8 40c7960854 fix(tenant): stage instrumental for the GCE encoder + daily tenant E2E (#828)
@coderabbitai ignore

## Final render blocker (bug #8)

With #824/#825/#826/#827, tenant jobs now render the karaoke video — but
the GCE encoder failed with **"No instrumental audio found"**.

**Root cause:** the user-supplied instrumental lives under
`uploads/{job}/audio/`, but the encoder only pulls input from
`jobs/{job}/` (`input_gcs_path`) and locates the instrumental via
`find_file("*existing_instrumental*", "*Instrumental User*")`. On top of
that, the running encoder VM (`encoding-worker-b`) boots a **2026-01-23
image** that predates the encoder's own `existing_instrumental`
config-download path (added 2026-03-04), so it could never fetch it from
`uploads/` either.

**Fix (image-agnostic — no risky shared-VM recreation, no consumer
impact):** `_encode_via_gce` server-side copies the instrumental into
`jobs/{job_id}/existing_instrumental.<ext>` before submitting. The
encoder downloads `jobs/{job}/` and its **recursive** `find_file` then
locates it on any image version. Added `StorageService.copy_blob` + unit
test.

> Note: the stale encoder VM image is a separate infra item — the
`encoding-worker` VMs should be recreated from the image family's newest
image (which has 3 months of encoder fixes). Tracked for a Pulumi pass;
not required for this fix.

## Daily tenant E2E (second ask)
- `scripts/e2e/tenant_e2e.py` — full per-tenant prod health check:
submit → approve review → render → assert (completes, Dropbox link
present, no YouTube, downloads available) → clean up job + uploads.
- `.github/workflows/e2e-tenant-daily.yml` — daily matrix (vocalstar,
singa), WIF auth, Discord + email on failure.
- `docs/archive/2026-06-15-tenant-portals-e2e.md` — the full story (8
bugs, all fixed).

## Testing
- `copy_blob` unit test; storage suite green (20).
- Full tenant E2E re-run to completion after deploy.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 06:49:46 -04:00
Andrew Beveridge ☄️andClaude Opus 4.8 d6bd2f1787 fix(tenant): complete the render for jobs that supply their own instrumental (#827)
@coderabbitai ignore

## Two final render blockers (found by the Vocal Star E2E)

With attribution (#824), the audio slot (#825), and transcription
padding (#826) fixed, the first real tenant job advanced through
transcription → screens → review → render, surfacing the last two
never-exercised issues:

### 1. Render prerequisites — "No instrumental selected"
existing-instrumental jobs (every tenant submission) have no
instrumental-selection UI step, so `instrumental_selection` was never
written to state_data. The video orchestrator then defaults to `'clean'`
(a separated stem that was never produced for these jobs), and
`_validate_prerequisites` rejects the job, leaving it stuck retrying.

**Fix:** `complete-review` now defaults `instrumental_selection` to
`'custom'` (use the provided instrumental) when
`existing_instrumental_gcs_path` is set. An explicit request value still
wins. The render path already supports `'custom'`.

### 2. Theme asset paths (reproducibility)
The tenant setup scripts wrote full `gs://bucket/themes/.../assets/X`
URLs, but `theme_service._build_style_assets_mapping` only resolves bare
basenames (the consumer convention). With `gs://` URLs, `style_assets`
came back empty → assets never downloaded → encoder failed with "Video
background image not found". Scripts now emit basenames, and add the
newly-required `extra_text_text_transform` field the strict theme
validator expects.

(The two live GCS themes — vocalstar, singa — were corrected out-of-band
to match.)

## Testing
- New regression tests: complete-review defaults to `custom` for
existing-instrumental jobs; explicit selection still wins.
- Review suites green (102). Full E2E re-run to completion after deploy.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 05:34:20 -04:00
Andrew Beveridge ☄️andClaude Opus 4.8 fea6ebec1a fix(tenant): make white-label B2B portals attribute jobs to the tenant (Vocal Star + Singa) (#824)
@coderabbitai ignore

## Problem

The white-label tenant portals (`vocalstar.nomadkaraoke.com`,
`singa.nomadkaraoke.com`) looked "done" but **had never worked
end-to-end**: 0 of the last 484 production jobs carried a `tenant_id`.
No tenant job had ever run.

**Root cause:** the frontend API client never sent the `X-Tenant-ID`
header. Tenant portals call the shared `api.nomadkaraoke.com`, whose
Host isn't the tenant subdomain, and the `?tenant=` query param is
disabled in prod — so the backend could never detect the tenant. Every
tenant sign-in and job submission silently fell back to consumer
behavior (default theme, no tenant Dropbox, etc.). The
`getTenantHeaders()` helper existed and was unit-tested, but was never
wired into requests.

## Changes

**Frontend** (`lib/api.ts`, `lib/tenant.ts`)
- Attach `X-Tenant-ID` on all authenticated requests **and** the
magic-link request, sourced from `window.__TENANT_CONFIG__`
(edge-injected by the Cloudflare function; mirrored from the tenant
store for dev/preview). Consumer build resolves no tenant → no header →
**behavior unchanged**.

**Backend** (`_apply_tenant_overrides` in `file_upload.py`)
- Make tenant config **authoritative** over global distribution
defaults. Prod sets `DEFAULT_DROPBOX_PATH` / `DEFAULT_GDRIVE_FOLDER_ID`
/ `DEFAULT_BRAND_PREFIX` globally; these seed `dist`, and the old "fill
only when unset" overlay let them leak — so every tenant job would have
landed in the **shared consumer Dropbox folder** and uploaded to the
**global Google Drive**. Now Dropbox/GDrive routing is gated by the
tenant feature flags and cleared when disabled.

**Tenant configs** (GCS + setup scripts, Vocal Star + Singa)
- Disable Google Drive (`gdrive_upload=false`, `gdrive_folder_id=null`),
keep YouTube off, enable Dropbox to each tenant's own folder. B2B =
locked theme, no YouTube/GDrive, Dropbox + portal download only.

## Why it's consumer-safe
- Frontend: header only attached when a tenant is resolved (never on
`gen.nomadkaraoke.com`).
- Backend: `_apply_tenant_overrides` early-returns when `tenant_config
is None` (all consumer jobs).

## Testing
- New: frontend header-attachment tests (tenant build vs consumer);
backend authoritative-override + feature-flag-gating regression guards
(incl. the global-default-leak case).
- Full suites green: **3466 backend**, **1013 frontend**.
- Verified live tenant configs via API after update.

Manual prod E2E (real track through each portal) + a daily per-tenant
E2E workflow follow in a second PR, once this deploys and is verified in
prod.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 02:40:08 -04:00
Andrew Beveridge ☄️andClaude Opus 4.8 39fb93653f fix(auto-correct): switch active model from Fable 5 to Opus 4.8 (#822)
## Summary

Claude **Fable 5** (`claude-fable-5`) is no longer available on the
Anthropic API. Verified live against `api.anthropic.com`:

```
404 not_found_error: "Claude Fable 5 is not available. Please use Opus 4.8.
Learn more: https://www.anthropic.com/news/fable-mythos-access"
```

`claude-opus-4-8` responds normally and is the API's own prescribed
replacement. Opus 4.8 was also ≈ Gemini 3.1 Pro in our 30-job offline
eval, so switching to it preserves the two-model compare quality.

## Changes

- **`.github/workflows/ci.yml`** (load-bearing):
`AUTO_CORRECT_MODEL=claude-opus-4-8` and
`AUTO_CORRECT_COMPARE_MODELS=claude-opus-4-8;gemini-3.1-pro-preview`.
Compare mode now runs **Opus 4.8 + Gemini 3.1 Pro** instead of Fable +
Gemini.
- **`scripts/measure_autocorrect_cost.py`**: default models + docstring
example → Opus, so the cost tool measures what prod runs.
- **`docs/API.md`**: example auto-correct response reflects Opus.
- **`pyproject.toml`**: `0.182.1` → `0.182.2`.

## Re-enable path

Fable is **disabled, not deleted** — its pricing entry in `pricing.py`
and all Fable unit tests are intentionally left intact. If access
returns, flip it back by reverting the two env-var values in `ci.yml`.

## Testing

- Auto-correct suite: 73 passed (`backend/tests/services/auto_correct/`
+ `test_proactive_auto_correct.py`).
- Live API probe confirmed Fable 404s and Opus 4.8 works.
- CodeRabbit (uncommitted): 0 findings.

@coderabbitai ignore

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-13 10:27:58 -04:00
Andrew Beveridge ☄️andClaude Opus 4.8 364e4ebd5f feat(auto-correct): proactive multi-model suggestions + cost logging + de-AI-ified review UI (#816)
## Summary

Makes AI lyrics auto-correction run **automatically on every job**
(multi-model), so suggestions are cached and ready the moment the
reviewer opens the lyrics UI — plus token/cost instrumentation and a UI
cleanup that removes all "AI"/"model" wording.

Measured cost first (`scripts/measure_autocorrect_cost.py`):
**~$0.13/track** multi-model on normal songs — cheap enough to run
proactively.

## Changes

**Cost instrumentation**
- Capture per-model `usage` from Anthropic + Gemini; log greppable
`auto-correct usage` / `auto-correct cost … total_est_cost_usd=…` lines
from `suggest()`.
- `backend/services/auto_correct/pricing.py` (Anthropic rates
authoritative; Gemini estimated) + `scripts/measure_autocorrect_cost.py`
offline harness.

**Proactive generation (every job)**
- `lyrics_worker._run_proactive_auto_correct()` runs the multi-model
path after `mark_lyrics_complete` (downstream already triggered →
non-blocking), gated by `AUTO_CORRECT_PROACTIVE_ENABLED`, bounded by a
180s timeout, and **swallows every error** — it never blocks or fails
the karaoke job (flaky AI / out of credits / timeout all degrade
gracefully).
- Cache-key aligned with the UI request, so the review page's on-load
call is an instant cache hit (no double spend).
- `ci.yml`: wires `AUTO_CORRECT_*` env + the `anthropic-api-key` secret
onto `lyrics-transcription-job` (same SA as the API service — no IAM
change).

**Review UI**
- Always multi-model (removed the model-selection toggle); button
renamed **"Auto-correct"**; stripped model/consensus/cached attribution
badges; reworded strings to drop "AI"/"model" (translated to all 32
locales).
- Auto-runs once on review load so suggestions are ready without a
click; the button stays for the paste-references-later flow.

## Testing
- Backend: **3460 passed** (`backend/tests/`, emulator excluded) — incl.
new `test_pricing.py` + `test_proactive_auto_correct.py` (gating,
graceful failure, cache alignment).
- Frontend: **1006 jest tests pass**; type-clean; 33-locale key parity
verified.

## Notes / follow-ups
- Proactive activates on the **next deploy** (the lyrics-job env/secret
apply via this PR's ci.yml on merge).
- Left "AI" wording in two *separate* features untouched (different
subsystems): `lyricsReview.agenticMetrics` and
`lyricsReview.modals.correctionAnnotation` — flagged for a separate
decision.
- The production E2E spec's click-to-open flow may want a tweak now that
suggestions auto-run on load.

@coderabbitai ignore

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-11 18:22:51 -04:00