Commit Graph
133 Commits
Author SHA1 Message Date
Andrew Beveridge ☄️andClaude Opus 5.5 e4dad6e3c5 fix(delivery): hold completion email when Dropbox delivery failed (NOMAD-1800) (#1213)
## Summary

Job `dd892383` (NOMAD-1800) finished while the business Dropbox was at
its 6 TiB quota. Both Dropbox upload passes failed with
`insufficient_space`, but the job still went COMPLETE and emailed the
customer a delivery email whose link was the literal `[Dropbox URL not
available]`. The only #alerts signal was a raw traceback.

Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562874010939949110

## Changes

- **New `backend/services/delivery_guard.py`**
- `completion_email_hold_reason(job)`: if the job's *effective*
distribution settings deliver to Dropbox (`dropbox_path` and
`brand_prefix`, the same condition the workers use, so
private/NonPublished jobs count too) and `state_data.dropbox_link` is
missing, the email is **held**.
- `hold_completion_email(...)`: records
`state_data.completion_email_held = {reason_code, reason, recipient,
at}`, logs a WARNING, and sends **one actionable #alerts message per
job** (`send_throttled_ops_alert`, keyed per job). The message covers
the job, brand code, the masked recipient whose email was held, the
reason (`insufficient_space` is classified as "Dropbox is FULL"), the
fix steps and the runbook. The Pushbullet push is unchanged.
- `check_dropbox_space_usage()`: calls `users_get_space_usage()`, using
team allocation for the business account. It posts one #alerts message
when usage reaches 95% and records its state in Firestore
(`ops_failure_alerts/dropbox-space-usage`), so it stays quiet while
usage stays high and re-arms once usage drops below 95%. It runs on the
existing hourly `process-stale-reviews` cron, so no infra change is
needed.
- **`JobManager._schedule_completion_email`**: checks the hold before
sending. Made-for-you ownership still transfers to the customer.
- **Admin `POST /api/admin/jobs/{id}/send-completion-email`**: returns
409 when the link is missing, unless `allow_missing_dropbox_link: true`
is passed. A successful send clears `completion_email_held`.
- **`TemplateService.render_job_completion`**: no more `[Dropbox URL not
available]` / `[YouTube URL not available]` placeholders. When a link is
missing, that whole section is dropped (English default and GCS wording,
plus the localized template). Jobs with no Dropbox delivery at all
(tenant, no Dropbox configured) still get their email, just without a
Dropbox paragraph. The YouTube-queued and private paths are unchanged.
- Dropbox upload failure logs (orchestrator and native pass) go from
ERROR to WARNING, so one root cause raises one alert (the held-delivery
alert) instead of an error-monitor traceback.
- Docs: `TROUBLESHOOTING.md` § "Held completion email (Dropbox delivery
failed)" with the manual recovery steps, and `API.md` for the new 409
and override.
- Version 0.282.0.

Not done: item 5 of the handoff (an admin "redistribute Dropbox only"
action). The existing admin re-render also re-uploads YouTube, so it
isn't a clean fix for public jobs. The manual recovery steps are in
TROUBLESHOOTING.md instead.

## Testing

- New `backend/tests/test_delivery_guard.py` (31 tests): failure
classification, hold/no-hold matrix (no link, link present, no Dropbox,
tenant, private defaults, settings error), held marker and alert
content, quota threshold, alert-once and re-arm, failed send does not
latch, API error never raises.
- `test_made_for_you.py`: email held when the link is missing (ownership
still transfers, marker recorded, alert sent); email sent when the link
is present; email sent for a job with no Dropbox delivery.
- `test_admin_email_endpoints.py`: 409 without the link; override sends
and clears the marker; fixed link sends and clears the marker; job with
no Dropbox delivery sends normally.
- `test_template_service.py`: missing links never render placeholders
(en, de, unknown GCS template).
- Backend unit suite (`backend/tests`, excluding emulator tests): 6637
passed, 7 failed, 25 skipped. All 7 failures are
`test_internal_api.py::*requires_auth` / `*requires_admin`, which fail
the same way on unmodified `origin/main` locally (they get 200 instead
of 401, apparently from the local environment), and this PR doesn't
touch them.
- The local CodeRabbit CLI failed ("Invalid token"), so this PR is left
open for the bot review.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_016F41fLoBotUw2wUjGCuAok

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 17:53:08 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 6c53f0471a feat(youtube): auto-reply to song requests in YouTube comments with requests-board links (v0.280.0) (#1205)
## Summary

Backlog Frog #6 (YouTube channel → gen funnel). Andrew's spec: *"we
could probably also solve that by building some automation to detect
youtube comments asking for songs and auto-reply with a link to the
requests board"*.

An hourly bot reads new top-level comments on the Nomad Karaoke channel
and, when someone asks for a song:
- **Song already on the channel:** replies with that video's link.
- **Otherwise:** adds the song to the requests board (or finds the
existing request) and replies with a link straight to it, e.g. `Added
"Tiny Hitman" by Bear Ghost to our requests board 🥳 upvote it here and
we'll email you when it's made: <link>`.
- **Vague asks** ("more Johnnie please") get a generic board link.
- **Non-English comments** get a translated reply, which falls back to
English if a URL doesn't survive translation.

The reply promotes only the board, not gen (Andrew's call). The goal is
turning anonymous YouTubers into board users with an email address.

**Andrew signed this off on a 6-month local replay**
(`scripts/youtube-comments/replay.py`, read-only):
- 265 threads, 33 song requests.
- It flagged all 29 requests Andrew had answered by hand, and all 7
vague asks.
- It posted no replies to non-requests.
- His calls on borderline cases became prompt rules: yes to Playa Fly,
no to "Joe Cuervo Karaoke 🎤" on an unrelated video.

## Changes

- **Decision core:** `backend/services/youtube_comment_bot.py`, a pure
module with injectable ports (Gemini classifier/translator, board,
channel index), shared by the replay and the worker.
- **Worker:** `backend/workers/youtube_comment_bot_worker.py`, called
through `POST /api/internal/youtube-comments/run` (`?dry_run=true`).
- Creates a create-only action doc per comment **before** posting, so
nothing gets a double reply.
- Keeps a high-water mark with a 6h overlap. A Gemini error causes a
retry, not a permanent skip.
  - Per-run cap and quota coordination.
  - Posts to #noise per reply; failures go to #alerts, throttled.
- **Kill switch:** `YOUTUBE_COMMENT_AUTOREPLY_ENABLED` defaults to
**false**, and this PR leaves it off. A follow-up PR switches it on
after a prod dry run. `YOUTUBE_COMMENT_AUTOREPLY_START` enables a
one-time backfill of unanswered requests.
- **Requests board:**
- New `source="youtube_comment"`; bot-added requests are owned by the
placeholder `youtube-comment@nomadkaraoke.com` with no votes.
- The **first real up-voter claims the request**, inside the vote
transaction. The daily picker skips unclaimed requests, so a free credit
never goes to the placeholder.
- Frontend `?request=<id>` deep link: highlight, scroll, banner (33
locales).
- Signing in from the link uses purpose `requests_board:<id>`, so the
magic link returns to the request with `&upvote=1` and the upvote is
cast once.
- **OAuth:** the bot uses its **own** Desktop OAuth client and secret
`youtube-comment-bot-credentials` (`youtube` + `youtube.force-ssl`),
because Google refuses `force-ssl` in the device flow. The upload token
is untouched.
- **Infra (Pulumi, already applied locally):** secret container + Cloud
Scheduler `youtube-comment-autoreply-hourly`.
- **Description template:** a make-your-own line moves above YouTube's
"...more" fold (`utm_campaign=make_cta_top`). The daily backfill
re-drains the channel over about 11 days.
- **Docs:** `docs/REQUESTS-BOARD.md` § YouTube comment auto-reply,
API.md, README.md, plus the plan in
`docs/archive/2026-10-22-youtube-comment-autoreply-plan.md`.

## Testing

- `backend/tests/test_youtube_comment_bot.py` (33): skips, routing,
vague asks, translation fallback, malformed classifier output, API
helpers.
- `backend/tests/test_youtube_comment_bot_worker.py` (12): kill switch,
dry run, idempotency (second run and concurrent claim), go-live start,
cap, quota, missing scope, post failure, classifier-error retry.
- `backend/tests/emulator/test_song_request_service.py`: bot add/dedupe,
claim on first upvote, downvote doesn't claim, picker skips unclaimed,
human resubmit claims, `find_published_by_key`.
- `backend/tests/test_admin_login_link.py`: `requests_board:<id>`
redirect and purpose helper.
- `frontend/e2e/regression/requests-board-deep-link.spec.ts` (3):
highlight + sign-in purpose, upvote-once on return + param dropped, no
toggle-off when already upvoted.
- Full `make test`: backend 6584 passed; the only failures are the
local-environment `test_emulator_integration` cases, which fail
identically on `origin/main` (port 8080 is held by another session's
emulator). Frontend Jest: 1760 passed.
- Behavioural acceptance: the 6-month replay above.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_016E3ibN2EcKxnGQvEf6T4V8

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 01:51:15 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 b67b273056 feat(kjbox): decide partner secret for catalog resolve + fake judge + error monitor for decide search (v0.278.10) (#1190)
## ⚠️ Merge prerequisite (do BEFORE merging)

The Secret Manager secret `decide-gen-partner-secret` must have a
version before this merges, otherwise the CI deploy's `--set-secrets
...DECIDE_GEN_PARTNER_SECRET=decide-gen-partner-secret:latest` fails.

```
gcloud auth login admin@nomadkaraoke.com
cd ~/Projects/nomadkaraoke/karaoke-gen-decide-partner-resolve/infrastructure
GOOGLE_OAUTH_ACCESS_TOKEN=$(gcloud auth print-access-token) pulumi up     # review: should only add the decide-gen-partner-secret Secret
openssl rand -hex 32 | tr -d '\n' | gcloud secrets versions add decide-gen-partner-secret --project=nomadkaraoke --data-file=-
```

No extra IAM grant needed: gen's API runs as `karaoke-backend@`, which
already has project-level `secretmanager.secretAccessor`.

## Summary
- `POST /api/kjbox/catalog/resolve` now also accepts karaoke-decide via
header `X-Partner-Secret` (new setting `decide_partner_secret` / env
`DECIDE_GEN_PARTNER_SECRET`). Other kjbox endpoints stay
`X-Kjbox-Secret` only. decide gets its own AI-call budget (60/min) so
kjbox nights are never starved; the verdict cache is shared.
- Fake match judge / resolve for the local E2E stack (`FAKE_WORKERS`):
`judge_kind` / `judge_canonical_*` knobs; no Gemini or catalog calls.
- Error monitor now watches decide's new `karaoke-decide-search` service
and `song-index-build` job.
- Pulumi: declares the `decide-gen-partner-secret` Secret (value added
manually); CI deploy mounts it.
- Version 0.278.10.

## Testing
- [x] Changed-module + kjbox/catalog/error-monitor/fake-worker backend
tests pass (133 passed). Full `backend/tests` suite exceeds 10 min
locally, so left to CI.

## Review
- CodeRabbit CLI failed twice (WebSocket closed / connection error), so
no local CodeRabbit review completed; I did a manual diff review instead
(no issues found). Deliberately NO `@coderabbitai ignore`, so the GitHub
bot reviews this PR.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01VsLw9cTTMBab1tiRxCsYrm

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 21:42:27 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 2838d2dd1d fix(credits): keep credits_charged through job creation; one helper for confirm/refund/select (v0.278.2) (#1180)
## Summary
Fixes `credits_charged` being wiped from `state_data`, which made
duration confirm, refunds and the web cost dialog disagree. Mobile E2E
against the local stack found it: a 15-minute track showed "1 credit in
total (1 more)", and confirming the true total was rejected with 409.

**Root cause:** `create_job` debits the credit and records
`state_data.credits_charged`. But create-from-search, audio-search
create, made-for-you orders (which also lost `payment_bypassed`) and the
admin source override then replaced the whole `state_data` map, deleting
the field. Readers filled the gap with different defaults:
- reconcile: 1
- confirm-duration: 0
- 48h expiry: 0

**Fix**
- **Writers:** per-key `state_data.<key>` updates instead of whole-map
writes. Retry-from-scratch keeps the billing keys. Reconcile records
`credits_charged` when it pauses a job.
- **Readers:** one helper, `backend/services/job_credits.py`
(`get_credits_charged`, `is_payment_bypassed`). Every reader uses it:
reconcile, confirm-duration, `/select`, the bulk worker,
`_refund_credit_for_job` and the 48h expiry.
- For legacy jobs that lack the field, it returns what `create_job`
actually debited: 0 for bypassed, admin, tenant or no-email jobs,
otherwise 1.
- **`/select` and the bulk worker** never bill payment-bypassed jobs
(made-for-you, tenant, admin-created). This includes legacy ones whose
flag was wiped.
- **API responses:** `GET /api/jobs/{id}` and the job lists return the
resolved `credits_charged` when the stored field is missing. This is
response-only and never persisted, so the web and mobile clients
acknowledge exactly `credits_charged + pending`.
- **Confirm-duration** records the full total, so cancelling after a
confirm refunds it all. The 48h expiry refunds once:
`cancel_job(refund=False)` stops a second refund.
- **Admin source override** clears the old source's duration, selection
and confirm markers and keeps `credits_charged`. The next `/select` then
charges only the difference.
- **Web cost hook:** uses the server's resolved figure.
- Version 0.278.2.

## Testing
- New `test_credits_charged_consistency.py` covers:
- reconcile, then confirm with total 2, then cancel: 2 refunded, for new
and legacy jobs
  - 409 on the old total
  - refund amounts unchanged for jobs that have the field
  - `/select` and the bulk worker never deducting on bypassed jobs
  - the override followed by `/select` charging only the difference
- Stale-expiry legacy tests, and an emulator test for the dotted-path
merge (runs in CI).
- Backend unit tests: 6411 passed. Emulator integration tests run in CI;
locally they collided with a running local stack. Frontend jest: 1754
passed.
- An independent review found one Critical issue (`/select` billing
bypassed jobs) and one Important issue (web/server default mismatch).
Both are fixed and re-reviewed clean. Local CodeRabbit raised one minor,
which is fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_011yLH4kWXMqjeKfZUG3ZiWZ

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 08:38:35 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 1616bc997b fix(api): sign final MP4s in GET /api/jobs/{id} download_urls (v0.278.1) (#1179)
## Summary
`GET /api/jobs/{id}` has always returned an empty `download_urls` for
real jobs. `JobManager.get_output_urls` signs only `job.output_files`,
and current workers never populate that field. Final videos live in
`file_urls.finals`.

This change also signs each `file_urls.finals` entry whose key ends in
`_mp4` (for example `lossy_720p_mp4`):
- URLs expire after 120 minutes.
- When a key exists in both, the `output_files` entry wins.
- Entries that aren't strings are skipped.
- After a real signing failure, it stops signing further entries, so a
stalled signer can't add a timeout per MP4.

**Why:** the mobile app (nomad-mobile Phase 1) streams the 720p MP4 in
`expo-video` and shows download progress. Both need a URL that supports
HTTP Range requests. The authenticated `/download/` proxy doesn't
support them.

Signing still goes through the bounded executor in `storage_service.py`
with its 12s timeout. The route is a sync `def`, so the event loop is
never blocked.

Version 0.278.1.

## Testing
- 6 new `TestGetOutputUrls` cases in
`backend/tests/test_job_manager.py`. That file passes: 58 tests.
- Local CodeRabbit raised one finding (the type guard), which is fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_011yLH4kWXMqjeKfZUG3ZiWZ

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 05:25:45 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 695b8ddf84 feat: mobile app backend — handoff token, app checkout return, Google/Apple sign-in (v0.278.0) (#1178)
## Summary
Backend support for the Nomad Karaoke mobile app (nomad-mobile Phase 1,
"Make it").

- **`POST /api/users/auth/handoff-token`**: the app opens a job's web
review already signed in. The token is a single-use, 120s
`job_review:<id>` magic link, returned as
`{FRONTEND_URL}/{locale}/auth/verify?token=…&handoff=1`.
- The verify page skips the "Complete Sign-In" tap only when the URL has
`handoff=1` **and** the server confirms the token was minted as a
handoff (`link-status` → `handoff: true`). Emailed links keep the
scanner guard.
  - The web locale is validated against the 33 web locales.
- **Checkout `return_to: "app"`**: Stripe success and cancel go to `GET
/api/users/credits/app-return`, which 302s to
`nomadkaraoke://checkout?status=success|cancelled`.
  - The client never supplies a URL, so this adds no open redirect.
  - `session_id` is never forwarded.
- **`POST /api/users/auth/google` and `/auth/apple`**: ID-token sign-in
(PyJWT + JWKS: RS256, issuer, audience, expiry, `email_verified`). It
returns the same `VerifyMagicLinkResponse` as email-code sign-in,
through a shared `finish_verified_sign_in`.
- Email-code, Google and Apple all share the same abuse checks
(`_sign_in_checks`).
  - A JWKS outage returns 503 `identity_provider_unavailable`.
- Google audiences default to the new mobile web and iOS OAuth clients,
which are recorded in ARCHITECTURE.md.
- **Job summary** now includes `credit_refunded`.
- **Local stack:**
- fake-config `search_confidence` (perfect / recommended / unsure), with
a drift guard tied to the frontend `getSearchConfidence`
  - `youtube_url_enabled`
  - `fake:<email>` social tokens, accepted only in fake mode
- PyJWT is now a direct dependency (`pyjwt[crypto]`). Before this it was
only present transitively.
- Version 0.278.0.

## Testing
- `make test` is green:
  - CLI: 2088 passed
  - backend unit: 6360 passed
  - emulator: 156 passed, plus 4 handoff lifecycle tests
  - frontend jest: 1751 passed
- New suites:
  - `test_handoff_token.py`
  - `emulator/test_handoff_token_lifecycle.py`
  - `test_checkout_app_return.py`
  - `test_social_signin.py`
  - `test_magic_link_status.py`
  - `fake-search-tiers.test.ts`
  - verify-page tests
- Reviews:
- An independent code review was approved, and its hardening suggestions
were applied.
  - A local CodeRabbit review found nothing.

Plan: `docs/archive/2026-10-17-mobile-app-auth-handoff-plan.md`. Spec:
nomad-mobile
`docs/superpowers/specs/2026-10-17-mobile-phase1-make-it-design.md`.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_011yLH4kWXMqjeKfZUG3ZiWZ

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 04:03:05 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 8ec789a9aa fix(bulk): album lookup reads our MusicBrainz mirror, never the live API (v0.277.0) (#1176)
## Summary
Bulk → By album → "Britney Spears" showed **"Lookup failed. Please try
again."** `/api/bulk/album/artists` returned 502 because every attempt
to `musicbrainz.org/ws/2` from Cloud Run failed with
`httpx.ConnectError('')`. The same request from a home IP returns 200.
It last worked on 9/29. Most likely MusicBrainz is refusing our shared
GCP egress.

Per Andrew, we should only query our own infra, with no live-API
fallback. `musicbrainz_service` now reads **only our BigQuery
MusicBrainz mirror** (`karaoke_decide.mb_*`, which karaoke-decide's
`mb-refresh` rebuilds weekly from the official dumps).

## Changes
- **`backend/services/musicbrainz_service.py`**
- The HTTP client, throttle and retries for musicbrainz.org are removed.
Queries go through the BigQuery REST `jobs.query` endpoint (httpx + ADC;
no new dependency).
- Queries are parameterized, capped with `maximumBytesBilled` = 2 GiB,
and cached in memory for 10 min.
  - Lookups by table:
- artists: `mb_artists_normalized`. Ranking is exact match (including
"The "+q), then prefix, then substring, then popularity.
    - albums: `mb_artist_credit_artists` → `mb_release_groups`.
    - editions: `mb_releases`.
- tracks: `mb_release_tracks`, the new table in karaoke-decide#158,
clustered by release group + release.
- Measured against prod data: about 0.6–1 s per query, MB to 0.4 GB
billed.
- Canonical-pressing selection now ranks an edition by its best
release-event country.
- **Route:** passes `release_group_mbid` through on edition switches. An
unknown release returns **404** instead of 502.
- **Frontend:** `switchEdition` sends the album's `release_group_mbid`
so the lookup stays cluster-pruned. Older clients still work through a
one-off lookup that scans about 0.2 GB.
- **Infra:** the `karaoke-backend` SA gets project-level
`bigquery.dataViewer` + `bigquery.jobUser`. decide's Pulumi owns the
dataset ACL, so a dataset-scoped grant would fight it.
- **Docs:** `docs/API.md` and
`docs/archive/2026-10-16-bulk-album-musicbrainz-mirror.md`.

## Rollout (order matters)
1. Merge karaoke-decide#158, then build `mb_release_tracks`: by hand
(SQL in the archive doc, about $0.08) or at Sunday's refresh.
2. Run `pulumi up` for the IAM change. The agent's credentials are
read-only, so this needs Andrew.
3. Merge this PR.

## Testing
- `make test`: all passed (backend + 1735 frontend tests).
- New and rewritten unit tests cover normalization, BigQuery row
decoding, the REST request shape (no musicbrainz.org, billing cap,
params, cache), the HTTP/timeout errors, every lookup, the release-group
lookup path, and the 404.
- Ran the service live against the prod mirror: Britney Spears → 39
albums → …Baby One More Time (11 tracks, 6 variants); also checked
"beatles", "Beyoncé", "ac/dc", "Sigur Rós".
- CodeRabbit CLI: 1 minor finding (404 vs 502), fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01C1sMXk2MaxBLbep5jgiFab

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 00:06:16 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 b2ef88a861 fix(review): unify backing-vocals recommendation on the 3-stem scorer (v0.276.2) (#1169)
## Summary
- `backing_vocals_analysis.recommended_selection` (legacy
`AudioAnalyzer` rule: keep only if >20% audible or a segment > −20 dB)
disagreed with the auto-approval `score_backing` verdict on **59/389
jobs (15%, Jul–Oct)** — every case 0.5–20% audible with no loud segment.
The finish modal followed the scorer, the instrumental screens followed
the heuristic, so the same job showed different "✓ recommended" stems
(job 58555663).
- Where they disagreed, admin/Andrew picks matched the scorer 14/18;
customer picks lean clean regardless (38% clean even when both signals
say keep). The heuristic also recommended keeping on 5 jobs the scorer
flagged as "backing stem IS the lead" (humans picked clean 4/5).
- 58555663 itself: the intro backing (16–29s) is a voiced B3–C♯4 hum
motif with the lead stem silent, pitch-matched to the "Mm-mm-mm" hum
lines — `with_backing` was right.

## Changes
- `unified_backing_recommendation` / `with_unified_recommendation` in
`auto_approval/instrumental.py`: scorer's confident verdict → `clean` if
backing-is-lead → else the heuristic (preserved as
`heuristic_recommended_selection`). Idempotent.
- Applied at write time (`screens_worker._analyze_backing_vocals`) and
read time (correction-data, `/instrumental-urls`,
`/instrumental-analysis`) so already-analysed jobs are consistent too.
- `backing_is_lead()` predicate extracted from `score_backing` and
shared.
- No change to auto-completion / enforcement (`backing_decision`
untouched); frontend unchanged.
- Docs: API.md note, LESSONS-LEARNED entry. Version 0.276.2.

## Testing
- [x] `make test` — backend 2054 + 6243 + emulator 152, frontend 1730,
all passing
- [x] New unit tests (helpers matrix), worker write-path test, route
test with the 58555663 signal profile

## Review
- [x] CodeRabbit CLI review completed locally — no findings

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_018aUxu88FZajta46BRdmpcc

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 13:08:49 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 abf7b693b4 feat(review): instrumental chooser appears when stems land after review opens (v0.276.0) (#1167)
## Summary

Recent jobs didn't show the instrumental chooser / backing-vocals
waveform in the Preview Video modal. Root cause: review opens as soon as
**lyrics** finish (~2 min after download), while audio separation lands
2-3 min later. The review page fetches `instrumental_options` once at
load, so stems that arrive mid-review never reach the modal, which
silently fell back to "Proceed to Instrumental Review".

Evidence (last 5 admin jobs): in 4/5 the review page was opened 11-23 s
after ready-for-review, and separation finished 1m46s-3m00s **after**
the page opened; the one opened after separation (6cb0f55f) was fine.

## Changes

- **Backend:** `GET /api/review/{id}/instrumental-urls` also returns
`audio_complete`, `job_status`, `backing_vocals_analysis` and a freshly
derived `auto_approval` summary (backwards-compatible).
- **Frontend:** new `useLateInstrumentalStems` hook. When the page
loaded without stems it polls every 15 s (max 10 min) until
`audio_complete`, which is set only after stems, OGGs and backing
analysis are all written, so the chooser never shows half its data.
Fresh data is overlaid as a separate modal prop, not merged into `data`,
so undo history, saved sessions and the in-flight preview render are
untouched. The getter is held in a ref because the page re-creates its
API client each render.
- **Modal:** while waiting, a spinner box ("Still separating the music
from the vocals…") and a disabled CTA ("Waiting for the instrumental…").
The chooser, ✓ badge, backing waveform and preview toggle appear in
place when ready. On failure or timeout it falls back to the old path
with a short note.
- i18n: 4 new strings, 33 locales.
- Fixed an existing Playwright test that expected "Proceed to
Instrumental Review" with both stems present (stale since v0.214; it now
picks Advanced mode).

## Testing

- `make test`: all pass (backend, plus frontend 140 suites / 1729 tests)
- New: 2 backend endpoint tests, 6 hook tests, 3 modal tests, 1
Playwright regression test (loads without stems → waiting state →
chooser appears)
- Local CodeRabbit: 1 finding (false positive: `advancedModeClean`
exists in all 33 locales)

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01QKJ7TxzFoUeNTug1PR8Cwh

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 00:21:35 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 c277bc8ee3 feat: local E2E stack (fake workers + test API) + email-code sign-in (v0.275.0) (#1165)
## Summary
- **Local stack:** `scripts/local-stack.sh up --frontend` runs the
**real** backend on the Firestore + GCS emulators with `FAKE_WORKERS=1`.
In-process fake workers move jobs through their real states in seconds,
using tiny real media, so web Playwright and mobile Maestro full-journey
tests can run offline against the same endpoints.
- **Email-code sign-in** (nomad-mobile spec §4.1): `POST
/api/users/auth/send-code` + `verify-code`. The code logic is shared
with kjbox via `LoginCodeStore`, and sends are throttled per email and
per IP.
- **Bug fix found by the new suite:** `confirm-duration` moved jobs to
`separating_stage1`, from which they can never reach review. It now
resumes at `downloading`. No prod job had taken this path yet.

## Changes
- `backend/services/fake_workers/`: dispatch seam in
`WorkerService.trigger_*`, plus fake search, download, checkout and
outbox, scenarios, and per-user config. The real stem upload, transcode,
backing-vocals analysis and preview render are reused. Every check is
strict (`is True`), so a MagicMock settings object never enables the
fakes, and startup **refuses FAKE_WORKERS in production**.
- `/api/test/*` router (mounted only in that mode): `status`,
`scenario`, `fake-config`, `emails`, `reset`, plus the fake Stripe
checkout page. Paying it runs the real webhook handler.
- `frontend/e2e/local-stack/` (9 specs, ~50s):
  - sign-up → make → review → instrumental → download
  - buy and cancel credits
  - duration confirm, failed, complete and review states
  - code sign-in and lockout
- New required CI job `local-stack-e2e`.
- Docs: TESTING.md § Local stack, DEVELOPMENT.md, API.md, README,
LESSONS-LEARNED.
- `MAX_SIGNUPS_PER_IP` can now be set by env. Only the local stack
raises it; production keeps the default of 2.

## Testing
- [x] Backend unit: 6175 passed
- [x] Emulator: 152 passed
- [x] Frontend Jest: 1706 passed
- [x] Local-stack Playwright: 9/9 passed (CI mode)

## Review
- [x] CodeRabbit CLI review completed locally. 1 minor finding declined:
it suggested swallowing infrastructure errors in send-code as fake
"sent" responses, which would hide real outages from users.

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01AGuxtgoscF6RJGtpjHPg3W

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 16:00:24 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 bfc8f7fb0f feat(cdg): configurable title card + per-theme CDG lyrics font (Vocal Star up card) (v0.273.0) (#1157)
## Summary
Vocal Star (and other commercial producers) need CDG files in their own
house style. Matt's 2025-02-20 email showed our CDG up card overflowing
its bordered design. He asked for his simpler card instead: logo on top,
then title, a small "MADE FAMOUS BY" line, then the artist.

Two pipeline limits blocked that:
- the CDG title card was hardcoded;
- a theme could only have **one font** for everything. The shared `font`
asset (`intro.font`) overwrites `karaoke.font_path` and `cdg.font_path`
at render time.

## Changes
- **Title card (cdgmaker composer):** new optional keys; all defaults
reproduce today's output.
- `title_separator_text` / `_color` / `_font_size`: a line between title
and artist.
  - `title_font_size`, `artist_font_size`, `title_max_height`.
- `title_block_top`: centre the block below a logo baked into the
background.
- It shrinks the title first, then the artist, so it can't loop forever.
- **Outro:** respects `title_block_top` and shrinks long text. Before,
an `assert` crashed when the text was taller than 200px.
- **Fonts:**
- The CDG title card and outro use the theme's main font
(`cdg.title_font_path`, fed by the shared `font` asset).
- A theme may keep a **distinct CDG lyrics font** via a new `cdg_font`
asset. The theme service emits it only when it differs from
`intro.font`, and `update_asset_paths` applies the shared font first so
section overrides win.
  - No visible change for existing themes, which all use one font.
- **Tenant theme editor:** keeps a distinct, available CDG font on save,
and copies it into the theme's assets if it's bundled.
- Docs: API.md.

## Testing
- New `tests/unit/lyrics_transcriber/output/test_cdg_title_card.py`
renders **real CDG files**, decodes the packets back into frames and
checks:
  - the separator sits between title and artist;
  - text stays below `title_block_top` and inside the safe area;
  - the title font is applied, and a missing one falls back;
  - long outro text shrinks instead of crashing.
- New tests for asset-mapping order, `cdg_font` emission, theme-editor
font handling, and the updated mapping assertion.
- Package unit tests: 2053 passed. Backend: 6056 passed (emulator tests
excluded locally).
- Local CodeRabbit with `--fresh`: no findings.
- Rendered Vocal Star samples (short, wrapped-long and very long
title/artist), decoded and visually checked against Matt's design.

After deploy, the Vocal Star theme gets: a new up-card background (logo
cropped from Matt's own card), separator "MADE FAMOUS BY",
`title_block_top`, black border, Avenir as the main font and Oswald kept
for CDG lyrics.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Gp5fBmsmWJxgkEHb4JfhXR

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 00:58:42 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 5047f39da2 feat(tenant): per-track catalogue codes prefix tenant outputs (Vocal Star VS codes) (v0.272.0) (#1153)
## Summary
Vocal Star (Matt, April 2025) asked for their own VS catalogue code at
the start of every output file, with all files in one folder (e.g.
`VS6588 - Gerry and the Pacemakers - Ferry Cross The Mersey`). Tenant
operators can now set a per-track **Track code**. It becomes the job's
preserved brand code (`keep_brand_code`, which both render paths already
honour), so the Dropbox folder and every delivered file use it instead
of an allocated `VSTAR-NNNN`.

## Changes
- **Backend:** `POST /api/jobs/create-with-upload-urls` accepts
`track_code`.
  - Tenant portals only; `400` elsewhere.
  - Strict filesystem-safe validation, upper-cased.
  - Maps to `JobCreate.keep_brand_code`.
- **Bulk analyze:** a leading code in filenames (`VS6588 Artist - Title
Guide.mp3`) is split off and returned as `track_code`.
  - Files pair only when code **and** song match.
- LLM pairs with conflicting codes are rejected, and the code is
stripped from LLM-extracted artists.
- **Frontend:** an optional Track code field on the tenant single-track
form and on each bulk row.
  - Client validation mirrors the backend, with a visible error.
  - Resumed bulk rows keep their code, locked.
  - 33 locales translated.
- Docs: API.md. Version 0.271.0 → 0.272.0.

## Testing
- Backend: new tests for endpoint mapping, rejection outside tenants,
unsafe codes, normalisation, filename parsing, the band-name false
positive (UB40), same-code/different-song and LLM conflicts.
- Full backend suite: 5985 passed. The only 30 failures are emulator
tests, because the local Firestore emulator wasn't running.
- Frontend: full Jest 1680/1680 passing, plus new TenantJobFlow /
TenantBulkFlow tests. `tsc` shows no new errors.
- Local CodeRabbit: 3 cycles, all real findings fixed. The one remaining
note is on a pre-existing string.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Gp5fBmsmWJxgkEHb4JfhXR

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 00:05:57 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 ff9b716435 feat: first-touch UTM/referrer signup attribution + UTM on YouTube description (v0.269.0) (#1154)
## Summary
This is BACKLOG Frog #2 ("plug the gen revenue leaks"), attribution
track. Until now we stored no acquisition source, so we couldn't tell
which channel brings the people who pay.

- **Capture (frontend):** `lib/attribution.ts` stores the first-touch
`utm_*` params, the external referrer host and the landing path for 30
days. It runs in the `[locale]` layout and in `LocaleRedirect`, before
`location.replace` wipes `document.referrer`. The data is sent with
`POST /auth/magic-link`.
- **Store (backend):** `services/signup_attribution.py` whitelists,
length-caps and sanitises the data. On account creation it sets
`signup_attribution` and `signup_source`: the `utm_source`, or a
referrer channel label such as `youtube`, `google`, `website`, `decide`
or `kjbox`. Malformed input never blocks sign-in, and the request field
is bounded (16 keys, values up to 500 chars).
- **Report:** `GET /api/users/admin/attribution-summary?days=90` returns
signups, verified, made a job, paid and revenue, per source. The admin
user detail shows the signup source and attribution.
- **YouTube:** description template links now carry
`utm_source=youtube&utm_medium=description&utm_campaign=requests|make_cta`.
They're static, so they're still derivable from the title, and the daily
backfill rolls them out. The `requests.nomadkaraoke.com` Cloudflare
redirect now has `preserve_query_string` (Pulumi `nomad-cloudflare`,
**already applied** locally).
- **Docs:** STRIPE-SETUP.md gets a "Legacy Payment Links" section. The
old order-form link `dR67sx…` was **deactivated 2026-10-14** and points
people to gen; it had 0 completed checkouts in about 30 sessions since
Dec. Also updated API.md.

Companion: nomadkaraoke/public-website#34 makes
`nomadkaraoke.com/r/CODE` pass the UTM params through.

## Testing
- `make test`: backend 5986 passed, emulator 150, frontend Jest 1683,
package 2044. All green.
- New tests: `backend/tests/test_signup_attribution.py` (cleaning,
channel labels, magic-link wiring, bounded payload, summary endpoint)
and `frontend/lib/__tests__/attribution.test.ts`.
- Local CodeRabbit: 3 findings fixed (malformed referrer, NaN expiry,
unbounded payload); re-review found 0.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_019KuzHAU4rxkv8JfD8NmHTN

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 23:31:16 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 3566fd136e fix: loop-stall alert flapped on idle autoscaled instances (v0.268.5) (#1140)
## Summary
The `>=2 event-loop stalls of 10s+ in 30 minutes` alert flapped on
2026-10-12 and 2026-10-13. Investigation:
- All 11 ERROR-level `EVENT_LOOP_STALL ended` lines in the last 4 days
(10-40s each) were on instances with **zero requests** in or near the
stall window. The check joined `labels.instanceId` between the stall
lines and the request logs.
- None of them had a culprit. A process freeze advances the 1s/5s/15s
sample schedule without taking a sample.
- **Pattern:** Cloud Run autoscaling starts an extra instance, it serves
one request (e.g. `/api/tenant/config`), then it sits idle and
CPU-throttled. The startup warmup thread competes for CPU slices
meanwhile (SpaCy preload took 690s on one instance). The watchdog reads
that as a loop stall.

## Changes
- `RequestTracker` + pure-ASGI `RequestTrackingMiddleware` (outermost)
count in-flight and started requests.
- `LoopWatchdog(request_tracker=...)`: a stall logs at ERROR, the level
the alert counts, only when a request was in flight at its start or
arrived during it. The verdict waits 0.5s after the loop resumes so
requests queued in the socket are counted. The same rule applies to `in
progress` samples, so they don't show up in the error monitor either.
- Idle stalls still log at WARNING with `requests_affected=0` and are
still recorded to `client_events` (`detail.requests_affected`).
- Without a tracker, behaviour is unchanged.
- Docs: TROUBLESHOOTING, LESSONS-LEARNED, API.

No infra change: the alert metric already filters on `severity>=ERROR`.

## Testing
- 10 new unit tests:
  - idle → WARNING;
  - in-flight request → ERROR;
  - request queued during the stall and accepted after resume → ERROR;
  - request finished mid-stall → still ERROR;
  - in-progress sample levels;
  - no-tracker behaviour unchanged;
  - middleware counting, including exceptions and non-http scopes;
  - middleware is installed outermost.
- `make test`: all passed.
- Local CodeRabbit review: 2 findings, both fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01DTfrTaaZQfXUUduAhZpLeq

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 12:38:01 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 983f6ecdcd fix(storage): keep user-uploaded inputs forever; uploads/ expires after 180d not 7d (v0.268.2) (#1137)
## Summary
`uploads/` had a bucket lifecycle **delete after 7 days**, and user
inputs were only partly copied out of it. The only copy step was the
audio worker (stem separation) copying the mix to `jobs/{id}/input/`,
best-effort. Everything else stayed in `uploads/`:
- bring-your-own-instrumental / tenant jobs, which skip separation;
- the uploaded instrumental, which was only staged at render time;
- lyrics and style uploads, which were never copied.

Result:
- randy-vild re-renders 404'd on their own input (2026-10-08). That data
was restored from Andrew's Dropbox.
- Firestore shows 64 completed Jan–Feb 2026 consumer jobs, plus
BYO-instrumental jobs, whose inputs are gone permanently.

## Changes
- **`backend/services/input_persistence.py`** (new):
- Server-side copies `uploads/{id}/**` to `jobs/{id}/input/**` and
repoints every job field that referenced the upload:
`input_media_gcs_path`, `existing_instrumental_gcs_path`,
`lyrics_file_gcs_path`, `style_params_gcs_path`, plus any matching value
under `file_urls` / `state_data` / `style_assets`.
  - Idempotent: unchanged copies are skipped.
- The audio, lyrics, screens, render-video and video workers all call it
on entry, via `asyncio.to_thread`. On failure it logs an ERROR, which
alerts, but doesn't fail the job.
- **`POST /api/admin/persist-uploads`** (dry run by default): backfill
for jobs no worker will touch again. Lookup failures are isolated per
job.
- **GCE encoder:** prefers the explicitly downloaded top-level
`existing_instrumental.*`. The recursive glob could otherwise return the
raw, unconformed copy now kept under `input/audio/`.
- **Pulumi:** `uploads/` lifecycle goes from 7 to **180 days**; `temp/`
stays at 7. **Already applied** and verified directly on the bucket.
- **Docs:** LESSONS-LEARNED, DISASTER-RECOVERY (new user-inputs row; the
stale "30d noncurrent" figure corrected to 7d), API, README.

## Verification
- `make test` all green.
- 16 new tests:
- copy mapping, repointing, idempotency, re-copy when a kept copy
differs, already-expired paths left alone;
  - the worker wrapper never raises;
- an AST guard that every pipeline worker calls the copy step via
`to_thread`;
  - backfill dry run / apply / error isolation;
  - the encoder's instrumental preference.
- Local CodeRabbit: 2 findings, both fixed (lyrics path, per-job
lookup); re-review clean.

## After merge
Run `POST /api/admin/persist-uploads` (dry run, then `?dry_run=false`)
to copy the inputs of the 51 jobs that still have objects in `uploads/`.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 17:33:06 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 f2612ac110 fix: close edit→delete refund loop, retry lost YouTube uploads, unblock event loop (v0.268.1) (#1135)
## Summary
Investigation of two 2026-10-08 Discord alerts (plus a third that
arrived mid-session) turned up four issues:

1. **Credit-refund loop.** A brand-new welcome-credit account received a
finished video (job 8a10a22d), clicked **Edit** (status back to
`awaiting_review`), then **deleted** the job and got its credit back.
`delete_job` refunds any non-complete status, and Edit makes a delivered
job look "never completed".
2. **YouTube 410 "Gone" dropped the upload.** The resumable session was
lost 1.3s into the upload. `upload_video` called `execute()` once with
no retries, and the orchestrator only queued quota errors, so any other
failure lost the upload.
3. **EVENT_LOOP_STALL 21s in `upload_json`.** `complete_review` (an
async route) wrote the corrections JSON to GCS directly on the event
loop. Ranking a week of watchdog stacks by route showed the same pattern
in `recover_stuck_jobs` (~260 stalls/week), the YouTube queue processor
(its `_process_single_upload` ran a whole upload on the loop),
`create_job_from_search` (theme prep), `_validate_audio_durations`
(15s), and the sync `run_job` / generation bump inside worker triggers.
4. **False 30s "stall" in asyncio `get_debug`.** Cloud Run CPU
throttling froze the whole instance mid-callback with no request in
flight. The watchdog only discounted freezes where the loop was parked
in `select()`.

## Changes
- **Refund:** `_refund_credit_for_job` skips jobs with `edit_count > 0`
(already delivered). This covers delete, cancel, fail and the 48h review
auto-expiry.
- **YouTube:** on 404/410 the upload restarts with a fresh session (3
attempts, with backoff). `execute(num_retries=3)` lets googleapiclient
resume chunks on 5xx/429 within a session. Any other failure is queued
in `youtube_upload_queue` (`reason="upload_error"`) for the existing
processor, which retries up to 5 times. Admins still get a Discord
warning: "queued for automatic retry".
- **Event loop:**
- New `backend/utils/loop_bridge.py`: `run_in_body_thread` runs sync
route bodies on a dedicated 16-thread pool, so they can't exhaust the
default executor the dispatched coroutines need. `run_on_loop`
dispatches coroutines back to the API loop (`run_coroutine_threadsafe`,
120s timeout).
- `complete_review`: its sync save runs via `to_thread`, and the render
trigger stays on the loop.
- `create_job_from_search`, `recover_stuck_jobs` and
`retry_pending_render_jobs`: bodies run in the route-body pool.
`_retrigger_parked_download` now does its writes in that thread.
- `_validate_audio_durations` and the YouTube queue's
`_process_single_upload` keep their async signatures but run in threads.
The queue's quota and Firestore calls go through `to_thread`.
- `worker_service`: the generation bump and `client.run_job` go through
`to_thread`.
- **Watchdog:** if the watchdog thread itself oversleeps by 2s or more
while the process uses less than 25% CPU, the whole process was frozen.
That time is subtracted from the stall and isn't sampled. Blocking I/O
(the watchdog still wakes on time) and GIL-hogging CPU work (CPU keeps
advancing) are still reported.
- Docs: `API.md` (edit/refund note) and `LESSONS-LEARNED.md`
(mostly-sync routes that await one trigger; throttling freezes).

## Testing
- Backend unit suite: **5863 passed**, 36 skipped (`pytest backend/tests
--ignore=backend/tests/emulator`).
- New tests:
- edited-track refund cases (delete in awaiting_review /
lyrics_complete, cancel);
  - YouTube restart on 410, give up after 3 attempts, no restart on 403;
  - orchestrator queues failed uploads and warns when queueing fails;
  - loop-responsiveness ticker tests for each offloaded path;
  - `run_on_loop` result, exception and timeout-with-cancel;
  - `run_in_body_thread` copies context and uses its own pool;
- watchdog: freeze ignored, I/O stall and CPU hog still reported, freeze
subtracted from a real stall.
- Local CodeRabbit: 3 cycles. Fixed the nested-executor deadlock risk,
the missing `run_on_loop` timeout, the parked-download writes on the
loop, and a coroutine leak. The last cycle had 0 findings.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 16:15:47 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 60687d6cd6 feat(theme): editable + previewed end screen; optional title/end screens (v0.268.0) (#1134)
## Summary
Randy Vild (tenant `randy-vild`) emailed: *"I have completed 3 songs but
all three songs produce the large text at the end even though I have in
my settings smaller text like in my beginning."*

Root cause: the theme editor's size/position sliders only edit
`intro.*`. The end screen has its own `end.*` layout (title 350 /
closing message 400 / yellow capitals artist in Randy's theme). The
editor neither exposed it nor previewed it ("The end screen isn't
previewed").

## Changes
- **End screen tab**: own title/artist colours, size + position sliders
(closing message, song title, artist), show/hide song title / artist
name, capitals toggles, and **Copy title card layout**. That copies the
title card's regions/colours/capitals and moves the closing message just
above the title, or below the artist if there's no room, so they never
overlap.
- **End screen preview**: `theme_preview_service.render_end_screen` uses
the same `VideoGenerator.create_end_video` as `screens_worker`. The
preview column shows the screen for the active tab plus the lyrics
frame; Advanced shows all three.
- **Leave out the title card / end screen**: optional `intro.enabled` /
`end.enabled` (absent = included; not added to the DEFAULT_*
required-field lists). The setting is honoured end to end:
- The screens worker skips the omitted screen and records
`state_data.screens_included`.
- The video worker's prerequisite check and the `/retry` ladder
(`_has_title_screen`, which also requires `screens_progress: complete`)
treat an omitted screen as done.
- The orchestrator sets the omitted screen's path to `None` and gives
the original-vocals guide an intro offset of 0.
- Encoding passes explicit `include_title_screen` /
`include_end_screen=false` to the GCE worker. Without them, its loose
`*title*.png` / `*end*.png` globs would rebuild the screen from
`style/*_background.png`.
- `LocalEncodingService.encode_lossless_mp4` builds the concat from
whichever segments exist (1–3).
  - The portrait renderer drops omitted cards.
- 11 new strings × 33 locales (`endScreenHint` removed). Docs: API.md,
README.md.

## Verification
- `make test` all green: frontend 1665 tests; backend unit, integration
and emulator suites.
- New `backend/tests/test_theme_screen_omission.py`, 28 tests:
  - screens worker for all 4 include combinations;
  - prerequisites and the retry helper;
  - orchestrator config and encoder options;
  - GCE `run_encoding` with decoy theme PNGs present;
  - concat for n = 1/2/3.
- Real-ffmpeg renders:
- end-screen preview (its own background; text size changes only the end
screen);
  - portrait durations with screens omitted.
- Frontend: editor tests for the end tab, copy layout, region toggles
and enable switches, plus unit tests for the `copyTitleCardLayoutToEnd`
util.
- Rendered Randy's real theme locally:
  - as saved, the end screen reproduces the large text;
  - after Copy title card layout, it matches his title card.
- Local CodeRabbit review: 0 findings.

## Deploy notes
- The worker paths changed (`gce_encoding/`,
`local_encoding_service.py`, `karaoke_gen/`), so CI blue-green redeploys
the encoding worker.
- Existing themes are unaffected: they have no `enabled` key.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 15:09:25 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 9d04274de7 feat(tenant): theme edits reach in-progress tracks + outdated-track re-render prompts (v0.267.0) (#1132)
## Why
Randy Vild edited his portal theme and his next video (RVILD-0003) still
had the old title card. Each job copies the theme when it's created.
This one was created 30 minutes before his edit and was waiting for
encoding capacity when he saved. Edits only reached jobs created
afterwards, finished tracks gave no sign they were outdated, and admins
saw two identical "Re-render" buttons on portal tracks.

## What
- **Theme save → in-progress tracks**
(`services/theme_change_service.py`): every tenant track that hasn't
started rendering gets a fresh copy of the theme. Tracks whose screens
may already exist (`generating_screens`…`in_review`,
`render_pending_capacity`) are flagged `state_data.theme_screens_stale`.
- **Render-worker divert:** a flagged job in `review_complete` goes once
through the screens worker (`LYRICS_COMPLETE` +
`regen_restore_status=review_complete`, the same bounce the theme
re-render uses). The screens worker regenerates the screens and
re-triggers the render. Doing this at render time covers every path into
render (review submit, auto-approve, retry, capacity retry) and never
interrupts someone mid-review. Stale `finals/*.mov` files are deleted
too, so the encoder can't splice the old intro back in.
- **Edit of a finished tenant track** picks up the current theme when
the track's copy is outdated.
- **New endpoints:** `GET /api/tenant/theme/outdated-jobs`, plus `POST
/api/tenant/theme/rerender-outdated`, which is quiet (no per-track
emails), capped at 20 per call, and keeps its blocking I/O off the event
loop. The save response now includes `refreshed_jobs` and
`outdated_job_ids`. Outdated means the theme's `style_params.json` was
written after the job's `theme_applied_at` (or `created_at` if that's
unset).
- **UI:** after a save, the theme editor says how many in-progress
tracks were updated and offers **Re-render all N**. Outdated job cards
get an "Older theme" badge on Re-render. On portal cards, admins see
only the portal Re-render, not the duplicate admin one.
- i18n: 12 new strings, translated into all 33 locales; 3 superseded
keys removed.

## Testing
- `make test` passes (backend unit, emulator and e2e; frontend Jest and
Playwright).
- New `backend/tests/test_theme_change.py` (36 tests, 98% coverage of
the new service), plus a render-worker divert test, editor dialog tests
and OutputLinks tests.
- Local CodeRabbit review: 1 major finding (bulk re-render blocking the
event loop / unbounded), fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 18:32:27 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 6c3c70654d feat(storage): retention purge (dry-run) and on-demand regenerate (v0.265.0) (#1125)
## Summary
Implements the approved storage-retention plan (2026-10-03): purge
regenerable files from jobs completed more than 30 days ago, and
regenerate them on demand.

### Purge (`backend/services/storage_retention.py`)
- Daily Cloud Scheduler `storage-retention-daily` calls `POST
/api/internal/storage-retention/run` (Pulumi, already applied).
- **DRY-RUN by default** (`STORAGE_RETENTION_DRY_RUN`, default true). It
writes a report to
`gs://karaoke-gen-storage-nomadkaraoke/storage-retention/reports/<ts>-dry-run.json`.
Real deletion needs the env flipped after a human review.
- Walks completed Firestore jobs. Completion time is the newest
`complete` timeline entry.
- **Keeps:** input (incl. `edited.flac`), lyrics, style, review
sessions, audio_edit, uploads, packages, analysis, screen images, the
720p final (both copies), `stems/custom_instrumental.*`,
`stems/vocals_derived.*`, and the root custom/existing instrumentals.
Unknown files are kept too.
- **Purges:** other finals, `videos/`, `previews/`, `encoded/`,
`quick/*.mp4`, `screens/*.mov`, `review-audio/`, and separated stems
(only when the input is kept in `jobs/{id}/input/`).
- **Skips:**
  - finalise-only jobs
  - jobs whose input audio is gone (66 old jobs)
- jobs with an in-flight edit, re-render, visibility change or
regenerate
  - pending YouTube uploads
  - excluded tenants (env)
- Real runs:
  - batched, with a resumable cursor
  - re-check eligibility atomically via a claim
- record `renders_purged_at`, `stems_purged_at` and a `storage_purge`
manifest
  - drop purged `file_urls` entries
- `?job_ids=…&dry_run=false&min_age_days=0` runs a scoped real pass, for
testing. A global real purge only happens through the setting.

### Regenerate (`backend/services/regenerate_service.py`)
- `POST /api/jobs/{id}/regenerate` (owner or admin, rate-limited per job
and per user) plus a job-page **Regenerate video** button and progress
notice. There is also `POST /api/admin/jobs/{id}/regenerate`.
- GCS-only: claim, then screens, render, encode. It uses the existing
lyrics, instrumental selection and style. No brand-code allocation, no
YouTube/Dropbox/GDrive/Discord. The owner is notified when it's done.
- **Stems gate** in the screens worker (`stems_restore.py`): if
`stems_purged_at` is set, it runs the audio-separation job in restore
mode first, and the audio worker re-triggers screens afterwards. This
covers regenerate, admin/theme re-render, private→public, Edit (which
now restores stems before review) and retry.
- Public→private on a job with archived finals regenerates first, then
redistributes (no more "No finals" failure).
- Archived downloads return **410** `output_archived` with a
`regenerate_url`.

### Fixes
- Encoder: the with-backing and clean instrumental lookups no longer
fall back to another instrumental (that was a silent wrong-instrumental
bug); a missing instrumental now fails loudly.
- `existing_instrumental_gcs_path` is persisted at upload to
`jobs/{id}/custom_instrumental.<ext>`, because `uploads/` expires after
7 days. Older jobs get a resolve-and-repoint fallback in the video
worker and orchestrator.
- Bucket noncurrent-version retention reduced from 30 to 7 days (Pulumi,
applied).

## Testing
- New: `test_storage_retention.py` (108), `test_regenerate_service.py`
(66), `test_instrumental_resolution.py` (16), plus edit-flow, upload and
visibility updates. Full backend unit suite: 5731 passed.
- Frontend: OutputLinks regenerate tests and job-status helper tests
(Jest). All 33 locales translated.
- Local read-only dry-run against prod: 538 jobs eligible, about 401 GiB
purgeable.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added an option to regenerate archived or missing video outputs from a
completed job. Existing publishing links remain unchanged, and job
progress is shown while processing.
* Added automatic restoration of archived audio stems when needed for
regeneration or job edits.
* **Improvements**
* Added scheduled cleanup of eligible files from older completed jobs
while retaining key source and review assets.
* Improved recovery when requested downloads have been archived by
offering a regeneration option.
* Improved handling of saved instrumentals so they remain available for
future processing.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-03 23:59:23 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 4e43449559 fix(backend): stop API event-loop freezes + identify users in degradation telemetry (v0.264.0) (#1120)
## Summary
The "Reconnecting" pill and the "servers unavailable" banner (164 / 49
events, 2026-09-19 → 10-02) were mostly caused by the single API
instance's **event loop freezing for 20–45s**. Every request on the
instance, `/api/health` included, completed at the same moment. The full
evidence and plan are in
`docs/archive/2026-10-03-backend-loop-freezes-telemetry-plan.md`.

### What froze the loop, and the fix for each
- **Screens worker:** it ran as a post-response `async def`
BackgroundTask doing 4K PIL renders, GCS transfers and ffmpeg transcodes
on the loop. Under `--cpu-throttling` it was also CPU-starved. **Fix:**
it now runs inline in its Cloud Tasks request (CPU stays allocated,
failures still return 200 so Cloud Tasks doesn't double-run it), with
the heavy steps in `asyncio.to_thread`. Also offloaded: style load,
timing signals, pre-apply.
- **Matplotlib font cache:** it was rebuilt on every new instance (17/28
episodes). **Fix:** it's pre-built in `Dockerfile.base`
(`MPLCONFIGDIR=/opt/mplconfig`), and pyplot is imported lazily.
- **Hourly stale-review cron:** an `async def` with zero awaits doing
≤1500 reads and SMTP sends. **Fix:** it runs in a thread.
- **166 `async def` routes that never awaited** (sync
Firestore/GCS/email/Gemini on the loop, e.g. magic-link verify).
**Fix:** they're now plain `def`, so they run in FastAPI's threadpool.
The new AST guard `test_no_blocking_async_routes.py` fails CI on new
ones. It also flags sync routes that call
`asyncio.create_task`/`get_running_loop`; that caught `confirm_duration`
and `trigger_youtube_queue_processing`, which stay async.

### Safety net
- `backend/services/loop_watchdog.py`: a heartbeat plus a daemon thread.
Any stall over 1s logs `EVENT_LOOP_STALL` with the loop thread's stack
and a `culprit=file:line`. Stalls of 5s or more are written to
`client_events` as `server_loop_stall`.
- Pulumi: the log metric `backend/event_loop_stalls` plus an alert on ≥2
stalls of ≥10s in 30 min. **This still needs a local `pulumi up`.**
Pulumi is never applied by CI, and the admin gcloud login has expired.

### Telemetry gaps fixed
- **Identity:** `user_email` used to be hard-coded to null. The frontend
now sends its session token, and the server resolves `user_email`,
`is_admin`, `is_internal` and `is_test`. A client-sent email is ignored.
- **New fields:** `device_fingerprint` (the existing FingerprintJS
visitorId), `tab_id`, `episode_id`, and `tenant`.
- **New event:** `banner_recovered`, carrying `duration_ms` and
`peak_status`.
- **Report:** `scripts/client_events_report.py` summarises the data,
excluding admin/internal/test accounts by default.

## Testing
- `make test` passes: backend 5529, plus the emulator suite, karaoke_gen
unit tests, and frontend Jest 1623.
- New tests:
- Watchdog: a deterministic state machine plus detection of a real
blocking call.
- Event-loop responsiveness: a ticker keeps ticking while the heavy
steps run.
  - Screens inline and failure semantics.
  - `client_events` identity and fields.
  - The AST guard.
  - The report summary.
  - Frontend reporter and banner episode tests.
- Local CodeRabbit CLI review: 0 findings.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

@coderabbitai ignore

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 14:17:51 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 0fa10e4330 feat(frontend): friendly cold-start waking state and backend pre-warm (v0.263.0) (#1110)
## Summary
`karaoke-backend` now runs with min-instances 0, so a request that hits
a cold backend waits ~15-20s (measured 18.4s) while Cloud Run boots an
instance. Cloud Run queues requests during that window, so the health
probe times out exactly as it does during a recycle, and users saw the
amber "Reconnecting…" / "having trouble reaching our servers" banner.
This replaces that with a calm cold-start experience and pre-warms the
backend.

### Behaviour (`frontend/lib/backend-status.ts`)
- New `waking` status. **Cold-start rule:** if no backend response
succeeded in the `REACHABLE_FRESH_MS` (10 min) before the oldest stalled
read began (fresh page load, or a tab idle long enough for
scale-to-zero), a stall is treated as a cold start:
- probe `/api/health` after `COLD_PROBE_AFTER_MS` (1s) instead of 10s,
so a normal warm load (reads finish in <1s) never sends an extra probe;
- once the read is ≥3s old **and** the probe failed (4s timeout), show
the waking card (~5s into a cold start);
  - it clears the instant reads complete.
- **Escalation:** past `WAKING_ESCALATE_MS` (60s) the normal
reconnecting/unavailable logic takes over. A stall that starts within 10
min of a successful response is a recycle/outage, so the existing banner
applies unchanged.
- The probe still gates everything: a warm backend with a merely slow
endpoint shows nothing.
- `apiFetch` marks the backend reachable on every non-transient
response, for any method. GET hard timeout (45s) and retry logic are
unchanged and already ride out a ~20-30s cold start (the queued request
simply completes).

### Pre-warm
- `prewarmBackend()`: an untracked, fire-and-forget `GET /api/health`
(`keepalive`, `cache: no-store`, no custom headers, so no CORS
preflight). It runs on app load via the global `BackendStatusBanner`
(all `[locale]` pages) and the admin layout, and again when a tab
becomes visible after >10 min hidden. It's skipped if the backend
answered in the last 10 min. Success marks the backend reachable.
- Companion PR nomadkaraoke/public-website#31 pre-warms from the
marketing homepage Hero (no-cors, idle-time).

### UI
- Waking card: neutral card colours plus a tenant-primary spinner.
"Starting up our karaoke servers…" / "This usually takes about 20
seconds. Thanks for your patience!". No warning colours and no dismiss
button.
- New `banner_waking` degradation event, added to the frontend type and
the backend `client-events` allow-list. It includes
`since_reachable_ms`.
- Strings translated for all 33 locales (Gemini Developer API path from
#1108).

## Testing
- Store unit tests with fake timers. Cold start → waking (~5s) → clears.
Fast cold read shows nothing. Warm slow endpoint shows nothing.
Prolonged stall → `unavailable` after 60s. Previously reachable then
stall → `reconnecting`. Reachable >10 min ago → waking. Pre-warm
de-dupe, failure and visibility re-warm are covered.
- apiFetch tests: a fresh session hitting a cold backend shows `waking`
(never the warning) and the GET completes at 18s. The pre-warm request
shape (no headers, keepalive) is checked.
- Backend: `banner_waking` accepted by `/api/client-events`.
- `make test` passes.
- Real browser check: `next dev` against a local stub that holds every
request for 18s from the first hit. The waking card appears by ~7s, the
dashboard shows "Loading jobs…" (no error or empty state) and everything
resolves at 18.3s.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added a startup status banner for cold starts, with an estimate that
startup usually takes about 20 seconds. The banner is available in
multiple languages.
* The app now checks backend availability when needed and after a tab
has been hidden for an extended period, helping requests recover from
cold starts.
* **Documentation**
* Updated API telemetry documentation to include the new startup event.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 22:27:26 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 8490ab75eb chore(gemini): move every Gemini call from Vertex AI to the Gemini Developer API (#1108)
## Summary
Moves **every** Gemini call in gen off **Vertex AI** (billed to the
`nomadkaraoke` GCP project, ~$85/mo) onto the **Gemini Developer API**
using the `gemini-api-key` Secret Manager secret (an AI Studio key
billed outside this project). Once deployed and verified,
`aiplatform.googleapis.com` / `generativelanguage.googleapis.com` get
disabled in nomadkaraoke.

### Key injection
- `backend/services/gemini_client.get_genai_client(timeout_ms)` — key
from `GEMINI_API_KEY`, else Secret Manager `gemini-api-key` via
`settings.get_secret` (runtime SA already has project-level
`secretAccessor`). No `project`/`location`.
- Service: CI `--set-secrets ... GEMINI_API_KEY=gemini-api-key:latest`.
- Jobs: Pulumi secret env on `lyrics-transcription-job`,
`video-encoding-job` (renders + translations), `nomad-error-monitor` —
**already applied** (targeted `pulumi up`, only that env added). Other
jobs fall back to Secret Manager.
- Local scripts (`frontend/scripts/translate.py`,
`scripts/measure_autocorrect_cost.py`,
`scripts/test_agentic_correction.py`): env, else `gcloud secrets
versions access` (in-process only).
- **Rotation:** add a new secret version; new instances / job executions
pick it up. No code change.

### Degradation (quota / billing / invalid key; per-minute 429s stay
retryable)
| Caller | Behaviour |
|---|---|
| auto-correct | non-retryable 503 → compare mode drops the Gemini leg,
keeps Opus |
| match-judge / free-text / parse-titles / tenant bulk | existing
fallbacks (no suggestion / blanks / regex) |
| credit eval | `pending_review` (fail-closed, unchanged) |
| lyrics translation | no retry; render proceeds without translation,
status recorded |
| custom lyrics | clean 503 "temporarily unavailable…" (+ 5-min timeout
so it can't hang) |
| error-monitor LLM | no retry; alerts sent without AI summary |
| translate.py | exit 2 "Gemini quota/credit exhausted — top up AI
Studio or rotate gemini-api-key" |

`note_gemini_failure()` also sends one Discord ops alert per 6h
(Firestore-throttled across instances via new
`ops_alerts.send_throttled_ops_alert`).

### Infra
- Pulumi: removes `roles/aiplatform.user` from `karaoke-backend` —
**will be applied after this deploys and Vertex traffic is confirmed at
zero**.
- Agentic correction default model `gemini/gemini-3.8-flash`
(`vertexai/` prefix still accepted, routed to the Developer API).
- Version 0.261.1 → 0.262.0. Docs: CLAUDE.md, LESSONS-LEARNED, README,
ARCHITECTURE, API, etc.

## Testing
- New: `backend/tests/services/test_backend_gemini_client.py`
(classifier, key resolution, no-Vertex client kwargs, alert throttle,
per-caller degradation),
`tests/unit/lyrics_transcriber/test_agentic_gemini_provider.py`,
`frontend/scripts/test_gemini_client.py`, lyrics-translation quota test.
- `pytest backend/tests --ignore=backend/tests/emulator` — all pass;
`pytest tests/unit` — 2021 passed.
- Verified `gemini-3.8-flash` is listed on the Developer API and
answered a 5-token test call with the key.
- `translate.py --dry-run` on `frontend/messages` and
`backend/translations` OK.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Updates**
* AI-powered lyric translation, custom lyrics, auto-correction, title
parsing, match judging, and related features now use the Gemini
Developer API.
* AI features use the configured Gemini API key. Quota, billing, or key
issues may show an unavailable message or use an existing fallback
instead of repeated attempts.
* **Documentation**
* Updated product, API, setup, and recovery guidance to reflect Gemini
API key configuration and usage.
* Updated guidance for handling AI service availability and quota
issues.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 21:58:53 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 f8d0febcf5 Translated lyrics beneath the karaoke lyrics for language learners (v0.260.0) (#1101)
## Summary
New job-submission option: **Add translated lyrics** → pick a language
(33 UI locales) → the karaoke videos show an AI translation of each
lyric line in smaller text beneath the synced lyrics, so language
learners can sing songs in another language while learning what the
words mean.

- **Job creation**: toggle + language picker (default = UI language) on
the Customize step, with a **live preview** of a real rendered frame
(`GET /api/themes/translation-preview?language=xx`; pre-generated sample
translations, no LLM call at preview time). Any job, public or private,
free for now. `translation_language` accepted by every creation route;
admin can edit it.
- **Post-review translation**: right after the job enters
RENDERING_VIDEO, the render worker translates the *final reviewed*
lyrics line by line with Gemini
(`backend/services/lyrics_translation.py`, `LYRICS_TRANSLATION_MODEL`,
GCS-cached so theme/admin re-renders are free) into
`jobs/{id}/lyrics/translations.json`. Not editable. A failure never
blocks the render (renders without translations, recorded in
`state_data.lyrics_translation`); a song already in the target language
gets no row.
- **Rendering** (landscape + portrait, GCE worker + local fallback):
`LyricsSegment.translation`; when present, slots get taller and screens
show 3 lines; a `Karaoke.Translation` style (unsung colour, ~½ size)
draws one static event beneath each line, sanitized, wrapped/scaled to
fit. Jobs without translations render identically.
- **RTL + LTR together**: the translation is its own ASS event with auto
paragraph direction, so e.g. a Hebrew song keeps its right-to-left
highlight with an English translation beneath (and vice versa). Long
lines split by the resizer share their translation across the pieces
(grapheme-safe for Thai/CJK).

Design: `docs/archive/2026-10-01-translated-lyrics-plan.md`.

## Testing
- `make test` all green (root 2014, backend 5416, emulator 150, frontend
1583).
- New: `tests/unit/lyrics_transcriber/output/test_translated_lyrics.py`
(layout/positions/style/sanitizing/fit/mixed RTL-LTR/Thai/portrait),
`test_translated_lyrics_render.py` (libass pixel tests incl.
Hebrew/Arabic lyrics with English/Arabic translations and CJK),
`backend/tests/test_lyrics_translation.py`,
`backend/tests/test_render_video_worker_translations.py`, GCE endpoint
test, frontend CustomizeStep/GuidedJobFlow/api tests, prod E2E
`e2e/production/translated-lyrics.spec.ts`.
- `scripts/run-render-tests-linux.sh`: 135 passed on Debian ffmpeg AND
prod's static ffmpeg (no skips).
- Real-data check: job 5710831e (Hebrew) translated to English by Gemini
and rendered with the nomad theme — RTL highlight intact, English
beneath, split line shares translation.

Local CodeRabbit CLI unavailable in this workspace (no seat), so leaving
this PR for the CodeRabbit bot; an independent local review was done and
its findings fixed in the second commit.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added an opt-in translated-lyrics option to job setup, with language
selection and a preview.
* Translations are generated after lyric review and appear beneath
lyrics in landscape and portrait videos, with text layout for multiple
languages and writing directions.
  * Translation settings can be changed or cleared by admins.
* **Bug Fixes**
* If translation fails or is unavailable, video rendering continues
without translated lyrics.
* **Documentation**
* Updated API and product documentation with translated-lyrics setup and
behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 13:41:54 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 f89de165cc Admin: re-render any completed job without review (v0.257.0) (#1096)
## Summary

An admin-only **Re-render** button on completed jobs in the `/app` jobs
list. It regenerates title/end cards, the karaoke video, all encoded
formats, packaging and distribution from the job's existing reviewed
lyrics, instrumental selection and style snapshot, **with no
lyrics/instrumental review**. Motivating case: re-rendering
already-delivered videos after a renderer fix (RTL/Hebrew, v0.255.0).

`POST /api/admin/jobs/{job_id}/rerender` with body `{"notify_customer":
false}`.

**Behaviour (decided with Andrew)**
- Works for consumer and tenant jobs, public or private. Uses the job's
existing style; no theme re-snapshot.
- **Published outputs are deleted up front, like Edit (YouTube, GDrive,
Dropbox), but the brand code is kept, not recycled.** The pipeline
re-publishes under the same NOMAD-xxxx. YouTube gets a new upload and
therefore a new URL.
- Outputs are only deleted for destinations the re-render will actually
re-publish (`plan_republish` uses the same conditions as the video
worker). Anything else is kept, with a warning shown to the admin.
- **No customer email/push/Discord by default.** The confirm dialog has
an "Email the customer when done" checkbox. Already-notified
requests-board voters aren't re-emailed; voter emails still owed from
the original publish do go out.
- A failed admin re-render is set to FAILED with an in-progress marker
(scoped to `review_token`, so it can't leak into later runs). It can be
retried by **admins only**; `/retry` returns 403 to customers while the
marker is active.
- **Mutual exclusion with the YouTube upload queue:** the claim and the
queue's `mark_processing` are Firestore transactions that each read both
documents. A re-render is refused with 409 while an upload is in flight.
A queued or failed deferred upload is cancelled atomically when YouTube
will be re-published. The queue never starts an upload for a job that is
mid re-render.
- After a successful upload, the queue never re-queues an entry, so a
later error can't cause a duplicate upload.
- Admin re-render completions don't increment the customer's
completed-jobs count.

**Refactors**
- New `backend/services/published_outputs_cleanup.py`, shared by Edit,
admin delete-outputs and the re-render.
- The Dropbox folder delete now uses the uploader's sanitised name and
also tries the legacy raw name. This is a deliberate fix for titles with
special characters.
- Remaining duplicate copies (`jobs.py` ~2440,
`visibility_change_service`) are listed as follow-ups in the design doc.

## Testing
- 173 tests in `test_admin_rerender.py` +
`test_published_outputs_cleanup.py`, built through three local
code-review rounds. They cover:
  - claim and validation
  - retry gating
  - marker scoping
  - publish planning (each mismatch case)
- queue exclusion in both directions, no starvation, and that the
re-render's own later upload is never cancelled
  - the no-re-queue-after-upload rule
  - notification gating on both pipelines
  - community voter cases plus reconcile
  - restart atomicity
  - legacy Dropbox naming
  - tenant re-render and Edit unchanged
- Frontend Jest: button visibility (admin vs non-admin, complete vs
not), dialog, payload, and warnings toast.
- `make test`: backend 1972 + 5372 + 150, frontend 1506.
- i18n: new strings in all 33 locales (frontend) plus backend
translations.

Docs: `docs/API.md`,
`docs/archive/2026-10-01-admin-rerender-any-job.md`.

No local CodeRabbit review (the CLI has no seat for this org), so this
is left open for the CodeRabbit bot. Three local code-review rounds ran;
their findings are addressed in the fix commits.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Administrators can re-render eligible completed tracks without another
review, retaining existing track settings and brand code.
* Choose whether to notify the customer; notifications are off by
default. Published outputs are replaced when applicable, and warnings
identify outputs that remain unchanged.
* Administrators can resume eligible failed or cancelled re-renders.
Re-render controls and status messages are available in multiple
languages.
* **Bug Fixes**
* Improved cleanup and handling of published files and queued YouTube
uploads during re-renders, with safer retries and clearer failure
reporting.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 23:37:51 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 e2f7dbd13c Detect sung stretches with no transcribed lyrics (shadow) + backfill endpoint (v0.256.0) (#1095)
## Summary

Transcription can silently **drop whole sung lines**. On job 5710831e
(Hebrew), the words stop at 20.66s and resume at 32.40s while the lead
vocal is ~96% active, and both reference sources have three pre-chorus
lines there. `SectionDetector` (gap ≥ 10s, no audio check) then rendered
"♪ INSTRUMENTAL ♪" over the singing. Nothing caught it:
- the scorer's anchor coverage measures how much of the *transcribed*
text matches the reference, so omissions don't lower it (98.8% here);
- `timing_check` G3 (`max_unclaimed_run_s`) does fire on this job but is
shadow-only and only computed for AUTO-bound jobs;
- the review UI shows nothing at the gap.

This PR is **phase 1: a shadow signal plus server-side backfill for
calibration**. It changes no verdicts or rendering.

- `backend/services/auto_approval/vocal_gaps.py`: for every
transcription gap ≥ 3s, records the lead-vocal active fraction, the
longest breath-bridged vocal run (with a 1.5s allowance for a held note
carried over from the previous word), and the reference lines between
the neighbouring words' reference positions. Section headers are dropped
and ±1-word anchor drift at line boundaries is snapped. `suspect` = run
≥ 3s.
- The executor analyzes every job once stems exist (any verdict), off
the event loop, and stores `state_data.vocal_gaps` keyed by an
`input_key` fingerprint so resets/edits recompute. A summary goes in
`processing_metadata.auto_approval.vocal_gaps`.
- `POST /api/internal/jobs/{id}/vocal-gaps[?dry_run=true]` (admin) plus
`scripts/audit_vocal_gaps.py` for backfill/calibration. Cloud Run does
the stem work.

**On the real job:** one suspect gap, 20.66–32.40s (10.26s run after the
allowance), whose reference lines are exactly the three missing Genius
lines. The three genuine instrumentals have ≤ 0.16s runs.

## Testing
- `test_vocal_gaps.py` (25 tests, synthesized audio + synthetic
correction data):
  - dropped lines vs instrumental vs ad-lib vs held note
  - a short dropped line diluted by a long gap
  - breath bridging
  - silent stem = error
  - malformed reference data
- reference snapping (off-by-one, single-word line, headers, reversed
chorus mapping)
- Executor tests: stored without changing the outcome; analyzed for
review-bound jobs; cache hit only for the same `input_key`, stale
recompute; pending audio / no stem / error / exception all non-fatal;
backfill uses the reviewed lyrics + original references; dry run.
- Internal endpoint tests (store / dry_run / admin).
- Full backend suite: 5199 passed.
- A local code review's findings were addressed in the second commit:
event loop, stale cache, silent stem, held notes, reference robustness,
audit ordering.

## Next (after deploy)
Run the audit → hand-check suspects → choose thresholds → phase 2:
scorer REVIEW gate with an explicit reason, no fake INSTRUMENTAL over
vocals, review-UI marker with one-click insert of the reference lines.
Also noted: anchor reference ids look shifted by one word at `[Section]`
header boundaries. Separate issue, documented in the plan doc.

No local CodeRabbit review (the CLI has no seat for this org), so this
is left open for the CodeRabbit bot.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added vocal-gap analysis to identify stretches of missing lyrics that
overlap with lead vocals. Results are saved with job data and can be
requested through an admin-only internal API.
* Analysis also runs automatically during auto-approval when vocal stems
are available. It does not change job status or approval decisions.
* **Documentation**
* Documented the internal API, its dry-run option, and the analysis
result statuses.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 22:10:47 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 e15a485007 Re-render: refresh tenant Dropbox archive in place instead of rejecting (v0.254.1) (#1093)
## Summary
Follow-up to #1091, found by testing it in prod. randy-vild jobs are
archived to Dropbox under a brand code (`RVILD-0001 - Randy Vild - What
Goes Up`). The "never distributed" guard therefore rejected the very
jobs the Re-render button is for (`POST /api/jobs/2579a1ea/rerender` →
400 "published to YouTube/Dropbox/Drive").

- Dropbox-archived tenant jobs can now be re-rendered. The re-render
stores the job's `brand_code` in `state_data.theme_rerender`, and the
orchestrator config and native distribution use it as `keep_brand_code`.
The same folder is overwritten (Dropbox `WriteMode.overwrite`) and no
new code is allocated.
- The code is scoped to the re-render marker (cleared on success), not
written to the job's `keep_brand_code` field. That way a later Edit,
which recycles the brand code, can't reuse it.
- YouTube/GDrive-published jobs are still rejected. The frontend hides
the button only when a YouTube link shows.

## Testing
- Backend 5158 passed, 36 skipped; `test_theme_rerender.py` 35 cases,
including Dropbox-archived jobs being re-renderable, the marker
recording the brand code, and the orchestrator keeping it.
- Frontend `OutputLinks.test.tsx` 28 passed.
- CodeRabbit CLI isn't available on this org, and the bot has been
rate-limited. This is a small change on top of the reviewed #1091.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 01:37:33 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 9085a3005e feat(tenants): configure Dropbox output folder in tenant setup (v0.254.0) (#1092)
## Summary
The Randy Vild tenant's completion email said "Here's the dropbox
folder…" then `[Dropbox URL not available]`: the tenant was created with
the (optional) Dropbox path left blank, so every job was download-only.
The admin form already had the fields, but nothing guided, validated, or
provisioned them.

### Changes
- **Tenant create/update** (`tenant_admin_service`): `dropbox_path` +
`brand_prefix` must be set together (the worker silently skips upload
otherwise); path normalized, prefix uppercased/validated (2-12 alnum)
and rejected if it's the consumer default (`NOMAD`/`NOMADNP`) or another
tenant's (brand-code counters are keyed by prefix alone). The Dropbox
folder is **created up front**; `features.dropbox_upload` always follows
the path.
- **DropboxService**: `ensure_folder()` (idempotent); `list_folders()`
treats a missing path as empty — previously the *first* brand-code
allocation for a new tenant would raise `not_found`.
- **Admin UI** (`/admin/tenants`): pre-fills
`/MediaUnsynced/Karaoke/Tracks-<Name>` + prefix from the name (Vocal
Star→VSTAR, Singa→SINGA, Randy Vild→RVILD), blocks half-set delivery,
warns that download-only tenants get no email link; table shows `PREFIX
→ path`.
- Update route maps conflict→409, Dropbox provisioning failure→502.

### Prod data fix (already applied, outside this PR)
- `randy-vild` tenant → `/MediaUnsynced/Karaoke/Tracks-RandyVild`,
`RVILD`; `brand_code_counters/RVILD` seeded at 3.
- 10 in-review Randy Vild jobs backfilled with
`dropbox_path`/`brand_prefix` (jobs snapshot these at creation).
- 2 completed jobs copied into `RVILD-0001 - Randy Vild - What Goes Up`
/ `RVILD-0002 - Randy Vild - Embers in the wind` (24 files each via
Dropbox `save_url` from signed GCS URLs),
`state_data.brand_code`/`dropbox_link` set.

## Testing
- `make test`: backend 5121 + 1846 + 150 passed, frontend 1476 passed
- New unit tests: tenant delivery validation/folder creation/prefix
conflicts, DropboxService ensure_folder + missing-path listing (real SDK
error types), admin page pre-fill + half-set block + download-only
warning, `suggestDelivery`/`deliveryError`.
- Local CodeRabbit review: 2 minor findings, both fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 01:22:19 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 31a8838b07 Tenant: re-render tracks with the current theme; edited tenant tracks keep their downloads (v0.253.0) (#1091)
## Summary
- **Bug: finished tenant track showed no download buttons** (randy-vild
job 2579a1ea). The Edit flow sets `outputs_deleted_at`, and it was only
cleared when the re-render uploaded to YouTube/Dropbox/GDrive. Tenant
jobs never distribute, so every download stayed hidden after the
re-render. A customer's private job (6452888e) had the same problem for
a month. Every successful render now clears the flag
(`video_worker.py`). Both prod jobs were cleared by hand.
- **New: "Re-render" button** on finished tenant-portal tracks (`POST
/api/jobs/{id}/rerender`). It re-applies the tenant's **current** theme
and rebuilds the title/end screens, lyrics video, all encodes and
CDG/TXT. It reuses the reviewed lyrics and instrumental selection, so
there's no review step. Same mechanism as the private→public visibility
flow (`regen_restore_status=review_complete` → screens → render →
encode). Owner (still on the tenant allowlist) or admin. Free. The theme
editor now points to the button.
- **Encoding-worker cache:** the worker caches finished jobs by id, so a
re-run submitted while the VM was still up could get the previous render
or encode back. Render and encode now submit as
`{job_id}_g{worker_generation}`.
- **Old title/end cards:** the encoder's `**/*Title*.mov` lookup would
have picked up the old `finals/… (Title).mov`. The re-render now deletes
the MOVs in `finals/`.
- **Failed re-render:** it can be re-run. Retry re-runs it if the new
screens weren't made yet, and otherwise resumes from render/video as
usual.

## Testing
- Backend: 5128 passed, 36 skipped. New `test_theme_rerender.py` (31
cases: validation, claim/conflict, artifact cleanup incl. finals MOVs,
trigger rollback, route auth/allowlist, retry branch, generation-keyed
ids) and `test_video_worker_outputs_deleted_clear.py` (fails without the
fix).
- Frontend: `OutputLinks.test.tsx` 28 passed (button visibility, confirm
→ API → refresh). Strings translated to all 33 locales.
- Review: CodeRabbit CLI isn't available on this org. An independent
reviewer agent went over the diff: 1 must-fix (stale finals MOVs) fixed;
the failed-re-render recovery fixed.

## Not done (follow-ups)
- Re-render re-sends the completion email and push notification (same as
the visibility flow).
- No per-tenant rate limit. One re-render per job at a time is enforced
by the status claim.
- The status briefly shows "Lyrics ready" before "Generating screens".

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 01:06:54 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 c831bd5e89 feat(uploads): line up mismatched user instrumentals + tenant bulk upload modal (v0.251.0) (#1087)
## Summary
A Randy Vild tenant bulk upload lost 3 of 10 tracks to the 0.5s
mix/instrumental duration check, and the bulk UI showed only
"Submitting…" while uploading. All three rejected tracks were valid
edits of the same song:

| Track | Difference | Now |
|---|---|---|
| Keep Smiling | instrumental has 2.97s extra lead-in | start trimmed →
matches mix exactly |
| Simulation | instrumental 6.5s longer outro | cut at mix end with 2s
fade |
| Faithful and True | instrumental stops 4.45s early | end padded with
silence |

## Changes
**Backend: conform instead of cancel**
(`backend/services/instrumental_conform.py`, `uploads-complete`)
- On a >0.5s mismatch, the start offset is found by FFT
cross-correlation (reusing `derived_vocals.find_offset`) at **25% and
75%** of the song, searching ±15s. The start is then trimmed or padded,
and the end is faded and cut or padded to the mix length. All of this is
one ffmpeg pass on the original file, keeping its sample rate and bit
depth, and written as FLAC.
- Stored at `uploads/{job}/conformed/existing_instrumental.flac`, which
becomes `existing_instrumental_gcs_path`. The original upload is kept,
and details go in `state_data.instrumental_conformed`.
- Still rejected (job cancelled and refunded, 400):
- correlation < 0.3 (wrong file; real pairs score 0.78–0.90, wrong pairs
about 0.01);
- offsets at 25% and 75% differ by more than 20 ms (edited differently
mid-song).
- Conforming runs fully off the event loop, with at most 2 at a time per
instance. GCS or ffmpeg failures return a retryable 503 instead of
cancelling the job.
- The orchestrator now stages the recorded
`existing_instrumental_gcs_path` for the GCE encoder, so the conformed
copy is what gets rendered.

**Frontend**
- Tenant bulk shows `UploadProgressModal` for the whole batch: total
bytes, speed and ETA, tracks submitted and failed, and an offline or
retrying notice. The modal gained optional `title`, `detail` and
`notice` props and uses the tenant's primary colour.
- Bulk retry bug:
- **Before:** a server-cancelled job kept its ID. The retry's 400 ("not
pending") was read as "already processing", so the row showed
**Submitted** for a dead job.
- **Now:** any 400 checks the job's status. A processing job counts as
done. A cancelled or failed job is dropped, so the retry creates a fresh
job. Otherwise the row keeps the job for retry, so no duplicate is
created.
- The single-track tenant form and the own-instrumental field now check
only the size cap in the browser; the server handles length differences.

## Testing
- Backend: 5086 passed (new: synthetic lead-in, long/short outro, late
start, wrong-file and mid-song-edit cases; route and helper guards;
orchestrator staging). Frontend jest: 1459 passed (new: batch modal
progress and offline notice, failed rows, retry and no-duplicate cases).
- Ran `conform_instrumental` on Randy's real files:
  - offsets at 25% and 75% agree to the sample;
  - output is exactly the mix length at 24-bit / 44.1 kHz;
  - after conforming, each file lines up with its mix at 0 offset;
  - a wrong pairing is rejected (correlation 0.02).
- Real frontend (`next dev`, randy-vild tenant, mocked API, slowed
upload server), 8 tracks / 192 MB: the modal showed progress, track
counts, speed and ETA, then closed onto the "All tracks submitted"
screen.
- Review: an independent reviewer agent found 1 must-fix and 4
should-fix issues, all addressed in the last commit. The CodeRabbit CLI
has no seat here, so this PR is left open to the CodeRabbit bot.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Uploaded instrumentals with different durations can now be aligned and
adjusted to match the mix when the audio matches.
* Bulk uploads now display overall progress, completed and failed track
counts, connection updates, and retry guidance.
* Failed bulk-upload tracks can be retried, with clearer handling for
submissions already in progress.

* **Bug Fixes**
* Instrumental duration checks now occur during upload processing
instead of blocking the upload in the browser. Files that cannot be
matched are reported as a mismatch.

* **Documentation**
* Updated upload instructions and related messages across supported
languages.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 23:42:08 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 f4eca80551 feat(audio-download): stalled torrent → 'Keep trying?' prompt (v0.250.0) (#1085)
## Why
Job 64be8a87 (a Fiverr order): the only seeder of a 1-seeder RED single
was offline. The torrent sat at **0 peers through 5 re-announces for 10
min**, flacfetch aborted, the job showed a raw error, and a Discord "Job
failed" alert fired even though a Cloud Run auto-retry was already
queued. That retry got the file in 17s. Andrew's ask: 10 min is too
short, and a stalled download should explain itself and offer "Keep
trying?" for up to another hour.

## What
- **Longer first wait:** 20 min with no progress (was 10), sent to
flacfetch per request as `max_stall_seconds` (needs flacfetch v0.31.0,
nomadkaraoke/flacfetch#51, already deployed to flacup). The worker's
wait and the Cloud Run **per-execution timeout override** come from the
same budget (`backend/services/audio_download_limits.py`), so the job's
Pulumi config is untouched.
- **Stall → user prompt, no silent retry:** flacfetch now reports
`error_code: torrent_stalled` (older versions are detected from the
error text). The worker fails the job with `error_details = {stage:
audio_download, code: audio_download_stalled, stall_minutes,
keep_trying}` and exits 0, so Cloud Run doesn't retry the same torrent.
Other download failures keep their auto-retries.
- **Job card:** `StalledDownloadNotice` replaces the raw error with an
amber explanation and two buttons:
- **Keep trying (up to 1 hour)**: `POST /api/jobs/{id}/retry
{"keep_trying": true}` (the body is optional and backward compatible).
This gives a 3600s stall budget and a 1h25m execution timeout.
- **Choose different audio**: new `POST
/api/jobs/{id}/choose-different-audio` (owner-checked). It moves FAILED
to `awaiting_audio_selection` (new transition) using the saved search
results and opens the Select Audio dialog.
- The generic Retry button is hidden for stalled jobs.
`error_details.{code,stage,stall_minutes,keep_trying}` were added to the
summary fields. Strings are translated into all 33 locales.
- **Heartbeat:** while waiting on a torrent, the worker writes
`state_data.audio_download_progress` (progress, peers) every 2 min.
Without this, recover-stuck-jobs (which parks a job with no update for
10 min) would park a live 20–60 min wait and start a duplicate download.
- **Ops alert:** the alert is skipped while a Cloud Run auto-retry is
pending (the final attempt clears the marker and alerts then). Stall
alerts get a 🟠 "Audio download stalled" headline.
- Docs updated: API.md, TROUBLESHOOTING.md, ARCHITECTURE.md.

## Tests
- `make test`: backend 1846 + 5078 + 150 passed; frontend 1451 passed.
- New tests:
- `test_audio_download_limits.py`,
`test_routes_jobs_stalled_download.py` (real HTTP via TestClient),
`test_worker_service_audio_download_timeout.py`.
- Stall, keep-trying, heartbeat and `main()` exit-code cases in
`test_audio_download_worker.py`.
- Stall detection and payload in `test_flacfetch_client.py`;
retry-pending suppression in `test_ops_alerts.py`.
- Frontend: `StalledDownloadNotice.test.tsx`, plus JobCard/JobActions
cases.

## Notes
- Credits are unchanged: a failed job is still auto-refunded and retries
don't re-charge. That's existing behaviour for every retry.
- A local CodeRabbit review couldn't run (the CLI seat is on another
org), so this PR has no ignore line and the GitHub bot will review it.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Failed audio downloads now offer options to keep trying for up to an
hour or choose a different audio source.
* Retry progress and clearer stalled-download guidance are available in
the job view, with messages translated across supported languages.

* **Bug Fixes**
* Stalled downloads can be recovered without triggering unnecessary
automatic retries or duplicate failure alerts.
* Choosing different audio returns eligible failed jobs to audio
selection.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 21:51:50 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 80f6d34007 feat(tenant): self-service theme editor with exact server-rendered previews (#1080)
## Summary
Tenant portal users (anyone on the tenant's allowlist) can now edit
their tenant's theme from **Theme & style** in the user menu.
- **Guided form:**
- backgrounds (upload or solid colour) for the title card, lyrics video
and end screen
  - font (bundled or uploaded TTF/OTF)
  - title and artist colours, positions and sizes
  - sung, upcoming and outline lyric colours
  - lyric size, top position and max line length
  - the end-screen message
- **Advanced:** validated full-theme JSON.
- **Exact previews:** the title card and a lyrics-video frame (first
line half-sung) are rendered **by the production renderers** from the
unsaved draft, in about 2 s, debounced. The plan is in
`docs/archive/2026-09-29-tenant-theme-editor-plan.md`.
- **Save** applies to jobs created afterwards.

## Backend
- New router: `GET/PUT /api/tenant/theme`, `POST
/api/tenant/theme/assets`, `POST /api/tenant/theme/preview`.
- **Membership guard:** `X-Tenant-ID` is client-controlled, so every
call re-checks the tenant allowlist (admins always pass). Non-members
and spoofed headers get 403. The tenant bulk router only checks that the
tenant exists, so this pattern wasn't copied from there.
- `theme_preview_service`:
  - title card via `VideoGenerator.create_title_video(duration=0)`
- karaoke frame via `SegmentResizer` → `SubtitlesGenerator.generate_ass`
→ one ffmpeg frame, using `setpts` to jump straight to time T and
scaling inside the same pass (11.8 s → 0.66 s)
  - bounded concurrency, and a cache for identical drafts
- `tenant_theme_service`, stricter than the admin console:
- asset fields must be basenames that exist in the theme (no paths or
`gs://`)
  - bounds, colours and regions are validated
- **font consistency with the render pipeline:** the single `font` asset
is applied everywhere, `karaoke.font` is set to the TTF family name
libass matches, and a chosen bundled font is copied into the theme's
assets
  - required ASS keys are filled in
- uploads are content-addressed, so they never overwrite an asset that
live jobs use
  - tenants can't change their config or allowlist through this API

## Frontend
- `components/tenant-theme/TenantThemeEditorDialog` and `theme-utils`.
- `tenantThemeApi` in `lib/api.ts`.
- A menu item in `AuthStatus`, shown only on tenant portals.
- `tenantTheme.*` strings in all 33 locales.

## Tests
- **pytest:**
- real renders (the background colour appears in the frames; only the
karaoke frame changes when the sung colour changes; caching)
- sanitisation (paths, un-uploaded assets, bounds, colours, regions,
sections)
- uploads (content addressing, `no-store`, rejecting non-images and bad
fonts)
  - save (font family, bundled font copy, no config changes)
- the route guard for spoofed, other-tenant, inactive and no-tenant
cases, on every endpoint
- **Jest:** load, debounced preview, hex→rgba conversion, upload, region
sliders, Advanced JSON validation, save, discard, the unsaved-changes
guard, and preview errors.
- **Prod E2E:** `e2e/production/tenant-theme-editor.spec.ts`.
- Results: backend 4998 passed; frontend 1434 passed.

No local CodeRabbit review was run (no CLI seat in this org), so this PR
is left open for the GitHub bot.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added a theme editor to the signed-in menu, with controls for title
cards, karaoke videos, end screens, and advanced settings.
* Preview theme drafts, upload images and fonts, and save or discard
changes. Saved themes apply to future videos.
  * Added translations for the editor across supported languages.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 03:18:24 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 518126727b feat(tenant): tenant users never see or use consumer credits (#1076)
## Problem
A tenant user's first login on `randy-vild.nomadkaraoke.com` showed the
consumer "We weren't able to grant free credits" screen with **Buy
Credits**. The backend also sent a credits-denied email. Tenant jobs
were being charged consumer credits too, so the client's first submit
would fail with "out of credits". Tenants are companies or bulk clients
on a separate commercial agreement.

## Changes
- **Sign-in:**
- Tenant magic links skip the whole welcome-credit flow: no AI eval, no
denial or welcome email, no referral attribution. `credit_status` comes
back as `not_applicable`.
- Sending a tenant magic link no longer precomputes a credit evaluation.
- **Jobs:** `create_job` never credit-checks or charges jobs with a
`tenant_id`. They're marked `payment_bypassed`, so duration
reconciliation never pauses them for credits and refunds never mint
credits.
- **Access:** an empty tenant allowlist now means **admins only**. Free
jobs must never be open to arbitrary sign-ups. All 3 live tenants
already have allowlists. The admin UI copy is updated.
- **Frontend:** on tenant portals, the credits badge and menu, Buy
Credits, and Referrals are hidden. The verify page never shows credit
screens for tenant logins.

## Tests
- `test_tenant_no_consumer_credits.py`: a tenant job isn't charged and
is marked `payment_bypassed`; consumer jobs are still charged and still
rejected without credits.
- Verify route: a tenant login returns `not_applicable` and the grant is
never called.
- Model: an empty allowlist means admins only.
- `AuthStatus`: tenant mode hides credits.
- Results: backend 4953 passed; frontend 1423 passed.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Access**
* Tenant portals with no configured email addresses or domains are
restricted to administrator sign-ins. Configured allowlists continue to
limit access to listed users and administrators.
* **Tenant Portal**
* Tenant sign-ins no longer receive consumer welcome-credit evaluations
or related credit emails.
  * Tenant jobs bypass consumer credit checks and charges.
* Credit balances, credit-purchase options, and referral links are
hidden in tenant portals.
* Successful tenant sign-ins proceed to the app regardless of credit
status.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 12:50:28 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 fe217fda1f kjbox singer search: Gemini fallback with the job flow's catalog tidy + descriptive queries (v0.246.2) (#1065)
## Summary
The Gemini fallback for kjbox's singer song identification (kjbox
`docs/SONG-IDENTIFICATION.md`). kjbox's on-device matcher answers most
searches; this endpoint handles what it can't.

- `POST /api/kjbox/catalog/resolve` now runs `resolve_and_tidy`:
1. One Gemini call (`gemini-3.8-flash`) splits the singer's one-line
query into artist/title and fixes typos.
2. It now also resolves **descriptions**: "that song from titanic" →
Céline Dion — My Heart Will Go On; "is this the real life" → Queen —
Bohemian Rhapsody; "song from top gun" → ambiguous: Danger Zone / Take
My Breath Away / Hold My Hand (live-checked).
3. The chosen song goes through the job flow's own
`judge_match(stage="fast")` catalogue tidy, so formatting matches gen's
job form.
- AI kind `none` keeps its typed split, so kjbox can pre-fill its
make-it form. A failed catalogue pass is returned uncached.
- kjbox calls it only when on-device identification and karaoke search
both found nothing, or when the singer uses "Describe it". It's cached
and rate-limited as before.
- Docs: `docs/API.md` notes that the match judge's corrections are
Gemini in practice (its catalogue pass is exact-match only and decide's
catalogue is prefix-only).

## Tests
- `test_match_judge_free_text.py`: resolve_and_tidy cases, descriptive
prompt coverage
- `TestCatalogResolve`
- Local CodeRabbit: findings fixed, re-review clean.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 06:07:13 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 3a6e175fbf feat(tenant): fully automatic tenant lifecycle — subdomain provisioning, delete, per-email access (#1072)
## Summary
The admin tenant console now creates and deletes **fully working**
tenants.

- **Create:**
- Provisions `{id}.nomadkaraoke.com` through the Cloudflare API: a Pages
custom domain on `karaoke-gen-tenant` plus a proxied CNAME.
- If a non-tenant DNS record already owns the hostname, create returns
`409` before anything is written.
- If Cloudflare fails, the GCS tenant is rolled back and create returns
`502`.
- The subdomain must be `{id}.nomadkaraoke.com`, because the edge
function and middleware derive the tenant from the first hostname label.
- **Delete:** new `DELETE /api/admin/tenants/{id}` removes the subdomain
(only records that point at the tenant Pages project), the theme (unless
it's the default or shared), the registry entry and the config. Jobs are
kept.
- **Retry/backfill:** new `POST /api/admin/tenants/{id}/domain` is
idempotent.
- **Per-email access:** new `auth.allowed_emails`, alongside domains. If
either list has entries, only listed emails/domains can sign in; both
empty = open. `@nomadkaraoke.com` admins can always sign in; the
docstring already claimed this but the code didn't do it.
- **Admin UI:**
  - domain status badge with set-up/retry
  - access editor with an "anyone can sign in" warning
  - delete that requires typing the tenant id
  - after create, polls until the certificate is live
- **Infra:** secret `cloudflare-tenant-domains-token`. The Pulumi
container is already applied; the value is an account-owned Cloudflare
token scoped to Pages R/W on the account plus DNS R/W on the
nomadkaraoke.com zone only.

## Testing
- New `test_tenant_domain_service.py` runs against a stateful fake
Cloudflare (httpx MockTransport). It covers provisioning, idempotency,
foreign-record conflicts, deprovisioning only our records, API errors
and a missing token.
- Service tests cover:
  - create provisions the domain
  - a DNS conflict or unusable Cloudflare means no writes
  - a provisioning failure means a full rollback
- delete removes everything, aborts if Cloudflare fails, and keeps a
shared theme
  - access-list normalisation and enforcement
- Route tests cover delete (204/404/502), the domain endpoint (200/409),
502 on create, allowed_emails passthrough, and admin-only access.
- Model tests cover individual emails, combined lists, the legacy flag,
and the admin bypass.
- Frontend (Jest) covers access editing, the open-portal warning, domain
set-up, and type-to-confirm delete.
- Results: `pytest backend/tests --ignore=backend/tests/emulator` gives
4927 passed; `jest` gives 1421 passed.

No local CodeRabbit review was run, so this PR is left open for the
GitHub bot. A separate agent review was done before merge.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Tenant creation provisions a portal subdomain, with its setup status
shown in tenant management.
* Admins can retry domain setup and delete tenants after confirming the
tenant ID.
* Tenant access can be restricted to specific email addresses, allowed
email domains, or both. Addresses are normalized, and admin access is
retained.
* **Bug Fixes**
* Email restrictions are enforced for magic-link sign-in even when no
domain allowlist is configured.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 01:26:32 -04:00
Andrew Beveridge ☄️ 7ebe929d71 feat: quick scrolling-lyrics version for kjbox make-it jobs (v0.244.0) (#1064)
## Summary
kjbox singer "make it" jobs take ~30 min. For kjbox jobs (`X-Client-Id:
kjbox*`) the audio worker now also produces a rough **scrolling-lyrics
draft video** within a few minutes of the audio landing, so the singer
can choose to sing right away (kjbox side: nomadkaraoke/kjbox PR to
follow).

- **Quick separation** — one fast single-model pass
(`mel_band_roformer_instrumental_fv7z_gabox.ckpt`, already baked in the
GPU image, MDXC overlap 2; `QUICK_VERSION_MODEL` override) on the warm
L4 **before** the ensemble.
- **Background render** (`backend/services/quick_version/`, ported from
kjbox `fastgen/`) while the ensemble runs: LRCLIB lyrics (synced →
time-anchored scroll; plain → constant crawl; none → title card), PIL
crawl PNG + one ffmpeg overlay pass, 854×480, vocals mixed back at 30%
as a guide.
- Uploads `jobs/{id}/quick/quick.mp4` → `file_urls.quick.video_mp4`;
`state_data.quick_version = {status, lyrics_tier, ready_at, timings…}`.
- The worker joins the render (≤300 s) before temp-dir cleanup, even if
the ensemble fails. Quick-version failures never fail the job. Skipped
on retries once `ready`, in remote-API mode, or with
`KJBOX_QUICK_VERSION_ENABLED=false`.
- No new endpoint: kjbox polls `GET /api/jobs/{id}` and downloads
`/api/jobs/{id}/download/quick/video_mp4`.

Plan: workspace
`docs/archive/2026-09-26-make-it-quick-fastgen-version-plan.md`.

## Testing
- `backend/tests/test_quick_version.py` (37): lyrics parsing/LRCLIB
fallbacks, layout + scroll expression, ffmpeg cmd, real PIL+ffmpeg
render (skips without ffmpeg), gating, separator params/output
detection, render thread success/failure, audio-worker wiring (join on
ensemble failure, timeout → failed, remote mode skips).
- Full backend suite: 4831 passed.
- Local real runs: Fv7z single-model separation produced both stems with
the custom names; ABBA "Waterloo" rendered in ~9 s (LRCLIB synced).
- Not yet measured in prod: GPU job cold start + L4 quick-separation
time — first thing to check after deploy.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-09-26 22:24:03 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 455fb11a14 feat(upload): unified browser upload flow, bring-your-own instrumental, stranded-upload cleanup (v0.243.0) (#1058)
## Summary
Two invisible upload failures on the same song led to this:
- **Job 8729f19a**: large audio upload (signed-URL flow) abandoned
mid-upload. The only UI signal was the "Creating..." button text and the
card already said "Setting up", so the job sat `pending` forever with
the credit spent.
- **Job 2d0c1ae0**: 44 MB instrumental upload on the review screen was a
multipart POST through Cloud Run (32 MiB cap) and failed silently. Only
the CORS preflight was logged.

## Changes
**Shared upload layer (frontend)**: all user uploads now go through one
path:
- `lib/upload.ts`: signed-URL PUT with progress, sequential multi-file
upload with aggregate progress, client-side duration check.
- `hooks/useUploadTask.ts`: progress state plus a `beforeunload` guard.
- `components/upload/UploadProgressModal.tsx`: a modal the user can't
close, showing progress, speed, ETA, "file N of M" and a keep-tab-open
warning.
- Used by guided job creation, tenant single job and instrumental
review. Tenant bulk keeps its per-row table and uses the shared guard.

**Guided flow**
- Always uses the signed-URL path, which also fixes
`requires_audio_edit` being silently dropped for uploads of 25MB or
more.
- Private upload jobs can attach their own instrumental
(`existing_instrumental`). This skips separation and instrumental
review. The duration is checked before upload, and the backing-vocals
option is hidden.

**Instrumental review**
- New `instrumental-upload-url` and `instrumental-upload-complete`
endpoints: direct-to-GCS upload, then a finalize step.
- One shared processing helper with the multipart route: duration check,
FLAC conversion, `stems.custom_instrumental`.
- Returns a signed `audio_url`. The old post-upload preview URL had no
backend route.
- Inline error plus toast for mismatches and failures.

**Backend robustness**
- `state_data.awaiting_upload`: the dashboard shows "Waiting for upload"
instead of "Setting up".
- The hourly stale-review run also cancels and refunds non-tenant jobs
still awaiting upload after 2h with nothing uploaded.
- `uploads-complete` now checks ownership (previously any signed-in user
could finalize any pending job). On an instrumental duration mismatch it
cancels and refunds the job instead of stranding it.

## Testing
- `make test`: backend 1839 + 4788 + 150 emulator passed; frontend Jest
1391 passed across 117 suites.
- New tests:
- backend: stale-upload processor; signed instrumental endpoints (15
cases); `requires_audio_edit` persistence; uploads-complete ownership
and cancel-on-mismatch
- frontend: upload lib, `useUploadTask`, modal (speed/ETA, can't be
dismissed), `OwnInstrumentalField`, `TenantJobFlow`, API flows
- Manual check against the real frontend (local dev, prod read-only;
create, upload and finalize endpoints mocked, uploads sent to a
throttled local PUT server, so no prod data was changed):
  - guided flow with and without an instrumental
- instrumental review upload on 2d0c1ae0, including a mismatch caught
before upload
- Escape and outside clicks don't close the modal; navigating away
triggers the `beforeunload` prompt

Plan: `docs/archive/2026-09-26-unified-upload-flow-plan.md`

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-26 20:25:00 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 e9465c73d4 chore: gemini-3.8-flash everywhere + auto-correct on Opus 5.5 (cost) (#1054)
## Why
GCP credits expired 2026-09-19; Vertex Gemini 3.1 Pro spend had grown $9
→ $37 → $105/30d. `gemini-3.8-flash` is $0.75/$3.75 per 1M (vs $2/$12
for 3.1 Pro; rises to $1.50/$7.50 on 2027-01-01). Opus 5.5 is $4/$20 (vs
$5/$25 for 4.8).

## Changes
- All Gemini defaults → `gemini-3.8-flash`: credit eval, custom lyrics,
auto-correct (default + compare leg), match judge, parse titles, tenant
bulk analyze, agentic AI router, error-monitor LLM analysis (also
`us-central1` → `global`, required for Gemini 3.x), frontend
`translate.py`.
- Auto-correct Claude leg `claude-opus-4-8` → `claude-opus-5-5` (ci.yml
env), with `effort: "high"` pinned (5.5 defaults to medium). Pricing
table gains both new models.
- Docs updated; version 0.241.1.

## Evidence (12 recent prod jobs, pre-review lyrics scored against the
human-reviewed final)
| Model | Precision | Word-level recall | $/job |
|---|---|---|---|
| gemini-3.1-pro-preview | 91% | 74% | $0.067 |
| gemini-3.8-flash | 98% | 38–41% | $0.020 |
| gemini-3.8-flash @ high thinking | 97% | 60% (4/12 timed out, 504) |
$0.041 |
| claude-opus-4-8 | 94% | 33% | $0.080 |
| claude-opus-5-5 (effort high) | 93% | 58% | $0.048 |

Flash for the Gemini leg is a deliberate cost/recall trade-off (Andrew
chose it); Opus 5.5 more than compensates. Also verified:
`gemini-3.8-flash` works on Vertex `global` (rejects
`thinking_level=minimal`; nothing here uses it), match-judge latency
median 3.3s / max 6.2s (< 12s timeout), Japanese translation smoke test
preserves ICU placeholders. Translation cache is keyed on English text +
locale, so no re-translation churn.

Tests: auto_correct, credit eval, custom lyrics, parse_titles,
error_monitor, agentic router suites pass locally.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* AI-powered correction, translation, matching, title parsing, and
evaluation now use updated models, including Gemini 3.8 Flash and Claude
Opus 5.5.
* **Updates**
* AI service pricing reflects scheduled rate changes, including a Gemini
3.8 Flash rate change beginning January 1, 2027.
  * Application version updated to 0.241.1.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-26 14:53:04 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 b177c36641 feat(kjbox): free-text song resolve for singer search auto-correct (v0.241.0) (#1053)
## Summary
Andrew wants the kjbox singer song search to auto-correct like gen's job
creation does, e.g. "the strokes max picu" → The Strokes — Machu Picchu,
with Undo. gen's match-judge only takes a separate artist and title, and
its catalog stage only fixes formatting, not typos. So this adds:

- **`POST /api/kjbox/catalog/resolve`** (`X-Kjbox-Secret`) `{query}` →
match-judge verdict shape (`kind` cosmetic/content/ambiguous/none,
`confident`, `canonical_*`, `alternatives`) plus
`typed_artist`/`typed_title`.
- **`backend/services/match_judge/free_text.py`**: one small Gemini call
(the match judge's model and settings) that splits and corrects
together. Bad or incomplete model output, or any model failure, degrades
to `kind: "none"`, never a 5xx.
- **Cost and abuse controls:**
- Results are cached per case-folded query (in-process, 2000 entries);
transient failures are not cached.
- A partner-wide cap of 120 AI calls/min per instance (429
`rate_limited`); cache hits are free.
- kjbox only calls this after its own catalogue search came back empty.
- Docs: `docs/API.md` § kjbox Partner API.

## Testing
- `test_match_judge_free_text.py` (8)
- `TestCatalogResolve` (4)
- Full backend suite passes (emulator tests excluded)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added free-text song search resolution that identifies artist and
title, corrects typos, and can suggest alternatives when a match is
ambiguous.
* Repeated searches can use cached results. If resolution is
unavailable, the request returns without an error; uncached requests may
be rate-limited.
* **Documentation**
* Documented the song-resolution API, including its response behavior
and request limits.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-26 13:09:27 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 36e95c7fe9 feat(kjbox): on-demand review sign-in link + record who started the review (v0.240.0) (#1052)
## Summary
Backs Andrew's feedback on the kjbox singer make-it flow (kjbox
companion PR to follow):

- **`POST /api/kjbox/jobs/{job_id}/review-link`** (partner secret +
singer Bearer session): mints the same one-click `job_review:<id>`
sign-in link the kjbox review email carries, on demand. This powers
"**Tap here** to review it yourself" on the singer's My songs. It is
owner-only (404 otherwise, including unknown ids) and only while
`awaiting_review`/`in_review` (409 `not_in_review`). It also returns
`review_started_by`.
- **`state_data.review_started_by` / `review_started_at`** are recorded
on the first `AWAITING_REVIEW → IN_REVIEW` transition. The value is
`"admin"` when an admin other than the owner (the KJ) opened it, and
`"owner"` otherwise (review-token links count as owner). It is wired
into `GET /api/review/{id}/correction-data` (`require_review_auth` now
stashes the reviewer on `request.state`) and into the legacy
`start-review` / `corrections` routes. kjbox reads it through its admin
job poll to show "Andrew has already started reviewing".
- Docs: `docs/API.md` § kjbox Partner API.

## Testing
- New `TestReviewLink` (6 tests) in `test_kjbox_partner_api.py`
- New `test_review_started_by.py` (7 tests)
- New route-level `TestCorrectionDataRecordsReviewStarter` (2 tests)
- Full backend suite: 4671 passed (excluding emulator tests)

CodeRabbit CLI has no seat on this org, so this PR is opened without the
ignore line and the GitHub bot will review it.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added a partner API endpoint for job owners to request a localized
sign-in link for jobs awaiting or undergoing review.
* Review details now indicate whether the owner or an administrator
started the review.
* **Documentation**
* Updated the API documentation with the required partner secret header
for review-link requests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-26 03:28:22 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 d570b1f139 feat(kjbox): partner API for singer email-code sign-in + show credits (v0.239.0) (#1051)
## Summary
Partner API so the kjbox singer UI (live karaoke nights) can turn
patrons into real karaoke-gen customers: the singer verifies their email
with a 6-digit code typed into the kjbox page, and kjbox then searches
audio / creates the job **as that user** (normal gen delivery emails).

All under `/api/kjbox`, gated by `X-Kjbox-Secret` (Secret Manager
`kjbox-partner-secret`, wired in ci.yml; 503 when unset):
- `POST /auth/send-code` — emails a 6-digit code (hashed at rest, 10-min
expiry, new code supersedes old). Exempt from the 2-signups-per-IP cap
(a venue shares one IP) — replaced by a partner-wide 100 new accounts /
24h cap + 5 codes / email / hour. New users tagged `signup_source=kjbox`
(+ venue).
- `POST /auth/verify-code` — 5 attempts then burned, single-use; shares
the post-login path with magic-link verify (refactored into
`complete_verified_login`: welcome credit w/ AI eval, locale, referral,
session, welcome email).
- `POST /credits/show-credit` (+ user Bearer) — +1 credit, no email
("free at Andrew's shows"), idempotent per key, 5 / user / 24h.
- Review-needed + 24h reminder emails for `X-Client-Id: kjbox*` jobs
carry a one-click sign-in link to the review page (singers never signed
in on gen's site).

Paired kjbox PR: nomadkaraoke/kjbox (singer make-it flow, v0.116.0).
Plan: kjbox `docs/archive/2026-09-24-singer-make-it-auto-flow-plan.md`.

## Testing
- New: `backend/tests/test_kjbox_partner_api.py`,
`backend/tests/emulator/test_kjbox_partner_service.py` (+ additions to
existing tests for the verify refactor, review-email links, X-Client-Id
on create-from-search)
- Backend unit suite: 4644 passed; emulator suite passed.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added partner sign-in with one-time email codes and show-credit
claiming, with limits on code requests and credit grants to prevent
repeated or duplicate claims.
* Added localized sign-in emails and single-use review links for
eligible reminders.
  * Sign-in supports welcome-credit evaluation and referral attribution.
* **Improvements**
* Review reminders can use custom links, and new accounts can record
signup source and venue.
  * Credential-like request headers are redacted from stored metadata.
* **Documentation**
  * Added partner API usage and behavior details.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-26 00:03:10 -04:00
Andrew Beveridge ☄️andClaude Fable 5 a9c6de0617 feat: KaraokeHunt interceptor — one freebie per email + community-version short-circuit (v0.236.0) (#1039)
## Summary

Follow-ups from Andrew's live test of the v0.235.0 interceptor (#1037):

1. **Community short-circuit** — his "abba - Waterloo" test generated a
real job despite a community karaoke version existing. Now
`check_community_versions` gates the whole routing: existing version →
no job, no board; the email's main button is the YouTube link (outcome
`community_existing`), and new users keep their conversion credit for a
custom song.
2. **One freebie EVER per email** (Andrew's spec): the goal is
converting installs into direct gen.nomadkaraoke.com users, not keeping
the dead app alive. First-time emails (all variants) now state the app
is retired and they must uninstall it and use Gen directly. A second
request from a converted address gets no job/board/credit — just the
"please uninstall" email (`repeat_request`), throttled to one per
address per 7 days. Accidental same-song double-taps stay silent
(`duplicate` dedup runs first).

## Implementation

- `karaokehunt_conversion.py`: `CONVERTED_OUTCOMES` +
`_freebie_already_used` (fails open) + `_uninstall_email_recently_sent`
(fails closed — anti-spam) + community short-circuit before routing;
queried fields re-verified in Python (defense in depth).
- `send_karaokehunt_conversion`: new `community` and `uninstall`
variants; rewritten intro (retired app, one-time freebie); bold
uninstall note on every first-time variant; new keys translated to all
32 locales.
- Docs: API.md + interceptor README routing updated.

## Testing

42 karaokehunt tests (6 new: community short-circuit ×2,
repeat/uninstall ×2, email variants ×2 more); full `make test` green.
Post-merge: live POSTs through create.karaokehunt.com for (a) a song
with a community version and (b) a repeat request from a converted
address.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* KaraokeHunt requests now receive a one-time conversion per email
address. Repeat requests receive an uninstall reminder instead of a new
conversion, with reminders limited to once every seven days.
* When a community version is available, the email links directly to it;
new users keep their free credit.
* Conversion emails explain credit status and include a Nomad Karaoke
sign-in link.

* **Documentation**
* Updated guidance on request outcomes and the one-time conversion
process.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-22 23:38:43 -04:00
Andrew Beveridge ☄️andClaude Fable 5 3d1ee6a621 feat: KaraokeHunt retired-app request interceptor — auto-convert to Gen users (v0.235.0) (#1037)
## Summary

The retired KaraokeHunt mobile app still has installs submitting ~5-10
song requests/month. Each request POSTs to the dead
`create.karaokehunt.com`, fires a client-side Pushbullet push to Andrew,
and shows the user a hardcoded *"you should receive an email in 5-10
minutes with a link to your new karaoke video"* — a promise broken for
years. This PR makes it true and converts those users into Gen users
(Andrew's spec, BACKLOG Frog #2 extension).

## How

- **Cloudflare Worker**
(`infrastructure/cloudflare/karaokehunt-interceptor/`, deployed via
`deploy.sh` — the karaokehunt.com zone is not IaC-managed): answers the
app instantly, `waitUntil`-forwards the payload to the new backend
intake with a shared secret.
- **`POST /api/karaokehunt/request`**: unauthenticated intake, gated by
`X-KH-Forwarder-Secret` (Secret Manager `karaokehunt-forwarder-secret`,
already created; wired in ci.yml `--set-secrets`), 10/min/IP rate limit,
lenient payload parsing, Firestore `karaokehunt_requests` audit trail.
- **Conversion** (`backend/workers/karaokehunt_conversion.py`, mirrors
`community_daily_pick`):
- new email → account + 1 credit (`karaokehunt_app_conversion`) → job
created **as them** (consumes it); first sign-in still grants the
welcome credit (deliberate — next song covered)
- existing user with credits → job on their credit (email offers to
restore it on reply)
- no credits, or over `KARAOKEHUNT_DAILY_JOB_CAP` (default 3/day, bounds
unattended spend) → free community requests board (dedup → upvote)
- conservative `pick_auto_selection` (not `select_best`): unconfident
matches park in `AWAITING_AUDIO_SELECTION` and the email asks them to
pick — garbage app inputs must never auto-publish to YouTube
- 14-day per-email+song dedup; failures retryable via admin `POST
/api/karaokehunt/internal/reprocess/{doc_id}`
- **Email** `send_karaokehunt_conversion`: 3 variants (job started /
pick the recording / on the board), 7-day one-click sign-in link,
KaraokeNerds community-version bonus link when one exists. EN + all 32
locales via translate.py.

## Safety

Deploys **dark**: endpoint 503s until the secret env lands, and receives
no traffic until the Worker + DNS flip (done post-merge via
`deploy.sh`). Kill switch = delete the Worker route; backend-side =
remove the secret mapping or `KARAOKEHUNT_DAILY_JOB_CAP=0`.

## Testing

- 36 new tests (`test_karaokehunt_intake.py`,
`test_karaokehunt_conversion.py`): secret/rate-limit/JSON gates, all
conversion routings, credit-grant guards, park-on-unconfident-search,
email variants + HTML escaping, reprocess auth.
- Full `make test` green (backend + frontend).
- Post-deploy: live end-to-end POST through `create.karaokehunt.com`
mimicking the app payload, verifying account/job/email/Firestore.

Plan doc: `docs/archive/2026-09-22-karaokehunt-app-interceptor-plan.md`

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **New Features**
- Requests submitted through the retired KaraokeHunt app are now
received and converted into Nomad Karaoke jobs, accounts, or
community-board requests.
- Users receive localized email updates with job status, credit usage,
audio-selection prompts, and tracking links.
  - Requests can be retried by administrators when processing fails.
- Duplicate protection, request limits, and daily processing caps help
prevent repeated or excessive submissions.
- **Documentation**
- Added API and deployment documentation for the request-forwarding
workflow.
- **Chores**
  - Released version 0.235.0.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-22 21:26:18 -04:00
Andrew Beveridge ☄️andClaude Fable 5 d85b6e271c feat: precomputed vocals peaks — Waveforms strips paint sub-second (#1031)
## Summary

Final piece of the review-page speed work (#1024 → #1028 → #1029): the
Waveforms-mode strips no longer require downloading + WebAudio-decoding
the entire vocals stem (~7.5MB, ~4–6s) — the backend now serves a
**precomputed 400-peaks/s envelope as ~150KB of uint8/base64 JSON**,
cached in GCS and pre-warmed at prep time, so strips paint sub-second.

## How

- **`WaveformGenerator.generate_peaks()`** — numpy max-abs envelope,
bucket-for-bucket identical to the frontend's `computePeaks` (same
drift-free fractional-stride bucketing; exact match vs a naive reference
in tests).
- **`AudioAnalysisService.get_vocals_peaks()`** — GCS cache at
`review-audio/vocals_peaks_400.json`, validated against the stem path;
miss decodes the small transcoded OGG; upload best-effort.
- **`GET /api/review/{job_id}/vocals-peaks`** — off the event loop under
the shared waveform semaphore; `202 Retry-After: 15` while separation
runs (mirrors `/audio/vocals`); `Cache-Control: private, max-age=600`;
`peaks_per_second` clamped 50–1000.
- **`screens_worker`** pre-computes the envelope next to the existing
OGG + waveform pre-warm, so it's a pure cache read by the time a job
reaches review.
- **Frontend** — `VocalsAudioDataLoader` prefers the peaks endpoint (one
quick retry on transient errors) and **falls back to the original full
download+decode** on any terminal peaks failure, so a peaks-side problem
can never cost a user their waveforms. Local mode (no peaks URL) uses
the old path unchanged. Vocal stem selection factored into
`backend/utils/stems.py` (shared by routes + worker).

## Testing

- Package: 4 tests incl. exact naive-reference parity (**106 passed**)
- Backend: 4 service + 4 endpoint tests (**4518 passed**)
- Frontend: `fetchVocalsPeaks` units + 5 loader-behavior tests (prefers
peaks / falls back / one-retry-then-fallback / 202 polling / audio-only
mode) (**1307 passed**)
- `review-fast-load.spec.ts` prod E2E extended with a cached
vocals-peaks latency/shape check
- Post-deploy: will verify the endpoint on a real in-review job and
confirm payload size + latency

## Notes

- Existing in-review jobs have no cache yet — their first Waveforms open
computes + caches lazily (bounded); every open after that is sub-second.
New jobs are pre-warmed.
- 8-bit amplitude quantization is visually lossless at strip heights
(14px).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-20 13:08:19 -04:00
Andrew Beveridge ☄️andClaude Fable 5 30cf41e7d6 feat: concurrent-reliability phases 2-5 — Cloud Run shape, degradation telemetry, banner tuning, load-test guard (#1028)
## Summary

Phases 2–5 of the concurrent-load reliability initiative (Phase 1 =
#1024, verified live: the 10-tab meltdown went from 9/30 failed requests
to 30/30 OK). Verbatim spec:
`docs/archive/2026-09-17-concurrent-review-reliability-verbatim-record.md`;
plan:
`docs/archive/2026-09-18-concurrent-review-reliability-investigation-and-plan.md`.

## P2 — Cloud Run shape
- **`--concurrency 80 → 16`** for karaoke-backend: the service is one
uvicorn process per 2-vCPU instance, and review loads are heavy; at 80 a
burst never triggers scale-out and all lands on the single min-instance.
16 makes bursts fan out while staying well above normal per-instance
load. (Deploy flags live in ci.yml — the service is not Pulumi-managed.)
- **Local preview ffmpeg fallback bounded**
(`LOCAL_PREVIEW_MAX_CONCURRENCY=1`, 900s queue timeout, fast
RuntimeError after that): the GCE-outage fallback runs ffmpeg ON the API
instance; unbounded, a few concurrent Preview clicks during an encoder
outage pegged both vCPUs, stalled the health probe, and showed everyone
the orange banner (Andrew's banner-during-preview screenshot).

## P3 — degradation telemetry (the explicit ask)
- New **`POST /api/client-events`** (unauthenticated so review-token
users report too; 30/min/IP rate limit; bot-filtered; detail capped;
URLs query-stripped). Each event → Firestore `client_events` collection
**and** a structured `client_event type=…` log line, so "how often are
users seeing this?" is reviewable via Firestore aggregation or Cloud
Logging correlation.
- Frontend reporter `lib/degradation-events.ts` (60s per-type throttle,
localhost no-op, never throws), wired into:
- `BackendStatusBanner` → `banner_reconnecting` / `banner_unavailable`
with stall age, in-flight count, probe state
- review lyrics-load failure → `lyrics_load_failed` with message/status
- `VocalsAudioDataLoader` → `waveform_slow` (>20s to first strips) and
`waveform_failed` (gave up) with poll/retry counts

## P4 — banner tuning
- Escalating to the full "We're having trouble reaching our servers"
card now requires **two consecutive failed health probes**
(`UNAVAILABLE_PROBE_FAILURES=2`). One failed 4s probe against a
briefly-pegged instance shows only the calm "Reconnecting…" pill.
Single-failure behavior for the pill is unchanged, as is instant
clearing on recovery.

## P5 — regression guard
- **`scripts/load-test-review.py`** checked in: replays the 10-tab
incident (Firestore-ADC review tokens, curl request trios, pass/fail
thresholds, exit code). Documented in `docs/TESTING.md`. Validated
against prod post-Phase-1: 9/9 requests OK, tabs < 3s.

## Testing
- Backend: 6 new tests for client-events
(persist/validate/rate-limit/bot/fail-soft/detail-cap) + local-preview
bound test — full suite **4517 passed**.
- Frontend: new degradation-events suite, two-probe escalation +
debug-snapshot tests; one existing api-fetch test updated to the new
escalation contract — full suite **1298 passed**.
- Live validation of the load-test tool against prod (small 3-tab run):
PASS.

## Notes for review
- Cloud Run concurrency change means bursts now spin up extra instances
(billable during requests only; max-instances unchanged at 20).
- Firestore `client_events` has no TTL policy yet (needs a
gcloud/console action our read-only ADC can't do); volume is tiny
(throttled per client) — can add later if it ever matters.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Added automatic reporting for degraded experiences, including
reconnecting or unavailable backend status, failed lyrics loads, and
slow or failed waveform loads.
- Added concurrent review-page load testing to help verify performance
under multiple open tabs.

- **Improvements**
- Backend status now remains in “reconnecting” until multiple health
checks fail, reducing premature unavailable messages.
- Local preview processing is limited during heavy demand to improve
stability.

- **Documentation**
- Added API documentation for client telemetry and guidance for running
concurrent-load regression tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-19 12:48:21 -04:00
Andrew Beveridge ☄️andClaude Fable 5 b336f36f4e perf: survive concurrent review-page loads (10-tab meltdown fix, Phase 1) (#1024)
## Summary

Andrew can reliably take the backend down by opening ~10 lyrics-review
tabs at once: 1–2 load, the rest show "We're having trouble reaching our
servers" / "The server is temporarily unavailable". This PR is Phase 1
of the systematic fix (investigation + full phased plan in
`docs/archive/2026-09-18-concurrent-review-reliability-investigation-and-plan.md`;
Andrew's verbatim spec in
`docs/archive/2026-09-17-concurrent-review-reliability-verbatim-record.md`).

**Reproduced before the fix (prod, 2026-09-18 03:35Z):** 10 simulated
tabs → 8/30 requests HTTP 500, survivors took 44–72s (vs 0.8–1.3s solo),
Cloud Run logged 9× "request aborted — no available instance". Root
cause: `/waveform-data` decodes a whole stem (pydub→ffmpeg + Python RMS
loop) **synchronously on the single uvicorn event loop, uncached**,
while every other review endpoint also does blocking Firestore/GCS on
that loop — one wedged instance takes all traffic (concurrency=80 → no
scale-out at 10 tabs).

## Changes

**Backend**
- `/waveform-data`: persistent GCS cache
(`jobs/{id}/review-audio/waveform_review_{n}.json`, validated against
source path + num_points), computed via `asyncio.to_thread` under a
bounded semaphore (`REVIEW_WAVEFORM_MAX_CONCURRENCY`, default 4). Cache
miss decodes the ~3MB transcoded OGG instead of the ~35MB raw FLAC.
Pre-computed in `screens_worker` (next to the existing OGG pre-warm) so
review opens are pure cache reads.
- Blocking Firestore/GCS calls (`get_job`, `file_exists`,
`download_json`, `transition_to_state`) moved off the event loop via
`asyncio.to_thread` in the review hot-path endpoints (correction-data,
waveform-data, audio proxies, instrumental-urls/-analysis).
- `/audio/vocals`: now served from the bounded LRU byte cache
(previously a fresh multi-MB GCS download per request), honors HTTP
Range (206), and ranged review-audio responses carry `Cache-Control:
private, max-age=600`.

**Frontend**
- `VocalsAudioDataLoader`: transient failures (5xx/429/network) retry
with backoff `[5s, 15s, 30s, 60s]` instead of silently giving up — the
"Waveforms mode shows no waveforms until manual reload" symptom. 404
stays terminal via new typed `AudioFetchError`; 202 polling unchanged.

## Testing

- 12 new backend tests (waveform cache
hit/miss/stale-source/num_points/OGG-preference/upload-failure; vocals
LRU + Range 206 + Cache-Control; waveform-data endpoint routing) — full
backend suite **4500 passed**.
- 3 new frontend tests (transient retry → load, retry exhaustion, 404
terminal) — full Jest suite **1289 passed** across 105 suites.
- Post-deploy: will re-run the 10-tab burst repro against prod and
record before/after numbers in the plan doc.

## Follow-ups (later phases, tracked in the plan doc + BACKLOG)

Cloud Run shape (lower concurrency so bursts scale out),
`/api/client-events` degradation telemetry (Andrew's explicit ask),
banner threshold tuning, `instrumental-analysis` still signs 4 URLs
(13.7s baseline), load-test regression guard.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Review waveforms load faster with cached waveform data and improved
concurrent-load handling.
* Audio playback supports partial-range requests and short-term caching
for smoother streaming.
  * Transient audio failures are automatically retried up to four times.

* **Bug Fixes**
  * Reduced review-page failures and delays during simultaneous loads.
* Waveform generation falls back to available audio when backing vocals
are unavailable.

* **Documentation**
* Added API documentation for waveform data and updated audio streaming
behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-18 00:25:41 -04:00
Andrew Beveridge ☄️andClaude Opus 4.8 bb743b49e4 Review fast full load: remove IAM signBlob from the review hot path (v0.229.0) (#1018)
## Why

Root-cause fix for the **2026-09-17 signing-hang incident**. `GET
/api/review/{id}/correction-data` signed 2-3 GCS URLs per load via the
IAM **signBlob** API — a network round-trip that made review loads slow
and, when it stalled (~120s), exhausted the request thread pool and took
gen down (`Service Unavailable`, "Loading lyrics data…" hang).

`v0.228.1` bounded/timed signing and made audio + waveform
**best-effort** (the degraded fail-soft path). Andrew's ask was to fix
the root cause so the review page **always loads fast with all features
— audio + waveform — present on first paint**, not the degraded path.

## What (Option B — same-origin byte proxy)

Removes IAM signBlob from the review hot path **entirely** —
`correction-data` now performs **zero** signing calls.

**Backend** (`backend/api/routes/review.py`)
- New `GET /{job_id}/instrumental-audio/{option_id}`: streams the
transcoded OGG bytes with HTTP **Range (206)** support (so `<audio>` can
seek), authenticated by `require_review_auth` (token in the query string
for raw-media-src use, exactly like `/audio/vocals`). The stem GCS path
is resolved **server-side** from the option id — clients can't request
arbitrary blobs. Bounded **LRU byte cache** (prod-safe, unlike the
unbounded dev cache).
- `_build_instrumental_options`: returns **relative same-origin proxy
paths** as `audio_url`, no signing. Local-dev byte proxy path unchanged.
- Dropped the `backing_vocals_waveform_url` signing — it was a **dead
field** (0 consumers repo-wide; the waveform is drawn from the
Bearer-auth JSON `/waveform-data` endpoint).
- Also removed a now-unused `storage` param + refreshed stale
docstrings.

**Frontend** (`frontend/lib/api.ts`)
- Resolves each option's relative proxy path into an absolute,
`?token=`-authenticated `<audio>` src in the API layer
(`getCorrectionData` + `refreshInstrumentalUrls`) — **zero component
changes**, mirrors the proven `getVocalsAudioUrl` pattern. Also fixes
the CORS `Origin: null` issue for good, and the proxy URLs never expire
(the whole signed-URL expiry/refresh dance is now moot).

The `v0.228.1` signing timeout + bounded admission stays as
**defense-in-depth** for the remaining, non-review-hot-path signing
(e.g. preview-video).

## Tests
- Backend: new endpoint (bytes / Range 206 / auth / 404s), plus
**positive assertions that the hot path signs nothing**
(`generate_signed_url` / `get_review_audio_url_async` not called).
- Frontend: URL-resolution unit tests (relative→absolute+token; dev
absolute + null left untouched; refresh path).
- Prod E2E: `frontend/e2e/production/review-fast-load.spec.ts` —
env-gated contract (proxy `audio_url`s present, endpoint serves
`audio/ogg` + Range, loads fast).
- Full suites green: **backend 4469 passed**, **frontend jest 1286
passed**.

## Deploy note
Minor, self-healing skew: during the backend↔frontend deploy gap, only
the **instrumental preview player** for a job opened in that window may
not play (lyrics + everything else fine); a reload after both deploy
fixes it. Far less severe than the signBlob-stall outage this removes.

Plan + rationale:
`docs/archive/2026-09-17-review-fast-full-load-plan.md`. Runbook updated
in `docs/TROUBLESHOOTING.md`.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **New Features**
- Review pages now load instrumental audio through same-origin,
authenticated audio URLs.
- Audio playback supports streaming and range requests for faster, more
reliable access.
- Instrumental audio options automatically include usable
token-authenticated URLs.

- **Bug Fixes**
- Review data no longer depends on URL-signing round trips, preventing
loading delays caused by signing timeouts.
- Removed the unused backing-vocals waveform URL from review responses.

- **Documentation**
- Updated API and troubleshooting documentation for the new audio
streaming behavior.

- **Chores**
  - Version updated to 0.229.0.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-09-17 18:11:56 +00:00
Andrew Beveridge ☄️andClaude Opus 4.8 36b331f3d7 feat(requests-board): admin history view + fix community publish transition (#1008)
## Why

Andrew asked whether we can see past community-requests-board
submissions and whether our first real user got a video. Investigation
surfaced:

- **First real user was fully served.** `tas041964@gmail.com` requested
*Sam Cooke and the Soul Stirrers – Wonderful* (Sep 15); it was
auto-picked, generated, published to YouTube (NOMAD-1650), and he got
the completion email + emailed thanks. ✅
- **No admin view existed** to see this — you had to query Firestore
directly.
- **A real bug:** the community publish fan-out (`mark_published` +
voter emails) fired **only** on the `youtube_upload_queue` path. Jobs
published via the direct video-worker distribution path (like Sam
Cooke's) bypassed it, so the request stayed `in_progress`, never lit up
the board's public "Recently made" list, and other up-voters would
silently miss their "track is live" email.

## What

1. **Fix the publish transition** — extract the fan-out into
`backend/services/community_publish.py` and call it from **both**
publish paths (`youtube_queue_processor` + `video_worker`). Also fixes a
latent `len(int)` crash in the log line.
2. **Backfill / safety net** — `POST
/api/internal/community-requests/reconcile`: idempotent, publishes any
`in_progress` pick whose job is already live on YouTube. (Will be run in
prod post-deploy to fix the Sam Cooke request.)
3. **Admin history view** — `GET /api/admin/community-requests` + a
`/admin/community-requests` page: every request, all statuses, with
requester, votes, and job + YouTube links.

The public board's "Recently made" section already existed in the UI —
it was empty only because of bug #1; fixing the transition (+ reconcile
backfill) lights it up.

## Testing

- `backend/tests/test_community_voter_notify.py` — fan-out from the
shared module, both-path idempotency, voter dedupe/retry, YouTube-url
resolution, and the reconcile scan.
- `backend/tests/test_community_review_admin.py` — the new admin history
endpoint maps all statuses.
- All 31 requests-board + 158 worker tests pass. The one unrelated
failure in `test_audio_processor_local_gpu.py` is pre-existing
(confirmed with changes stashed).

## Notes

- Admin surface is English-only (no i18n changes); no `en.json` strings
added.
- Local CodeRabbit review couldn't run (CLI requires life360-org SSO; no
seat on this repo) — leaving to the GitHub bot.

Plan:
`docs/archive/2026-09-17-requests-board-history-and-publish-transition-plan.md`

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Added an admin Community Requests history page covering all statuses,
votes, submitters, review details, linked jobs, and YouTube videos.
  - Added admin reconciliation for eligible requests with live videos.
- Added automatic voter email notifications after community-request
publication.

- **Bug Fixes**
- Improved publication handling across direct and queued video
workflows.
- Added recovery for incomplete voter notifications and missed
publication transitions.

- **Documentation**
- Documented the community-request history and reconciliation API
endpoints.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-09-17 01:07:40 -04:00
Andrew Beveridge ☄️andClaude Opus 4.8 bedbcdc3bd Requests board: skip songs with existing community karaoke versions (submit + pick time) (#979)
## Summary

Stops the requests board from spending a free track — or wasting voters'
time — on a song that **already has a good community karaoke version
online**, reusing the same KaraokeNerds check the normal job-submission
flow uses, at two points.

Follow-up to the requests-board Phase 2 automation (#977). Addresses
Andrew's two asks:

**A) Submission time (soft, non-blocking).** As a signed-in user types
an artist/title on the board, we debounce-check KaraokeNerds and surface
any existing community versions in a dismissible banner with YouTube
links. They can still request it — but many will just watch the existing
one. Reuses the existing `POST /api/catalog/community-check` endpoint +
`CommunityVersionBanner` component (added an optional board-appropriate
subtitle).

**B) Pick time (review).** A song may have had no version when requested
but gained one by the time it's upvoted to the top. So the daily picker
now KaraokeNerds-checks each candidate **in rank order**:
- A candidate that already has a community version is **flagged for
review** (`review_state=pending`) instead of auto-made, and the picker
**skips to the next clean request** — so a fresh, non-duplicate free
track still ships that day.
- Andrew is emailed a summary and resolves held picks at
**`/admin/community-reviews`**: **Make ours anyway** (creates the job
like a normal pick), **Reject** (declines and emails every up-voter a
link to the existing version), or **Keep on board** (leaves it votable,
snoozing re-review for 30 days).
- Held requests stay visible/votable on the board; they just aren't
auto-made. Flag/email is idempotent (no daily re-nag).

## Andrew's decisions (via AskUserQuestion)
Held pick → **skip to next clean request**. Review surface → **admin
queue page + email**. "Keep" → **return to board, snooze re-review (30
days)**.

## Changes
- **Model:** `SongRequest.review_state` (pending|snoozed),
`community_versions`, `community_checked_at`, `review_snoozed_until`.
- **Service:** `list_pick_candidates` (skips review-blocked),
`set_review_pending` (transactional/idempotent), `snooze_review`,
`clear_review`, `reject_request`, `list_pending_reviews`.
- **Picker:** candidate loop + community check; extracted
`_provision_and_start` shared by the picker and the admin "make" action.
- **Admin:** `GET/POST /api/admin/community-reviews`,
`/admin/community-reviews` page + sidebar item. New existing-version
email to up-voters (33 locales).
- **Frontend:** board submit form debounced check + banner.

## Testing
- Picker candidate loop (flag dup → make next clean; all-dup → empty +
flagged; resume skips re-check), admin actions (make/reject/keep, 404,
500), emulator coverage for the new service methods. 86 passing across
touched suites; frontend typechecks clean; 32/32 locale parity.

## Deploy
No `infrastructure/` changes — merge deploys it. The daily picker is
already enabled in prod, so the pick-time check goes live with the
backend deploy; the submission banner ships with the frontend.

## Review
CodeRabbit reviewed locally (one minor grammar nit, fixed).

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-09-03 16:59:16 -04:00
Andrew Beveridge ☄️andClaude Opus 4.8 05f024bdd4 Requests voting board Phase 2: daily free-community-track automation (#977)
## Summary

Phase 2 of the requests voting board (`requests.nomadkaraoke.com`) — the
daily automation that turns the top-voted request into a free,
auto-generated, published karaoke video. **Deploys dark** behind a
kill-switch (`COMMUNITY_DAILY_PICK_ENABLED`, default off); the daily
Scheduler only shadow-logs until it's flipped on.

Picks up
`docs/archive/2026-09-03-requests-voting-board-phase2-handoff.md`.
Design + Andrew's decisions:
`docs/archive/2026-09-03-requests-board-phase2-plan.md`.

## What's included
- **Daily picker** (`community_daily_pick.py`, Scheduler noon US
Eastern, DST-aware): claims the UTC day with a create-only lock
(`daily_community_pick/{YYYY-MM-DD}` → **one free track/day, total**),
picks the top open request with net votes ≥ 0 (source-agnostic; skips
community-rejected), grants the requester a free credit, and submits the
job **as that user** (owned by them, credit consumed) via the same
search → auto-select → download path the web flow uses. Fully idempotent
(resumable per-day lock phase + per-request guard flags).
Shadow/`?dry_run=true` mode.
- **24h ownership handoff** (`community_handoff.py`, hourly): reassigns
a stale review to the next up-voter (compare-and-set guarded), up to **5
voters**, then parks the track (`stalled`). Community jobs excluded from
the normal stale-review auto-cancel.
- **Voter publish fan-out**: YouTube upload-queue hook marks the request
`published` and emails every up-voter (except the owner) a new "track
you voted for is live" email (33 locales). Retry-safe (per-voter success
tracking).
- **Two internal endpoints** + two Cloud Scheduler jobs (Pulumi).

## Andrew's decisions (via AskUserQuestion)
- Trigger: **noon US Eastern**. Trending-agent fallback source:
**deferred** to a follow-up. Net-negative requests: **skipped**. Handoff
cap: **5 voters then park**.

## Deferred / follow-ups (not in this PR)
- **Trending-agent fallback source** (auto-submit a candidate when the
board is empty) — deliberately deferred; must generate candidates the
careful way (karaoke-decide `candidates` + KaraokeNerds).
- **YouTube description board link** — left untouched to avoid
clobbering Andrew's parallel descriptions work; add
`requests.nomadkaraoke.com` to `default_youtube_description` once that
lands.

## Testing
- Worker unit tests (kill-switch shadow, empty board, day-lock
idempotency, credit-grant/job-create guards, handoff cap/stale/CAS,
voter fan-out retry-safety) + emulator tests for the new service methods
against real Firestore transactions.
- All touched existing suites (youtube queue, stale-review,
requests-board routes/logic) pass. 79 passed locally.

## Review
CodeRabbit reviewed locally (2 cycles completed; 2 major + 1 minor
findings all addressed — retry-safe fan-out, CAS owner assignment,
up-voter ordering). 3rd cycle hit the free-OSS rate limit; it would only
re-verify the applied fixes.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-09-03 01:34:31 -04:00
Andrew Beveridge ☄️andClaude Opus 4.8 d41854f787 feat(requests-board): public song-request voting board (Phase 1) (#975)
## What & why

Phase 1 of the **daily-free-community-track** initiative (BACKLOG:
*"Daily free community track via public voting board"*). A public,
Hacker-News-style voting board at **requests.nomadkaraoke.com** where
anyone signs in with an email magic link, submits a song request
(artist/title auto-corrected + deduped), and casts **one vote per
calendar day**. Publicly viewable so it can be linked from YouTube
descriptions to drive top-of-funnel awareness.

Ships the board first (this PR) so the linkable URL exists; the
automation (daily auto-picker, free-credit grant on pick, 24h ownership
handoff, voter publish-emails, and the trending-karaoke agent as a
fallback request source — one free track/day total) is **Phase 2**.
Design doc: `docs/archive/2026-09-02-requests-voting-board-plan.md`.

## Backend
- New `song_request` models + service (Firestore `song_requests` /
`song_request_votes`) with transactional vote arithmetic (create / move
/ flip / toggle-off), dedupe, and ranking. One-vote-per-day is enforced
structurally via the vote doc id (`{email}__{YYYY-MM-DD}`).
- New `/api/requests-board` router: `GET /requests` (public, annotates
the caller's vote), `POST /requests` (submit — auto-corrects via the
existing `match_judge`), `POST /requests/{id}/vote`, `GET /me`.
- Board sign-in (`purpose="requests_board"`): passwordless identity only
— **no welcome credit**, a **higher per-IP signup cap**
(`BOARD_MAX_SIGNUPS_PER_IP`, default 30) so a shared venue WiFi can
register many voters while the per-fingerprint cap stays strict, and
`redirect_path="/requests"`.
- New `POST /users/claim-welcome-credit` powers the "Make it yourself
now" conversion (grants the standard credit the normal way, idempotent).

## Frontend
- `app/[locale]/requests` board: submit form with "we tidied that to…"
feedback, ranked vote list with daily-vote UX, convert-to-gen CTA,
recently-made list with YouTube links.
- API client methods + types; verify page honors `redirect_path`; new
`requests` i18n namespace translated to **all 33 locales**.
- `requests` added to `nonTenantSubdomains`.

## Testing
- 20 backend unit tests (routes + pure logic + sign-in + board signup
limit) and 8 emulator tests exercising **real Firestore transactions**
(vote arithmetic, dedupe, idempotent resubmit, one-vote-per-day). Full
backend unit suite green (4247 passed); frontend typecheck clean.
- Reviewed locally with CodeRabbit CLI (3 cycles); all findings
addressed (auth headers, idempotent resubmit, signup-limit hardening,
blank-input validation, i18n particle, refresh race).

## Deploy note
The board is reachable at `gen.nomadkaraoke.com/en/requests` on deploy.
Wiring the vanity `requests.nomadkaraoke.com` is a **post-deploy
Cloudflare step** — see
`docs/runbooks/requests-subdomain-cloudflare.md`.

@coderabbitai ignore

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-09-02 22:02:35 -04:00