2052 Commits
Author SHA1 Message Date
Andrew Beveridge ☄️andClaude Opus 5.5 4e43449559 fix(backend): stop API event-loop freezes + identify users in degradation telemetry (v0.264.0) (#1120)
## Summary
The "Reconnecting" pill and the "servers unavailable" banner (164 / 49
events, 2026-09-19 → 10-02) were mostly caused by the single API
instance's **event loop freezing for 20–45s**. Every request on the
instance, `/api/health` included, completed at the same moment. The full
evidence and plan are in
`docs/archive/2026-10-03-backend-loop-freezes-telemetry-plan.md`.

### What froze the loop, and the fix for each
- **Screens worker:** it ran as a post-response `async def`
BackgroundTask doing 4K PIL renders, GCS transfers and ffmpeg transcodes
on the loop. Under `--cpu-throttling` it was also CPU-starved. **Fix:**
it now runs inline in its Cloud Tasks request (CPU stays allocated,
failures still return 200 so Cloud Tasks doesn't double-run it), with
the heavy steps in `asyncio.to_thread`. Also offloaded: style load,
timing signals, pre-apply.
- **Matplotlib font cache:** it was rebuilt on every new instance (17/28
episodes). **Fix:** it's pre-built in `Dockerfile.base`
(`MPLCONFIGDIR=/opt/mplconfig`), and pyplot is imported lazily.
- **Hourly stale-review cron:** an `async def` with zero awaits doing
≤1500 reads and SMTP sends. **Fix:** it runs in a thread.
- **166 `async def` routes that never awaited** (sync
Firestore/GCS/email/Gemini on the loop, e.g. magic-link verify).
**Fix:** they're now plain `def`, so they run in FastAPI's threadpool.
The new AST guard `test_no_blocking_async_routes.py` fails CI on new
ones. It also flags sync routes that call
`asyncio.create_task`/`get_running_loop`; that caught `confirm_duration`
and `trigger_youtube_queue_processing`, which stay async.

### Safety net
- `backend/services/loop_watchdog.py`: a heartbeat plus a daemon thread.
Any stall over 1s logs `EVENT_LOOP_STALL` with the loop thread's stack
and a `culprit=file:line`. Stalls of 5s or more are written to
`client_events` as `server_loop_stall`.
- Pulumi: the log metric `backend/event_loop_stalls` plus an alert on ≥2
stalls of ≥10s in 30 min. **This still needs a local `pulumi up`.**
Pulumi is never applied by CI, and the admin gcloud login has expired.

### Telemetry gaps fixed
- **Identity:** `user_email` used to be hard-coded to null. The frontend
now sends its session token, and the server resolves `user_email`,
`is_admin`, `is_internal` and `is_test`. A client-sent email is ignored.
- **New fields:** `device_fingerprint` (the existing FingerprintJS
visitorId), `tab_id`, `episode_id`, and `tenant`.
- **New event:** `banner_recovered`, carrying `duration_ms` and
`peak_status`.
- **Report:** `scripts/client_events_report.py` summarises the data,
excluding admin/internal/test accounts by default.

## Testing
- `make test` passes: backend 5529, plus the emulator suite, karaoke_gen
unit tests, and frontend Jest 1623.
- New tests:
- Watchdog: a deterministic state machine plus detection of a real
blocking call.
- Event-loop responsiveness: a ticker keeps ticking while the heavy
steps run.
  - Screens inline and failure semantics.
  - `client_events` identity and fields.
  - The AST guard.
  - The report summary.
  - Frontend reporter and banner episode tests.
- Local CodeRabbit CLI review: 0 findings.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

@coderabbitai ignore

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 14:17:51 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 4015b24857 feat(audio-search): title-matched best pick; Spotify track match is Recommended, not 'Limited sources' (v0.264.0) (#1122)
## Summary
Live kjbox night 2026-10-01: a singer's make-it for **Braxton Keith -
The Chair** got 10 Spotify results (3 were the right track) and no
torrents. Because Spotify counts as "not lossless", the search was
**tier 3 "Limited sources found"** with a wrong-track best pick ("I Own
This Bar"), so the singer pasted a YouTube link. That then hit the
YouTube bot-check.

- `getBestResult(results, searchTitle)`: when any torrent/Spotify result
is a **confirmed** track-name match, only those compete. YouTube is
never promoted this way. Ties go to seeders, then popularity
(`view_count`). If nothing in the pool ranks, the fallback stays within
the matches.
- `isConfirmedTitleMatch`: the track filename must actually be compared
and equal to the title, ignoring only trailing version suffixes (`(feat.
…)`, `[…]`, ` - 2013 Remaster`). "The Chairman" ≠ "The Chair"; "7
Rings", "Mr. Brightside" and "(I Can't Get No) Satisfaction" work.
- `checkFilenameMismatch` now strips only real audio extensions. Before,
"Mr. Brightside" lost ". Brightside" as an "extension".
- `getSearchConfidence`: a title-matched Spotify best pick is **tier 2**
(Recommended), with no "No lossless sources" warning and a "Spotify
audio" reason.

kjbox ports this 1:1 in `static-sing/audio_rank.js` (separate PR).

## Test plan
- [x] `audio-search-utils.test.ts`: 10 new tests from the real Braxton
Keith results plus edge cases (86 total)
- [x] Full frontend jest: 1617 passed
- [x] Local CodeRabbit: 3 cycles, all findings fixed

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.264.0
2026-10-03 14:08:12 -04:00
github-actions[bot] 7ea86a8dda test(e2e): self-heal Stripe Checkout (fillPostalCode) (#1119)
## 🩹 Stripe Checkout self-heal

Stripe changed its hosted checkout page. In [run
#197](https://github.com/nomadkaraoke/karaoke-gen/actions/runs/37140934446)
every deterministic variant of **fillPostalCode** failed; the Gemini
fallback completed the step(s), and the purchase was then **verified
server-side** (credit granted).

This PR only adds the recipe(s) to `learned-variants.json`, so future
runs replay them deterministically (no LLM). It auto-merges once CI
passes.

### Learned recipes
```json
[
  {
    "stepId": "fillPostalCode",
    "model": "gemini-3.5-flash-lite",
    "llmCalls": 1,
    "actions": [
      {
        "type": "fill",
        "target": {
          "by": "css",
          "css": "#billingPostalCode",
          "frameUrlIncludes": "checkout.stripe.com/c/pay"
        },
        "value": "{{POSTAL_CODE}}"
      }
    ]
  }
]
```

@coderabbitai ignore

🤖 Generated by the E2E Daily self-healing workflow

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-10-03 18:07:21 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 c8461e56d0 fix(e2e): self-heal CI approval — correct pending detection, re-approve open PRs (#1121)
## Summary
This fixes the approval step from #1118, found by the live test on run
37140934446. That run reported `approved=true`, but bot PR #1119's CI
run was still `action_required`: the old check treated *any* existing
`pull_request` run as already running.

- `approve_ci` now succeeds only once a `pull_request` CI run exists
that is past `action_required`. It approves pending runs as it finds
them.
- When a self-heal PR is already open, the job now runs `approve_ci` for
that PR instead of exiting. So a stuck bot PR gets unstuck on the next
run.
- Polling is tolerant of transient `gh`/`jq` failures (from the local
CodeRabbit review). An empty branch name fails fast.

## Testing
- `actionlint` is clean for the changed lines; local CodeRabbit review:
2 findings, both fixed.
- After merge: a forced-LLM run on `main` will hit the "PR already open"
path for #1119. That approves its CI with the workflow's own token and
should let it auto-merge with no manual input.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 17:50:38 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 1c9104a79d fix(e2e): self-heal job approves its PR's CI run (no manual step) (#1118)
## Summary
This follows #1116. The live test of the self-heal loop (bot PR #1117)
merged, but only after one manual step: approving its CI run.

- **Approve the bot PR's CI run.** A bot-authored PR's `pull_request` CI
run is created as **action_required** (waiting for approval), and until
it runs it outranks the passing CI Gate from the dispatched run. The
`self-heal-pr` job now approves that run via `POST
/actions/runs/{id}/approve`. If approval fails, it dispatches `ci.yml`
and the Discord alert says the PR needs a one-click approval, with a
link.
- **Ruleset dependency, now documented.**
`require_extra_approval_for_unattributed_changes` is now set to `false`
on the `protect main` ruleset (11423976). It's a newer GitHub setting
that defaults to true and blocks bot-authored PRs until a human
approves. This is recorded in `docs/TESTING.md` and
`docs/LESSONS-LEARNED.md`.

## Testing
- `actionlint` is clean for the changed lines.
- After merge: a forced-LLM run on `main` on a step with no learned
recipe yet (`fillPostalCode`) will open a fresh bot PR and exercise
approval → CI → auto-merge without manual input.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 17:30:35 +00:00
github-actions[bot] 84f709ea0e test(e2e): self-heal Stripe Checkout (selectCard) (#1117)
## 🩹 Stripe Checkout self-heal

Stripe changed its hosted checkout page. In [run
#196](https://github.com/nomadkaraoke/karaoke-gen/actions/runs/37139075045)
every deterministic variant of **selectCard** failed; the Gemini
fallback completed the step(s), and the purchase was then **verified
server-side** (credit granted).

This PR only adds the recipe(s) to `learned-variants.json`, so future
runs replay them deterministically (no LLM). It auto-merges once CI
passes.

### Learned recipes
```json
[
  {
    "stepId": "selectCard",
    "model": "gemini-3.1-flash-lite",
    "llmCalls": 3,
    "actions": [
      {
        "type": "click",
        "target": {
          "by": "css",
          "css": "[data-testid='card-accordion-item']",
          "frameUrlIncludes": "checkout.stripe.com/c/pay"
        }
      }
    ]
  }
]
```

@coderabbitai ignore

🤖 Generated by the E2E Daily self-healing workflow

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-10-03 17:29:06 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 4768164a94 feat(e2e): self-healing Stripe Checkout (LLM fallback + auto-merging fix PRs) (#1116)
## Summary
Stripe's hosted checkout page has broken the daily E2E **5 times since
April** (#916, #919, #945, #1115…). Each fix added another selector, and
each new Stripe change still needed a human. This PR makes the Stripe
stage repair itself, with no manual steps.

**How it works.** `stripe-checkout.ts` now runs the checkout as verified
steps (`selectCard`, `fillCardNumber`, `fillCardExpiry`, `fillCardCvc`,
`fillCardholderName`, `fillPostalCode`, `uncheckSaveInfo`,
`submitPayment`). Each step tries, in order:
1. **learned variants** (`learned-variants.json`, newest first),
2. the existing hand-written **built-in** selectors,
3. a **Gemini fallback** that drives the page one action at a time until
the step's check passes. Model list: `gemini-3.5-flash-lite` →
`gemini-3.1-flash-lite` → `gemini-3.8-flash`; IDs that aren't available
are skipped.

A step only counts as done when its **check** passes, never just because
a click didn't throw (the root cause of run #192's misleading failure).

**Fully self-healing loop.** When the LLM repairs a step *and* the
purchase is confirmed server-side (credit granted), the new
`self-heal-pr` job (on `main` only):
- opens a **data-only** PR that changes only `learned-variants.json`
(label `e2e-self-heal`, `@coderabbitai ignore`),
- enables auto-merge,
- dispatches `ci.yml` on the branch, because PRs opened with the
workflow token don't trigger CI.

The next run replays the recipe without the LLM. Discord and the daily
email report 🩹 when this happens, and only one self-heal PR is open at a
time.

**Safety**
- Card data never reaches the LLM. It may only use per-step
`{{CARD_NUMBER}}`-style placeholders, which are substituted locally.
Field values and card-number-like digit runs are redacted from what it
sees, and inputs are masked in screenshots. Tests assert no secret
appears in the prompt.
- Guardrails refuse clicks on Link/Klarna/Cash App/Bank/"Back to…", and
on the final Pay button outside `submitPayment`. They check the
**resolved element** (text, aria-label, test ID, submit type), not just
the model's label.
- The Pay step's check is "redirected to the success page OR credit
granted server-side" (throttled), so the fallback never submits twice.
The cancel URL doesn't count as success.

**Gemini key:** the AI Studio `gemini-api-key`, billed outside the
nomadkaraoke project. A `GEMINI_API_KEY` repo secret is used if set;
otherwise it's read from Secret Manager via keyless login, and the
credential file is deleted before the tests run. With no key, the test
runs deterministic-only, as before.

Also passes the existing `E2E_STRIPE_ZIP` secret, which was never wired
up.

## Testing
- **Offline:** `playwright.selfheal.config.ts`, 21 tests with fixture
Stripe pages served on fake checkout.stripe.com/js.stripe.com and a stub
LLM. Covers: known layout uses no LLM; unknown layout → LLM repairs it →
recipe learned → replayed without the LLM; no secrets reach the LLM;
guardrails, including CSS targets; deterministic-only failure mode; unit
tests for redaction, merging and frame keys. Added to CI's smoke job.
- **Live (real Stripe, real Gemini) on this branch,** with
`self_heal_force_llm=selectCard`:
- [run
37137960148](https://github.com/nomadkaraoke/karaoke-gen/actions/runs/37137960148):
`gemini-3.5-flash-lite` completed "choose Card" in one call, credit
granted 1→2. This exposed the session ID in Stripe's URL path, now
fixed.
- [run
37138459027](https://github.com/nomadkaraoke/karaoke-gen/actions/runs/37138459027):
repeated after the review fixes. Recipe
`[data-testid="card-accordion-item"]` @ `checkout.stripe.com/c/pay`,
credit granted.
  - The PR-opening job is skipped on branches by design.
- After merge, I'll dispatch once on `main` with
`self_heal_force_llm=selectCard` to exercise the PR → CI → auto-merge
path.
- Local CodeRabbit review: 2 findings, both fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 17:02:12 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 2b68cca183 fix(e2e): expand Stripe Checkout card accordion (daily E2E run #192) (#1115)
## Summary
Daily E2E Stage 1 failed (run #192): `Could not find card field:
cardNumber in any frame`.

**Root cause:** Stripe changed its Checkout payment-method accordion.
The Card row is now a plain `Card` text label with a visually-hidden
`button "Pay with card"`. There is no longer a radio/button named
`Card`, so the helper never expanded the accordion. This only affects
the test. Customers can still pay.

(Runs #190/#191 on Oct 1-2 failed for a different, already-fixed reason:
the Postmark 403 outage, so the magic-link email was never sent.)

## Changes
- `frontend/e2e/helpers/stripe-checkout.ts`: the chooser now matches
`Card` / `Pay with card` (radio, button, or exact label text). If the
fields still don't appear, it dispatches a click on the hidden `Pay with
card` button. On failure it writes
`test-results/stripe-checkout-dom.html` so the next markup change can be
diagnosed from the artifacts.

## Testing
- Type-checks; `playwright --list` loads the spec.
- Not run end to end yet: that needs a live Stripe charge ($0.50).
Verify with a `workflow_dispatch` of *E2E Daily Test* on this branch, or
the next scheduled run.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Tests**
* Automated checkout checks now recognize additional card-selection
options, including card choices presented as buttons or radio options,
and retry when card fields are slow to appear. If the fields remain
unavailable, the checks save diagnostic details from the checkout frames
to help investigate the issue. These updates affect the checkout test
process; no changes to the customer-facing payment flow are described.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 16:31:02 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 d6972f10aa infra(monitoring): alert when Postmark SMTP-fallback sends fail + document static-IP options (#1114)
## Summary
This is a follow-up to #1112/#1113 and decide#147. Postmark confirmed
(ticket #11562402) that the nginx 403s are an IP-reputation block on
shared Google us-central1 egress IPs, and they won't lift it. The SMTP
fallback is working in prod: on 2026-10-02 between 19:10 and 20:17 UTC,
12 of 12 blocked sends were delivered with 0 failures. To save money,
we're **deferring** a static egress IP and adding a free tripwire
instead.

## Changes
- `infrastructure/modules/monitoring.py`:
`create_email_delivery_observability()` adds:
- A log-based metric `email/postmark_smtp_fallback_failures`, plus an
alert policy ("Email - Postmark SMTP fallback failing (emails being
lost)") that fires on any failed fallback send in karaoke-backend **or**
karaoke-decide.
- The filter matches **both** `jsonPayload.message` (gen logs structured
JSON) and `textPayload` (decide logs plain text).
- `docs/archive/2026-10-02-postmark-ip-block-options.md`: full write-up
covering the evidence, Postmark's answer, how the fallback works, the
alert, and the static-IP options. It includes **measured costs**:
reusing the existing `github-runners-nat` with a reserved IP is
~$4–7/mo. It also has Pulumi steps, NAT sizing, and how to verify the
reserved IP is used.
- `docs/TROUBLESHOOTING.md` and `infrastructure/monitoring/README.md`
link to it.

## Testing / rollout
- Filter validated against real logs:
  - 0 failures, which is correct.
- Positive control on the gen JSON branch: 12/12 real fallback sends
matched.
  - Positive control on the decide text branch: 5/5 real lines matched.
- `pulumi preview` showed: +2 to create, 274 unchanged.
- **Already applied locally with `pulumi up`** (repo rule). The first
apply hit the log-metric propagation race, and re-running succeeded;
that gotcha is now documented in the code. The alert is live, enabled,
and has the email channel attached.
- Cost: $0 now. Alert policies cost $0.35/mo from 2027-09-01.

Local CodeRabbit review: 2 findings (Cloud NAT sizing, reserved-IP
verification in the options doc), both addressed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 18:42:29 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 5181e42b46 fix(email): failed SMTP QUIT after an accepted send doesn't fail the send (v0.263.2) (#1113)
## Summary
This is a follow-up to #1112. CodeRabbit found the issue while reviewing
the same fix in karaoke-decide. If Postmark accepts the message but the
SMTP `QUIT` fails during the `with` block's exit, `_send_via_smtp`
reported a failure, and a caller retry could then send the email twice.
Cleanup errors are now swallowed, falling back to `close()`, and only
errors from the send itself fail the send.

## Testing
- Added `test_quit_failure_after_send_still_reports_success`. Email
tests: 72 passed.

The same change was reviewed by local CodeRabbit in karaoke-decide#147.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.263.2
2026-10-02 14:38:42 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 eb1c914e4a fix(email): fall back to Postmark SMTP when the API edge returns HTML 403 (v0.263.1) (#1112)
## Summary
Since 2026-10-02 11:25 UTC **every** backend email has failed: magic
links, made-for-you order confirmations and admin alerts (e.g. job
`18bee12c`). Postmark's API edge answers our us-central1 Cloud Run
egress IPs with a bare nginx HTML 403. The retries added in v0.261.1
never get past it, because a block lasts hours.

- Postmark IP Allowlisting is **off** (checked in the UI at account and
server level). karaoke-decide (us-central1) gets the same 403s, but
us-east4 jobs and residential IPs don't. That points to a network-level
block on Postmark's side. Support ticket **#11562402** is filed.
- Postmark SMTP is served by separate infrastructure and is already
enabled on the server.

## Changes
- `PostmarkEmailProvider`: on an HTML 403 (edge block), go straight to
SMTP (`smtp.postmarkapp.com:587`, STARTTLS, server token as username and
password, `X-PM-Message-Stream: outbound`). The API retry was dropped
for this case; it only added about 4s of latency.
- Also fall back to SMTP when every API attempt hits a connect timeout
(the message never reached Postmark, so no duplicate risk).
- Unchanged: 429/5xx are still retried on the API. JSON Postmark errors
and read timeouts never fall back.
- SMTP sends return no MessageID and clear `last_message_id`.
Header-injection `ValueError`s are caught.
- `docs/TROUBLESHOOTING.md` runbook updated. Version bumped to 0.263.1.

## Testing
- 71 email unit tests, plus `make test-backend` (unit + emulator)
passing.
- **A real send through `_send_via_smtp` from a local machine** to
andrew@nomadkaraoke.com shows as `Sent` on the outbound stream in
Postmark's message API, and the Unicode subject came through intact.
- After deploy: trigger a magic-link email and confirm `via Postmark
SMTP fallback` in the backend logs.

Local CodeRabbit review: 1 finding (header-injection ValueError), fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.263.1
2026-10-02 13:50:33 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 b90ad82d90 docs: record Vertex and Gemini API shutdown in nomadkaraoke (#1111)
Records the manual (non-Pulumi) infra changes from the Gemini migration
(#1108): Vertex IAM removal, the cross-project Aquarius grant removal,
disabling aiplatform + generativelanguage in nomadkaraoke, deleting the
unused restricted API key, and how to rotate `gemini-api-key`. Docs
only, so no version bump.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Documentation**
* Updated deployment guidance to record recent API access and credential
changes, including disabled services and removal of an unused restricted
key. The guidance also describes the associated project and billing
setup, with steps for re-enabling services or rotating credentials. No
end-user product behavior changes are included.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 22:51:39 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 0fa10e4330 feat(frontend): friendly cold-start waking state and backend pre-warm (v0.263.0) (#1110)
## Summary
`karaoke-backend` now runs with min-instances 0, so a request that hits
a cold backend waits ~15-20s (measured 18.4s) while Cloud Run boots an
instance. Cloud Run queues requests during that window, so the health
probe times out exactly as it does during a recycle, and users saw the
amber "Reconnecting…" / "having trouble reaching our servers" banner.
This replaces that with a calm cold-start experience and pre-warms the
backend.

### Behaviour (`frontend/lib/backend-status.ts`)
- New `waking` status. **Cold-start rule:** if no backend response
succeeded in the `REACHABLE_FRESH_MS` (10 min) before the oldest stalled
read began (fresh page load, or a tab idle long enough for
scale-to-zero), a stall is treated as a cold start:
- probe `/api/health` after `COLD_PROBE_AFTER_MS` (1s) instead of 10s,
so a normal warm load (reads finish in <1s) never sends an extra probe;
- once the read is ≥3s old **and** the probe failed (4s timeout), show
the waking card (~5s into a cold start);
  - it clears the instant reads complete.
- **Escalation:** past `WAKING_ESCALATE_MS` (60s) the normal
reconnecting/unavailable logic takes over. A stall that starts within 10
min of a successful response is a recycle/outage, so the existing banner
applies unchanged.
- The probe still gates everything: a warm backend with a merely slow
endpoint shows nothing.
- `apiFetch` marks the backend reachable on every non-transient
response, for any method. GET hard timeout (45s) and retry logic are
unchanged and already ride out a ~20-30s cold start (the queued request
simply completes).

### Pre-warm
- `prewarmBackend()`: an untracked, fire-and-forget `GET /api/health`
(`keepalive`, `cache: no-store`, no custom headers, so no CORS
preflight). It runs on app load via the global `BackendStatusBanner`
(all `[locale]` pages) and the admin layout, and again when a tab
becomes visible after >10 min hidden. It's skipped if the backend
answered in the last 10 min. Success marks the backend reachable.
- Companion PR nomadkaraoke/public-website#31 pre-warms from the
marketing homepage Hero (no-cors, idle-time).

### UI
- Waking card: neutral card colours plus a tenant-primary spinner.
"Starting up our karaoke servers…" / "This usually takes about 20
seconds. Thanks for your patience!". No warning colours and no dismiss
button.
- New `banner_waking` degradation event, added to the frontend type and
the backend `client-events` allow-list. It includes
`since_reachable_ms`.
- Strings translated for all 33 locales (Gemini Developer API path from
#1108).

## Testing
- Store unit tests with fake timers. Cold start → waking (~5s) → clears.
Fast cold read shows nothing. Warm slow endpoint shows nothing.
Prolonged stall → `unavailable` after 60s. Previously reachable then
stall → `reconnecting`. Reachable >10 min ago → waking. Pre-warm
de-dupe, failure and visibility re-warm are covered.
- apiFetch tests: a fresh session hitting a cold backend shows `waking`
(never the warning) and the GET completes at 18s. The pre-warm request
shape (no headers, keepalive) is checked.
- Backend: `banner_waking` accepted by `/api/client-events`.
- `make test` passes.
- Real browser check: `next dev` against a local stub that holds every
request for 18s from the first hit. The waking card appears by ~7s, the
dashboard shows "Loading jobs…" (no error or empty state) and everything
resolves at 18.3s.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added a startup status banner for cold starts, with an estimate that
startup usually takes about 20 seconds. The banner is available in
multiple languages.
* The app now checks backend availability when needed and after a tab
has been hidden for an extended period, helping requests recover from
cold starts.
* **Documentation**
* Updated API telemetry documentation to include the new startup event.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
v0.263.0
2026-10-01 22:27:26 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 8490ab75eb chore(gemini): move every Gemini call from Vertex AI to the Gemini Developer API (#1108)
## Summary
Moves **every** Gemini call in gen off **Vertex AI** (billed to the
`nomadkaraoke` GCP project, ~$85/mo) onto the **Gemini Developer API**
using the `gemini-api-key` Secret Manager secret (an AI Studio key
billed outside this project). Once deployed and verified,
`aiplatform.googleapis.com` / `generativelanguage.googleapis.com` get
disabled in nomadkaraoke.

### Key injection
- `backend/services/gemini_client.get_genai_client(timeout_ms)` — key
from `GEMINI_API_KEY`, else Secret Manager `gemini-api-key` via
`settings.get_secret` (runtime SA already has project-level
`secretAccessor`). No `project`/`location`.
- Service: CI `--set-secrets ... GEMINI_API_KEY=gemini-api-key:latest`.
- Jobs: Pulumi secret env on `lyrics-transcription-job`,
`video-encoding-job` (renders + translations), `nomad-error-monitor` —
**already applied** (targeted `pulumi up`, only that env added). Other
jobs fall back to Secret Manager.
- Local scripts (`frontend/scripts/translate.py`,
`scripts/measure_autocorrect_cost.py`,
`scripts/test_agentic_correction.py`): env, else `gcloud secrets
versions access` (in-process only).
- **Rotation:** add a new secret version; new instances / job executions
pick it up. No code change.

### Degradation (quota / billing / invalid key; per-minute 429s stay
retryable)
| Caller | Behaviour |
|---|---|
| auto-correct | non-retryable 503 → compare mode drops the Gemini leg,
keeps Opus |
| match-judge / free-text / parse-titles / tenant bulk | existing
fallbacks (no suggestion / blanks / regex) |
| credit eval | `pending_review` (fail-closed, unchanged) |
| lyrics translation | no retry; render proceeds without translation,
status recorded |
| custom lyrics | clean 503 "temporarily unavailable…" (+ 5-min timeout
so it can't hang) |
| error-monitor LLM | no retry; alerts sent without AI summary |
| translate.py | exit 2 "Gemini quota/credit exhausted — top up AI
Studio or rotate gemini-api-key" |

`note_gemini_failure()` also sends one Discord ops alert per 6h
(Firestore-throttled across instances via new
`ops_alerts.send_throttled_ops_alert`).

### Infra
- Pulumi: removes `roles/aiplatform.user` from `karaoke-backend` —
**will be applied after this deploys and Vertex traffic is confirmed at
zero**.
- Agentic correction default model `gemini/gemini-3.8-flash`
(`vertexai/` prefix still accepted, routed to the Developer API).
- Version 0.261.1 → 0.262.0. Docs: CLAUDE.md, LESSONS-LEARNED, README,
ARCHITECTURE, API, etc.

## Testing
- New: `backend/tests/services/test_backend_gemini_client.py`
(classifier, key resolution, no-Vertex client kwargs, alert throttle,
per-caller degradation),
`tests/unit/lyrics_transcriber/test_agentic_gemini_provider.py`,
`frontend/scripts/test_gemini_client.py`, lyrics-translation quota test.
- `pytest backend/tests --ignore=backend/tests/emulator` — all pass;
`pytest tests/unit` — 2021 passed.
- Verified `gemini-3.8-flash` is listed on the Developer API and
answered a 5-token test call with the key.
- `translate.py --dry-run` on `frontend/messages` and
`backend/translations` OK.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Updates**
* AI-powered lyric translation, custom lyrics, auto-correction, title
parsing, match judging, and related features now use the Gemini
Developer API.
* AI features use the configured Gemini API key. Quota, billing, or key
issues may show an unavailable message or use an existing fallback
instead of repeated attempts.
* **Documentation**
* Updated product, API, setup, and recovery guidance to reflect Gemini
API key configuration and usage.
* Updated guidance for handling AI service availability and quota
issues.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
v0.262.0
2026-10-01 21:58:53 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 49464e8fcc chore(cost): GCP cost cuts round 2 (v0.261.4) (#1107)
Round 2 of the GCP cost cuts, approved by Andrew. Target: the whole
project under $300/mo; the current run-rate is about $400.

## Changes
- **karaoke-backend min-instances 1 → 0** (`ci.yml`), about $24/mo. The
backend Cloud Run service isn't defined in Pulumi, so the CI flag is the
only place this is set. Andrew accepts a ~15s cold start. The 60s
`/api/health` uptime check and the `*/10` schedulers keep an instance
warm most of the time. With `--cpu-throttling` (request-based billing),
a warm instance above the minimum costs nothing while idle.
- **audio-separator Cloud Run GPU service shut down** (us-east4), about
$22/mo. It is now behind the Pulumi flag `audioSeparatorServiceEnabled`,
off by default. The service, SA, IAM bindings, output bucket and AR repo
were already removed with a targeted `pulumi up`; the bucket needed the
break-glass SA. `audio-separation-job` is untouched; it uses
`karaoke-backend-gpu/karaoke-backend`. The separator health check no
longer depends on `AUDIO_SEPARATOR_API_URL`, which also fixes a status
card that always showed "offline". The CLI now logs a clear hint when
the remote API fails. Companion PR:
nomadkaraoke/python-audio-separator#309 makes its Cloud Run deploy
manual-only.
- **backup-to-aws no longer copies job files to S3**, about $15/mo
egress. Job files now rely on GCS versioning plus soft-delete.
Firestore, secrets, BigQuery, kn-data and git-bundle backups are
unchanged. The freshness monitor no longer checks `gcs/job-files/`, and
DISASTER-RECOVERY.md is updated.
- **Artifact Registry cleanup policies**, about $8–10/mo. Per package:
keep the 10 most recent versions plus anything tagged `latest`,
`content-` or `cache`; delete all other versions older than 7d. Until
now, tagged per-commit images were never deleted. Dropped the CI
fallback that copied the GPU base once from us-east4, and deleted the
legacy 13 GB us-east4 GPU base package.
- Docs: GCP-COST-OPTIMIZATION.md (new section, including how to redeploy
the separator on demand), ARCHITECTURE.md, DISASTER-RECOVERY.md.

Changes made outside Pulumi (gcf-artifacts cleanup policy, destroying
superseded secret versions) are recorded in GCP-COST-OPTIMIZATION.md.

## Tests
- `backend/tests/test_system_status_endpoint.py` (new separator status
tests)
- `infrastructure/functions/backup_to_aws` (49 tests, including a new
one that checks job files aren't synced)
- `tests/unit/test_audio_remote.py`, `test_audio_processor_local_gpu.py`

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Service Availability**
* The hosted audio-separation service is no longer available. Audio
separation can run locally; users relying on the hosted service will
need another deployment.
* **Backup and Recovery**
* New job files are no longer copied to off-site S3 backups. S3 recovery
is limited to files backed up through September 30, 2026; newer outputs
may need to be regenerated.
* **System Status**
* Audio-separator health status now reports the locally installed
package version without checking a remote service.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
v0.261.4
2026-10-01 21:22:49 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 e4dc50d9e3 fix(ci): sanitize PR title in deploy env vars (v0.261.3) (#1109)
The v0.261.2 main deploy (run 36945703908) failed at **Deploy to Cloud
Run (candidate revision)** with:

`argument --set-env-vars: Bad syntax for dict arg`

The #1106 PR title contained a comma, and `PR_TITLE` is spliced into the
comma-delimited `--set-env-vars`. The bug was already there and that
title was the first to trigger it. Its Cloud Build step succeeded.

Fix: in both build-meta steps, commas become `;`, and newlines and
shell-active characters (`\" \` $ \\`) are removed. Version bumped to
0.261.3.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
v0.261.3
2026-10-01 20:50:43 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 dd795c66a0 ci(cost): build backend images in Cloud Build, run deploy-backend on ubuntu-latest (#1106)
## Summary
Retires the self-hosted `gha-build-*` runner VMs (e2-standard-8,
`docker-build` label, ~$21/mo). `deploy-backend` now runs on free
`ubuntu-latest`; the Docker images are built by **Cloud Build in
us-central1**.

## Why not build on the hosted runner? (egress math)
The app images are built `FROM` bases in us-central1 Artifact Registry,
and the app layer has a `RUN pip install`, so the full base must be
pulled each deploy:
- CPU base `karaoke-backend-base`: **4.22 GB** compressed
- GPU base `karaoke-backend-gpu-base`: **13.1 GB** compressed
- ~120 main deploys/month (119 in Sept, 11 so far today)

On a GitHub-hosted runner that is GCP internet egress: 17.3 GB × 120 ×
$0.12/GB ≈ **$250/mo** (CPU-only would still be ≈ $61/mo). Both are
worse than the $21 VM.

Cloud Build in us-central1 pulls same-region (free). Build-minutes on
the default-pool e2-standard-2 machine are in the **2,500 min/month free
tier** ($0.006/min after that). A test build of the real images (CPU and
GPU in parallel) took **6m55s**, about the same as today's ~7.3 min of
sequential builds on the VM. That's ~840 min/mo, so the expected cost is
~$0 (worst case ~$5/mo).

## Changes
- `infrastructure/cloudbuild/backend-images.yaml` +
`build-backend-images.sh`: buildx (docker-container) builds with the
**same tags and registry caches** as before. CPU and GPU build as
parallel steps, and each build gets a fresh metadata-server token.
- `ci.yml` `deploy-backend`: `runs-on: ubuntu-latest`. It keeps the hash
checks (manifest + ~10 KB config blob only) and submits `gcloud builds
submit --async`, then streams the logs and checks the status.
- Base rebuilds (≈ monthly) use e2-highcpu-8 with 300 GB disk and a 2 h
timeout.
- A Cloud Build still running when the job is cancelled or fails is
cancelled. CI concurrency cancels superseded deploys, and an orphaned
build would otherwise still push `:latest`.
- Removed the disk-space step, the local base pull, and the one-time
us-east4→us-central1 GPU-base copy. The copy finished 2026-09-26, and on
a hosted runner it would stream 13 GB through egress.
- `build-runner-images.yml`: `variants=all` no longer bakes the unused
`build` family. It can still be built with an explicit `variants=build`.
- Docs: EPHEMERAL-GHA-RUNNERS, SELF-HOSTED-RUNNERS,
GCP-COST-OPTIMIZATION.

No Pulumi or IAM changes were needed. `github-actions-deployer` already
has `cloudbuild.builds.editor`, `storage.admin` and
`iam.serviceAccountUser`, and the default Cloud Build SA (compute
default) already has `artifactregistry.writer` on both repos.

## Rollback (one commit)
`git revert <squash-merge sha of this PR>` and push to main. That
restores `runs-on: [self-hosted, linux, gcp, docker-build]` and the old
in-job buildx steps. Nothing outside the repo was changed: the
runner-manager `build` family is untouched, and the `gha-runner-build`
GCE images (20260812, 20260901) are kept so the dispatcher can still
launch `gha-build-*` VMs. If the build image has been pruned by then,
re-bake it with Actions → Build GHA Runner Images → `variants=build`
(only `variants=all` no longer includes it).

## Testing
- I ran the real CPU and GPU builds from this branch's config in Cloud
Build us-central1, with `cbtest-*` tags and no cache writes. Both
succeeded in 6m55s, and the us-east4 push uploaded only the app layers.
- `actionlint` and `shellcheck` report no new findings.
- Full verification is the first main deploy after merge.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Improvements**
* Backend image builds now run through a managed build service in the
central US region, with CPU and GPU images published as part of the
deployment workflow.
* The default “all variants” selection now excludes the build variant;
explicitly requested variants remain available.
* **Documentation**
* Updated runner and cost documentation to reflect the current build and
deployment setup.
* **Maintenance**
  * Updated the package version to 0.261.2.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
v0.261.2
2026-10-01 20:22:59 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 17d60c58a0 fix(cost): low Gemini thinking for simple callers + dedupe proactive auto-correct (#1105)
## Summary
GCP cost cut: Vertex Gemini 3.8 Flash is now the largest variable cost
(~$85-90/mo), mostly thinking tokens.

### 1. `thinking_level="low"` for simple extraction/classification
callers
| Caller | Model setting | Before | After |
|---|---|---|---|
| `match_judge/ai.py`, `match_judge/free_text.py` | `match_judge_model`
| default (high/dynamic) | low |
| `tenant_bulk/analyze.py` (filename pairing) | `match_judge_model` |
default | low |
| `parse_titles/ai.py` | `parse_titles_model` | default | low |
| `credit_evaluation_service.py` (grant/deny) | `credit_eval_model` |
default | low |
| `frontend/scripts/translate.py` | n/a | medium | low |

**Unchanged on purpose:** auto-correct (quality eval), custom lyrics,
lyrics translation, and agentic correction (`agentic_ai_model` is
dormant because `SKIP_CORRECTION=true`; its router makes no LLM call).

Timeouts: low thinking only makes these calls faster, so the existing
deadlines (`match_judge_timeout_ms` 12s, `parse_titles_timeout_ms` 20s)
need no change.

**Live check (Vertex, ADC):** 8 match-judge cases plus a 4-file
parse-titles batch, default vs low. Same canonical artist/title on every
case and identical parses. One verdict changed: `beatles/hey jude` went
from content to cosmetic, and cosmetic is the correct label for a
casing/"The" fix. Thinking tokens dropped from 1519 to 167, and max
latency from 5.5s to 2.7s.
**translate.py A/B:** 5 ICU-plural/placeholder strings into es/de/ja/ar.
Low used 0 thinking tokens and medium used ~33k. Both preserved all
placeholders and produced equivalent quality.

### 2. Duplicate proactive auto-correct (job 96cf1100 and about 55% of
jobs)
Root cause: the lyrics worker calls `advance_to_screens_if_ready`
*before* `_trigger_proactive_auto_correct`. The screens worker's
`ensure_and_pre_apply` checks the suggestion cache while the proactive
run is still in flight, gets a miss, and starts a second identical
multi-model run (Claude Opus plus Gemini). In job 96cf1100 both runs
were on the same instance and finished 4s apart. From prod logs for
2026-09-17 to 10-01: **92 of 167 jobs generated twice** (9 of them
across instances), at about $0.07 per duplicate.

Fix (`backend/workers/auto_correct_worker.py`):
- In-process single-flight per job. A concurrent caller awaits the
in-flight task, shielded so it can't cancel it.
- Cross-instance GCS lease
(`jobs/{id}/lyrics/auto_correct_inflight.json`) with generation fencing
(create-only; stale takeover and release only against the observed/own
generation; owner token; 240s TTL; timed-out run keeps its lease). A
non-holder polls off the event loop for the holder's cache for up to
150s instead of regenerating. If the holder fails, the UI's existing
on-demand fallback takes over.

## Tests
- `backend/tests/test_proactive_auto_correct.py`: concurrent calls share
one run (fails on old code), waiting on a peer's lease, taking over a
stale lease, and not regenerating after a peer fails.
- `backend/tests/test_gemini_thinking_level.py`: each changed caller
sends `thinking_level=low`.
- Related suites pass locally. `test_internal_api.py` has 7 failures
locally that also fail on `origin/main` (local env).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Improvements**
* AI-powered features now use a lower thinking level for tasks such as
title parsing, match evaluation, credit evaluation, bulk analysis, and
translation.
* Proactive auto-correct now coordinates concurrent suggestion requests,
reuses completed suggestions, and avoids starting a duplicate run when
another request is already in progress.
* If another request ends without producing suggestions, auto-correct
skips the request rather than immediately generating them again.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 19:35:22 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 7e51ba7cc1 fix(email): retry transient Postmark failures; diagnose missing E2E emails (v0.261.1) (#1104)
## Summary
E2E Daily #190 (2026-10-01) failed in Stage 1 because Postmark rejected
all 3 magic-link sends with a bare **nginx HTML 403**. That's an
edge/IP-level block, not Postmark's JSON error format. Since 2026-09-30
05:26 UTC there have been 11 such failures from `karaoke-backend`
(us-central1), in bursts. They also dropped real users' magic links and
a lyrics reminder. `video-encoding-job` (us-east4) was never affected.

## Changes
- **`PostmarkEmailProvider` retry**: 429, 5xx, non-JSON 403 and connect
timeouts are retried with backoff (1s, 3s; 3 attempts total).
- JSON Postmark errors (bad token, invalid request, suppressed
recipient) are not retried: they're deterministic.
- Read timeouts are not retried: Postmark may already have accepted the
email and has no idempotency key, so a retry could send it twice.
- **E2E `waitForEmail` diagnosis**: on timeout, it queries `GET
/api/admin/users/{email}/emails` and reports either "backend has NO
record of sending" (a server-side send failure) or "backend sent N
email(s) but none reached testmail.app". This replaces the misleading
testmail `AbortError`. Applies to every spec that waits for an email.
- `docs/TROUBLESHOOTING.md`: new section on Postmark HTML 403s.
- Version 0.261.0 → 0.261.1.

## Testing
- New `TestPostmarkRetry` unit tests: HTML 403 then success, giving up
after 3 attempts, 429/5xx, no retry on JSON 403/422/406, connect timeout
retried, read timeout not retried.
- `make test` passes: backend 2017 + 5436 + 150; frontend Jest 1591.
- Diagnosis helper: every branch checked with a stubbed `fetch`. Against
the real prod endpoint, today's failed E2E address shows 0 emails and an
address from a passing run shows 2.

## Not in this PR
The underlying block is probably IP-based. If retries don't absorb it,
the follow-ups are Postmark support and/or a static outgoing IP (Direct
VPC egress + Cloud NAT).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Email delivery now automatically retries certain temporary service and
connection failures, with brief delays between attempts.
* Email wait timeouts in end-to-end testing now include additional
diagnostic details, and temporary polling errors no longer stop checks
early.

* **Documentation**
* Added troubleshooting guidance for missing emails, including how to
distinguish delivery blocks from authentication errors and what to check
when retries do not resolve the issue.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.261.1
2026-10-01 18:59:19 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 d4fcab521c Translated lyrics follow-ups + review Waveforms 'separating' spinner (v0.261.0) (#1103)
## Summary
Follow-ups from Andrew's test of translated lyrics (job 948b6760), plus
a review-page UX gap found while investigating that job.

- **Translation row is now secondary:** it renders at 70% of the unsung
colour's opacity, with outline and shadow faded to match. A theme's
`karaoke.translation_color` still wins.
(`subtitles.py:TRANSLATION_OPACITY`)
- **YouTube label:** published videos with translations get "(With
Translation into X)" in the title, before "(Karaoke)". Only the
artist/title get truncated, never the label. The description gets an
up-front notice. This covers the live upload and the quota queue, and
applies only when `state_data.lyrics_translation.status ==
"translated"`. Turning translation off and re-rendering clears the
outcome.
- **Review preview modal:** an always-visible note explains the preview
is a quick black-background sync check, not the final look. Jobs with a
translation language also get a note that translations are added after
review.
- **Waveforms toggle:** shows a spinner and tooltip while audio
separation is still producing the vocal stem (peaks endpoint returns
202). Previously the rows just stayed empty.

## Testing
- Jest: lyrics-review suite (245+) incl. new
TranscriptionView/Preview/loader tests
- pytest: youtube description/queue/orchestrator, lyrics_translation,
translated-lyrics unit tests
- Linux libass render tests (Debian + prod static ffmpeg) pass
- Independent review agent: 2 findings fixed (Tooltip data-state
clobbering toggle state; stale translation outcome)

No local CodeRabbit run (CLI has no seat on this org), so leaving this
open to the CodeRabbit bot.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* YouTube titles and descriptions now identify the language of
translated lyrics. Titles are kept within the platform’s length limit.
  * Translated lyrics appear with softer opacity by default.
* The lyric-review screen explains how the quick preview differs from
the final video, including that translated lyrics are added after
review.
* A spinner and notice indicate when vocal waveforms are still being
prepared.

* **Bug Fixes**
* Translation status now reflects when translation is turned off,
preventing stale translated-language labels.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.261.0
2026-10-01 16:39:56 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 bc1fc774eb Translated lyrics: don't split Japanese/Chinese translations mid-word (v0.260.1) (#1102)
## Summary
Follow-up to #1101, found while verifying it on prod (job ca2e881a,
English song → Japanese): when a long lyric line is split across rows,
its translation is shared between the pieces, and unspaced scripts were
cut at an arbitrary character — e.g. 無|駄 ("waste") across two rows,
worse in portrait.

`split_text_proportionally` now scores each split point for unspaced
scripts: free after punctuation (、。!?…), cheap after a Japanese particle
(を は が に…) or before a kanji/katakana word start following hiragana, a
full "part" of cost for a mid-word cut. It also never leaves a piece
empty when there are enough characters.

Before → after (real prod line): `ねえ、おかしいでしょ、無` / `駄な時間を省いてあげる` →
`ねえ、おかしいでしょ、` / `無駄な時間を省いてあげる`.

Thai can still split inside a word (no word-boundary data without a
dictionary).

## Testing
- New unit tests for the prod case and a particle-boundary case; all 201
lyrics output tests pass.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Improved how translated lyrics are split across rows for unspaced
text. Splits now favor punctuation and natural language boundaries,
helping avoid cutting through words where possible.
  * Thai text may still be split within words.
* **Chores**
  * Updated the project version to 0.260.1.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
v0.260.1
2026-10-01 14:39:13 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 f8d0febcf5 Translated lyrics beneath the karaoke lyrics for language learners (v0.260.0) (#1101)
## Summary
New job-submission option: **Add translated lyrics** → pick a language
(33 UI locales) → the karaoke videos show an AI translation of each
lyric line in smaller text beneath the synced lyrics, so language
learners can sing songs in another language while learning what the
words mean.

- **Job creation**: toggle + language picker (default = UI language) on
the Customize step, with a **live preview** of a real rendered frame
(`GET /api/themes/translation-preview?language=xx`; pre-generated sample
translations, no LLM call at preview time). Any job, public or private,
free for now. `translation_language` accepted by every creation route;
admin can edit it.
- **Post-review translation**: right after the job enters
RENDERING_VIDEO, the render worker translates the *final reviewed*
lyrics line by line with Gemini
(`backend/services/lyrics_translation.py`, `LYRICS_TRANSLATION_MODEL`,
GCS-cached so theme/admin re-renders are free) into
`jobs/{id}/lyrics/translations.json`. Not editable. A failure never
blocks the render (renders without translations, recorded in
`state_data.lyrics_translation`); a song already in the target language
gets no row.
- **Rendering** (landscape + portrait, GCE worker + local fallback):
`LyricsSegment.translation`; when present, slots get taller and screens
show 3 lines; a `Karaoke.Translation` style (unsung colour, ~½ size)
draws one static event beneath each line, sanitized, wrapped/scaled to
fit. Jobs without translations render identically.
- **RTL + LTR together**: the translation is its own ASS event with auto
paragraph direction, so e.g. a Hebrew song keeps its right-to-left
highlight with an English translation beneath (and vice versa). Long
lines split by the resizer share their translation across the pieces
(grapheme-safe for Thai/CJK).

Design: `docs/archive/2026-10-01-translated-lyrics-plan.md`.

## Testing
- `make test` all green (root 2014, backend 5416, emulator 150, frontend
1583).
- New: `tests/unit/lyrics_transcriber/output/test_translated_lyrics.py`
(layout/positions/style/sanitizing/fit/mixed RTL-LTR/Thai/portrait),
`test_translated_lyrics_render.py` (libass pixel tests incl.
Hebrew/Arabic lyrics with English/Arabic translations and CJK),
`backend/tests/test_lyrics_translation.py`,
`backend/tests/test_render_video_worker_translations.py`, GCE endpoint
test, frontend CustomizeStep/GuidedJobFlow/api tests, prod E2E
`e2e/production/translated-lyrics.spec.ts`.
- `scripts/run-render-tests-linux.sh`: 135 passed on Debian ffmpeg AND
prod's static ffmpeg (no skips).
- Real-data check: job 5710831e (Hebrew) translated to English by Gemini
and rendered with the nomad theme — RTL highlight intact, English
beneath, split line shares translation.

Local CodeRabbit CLI unavailable in this workspace (no seat), so leaving
this PR for the CodeRabbit bot; an independent local review was done and
its findings fixed in the second commit.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added an opt-in translated-lyrics option to job setup, with language
selection and a preview.
* Translations are generated after lyric review and appear beneath
lyrics in landscape and portrait videos, with text layout for multiple
languages and writing directions.
  * Translation settings can be changed or cleared by admins.
* **Bug Fixes**
* If translation fails or is unavailable, video rendering continues
without translated lyrics.
* **Documentation**
* Updated API and product documentation with translated-lyrics setup and
behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
v0.260.0
2026-10-01 13:41:54 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 b7a3864a37 Missing-lyrics insert: keep synced-reference timings inside the gap (v0.259.1) (#1099)
## Summary
Follow-up to #1098, found while fixing job 5710831e.

LRCLIB references are line-synced, and the last reference line in a gap
can end **after** the next transcribed line starts. On 5710831e it ends
at 33.07s while the next line starts at 32.40s. "Insert these lines"
copied those timings verbatim, so the inserted words overlapped the
following line.

`fitTimedLinesToWindow` clamps each inserted timed line to the window
between the neighbouring lines' words, with a 0.05s pad:
- A line that fits keeps its exact reference timing.
- An overflowing line is compressed affinely, keeping its word
proportions.
- A line with no room left falls back to **untimed**, so the existing
submit guard forces a Tap To Sync.

## Testing
- 4 new unit tests, including a 5710831e regression using the real
numbers: inserted words never overlap the next line, lines that fit keep
exact timing, push-after-previous, the untimed fallback, and untimed
lines passing through unchanged.
- Jest: 1561/1561.
- Playwright missing-lyrics scenarios: 4/4.

No local CodeRabbit review (the CLI has no seat for this org), so this
is left open for the CodeRabbit bot.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Timed lyrics inserted into gaps now stay within the available interval
and avoid overlapping neighboring lyrics. Timings are preserved when
they already fit; overflowing lines are adjusted to fit when there is
enough room.
* If there isn’t enough room to assign valid timings, or the original
timing has no duration, inserted lyrics are left untimed instead. Lyrics
that are already untimed continue to pass through unchanged.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
v0.259.1
2026-10-01 11:47:52 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 08b574a86c Review UI: flag possible missing lyrics and insert the reference lines; mount sonner toasts (v0.259.0) (#1098)
## Summary
Phase 2b of the missing-lyrics work (#1095 detector, #1097 gate). When
the job's `vocal_gaps` analysis has an **evidenced** gap (the lead vocal
is singing, there are no lyrics, and the reference lyrics place lines
there), the lyrics review UI now shows:

- **An amber callout:** "Possible missing lyrics at 0:20–0:32 — the lead
vocal is singing here but there are no lyrics." It lists the expected
lines and their source (synced reference lines first), with **Play** and
**Insert these lines** buttons.
- **A marker row** at the gap's position in Synced Lyrics (Simple,
Advanced and Waveforms views).

**Insert these lines:**
- Adds one segment per reference line, at the right point in time.
- If the gap falls inside a segment, that segment is split.
- In duets, the singer follows Add Segment's rule.
- It's one undo step, logged in the edit log with `origin:
missing_lyrics_gap`.

**Timing of inserted words:**
- If a **synced reference** (e.g. LRCLIB) has the same line starting
inside the gap, its word timings are used.
- Otherwise the words go in **untimed**, so the existing "words have no
timing" submit guard forces a Tap To Sync. Timings are never invented.
- A re-sync hint lists inserted lines with **Sync timing** buttons that
open the existing Edit modal / Tap To Sync. It accumulates across gaps.

**When the marker hides:**
- When words exist inside the gap, with a 0.05s edge tolerance for the
backend's rounding.
- When the analysis doesn't match the lyrics as loaded (stale guard).
- For audio-only gaps, which are too noisy to show.
- In read-only/replay mode, the marker shows but Insert is disabled,
with an accessible explanation.
- The displayed range and Play use the review's timing offset.

**Also fixed (pre-existing bug):** sonner's `<Toaster>` was never
mounted, so **every sonner toast was silently dropped**. That covers
lyrics and instrumental review and the admin pages, and includes the
untimed-words submit guard this feature relies on. It's now mounted in
the root layout, and the guard message is translated (it was hardcoded
English).

## Testing
- Jest: 51 tests for the feature (gap detection with tolerance,
ordering/split, untimed vs synced timing, submit guard counting, duet
singer, offset, stale guard, read-only tooltip, memoisation, re-sync
hint).
- Playwright (`e2e/regression/lyrics-review.spec.ts`), 4 scenarios:
untimed insert → pending sync → undo; Sync timing opens Tap To Sync;
LRCLIB-timed insert closes the gap → undo; **blocked submit shows the
toast** (verified to fail without the Toaster mount).
- `make test`: frontend 1557/1557 plus all backend stages.
- i18n: all new strings in 33 locales.
- Two local code-review rounds; findings addressed.

No local CodeRabbit review (the CLI has no seat for this org), so this
is left open for the CodeRabbit bot.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Lyrics review now highlights vocal gaps that may be missing lyrics,
with time ranges and available reference lines.
* Insert suggested lines into the transcript, undo the change, and open
inserted lyrics in the timing editor. Playback and timing-sync options
help review each gap.
* **Bug Fixes**
* Video submission now warns when lyrics contain untimed words that need
synchronization.
  * Improved accessibility labels for undo and redo controls.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
v0.259.0
2026-10-01 00:41:53 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 777daaf25f Missing-lyrics gate: block auto-approval when singing has no lyrics and the reference has lines there (v0.258.0) (#1097)
## Summary
Phase 2a of the missing-lyrics work (phase 1: #1095, shadow detector +
backfill).

**Audit** (262 completed jobs, 60 days, server-side
`scripts/audit_vocal_gaps.py`):
- Longest unlyricked vocal run per job: p50 0.7s, p90 5.1s, p99 22.4s.
47 jobs (18%) have a run of 3s or more, far too many to gate on audio
alone.
- Cross-checked against **synced** reference lyrics (LRCLIB line
timestamps): of 57 suspect gaps that have a synced reference, only 15
contain a reference line. Audio-only is ~25% precise. The rest are vocal
chops, samples, ad-libs and synth bleed, mostly in drum & bass tracks.
- **Evidence-based rule:** a gap is reliable when the reference lyrics
place lines in it.

**Changes**
- `vocal_gaps` 0.2.0: records the synced reference lines inside each
gap. `evidenced` = suspect vocal run AND reference lines, either synced
or between the neighbouring anchors.
- Scorer 0.5.0: new never-auto `missing-lyrics-gate` with a readable
reason, e.g. *"possible missing lyrics: ~10s of singing at 0:20-0:32 has
no transcribed words, and the reference lyrics have 3 line(s) there"*.
Audio-only gaps don't gate.
- The executor analyzes vocal gaps *before* scoring (off the event loop,
fingerprinted cache) and passes them to both scoring passes. It fails
open on pending audio or analysis errors.
- `GET /api/review/{id}/correction-data` now includes `vocal_gaps`, for
the upcoming review-UI marker (phase 2b).

On job 5710831e, the synced LRCLIB lines inside the gap are exactly the
three dropped lines. The genuine instrumentals have none.

## Testing
- `test_vocal_gaps.py`: synced lines with edge tolerance; section
headers ignored; evidence requires both singing and reference
(instrumental with misplaced reference lines doesn't gate; singing
without reference doesn't gate).
- `test_scorer.py`: reasons only for evidenced gaps; the gate overrides
confident lyrics; non-evidenced gaps don't gate.
- `test_executor.py`: evidenced gaps block auto-approval with the reason
and skip the timing check; cached evidenced gaps still gate.
- `test_combined_review_flow.py`: `vocal_gaps` is included in
correction-data.
- Full backend suite: 5211 passed.

Versioned 0.258.0 to follow #1096 (0.257.0).

No local CodeRabbit review (the CLI has no seat for this org), so this
is left open for the CodeRabbit bot.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
v0.258.0
2026-09-30 23:56:48 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 f89de165cc Admin: re-render any completed job without review (v0.257.0) (#1096)
## Summary

An admin-only **Re-render** button on completed jobs in the `/app` jobs
list. It regenerates title/end cards, the karaoke video, all encoded
formats, packaging and distribution from the job's existing reviewed
lyrics, instrumental selection and style snapshot, **with no
lyrics/instrumental review**. Motivating case: re-rendering
already-delivered videos after a renderer fix (RTL/Hebrew, v0.255.0).

`POST /api/admin/jobs/{job_id}/rerender` with body `{"notify_customer":
false}`.

**Behaviour (decided with Andrew)**
- Works for consumer and tenant jobs, public or private. Uses the job's
existing style; no theme re-snapshot.
- **Published outputs are deleted up front, like Edit (YouTube, GDrive,
Dropbox), but the brand code is kept, not recycled.** The pipeline
re-publishes under the same NOMAD-xxxx. YouTube gets a new upload and
therefore a new URL.
- Outputs are only deleted for destinations the re-render will actually
re-publish (`plan_republish` uses the same conditions as the video
worker). Anything else is kept, with a warning shown to the admin.
- **No customer email/push/Discord by default.** The confirm dialog has
an "Email the customer when done" checkbox. Already-notified
requests-board voters aren't re-emailed; voter emails still owed from
the original publish do go out.
- A failed admin re-render is set to FAILED with an in-progress marker
(scoped to `review_token`, so it can't leak into later runs). It can be
retried by **admins only**; `/retry` returns 403 to customers while the
marker is active.
- **Mutual exclusion with the YouTube upload queue:** the claim and the
queue's `mark_processing` are Firestore transactions that each read both
documents. A re-render is refused with 409 while an upload is in flight.
A queued or failed deferred upload is cancelled atomically when YouTube
will be re-published. The queue never starts an upload for a job that is
mid re-render.
- After a successful upload, the queue never re-queues an entry, so a
later error can't cause a duplicate upload.
- Admin re-render completions don't increment the customer's
completed-jobs count.

**Refactors**
- New `backend/services/published_outputs_cleanup.py`, shared by Edit,
admin delete-outputs and the re-render.
- The Dropbox folder delete now uses the uploader's sanitised name and
also tries the legacy raw name. This is a deliberate fix for titles with
special characters.
- Remaining duplicate copies (`jobs.py` ~2440,
`visibility_change_service`) are listed as follow-ups in the design doc.

## Testing
- 173 tests in `test_admin_rerender.py` +
`test_published_outputs_cleanup.py`, built through three local
code-review rounds. They cover:
  - claim and validation
  - retry gating
  - marker scoping
  - publish planning (each mismatch case)
- queue exclusion in both directions, no starvation, and that the
re-render's own later upload is never cancelled
  - the no-re-queue-after-upload rule
  - notification gating on both pipelines
  - community voter cases plus reconcile
  - restart atomicity
  - legacy Dropbox naming
  - tenant re-render and Edit unchanged
- Frontend Jest: button visibility (admin vs non-admin, complete vs
not), dialog, payload, and warnings toast.
- `make test`: backend 1972 + 5372 + 150, frontend 1506.
- i18n: new strings in all 33 locales (frontend) plus backend
translations.

Docs: `docs/API.md`,
`docs/archive/2026-10-01-admin-rerender-any-job.md`.

No local CodeRabbit review (the CLI has no seat for this org), so this
is left open for the CodeRabbit bot. Three local code-review rounds ran;
their findings are addressed in the fix commits.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Administrators can re-render eligible completed tracks without another
review, retaining existing track settings and brand code.
* Choose whether to notify the customer; notifications are off by
default. Published outputs are replaced when applicable, and warnings
identify outputs that remain unchanged.
* Administrators can resume eligible failed or cancelled re-renders.
Re-render controls and status messages are available in multiple
languages.
* **Bug Fixes**
* Improved cleanup and handling of published files and queued YouTube
uploads during re-renders, with safer retries and clearer failure
reporting.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
v0.257.0
2026-09-30 23:37:51 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 e2f7dbd13c Detect sung stretches with no transcribed lyrics (shadow) + backfill endpoint (v0.256.0) (#1095)
## Summary

Transcription can silently **drop whole sung lines**. On job 5710831e
(Hebrew), the words stop at 20.66s and resume at 32.40s while the lead
vocal is ~96% active, and both reference sources have three pre-chorus
lines there. `SectionDetector` (gap ≥ 10s, no audio check) then rendered
"♪ INSTRUMENTAL ♪" over the singing. Nothing caught it:
- the scorer's anchor coverage measures how much of the *transcribed*
text matches the reference, so omissions don't lower it (98.8% here);
- `timing_check` G3 (`max_unclaimed_run_s`) does fire on this job but is
shadow-only and only computed for AUTO-bound jobs;
- the review UI shows nothing at the gap.

This PR is **phase 1: a shadow signal plus server-side backfill for
calibration**. It changes no verdicts or rendering.

- `backend/services/auto_approval/vocal_gaps.py`: for every
transcription gap ≥ 3s, records the lead-vocal active fraction, the
longest breath-bridged vocal run (with a 1.5s allowance for a held note
carried over from the previous word), and the reference lines between
the neighbouring words' reference positions. Section headers are dropped
and ±1-word anchor drift at line boundaries is snapped. `suspect` = run
≥ 3s.
- The executor analyzes every job once stems exist (any verdict), off
the event loop, and stores `state_data.vocal_gaps` keyed by an
`input_key` fingerprint so resets/edits recompute. A summary goes in
`processing_metadata.auto_approval.vocal_gaps`.
- `POST /api/internal/jobs/{id}/vocal-gaps[?dry_run=true]` (admin) plus
`scripts/audit_vocal_gaps.py` for backfill/calibration. Cloud Run does
the stem work.

**On the real job:** one suspect gap, 20.66–32.40s (10.26s run after the
allowance), whose reference lines are exactly the three missing Genius
lines. The three genuine instrumentals have ≤ 0.16s runs.

## Testing
- `test_vocal_gaps.py` (25 tests, synthesized audio + synthetic
correction data):
  - dropped lines vs instrumental vs ad-lib vs held note
  - a short dropped line diluted by a long gap
  - breath bridging
  - silent stem = error
  - malformed reference data
- reference snapping (off-by-one, single-word line, headers, reversed
chorus mapping)
- Executor tests: stored without changing the outcome; analyzed for
review-bound jobs; cache hit only for the same `input_key`, stale
recompute; pending audio / no stem / error / exception all non-fatal;
backfill uses the reviewed lyrics + original references; dry run.
- Internal endpoint tests (store / dry_run / admin).
- Full backend suite: 5199 passed.
- A local code review's findings were addressed in the second commit:
event loop, stale cache, silent stem, held notes, reference robustness,
audit ordering.

## Next (after deploy)
Run the audit → hand-check suspects → choose thresholds → phase 2:
scorer REVIEW gate with an explicit reason, no fake INSTRUMENTAL over
vocals, review-UI marker with one-click insert of the reference lines.
Also noted: anchor reference ids look shifted by one word at `[Section]`
header boundaries. Separate issue, documented in the plan doc.

No local CodeRabbit review (the CLI has no seat for this org), so this
is left open for the CodeRabbit bot.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added vocal-gap analysis to identify stretches of missing lyrics that
overlap with lead vocals. Results are saved with job data and can be
requested through an admin-only internal API.
* Analysis also runs automatically during auto-approval when vocal stems
are available. It does not change job status or approval decisions.
* **Documentation**
* Documented the internal API, its dry-run option, and the analysis
result statuses.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
v0.256.0
2026-09-30 22:10:47 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 354e43bab2 Render Hebrew/Arabic karaoke right-to-left; fix title-card tofu (v0.255.0) (#1094)
## Summary

A Hebrew KaraokeHunt job (5710831e, Omer Adam – "שני משוגעים") shipped
with its lyrics **laid out and highlighted left-to-right** and a **title
card of "?" boxes**. Diagnosed by measuring rendered pixels (not by
eye), reproduced locally and with prod's static ffmpeg 7.0.2 in Docker.

**Root causes**
- ASS style `Encoding=0` makes libass force LTR paragraphs; the `{\kf}`
before each word splits the line into runs, so each word is shaped
correctly but the words go left→right.
- `\kf` always wipes left→right within a word
([libass#406](https://github.com/libass/libass/issues/406)).
- PIL (title/end cards) has no per-glyph font fallback; the theme font
(Avenir Next) has no Hebrew glyphs.

**Fixes**
- `build_karaoke_styles` forces `Encoding=-1` (libass auto paragraph
direction; LTR unchanged).
- RTL lines get `{\frz(180+angle)\frx180\fry180}` — a visual no-op that
makes libass ≥0.15 sweep fills right→left without splitting words
(Arabic still joins). Theme angle preserved; re-emitted after duet
`{\r}` resets.
- Lead-in indicator mirrored for RTL lines.
- Text measurement mirrors libass: per-glyph fontconfig fallback, size
scaled from each font's metrics (old fixed 0.70 was 20%+ off for
fallback fonts).
- New `karaoke_gen/utils/font_fallback.py` (fontTools cmap +
fc-match/fc-list) used by title/end cards, portrait headers and ASS
measurement.
- Found by the new tests: inter-word gaps ≤0.1s were dropped, so the
highlight ran up to ~0.2s early by line end (all languages). Karaoke
tags now derive from one absolute centisecond timeline.
- Fixed pre-existing `test_render_video_worker_gce` failure on main
(MagicMock generation → `_g<mock>` id).

## Testing
- **Pixel-level render tests** (`test_ass_render_highlight.py` +
`ass_render_harness.py`): real `SubtitlesGenerator` output rendered with
ffmpeg/libass in a pure-colour palette, asserting on frames for English,
Hebrew, Arabic, Chinese, Japanese, Korean + an RTL duet line: highlight
starts at the reading-start edge, sung pixels always precede unsung in
reading order (word order + in-word fill direction), monotonic and in
time with the words, lead-in on the correct side, no missing glyphs, RTL
tags don't move the line (incl. rotated themes).
- 32 fail on pre-fix main; mutation checks ("Encoding only", "no re-emit
after `\r`", "ignore theme angle") each fail exactly the expected cases.
- 123/123 pass on Linux against **both** Debian ffmpeg (CI-equivalent)
and **prod's johnvansickle static ffmpeg** via
`scripts/run-render-tests-linux.sh`.
- CI now installs `fonts-noto-core fonts-noto-cjk`; under `CI=true`
these tests fail rather than skip on missing preconditions. (macOS
libass uses CoreText → font-sensitive checks skip locally.)
- Title-card tests (`test_font_fallback_render.py`): no `.notdef` tofu
for Hebrew/Arabic/CJK titles; Hebrew title reads right-to-left.
- Unit tests: `test_rtl_ass_output.py`, `test_font_fallback.py`.
- `make test`: unit 1972 passed; emulator 150 passed; frontend 1491
passed.
- Full local re-render of job 5710831e reviewed by Andrew: RTL highlight
smooth and correct, title card correct.

## Follow-ups (not in this PR)
- CDG output still renders RTL left-to-right / tofu (separate renderer).
- Separate investigation: transcription dropped 3 sung lines in this
job, which the section detector then labelled INSTRUMENTAL.

No local CodeRabbit review (CLI has no seat for this org), so leaving
this open for the CodeRabbit bot. A local code review ran; its findings
are addressed in the second commit.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Karaoke lyrics now render more reliably for Hebrew, Arabic, Chinese,
Japanese, and Korean, with improved reading-direction highlighting,
timing, and lead-in placement.
* Titles and portrait text can use fallback fonts when the selected font
lacks the necessary characters.

* **Bug Fixes**
* Improved lyric centering, word highlighting, and glyph rendering
across supported scripts.

* **Documentation**
  * Added guidance on lyric rendering and multilingual test coverage.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
v0.255.0
2026-09-30 21:02:53 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 e15a485007 Re-render: refresh tenant Dropbox archive in place instead of rejecting (v0.254.1) (#1093)
## Summary
Follow-up to #1091, found by testing it in prod. randy-vild jobs are
archived to Dropbox under a brand code (`RVILD-0001 - Randy Vild - What
Goes Up`). The "never distributed" guard therefore rejected the very
jobs the Re-render button is for (`POST /api/jobs/2579a1ea/rerender` →
400 "published to YouTube/Dropbox/Drive").

- Dropbox-archived tenant jobs can now be re-rendered. The re-render
stores the job's `brand_code` in `state_data.theme_rerender`, and the
orchestrator config and native distribution use it as `keep_brand_code`.
The same folder is overwritten (Dropbox `WriteMode.overwrite`) and no
new code is allocated.
- The code is scoped to the re-render marker (cleared on success), not
written to the job's `keep_brand_code` field. That way a later Edit,
which recycles the brand code, can't reuse it.
- YouTube/GDrive-published jobs are still rejected. The frontend hides
the button only when a YouTube link shows.

## Testing
- Backend 5158 passed, 36 skipped; `test_theme_rerender.py` 35 cases,
including Dropbox-archived jobs being re-renderable, the marker
recording the brand code, and the orchestrator keeping it.
- Frontend `OutputLinks.test.tsx` 28 passed.
- CodeRabbit CLI isn't available on this org, and the bot has been
rate-limited. This is a small change on top of the reviewed #1091.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
v0.254.1
2026-09-30 01:37:33 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 9085a3005e feat(tenants): configure Dropbox output folder in tenant setup (v0.254.0) (#1092)
## Summary
The Randy Vild tenant's completion email said "Here's the dropbox
folder…" then `[Dropbox URL not available]`: the tenant was created with
the (optional) Dropbox path left blank, so every job was download-only.
The admin form already had the fields, but nothing guided, validated, or
provisioned them.

### Changes
- **Tenant create/update** (`tenant_admin_service`): `dropbox_path` +
`brand_prefix` must be set together (the worker silently skips upload
otherwise); path normalized, prefix uppercased/validated (2-12 alnum)
and rejected if it's the consumer default (`NOMAD`/`NOMADNP`) or another
tenant's (brand-code counters are keyed by prefix alone). The Dropbox
folder is **created up front**; `features.dropbox_upload` always follows
the path.
- **DropboxService**: `ensure_folder()` (idempotent); `list_folders()`
treats a missing path as empty — previously the *first* brand-code
allocation for a new tenant would raise `not_found`.
- **Admin UI** (`/admin/tenants`): pre-fills
`/MediaUnsynced/Karaoke/Tracks-<Name>` + prefix from the name (Vocal
Star→VSTAR, Singa→SINGA, Randy Vild→RVILD), blocks half-set delivery,
warns that download-only tenants get no email link; table shows `PREFIX
→ path`.
- Update route maps conflict→409, Dropbox provisioning failure→502.

### Prod data fix (already applied, outside this PR)
- `randy-vild` tenant → `/MediaUnsynced/Karaoke/Tracks-RandyVild`,
`RVILD`; `brand_code_counters/RVILD` seeded at 3.
- 10 in-review Randy Vild jobs backfilled with
`dropbox_path`/`brand_prefix` (jobs snapshot these at creation).
- 2 completed jobs copied into `RVILD-0001 - Randy Vild - What Goes Up`
/ `RVILD-0002 - Randy Vild - Embers in the wind` (24 files each via
Dropbox `save_url` from signed GCS URLs),
`state_data.brand_code`/`dropbox_link` set.

## Testing
- `make test`: backend 5121 + 1846 + 150 passed, frontend 1476 passed
- New unit tests: tenant delivery validation/folder creation/prefix
conflicts, DropboxService ensure_folder + missing-path listing (real SDK
error types), admin page pre-fill + half-set block + download-only
warning, `suggestDelivery`/`deliveryError`.
- Local CodeRabbit review: 2 minor findings, both fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
v0.254.0
2026-09-30 01:22:19 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 31a8838b07 Tenant: re-render tracks with the current theme; edited tenant tracks keep their downloads (v0.253.0) (#1091)
## Summary
- **Bug: finished tenant track showed no download buttons** (randy-vild
job 2579a1ea). The Edit flow sets `outputs_deleted_at`, and it was only
cleared when the re-render uploaded to YouTube/Dropbox/GDrive. Tenant
jobs never distribute, so every download stayed hidden after the
re-render. A customer's private job (6452888e) had the same problem for
a month. Every successful render now clears the flag
(`video_worker.py`). Both prod jobs were cleared by hand.
- **New: "Re-render" button** on finished tenant-portal tracks (`POST
/api/jobs/{id}/rerender`). It re-applies the tenant's **current** theme
and rebuilds the title/end screens, lyrics video, all encodes and
CDG/TXT. It reuses the reviewed lyrics and instrumental selection, so
there's no review step. Same mechanism as the private→public visibility
flow (`regen_restore_status=review_complete` → screens → render →
encode). Owner (still on the tenant allowlist) or admin. Free. The theme
editor now points to the button.
- **Encoding-worker cache:** the worker caches finished jobs by id, so a
re-run submitted while the VM was still up could get the previous render
or encode back. Render and encode now submit as
`{job_id}_g{worker_generation}`.
- **Old title/end cards:** the encoder's `**/*Title*.mov` lookup would
have picked up the old `finals/… (Title).mov`. The re-render now deletes
the MOVs in `finals/`.
- **Failed re-render:** it can be re-run. Retry re-runs it if the new
screens weren't made yet, and otherwise resumes from render/video as
usual.

## Testing
- Backend: 5128 passed, 36 skipped. New `test_theme_rerender.py` (31
cases: validation, claim/conflict, artifact cleanup incl. finals MOVs,
trigger rollback, route auth/allowlist, retry branch, generation-keyed
ids) and `test_video_worker_outputs_deleted_clear.py` (fails without the
fix).
- Frontend: `OutputLinks.test.tsx` 28 passed (button visibility, confirm
→ API → refresh). Strings translated to all 33 locales.
- Review: CodeRabbit CLI isn't available on this org. An independent
reviewer agent went over the diff: 1 must-fix (stale finals MOVs) fixed;
the failed-re-render recovery fixed.

## Not done (follow-ups)
- Re-render re-sends the completion email and push notification (same as
the visibility flow).
- No per-tenant rate limit. One re-render per job at a time is enforced
by the status claim.
- The status briefly shows "Lyrics ready" before "Generating screens".

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
v0.253.0
2026-09-30 01:06:54 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 eaa70e04ea feat(review): hold a word's resize handle at the row edge to keep extending it (v0.252.0) (#1090)
## Summary
In the lyrics review Waveforms rows, the view is the segment ±1s and it
was fixed during a drag (and leaving the row ended the drag). So a word
ending 5s+ too early took several drag-release cycles to fix.

Now, holding a word's resize handle within 24px of the edge it's heading
toward (or past it) keeps growing the word at 1–4 s/s (faster nearer the
edge). The view zooms out so the handle stays under the cursor. It stops
when:
- the pointer leaves the edge zone (then the handle tracks the pointer
normally in the zoomed view),
- the mouse is released (commits once, as before), or
- it reaches the neighbouring word, including the **next line's first
word** even when that's beyond the drawn context padding (new
`computeNeighbourBoundsBySegment`), else the audio end/start.

Works for both handles and also in the Edit Segment modal timeline (same
`TimelineEditor`).

## Changes
- `TimelineEditor.tsx`: absolute pointer→time mapping for resizes (grab
offset), a rAF auto-extend loop, a `dragView` override that widens the
view mid-drag, and window-level mousemove/mouseup (so drags survive
leaving the row, with text selection prevented). Resizes now clamp to
the neighbour instead of freezing short of it on overshoot, and never
grow into the adjacent line (pre-existing overlaps aren't snapped back).
- `contextWords.ts`: `computeNeighbourBoundsBySegment` (prev line end /
next line start per segment), wired through `TranscriptionView` →
`WaveformSegmentRow`.
- Version 0.251.1 → 0.252.0; docs/README.md recent-changes entry.

## Testing
- 7 new `TimelineEditor` tests (fake rAF): extends while held at the
edge, stops at the next line / same-segment neighbour, stops when
leaving the zone, respects the drag threshold, left handle bounded by
the previous line, commits once when released outside the row. Plus 2
new util tests.
- `make test`: all backend + 1476 frontend tests pass.
- Checked in the browser against the local fixture harness: the last
line's "shoes" extended with the handle glued to the edge while the row
zoomed out; "alright" stopped exactly at the next line's "Because".

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* In lyrics review waveforms and the Edit Segment timeline, dragging a
resize handle within 24 pixels of the direction of travel now
automatically extends the word. The timeline zooms to keep the handle
under the cursor as the word grows.
* Resizing stops at neighboring words, adjacent segment boundaries, or
the audio limit. You can continue dragging outside the row, and
releasing completes the resize.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
v0.252.0
2026-09-30 00:54:00 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 69986bb844 fix(youtube): canonical description template wins over stale theme youtube_description.txt (v0.251.1) (#1088)
## Summary
The daily YouTube description backfill finished its full-channel rewrite
on 2026-09-15, but kept rewriting 1-4 **brand-new** customer videos most
days. Root cause: the `nomad` theme's legacy
`gs://karaoke-gen-storage-nomadkaraoke/themes/nomad/youtube_description.txt`
(old Fiverr text: "...my cheapest package is only \$10!
fiverr.com/share/...") was returned by `_prepare_theme_for_job` and
written into `job.youtube_description_template` by every themed
job-creation path (made-for-you, bulk search, file upload x4, guided
search), overriding `Settings.default_youtube_description`. New videos
were published with the stale description and only corrected ~1 day
later by the backfill worker.

## Change
- `_prepare_theme_for_job` now only returns a theme's
`youtube_description.txt` when no canonical template is configured. All
callers already handle `None` (they keep the canonical template set at
creation, or leave it `None`, which the orchestrator renders with the
default).
- Tests: 2 new unit tests covering both precedence branches.
- Version 0.251.0 → 0.251.1.

## Evidence
Post-completion backfill updates (e.g. `l8xUGZqCYSY`, `jCyfwQorgOY`,
`44Qa2Pt-gpU`, `hMQEV2vCgLY`, `MFu7t-F0VR0`) all map to customer jobs on
theme `nomad` whose `youtube_description_template` starts "This is a
karaoke (instrumental) version of the song, with vocals removed using
AI...".

## Testing
- `pytest backend/tests/ --ignore=backend/tests/emulator`: 5080 passed,
36 skipped
- Local review (subagent fallback; CodeRabbit unavailable): no
must/should-fix findings

## Follow-ups (optional)
- The stale GCS `themes/nomad/youtube_description.txt` can be deleted
(also served by the unused `GET /api/themes/{id}/youtube-description`).
- Verify after deploy: backfill worker should report 0 pending on
subsequent days.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
v0.251.1
2026-09-30 03:55:31 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 780241ad03 feat(dashboard): show other accounts' awaiting-audio jobs + admin toggle (v0.251.0) (#1089)
## Why
fiverrbot (v0.3.0) parks Fiverr orders with no recommended audio at
`awaiting_audio_selection` (owner `fiverr@nomadkaraoke.com`) and tells
the admin to click **Select Audio** in `/app`. But
`shouldShowJobOnDashboard` hid every job at that status except
`made_for_you` orders, so the card never appeared (e.g. job `bdf35771`,
order FO62A31A27AC6). The API returned the job; only the frontend filter
dropped it.

## Changes
- `shouldShowJobOnDashboard(job, { viewerEmail, showOthersAwaitingAudio
})`: every `awaiting_audio_selection` job is now shown. The old wizard
reason for hiding them is gone, since the guided flow searches via
`searchStandalone` and never creates a job at this status. This also
fixes a bug already on main: "Choose different audio" (v0.250.0) moved
your own job to this status, and the card with its open picker
disappeared.
- With the admin switch off, only **other accounts'** non-made-for-you
jobs are hidden. Your own jobs and made-for-you orders stay visible.
- New admin-only **Show Jobs Awaiting Audio** switch in the account
menu, next to Show Test Jobs. It's persisted in `admin-settings`
(`showAwaitingAudioJobs`, default on). Turning it off hides other
accounts' jobs at this status.
- i18n: `auth.status.showAwaitingAudioJobs` in all 33 locales.
- LESSONS-LEARNED: added a recurrence note to the Aug 2026 made-for-you
entry.
- Version 0.250.0 → 0.251.0.

`user_email` is already in `SUMMARY_FIELD_PATHS`, so no backend change
is needed. Non-admin users only ever receive their own jobs, so nothing
changes for them.

## Testing
- New unit tests for the filter: shown by default, own job shown, toggle
off hides others but keeps own (case-insensitive) and made-for-you,
unknown owner hidden when off, other statuses unaffected. Plus a store
test for the new setting.
- Full frontend Jest: 124 suites / 1459 tests pass.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 23:51:49 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 c831bd5e89 feat(uploads): line up mismatched user instrumentals + tenant bulk upload modal (v0.251.0) (#1087)
## Summary
A Randy Vild tenant bulk upload lost 3 of 10 tracks to the 0.5s
mix/instrumental duration check, and the bulk UI showed only
"Submitting…" while uploading. All three rejected tracks were valid
edits of the same song:

| Track | Difference | Now |
|---|---|---|
| Keep Smiling | instrumental has 2.97s extra lead-in | start trimmed →
matches mix exactly |
| Simulation | instrumental 6.5s longer outro | cut at mix end with 2s
fade |
| Faithful and True | instrumental stops 4.45s early | end padded with
silence |

## Changes
**Backend: conform instead of cancel**
(`backend/services/instrumental_conform.py`, `uploads-complete`)
- On a >0.5s mismatch, the start offset is found by FFT
cross-correlation (reusing `derived_vocals.find_offset`) at **25% and
75%** of the song, searching ±15s. The start is then trimmed or padded,
and the end is faded and cut or padded to the mix length. All of this is
one ffmpeg pass on the original file, keeping its sample rate and bit
depth, and written as FLAC.
- Stored at `uploads/{job}/conformed/existing_instrumental.flac`, which
becomes `existing_instrumental_gcs_path`. The original upload is kept,
and details go in `state_data.instrumental_conformed`.
- Still rejected (job cancelled and refunded, 400):
- correlation < 0.3 (wrong file; real pairs score 0.78–0.90, wrong pairs
about 0.01);
- offsets at 25% and 75% differ by more than 20 ms (edited differently
mid-song).
- Conforming runs fully off the event loop, with at most 2 at a time per
instance. GCS or ffmpeg failures return a retryable 503 instead of
cancelling the job.
- The orchestrator now stages the recorded
`existing_instrumental_gcs_path` for the GCE encoder, so the conformed
copy is what gets rendered.

**Frontend**
- Tenant bulk shows `UploadProgressModal` for the whole batch: total
bytes, speed and ETA, tracks submitted and failed, and an offline or
retrying notice. The modal gained optional `title`, `detail` and
`notice` props and uses the tenant's primary colour.
- Bulk retry bug:
- **Before:** a server-cancelled job kept its ID. The retry's 400 ("not
pending") was read as "already processing", so the row showed
**Submitted** for a dead job.
- **Now:** any 400 checks the job's status. A processing job counts as
done. A cancelled or failed job is dropped, so the retry creates a fresh
job. Otherwise the row keeps the job for retry, so no duplicate is
created.
- The single-track tenant form and the own-instrumental field now check
only the size cap in the browser; the server handles length differences.

## Testing
- Backend: 5086 passed (new: synthetic lead-in, long/short outro, late
start, wrong-file and mid-song-edit cases; route and helper guards;
orchestrator staging). Frontend jest: 1459 passed (new: batch modal
progress and offline notice, failed rows, retry and no-duplicate cases).
- Ran `conform_instrumental` on Randy's real files:
  - offsets at 25% and 75% agree to the sample;
  - output is exactly the mix length at 24-bit / 44.1 kHz;
  - after conforming, each file lines up with its mix at 0 offset;
  - a wrong pairing is rejected (correlation 0.02).
- Real frontend (`next dev`, randy-vild tenant, mocked API, slowed
upload server), 8 tracks / 192 MB: the modal showed progress, track
counts, speed and ETA, then closed onto the "All tracks submitted"
screen.
- Review: an independent reviewer agent found 1 must-fix and 4
should-fix issues, all addressed in the last commit. The CodeRabbit CLI
has no seat here, so this PR is left open to the CodeRabbit bot.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Uploaded instrumentals with different durations can now be aligned and
adjusted to match the mix when the audio matches.
* Bulk uploads now display overall progress, completed and failed track
counts, connection updates, and retry guidance.
* Failed bulk-upload tracks can be retried, with clearer handling for
submissions already in progress.

* **Bug Fixes**
* Instrumental duration checks now occur during upload processing
instead of blocking the upload in the browser. Files that cannot be
matched are reported as a mismatch.

* **Documentation**
* Updated upload instructions and related messages across supported
languages.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
v0.251.0
2026-09-29 23:42:08 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 d182151626 infra: remove legacy us-central1 CPU Cloud Run Jobs (Pulumi already applied) (#1086)
## Summary
Follow-up to #1077 and #1083. This removes the four legacy
**us-central1** Cloud Run Jobs, which were kept only as a rollback path
while the us-east4 copies were verified:
- audio-download-job
- lyrics-transcription-job
- bulk-search-job
- video-encoding-job

The us-east4 jobs were verified in prod on 2026-09-29:
- A normal job's download and lyrics jobs started in about 20 s, and a
kjbox make-it worked end to end.
- Five render and final-video runs started in 10–24 s, and all three
jobs completed with YouTube, Dropbox and GDrive delivery (the tenant job
skips those, as expected).
- The legacy jobs have had no runs since each cutover.

## Pulumi
**Already applied** (`pulumi up`: 4 deleted). A preview afterwards shows
**286 unchanged**, so this merge introduces no drift. The us-east4 jobs
keep their suffixed logical names so they aren't replaced.

## Changes
- `infrastructure/__main__.py`: removed the `legacy_*` job resources;
comment updated.
- `infrastructure/modules/cloud_run.py`: comment-only updates.
- `backend/config.py`: comment-only update to the rollback guidance.
- `docs/TROUBLESHOOTING.md`: rollback runbook now describes recreating
the jobs via Pulumi, and notes that rolling the service back to before
v0.247.1 needs them.

## Testing
- `test_worker_service_cpu_jobs_region.py` and
`test_audio_download_worker.py` (which parses `cloud_run.py`) pass.
- The Pulumi preview and apply both match the expected changes exactly.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Infrastructure**
* Removed the legacy Cloud Run job copies in `us-central1`; active jobs
continue to run in their configured region.

* **Documentation**
* Updated rollback guidance to cover recreating the `us-central1` jobs
before rolling back the backend to a version before v0.247.1. The
instructions now describe the staged transition between regions,
including updates to backend and CI job settings, region defaults, and
the rollback script.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 22:12:28 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 f4eca80551 feat(audio-download): stalled torrent → 'Keep trying?' prompt (v0.250.0) (#1085)
## Why
Job 64be8a87 (a Fiverr order): the only seeder of a 1-seeder RED single
was offline. The torrent sat at **0 peers through 5 re-announces for 10
min**, flacfetch aborted, the job showed a raw error, and a Discord "Job
failed" alert fired even though a Cloud Run auto-retry was already
queued. That retry got the file in 17s. Andrew's ask: 10 min is too
short, and a stalled download should explain itself and offer "Keep
trying?" for up to another hour.

## What
- **Longer first wait:** 20 min with no progress (was 10), sent to
flacfetch per request as `max_stall_seconds` (needs flacfetch v0.31.0,
nomadkaraoke/flacfetch#51, already deployed to flacup). The worker's
wait and the Cloud Run **per-execution timeout override** come from the
same budget (`backend/services/audio_download_limits.py`), so the job's
Pulumi config is untouched.
- **Stall → user prompt, no silent retry:** flacfetch now reports
`error_code: torrent_stalled` (older versions are detected from the
error text). The worker fails the job with `error_details = {stage:
audio_download, code: audio_download_stalled, stall_minutes,
keep_trying}` and exits 0, so Cloud Run doesn't retry the same torrent.
Other download failures keep their auto-retries.
- **Job card:** `StalledDownloadNotice` replaces the raw error with an
amber explanation and two buttons:
- **Keep trying (up to 1 hour)**: `POST /api/jobs/{id}/retry
{"keep_trying": true}` (the body is optional and backward compatible).
This gives a 3600s stall budget and a 1h25m execution timeout.
- **Choose different audio**: new `POST
/api/jobs/{id}/choose-different-audio` (owner-checked). It moves FAILED
to `awaiting_audio_selection` (new transition) using the saved search
results and opens the Select Audio dialog.
- The generic Retry button is hidden for stalled jobs.
`error_details.{code,stage,stall_minutes,keep_trying}` were added to the
summary fields. Strings are translated into all 33 locales.
- **Heartbeat:** while waiting on a torrent, the worker writes
`state_data.audio_download_progress` (progress, peers) every 2 min.
Without this, recover-stuck-jobs (which parks a job with no update for
10 min) would park a live 20–60 min wait and start a duplicate download.
- **Ops alert:** the alert is skipped while a Cloud Run auto-retry is
pending (the final attempt clears the marker and alerts then). Stall
alerts get a 🟠 "Audio download stalled" headline.
- Docs updated: API.md, TROUBLESHOOTING.md, ARCHITECTURE.md.

## Tests
- `make test`: backend 1846 + 5078 + 150 passed; frontend 1451 passed.
- New tests:
- `test_audio_download_limits.py`,
`test_routes_jobs_stalled_download.py` (real HTTP via TestClient),
`test_worker_service_audio_download_timeout.py`.
- Stall, keep-trying, heartbeat and `main()` exit-code cases in
`test_audio_download_worker.py`.
- Stall detection and payload in `test_flacfetch_client.py`;
retry-pending suppression in `test_ops_alerts.py`.
- Frontend: `StalledDownloadNotice.test.tsx`, plus JobCard/JobActions
cases.

## Notes
- Credits are unchanged: a failed job is still auto-refunded and retries
don't re-charge. That's existing behaviour for every retry.
- A local CodeRabbit review couldn't run (the CLI seat is on another
org), so this PR has no ignore line and the GitHub bot will review it.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Failed audio downloads now offer options to keep trying for up to an
hour or choose a different audio source.
* Retry progress and clearer stalled-download guidance are available in
the job view, with messages translated across supported languages.

* **Bug Fixes**
* Stalled downloads can be recovered without triggering unnecessary
automatic retries or duplicate failure alerts.
* Choosing different audio returns eligible failed jobs to audio
selection.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
v0.250.0
2026-09-29 21:51:50 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 fc05571779 fix(theme-preview): skip ffprobe for no-audio ASS generation (v0.249.1) (#1084)
## Summary
Error-monitor alert: `Failed to get audio duration: ...
/tmp/theme-preview-*/no-audio.flac`. The tenant theme editor (#1080)
passed a nonexistent audio path to `SubtitlesGenerator.generate_ass`, so
every uncached preview ran ffprobe on a missing file and logged at ERROR
before falling back to the segment-derived duration. Previews rendered
fine (200); this was pure log noise.

## Changes
- `generate_ass` / `_get_audio_duration` accept `audio_filepath=None` →
skip ffprobe, use `last segment end + 30s` silently. Real-audio callers
unchanged (a failing probe still logs ERROR).
- `theme_preview_service` passes `None`.
- Version 0.249.0 → 0.249.1.

## Testing
- New
`tests/unit/lyrics_transcriber/output/test_subtitles_audio_duration.py`
(None path: no probe, no error log; failed probe still logs;
`generate_ass(..., None)` end-to-end).
- `make test-unit test-backend-unit`: 1846 + 5016 passed.
- Local CodeRabbit review: no findings.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
v0.249.1
2026-09-29 16:34:46 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 badd845a1d perf(jobs): move video-encoding-job (video + render) to us-east4 (v0.249.1) (#1083)
## Summary
Follow-up to #1077. `video-encoding-job` runs both the post-review
render (`render_video_worker`) and the final video (`video_worker`). In
us-central1 each run waited **2–5 min** for Cloud Run Jobs to start it,
so every customer job paid that twice. This moves it to us-east4
alongside the download, lyrics and bulk-search jobs.

## Cost (measured, not estimated)
- Over the last 30 days the job read **156 GB** from Google services,
mostly the us-central1 bucket, across 494 runs (Cloud Monitoring
`container/network/received_bytes_count`, kind=google). That's about
0.32 GB per run.
- Pricing from the Billing Catalog: GCS inter-region within North
America is **free for the first 100 GB/month, then $0.02/GB**. Cloud Run
writes to Google APIs are **$0**.
- Internet egress (224 GB/month, Dropbox etc.) and Cloud Run CPU cost
the same in either region.
- Existing cross-region reads are about 24 GB/month (GPU job and
lyrics), which the billing export shows at about $0.
- **Net: about $1.60/month**, or about $3/month worst case. Time saved:
roughly 4–8 min per customer job.

## Changes
- **backend**: `trigger_video_worker` (via `_trigger_cloud_run_job`) and
`trigger_render_video_worker` now use `cpu_jobs_region`. `GCP_REGION`
stays us-central1; encoding VM and zone logic are unchanged.
- **infra (Pulumi, already applied)**: new us-east4
`video-encoding-job`, with the legacy us-central1 copy kept for rollback
and its deletion protection turned off. A preview after applying showed
290 unchanged.
- **ci**: the video job image update now targets us-east4, using the
us-east4 CPU image copy.
- **Also updated**: rollback.sh, tests (video and render routing),
TROUBLESHOOTING/ARCHITECTURE/LESSONS-LEARNED.

## Testing
- `make test` passes: backend tests plus 1443 frontend tests.
- Prod smoke test of the new us-east4 job: it imports `video_worker` and
`render_video_worker` and calls the encoding VM's `/health` on its
public IP (port 8080 is open to 0.0.0.0/0 with API-key auth). **Started
in 25 s, exit 0.**

🤖 Generated with [Claude Code](https://claude.com/claude-code)


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Updates**
- Video encoding and render jobs now run in the `us-east4` region,
alongside other CPU jobs.
- Video encoding continues to access storage and encoding resources in
`us-central1`; cross-region transfer may incur costs.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 16:33:23 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 cb5ce7f162 chore(pypi): prune keep-window default 60→30 days (#1082)
Ran the PyPI prune runbook today (2026-09-29): publishing was blocked
(`Project size too large`). The 60-day policy would only have freed 1.08
GB (→9.64 GB), refilling within ~a week, so pruned with `--keep-days 30`
instead (69 releases deleted, 0.188.5–0.207.0; newest release of each
older month retained) → ~6.8 GB. Re-ran the failed publish; 0.249.0 is
live on PyPI.

This PR makes 30 days the default in the planner script, the monthly
reminder workflow, and the runbook. Unit tests pass (13).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 16:17:43 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 385a997aba test(e2e): fix tenant theme editor prod spec selectors (#1081)
Follow-up to #1080. The production spec failed to find its targets on
the live `randy-vild` portal:
- `AuthStatus` renders twice, once in the desktop header and once in the
mobile header, so `user-menu-trigger` matched two elements. The spec now
targets the visible one.
- The dialog's corner ✕ is also labelled "Close". The spec now targets
the footer button: `exact: true`, then `.first()`.

I verified this against production with `DEBUG=pw:api`: every step and
assertion passes (both previews render, only the lyrics preview changes
after a sung-colour edit, Discard and Close work).

Test-only change, so no version bump.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Tests**
* Updated the tenant theme editor end-to-end test to open the user menu
and close the editor using more specific controls.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
v0.249.0
2026-09-29 04:21:10 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 80f6d34007 feat(tenant): self-service theme editor with exact server-rendered previews (#1080)
## Summary
Tenant portal users (anyone on the tenant's allowlist) can now edit
their tenant's theme from **Theme & style** in the user menu.
- **Guided form:**
- backgrounds (upload or solid colour) for the title card, lyrics video
and end screen
  - font (bundled or uploaded TTF/OTF)
  - title and artist colours, positions and sizes
  - sung, upcoming and outline lyric colours
  - lyric size, top position and max line length
  - the end-screen message
- **Advanced:** validated full-theme JSON.
- **Exact previews:** the title card and a lyrics-video frame (first
line half-sung) are rendered **by the production renderers** from the
unsaved draft, in about 2 s, debounced. The plan is in
`docs/archive/2026-09-29-tenant-theme-editor-plan.md`.
- **Save** applies to jobs created afterwards.

## Backend
- New router: `GET/PUT /api/tenant/theme`, `POST
/api/tenant/theme/assets`, `POST /api/tenant/theme/preview`.
- **Membership guard:** `X-Tenant-ID` is client-controlled, so every
call re-checks the tenant allowlist (admins always pass). Non-members
and spoofed headers get 403. The tenant bulk router only checks that the
tenant exists, so this pattern wasn't copied from there.
- `theme_preview_service`:
  - title card via `VideoGenerator.create_title_video(duration=0)`
- karaoke frame via `SegmentResizer` → `SubtitlesGenerator.generate_ass`
→ one ffmpeg frame, using `setpts` to jump straight to time T and
scaling inside the same pass (11.8 s → 0.66 s)
  - bounded concurrency, and a cache for identical drafts
- `tenant_theme_service`, stricter than the admin console:
- asset fields must be basenames that exist in the theme (no paths or
`gs://`)
  - bounds, colours and regions are validated
- **font consistency with the render pipeline:** the single `font` asset
is applied everywhere, `karaoke.font` is set to the TTF family name
libass matches, and a chosen bundled font is copied into the theme's
assets
  - required ASS keys are filled in
- uploads are content-addressed, so they never overwrite an asset that
live jobs use
  - tenants can't change their config or allowlist through this API

## Frontend
- `components/tenant-theme/TenantThemeEditorDialog` and `theme-utils`.
- `tenantThemeApi` in `lib/api.ts`.
- A menu item in `AuthStatus`, shown only on tenant portals.
- `tenantTheme.*` strings in all 33 locales.

## Tests
- **pytest:**
- real renders (the background colour appears in the frames; only the
karaoke frame changes when the sung colour changes; caching)
- sanitisation (paths, un-uploaded assets, bounds, colours, regions,
sections)
- uploads (content addressing, `no-store`, rejecting non-images and bad
fonts)
  - save (font family, bundled font copy, no config changes)
- the route guard for spoofed, other-tenant, inactive and no-tenant
cases, on every endpoint
- **Jest:** load, debounced preview, hex→rgba conversion, upload, region
sliders, Advanced JSON validation, save, discard, the unsaved-changes
guard, and preview errors.
- **Prod E2E:** `e2e/production/tenant-theme-editor.spec.ts`.
- Results: backend 4998 passed; frontend 1434 passed.

No local CodeRabbit review was run (no CLI seat in this org), so this PR
is left open for the GitHub bot.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added a theme editor to the signed-in menu, with controls for title
cards, karaoke videos, end screens, and advanced settings.
* Preview theme drafts, upload images and fonts, and save or discard
changes. Saved themes apply to future videos.
  * Added translations for the editor across supported languages.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 03:18:24 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 835fdf6096 kn-data-sync: ListenBrainz popularity + inclusion in the kjbox song-id index (v0.248.1) (#1078)
## Summary
Adds ListenBrainz (karaoke-decide `lb-refresh` →
`lb_recording_popularity` / `lb_artist_popularity`) to `SONG_ID_SQL`,
fixing the index's freshness gap (design: kjbox
`docs/SONG-IDENTIFICATION.md` D5a/D12).

- **Score:** each stats range maps onto Spotify's 0–100 as `offset +
19·log10(listeners)`. Offsets: all_time 8, this_year 11,
year/half_yearly 12, month 14, this_month 20. They were fitted to the
median Spotify popularity of songs that have both. Ranges under 3
listeners are ignored.
- **Song popularity** = max(Spotify track score, ListenBrainz score).
The capped artist estimate stays the fallback, and it can now also use
the ListenBrainz artist score.
- **Inclusion:** a MusicBrainz song is also kept with ≥ 5 listeners
this/last year or ≥ 3 last/this month. That's how brand-new artists get
in (Ninajirachi, Geese, CORTIS). ListenBrainz artist popularity
deliberately does **not** gate inclusion: it would add ~440K unlistened
songs.
- **Examples:** Olivia Dean "Man I Need" 50 → 75; "The Fate of Ophelia"
51 → 77; Tame Impala "Dracula" 0 → 80.
- **Size / cost:** 5.90M → 6.30M rows exported (+7%); 8.0 GB scanned
(~$0.05); `EXPORT DATA` ~23 s, well under the 300 s function timeout.
- `deploy.sh` now runs `gcloud functions deploy` itself. Pulumi's source
object has no generation pin, so a new upload never rebuilt the
function, and the script's "updated on next Pulumi deploy" message was
wrong.
- Also: an SQL-contract test, a blank-title filter, and the
ARCHITECTURE.md row.

## Evaluation
The candidate SQL was exported to a scratch prefix
(`gs://nomadkaraoke-kn-data/song-id-eval/lb1/`), then built and
evaluated on the NomadPC with kjbox #261:
- Labelled: 72% → 80% auto-identified, wrong 2 → 0.
- Freshness cases: 18 → 26 of 26.
- Frozen held-out: 84% unchanged, wrong 8 → 7.

## Deploy
After merge: run `infrastructure/functions/kn_data_sync/deploy.sh`, then
`gcloud scheduler jobs run kn-data-sync-full-daily --location
us-central1`. The NomadPC picks the new run up at its next sync.

## Testing
`cd infrastructure/functions/kn_data_sync && python -m pytest
test_main.py`: 6 pass. The SQL was dry-run and executed in prod BigQuery
(read-only queries plus the scratch export).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* The song-identification index now includes eligible MusicBrainz songs
using ListenBrainz listener activity and popularity alongside Spotify
popularity. This expands coverage beyond the previous
Spotify-popularity-based selection and includes all KaraokeNerds
entries.
* Song popularity can reflect recording-level ListenBrainz scores, with
artist popularity used as a fallback.

* **Documentation**
  * Updated the documented index size to approximately 6.6 million rows.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 01:59:37 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 48e5b6bfe2 feat(review): Edit Segment timeline — click waveform to play, Ctrl/Cmd-click word to delete (v0.248.0) (#1079)
## Summary
Brings the Edit Segment modal's timeline to parity with the main
Waveforms review rows:

1. **Click the waveform to play from that point** — previously only the
ruler/timeline strip was a click target in the modal; the waveform only
was in compact rows.
2. **Ctrl/Cmd-click any word bar to delete it** — no more scrolling the
word list to hit delete.

## Changes
- `TimelineEditor`: waveform wrapper always handles click-to-play
(pointer cursor when `onPlaySegment` is set); new `onWordCtrlDelete`
prop fires on Ctrl/Cmd+mousedown on a word bar (incl. its resize
handles). Modifier is read from the event, so no key-state tracking.
- `EditTimelineSection` → `EditModal`: wired to the existing
`handleRemoveWord`; disabled during Tap To Sync so indices don't shift
mid-sync.

## Testing
- New `TimelineEditor.test.tsx` (waveform click → time mapping in
compact + non-compact; Ctrl/Meta delete; plain click doesn't delete;
handles delete) + `EditModal` test (Ctrl-delete removes word, recomputes
text/start on save). 5 of the new tests fail without the source change.
- Full frontend jest: 120 suites / 1430 tests pass.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Ctrl/Cmd-click a word or its resize handle to delete that word from
the timeline.
* Click the waveform to play from the selected point in both standard
and compact layouts.

* **Bug Fixes**
* Word deletion updates the saved segment’s text and start time to
reflect the remaining words.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 01:54:15 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 3e855ff65d perf(jobs): run download/lyrics/bulk-search Cloud Run Jobs in us-east4 — cut 2-5 min job start latency (v0.247.1) (#1077)
## Summary
Every gen job waited **2–5 min** before `audio-download-job` (and then
`lyrics-transcription-job`) even started. That blocked kjbox's goal of
singing within about 4 minutes of making a song.

**Root cause: Cloud Run Jobs in us-central1 are slow to start on
Google's side.** Our image, CPU and code aren't the cause.
- The `Started` condition fires about 10 s before the first Python log,
so our process startup is fine.
- decide's 203 MB image on 4 CPU is just as slow in us-central1.
- A throwaway job running Google's tiny sample image, 4 runs per region:
**us-central1 4m–5m07s**, **us-east4 8–19 s**, **us-east1 8–25 s**. All
four us-central1 runs started in the same second, as if released from a
queue. Quotas are far from their limits.

## Changes
- **backend**: new `CPU_JOBS_REGION` setting (default `us-east4`) for
triggering `audio-download-job`, `lyrics-transcription-job` and
`bulk-search-job`. `GCP_REGION` is unchanged. `video-encoding-job` stays
in us-central1, because it pulls multi-GB outputs from the us-central1
bucket.
- **infra (Pulumi, already applied)**: the three jobs are created in
us-east4 as new resources. The legacy us-central1 copies are kept as the
rollback path, with deletion protection turned off so a follow-up can
remove them. The preview after applying showed 289 unchanged, so there's
no drift.
- **ci**: the CPU app image is also pushed to
`us-east4-docker.pkg.dev/nomadkaraoke/karaoke-backend-gpu/karaoke-backend-cpu`,
so jobs pull from their own region. The lyrics job image update now
targets us-east4. After each deploy, CI starts a no-op
`audio-download-job` execution so the one-time ~2.5 min image import
doesn't land on the first customer.
- `scripts/rollback.sh`: lyrics job moves to the us-east4 image, and
jobs whose target tag is missing are skipped instead of aborting the
loop.
- Docs: TROUBLESHOOTING runbook (measure + rollback), ARCHITECTURE,
LESSONS-LEARNED.

## Testing
- `make test`: all backend tests pass, plus 1422 frontend tests.
- New tests in `backend/tests/test_worker_service_cpu_jobs_region.py`:
routing of each moved job, video job stays in `GCP_REGION`, env override
for rollback, and the Pulumi/backend region constants must match.
- Prod smoke test of the new us-east4 `audio-download-job` with the real
image (`import backend.workers.audio_download_worker`): **Started in 7.0
s, 8.6 s, 10.0 s** (first run of a new digest adds a one-time ~2.5 min
import, which the CI step covers).
- Local review via a reviewer agent: no must-fix findings; the
rollback-path suggestions are addressed.

## Rollout
One-time image seed (Cloud Build gcrane copy) and `pulumi up` are done.
After merge, CI deploys the backend, which starts triggering the
us-east4 jobs.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Performance**
* Audio downloads, bulk searches, and lyric transcriptions now run in a
region with faster job startup times, reducing delays before processing
begins. Video encoding continues to run in its existing region.
* **Reliability**
* Rollbacks now skip jobs when the target image is unavailable and
continue with the remaining jobs. Other image lookup errors still stop
the rollback.
* **Documentation**
* Updated architecture and troubleshooting guides with regional job
details, startup-delay guidance, and rollback instructions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 01:02:19 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 518126727b feat(tenant): tenant users never see or use consumer credits (#1076)
## Problem
A tenant user's first login on `randy-vild.nomadkaraoke.com` showed the
consumer "We weren't able to grant free credits" screen with **Buy
Credits**. The backend also sent a credits-denied email. Tenant jobs
were being charged consumer credits too, so the client's first submit
would fail with "out of credits". Tenants are companies or bulk clients
on a separate commercial agreement.

## Changes
- **Sign-in:**
- Tenant magic links skip the whole welcome-credit flow: no AI eval, no
denial or welcome email, no referral attribution. `credit_status` comes
back as `not_applicable`.
- Sending a tenant magic link no longer precomputes a credit evaluation.
- **Jobs:** `create_job` never credit-checks or charges jobs with a
`tenant_id`. They're marked `payment_bypassed`, so duration
reconciliation never pauses them for credits and refunds never mint
credits.
- **Access:** an empty tenant allowlist now means **admins only**. Free
jobs must never be open to arbitrary sign-ups. All 3 live tenants
already have allowlists. The admin UI copy is updated.
- **Frontend:** on tenant portals, the credits badge and menu, Buy
Credits, and Referrals are hidden. The verify page never shows credit
screens for tenant logins.

## Tests
- `test_tenant_no_consumer_credits.py`: a tenant job isn't charged and
is marked `payment_bypassed`; consumer jobs are still charged and still
rejected without credits.
- Verify route: a tenant login returns `not_applicable` and the grant is
never called.
- Model: an empty allowlist means admins only.
- `AuthStatus`: tenant mode hides credits.
- Results: backend 4953 passed; frontend 1423 passed.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Access**
* Tenant portals with no configured email addresses or domains are
restricted to administrator sign-ins. Configured allowlists continue to
limit access to listed users and administrators.
* **Tenant Portal**
* Tenant sign-ins no longer receive consumer welcome-credit evaluations
or related credit emails.
  * Tenant jobs bypass consumer credit checks and charges.
* Credit balances, credit-purchase options, and referral links are
hidden in tenant portals.
* Successful tenant sign-ins proceed to the app regardless of credit
status.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 12:50:28 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 2b8439a0ae fix(auth): own domain never disposable; tenant-allowlisted emails skip disposable check (#1075)
## Problem
Signing in as `randy@nomadkaraoke.com` on the Randy Vild portal showed
"We don't support disposable email addresses". `nomadkaraoke.com` had
been **auto-learned into the disposable-domain blocklist**
(`blocklists/config.manual_domains`). Its `verified_clean_domains` entry
dated from 2026-07-04 and has a 7-day TTL. After it expired, an external
API (DeBounce/verifymail) flagged the domain. The result: **every
`@nomadkaraoke.com` magic-link sign-in was rejected.**

## Fix
- **Prod data (already done through the admin API):** removed
`nomadkaraoke.com` from `manual_domains` and added it to
`allowlisted_domains`.
- **Code:**
- New `NEVER_DISPOSABLE_DOMAINS = {"nomadkaraoke.com"}`: a tier-0
short-circuit before any list or API check, and `_auto_learn_domain`
refuses it (and well-known providers).
- On tenant portals, an email the tenant explicitly allowlisted
(`allowed_emails` or an allowed domain) skips the disposable checks,
because an admin vouched for it. Everyone else is still checked.

## Tests
- Regression tests: our domain isn't flagged even when it's blocklisted
and the API says it's disposable; auto-learn refuses it; a
tenant-vouched email skips the check while other emails on the same
portal still get 422.
- `pytest backend/tests --ignore=backend/tests/emulator`: 4940 passed.

No local CodeRabbit review was run, so this PR is left open for the
GitHub bot.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Bug Fixes**
- Explicitly allowlisted email addresses can now request magic links
even if their domain is flagged as disposable. Other addresses on the
same tenant portal remain subject to the existing check.
- `nomadkaraoke.com` is no longer treated as a disposable domain or
automatically recorded as one.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
v0.246.2
2026-09-28 02:08:24 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 fe217fda1f kjbox singer search: Gemini fallback with the job flow's catalog tidy + descriptive queries (v0.246.2) (#1065)
## Summary
The Gemini fallback for kjbox's singer song identification (kjbox
`docs/SONG-IDENTIFICATION.md`). kjbox's on-device matcher answers most
searches; this endpoint handles what it can't.

- `POST /api/kjbox/catalog/resolve` now runs `resolve_and_tidy`:
1. One Gemini call (`gemini-3.8-flash`) splits the singer's one-line
query into artist/title and fixes typos.
2. It now also resolves **descriptions**: "that song from titanic" →
Céline Dion — My Heart Will Go On; "is this the real life" → Queen —
Bohemian Rhapsody; "song from top gun" → ambiguous: Danger Zone / Take
My Breath Away / Hold My Hand (live-checked).
3. The chosen song goes through the job flow's own
`judge_match(stage="fast")` catalogue tidy, so formatting matches gen's
job form.
- AI kind `none` keeps its typed split, so kjbox can pre-fill its
make-it form. A failed catalogue pass is returned uncached.
- kjbox calls it only when on-device identification and karaoke search
both found nothing, or when the singer uses "Describe it". It's cached
and rate-limited as before.
- Docs: `docs/API.md` notes that the match judge's corrections are
Gemini in practice (its catalogue pass is exact-match only and decide's
catalogue is prefix-only).

## Tests
- `test_match_judge_free_text.py`: resolve_and_tidy cases, descriptive
prompt coverage
- `TestCatalogResolve`
- Local CodeRabbit: findings fixed, re-review clean.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 06:07:13 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 5a9d0024a5 fix(quick-version): ffmpeg 4.4 never-ending render + fast ONNX model (v0.246.1) (#1074)
## Summary
First production make-it quick version (job `405eed71`, Rihanna "Push Up
On Me") failed with `render timed out`:
- **Render never ended**: the GPU image ships ffmpeg **4.4** (Ubuntu
22.04). With the guide-vocals `amix` in the filtergraph plus the
infinite lavfi colour + looped PNG inputs, it ignores `-shortest` —
reproduced in `ubuntu:22.04`: a 20 s song produced 540 s+ of video until
killed. ffmpeg 8 (local dev) stops fine, which is why local tests
passed. Fix: every input and the output are bounded with `-t <duration>`
(colour source `d=`). Verified in `ubuntu:22.04`: 3.7 s render, exactly
20.0 s audio+video.
- **Quick separation too slow**: the baked roformer took 14 s load + 124
s for a 3:15 song on the L4. Switched to fastgen's
`UVR-MDX-NET-Inst_HQ_4.onnx`. It's downloaded on load until the next GPU
base rebuild bakes it (`download_models.py` `EXTRA_MODELS`), with a
fallback to the baked roformer if the load fails.

Prod timeline observed for that job (for the <4 min goal): job created
05:22:12 → **audio-download-job container took 3m46s to start**
(existing issue, 1m40–4m across recent executions; separate follow-up) →
audio in GCS 05:26:46 → GPU worker started 05:27:10 → quick separation
done 05:29:49 → render hung.

## Testing
- `backend/tests/test_quick_version.py`: 40 pass (new: every stream
bounded to the duration; model fallback; both-fail raises).
- Docker `ubuntu:22.04` ffmpeg 4.4.2 reproduction before/after (see
above).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Quick audio separation now uses a faster model by default, with an
alternate model available if the primary model cannot be loaded.
* **Bug Fixes**
* Quick video rendering is now limited to the song’s duration, helping
prevent renders from continuing beyond the audio or failing to finish.
* Model loading failures are handled more reliably: if both the primary
and alternate models fail to load, the error is reported.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
v0.246.1
2026-09-28 01:51:13 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 37f7d578ee kn-data-sync: daily export of the kjbox song-identification index (#1066)
## Summary
After the daily KN `full` refresh, `kn-data-sync` now also exports the
**kjbox song-identification index**: every song a singer might mean.
- **Contents:** `spotify_tracks_normalized` (popularity ≥ 30) FULL OUTER
JOIN `karaokenerds_raw`, deduped on normalised artist+title. About 2.0M
rows: artist, title, popularity (blank = KN-only), karaoke (0/1).
- **Export:** `EXPORT DATA` to
`gs://nomadkaraoke-kn-data/song-id/<run>/songs-*.tsv.gz`, then a
manifest at `song-id/latest.json` naming that run's shards, so a device
never mixes runs. The last 3 runs are kept.
- **Cost and safety:** it's a server-side export; no rows pass through
the function (512M memory is fine). The dry run scans about 160 MB
(~$0.001/day). A failed export is reported in the response and doesn't
fail the KN sync.
- No IAM change: the SA already has project `bigquery.dataEditor` +
`jobUser` and write access to the bucket.

Design and reasoning: kjbox `docs/SONG-IDENTIFICATION.md` (on-device
matcher first, Gemini fallback). The NomadPC side (download → build
`song_id.db`) is in the kjbox PR.

## ⚠️ Deploy (manual, needs GCP write)
1. `infrastructure/functions/kn_data_sync/deploy.sh` (uploads the source
zip)
2. `pulumi up`, or redeploy the function. The source object name doesn't
change, so force the function to pick up the new zip.
3. Optionally trigger it once: `gcloud scheduler jobs run
kn-data-sync-full-daily --location us-central1`, then check
`gs://nomadkaraoke-kn-data/song-id/latest.json`.

## Tests
- `infrastructure/functions/kn_data_sync/test_main.py` (5): manifest
names only this run's shards, prune keeps 3 runs, empty export raises,
full sync survives an export failure, community mode doesn't export. Run
manually (function tests aren't in the repo testpaths).
- The rendered SQL passes a BigQuery dry run.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Full catalog syncs now generate a song-identification index covering
popular Spotify tracks and KaraokeNerds entries. The index is exported
to cloud storage, with a manifest pointing to the latest run and the
three most recent runs retained.
* **Bug Fixes**
* If index export fails, the catalog sync still completes and reports
the export failure. Community syncs do not generate an index.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 01:48:50 -04:00