Commit Graph
1982 Commits
Author SHA1 Message Date
Andrew Beveridge ☄️andClaude Sonnet 5.5 83e097fa3b feat(local-stack): fake review fixtures and knobs for mobile native review (v0.279.5) (#1202)
## Summary
Local-stack (FAKE_WORKERS=1) support for nomad-mobile Phase 2, which is
the native lyrics and instrumental review. Maestro can now drive every
review state against the real review endpoints. Nothing changes in prod:
all hooks are gated on `fake_workers_enabled()`.

## Changes
- **New per-user `FakeConfig` knobs:**
- `review_fixture` (`default` / `corrections` / `duet` / `rtl` /
`large`). The fixtures are built from the real `CorrectionResult` types:
AI spans with `undo_info`, an untimed word, an evidenced vocal gap with
synced reference lines, duet singers, Hebrew/Arabic lines, and 300
lines.
  - `backing_vocals` (`silent` / `audible`)
  - `stems_delay_seconds` (stems land after review opens)
- `review_mode` and `auto_approval` (writes
`processing_metadata.auto_approval`, so the confident and skipped-step
states can be reached)
  - `review_complete_fail_once`
- **Fake media gets an amplitude envelope**, so waveforms aren't flat.
There is also an audible backing stem variant.
- **`/api/review/{id}/audio` dev-proxy branch** now uses
`_ranged_response` (Range → 206), so seeking works on iOS. The prod
signed-URL path is untouched.
- **`GET /api/test/jobs/{id}/review-result`** returns the submitted
review (selection, `is_duet`, corrected lines) for Maestro assertions.
- `docs/TESTING.md` § Local stack documents the knobs.

## Testing
- `make test`: backend 2184 + 6532 + 163 passed, frontend 1760 passed.
- 29 new tests in `backend/tests/fake_workers/test_review_fixtures.py`.
- Smoke-tested on the local stack: every fixture loads; `with_backing`
analysis; Range 206; 503 then 200 on complete; `review-result`;
`auto_approval` states; late stems.

Reviewed locally (CodeRabbit CLI, 1 finding fixed).

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01G81fc5sBB8GwG5Xhz2UMgc

---------

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
v0.279.5
2026-10-20 00:20:38 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 e050cb8b3d fix: rescue repetitive-song reference lyrics the relevance filter undercounts (v0.279.4) (#1201)
@coderabbitai ignore

## Why
Andrew: job 5c1d24eb loaded no reference lyrics. LRCLIB and Genius both
returned the correct song, but the relevance filter scored them 26% and
25%, below its 30% threshold. Anchor coverage credits a repeated phrase
only once, so chorus-heavy songs undercount. The transcription was
decent, with a few consistent mishearings.

## What
`filter_irrelevant_sources(..., transcribed_text=)` now does a second,
**additive** check. A source the anchor filter rejects is kept when its
word-trigram overlap with the transcription clears four guards:
- **≥ 25%** of reference trigrams matched, repeats counted;
- **≥ 20** reference trigrams, so tiny references can't score by chance;
- **≥ 10** matches;
- **≥ 15 distinct** matched trigrams, so a shared refrain alone can't
pass.

Trigrams must contain three distinct words, so "na na na" and "oh oh oh"
don't count. Tokens are NFC-normalized and keep combining marks.

Both call sites pass the transcription: the corrector, which also serves
the review UI's lyrics search and add-source flow, and the controller's
uncorrected path. Without `transcribed_text`, behaviour is unchanged.
The rescue **never rejects** a source the anchor filter keeps. The
serialized `rejected_sources` metadata is unchanged.

## Calibration (459 recent production jobs; per-job reference txt vs
`(Lyrics Uncorrected).txt`)
- **Wrong-song baseline:** 491k cross-job pairs, excluding pairs with
the same title.
  - Worst genuine pair: 9.4%; p99.99: 5.5%.
- The only cross-job pairs that pass are one job whose audio is another
job's song under a wrong title. Their transcriptions overlap 45%, where
unrelated songs score under 10%.
- A naive n-gram rule would have admitted a wrong song at 41% through
shared "na na" filler. Requiring three distinct words removes it.
- **Rescued:** 22 sources that are currently rejected, **all the correct
song**, scoring 26–88%. Examples: the 5c1d24eb song, Deacon Blue, UB40,
Atomic Kitten, Sezen Aksu, Grateful Dead, Etherwood, Sub Focus, Terry
Allen.
- **Left rejected by the distinct-trigram guard:** one refrain-only drum
& bass track (8 distinct matches; it's the right song but has the risk
profile). This was a deliberate, cautious choice.
- No lyrics text is committed; the tests use invented lyrics.

## Testing
- `tests/correction/test_relevance.py`, 17 passed (8 new):
  - a repetitive correct song is rescued;
  - no transcription means unchanged behaviour;
  - a wrong song is still rejected;
  - filler doesn't count;
  - a tiny reference isn't rescued;
  - a source the anchor filter keeps is never rejected;
  - a shared refrain alone can't rescue;
  - combining marks are kept.
- `tests/unit`: 2184 passed. Running `tests/unit`, `tests/correction`
and the backend review tests together gives the same 12 order-dependent
failures on untouched main; they pass separately, as CI runs them.
- Local CodeRabbit: 2 cycles. Fixed the combining-marks and
distinct-trigram findings.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Jw21syo4HizJJjYw2mg2Vp

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.279.4
2026-10-20 00:08:38 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 b67b273056 feat(kjbox): decide partner secret for catalog resolve + fake judge + error monitor for decide search (v0.278.10) (#1190)
## ⚠️ Merge prerequisite (do BEFORE merging)

The Secret Manager secret `decide-gen-partner-secret` must have a
version before this merges, otherwise the CI deploy's `--set-secrets
...DECIDE_GEN_PARTNER_SECRET=decide-gen-partner-secret:latest` fails.

```
gcloud auth login admin@nomadkaraoke.com
cd ~/Projects/nomadkaraoke/karaoke-gen-decide-partner-resolve/infrastructure
GOOGLE_OAUTH_ACCESS_TOKEN=$(gcloud auth print-access-token) pulumi up     # review: should only add the decide-gen-partner-secret Secret
openssl rand -hex 32 | tr -d '\n' | gcloud secrets versions add decide-gen-partner-secret --project=nomadkaraoke --data-file=-
```

No extra IAM grant needed: gen's API runs as `karaoke-backend@`, which
already has project-level `secretmanager.secretAccessor`.

## Summary
- `POST /api/kjbox/catalog/resolve` now also accepts karaoke-decide via
header `X-Partner-Secret` (new setting `decide_partner_secret` / env
`DECIDE_GEN_PARTNER_SECRET`). Other kjbox endpoints stay
`X-Kjbox-Secret` only. decide gets its own AI-call budget (60/min) so
kjbox nights are never starved; the verdict cache is shared.
- Fake match judge / resolve for the local E2E stack (`FAKE_WORKERS`):
`judge_kind` / `judge_canonical_*` knobs; no Gemini or catalog calls.
- Error monitor now watches decide's new `karaoke-decide-search` service
and `song-index-build` job.
- Pulumi: declares the `decide-gen-partner-secret` Secret (value added
manually); CI deploy mounts it.
- Version 0.278.10.

## Testing
- [x] Changed-module + kjbox/catalog/error-monitor/fake-worker backend
tests pass (133 passed). Full `backend/tests` suite exceeds 10 min
locally, so left to CI.

## Review
- CodeRabbit CLI failed twice (WebSocket closed / connection error), so
no local CodeRabbit review completed; I did a manual diff review instead
(no issues found). Deliberately NO `@coderabbitai ignore`, so the GitHub
bot reviews this PR.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01VsLw9cTTMBab1tiRxCsYrm

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.279.3
2026-10-19 21:42:27 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 4286d98dbe fix: LRCLIB as a registered server-side app; honour Retry-After, retry 5xx (v0.279.2) (#1200)
@coderabbitai ignore

## Why
LRCLIB now applies stricter rate limits to unregistered clients
(https://lrclib.net/docs#implementation-requirements-rate-limiting), and
Cloud Run shares egress IPs. On 10-19, prod jobs got intermittent 503s,
and job 80e7ffdf lost its LRCLIB lyrics. LRCLIB is our only free source;
it supplied lyrics for 165 of 220 jobs in 30 days. Andrew registered a
server-side app.

## What
- **`karaoke_gen/lyrics_transcriber/lyrics/lrclib_client.py`** (new):
- Sends `Lrclib-App-Id` + `Lrclib-Api-Key` from `LRCLIB_APP_ID` /
`LRCLIB_SECRET_KEY`.
- Applies LRCLIB's rules: every 429 sets a **process-wide cooldown** of
`Retry-After` plus up to 30% jitter. The wait is never shortened, and
overlapping 429s keep the latest expiry. Nothing is sent until the
cooldown has passed. It waits and retries once only when the wait is
≤30s.
  - Retries 5xx after 1s/2s, honouring a 5xx `Retry-After`.
- A **401 on our credentials** logs a WARNING and drops to anonymous
access for the process, so a revoked key degrades instead of losing
LRCLIB.
- `LRCLIBProvider` (both `get` and `search`) uses it. The
`quick_version` renderer uses it with `retry=False`, so the fast path
gets the auth and the cooldown but never blocks.
- **Infra:**
  - Pulumi secrets `lrclib-app-id` / `lrclib-api-key`.
  - Env vars on the `lyrics-transcription-job`.
- The backend service deploy's `--set-secrets` (`ci.yml`). The
karaoke-backend SA already has project-wide `secretAccessor`.

## Deploy order (do NOT merge before steps 1–2)
1. `pulumi up` to create the two secrets and the job env. Needs
`/gcp-admin`.
2. Add the secret versions from Andrew's `.envrc` values.
3. Merge. The CI backend deploy references the secrets.

## Testing
- New `tests/unit/test_lrclib_client.py` (14 tests): headers; anonymous
use; Retry-After with jitter; a long Retry-After is not shortened and
blocks further calls; no-retry mode; a second 429; 5xx backoff; 5xx
Retry-After; 401 fallback; overlap merge (mutation-checked); provider
routes through the client.
- 164 passed across the client, `quick_version` and `search_lyrics`
tests.
- **Live** with the real app key: Dar Williams - Iowa and Wolfmother -
Where Eagles Have Been were found. One 503 was recovered by the retry.
- Local CodeRabbit, 3 cycles. I fixed the Retry-After cap, the cooldown
after a final 429, the `quick_version` 401 path, the overlap merge and
5xx Retry-After. I skipped one false positive about the tests' fake
clock (the retry path sleeps, then sends without re-checking the clock).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Jw21syo4HizJJjYw2mg2Vp

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.279.2
2026-10-19 17:33:19 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 653801cd04 fix: Musixmatch provider uses the RapidAPI v2 /lyrics?artist=&title= endpoint (v0.279.1) (#1199)
@coderabbitai ignore

## Why
BACKLOG "Musixmatch lyrics provider dead for 30+ days (0/287 requests
succeed)". We're still paying for the RapidAPI listing
(`cavsn/musixmatch-song-lyrics-api`, Pro $14.99/mo, active), but the old
`GET /lyrics/{artist}/{title}/` path now 404s for every song. I checked
the listing's playground (via Andrew's RapidAPI login). The API moved to
`GET /lyrics?artist=&title=` (optional `trackId`, `translate`) and
returns a flat `data` object.

## What
- `MusixmatchProvider._fetch_data_from_source` calls the new endpoint
with query params. A miss comes back as **HTTP 200 with `data.error`**,
or with empty `lyrics`. Both are now treated as not-found and logged at
INFO, so they don't alert.
- `_convert_result_format` reads `trackName` / `artistName` / `trackId`
/ `lyrics`. Raw cache entries in the old `macro_calls` shape still
convert.

## Testing
- New `tests/unit/test_musixmatch_provider.py` uses real response shapes
captured today: endpoint and params, the 200 miss logged with no ERROR,
empty lyrics, no key, v2 conversion, legacy conversion.
- 47 passed across the provider, base provider and `search_lyrics`
tests.
- **Live check** with the production key: Queen - Bohemian Rhapsody 47
lines, Adele - Hello 48 lines. The upstream misses some songs (piri -
dog; Taylor Swift - Shake It Off), so coverage is partial but no longer
zero.
- Local CodeRabbit: 0 findings.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Jw21syo4HizJJjYw2mg2Vp

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.279.1
2026-10-19 16:34:13 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 867617626a feat: requests board picks the visitor's language + header language switcher (#1198)
## Summary
Follow-up to #1193. `requests.nomadkaraoke.com` always served the
English board (the old redirect was also hard-coded to `/en/requests`).

- **Worker (`requests-host`, already deployed):** `/` serves
`/{locale}/requests/`. The locale comes from the `nk_requests_locale`
cookie, else `Accept-Language` (primary-subtag match like gen's
`LocaleRedirect`, so `pt-BR` → `pt` and `nn` → `nb`), else `en`.
`/{locale}/` (e.g. `/es/`) forces a language. Responses send `Vary:
Accept-Language, Cookie`.
- **Board header:** gen's `LanguageSwitcher` sits top-right. On the
vanity host it sets the cookie and reloads `/`, so the copyable URL
stays bare. On gen it behaves as before (`router.replace` to
`/{locale}/requests`).
- **`LanguageSwitcher`:** new optional `onSelectLocale` prop that
replaces the default navigation.

## Testing
- Worker: `node --test infrastructure/cloudflare/requests-host/`, 10
pass. This includes a check that the Worker's locale list matches
`frontend/i18n/routing.ts`.
- New `components/__tests__/LanguageSwitcher.test.tsx` (2 tests); the
703 component tests pass.
- Live checks against the deployed Worker: `es`, `ja`, `pt-BR`, `en-US`
each get the translated board, and the cookie beats `Accept-Language`.
- Mobile (390px) screenshot of a local build: the switcher shows as a
compact flag top-right and the dropdown fits the screen.
- Prod E2E spec extended with a browser-language check and a switcher
round-trip. It will run after deploy.

Local CodeRabbit review: no findings.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_016FYpoNVXzJzqKFaowqpwxr

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.279.0
2026-10-19 16:15:08 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 37b4f5126a fix: serialise translation-preview renders and keep them in GCS (OOM SIGKILLs) (v0.278.17) (#1197)
@coderabbitai ignore

## Why
BACKLOG "translation-preview renders get the API instance SIGKILLed
(OOM)" (#alerts msgs 1557981404242116608, 1560563690053439628). On 10-09
and 10-16, two concurrent `GET /api/themes/translation-preview` renders
for **different** languages, each a 4K ffmpeg frame taking about 20s,
were followed by `Container terminated on signal 9`. Every in-flight
request got a 503, including a real user's `create-from-search` on
10-09. The existing per-key lock only prevented duplicate renders of the
*same* language.

## What
- `_render_slot = BoundedSemaphore(1)`: one render at a time per
instance.
- **Persistent cache:** rendered frames go to GCS at
`themes/_translation_previews/<theme>/<lang>-<sha256(style_params +
sample)[:16]>.jpg`. Each language renders once per theme change rather
than once per cold instance, so very few requests ever wait on the slot.
A theme edit changes the digest, so stale frames are never served. Cache
reads and writes are best-effort: failures log a WARNING and fall back
to rendering. They use the bucket directly, so a cache miss isn't logged
as a "not found".
- **Fixed an existing test blind spot:** this module's autouse
`no_sleep` fixture stubs the global `time.sleep`. As a result
`test_concurrent_cold_previews_render_once` passed without any threads
overlapping, and so did my first version of the new concurrency test.
Both now wait on `threading.Event().wait()`.

## Testing
- New: a GCS hit serves without rendering; a theme change gets a new
key; four languages never render concurrently. Mutation-checked: with
`BoundedSemaphore(8)` it fails with `assert 4 == 1`.
- `test_lyrics_translation.py`: 33 passed. Local CodeRabbit: 0 findings.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Jw21syo4HizJJjYw2mg2Vp

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.278.17
2026-10-19 15:38:28 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 acb02e138a fix: un-pin audio-separator to 0.47.1 with RoFormer overlap=2 (v0.278.16) (#1196)
@coderabbitai ignore

## Why
BACKLOG "audio-separator 0.45–0.47 regression: mel-band RoFormer karaoke
models 3–5× slower on L4". The root cause is **python-audio-separator
#299 (0.47.0)**, not #298. #299 made RoFormer `overlap` a *window count*
that defaults to each model YAML's `num_overlap`. Before that, `overlap`
was effectively an 8-second step, which meant almost no overlap. The
karaoke models' YAMLs set `num_overlap` 4/8, so a 3-minute track went
from 23 forward passes to 87 (aufr33) and 124 (gabox_v2, becruily). That
matches the measured 25→75s, 26→149s and 27→157s. 0.45.0 and 0.46.0
still had the old step formula.

python-audio-separator #313 (released as **0.47.1**) keeps #299. It adds
a per-model forward-pass budget test on every PR and a GPU per-pass
timing test (T4: 1.21 s/pass), plus a README note. Andrew chose
overlap=2 for gen: smoother chunk seams than 0.44's no-overlap, at about
1.5× the 0.44 compute instead of 4–6×.

## What
- `karaoke_gen/audio_processor.py`: `SEPARATOR_MDXC_PARAMS` (overlap=2)
goes to all four `Separator(...)` constructions (stage 1, stage 2
karaoke ensemble, single-model paths) **only when audio-separator ≥
0.47**. On older versions `overlap=2` meant a 2-second step, which is 4×
slower, so they keep their defaults. Ensembles load each model through
`load_model`, so the param reaches every model in the preset.
- `backend/Dockerfile.gpu-base`: `AUDIO_SEPARATOR_VERSION=0.47.1`, with
the comment rewritten to explain the root cause. 0.47.1 needs
`torch>=2.3` on Linux, so the pre-installed `torch==2.6.0+cu126`
(required by the L4 driver) is kept.
- The poetry lock stays on 0.44.1. The version-aware params keep CI and
CPU images on the old semantics, and there's no torch churn.

## Expected cost per ~3-min track (forward passes)
| model | 0.44.2 | 0.47 YAML default | 0.47 + overlap=2 |
|---|---|---|---|
| karaoke_aufr33 | 23 | 87 | 44 |
| karaoke_gabox_v2 | 23 | 124 | 32 |
| karaoke_becruily | 23 | 124 | 32 |
| instrumental fv7z / resurrection | 23 | 32 / 21 | 32 / 21 |

## Verification plan (before /backlog-finish)
After the deploy, time the separation of the daily E2E or a real job on
the audio-separation-job logs. I expect roughly 1.5× the 0.44.2
karaoke-stage times (~40–50s per karaoke model rather than 75–157s). If
it's worse, revert the pin to 0.44.2. That's one line.

## Testing
- `tests/unit/test_audio_processor_local_gpu.py`: version gating
(0.47.x/0.48/1.0 → overlap 2; 0.44/0.46 → defaults; unknown → defaults;
fresh copy), and both stages' constructors receive `mdxc_params.overlap
== 2`.
- `make test`: the backend unit suite passes. The only failures are 8
`test_emulator_integration.py` tests, caused by my local environment:
another session's emulator setup is on :8080/:4443, and they fail
identically on untouched main. CI is green on main.
- Local CodeRabbit: 0 findings.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Jw21syo4HizJJjYw2mg2Vp

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.278.16
2026-10-19 15:09:41 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 92fd647203 fix: preview with untimed in-browser edits fills timings or 422s, not a KeyError 500 (v0.278.15) (#1195)
@coderabbitai ignore

## Why
BACKLOG "Lyrics preview 500 `KeyError 'end_time'` on unsaved edits"
(#alerts msg 1559340473439617117). On 10-12 a paying user's review
preview failed 3× (job 16792df7). Both the GCE and the local fallback
failed with `'end_time'`. `JSON.stringify` drops `undefined`, so a word
or line edited in the browser can arrive without its timing keys, and
`CorrectionOperations.update_correction_result_with_data` indexes
`w["end_time"]` / `s["end_time"]`.

## What
- `timing_sanitizer.sanitize_corrections`: a segment with missing or
non-finite bounds now gets them from its words' timings, or else from
the neighbouring lines' gap. It used to be skipped. The existing word
clamp then fills untimed words inside that window.
- New `find_missing_timings()` returns the text of a line that still has
no usable timing.
- `POST /api/review/{id}/preview-video` now sanitizes the payload up
front, as review submit already did. If a line can't be recovered, it
returns **422** `The line "…" has a word with no timing. Sync that line,
then try the preview again.` (en/es/de) instead of a 500.
- The frontend payload is unchanged. The backend now treats a missing
key and a null value the same way, so a frontend change would add
nothing.

## Testing
- `test_timing_sanitizer.py`: bounds from words, missing `end_time`
**key** (the 16792df7 shape), untimed line filled from its neighbours'
gap, unrecoverable line reported. The old "non-finite segment bounds
left untouched" test is replaced, because that is the behaviour this PR
changes.
- `test_routes_review.py::TestPreviewVideoMissingTimings`: a 422 naming
the line, with no render attempted; fillable gaps reach the renderer
filled.
- 131 passed across review routes, sanitizer, and submit-corrections
tests. Local CodeRabbit: 0 findings.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Jw21syo4HizJJjYw2mg2Vp

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.278.15
2026-10-19 14:57:13 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 3bbf397dc4 fix: keep auth, tenant lookup and job-log writes off the API event loop (v0.278.14) (#1194)
@coderabbitai ignore

## Why
The residual event-loop freezes BACKLOG item. Seven days of
`EVENT_LOOP_STALL in progress` stacks (2026-10-12→19, 355 samples),
grouped by their outermost app frame, ranked the app-level loop
blockers:

| Source | Samples | Max |
|---|---|---|
| in-process screens worker → job-log handler → sync Firestore write per
log line | 30 | 2.2s |
| `require_auth` → `validate_token_full` (session read on **every**
authed request) | 19 | 1.9s |
| preview encodes
(`_run_preview_encode`/`_encode_preview_in_background`) → job-log write
| 15 | 2.4s |
| `change_visibility` → sync Dropbox/GCS deletes | 6 | 1.2s |
| tenant middleware → `get_tenant_config` (GCS on cache miss) | 5 | 1.2s
|
| `require_review_auth` → `get_job` | 5 | 2.2s |
| `auto_correct_worker._run_suggest` → `get_job` (the only stall since
#1192) | 1 | 1.2s |

(The render/video in-process stalls stopped with #1192.)

## What
- **`backend/services/log_write_offload.py`**: when a job-log write is
made on a running event loop in the API process, it goes to one
background writer thread. The queue is bounded at 5000 and a drop is
logged at WARNING, rate-limited; `atexit` drains the queue. Writes stay
inline in threads and Cloud Run Jobs, so the worker code paths behave
exactly as before. Logs are read back ordered by their own `timestamp`,
so the hand-off doesn't reorder them.
- **`asyncio.to_thread`** for
`require_auth`/`require_auth_legacy`/`require_review_auth`, the tenant
middleware, `rerender_with_current_theme`'s tenant lookup,
`auto_correct._run_suggest`, and `change_visibility`'s `get_job`.
- **Visibility change:** `_delete_*_outputs` are now plain `def`, and
the sync steps run in one worker thread. Moving them off the loop
removed the implicit serialisation, so both directions now **claim
`visibility_change_in_progress` atomically** before touching anything.
They use the new `FirestoreService.try_set_job_state_flag`, a
transactional test-and-set. A lost claim returns 409. The
regenerate→private continuation passes `holds_claim=True`, because its
regenerate already set the flag. The claim is released if the reset
fails, and a raising screens trigger rolls back like a `False` one.
- **Guard (`test_no_blocking_async_routes.py`):** `BLOCKING_CALLS` gains
`validate_token(_full)`, `validate_session`, `get_tenant_config`,
`append_log_to_subcollection` and `append_worker_log`. Every `async def`
in `api/dependencies.py` and `middleware/` is now checked, not just
decorated routes. Mutation-checked: reverting `require_auth` fails the
guard.
- `docs/LESSONS-LEARNED.md` entry. About 70 direct
`get_job`/`update_job` calls in async routes remain. They're single
round-trips that rarely show in the watchdog, and the doc says to move
them when one does.

## Testing
- New: `test_log_write_offload.py` (inline vs offloaded, Cloud Run Job
inline, ordering, full-queue drop, Firestore append off the loop
thread), claim tests in `test_change_visibility.py` (lost claim, release
on failure, chained continuation, trigger exception, route 409), and
emulator `test_try_set_job_state_flag.py` (once-only, already set,
missing job, one winner among 4 concurrent callers).
- Backend: 6478 passed. The 8 failures are
`test_emulator_integration.py`, caused by my local environment: another
session's emulator setup has an `ssh` forward on :4443 instead of
fake-GCS, and they fail identically on untouched main. CI's emulator job
covers them and the new emulator test.
- Local CodeRabbit: 3 cycles. I fixed the claim race (two findings),
trigger-exception rollback, and the guarded claim release. I skipped two
minor nitpicks on the pre-existing trigger-failure rollback.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Jw21syo4HizJJjYw2mg2Vp

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.278.14
2026-10-19 14:41:34 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 b02cce3cc7 feat: serve the requests board on requests.nomadkaraoke.com itself (#1193)
## Summary
`requests.nomadkaraoke.com` used to 302 to
`gen.nomadkaraoke.com/en/requests`, so the short link couldn't be copied
out of a mobile address bar (Andrew pastes it into YouTube comment
replies). Now the board is served on the vanity host itself.

- **New Worker `requests-host`**
(`infrastructure/cloudflare/requests-host/`): `/` serves gen's
`/en/requests/` (query kept), paths with an extension are proxied from
gen, and every other path 302s to the same path on gen.
- **Pulumi (already applied):** Worker route
`requests.nomadkaraoke.com/*`; old redirect rule removed; DNS comment
updated.
- **Tenant detection:** `requests` added to the non-tenant subdomains.
The browser and Pages Function now share one list
(`frontend/lib/edge/nonTenantSubdomains.ts`); the browser copy had
drifted and was missing `requests`. Backend list updated to match.
- **Board "Make it yourself"** goes to `gen.nomadkaraoke.com/app` when
on the vanity host.
- **Docs:** runbook rewritten, including why a Pages custom domain
didn't work. Attaching the domain bypassed zone redirect rules and
served the gen homepage for ~3 min before rollback.

## Testing
- `node --test infrastructure/cloudflare/requests-host/`: 4 pass
- Jest tenant + new non-tenant list tests: pass. Backend
`test_tenant_middleware.py`: 36 pass
- `make test`: 6465 pass. 8 fail in `test_emulator_integration.py`, and
the same 8 fail on a clean `origin/main` checkout (pre-existing,
env-related)
- New prod E2E `e2e/production/requests-vanity-host.spec.ts`: board at
`/` with no redirect and the API working, other paths 302 to gen. The
pass/fail status of the "not a tenant" check (still sends `X-Tenant-ID:
requests` until this frontend deploys) is listed below.
- Manual browser check on prod: URL stays
`requests.nomadkaraoke.com/?utm_source=…`, board renders, no console
errors

| Prod E2E check | Before this deploy |
|---|---|
| serves board at `/` | ✅ |
| other paths → gen | ✅ |
| not treated as tenant | ❌ expected, fixed by this PR |

Local CodeRabbit review done (1 minor doc finding, fixed).

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_016FYpoNVXzJzqKFaowqpwxr

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.278.13
2026-10-19 14:18:57 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 1344a086da fix: worker Cloud Run Jobs never dispatch video/render into the API process (v0.278.12) (#1192)
## Problem: a regression from #1191, caught by #alerts triage
At 07:03 UTC on 2026-10-19, two new freeze alerts fired: **52s with 29
requests affected** (culprit
`firestore_service.append_log_to_subcollection`, during CDG packaging)
and **81s with 33 requests** (culprit `dropbox_service.upload_file`).
The stacks show `video_worker.generate_video` running **inside
karaoke-backend**.

Cause: in today's daily E2E run (job 5a5e5181), #1191's park/resume
worked as designed (parked 06:55:04, resumed 06:57:50). But the resume
runs in `audio-separation-job`, which doesn't set
`USE_CLOUD_RUN_JOBS_FOR_VIDEO`. So `trigger_video_worker` fell back to
Cloud Tasks, which runs it as a `BackgroundTask` on the API, putting the
whole video worker on the API's event loop. The same latent bug already
applied to the audio worker's auto-approval → render trigger.

## Fix
- `backend/config.py`: inside a Cloud Run Job (`CLOUD_RUN_JOB` is set by
the platform), `use_cloud_run_jobs_for_video` and
`use_cloud_run_jobs_for_render` default to **true**. An explicit env
value still wins. This covers every worker job (audio, lyrics, download,
...) with no infra change, through the normal image deploy.
- Error monitor normalizer: `requests_affected=N` →
`requests_affected=<N>`, so a stall culprit is one alert pattern instead
of one per request count.

## Testing
- New `backend/tests/test_config_cloud_run_job_dispatch.py`: inside a
job → true; outside → unchanged; explicit env wins.
- New normalizer test.
- `backend/tests` + `tests/unit/services/error_monitor`: 6871 passed.
- Local CodeRabbit review: 0 findings.

## Verify after deploy
The next parked-then-resumed job logs `Started Cloud Run Job
video-encoding-job` from audio-separation-job, and there are no
`EVENT_LOOP_STALL` lines on karaoke-backend with `video_worker` in the
stack.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_019pmG2XAgtUguEJWLFpkKHN

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.278.12
2026-10-19 13:17:49 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 0d0605d7b5 fix: final encode waits for stem separation instead of stranding the job; pin audio-separator 0.44.2 (v0.278.11) (#1191)
## Problem
The daily E2E Stage 2 failed on 2026-10-17 (run #216) and 2026-10-18
(#217). When a customer finishes review **before** stem separation does,
`video-encoding-job` fails its prerequisites (`Missing instrumental:
instrumental_clean`) 3× within seconds. The job then sits at
`instrumental_selected` ("Starting final encode") forever: it's never
marked failed, nothing alerts, and separation finishing doesn't
re-trigger the encode.

The race became common when the GPU image's unpinned `pip install
"audio-separator[gpu]"` picked up **0.47.0** on the 10-17 rebuild. On
L4, the mel-band RoFormer **karaoke** models run 3–5× slower than on
0.44.2, so total separation went from ~5 to ~10–16 min:

| model (~3-min track) | 0.44.2 | 0.47.0 |
|---|---|---|
| karaoke_aufr33_viperx | ~25s | 75s |
| karaoke_gabox_v2 | ~26s | 149s |
| karaoke_becruily | ~27s | 157s |
| instrumental fv7z / resurrection | ~50–60s | ~62s |

The model YAMLs still say `num_overlap: 8, batch_size: 1`, so this isn't
the #299 YAML-defaults change; the cause is somewhere in 0.45–0.47's
RoFormer inference changes. That root cause is tracked separately.

## Fix
- **Pin** `audio-separator[gpu]==0.44.2` in
`backend/Dockerfile.gpu-base` (`ARG AUDIO_SEPARATOR_VERSION`). Changing
this file triggers the GPU base image rebuild in CI.
- **Video worker parks instead of failing.** When the selected stem is
missing *and* separation is still running (`audio_complete` false, valid
status), it sets `state_data.video_waiting_for_stems`, puts a
user-visible message ("Waiting for audio separation to finish; the final
video starts automatically") and exits 0, so Cloud Run doesn't retry. It
re-checks after parking, and an atomic Firestore test-and-clear
(`claim_job_state_flag`) means exactly one of the video worker and the
audio worker proceeds if the stems land mid-park.
- **Audio worker resumes it.** After `mark_audio_complete`,
`resume_video_if_waiting_for_stems` re-triggers a parked encode. The
fake-workers pipeline does the same, so the local stack matches prod.
- **Backstop in `recover_stuck_jobs`.** `instrumental_selected` for more
than 10 min with the stem present → re-trigger the video worker (covers
a lost resume or lost dispatch). More than 60 min with separation never
finishing → `mark_job_failed` (`stage=video_waiting_for_stems`), so the
job fails loudly. Jobs older than 24h are never revived. (No jobs are in
`instrumental_selected` in prod right now.)

## Testing
- New `backend/tests/test_video_waits_for_stems.py` (16 tests): pending
detection (including that a wrong status never parks), park / continue /
claim-race outcomes, and resume (flag claimed / no flag / job moved on).
- `test_recover_stuck_jobs_repark.py`: 5 new sweep tests (re-trigger,
fresh, still separating, never-ready → fail, >24h ignored).
- Full `backend/tests` (excluding emulator): 6455 passed.
- Local CodeRabbit review: 0 findings.

## Verify after deploy
- The GPU job logs `Separator version 0.44.2`, and karaoke-model
separation is back to ~20–30s.
- The next daily E2E Stage 2 passes.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_019pmG2XAgtUguEJWLFpkKHN

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.278.11
2026-10-19 02:21:52 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 70b2f4279a feat(alerts): collapse error bursts, suppress link previews, de-noise #alerts (v0.278.9) (#1188)
Andrew's goal: "I'd like to make sure that our alerts channel only has
alerts in it that are actionable and important, or are just signal and
not noise." This follows #1186 (#noise routing).

## Changes
1. **Burst collapse** (`error_monitor/monitor.py`, `discord.py`,
`config.py`). When a run has more than 3 new patterns for one service,
including leftover un-alerted ones, the monitor sends ONE #alerts
message. It has the count, total occurrences, the top 5 by count,
first/last seen, and a `scripts/query-error-patterns.py --service X
--status new` pointer. All of those patterns are marked alerted. The
grouping is deterministic, since LLM grouping is off in prod. The
10-messages-per-run cap still applies, and unsent patterns are picked up
again next run. Evidence: on 2026-10-07/08, one mb-refresh run produced
73 separate alerts.
2. **No link previews.** Every error-monitor post (`_post`) and every
`ops_alerts` post now sets `flags: 4` (SUPPRESS_EMBEDS), through a new
`suppress_embeds` kwarg on `DiscordNotificationService.post_message`.
The YouTube "New upload" post keeps its embed.
3. **Loop watchdog.** `EVENT_LOOP_STALL in progress` is now always
WARNING. The `ended` line keeps the existing ERROR rule. In the last two
weeks there were 24 "in progress" pattern alerts on top of 15 "ended"
ones. I checked `infrastructure/modules/monitoring.py`
`EVENT_LOOP_STALL_LOG_FILTER`: it keys on `"EVENT_LOOP_STALL ended"` +
`severity>=ERROR`, so this needs no Pulumi change.
4. **Frontend chunk-load errors** (`frontend_ingestion.py`,
`firestore_adapter.py`). Reports matching `isChunkLoadError` collapse
into one pattern per build with `alert_min_hourly_count=10`. The monitor
holds that pattern un-alerted until more than 10 arrive within an hour,
which means a broken deploy. Separately, Safari/Firefox stacks now get
the message prepended before hashing (a pattern was named
`@https://gen.nomadkaraoke.com`). Chrome hashes are unchanged. Existing
Safari/Firefox patterns will re-key once, and burst collapse groups
them.
5. **known_issues.** Added a service-scoped ignore for
`github-runner-manager` "no available instance" aborts during scale-up
(seen 09-20, 09-23, 09-27, 10-08). The same message from other services
still alerts.
6. **divebar-mirror schedulers.** Added `attempt_deadline="600s"` on
both `divebar-mirror-daily` and `divebar-mirror-refresh`; the function
timeout is 540s. With the 180s default, Cloud Scheduler retried syncs
that were still running, and the overlapping runs hit OOM (4x in 30
days).

## ⚠️ Pulumi
`infrastructure/modules/divebar_mirror.py` changed, so **this needs
`pulumi up` by Andrew before merge**. Auto-merge is NOT enabled.

## Testing
- New and updated unit tests in
`tests/unit/services/error_monitor/{test_monitor,test_discord,test_frontend_ingestion,test_firestore_adapter,test_known_issues}.py`,
`backend/tests/{test_loop_watchdog,test_discord_service,test_ops_alerts}.py`,
and `tests/unit/test_divebar_mirror_scheduler.py` (an AST guard:
deadline > function timeout).
- `pytest backend/tests --ignore=backend/tests/emulator`: 6421 passed,
36 skipped.
- `tests/unit/services/error_monitor`: all pass.
- Local CodeRabbit review (`--committed`) completed with 0 findings.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_019pmG2XAgtUguEJWLFpkKHN

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.278.10
2026-10-19 05:13:13 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 9af5c8224e docs(medley): recommend backing_vocals_db -8, not -14 (#1189)
The customer reported that -14 dB made the backing singers inaudible
(about 25 dB under the band). -8 dB leaves them about 18 dB under: quiet
but present. This updates the MEDLEYS.md spec-table example and adds the
guidance. Docs only, so no version bump.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01QV2TQwyC7pwVkGPCk98Bqg

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 23:58:23 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 0e507852af fix(alerts): demote expected conditions off ERROR + Payment Link webhook fix (v0.278.9) (#1187)
## Summary

Alert hygiene, part of keeping Discord #alerts to actionable signal
only. ERROR is page-worthy: the error monitor turns every new ERROR
pattern into an #alerts message. This PR moves expected conditions down
to INFO/WARNING and fixes the one real bug among them, which was a
Payment Link checkout crashing the Stripe webhook. It covers the BACKLOG
items "Payment Link checkouts crash the Stripe webhook" and part of
"Alert-hygiene bundle".

## Changes

1. **Stripe webhook, non-gen checkouts** (`users.py`,
`stripe_service.py`, `stripe_admin_service.py`)
- A session that gen didn't create (a Stripe Payment Link:
`payment_link` is set, there's no gen metadata, and
`customer_email=None`) used to log ERROR "No user email in checkout
session". `store_payment` then crashed on `None.lower()`, so the payment
never reached `stripe_payments` or revenue.
- New `StripeService.is_gen_checkout()`. A non-gen checkout now logs
INFO "non-gen checkout (payment link ...)" and grants no credits. It is
marked processed for idempotency and is still stored, with
`customer_details.email`, `order_type="payment_link"`, `payment_link_id`
and the line-item description.
- `store_payment` now handles None for metadata, customer_details,
total_details and payment-method fields.
- **Not backfilled:** session `cs_live_a1ENsua3w6mDH04PPc…` (2026-10-13,
a $36 tip, no customer action needed). It could be backfilled through an
admin path that calls `store_payment` on the expanded session. GCP
writes from agents are restricted, so this PR doesn't do it.
2. **Lyrics providers** (`base_lyrics_provider._log_request_exception`):
a 404 means "no lyrics for this track" and now logs INFO. 429 moves to
WARNING alongside 5xx, connection errors and timeouts. Auth errors and
other 4xx stay ERROR. After 5 404s from one provider in a process, it
logs a single WARNING saying the provider may be dead (TODO points at
the Musixmatch backlog item).
3. **research_audio** (`audio_search.py`): no more
`AWAITING_AUDIO_SELECTION -> SEARCHING_AUDIO ->
AWAITING_AUDIO_SELECTION` transitions, which were invalid and logged 2
ERRORs per re-search. The job stays in AWAITING_AUDIO_SELECTION with no
status writes, so a crashed request can't strand it. After the search
the job is re-read. If a selection landed in the meantime, the fresh
results are discarded with a 409 instead of replacing the results that
selection indexed into.
4. **audio-download-job on a deleted job** (`audio_download_worker.py`):
if `get_job()` returns None at the start or after the download, or a
failure happens and a fresh read confirms the doc is gone, the worker
logs INFO and returns success, so it exits 0. That means no `fail_job`,
no further writes, no worker triggers and no Cloud Run retry. A GCS
NotFound on a job that still exists stays an ERROR. Seen with job
6ffe88ad on 2026-10-14.
5. **Loop watchdog**: "Failed to record loop stall (non-fatal)" now logs
WARNING with exc_info. Only that line changed, to keep the conflict with
feat/alert-noise-monitor trivial.
6. **Job-list queries** (`firestore_service.list_jobs`,
`list_jobs_summary`, `list_jobs_by_batch_id`): `query.stream()` now gets
`retry=` and `timeout=` of 30s. The client default is 300s for both,
which caused the "Error listing jobs summary: Timeout of 300.0s
exceeded" errors.

Version 0.278.9 (0.278.8 is taken by the still-open #1186).

## Testing

- New/updated unit tests:
- `test_stripe_webhook_non_gen_checkout.py`: webhook stores the payment,
logs INFO, logs nothing at ERROR, and marks the session processed.
- `test_stripe_admin_service.py`: Payment Link session and null fields.
- `tests/unit/test_base_lyrics_provider.py`: log levels for 404, 429 and
401, plus the dead-provider WARNING.
- `test_audio_search_research_attach.py`: no status writes, and a
selection made mid-search gets a 409.
- `test_audio_download_worker.py`: job deleted at start, mid-download,
or with NotFound on write.
  - `test_loop_watchdog.py`: recorder failure logs WARNING.
- `test_firestore_list_query_timeout.py`: bounded retry/timeout on all
three list queries.
- `pytest backend/tests --ignore=backend/tests/emulator`: 6434 passed.
`pytest tests/unit`: 2108 passed.
- Local CodeRabbit review: cycle 1 raised 2 findings (non-gen session
idempotency, re-search race), both fixed. Cycle 2 was clean.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_019pmG2XAgtUguEJWLFpkKHN

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.278.9
2026-10-19 02:58:58 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 a6d4163d2e feat(alerts): route non-signal Discord posts to #noise (v0.278.8) (#1186)
## Summary
Andrew asked for #alerts to carry only real problems. Routine and
informational posts now go to the new **#noise** channel. The GitHub org
PR webhook was already moved to #noise separately.

| Message | Before | After |
|---|---|---|
| Error monitor: New Error Pattern / Spike / Incident | #alerts |
#alerts |
| Error monitor: ✅ Patterns Auto-Resolved | #alerts | **#noise** |
| Error monitor: 📊 Daily Error Digest | #alerts | **#noise** |
| Nightly Backup Report (clean run) | #alerts | **#noise** |
| Nightly backup failed/partial (title is now "🚨 Nightly Backup FAILED")
| #alerts | #alerts |
| ✅ E2E Daily Test PASSED | #alerts | **#noise** |
| 🩹 E2E self-healed Stripe (auto-merge PR opened) | #alerts | **#noise**
|
| 🩹 self-heal PR needs CI approval click / could not open | #alerts |
#alerts |
| E2E / Tenant E2E failures, DR freshness stale, Job failed, Publish
completeness SHADOW, Gemini unavailable, OAuth credential alerts,
quarterly DR drill reminder | #alerts | #alerts (unchanged) |

## How the noise webhook is resolved
- **Backend / error monitor and the backup function** read it at
runtime: env `DISCORD_NOISE_WEBHOOK_URL` first, then the Secret Manager
secret `discord-noise-webhook` (the name can be overridden with
`DISCORD_NOISE_WEBHOOK_SECRET_NAME`). No Pulumi change is needed,
because both service accounts already have project-wide
`secretAccessor`.
- **GitHub Actions** uses `secrets.DISCORD_NOISE_WEBHOOK_URL ||
secrets.DISCORD_ALERT_WEBHOOK_URL`.
- **Nothing is silently dropped.** If the noise webhook can't be
resolved, or a post to it fails, the message goes to #alerts instead. In
the workflow this uses `curl -fsS` with a retry on the alert webhook.
- The error monitor now rate-limits each channel separately (10 per run
each), so noise can't use up the alert budget during an incident.

## Deploy notes
- **Backup Cloud Function:** CI doesn't deploy its source. Run
`infrastructure/functions/backup_to_aws/deploy.sh` after merge.
- **Error monitor job:** runs `karaoke-backend:latest`, so it picks up
the change once CI pushes the image.
- **No `pulumi up` needed.**

## Testing
- `tests/unit/services/error_monitor`: 376 passed. New tests cover noise
vs alert routing, fallback when the noise webhook is missing or fails,
separate rate limits, and secret resolution.
- `backend/tests` (excluding emulator): 6418 passed, 36 skipped.
- `infrastructure/functions/backup_to_aws`: 60 passed, including the new
`test_discord_alert.py` and routing tests in `test_main.py`. These were
run in a scratch venv because `functions_framework` isn't in the conda
env, which was already the case before this change.
- `actionlint` on `e2e-daily.yml`: no new findings. The only findings
are info-level shellcheck notes that were already there.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_019pmG2XAgtUguEJWLFpkKHN

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.278.8
2026-10-19 02:46:50 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 290f927175 fix(backend): move remaining event-loop blockers off the loop (v0.278.7) (#1185)
## Summary
Found while triaging Discord #alerts and 14 days of freeze telemetry
(2026-10-18). The event-loop watchdog still caught real freezes that hit
users:

- **151s freeze, 32 requests affected (2026-10-17 20:10 UTC).** `POST
/api/review/{id}/custom-lyrics/generate` called
`CustomLyricsService.generate` (a sync Gemini call) on the loop. Every
user on the instance saw "server unavailable". It now runs via
`asyncio.to_thread`.
- **14.9s freeze, 7 requests (2026-10-16).** The first `from
google.cloud import run_v2` in
`worker_service._trigger_worker_cloud_run_job` ran on the loop.
`google.cloud.run_v2` and `httpcore` (imported lazily by httpx) are now
in the startup `_LAZY_ROUTE_IMPORTS`, so they are imported off-loop
before readiness.
- **GCS on the loop in async routes.** `_prepare_theme_for_job` ran on
the loop in `upload_and_create_job`, `create_job_with_upload_urls`,
`create_job_from_url` and audio-search `search_audio`;
`submit_audio_edit` also uploaded `edit_log.json` on the loop. These now
go through `to_thread`.
- **Guard.** `upload_json`, `_prepare_theme_for_job` and `generate` are
added to the `BLOCKING_CALLS` denylist in
`test_no_blocking_async_routes.py`. The denylist caught the two extra
routes above.

## Testing
- `pytest backend/tests --ignore=backend/tests/emulator`: 6418 passed,
36 skipped.
- `test_audio_search` source check updated to accept the
`to_thread(_prepare_theme_for_job, …)` form.
- Local CodeRabbit review: 0 findings.

## Verify after deploy
No new `EVENT_LOOP_STALL ended … requests_affected>0` lines with
culprits `_call_gemini`, `run_v2` or `upload_json`.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_019pmG2XAgtUguEJWLFpkKHN

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.278.7
2026-10-19 01:56:13 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 9d6aa3df7d feat(medley): per-section backing_vocals_db for the karaoke track (v0.278.6) (#1184)
## Summary
The Fiverr medley customer said the backing singers in two songs were
too loud to sing over in tune, and asked for them as a faint background.

- **New option:** a per-section `backing_vocals_db` in the medley spec.
For the karaoke track only, the section is rebuilt from the clean
instrumental plus the backing-vocal stem at that level, e.g. `-14` for a
faint background. The rest of the karaoke track, the clean instrumental
and the guide are unchanged.
- **Cached mix:** the mix is cached per job/level, written to a
`.partial.flac` and renamed, so an interrupted run never leaves a
truncated mix to be reused. If the job has no backing stem, it stops
with a clear error.
- **Docs:** `docs/MEDLEYS.md` spec table.

## Testing
- `test_make_medley.py`: parsing, the ffmpeg graph, only the karaoke
variant changing, the atomic write and the cache. 20 + chord-chart tests
pass.
- Real data: the customer's medley with `-14` on Remember When and On
the Other Hand.
- Only those two sections change; the others differ by ≤ -43 dB, which
is crossfade overlap.
  - Overall section loudness is unchanged.
- A local CodeRabbit review found 1 issue (stale or partial cache). It
is fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.278.6
2026-10-18 15:04:43 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 03ee693f28 feat(medley): customer chord charts + static-gain master (v0.278.5) (#1183)
## Summary
Two follow-ups on the medley tooling, both from the Fiverr medley
customer's feedback.

- **Fixes the robotic artifact at the start of the karaoke audio.** The
final single-pass `loudnorm` ran in dynamic mode. On a song that opens a
cappella (HSLHT sings from 0.4s, band at 3s), the instrumental's first
3s is near-silent separation residue at -70 dB, and `loudnorm` boosted
it by about 50 dB into a loud warble. The master is now one static gain
to -14 LUFS plus a 4×-oversampled limiter at -1 dB (`master_filter`).
After the fix, the first 3s measure -61 dB (were -15 dB). Sync is
sample-identical and true peak is -0.9 dB.
- **Imports the customer's own chord chart.**
`scripts/medley/chord_chart.py` plus a new `make_medley.py chart` step
read a chords-over-lyrics chart (.txt or .docx in Courier, capo shapes,
`Capo N` / `Key change Capo 1` lines). They align it word-by-word to the
reviewed lyrics, so chords come out timed on the words the chart puts
them on. Detected chords fill intros, breaks and spoken verses,
transposed into the capo shapes.
- **Shows the capo in the chord box.** New `ChordChange.capo` field,
shown as "capo N" under the chord.
- **Docs:** `docs/MEDLEYS.md` gets a new section on importing a customer
chart, and the mastering guidance now says never to use single-pass
`loudnorm`. `docs/README.md` has an entry.

## Testing
- Unit tests:
  - New: `tests/unit/test_chord_chart.py` (9).
- Added: capo tests in `test_guitar_chords.py`, plus master and capo
tests in `test_make_medley.py`.
- Full `tests/unit` suite: 2100 passed, 65 skipped (run before the last
CodeRabbit fix; only the 9 chord-chart tests were re-run after it).
- Real data: the customer's 4-song chart against the medley. The
lyrics-with-chords output matched the chart line by line, and 4K frames
rendered with capo labels.
- A local CodeRabbit review found 1 issue (two chord groups after the
same word could come out in the wrong order). It is fixed and has a
test.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01QV2TQwyC7pwVkGPCk98Bqg

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.278.5
2026-10-17 14:24:45 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 9a44a30368 fix(review): idempotent correction-data review transition (v0.278.4) (#1182)
## Summary
The same race as #1181, on the other door into review. `GET
/api/review/{job_id}/correction-data` moves `AWAITING_REVIEW` to
`IN_REVIEW`. When two loads arrive concurrently, both see
`AWAITING_REVIEW`, so the second transition raises `in_review ->
in_review`. That returns a 500 and the review page shows "Failed to load
lyrics". The iOS in-app browser hits this every time; mobile E2E caught
it.

**Fix:** a shared `ensure_in_review()` helper in
`backend/services/review_state.py`, now used by both `correction-data`
and `start-review`.
- An already-`IN_REVIEW` job does not transition again.
- A lost race re-reads the job and carries on.
- Any other invalid transition still errors.
- `record_review_started` runs only for the winning transition.

Version 0.278.4.

## Testing
- 4 new `TestCorrectionDataReviewTransitionIdempotent` cases.
- 493 review and correction backend tests pass.
- Local CodeRabbit found nothing.
- Verified against the local stack via the mobile Maestro review-bridge
flow.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_011yLH4kWXMqjeKfZUG3ZiWZ

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.278.4
2026-10-17 12:07:03 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 072f85cad9 fix(review): idempotent start-review (no more in_review -> in_review error) (v0.278.3) (#1181)
## Summary
Opening a lyrics review could fail with `Invalid state transition
in_review -> in_review`.

`POST /api/jobs/{job_id}/start-review` always called
`transition_to_state(IN_REVIEW)` and never checked for `in_review`. So
any second open failed: a duplicate request, a double-mount, a second
tab, or the mobile app's in-app browser opening the review again. Mobile
E2E found this, and a screenshot captured gen's error page.

The endpoint is now idempotent:
- If the job is already `in_review`, it returns `{"status": "success",
"job_status": "in_review"}`.
- If the transition raises because a concurrent call won the race, it
re-reads the job. If the job is now `in_review`, it returns the same
success.
- Any other invalid transition still raises.
- `record_review_started` runs only for the winning `awaiting_review →
in_review` transition.

Version 0.278.3.

## Testing
- 4 new `TestStartReview` cases in `backend/tests/test_jobs_api.py`.
- The review-related backend tests pass: 452.
- Local CodeRabbit found nothing.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_011yLH4kWXMqjeKfZUG3ZiWZ

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.278.3
2026-10-17 10:20:58 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 2838d2dd1d fix(credits): keep credits_charged through job creation; one helper for confirm/refund/select (v0.278.2) (#1180)
## Summary
Fixes `credits_charged` being wiped from `state_data`, which made
duration confirm, refunds and the web cost dialog disagree. Mobile E2E
against the local stack found it: a 15-minute track showed "1 credit in
total (1 more)", and confirming the true total was rejected with 409.

**Root cause:** `create_job` debits the credit and records
`state_data.credits_charged`. But create-from-search, audio-search
create, made-for-you orders (which also lost `payment_bypassed`) and the
admin source override then replaced the whole `state_data` map, deleting
the field. Readers filled the gap with different defaults:
- reconcile: 1
- confirm-duration: 0
- 48h expiry: 0

**Fix**
- **Writers:** per-key `state_data.<key>` updates instead of whole-map
writes. Retry-from-scratch keeps the billing keys. Reconcile records
`credits_charged` when it pauses a job.
- **Readers:** one helper, `backend/services/job_credits.py`
(`get_credits_charged`, `is_payment_bypassed`). Every reader uses it:
reconcile, confirm-duration, `/select`, the bulk worker,
`_refund_credit_for_job` and the 48h expiry.
- For legacy jobs that lack the field, it returns what `create_job`
actually debited: 0 for bypassed, admin, tenant or no-email jobs,
otherwise 1.
- **`/select` and the bulk worker** never bill payment-bypassed jobs
(made-for-you, tenant, admin-created). This includes legacy ones whose
flag was wiped.
- **API responses:** `GET /api/jobs/{id}` and the job lists return the
resolved `credits_charged` when the stored field is missing. This is
response-only and never persisted, so the web and mobile clients
acknowledge exactly `credits_charged + pending`.
- **Confirm-duration** records the full total, so cancelling after a
confirm refunds it all. The 48h expiry refunds once:
`cancel_job(refund=False)` stops a second refund.
- **Admin source override** clears the old source's duration, selection
and confirm markers and keeps `credits_charged`. The next `/select` then
charges only the difference.
- **Web cost hook:** uses the server's resolved figure.
- Version 0.278.2.

## Testing
- New `test_credits_charged_consistency.py` covers:
- reconcile, then confirm with total 2, then cancel: 2 refunded, for new
and legacy jobs
  - 409 on the old total
  - refund amounts unchanged for jobs that have the field
  - `/select` and the bulk worker never deducting on bypassed jobs
  - the override followed by `/select` charging only the difference
- Stale-expiry legacy tests, and an emulator test for the dotted-path
merge (runs in CI).
- Backend unit tests: 6411 passed. Emulator integration tests run in CI;
locally they collided with a running local stack. Frontend jest: 1754
passed.
- An independent review found one Critical issue (`/select` billing
bypassed jobs) and one Important issue (web/server default mismatch).
Both are fixed and re-reviewed clean. Local CodeRabbit raised one minor,
which is fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_011yLH4kWXMqjeKfZUG3ZiWZ

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.278.2
2026-10-17 08:38:35 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 1616bc997b fix(api): sign final MP4s in GET /api/jobs/{id} download_urls (v0.278.1) (#1179)
## Summary
`GET /api/jobs/{id}` has always returned an empty `download_urls` for
real jobs. `JobManager.get_output_urls` signs only `job.output_files`,
and current workers never populate that field. Final videos live in
`file_urls.finals`.

This change also signs each `file_urls.finals` entry whose key ends in
`_mp4` (for example `lossy_720p_mp4`):
- URLs expire after 120 minutes.
- When a key exists in both, the `output_files` entry wins.
- Entries that aren't strings are skipped.
- After a real signing failure, it stops signing further entries, so a
stalled signer can't add a timeout per MP4.

**Why:** the mobile app (nomad-mobile Phase 1) streams the 720p MP4 in
`expo-video` and shows download progress. Both need a URL that supports
HTTP Range requests. The authenticated `/download/` proxy doesn't
support them.

Signing still goes through the bounded executor in `storage_service.py`
with its 12s timeout. The route is a sync `def`, so the event loop is
never blocked.

Version 0.278.1.

## Testing
- 6 new `TestGetOutputUrls` cases in
`backend/tests/test_job_manager.py`. That file passes: 58 tests.
- Local CodeRabbit raised one finding (the type guard), which is fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_011yLH4kWXMqjeKfZUG3ZiWZ

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.278.1
2026-10-17 05:25:45 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 695b8ddf84 feat: mobile app backend — handoff token, app checkout return, Google/Apple sign-in (v0.278.0) (#1178)
## Summary
Backend support for the Nomad Karaoke mobile app (nomad-mobile Phase 1,
"Make it").

- **`POST /api/users/auth/handoff-token`**: the app opens a job's web
review already signed in. The token is a single-use, 120s
`job_review:<id>` magic link, returned as
`{FRONTEND_URL}/{locale}/auth/verify?token=…&handoff=1`.
- The verify page skips the "Complete Sign-In" tap only when the URL has
`handoff=1` **and** the server confirms the token was minted as a
handoff (`link-status` → `handoff: true`). Emailed links keep the
scanner guard.
  - The web locale is validated against the 33 web locales.
- **Checkout `return_to: "app"`**: Stripe success and cancel go to `GET
/api/users/credits/app-return`, which 302s to
`nomadkaraoke://checkout?status=success|cancelled`.
  - The client never supplies a URL, so this adds no open redirect.
  - `session_id` is never forwarded.
- **`POST /api/users/auth/google` and `/auth/apple`**: ID-token sign-in
(PyJWT + JWKS: RS256, issuer, audience, expiry, `email_verified`). It
returns the same `VerifyMagicLinkResponse` as email-code sign-in,
through a shared `finish_verified_sign_in`.
- Email-code, Google and Apple all share the same abuse checks
(`_sign_in_checks`).
  - A JWKS outage returns 503 `identity_provider_unavailable`.
- Google audiences default to the new mobile web and iOS OAuth clients,
which are recorded in ARCHITECTURE.md.
- **Job summary** now includes `credit_refunded`.
- **Local stack:**
- fake-config `search_confidence` (perfect / recommended / unsure), with
a drift guard tied to the frontend `getSearchConfidence`
  - `youtube_url_enabled`
  - `fake:<email>` social tokens, accepted only in fake mode
- PyJWT is now a direct dependency (`pyjwt[crypto]`). Before this it was
only present transitively.
- Version 0.278.0.

## Testing
- `make test` is green:
  - CLI: 2088 passed
  - backend unit: 6360 passed
  - emulator: 156 passed, plus 4 handoff lifecycle tests
  - frontend jest: 1751 passed
- New suites:
  - `test_handoff_token.py`
  - `emulator/test_handoff_token_lifecycle.py`
  - `test_checkout_app_return.py`
  - `test_social_signin.py`
  - `test_magic_link_status.py`
  - `fake-search-tiers.test.ts`
  - verify-page tests
- Reviews:
- An independent code review was approved, and its hardening suggestions
were applied.
  - A local CodeRabbit review found nothing.

Plan: `docs/archive/2026-10-17-mobile-app-auth-handoff-plan.md`. Spec:
nomad-mobile
`docs/superpowers/specs/2026-10-17-mobile-phase1-make-it-design.md`.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_011yLH4kWXMqjeKfZUG3ZiWZ

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.278.0
2026-10-17 04:03:05 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 f8e9a8b3b6 feat(bulk): guided Artist → Album → Tracks steps in bulk album mode (v0.277.1) (#1177)
## Summary
Bulk album mode was hard to figure out — the one-line text hint was easy
to skip and it wasn't obvious the list was albums. This adds visual
cues:

- **Step strip** `1. Artist › 2. Album › 3. Tracks` with icons; current
step highlighted, completed steps show the pick and are clickable to
jump back (replaces back buttons)
- **Per-step headings** with icons ("Choose an album — Albums by Britney
Spears")
- **Artist/album results as clickable cards** (icon + chevron, hover
border)
- **Tracklist**: Select all / none + "N of M tracks selected"
- Icons on **By album / By text** tabs
- Album mode **hides batch settings + submit until a tracklist is
loaded** (no more "Create 0 karaoke videos" on step 1)
- Leaving the tracklist clears its rows; stale in-flight lookups are
dropped (CodeRabbit finding) so a previous album can't be submitted

Cover art deliberately not used (would hit MetaBrainz's Cover Art
Archive live).

## Testing
- New `frontend/__tests__/bulk-album-mode.test.tsx` (7 tests: step
navigation, select all/none, rows cleared on back, stale-response drop,
settings/submit gating)
- Full frontend Jest suite: 142 suites / 1741 tests passed
- Visually verified locally against prod API (desktop + 390px)
- 33 locales translated

Local CodeRabbit review completed; 1 finding fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01BrgLoS3bpnku9n21g6YGgw

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.277.1
2026-10-17 01:11:44 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 7cf5beede8 feat: guitar chords in karaoke videos + ChordMini ensemble chord detection (v0.277.0) (#1175)
## Summary
Adds optional guitar chords to karaoke videos, first used for a Fiverr
medley customer who wants chords on their video.

- **Rendering (gen library):**
`OutputGenerator.generate_outputs(chords=[ChordChange(start, end,
label)])`.
- Each chord change during a sung line is drawn above the word it lands
on, in a row reserved above every line (the same slot mechanism as
translated lyrics).
- Chords light up when played, using the same `\k` highlight as lyrics:
gold while upcoming (theme `chord_color`), sung-lyric colour once
played. Quick changes on one word never overlap.
- A top-right "now / next" chord box runs for the whole song, covering
intros and instrumental breaks.
- With no chords supplied, output is unchanged. A test asserts the ASS
is byte-identical.
- **Detection (`scripts/medley/detect_chords.py`):**
- Ensembles both ChordMini checkpoints (BTC + 2E1D, MIT; about 80%
majmin/MIREX, the best open weights we found) over the full mix, gen's
clean instrumental, and an htdemucs_6s harmonic mix (guitar + piano +
bass + other).
- Weighted per-frame vote, then merges short blips, snaps changes to
beats and spells chords for the key (Ab/Db/Eb7, F#m, borrowed
bIII/bVI/bVII as flats).
- **Medley script:** new `chords` step plus a `"chords": true` spec
option. Chords are clipped per section and hand over mid-crossfade, then
shifted for the countdown.
- **Docs:** `docs/MEDLEYS.md` § Guitar chords (setup, method, accuracy
caveats, theme keys) and a docs/README.md entry.

There's no job or UI option yet. The medley pipeline uses this for now;
a product version would need a chords toggle plus detection in a worker.

## Testing
- New: `test_guitar_chords.py` (12: chord timeline helpers, word
placement measured against the font, layout geometry, highlight timing,
now/next box, theme colour, no-chords path byte-identical),
`test_detect_chords.py` (7), and 2 medley chord-shifting tests.
- `tests/unit`: 2087 passed, 65 skipped.
- End to end on the 5 real medley songs: models/inputs agree on 91–99%
of major/minor labels, and the output matches the expected I–IV–V
progressions. Rendered 1080p and 4K medley videos with 146 chord
changes; spot-checked frames (verse, instrumental screen, chorus).
- Local CodeRabbit: 1 finding (doc section reference), fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01QV2TQwyC7pwVkGPCk98Bqg

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 00:24:11 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 8ec789a9aa fix(bulk): album lookup reads our MusicBrainz mirror, never the live API (v0.277.0) (#1176)
## Summary
Bulk → By album → "Britney Spears" showed **"Lookup failed. Please try
again."** `/api/bulk/album/artists` returned 502 because every attempt
to `musicbrainz.org/ws/2` from Cloud Run failed with
`httpx.ConnectError('')`. The same request from a home IP returns 200.
It last worked on 9/29. Most likely MusicBrainz is refusing our shared
GCP egress.

Per Andrew, we should only query our own infra, with no live-API
fallback. `musicbrainz_service` now reads **only our BigQuery
MusicBrainz mirror** (`karaoke_decide.mb_*`, which karaoke-decide's
`mb-refresh` rebuilds weekly from the official dumps).

## Changes
- **`backend/services/musicbrainz_service.py`**
- The HTTP client, throttle and retries for musicbrainz.org are removed.
Queries go through the BigQuery REST `jobs.query` endpoint (httpx + ADC;
no new dependency).
- Queries are parameterized, capped with `maximumBytesBilled` = 2 GiB,
and cached in memory for 10 min.
  - Lookups by table:
- artists: `mb_artists_normalized`. Ranking is exact match (including
"The "+q), then prefix, then substring, then popularity.
    - albums: `mb_artist_credit_artists` → `mb_release_groups`.
    - editions: `mb_releases`.
- tracks: `mb_release_tracks`, the new table in karaoke-decide#158,
clustered by release group + release.
- Measured against prod data: about 0.6–1 s per query, MB to 0.4 GB
billed.
- Canonical-pressing selection now ranks an edition by its best
release-event country.
- **Route:** passes `release_group_mbid` through on edition switches. An
unknown release returns **404** instead of 502.
- **Frontend:** `switchEdition` sends the album's `release_group_mbid`
so the lookup stays cluster-pruned. Older clients still work through a
one-off lookup that scans about 0.2 GB.
- **Infra:** the `karaoke-backend` SA gets project-level
`bigquery.dataViewer` + `bigquery.jobUser`. decide's Pulumi owns the
dataset ACL, so a dataset-scoped grant would fight it.
- **Docs:** `docs/API.md` and
`docs/archive/2026-10-16-bulk-album-musicbrainz-mirror.md`.

## Rollout (order matters)
1. Merge karaoke-decide#158, then build `mb_release_tracks`: by hand
(SQL in the archive doc, about $0.08) or at Sunday's refresh.
2. Run `pulumi up` for the IAM change. The agent's credentials are
read-only, so this needs Andrew.
3. Merge this PR.

## Testing
- `make test`: all passed (backend + 1735 frontend tests).
- New and rewritten unit tests cover normalization, BigQuery row
decoding, the REST request shape (no musicbrainz.org, billing cap,
params, cache), the HTTP/timeout errors, every lookup, the release-group
lookup path, and the 404.
- Ran the service live against the prod mirror: Britney Spears → 39
albums → …Baby One More Time (11 tracks, 6 variants); also checked
"beatles", "Beyoncé", "ac/dc", "Sigur Rós".
- CodeRabbit CLI: 1 minor finding (404 vs 502), fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01C1sMXk2MaxBLbep5jgiFab

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.277.0
2026-10-17 00:06:16 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 85390fd12b docs: medley-making guide + spec-driven make_medley script (#1174)
## Summary
Documents the multi-song medley process used for the first
classic-country medley, and turns it into a reusable script so the next
one only needs a spec file.

- `docs/MEDLEYS.md`: the full process. One gen job per original song,
then identical cuts on every audio variant, and the reviewed lyrics
shifted onto the medley timeline and rendered with gen's own
`OutputGenerator`. Also covers how to choose cut points (crossfades,
pot-out/pot-in, keys/tempos, checking the brief against the recording)
and gotchas.
- `scripts/medley/make_medley.py`: `fetch` (Firestore + gcloud CLI) →
`audio` (per-section LUFS gain, crossfade/gap/pot fades, `loudnorm` to
-14 LUFS / -1 dBTP, three variants) → `video` (shifted reviewed lyrics +
countdown + themed title/end screens + karaoke and guide MP4s).
- `scripts/medley/example-spec.json`: the country medley as a worked
example.
- Linked from `CLAUDE.md` Quick Reference and `docs/README.md`.

Ops tooling only. No backend or frontend change, so no version bump.

## Testing
- `tests/unit/test_make_medley.py`: 12 unit tests covering spec
validation, the timeline offset maths, lyric-segment shifting (including
the same song used twice) and the ffmpeg filtergraph.
- End-to-end run against the real prod jobs: `fetch` → `audio`
reproduces the hand-built v1 medley **bit-for-bit** (max sample diff 0.0
on both the instrumental and guide WAVs). `video --resolution 360p`
renders both MP4s and the LRC.
- Local CodeRabbit review: 1 finding (unchecked ffmpeg exit code in
`measure_lufs`), fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01QV2TQwyC7pwVkGPCk98Bqg

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 19:44:27 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 bb55efd3c2 fix(cloudflare): pin email catch-all name to stop refresh drift (#1173)
Every `pulumi preview --refresh` on the nomad-cloudflare stack planned a
no-op update on the 3 `EmailRoutingCatchAll` resources. The live rules
matched the code, but the provider reads back `name: ""`. Setting
`name=""` explicitly fixes it (same pattern as
`support_subaddress=False`).

Already applied locally. `pulumi preview --refresh` now reports **132
unchanged** (zero drift).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01QuVxnKWqXrdn3ux2VWLxS8

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Updated the configuration for catch-all email routing to specify an
empty name. This is an infrastructure-level change; no changes to
end-user email features or behavior are described.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 14:33:32 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 6b7e057e50 feat(cloudflare): Google login for Cloudflare Access (#1172)
Adds a Google identity provider to the Nomad Karaoke Zero Trust org, so
KJ Controller / FiverrBot login pages offer **Sign in with Google**
alongside One-time PIN and Cloudflare.

- OAuth web client `Cloudflare Access
(nomadkaraoke.cloudflareaccess.com)` created in GCP project
`nomadkaraoke` via the console (no API exists for web OAuth clients).
Redirect URI:
`https://nomadkaraoke.cloudflareaccess.com/cdn-cgi/access/callback`.
- Client ID in Pulumi config `googleOauthClientId`, secret encrypted as
`googleOauthClientSecret`.
- **Already applied locally** (`pulumi up`: 1 created, 131 unchanged),
so CI's run will be a no-op. Verified a full Google sign-in to
kjbox.nomadkaraoke.com.
- Access policies are unchanged; they still allow only the listed
emails/domains.

Infra-only change (no app code, no version bump). No local CodeRabbit
run, so the bot can review this one.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01QuVxnKWqXrdn3ux2VWLxS8

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Google sign-in is now available as an identity provider for Cloudflare
Zero Trust access, alongside the existing one-time PIN option.
* Production access now supports the Google OAuth credentials needed to
enable this sign-in option. Users can choose Google authentication where
it is configured, while existing one-time PIN access remains available.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 14:26:41 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 8a16e33a34 fix(review): show duet singer chips in Waveforms view (v0.276.3) (#1171)
## Summary
In duet mode, the Simple/Advanced lyrics views show a per-line singer
chip (● 1 / ● 2 / Both) to see and toggle which singer each line belongs
to. The Waveforms view was missing it, because `WaveformSegmentRow`
never got the duet props.

## Changes
- `WaveformSegmentRow`: new optional `singer` / `hasSingerOverrides` /
`onSingerChange` props render the same `SingerChip` after the
index/trash/play controls, plus `onFocus`/`onBlur`/`className` props.
- `TranscriptionView`: in duet mode, passes the singer, the singer row
tint and focus tracking to each waveform row, so the singer keyboard
shortcuts work there too. Moved the tint logic into a shared
`singerTintClass` helper.

## Testing
- New `TranscriptionView.waveformsDuet.test.tsx`: chips render per line
in duet mode, clicking cycles the singer, and no chips appear when duet
mode is off.
- `WaveformSegmentRow.test.tsx`: chip renders and cycles; focus and blur
get reported.
- Full frontend jest: 141 suites / 1735 tests pass.
- Local CodeRabbit review: no findings.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_015cvzKpvvadUsJGZFMv9DXy

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 13:38:18 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 3d4c9c4e6e fix(gdrive): retry Drive 5xx/429, check publish completeness after fallback pass (v0.276.3) (#1170)
## Summary
The Discord "Publish completeness SHADOW" alert for **NOMAD-1783** (job
`dd885ea9`) was a false alarm. Drive returned a one-off `HttpError 500
internalError` on resumable-upload init, and the worker's fallback Drive
pass re-uploaded all three files 43s later. Investigating it turned up
four gaps:

1. **No retry on Drive 5xx.** `gdrive_service` only retried connection
errors, so one 500 dropped the whole orchestrator upload.
2. **Check ran too early.** The shadow check ran inside the
orchestrator, before `_handle_native_distribution`'s fallback pass.
3. **Fallback dropped the original-vocals guide.** kjbox's "Original
Vocals" slider had no guide for NOMAD-1783.
4. **Recovered state was overwritten.** The main path persisted the
orchestrator's empty `gdrive_files` and the stale "Google Drive upload
failed" warning over the recovered state (`state_data.gdrive_files` was
`None`).

## Changes
- `is_transient_error`: Drive 429/500/502/503/504, plus 403 with
`rateLimitExceeded`/`userRateLimitExceeded`, now retry like connection
errors. Uploads get 4 attempts (2/4/8s backoff). Retries stay idempotent
because each attempt re-runs the same-name delete. With
`replace_existing=False`, only connection errors retry.
- `VideoWorkerOrchestrator.defer_publish_completeness_check`:
`generate_video_orchestrated` sets it and runs
`check_publish_completeness()` after the fallback pass.
`redistribute_video` is unchanged.
- `_adopt_fallback_gdrive_upload`: carries the fallback's file IDs onto
the result. The Drive-failure warning is dropped only when the recovered
set is complete.
- `OrchestratorResult.original_vocals_guide`: the fallback pass now
ships the guide too.

## Prod remediation (already done)
- Built the NOMAD-1783 vocals guide with the orchestrator's own
`_emit_original_vocals_guide` (5s intro, silent intro verified) and
copied it to `gs://nomadkaraoke-divebar-files/files/Nomad
Karaoke/vocals-padded/`.
- Restored `state_data.gdrive_files` from the actual Drive IDs (one file
per folder, no duplicate) and cleared the stale warning.
- Logs show no other `Google Drive upload failed` in the last 30 days.

## Testing
- New `backend/tests/test_gdrive_fallback_recovery.py` (26 tests): the
retry predicate, upload retry after a 500, no retry without cleanup,
deferred and default check, guide kept on a failed upload,
adopt/partial/no-recovery cases, and the fallback shipping the guide.
- Full backend suite: 6268 passed, 36 skipped.
- Local CodeRabbit: 2 cycles. Fixed partial-recovery warning handling,
403 rate-limit reasons, and the non-replace retry guard. Skipped
folder-create idempotency, since the MP4/, MP4-720p/ and CDG/ folders
always exist.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_0199crntgXrDkR93dQeqvnGY

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.276.3
2026-10-16 13:32:17 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 b2ef88a861 fix(review): unify backing-vocals recommendation on the 3-stem scorer (v0.276.2) (#1169)
## Summary
- `backing_vocals_analysis.recommended_selection` (legacy
`AudioAnalyzer` rule: keep only if >20% audible or a segment > −20 dB)
disagreed with the auto-approval `score_backing` verdict on **59/389
jobs (15%, Jul–Oct)** — every case 0.5–20% audible with no loud segment.
The finish modal followed the scorer, the instrumental screens followed
the heuristic, so the same job showed different "✓ recommended" stems
(job 58555663).
- Where they disagreed, admin/Andrew picks matched the scorer 14/18;
customer picks lean clean regardless (38% clean even when both signals
say keep). The heuristic also recommended keeping on 5 jobs the scorer
flagged as "backing stem IS the lead" (humans picked clean 4/5).
- 58555663 itself: the intro backing (16–29s) is a voiced B3–C♯4 hum
motif with the lead stem silent, pitch-matched to the "Mm-mm-mm" hum
lines — `with_backing` was right.

## Changes
- `unified_backing_recommendation` / `with_unified_recommendation` in
`auto_approval/instrumental.py`: scorer's confident verdict → `clean` if
backing-is-lead → else the heuristic (preserved as
`heuristic_recommended_selection`). Idempotent.
- Applied at write time (`screens_worker._analyze_backing_vocals`) and
read time (correction-data, `/instrumental-urls`,
`/instrumental-analysis`) so already-analysed jobs are consistent too.
- `backing_is_lead()` predicate extracted from `score_backing` and
shared.
- No change to auto-completion / enforcement (`backing_decision`
untouched); frontend unchanged.
- Docs: API.md note, LESSONS-LEARNED entry. Version 0.276.2.

## Testing
- [x] `make test` — backend 2054 + 6243 + emulator 152, frontend 1730,
all passing
- [x] New unit tests (helpers matrix), worker write-path test, route
test with the 58555663 signal profile

## Review
- [x] CodeRabbit CLI review completed locally — no findings

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_018aUxu88FZajta46BRdmpcc

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.276.2
2026-10-16 13:08:49 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 82a332dbe1 fix(review): proceed to instrumental review while stems separate; tolerate old backend (v0.276.1) (#1168)
Follow-up to #1167, from Andrew trying it on job 58555663.

## Changes
- **Don't block the CTA while stems are separating.** #1167 disabled
"Proceed to Instrumental Review" until the stems landed, but the full
instrumental screen waits for them itself, and reviewers in a hurry
(happy with clean) used to go straight there. The button is enabled
again, and the waiting box adds "In a hurry? You can continue to the
full instrumental review now instead."
- **Tolerate a backend without `audio_complete`.** The frontend deploys
well before Cloud Run, so the new page polled the 0.275.0 endpoint,
never saw `audio_complete`, and spun until the 10-min cap even though
separation had finished. The hook now treats "both stems present" as
ready when the field is absent.
- i18n: dropped `waitingForStems`, added `stemsWaitingProceed` (33
locales).

## Testing
- Jest: hook + modal 31/31 (new: legacy-backend fallback; waiting-state
CTA enabled + submits)
- Playwright `cloud-mode-review.spec.ts`: 13/13
- Local CodeRabbit: no findings

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01QKJ7TxzFoUeNTug1PR8Cwh

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.276.1
2026-10-16 00:42:50 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 abf7b693b4 feat(review): instrumental chooser appears when stems land after review opens (v0.276.0) (#1167)
## Summary

Recent jobs didn't show the instrumental chooser / backing-vocals
waveform in the Preview Video modal. Root cause: review opens as soon as
**lyrics** finish (~2 min after download), while audio separation lands
2-3 min later. The review page fetches `instrumental_options` once at
load, so stems that arrive mid-review never reach the modal, which
silently fell back to "Proceed to Instrumental Review".

Evidence (last 5 admin jobs): in 4/5 the review page was opened 11-23 s
after ready-for-review, and separation finished 1m46s-3m00s **after**
the page opened; the one opened after separation (6cb0f55f) was fine.

## Changes

- **Backend:** `GET /api/review/{id}/instrumental-urls` also returns
`audio_complete`, `job_status`, `backing_vocals_analysis` and a freshly
derived `auto_approval` summary (backwards-compatible).
- **Frontend:** new `useLateInstrumentalStems` hook. When the page
loaded without stems it polls every 15 s (max 10 min) until
`audio_complete`, which is set only after stems, OGGs and backing
analysis are all written, so the chooser never shows half its data.
Fresh data is overlaid as a separate modal prop, not merged into `data`,
so undo history, saved sessions and the in-flight preview render are
untouched. The getter is held in a ref because the page re-creates its
API client each render.
- **Modal:** while waiting, a spinner box ("Still separating the music
from the vocals…") and a disabled CTA ("Waiting for the instrumental…").
The chooser, ✓ badge, backing waveform and preview toggle appear in
place when ready. On failure or timeout it falls back to the old path
with a short note.
- i18n: 4 new strings, 33 locales.
- Fixed an existing Playwright test that expected "Proceed to
Instrumental Review" with both stems present (stale since v0.214; it now
picks Advanced mode).

## Testing

- `make test`: all pass (backend, plus frontend 140 suites / 1729 tests)
- New: 2 backend endpoint tests, 6 hook tests, 3 modal tests, 1
Playwright regression test (loads without stems → waiting state →
chooser appears)
- Local CodeRabbit: 1 finding (false positive: `advancedModeClean`
exists in all 33 locales)

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01QKJ7TxzFoUeNTug1PR8Cwh

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.276.0
2026-10-16 00:21:35 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 090bbf9ada feat(tenant-theme): CD+G instrumental/closing previews + transitions; per-tab previews, wider editor (v0.275.0) (#1166)
Follow-up to #1164, from Andrew's review of the theme editor.

## CD+G tab
- **All four CD+G screens previewed** from one real CDG: up card, lyrics
page, **instrumental break** (the sample lyrics include a gap ≥ a
preview threshold; the fully drawn frame is found by colour) and
**closing screen**. ~1.3 s.
- **"How it's drawn in"** for the up card, instrumental break and
closing screen. These are the "mask" images: cdgmaker's built-in
greyscale draw-order masks (`cdgmaker/transitions/*.png`, e.g.
`centertexttoplogobottomtext`) that decide the order tiles are revealed.
They change the animation, not the finished frame. Validated server-side
against the built-ins; the editor GET returns `cdg_transitions`.
- **Instrumental gap threshold** slider (3–120 s; stored in
centiseconds, validated 300–12000).

## Theme & style modal
1. ~96vw wide (max 1800px); the preview column is sticky.
2. **Each tab previews only its own screen** (Title card → title card,
Lyrics video → lyrics frame, End screen → end screen, CD+G → the 4 CD+G
screens, Advanced → all video screens). The server renders only what's
asked (new preview `kinds`), so previews are faster too.
3. CD+G previews in a compact 2×2 grid.

## Tests
- Backend: instrumental + closing frames render (distinct colours),
one-line sample still shows the break, `kinds` limits rendering,
transitions validated (incl. non-string), threshold bounds, editor lists
transitions, route `kinds` + 422 on unknown kinds.
- Frontend: per-tab previews/requests, Advanced shows all, modal width,
4 CD+G previews in a 2-col grid, transition picks saved, threshold
seconds↔centiseconds, unset placeholder. Full jest 1720 pass; backend
suite 6126 pass; package suite 2054 pass.
- 32 locales translated. CodeRabbit: 4 minor findings fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Gp5fBmsmWJxgkEHb4JfhXR

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.275.0
2026-10-15 16:01:56 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 c277bc8ee3 feat: local E2E stack (fake workers + test API) + email-code sign-in (v0.275.0) (#1165)
## Summary
- **Local stack:** `scripts/local-stack.sh up --frontend` runs the
**real** backend on the Firestore + GCS emulators with `FAKE_WORKERS=1`.
In-process fake workers move jobs through their real states in seconds,
using tiny real media, so web Playwright and mobile Maestro full-journey
tests can run offline against the same endpoints.
- **Email-code sign-in** (nomad-mobile spec §4.1): `POST
/api/users/auth/send-code` + `verify-code`. The code logic is shared
with kjbox via `LoginCodeStore`, and sends are throttled per email and
per IP.
- **Bug fix found by the new suite:** `confirm-duration` moved jobs to
`separating_stage1`, from which they can never reach review. It now
resumes at `downloading`. No prod job had taken this path yet.

## Changes
- `backend/services/fake_workers/`: dispatch seam in
`WorkerService.trigger_*`, plus fake search, download, checkout and
outbox, scenarios, and per-user config. The real stem upload, transcode,
backing-vocals analysis and preview render are reused. Every check is
strict (`is True`), so a MagicMock settings object never enables the
fakes, and startup **refuses FAKE_WORKERS in production**.
- `/api/test/*` router (mounted only in that mode): `status`,
`scenario`, `fake-config`, `emails`, `reset`, plus the fake Stripe
checkout page. Paying it runs the real webhook handler.
- `frontend/e2e/local-stack/` (9 specs, ~50s):
  - sign-up → make → review → instrumental → download
  - buy and cancel credits
  - duration confirm, failed, complete and review states
  - code sign-in and lockout
- New required CI job `local-stack-e2e`.
- Docs: TESTING.md § Local stack, DEVELOPMENT.md, API.md, README,
LESSONS-LEARNED.
- `MAX_SIGNUPS_PER_IP` can now be set by env. Only the local stack
raises it; production keeps the default of 2.

## Testing
- [x] Backend unit: 6175 passed
- [x] Emulator: 152 passed
- [x] Frontend Jest: 1706 passed
- [x] Local-stack Playwright: 9/9 passed (CI mode)

## Review
- [x] CodeRabbit CLI review completed locally. 1 minor finding declined:
it suggested swallowing infrastructure errors in send-code as fake
"sent" responses, which would hide real outages from users.

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01AGuxtgoscF6RJGtpjHPg3W

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 16:00:24 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 eeb08578c9 feat(tenant-theme): CD+G tab in the theme editor with real CD+G previews (v0.274.0) (#1164)
## Why
Commercial producers (Vocal Star first) care a lot about their CD+G
output, but the tenant theme editor only covered the MP4 title card /
lyrics / end screen. CD+G styling needed an admin to hand-edit the theme
JSON.

## What
**New "CD+G" tab** (only on portals with `features.enable_cdg` and a
`cdg` theme section):
- **Up card**: image, title/artist colours, title/artist sizes,
separator text between them (e.g. "MADE FAMOUS BY") + its colour, text
position from top (keeps text below a logo baked into the image),
title–artist gap.
- **Lyrics**: sung / unsung / screen / border colours, size, outline
width + colour, lyrics font ("Same as main font" or a distinct one, e.g.
condensed Oswald).
- **Instrumental breaks**: image, text, colour. **Closing screen**:
image, line colours.

**Real previews**: on the CD+G tab the preview request sets
`include_cdg`; the backend composes a short CDG with the real
`CDGGenerator` from the draft's `cdg` section and renders the up card +
a part-sung lyrics page with the new `karaoke_gen/.../cdg_decoder.py`
(single pass, multi-frame; moved out of the CDG test). 300×216 upscaled
3× with hard pixels, ~1 s. Other tabs never pay for it.

**Images**: uploads from the CD+G tab are fitted (letterboxed) to
300×216 PNG (`purpose=cdg`); the CD+G dropdowns only offer 300×216
images (`cdg_images` from the editor GET, header-only reads);
save/preview reject other sizes since the CDG renderer never resizes.
All 4 existing themes' CD+G images verified 300×216.

**Validation**: `cdg` hex colours, numeric bounds, single-line text
limits; distinct CDG font resolved for the preview the same way renders
do (`cdg_font` asset), title/outro use the main font.

## Tests
- Backend: CDG validation/fit/size-check/upload/editor list
(tenant_theme_service), real CD+G preview renders incl. colour change +
distinct font + missing asset + timeout (theme_preview_service), route
include_cdg + upload purpose. CDG title-card tests now use the shared
decoder.
- Frontend: new `TenantThemeEditorCdg.test.tsx` (10): tab gating,
includeCdg only on that tab, edits map to `cdg`, outline applies to both
strokes, font follow/distinct, fitted upload, CD+G-sized image list,
pixelated 300/216 previews. Full jest 1716 pass.
- Backend suite: all pass except `test_emulator_integration.py`, which
fails identically on main locally (a stray local Firestore emulator
enables it); CI runs emulator tests separately.
- 32 locales translated. CodeRabbit: 2 minor findings fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Gp5fBmsmWJxgkEHb4JfhXR

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.274.0
2026-10-15 15:02:24 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 35c0af444f fix(rerender): recover expired input audio before rendering (v0.273.6) (#1163)
## Problem
Re-rendering Vocal Star job `5fca18dd` (created 2026-08-25) failed in
the encoder:
`404 ... uploads/5fca18dd/audio/S1106-1 Ruby Murray - Softly Softly
Guide.mp3: No such object`.
Jobs created before v0.268.2 kept their uploaded mix and bring-your-own
instrumental only under `uploads/`, which had a 7-day lifecycle delete.
Every pre-October tenant job would fail the same way on re-render (incl.
the theme editor's "Re-render all").

## Fix
New `backend/services/input_recovery.recover_missing_inputs`, called by
the render worker (both GCE + local paths, off the event loop) and the
video worker, right after `ensure_job_inputs_persisted`:
- **Mix** (`input_media_gcs_path`) missing → repoint to the persisted
`jobs/{id}/input/` copy; else reuse a previous recovery; else rebuild
from the job's with-vocals render (`finals/with_vocals_mp4.mp4`,
`videos/with_vocals.*`): its audio track is the original mix (AAC).
Countdown padding (from `state_data.lyrics_metadata` /
`processing_metadata.rendering`) is trimmed; decoded to
`jobs/{id}/input/recovered_mix.flac` (input/ is keep-forever). Tries the
next video if one can't be decoded.
- **Instrumental** (`existing_instrumental_gcs_path`) missing →
persisted copy, else the staged `existing_instrumental.*` /
`custom_instrumental.*` at the job root or in `stems/` (kept by storage
retention).
- Nothing left → the job fails with an actionable message instead of a
raw 404. Recovery is recorded in `state_data.input_recovery`.

Verified on the real job's files: `with_vocals_mp4.mp4` audio = 148.69s,
same as `custom_instrumental.mp3`, same 0.61s leading silence (no
padding on this job).

## Tests
- `backend/tests/test_input_recovery.py` (17): no-op, persisted copy,
real ffmpeg rebuild, countdown trim, padding read from all known
locations, reuse of previous recovery, fall-through on an unreadable
video, actionable errors, instrumental repoint (4 locations), gs://
paths.
- Render-worker tests whose storage mocks report every blob missing stub
the recovery (autouse fixture).
- Full backend suite: 6083 passed.
- CodeRabbit: 3 findings fixed (padding location, to_thread,
fall-through);

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Gp5fBmsmWJxgkEHb4JfhXR

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.273.6
2026-10-15 14:35:52 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 23a0355363 fix(tenant): keep the tenant site title after Next.js rewrites <title> (v0.273.5) (#1162)
Follow-up to #1161. Prod check after #1161 showed the edge now serves
`<title>Vocal Star Karaoke Generator</title>` and the logo flash is
gone, but Next.js metadata rewrites the title on hydration and route
changes ("Nomad Karaoke: Generator", "Karaoke Generator"), overriding
both the edge-injected title and `applyTenantBranding`.

- `TenantProvider` pins `branding.site_title` on tenant portals
(MutationObserver on `<head>`); the default site is untouched.
- Tests: title restored after a simulated metadata rewrite; default
site's title left alone. Also unmount before resetting the store in
`afterEach` (an old mounted provider was refetching and polluting the
next test).
- CodeRabbit: one finding (my version bump had touched dev-dependency
pins) fixed; fresh re-review clean.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Gp5fBmsmWJxgkEHb4JfhXR

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.273.5
2026-10-15 13:28:55 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 c883173ca3 fix(tenant): no Nomad logo flash on tenant portals (v0.273.4) (#1161)
## Problem
Loading https://vocalstar.nomadkaraoke.com briefly showed the Nomad logo
(and on `/en/`, the whole consumer landing page) before the Vocal Star
branding appeared. The static export is prerendered with default
branding; the tenant swap only happens after hydration (measured: Nomad
logo in DOM at ~2ms, tenant logo at ~111ms, longer on cold loads). The
tab title also stayed "Nomad Karaoke: Generator".

## Fix
- Edge Pages Function (already knows the tenant) now also:
- sets `<html data-tenant-pending>` + `html[data-tenant-pending]
body{visibility:hidden}`, with a 4s safety reveal if JS never runs
  - preloads the tenant logo
  - replaces `<title>` with the tenant's `site_title`
- `TenantProvider` removes the marker once tenant config is initialised
(parent effect → runs after the tenant UI has committed; also on
config-fetch failure).
- Injection logic moved from `functions/[[path]].ts` into
`lib/edge/tenantHtmlInjection.ts` so it is unit-tested. Non-tenant hosts
are untouched.

## Testing
- New: `lib/__tests__/tenant-html-injection.test.ts` (hide/reveal script
executed in jsdom, timeout reveal, preload, title escaping, XSS
escaping, gs:// logo mapping, non-tenant passthrough) and
`components/__tests__/tenant-provider-reveal.test.tsx` (page is revealed
only after tenant branding rendered; revealed on fetch failure).
- Full frontend jest: 138 suites / 1704 tests pass. CodeRabbit local
review: no findings.
- Post-deploy: verify on vocalstar.nomadkaraoke.com with a Playwright
img/visibility timeline.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Gp5fBmsmWJxgkEHb4JfhXR

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.273.4
2026-10-15 13:03:57 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 0c1aa16469 Turn on conversion emails + fix doubled signature on credit-denied email (v0.273.3) (#1160)
## Summary
- **Conversion emails go live.** Andrew approved the copy, the 25%
offer, the timings and the 612 Joan St postal address. Adds
`CONVERSION_EMAILS_ENABLED=true,REVIEW_LAST_CHANCE_ENABLED=true,CONVERSION_OFFER_PERCENT=25`
to the backend `--set-env-vars` in ci.yml (feature shipped dark in
#1156).
- **Doubled signature fix.** The "About your Nomad Karaoke free credits"
(credit-denied) email appended `_get_email_signature()` itself and also
got it from the shared footer, so it showed twice. I audited all 22
`send_*` emails by rendering each one; this was the only duplicate.
- New regression test: renders every `EmailService.send_*` and asserts
the signature appears exactly once. It fails on main with
`send_credit_denied_email: 2`.

## Testing
- `make test` green (backend 2054 + 6071 + 150, frontend 1692).
- Local CodeRabbit review: no findings.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_019KuzHAU4rxkv8JfD8NmHTN

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.273.3
2026-10-15 12:44:43 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 182b4eace4 fix(tenant): second Dropbox pass keeps the tenant track-code prefix (v0.273.2) (#1159)
## Summary
Found by verifying a real Vocal Star portal job in prod after #1158 (job
`0976cc34`, code `E2E4635623`). The delivery folder had **both**
`E2E4635623 - Artist - Title (…)` and plain `Artist - Title (…)` copies
of every file.

The cause: `video_worker._handle_native_distribution` re-uploads the
folder after the orchestrator (adding `stems/` and `lyrics/`, with
`skip_unchanged`) without the tenant filename prefix. It now passes
`filename_prefix=tenant_track_code_prefix(job)`, so both passes write
the same names. The unchanged-skip then also matches, so files aren't
uploaded twice.

## Testing
- New test: the post-orchestrator pass passes the tenant prefix.
- Backend: 6069 passed (emulator tests excluded locally).
- Local CodeRabbit with `--fresh`: no findings.
- Will re-verify with a real prod job after deploy by listing the
Dropbox folder.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Gp5fBmsmWJxgkEHb4JfhXR

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.273.2
2026-10-15 02:53:21 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 621077474e feat(tenant): track code prefixes every delivered file + CDG title shrinks unbreakable words (v0.273.1) (#1158)
## Summary
Follow-up to #1153 and #1157, found by running a real Vocal Star job
through the portal in prod (job `8dc6945f`, E2E with track code
`E2E1531717`, which passed).

1. **Track code on every file.** #1153 put the tenant's code on the
Dropbox **folder** (`VS6588 - Artist - Title/`), but the files inside
were still `Artist - Title (…)`. Matt's April 2025 ask was the VS code
at the start of **each file**, and Andrew had confirmed "all files
prefixed with the track code".
- Tenant jobs with their own code now get `VS6588 - Artist - Title (…)`
on every Dropbox file.
- The entries inside the CDG/TXT zips are renamed too, so each CDG+MP3
pair still matches.
- Nomad's own `keep_brand_code` flows (finalise-only, re-render) are
unchanged, because this requires `tenant_id`.
2. **CDG title card:** a single word wider than the screen can't wrap,
so the card now shrinks the font instead of clipping it. Seen on the
first real prod render, where the E2E title has a 24-character timestamp
token.

## Changes
- `dropbox_service.upload_folder(filename_prefix=…)`.
- `prefix_zip_entries()` streams entries and keeps their
timestamps/compression, so a repack is deterministic.
  - `skip_unchanged` also works for repacked zips.
- `tenant_track_code_prefix(job)` is set into
`OrchestratorConfig.output_filename_prefix`. Tenant-only, and the code
is sanitized again before use.
- cdgmaker composer: width check in the title-card shrink loop.
- Version 0.273.0 → 0.273.1.

## Testing
- Prefixed upload, including zip entries and subfolders.
- No-prefix path unchanged.
- Deterministic, streamed repack.
- Tenant-only prefix, with sanitization.
- Real-render test for the unbreakable word.
- Package unit tests: 2054 passed. Backend: 6066 passed (emulator tests
excluded locally).
- CodeRabbit: 3 cycles, all findings fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Gp5fBmsmWJxgkEHb4JfhXR

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.273.1
2026-10-15 01:52:35 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 bfc8f7fb0f feat(cdg): configurable title card + per-theme CDG lyrics font (Vocal Star up card) (v0.273.0) (#1157)
## Summary
Vocal Star (and other commercial producers) need CDG files in their own
house style. Matt's 2025-02-20 email showed our CDG up card overflowing
its bordered design. He asked for his simpler card instead: logo on top,
then title, a small "MADE FAMOUS BY" line, then the artist.

Two pipeline limits blocked that:
- the CDG title card was hardcoded;
- a theme could only have **one font** for everything. The shared `font`
asset (`intro.font`) overwrites `karaoke.font_path` and `cdg.font_path`
at render time.

## Changes
- **Title card (cdgmaker composer):** new optional keys; all defaults
reproduce today's output.
- `title_separator_text` / `_color` / `_font_size`: a line between title
and artist.
  - `title_font_size`, `artist_font_size`, `title_max_height`.
- `title_block_top`: centre the block below a logo baked into the
background.
- It shrinks the title first, then the artist, so it can't loop forever.
- **Outro:** respects `title_block_top` and shrinks long text. Before,
an `assert` crashed when the text was taller than 200px.
- **Fonts:**
- The CDG title card and outro use the theme's main font
(`cdg.title_font_path`, fed by the shared `font` asset).
- A theme may keep a **distinct CDG lyrics font** via a new `cdg_font`
asset. The theme service emits it only when it differs from
`intro.font`, and `update_asset_paths` applies the shared font first so
section overrides win.
  - No visible change for existing themes, which all use one font.
- **Tenant theme editor:** keeps a distinct, available CDG font on save,
and copies it into the theme's assets if it's bundled.
- Docs: API.md.

## Testing
- New `tests/unit/lyrics_transcriber/output/test_cdg_title_card.py`
renders **real CDG files**, decodes the packets back into frames and
checks:
  - the separator sits between title and artist;
  - text stays below `title_block_top` and inside the safe area;
  - the title font is applied, and a missing one falls back;
  - long outro text shrinks instead of crashing.
- New tests for asset-mapping order, `cdg_font` emission, theme-editor
font handling, and the updated mapping assertion.
- Package unit tests: 2053 passed. Backend: 6056 passed (emulator tests
excluded locally).
- Local CodeRabbit with `--fresh`: no findings.
- Rendered Vocal Star samples (short, wrapped-long and very long
title/artist), decoded and visually checked against Matt's design.

After deploy, the Vocal Star theme gets: a new up-card background (logo
cropped from Matt's own card), separator "MADE FAMOUS BY",
`title_block_top`, black border, Avenir as the main font and Oswald kept
for CDG lyrics.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Gp5fBmsmWJxgkEHb4JfhXR

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.273.0
2026-10-15 00:58:42 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 5047f39da2 feat(tenant): per-track catalogue codes prefix tenant outputs (Vocal Star VS codes) (v0.272.0) (#1153)
## Summary
Vocal Star (Matt, April 2025) asked for their own VS catalogue code at
the start of every output file, with all files in one folder (e.g.
`VS6588 - Gerry and the Pacemakers - Ferry Cross The Mersey`). Tenant
operators can now set a per-track **Track code**. It becomes the job's
preserved brand code (`keep_brand_code`, which both render paths already
honour), so the Dropbox folder and every delivered file use it instead
of an allocated `VSTAR-NNNN`.

## Changes
- **Backend:** `POST /api/jobs/create-with-upload-urls` accepts
`track_code`.
  - Tenant portals only; `400` elsewhere.
  - Strict filesystem-safe validation, upper-cased.
  - Maps to `JobCreate.keep_brand_code`.
- **Bulk analyze:** a leading code in filenames (`VS6588 Artist - Title
Guide.mp3`) is split off and returned as `track_code`.
  - Files pair only when code **and** song match.
- LLM pairs with conflicting codes are rejected, and the code is
stripped from LLM-extracted artists.
- **Frontend:** an optional Track code field on the tenant single-track
form and on each bulk row.
  - Client validation mirrors the backend, with a visible error.
  - Resumed bulk rows keep their code, locked.
  - 33 locales translated.
- Docs: API.md. Version 0.271.0 → 0.272.0.

## Testing
- Backend: new tests for endpoint mapping, rejection outside tenants,
unsafe codes, normalisation, filename parsing, the band-name false
positive (UB40), same-code/different-song and LLM conflicts.
- Full backend suite: 5985 passed. The only 30 failures are emulator
tests, because the local Firestore emulator wasn't running.
- Frontend: full Jest 1680/1680 passing, plus new TenantJobFlow /
TenantBulkFlow tests. `tsc` shows no new errors.
- Local CodeRabbit: 3 cycles, all real findings fixed. The one remaining
note is on a pre-existing string.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Gp5fBmsmWJxgkEHb4JfhXR

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.272.0
2026-10-15 00:05:57 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 4ad946c3aa feat: free-to-paid conversion emails + last-chance review nudge, OFF until copy approved (v0.271.0) (#1156)
## Summary
This is BACKLOG Frog #2 ("plug the gen revenue leaks"), conversion-email
track. 44% of signups finish a free video, but only 6% ever pay, and
buyers decide fast (median about 5h). 45 users opened lyrics review and
let the job expire.

**Ships dark.** Nothing sends until Andrew approves the copy and
`CONVERSION_EMAILS_ENABLED` / `REVIEW_LAST_CHANCE_ENABLED` are added to
the ci.yml deploy env.

| Email | When | Stream |
|---|---|---|
| E1 "Your karaoke video is done. What's next?" + 25% off any pack for
48h | 2h after a never-paid user's **first** video completes | broadcast
|
| E2 "Your 25% off ends in 8 hours" | last 8h of the offer, if unpaid |
broadcast |
| R2 "6 hours left to finish your karaoke video" | 42h into review (the
existing reminder is at 24h; expiry is 48h) | transactional |

- `services/conversion_emails.py`:
- A `JobManager` hook marks the user on their first unpaid completion.
Tenant, Made For You and test accounts are excluded.
- The hourly `process-stale-reviews` cron sends the emails. It's
idempotent per user, and due times more than 24h stale are skipped, so
turning it on never emails a backlog. No infra change is needed.
- **Promo:** E1 grants `promo_discount_percent` /
`promo_discount_expires_at`. Checkout applies coupon
`conversion-offer-{pct}pct` when it beats a referral discount.
- The buy dialog shows a banner, and its prices use the effective
discount (CodeRabbit finding).
  - The email button links to `/app?buy=1`, which opens the dialog.
- **Compliance:** E1 and E2 go on Postmark's `broadcast` stream, where
Postmark manages unsubscribes (verified: it appends the link). They
include a postal-address footer (`COMMERCIAL_EMAIL_POSTAL_ADDRESS`).
`EmailProvider` gains `message_stream`, and email_log records the
stream.
- Copy is in `backend/translations` (33 locales). Docs:
`docs/CONVERSION-EMAILS.md`, covering how it works, measurement and how
to turn it on.

## Testing
- `make test`: backend 6017, emulator 150, frontend Jest 1688, package
2044. All green. Re-ran the targeted suites after rebasing onto the tip
PR.
- New tests:
- `test_conversion_emails.py` (26): eligibility, E1/E2 windows,
staleness, promo vs referral, broadcast stream, rendering.
- Stale-review R2 tests (3), `BuyCreditsDialog.test.tsx` (3), and an
AuthStatus `?buy=1` test.
- Rendered E1 and R2 HTML and checked them visually.
- Local CodeRabbit: 1 finding fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_019KuzHAU4rxkv8JfD8NmHTN

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.271.0
2026-10-14 23:50:59 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 542ad66b65 feat: opt-in 'tip the creator' add-on at gen checkout (v0.270.0) (#1155)
## Summary
This is BACKLOG Frog #2 ("plug the gen revenue leaks"), tip track. Tips
were **27% of old order-form revenue** ($950 on 178 of 231 orders), and
gen had nothing like them.

- Both gen checkouts (credit packs and Made For You) now offer an
**opt-in** Stripe Checkout `optional_items` tip: "Add to your order →
Tip the creator (optional) $5 [+ Add]".
- It's never pre-added. The old order form put a $5 tip line in the cart
at quantity 1 (opt-out), which is banned for UK/EU consumers.
- **Price:** $1/unit, found or created by
`lookup_key=gen_checkout_tip_usd_1`. It defaults to 5 units and is
adjustable from 1 to 100.
- Live product `prod_VRY9FhfNhiS0NN` already exists, with a description
explaining "each unit is $1".
- Any Stripe error means no tip is offered, so checkout is never
blocked. Kill switch: `CHECKOUT_TIP_ENABLED=false`.
- **Accounting:** credits are untouched. `stripe_payments.tip_cents`
records the tip separately.
- Referral earnings exclude tips. If the tip can't be read from Stripe,
the base is capped at the package list price (CodeRabbit finding).
- Docs: STRIPE-SETUP.md has a new "Checkout Tip" section.

## Testing
- Verified against **live Stripe**: created a real session (since
expired, nothing charged), and the hosted page shows the tip as an
opt-in "Add" item with the total unchanged.
- `make test`: backend 5977, emulator 150, frontend Jest 1676, package
2044. All green.
- New `backend/tests/test_checkout_tip.py`: price lookup and creation,
kill switch, optional item on both checkouts, opt-in only, tip_cents,
referral base.
- Local CodeRabbit: 1 finding fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_019KuzHAU4rxkv8JfD8NmHTN

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.270.0
2026-10-14 23:39:06 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 ff9b716435 feat: first-touch UTM/referrer signup attribution + UTM on YouTube description (v0.269.0) (#1154)
## Summary
This is BACKLOG Frog #2 ("plug the gen revenue leaks"), attribution
track. Until now we stored no acquisition source, so we couldn't tell
which channel brings the people who pay.

- **Capture (frontend):** `lib/attribution.ts` stores the first-touch
`utm_*` params, the external referrer host and the landing path for 30
days. It runs in the `[locale]` layout and in `LocaleRedirect`, before
`location.replace` wipes `document.referrer`. The data is sent with
`POST /auth/magic-link`.
- **Store (backend):** `services/signup_attribution.py` whitelists,
length-caps and sanitises the data. On account creation it sets
`signup_attribution` and `signup_source`: the `utm_source`, or a
referrer channel label such as `youtube`, `google`, `website`, `decide`
or `kjbox`. Malformed input never blocks sign-in, and the request field
is bounded (16 keys, values up to 500 chars).
- **Report:** `GET /api/users/admin/attribution-summary?days=90` returns
signups, verified, made a job, paid and revenue, per source. The admin
user detail shows the signup source and attribution.
- **YouTube:** description template links now carry
`utm_source=youtube&utm_medium=description&utm_campaign=requests|make_cta`.
They're static, so they're still derivable from the title, and the daily
backfill rolls them out. The `requests.nomadkaraoke.com` Cloudflare
redirect now has `preserve_query_string` (Pulumi `nomad-cloudflare`,
**already applied** locally).
- **Docs:** STRIPE-SETUP.md gets a "Legacy Payment Links" section. The
old order-form link `dR67sx…` was **deactivated 2026-10-14** and points
people to gen; it had 0 completed checkouts in about 30 sessions since
Dec. Also updated API.md.

Companion: nomadkaraoke/public-website#34 makes
`nomadkaraoke.com/r/CODE` pass the UTM params through.

## Testing
- `make test`: backend 5986 passed, emulator 150, frontend Jest 1683,
package 2044. All green.
- New tests: `backend/tests/test_signup_attribution.py` (cleaning,
channel labels, magic-link wiring, bounded payload, summary endpoint)
and `frontend/lib/__tests__/attribution.test.ts`.
- Local CodeRabbit: 3 findings fixed (malformed referrer, NaN expiry,
unbounded payload); re-review found 0.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_019KuzHAU4rxkv8JfD8NmHTN

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.269.0
2026-10-14 23:31:16 -04:00