433 Commits
Author SHA1 Message Date
Andrew Beveridge ☄️andClaude Opus 5.5 f4f0f6af77 feat(review): only show the review button once the job is fully prepared (v0.298.0) (#1273)
## Summary

The review button now appears only once a job is fully prepared: stems
separated, auto-approval re-scored with them, AI lyric corrections
generated and applied on the server, and review audio + waveforms
cached. The browser no longer applies AI corrections by itself, so there
is one code path.

Prompted by job ae995697 (a Made For You order): the review button
showed at 18:29, but the stems, backing verdict and vocal waveform only
existed from 18:32, and the server pre-apply had aborted on a
duplicate-word check, leaving the browser to apply corrections on load.

## Changes

- **Hold every job until stems land**
(`review_gate.route_after_scoring`). Before, only jobs whose lyrics
would auto-ship were held, so Made For You and lyrics-gated jobs opened
review early. Same kill switch (`AUTO_APPROVAL_AUDIO_HOLD_ENABLED`) and
20-minute backstop.
- **Generate AI suggestions on the API service before parking.** The
hold is released inside the audio-separation job, which has no LLM
credentials, so that job only applies an existing cache.
- **`enter_human_review` prepares everything before AWAITING_REVIEW**:
pre-apply (generating on a cache miss), then
`review_assets.prepare_review_assets` (OGG transcodes, backing waveform,
vocal peak envelope), then the transition.
- **Non-strict pre-apply for human review**
(`build_applied_segments(strict=False)`): what "Accept All" produces.
Stale/duplicate signals are recorded as `warnings`. The auto-ship path
stays strict.
- **Removed the browser's on-load auto-run/auto-apply** from
`useAutoCorrect`.
- **Job card shows "[5/10] Preparing review"** while held (new summary
field + 33 locales).
- Docs: ARCHITECTURE review-gate step,
`docs/archive/2026-11-19-review-ready-gate.md`.

Trade-off: Made For You and lyrics-gated jobs reach review (and send the
review-ready notification) ~3 minutes later.

## Testing

- Backend unit suite: 7294 passed. Locally failing: 8
emulator-integration tests (something unrelated listens on :8080 so they
don't skip; CI runs them against real emulators) and 1 locale test that
tripped on a gitignored translation snapshot (passes once removed).
- New/updated tests: review gate (review-verdict jobs held, kill switch,
prepare-before-transition order, suggestions generated before parking,
no double wait, missing job), non-strict apply, duplicate-word pre-apply
(ae995697 case), review_assets.
- Frontend Jest: 1834 passed; no type errors in touched files.
- Local CodeRabbit: 2 minor findings fixed, re-review clean.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVVdAxGitbw1FhYP4e4

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 16:12:58 -05:00
Andrew Beveridge ☄️andClaude Opus 5.5 ad64286004 feat(tenant): tenant portals are shared workspaces (v0.297.0) (#1266)
## Summary
Andrew's expectation: "all jobs in the tenant portal are accessible to
everyone who has permission to access that portal." Until now a
non-admin only saw/opened jobs whose `user_email` was their own, even on
a tenant portal. A client logging in with their own email never saw the
10 jobs bulk-uploaded for them under a separate `@nomadkaraoke.com`
address, and re-submitted every song.

## Rule (`backend/services/job_access.py`)
- Admins: every job.
- Owner: their jobs.
- **Tenant portal member** (currently allowed to sign in:
`allowed_emails` / `allowed_email_domains`, tenant active): every job of
that tenant. Re-checked per request against the live tenant config (~5
min cache), so removing someone from the allowlist removes access. Fails
closed on lookup errors.
- Consumer jobs (no tenant): owner-only, unchanged.

## Wired into
- `_check_job_ownership` (all job routes: view, edit, rerender, delete,
…)
- full-auth review access (`require_review_auth`:
lyrics/instrumental/audio-edit)
- file-upload guards (finalize, style uploads; style uploads keep
denying ownerless legacy jobs as before)
- `GET /api/jobs`: members on a tenant portal list all of that tenant's
jobs (still tenant-scoped)
- tenant theme refresh/outdated/rerender-outdated scope now covers the
whole portal (routes were already member-gated)

Unchanged on purpose: kjbox singer review links and the consumer
visibility toggle (consumer-only).

## Testing
- New `backend/tests/test_tenant_shared_jobs.py` (20): member/domain
member/case, outsider, consumer privacy, cross-tenant, inactive tenant,
no allowlist, removed member, fail-closed, token-without-email,
ownerless per-route, list scoping (member/outsider/consumer portal),
review auth 200/403, ownership wrapper.
- Updated theme tests that asserted the old owner-only scope; added
teammate-can-rerender.
- Local: 2,710 job/review/upload/tenant tests + 656 theme/tenant tests
pass (only `test_emulator_integration.py` fails locally, which needs the
Firestore emulator; same on main).
- Local CodeRabbit: 1 finding (ownerless legacy jobs on style uploads)
fixed; re-review clean.

## Prod cleanup done alongside (admin API)
Deleted (archived) the 10 stale Sept bulk-upload jobs for the randy-vild
tenant that duplicated his completed RVILD tracks.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVVUdF2J1bNeVa2uLXA

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 14:49:27 -05:00
Andrew Beveridge ☄️andClaude Opus 5.5 13619b7cc9 fix(review): preview shows the countdown intro like the final render (v0.296.0) (#1264)
## Summary
Songs whose vocals start within 3s get a "3... 2... 1..." intro in the
final render (`CountdownProcessor`: lyrics +3s, 3s leading silence), but
the **review preview never applied it**. So the preview opened straight
on vocals, and a customer (Randy Vild, "Caught Up Alive" / "Faithful and
True") asked us to add a countdown that was already coming.

## Changes
- `CountdownProcessor.add_countdown_to_lyrics_if_needed` (lyrics-only
half of `process`) + `padding_for_segments_data` (same rule on the
serialized segments the client sends).
- `CorrectionOperations.generate_preview_video` applies the countdown;
full local/CLI renders pad the audio too. Returns
`countdown_padding_seconds`. Countdown previews get a salted hash so
pre-fix cached previews aren't reused.
- GCE preview path: pads the already-downloaded audio on the API side
and uploads it to `previews/{hash}_audio.flac` for the encoder, so **no
encoding-worker change / version skew**.
- `POST /preview-video` responses carry `countdown_padding_seconds`.
- `PreviewVideoSection`: video time = song time + offset, so the
overlaid instrumental stem, waveform playhead and audition seeks stay
aligned (stem parked at 0 during the intro).
- Corrections that already contain the countdown aren't shifted again
but still get padded audio (CodeRabbit finding).

## Testing
- New:
`tests/unit/lyrics_transcriber/output/test_countdown_preview_helpers.py`,
`tests/unit/lyrics_transcriber/correction/test_operations_preview_countdown.py`,
GCE-path endpoint test in `backend/tests/test_routes_review.py`, 4
offset tests in `PreviewVideoSection.test.tsx`.
- Frontend jest (lyrics-review + lib): 892 passed.
- Real render check: local preview of job 508b38b2 (Faithful and True)
from prod corrections + audio with real ffmpeg shows the countdown, and
silence runs to 3.80s, matching the delivered final video's first word
(3.795s).
- Local CodeRabbit review: 2 findings fixed, re-review clean.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVVUdF2J1bNeVa2uLXA

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 13:56:20 -05:00
Andrew Beveridge ☄️andClaude Opus 5.5 2db5407a5d feat: archive deleted jobs (doc + logs) instead of losing them (v0.295.0) (#1261)
## Why
When NOMAD-1875 was deleted (2026-11-09), its Firestore record and logs
were gone for good, so we couldn't use them to debug the delete-outputs
result afterwards. Andrew's call: job records are **system data**. Keep
them forever, and on account deletion redact the user's identity rather
than deleting the record.

## What
- `FirestoreService.archive_job()` copies `jobs/{id}` to
`jobs_archive/{id}`, adding `archived_at`, `archived_by` and
`archive_reason`. It also copies the `logs` subcollection to
`archived_logs` without `ttl_expiry` (the TTL policy is on the `logs`
collection group, so otherwise they'd expire after 30 days), and the
`audio_edit_sessions` subcollection.
- `JobManager.delete_job()` archives first and **aborts the delete if
the archive fails**.
- Callers pass who deleted the job and why: `admin_delete` /
`owner_delete` from `DELETE /api/jobs/{id}`, and
`cleanup_distribution_delete`.
- **Account deletion** now also redacts archived jobs: owner/customer
email becomes `deleted:<uuid>`, and IPs, notes and contact fields are
removed. These are single-field queries, so no new composite index is
needed. Archived jobs are never deleted.
- Bulk test cleanup (`delete_jobs_by_filter`) still hard-deletes.
- `scripts/get_job.py` falls back to the archive. Documented in
`docs/ARCHITECTURE.md`.

No infra change: Firestore creates the collection on first write, and no
TTL policy is set because archived jobs are kept forever.

## Testing
- Unit: archive happens before delete; the delete is aborted when the
archive fails; archived jobs are redacted on account deletion (including
made-for-you customer jobs).
- Emulator: `backend/tests/emulator/test_job_archive_emulator.py` checks
the doc, logs (no `ttl_expiry`) and audio-edit sessions are copied, and
that a missing job returns False.
- `test_account_deletion`, `test_job_manager`, `test_jobs_api`,
`test_services`, `test_api_integration`: 195 passed.
- Local CodeRabbit: no findings.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVUwKapcjF8czVQrPD5

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 12:06:51 -05:00
Andrew Beveridge ☄️andClaude Opus 5.5 d63c9781e1 feat(replay): sync-triage verdict bar + vocal waveforms for stems-purged jobs (v0.292.0) (#1254)
## Summary
Andrew found the standalone triage page unintuitive, with no way to scan
visually. So the **real review UI in replay mode** becomes the triage
tool for the word-timing reference set (full-auto v2): anchors, gaps,
waveforms and audio he already knows.

## Changes
- **`ReplaySyncVerdict` bar** (replay only, admin): Perfectly synced /
Not perfect, reason chips, notes, Save / **Save & next job** through the
existing `queue` URL param.
  - It shows the saved verdict and flags unsaved changes.
- It saves via `POST /api/review/{id}/sync-reference`, the same store as
the review-modal checkbox from #1242, which now also accepts `reasons`
(trimmed, max 10).
- **Vocal waveform for old jobs:** storage retention purges stems after
30 days, so replaying old jobs showed no waveform. `_review_vocals_path`
now falls back to the backfilled `training/lead_vocals/{job_id}.flac`,
but only for stems-purged jobs. It's used by the vocals stream and
vocals-peaks.
- Docs: REPLAY.md § Sync triage.

## Testing
- Backend: reasons stored trimmed and capped; waveform fallback
(registered stem preferred, purged → training copy, unpurged never
checks storage). `test_routes_review` + `test_derived_vocals` +
sync-reference tests: 143 pass.
- Frontend: `ReplaySyncVerdict.test.tsx` (save & next, reasons + notes,
saved state, unsaved flag, disabled until chosen): 5 pass. tsc shows no
new errors.
- **Manual:** local replay (read-only against prod data) on a Feb-2026
stems-purged job (83414c76). The verdict bar renders, and Waveforms mode
shows the vocal waveform via the fallback (vocals-peaks 200).
- CodeRabbit ran locally; its one finding (unsaved state) is fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVVVPsvPbXFhRYY5VSJ

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 21:33:16 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 3a563ab3e6 fix(jobs): main is failing: restore the start-review guard removed twice (0.290.4) (#1250)
#1248 and #1249 were opened at the same time to resolve the same
duplicate guard. Each removed a different one, so `start_review` on main
now has none: 6 cases in `test_jobs_api.py` fail, no deploy can finish,
and a stale push on a finished song would 500 again.

Restores #1243's guard, as #1248 intended, and the test assertion #1249
loosened. `backend/tests/test_jobs_api.py` passes locally.

**Other sessions: please leave this route alone until this merges.**

No local CodeRabbit review on this small unblocking fix, so the bot may
review it.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVTvqXyL4QyA2EjzQMG

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 23:47:45 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 dd09aab941 fix(jobs): main is failing: resolve duplicate start-review guards (0.290.3) (#1249)
#1243 and #1246 both added a 409 guard to `start_review` and merged
minutes apart. #1243's test expects its own message, but #1246's guard
now runs first, so main's Backend Unit Tests fail (3 cases) and no
deploy can finish.

Keeps #1246's guard, because #1243's would refuse statuses that have a
valid transition to `in_review`. The race handler is unchanged. The test
still checks 409, no transition, and no enum text in the message.

`backend/tests/test_jobs_api.py` passes locally. No local CodeRabbit
review on this small unblocking fix, so the bot may review it.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVTvqXyL4QyA2EjzQMG

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* Review requests for jobs that cannot enter review continue to receive
a conflict response. Eligible jobs can proceed into review, with
review-start actions recorded only when a job transitions from awaiting
review.
* Updated the conflict-response test to check that internal status
naming is not exposed.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 23:35:29 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 5ae9d407b2 chore: remove duplicate start-review 409 guard (v0.290.3) (#1248)
#1243 and #1246 merged within minutes of each other and both added a 409
to `start-review`. This removes the one from #1246. The guard from #1243
stays, since it also handles losing the race to auto-approval. No
behaviour change for a finished job, and the #1246 test for it stays.

The two guards differ for statuses like `lyrics_complete`, which could
technically move to `in_review`: under the #1243 guard those already get
a 409.

Docs-free, small diff; no local CodeRabbit run, so the bot can review
it.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVTajJiZnzPUt9qsVEJ

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* Starting a review is now limited to jobs awaiting review or already in
review; other job statuses receive a conflict response.
* Repeated requests to open an eligible review remain safe, and
concurrent state changes are handled using the job’s current status.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 18:30:59 -05:00
Andrew Beveridge ☄️andClaude Opus 5.5 a10e548606 fix(url-jobs): retry a YouTube bot block, then point to Audio Search (v0.290.1) (#1244)
Implements fix C from section 4 of the 2026-11-03 alerts triage handoff.
Job 526bf46c, from a paying customer, failed within 3s of a YouTube bot
wall and showed the raw yt-dlp error. The credit was refunded, and the
customer then uploaded an MP3.

## Changes

- **Download:** `create-from-url` retries a bot-blocked download once
after 30s. Other download errors behave as before.
- **Still blocked:** the job fails with a translated message pointing to
Audio Search. The raw error goes in `error_details`. The response stays
a 500, because the frontend treats 503 as a transient backend blip.
- **Pre-check:** a bot-blocked availability check now waits 30s and
checks again before rejecting. No job or credit exists yet at that
point. A repeat block returns the same translated message instead of the
old hardcoded English. CodeRabbit raised this: without it, the retry
never ran when the pre-check caught the block.
- **Logging:** the bot-block paths log WARNING rather than ERROR,
because `fail_job` already raises the alert for the failure.

`audio_search.py` has its own bot-block handling, which this PR doesn't
change. The monitor and start-review changes are in a separate PR.

## Depends on flacfetch

The retries only help much with
https://github.com/nomadkaraoke/flacfetch/pull/61 deployed, since that's
what requests fresh cookies straight away. Without it, new cookies only
arrive from the keeper's 30-minute probe, so a 30s wait rarely helps.
This PR is safe to deploy first, but it adds up to 30s before the same
failure.

## Testing

- New `test_create_from_url_bot_block.py`, written test-first: retries,
final failure, pre-check, non-bot errors and all 33 locales.
- Backend tests selected with `-k "url or youtube or audio_search or
i18n or translation"`: 785 passed, 5 skipped. Emulator and integration
tests excluded. I did not run the full `make test`, so CI is the full
check.
- Backend translation key parity passes.
- Local CodeRabbit review: 1 finding, fixed. The fresh re-review had no
new findings.

**Not verified:**
- Nothing has run in production, and I created no test jobs.
- **The 32 new translations were written by hand, not by
`translate.py`.** gcloud auth had expired, so the Gemini key wasn't
available. To replace them, run `translate.py` for that key with a valid
login.
- ruff reports F823 for `asyncio` at about line 647 of `file_upload.py`.
It's also on main, in a different function, and I left it alone.

Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567241547739570298
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567252495535841386
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567252497620537415

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVTvqXyL4QyA2EjzQMG

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 23:18:13 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 08b146fce0 fix: no review push before AI suggestions are scored; start-review 409 after review ends (v0.290.1) (#1246)
Andrew's job `7f89f884` (2026-11-03) got a "review your lyrics" push,
then auto-approved about 3.7 minutes later. Tapping the push afterwards
left the app on its loader.

**False review push.** The screens worker's first score ran at 00:15:40,
before the proactive auto-correct finished at 00:15:46. Without
suggestions the lyrics scored `needs-review`, so `hold_for_audio` was
false and review opened with its push. The audio worker's re-score then
saw the suggestions (`auto/ai-resolved`) and auto-approved. In the last
14 days 5 jobs auto-approved, and 4 of them opened review first
(typically about 3.7 minutes before).

`route_after_scoring` now waits for the suggestions and scores again
when the first pass ran without them, then holds or auto-completes. If
suggestions can't be had, or the re-score reaches no verdict, the first
verdict stands and review opens as before.

**start-review on a finished job.** `complete -> in_review` raised an
invalid transition: a 500, an ERROR log (4 in prod for this job, which
the error monitor can turn into an #alerts message), and the app
retrying as if it were transient. It now returns 409 without attempting
the transition. The app's handling is in nomad-mobile #44.

**Tests:** `test_review_gate.py` and `test_jobs_api.py` pass with new
cases. `test_internal_api.py` has 7 auth test failures locally that also
fail on the main clone, so they aren't from this change.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVTajJiZnzPUt9qsVEJ

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 18:13:25 -05:00
Andrew Beveridge ☄️andClaude Opus 5.5 70bdd0e693 fix(alerts): start-review 409, per-cause crash patterns, one alert per job failure (v0.290.1) (#1243)
Section 1 of the 2026-11-03 #alerts triage handoff: a stale push on a
finished song got a 500, and the error monitor could stay silent on a
new crash.

Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566980744448843886
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566980746394996879
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566980747905073224
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1565697725280948294
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1565803187334815775
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1565818418072916110
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566225593354420226
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566256141074169987
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567207141994135654
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567252495535841386
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567252497620537415

## What changed

**1a. start-review:** a job not in review gets 409
(`jobs.jobNotInReviewState`) with no transition attempted. Losing the
auto-approver race is a 409 too. `correction-data` has the same race,
and its `except Exception` turned it into a 500; it now returns 409
(`except HTTPException: raise` added). Its status gate still returns
400, as before.

**1b. Crash patterns:** `Exception in ASGI application` and
`request_audit_error` become `Unhandled exception: <Type: message>`,
read from the entry's `exception` field. The handoff's suspicion is
confirmed in `upsert_pattern`: an existing `new` pattern returns
`is_new=False`, so a second, different crash stayed silent. Both log
lines of one crash now share one pattern. The path and method go in the
sample only.

I read the line after the last frame rather than the traceback's last
line. pydantic's last line is a docs link shared by every failure, and
group tracebacks end in a separator.

**1c. One alert per failure:**
- YouTube ids are normalized out of signatures, which also fixes the
false "first time" line in `ops_alerts`.
- `Job <id> failed` (both call sites in `job_manager`) logs WARNING,
since the status write sends the job-failed alert.
- `URL download failed:` is ignored in the monitor, not demoted, because
`file_upload.py` belongs to the section 4 PR.
- `[job:<id>] Download failed` logs WARNING for the expected
`DownloadError`/`FlacfetchServiceError` types. This wasn't named in the
handoff, but it is the same duplicate on the audio-search path.
Unexpected exceptions there stay ERROR with a traceback.

**1d. Suppressions and demotions:**
- Zalo's `zaloJSV2 is not defined`: dropped at `/api/client-errors`,
only with a Zalo UA.
- `python -c` experiments on `audio-separation-job`: ignored, using the
handoff's regex.
- Spotify 403 on `/lyrics`: WARNING with `response.text[:200]`. 401 on
`/lyrics` and 401/403 on `/search` stay ERROR.
- Dropbox token probe: WARNING. A rejected refresh used to log nothing,
so it now logs ERROR, as does having no refresh credentials.
- decide-search health aborts: this was a small change. `should_ignore`
takes an optional `request_url` and matches the exact path
`/api/search/health`.

**1e. Cold start:** `anyio._backends._asyncio` is added to
`_LAZY_ROUTE_IMPORTS`. The prod stack from 10-30 confirms the stall was
in that import. Lifespan also sends one `/api/health` request through
the app in a worker thread, bounded by the existing 30s timeout and
best-effort.

## Testing

- TDD: new tests failed first, then passed.
- Affected suites: 826 passed.
- `backend/tests` (excluding `backend/tests/emulator`): 7088 passed, 8
failed, 29 skipped. All 8 are in `test_emulator_integration.py`, and
that file also shows 8 failures at the base commit a8f9618a. I did not
compare the 8 test names one by one.
- `tests/unit` outside the listed files was not run.
`test_audio_fetcher.py` and `test_remote_cli.py` fail to collect in my
environment; I didn't change either file.
- Read-only replay of real prod entries through the new code: the 11-03
crash gave one specific pattern with `POST
/api/jobs/7f89f884/start-review` in the sample, the decide-search health
aborts and the experiment traceback were ignored.
- Local CodeRabbit review completed with 0 findings.

## Not verified

- Nothing is live, so the alert volume and the cold-start stall haven't
been measured. The warm request has only run in tests, not on a real
Cloud Run cold start.
- A request's first-use imports beyond the anyio backend aren't
individually checked.
- My test environment has no `flacfetch`, so its lazy imports only log a
warning here.
- Zalo: the trigger is one report (11-03).

## Known leftovers

- `validate_state_transition` still logs ERROR before raising, so the
auto-approver race (and the existing in_review race) still emits one
`Invalid state transition` ERROR even though the client now gets a clean
response. The stale-push case no longer reaches it. I left this because
other callers rely on that log.
- A crash that also logs its own ERROR first still gives two patterns,
the ERROR and `Unhandled exception`.
- The first bot block after deploy shows "first time" once, because the
normalized signature is new.
- Old `Exception in ASGI application` and `request_audit_error` patterns
stop recurring and will auto-resolve.
- If the section 4 PR rewords `URL download failed:`, the ignore entry
in `known_issues.py` needs the same wording.

## For Andrew (infra and cost, not changed here)

- Cloud Run startup probe and min-instances for `karaoke-backend`, if
cold-start stalls continue after this ships.
- decide-search's min-0 is why its health pings abort. The suppression
hides those, but real search paths will still alert on the same abort.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVTvqXyL4QyA2EjzQMG

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 23:04:04 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 a8f9618a07 feat(review): admin "perfectly synced" mark for the word-timing reference set (v0.290.0) (#1242)
## Summary
Andrew's review depth varies: Thursday live-event jobs often get only a
quick review. Treating every admin review as perfect sync truth
therefore poisons aligner training and evaluation (full-auto v2). This
adds an explicit, admin-only **"perfectly synced"** mark in the lyrics
review. Ticked jobs join the word-timing reference set; unticked means
unverified. Andrew's definition: a confident spot-check (anchors +
waveforms), not necessarily a full listen.

## Changes
- **Backend:** new `POST /api/review/{job_id}/sync-reference`
(`require_admin`).
  - Stores `state_data.sync_reference = {verified, by, at, notes}`.
- `verified: true` needs saved review output: the `file_urls` pointer,
or the direct GCS path for older jobs. Otherwise it returns 409.
Revoking is always allowed.
- **`correction-data`** returns `sync_reference`, so a re-review shows
the existing mark and can revoke it.
- **Frontend:**
- The review modal shows the checkbox plus an optional notes field, to
admins only.
- The modal passes the mark to `onSubmit`. `LyricsAnalyzer` saves it
**after** `submitCorrections` succeeds, so a mark can't sit on unsaved
timing. A failure shows a toast and never blocks the review.
- Once the box is touched, its current state is always sent, so it can
be revoked.
- **i18n:** 4 new strings × 33 locales.
- **Docs:** API.md. Version 0.290.0.

## Testing
- Backend: `test_review_sync_reference.py` (admin store, notes trim,
404, 422, 403 non-admin, 409 without review, direct-path fallback).
Related review suites pass (29).
- Frontend: `ReviewChangesModal.syncReference.test.tsx` (hidden for
non-admin, tick passes the mark, untick sends nothing, saved mark
pre-ticks and revokes, touched-then-unticked sends `false`). The modal
suites pass (29). tsc shows no new errors (21 pre-existing test-file
errors, the same on main).
- **CodeRabbit:** ran locally for 3 cycles and fixed 6 findings. One was
left by design: a pre-ticked mark on an edited re-review is visible in
the modal Andrew confirms, so submitting re-affirms it. The only user is
Andrew.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVVVPsvPbXFhRYY5VSJ

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 20:17:41 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 de5dfcd50d feat(encoding): AWS EC2 encoding worker as a pool candidate (v0.288.0) (#1236)
## Summary
Phase 1 of moving encoding to AWS. EC2 is paid for by the AWS Activate
credits (acct 790139457751, expiring 2028-10-31).
- **Plan:** `docs/archive/2026-10-29-aws-encoding-worker-plan.md`.
- **Benchmark** (3 real prod jobs replayed): an EC2 c7a.4xlarge matched
or beat GCE c4d-highcpu-16.

The EC2 box runs the **unchanged** `gce_encoding` worker service and
joins the existing candidate pool as `provider: aws`. The existing
failover, active override, lost-job detection and stockout cooldowns all
apply to it as they are.

## Changes
- **`backend/services/aws_encoding_compute.py` (new):**
`AwsComputeAdapter` looks to the manager like the
`compute_v1.InstancesClient` methods it uses.
  - EC2 states map to GCE statuses.
  - EC2 capacity errors are added to `CAPACITY_ERROR_CODES`.
  - A missing instance raises `NotFound`.
- Controller credentials come from Secret Manager
`aws-encoding-controller-credentials`.
- **`EncodingWorkerManager`:**
- `_compute_for(vm)` sends AWS VMs to the adapter; the GCE paths are
untouched.
- A shared `parse_fallback_vms` passes `provider`/`instance_id`/`region`
through.
- `ranked_candidates()` is used by both the encode path and the warmups.
- `ensure_preferred_running()` behaves **exactly like
`ensure_primary_running` whenever the GCE primary ranks first**, which
is today's case.
  - An AWS control-plane fault fails over to the GCE candidates.
- **Ranking:** `SPEED_RANK["c7a"] = 90`, so AWS is the last resort by
default. Firestore `config/encoding-worker.speed_rank_overrides =
{"c7a": 5}` makes it preferred, with no deploy.
- **Isolation from GCE-only tooling:**
  - The override now records `active_override_provider`.
- The CI blue-green deploy, `deploy_promote` and the idle Cloud Function
all ignore the AWS worker; it hot-updates its wheel per job.
  - A GCE promotion clears any stale provider tag.
- **Worker:**
- Uses `gcloud storage` on AWS, because gsutil can't use WIF credential
files. GCE keeps gsutil.
  - `infrastructure/encoding-worker/aws/` adds:
    - `provision.sh`, as EC2 user-data;
    - `startup-aws.sh`, uploaded by CI;
- `idle-watchdog.py`, an idle self-stop that runs as its own systemd
timer.
- **Infra (GCP), already applied with `/gcp-admin`:**
- The WIF pool `aws-encoding`, plus an AWS provider that only admits the
`nomad-encoding-worker` role, federated to `encoding-worker@`.
  - The secret `aws-encoding-controller-credentials`.
  - Preview and apply: 4 creates, 288 unchanged.
- **Infra (AWS):** a new Pulumi project `infrastructure/aws` (stack
`nomadkaraoke/prod`):
- the c7a.8xlarge **persistent Spot** worker, which stops on
interruption or shutdown;
  - an Elastic IP;
  - a security group on 8080 (API key, same as the GCE firewall);
  - the worker role with SSM;
  - a tag-scoped `nomad-encoding-controller` user.
- Preview: 10 creates. **I apply it after merge**, because the box
provisions from the deployed wheel and boot scripts.
- `boto3` is now an explicit dependency.
- CI also runs the `encoding_worker_idle` function tests, in the job
renamed "Infra - Cloud Function tests".

## Rollout (after merge)
1. **Stage A:** `pulumi up` the AWS stack. Store the controller key in
the secret, then add the worker's entry to
`encoding-worker-fallback-vms`. It's inert unless both c4d and c2d stock
out.
2. **Stage B:** a supervised `speed_rank_overrides {"c7a": 5}` window.
Run real jobs, then compare timings and outputs, including a visual
check of the portrait video.
3. **Stage C:** keep it preferred. After a clean week, delete the c4d
pair.

Phase 2, running distribution (Dropbox/YouTube/Drive uploads) from AWS,
follows in a separate PR.

## Testing
- **New tests:**
- adapter: botocore `Stubber`, covering the state map, capacity and
NotFound handling;
  - manager routing and ranking with overrides;
  - `ensure_preferred_running` branches;
  - failover on an AWS control-plane fault;
  - provider tagging;
  - deploy_promote exclusions;
  - the idle function skipping AWS;
  - watchdog decisions;
  - the gcs CLI choice.
- **Updated** the candidate and warmup-route tests to the new seams.
- **Full backend + unit suites** have no new failures. The local
emulator and cross-suite failures are identical on main.
- **Local CodeRabbit:** 2 findings, both fixed (failover on AWS faults;
stale provider tag).

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 17:42:35 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 bfcf1fd951 feat(auto-approval): hold confident-lyrics jobs out of review until stems land (v0.287.0) (#1232)
## Summary
In prod (2026-08-31 to 10-29), about a third of jobs were last scored
**before stems existed**. People opened the review link 15–40 s after
AWAITING_REVIEW. The audio worker's "second chance" re-score then hit
IN_REVIEW (`wrong_status`), so:
- 134 jobs never got a backing-vocals verdict;
- 7 jobs whose lyrics were in the auto tier went to a human anyway (e.g.
6c4be7d8, ed0b0c33, c4d8a256).

This is the first quick win from the full-auto v2 re-evaluation, which
targets >50% zero-touch jobs.

## Changes
- **`backend/services/auto_approval/review_gate.py` (new).**
- When the text verdict is AUTO and the only blocker is
`audio_incomplete`, the screens worker parks the job in
`GENERATING_SCREENS` with `state_data.auto_review_waiting_for_stems`. No
review link or notification goes out.
  - The hold is claimed atomically (`claim_job_state_flag`) by one of:
    - the audio worker once stems land (normal path);
    - the screens worker itself, if stems landed while it was parking;
    - `recover_stuck_jobs`, for holds older than 20 min.
- The claimant re-scores (`trigger="audio_hold"`), then either
auto-completes or opens review exactly like before, with server-side
pre-apply.
- If the post-claim re-score fails, the job falls back to human review.
If that also fails, the hold marker is restored so the sweep retries. A
job is never stranded.
- **Executor:** `hold_for_audio` is recorded in the verdict. It never
applies with `always_review`, `made_for_you`,
`backing_preference=review`, unconfident lyrics, or the kill switch on.
- **Kill switch:** `AUTO_APPROVAL_AUDIO_HOLD_ENABLED` (code default
true, because the executor runs in both the service and the
audio-separation job).
- Docs: LESSONS-LEARNED, README status row.

## Testing
- New: `test_review_gate.py` (11), `test_screens_worker_review_hold.py`
(2), executor hold cases (7), and recover-stuck-jobs hold sweep (3).
- Full backend suite locally: 6962 passed. The only failures are
`test_emulator_integration`, which needs the Firestore emulator and also
fails on pristine main.
- CodeRabbit CLI ran locally. Its one finding (stranded job on
post-claim failure) is fixed; the re-review is clean.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVVVPsvPbXFhRYY5VSJ

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 07:08:18 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 0d817509fa fix(api): event-loop stalls — visibility redistribution off-loop, route warmup, watchdog/Cloud Trace noise (0.286.2) (#1230)
## Summary

Fixes the 2026-10-29 #alerts event-loop stall batch (handoff
`docs/archive/2026-10-29-gen-event-loop-stalls-handoff.md` in the
workspace).

**1. Visibility change off the loop (39s stall, 13 requests incl.
/api/health)**
- `change_to_private` now runs claim, flag write, `redistribute_video`,
public-output delete and the final commit as **one** `asyncio.to_thread`
call (`_redistribute_and_commit_private`). `redistribute_video` runs on
that thread's own loop through the new `redistribute_video_off_loop`
(`asyncio.run(with_notification_drain(...))`).
- Is anything bound to the main loop? No. `redistribute_video` builds
its own `JobManager`/`StorageService`. The Dropbox, YouTube and GDrive
clients are sync. Job-log writes go through `log_write_offload`'s writer
thread. Notification tasks scheduled via `_schedule_notification` would
land on the private loop, so `with_notification_drain` drains them
before the loop closes, as the Cloud Run Job entry points do.
- Why one thread call and not just the redistribution (from CodeRabbit):
a thread can't be cancelled. So a cancelled request can no longer leave
a committed private archive with its public outputs still live, or a
claimed flag with no redistribution.
- Guard: `redistribute_video` is added to `BLOCKING_CALLS`, and
`visibility_change_service.change_to_private`/`change_to_public` are
added to `AWAITED_ROUTE_HELPERS`. I checked that putting back `await
redistribute_video(...)` fails the test.
- **Double Dropbox upload confirmed and fixed.** GCS `finals/` holds
every output twice: short canonical keys written by `_upload_results`
(`lossy_4k_mp4.mp4`, …) and the readable encoder names (verified on job
0f211e97: 17 objects / 525 MiB). Redistribution uploaded both. The
short-named `lossless_4k_mp4.mp4` also slipped past the `" (Final
Karaoke Lossless 4k).mp4"` Dropbox exclusion.
`_dedupe_redistribution_files` now keeps one copy per final, under its
readable name:
- a short name whose readable twin exists (twin found by suffix) is
dropped;
- a short name with no twin is renamed to the readable name (CDG/TXT
zips included).

**2. First-request stalls on new instances**
- Lifespan builds FastAPI's lazy route contexts
(`fastapi.routing.iter_route_contexts` +
`effective_low_priority_routes`) in a thread before readiness, with a
timeout and best-effort error handling. The results are cached by routes
version under a lock, so the OTel `_get_route_details` walk no longer
compiles route regexes on the loop. It is a no-op on FastAPI versions
without these internals. Note: the lock pins 0.128 but prod installs
unpinned (`pip install -e .`), so prod runs a newer FastAPI. I checked
against FastAPI 0.162: the first walk builds 304 contexts, the second
walk and the OTel lookup are cached (<1ms).
- `flacfetch.core.manager`, `flacfetch.downloaders.youtube` and `yt_dlp`
are added to `_LAZY_ROUTE_IMPORTS`.
- `search_audio` auto_download: `select_best` now goes through
`asyncio.to_thread`, with `select_best` added to `BLOCKING_CALLS`.

**3. Loop watchdog false positive**
- `_finish` logs WARNING, not ERROR, when all of these hold: no non-idle
sample was taken, a process freeze was observed (`frozen_s > 0`), and no
request was in flight when the stall began. Every "affected" request
therefore arrived while the instance was thawing. An unsampled stall
with no observed freeze, or with a request in flight at its start, stays
ERROR. The freeze requirement came from CodeRabbit.

**4. Cloud Trace exporter noise**
- `cloud_trace_export` ignore pattern for `Error while writing to Cloud
Trace`. CodeRabbit suggested matching only the transient gRPC detail. I
didn't: the error monitor only sees `jsonPayload.message` (the exception
is a separate field), and a persistent loss of telemetry isn't
#alerts-worthy anyway.

Smaller change: `change_to_private`'s Firestore `job_ref.update` writes
now also run off the loop.

## Testing
- New `backend/tests/test_redistribute_video.py`:
  - dedupe unit tests;
- an end-to-end `redistribute_video` run with mocks, checking the files
the Dropbox stage sees;
  - the off-loop runner uses its calling thread;
- `change_to_private` runs the redistribution on another thread and
loop;
  - a cancelled request still deletes the public outputs.
- `test_loop_watchdog.py`: frozen + unsampled + late arrival gives
WARNING; a request in flight at the start gives ERROR; no observed
freeze gives ERROR; a sampled stall with a late arrival gives ERROR.
- `test_main_startup_warmup.py`: lazy imports list; the route-context
walk; no-op without the API; failure is non-fatal; lifespan warms
off-loop before readiness.
- `test_known_issues.py`: the Cloud Trace pattern matches, and other
trace errors are not ignored.
- Local runs: `pytest backend/tests/ tests/unit/services/error_monitor
--ignore=backend/tests/emulator`: 7369 passed, 29 skipped. Not counted
in that total: `test_emulator_integration.py` fails 8/11 the same way on
the base commit, because a stray emulator from another session is on
:8080 locally. That's environmental, not caused by this PR.
- Local CodeRabbit: 3 cycles. Cycle 1: 3 findings (2 fixed, 1 skipped
with the reason above). Cycle 2: 1 finding, fixed. Cycle 3: 0 findings,
but CodeRabbit reported it reused cached results rather than a fresh
review.

## Not verified
- No prod run of a real public→private change yet. After deploy, the
Dropbox folder of the next one should contain each final exactly once,
and no `EVENT_LOOP_STALL` should appear during it.
- Route-context warmup was checked only against FastAPI 0.162 in a
scratch venv, not the exact prod version. Watch for the `Warmed N route
contexts` log line on the next cold start.
- Existing private Dropbox folders that already hold the duplicates are
not cleaned up by this PR.

Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1563719390849929330
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1563009782363721760
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1563039889354784882
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1563221363278348514
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1564308425033252895
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1563160849336762449

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVTiUPKxn3sQPyp3Teh

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 02:49:20 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 331d18ddf8 kjbox: one-tap Google/Apple sign-in for singers; sessions last 30 idle days / 365 max (v0.286.0) (#1226)
## Summary

Backend for one-tap sign-in in the kjbox singer page
(sing.nomadkaraoke.com). At the 2026-10-22 show, typing an email and a
6-digit code was the biggest friction point for singers making a karaoke
video on demand. Plan: kjbox
`docs/archive/2026-10-28-singer-make-it-one-screen-plan.md`.

- **`POST /api/kjbox/auth/google` and `/api/kjbox/auth/apple`** take
`{id_token, locale?, venue?}` and return the same body as
`/api/kjbox/auth/verify-code`, with no code needed.
- The token is checked by `login_code._verify_identity`, so the
audiences match the app's (and `fake:` tokens work only on the local
stack).
- Venue rules are the same as verify-code: no per-IP signup cap,
`KJBOX_SIGNUP_CAP_PER_24H` on new accounts, `signup_source="kjbox"` +
`signup_venue`, no `signup_ip`.
- Send-code's eligibility checks are applied explicitly. A disposable
domain gets 422. A blocked email or IP gets `422 sign_in_not_allowed`.
Apple relay addresses are allowed.
- Code sign-in and social sign-in now share one `_finish_sign_in`
helper.
- **`SESSION_EXPIRY_DAYS` (inactivity) 7 → 30, globally** (Andrew's
call). With weekly shows, every returning singer was signed out by the
next show. **`SESSION_ABSOLUTE_EXPIRY_DAYS` 30 → 365** (Andrew,
2026-10-28), so a weekly regular isn't signed out once a month.
- Version 0.286.0. API.md updated.

This stays inert until kjbox calls the endpoints. Before the singer page
can use them, Andrew needs to:
- add `https://sing.nomadkaraoke.com` as an Authorized JavaScript origin
on the "Nomad Karaoke Web" OAuth client;
- create an Apple web Services ID and add it to
`APPLE_SIGNIN_AUDIENCES`.

## Testing

`backend/tests/test_kjbox_partner_api.py` covers:
- a new singer via Google and via Apple;
- an Apple relay address, which is never treated as disposable;
- a code user who later signs in with Google;
- the signup cap applying to new accounts only;
- an invalid token returning 401;
- the partner secret being required;
- disposable and blocked email/IP refusals;
- 20 days idle still valid and 31 days idle expired; a new session
expires a year from sign-in; a weekly regular is still signed in two
months on.

259 auth/session/kjbox tests pass locally. Local CodeRabbit: 2 cycles.
The first found the missing eligibility checks (fixed) and the
absolute-expiry question (now 365 days). The second found nothing new.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 19:34:00 -04:00
Andrew Beveridge ☄️andClaude Sonnet 5.5 cd1fa3feaf fix: provide-url YouTube downloads failing with 'No audio source selection found' (v0.285.1) (#1224)
Pasting a YouTube URL in the audio modal (job 258ba59b) failed:
provide-url set source_name/download_url but not source_id, and the
worker fallback required source_id. Now sets source_id and the worker
accepts download_url. The retry loop could never succeed before.

Tests: worker regression test for the provide-url job shape; provide-url
test asserts source_id. Emulator test_e2e_cli_backend needs Firestore
emulator, left to CI.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* YouTube audio jobs submitted with a pasted URL can now proceed when no
video ID is available.
* YouTube video IDs are saved with provided-URL jobs when available, and
stale search selections no longer override the pasted URL.

* **Chores**
  * Updated the application version to 0.285.1.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-26 20:07:38 -04:00
Andrew Beveridge ☄️andClaude Sonnet 5.5 3230bfbe0b feat(requests-board): real-user-only requests, 1 request + 1 vote/day, delete on last vote, confident unattended audio (v0.285.0) (#1223)
## Why
Investigating two bad community picks for one user (jobs 8695c32e,
94c83a7e) found:
- a YouTube-bot request was "claimed" by a vote that was later moved,
leaving a 0-vote request with a stale owner that was still pickable
(`vote_count >= 0`);
- the daily pick's `select_best` raised on remote (dict) results and
silently fell back to result #0 (logged as a WARNING on 7 days since 10
Oct).

## What
- **Real users only:** every request must belong to a real user email
(`require_real_email`, enforced in the service); the YouTube comment bot
no longer creates requests, it replies with how to request on the board.
Legacy placeholder rows are never listed/picked/credited.
- **Limits:** 1 request + 1 vote per day (submitting spends the vote →
`409 vote_already_used`; `429 submission_limit`). The cap uses a
create-only per-day marker, so deleting a request doesn't give the
submission back (purged on account deletion).
- **Last up-vote deletes the request** after confirmation (`409
would_delete_request` + `request_id` until `confirm_delete=true`),
freeing remaining voters' votes. Replaces the claim logic.
- **Picker:** needs `vote_count >= 1` and an owner who still up-votes it
and hasn't owned it already; deleting a community job reopens its
request (or deletes it if nobody backs it).
- **Audio:** unattended pick only on an exact artist+title match
(torrent ≥3 seeders, else Spotify; ambiguity → none). Otherwise the job
parks in `AWAITING_AUDIO_SELECTION` and the owner is emailed.
- **Email:** community-pick review email says the song came from the
requests board.
- **UI:** confirm before spending the daily vote / submitting, thank-you
panel with credit-pack CTA (`/app?buy=1`), form disabled once the vote
is spent, delete-request warning. Strings in all 33 frontend locales +
en/es/de backend.

## Testing
- Emulator tests for all service rules (real Firestore transactions),
unit tests for routes, audio pick, daily pick parking, notifications;
Jest tests for the board client.
- Local CodeRabbit review run; all 7 findings fixed.
- Note: a handful of existing job-creation emulator tests and
`test_internal_api` auth tests fail identically on clean `origin/main`
in my local env; CI is the judge.

Plan: docs/archive/2026-10-26-requests-board-overhaul-plan.md. Docs:
REQUESTS-BOARD.md, API.md, LESSONS-LEARNED.md.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-26 18:42:10 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 d76a136ee6 feat: mobile Phase 6 — RevenueCat IAP webhook, Expo push, self-serve account deletion (flagged off), App Review demo login (v0.284.0) (#1220)
Phase 6 (mobile release readiness) — gen side. Spec: nomad-mobile
`docs/superpowers/specs/2026-10-25-mobile-phase6-release-readiness-design.md`
(§4 payments, §5 push, §6 deletion, §9 demo account).

## What
- **`/me` app fields:** opaque `app_user_id` (UUID, used for
RevenueCat/Sentry — no email to third parties),
`capabilities.account_deletion`, `push_prefs`.
- **In-app purchases:** `POST /api/users/webhooks/revenuecat` (shared
Authorization header). Exactly-once grants keyed by
`{store}:{transaction_id}` inside the same Firestore transaction as the
balance change; refunds (REFUND / CANCELLATION with CUSTOMER_SUPPORT or
negative price) debit the granted account, refund tombstones block late
grants; other cancellations ignored. `store_payments` for accounting +
revenue sums; referral earnings on **net** proceeds; sandbox grants but
excluded from revenue/referrals. Credit packages gain `store_product_id`
+ server-driven `web_link` (iOS US only for now).
- **Push:** `POST/DELETE /api/push/expo-token` (token moves between
users atomically), `PATCH /api/users/me/push-prefs` (ready / review /
problems, gates Expo **and** Web Push), Expo sender (batched,
DeviceNotRegistered cleanup), new FAILED / AWAITING_AUDIO_SELECTION
triggers.
- **Account deletion** (`ACCOUNT_DELETION_ENABLED`, **off in prod**
until Andrew's manual pass): `GET /api/users/me/deletion-preview`,
`DELETE /api/users/me`. Refuses admin / unpaid referral balance / Stripe
Connect / made-for-you in progress (409). Cancels active jobs, deletes
decide data (partner call, twice), RevenueCat subscriber, revokes Sign
in with Apple (when a code is given), queues kjbox
(`/api/kjbox/account-deletions` pull + ack), deletes **inputs only**
(uploads, input/, audio_edit/, style/, staged instrumentals) — **never
published outputs** — strips personal fields, anonymises
jobs/referrals/song requests/show-credit records, keeps payment records.
Every step idempotent; failures → 503 retry; sessions revoked last.
- **Web:** `/account/delete` page (Google Play web deletion path),
hidden while the flag is off.
- **App Review demo login:** fixed code for one configured email — inert
until `APP_REVIEW_DEMO_EMAIL` is set (not set in prod).
- **CI:** mounts `REVENUECAT_WEBHOOK_AUTH`, `REVENUECAT_SECRET_API_KEY`,
`APP_REVIEW_DEMO_CODE` (secrets exist; runtime SA already has accessor).

## Tests
6838 backend unit tests pass (emulator suite needs emulators;
iap/deletion emulator tests pass locally), frontend jest 1785/1785.
Per-task reviews + local CodeRabbit review (5 minor findings fixed).

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01CZf1cM9vefnNw5djARRhyA

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 17:31:01 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 eab20db79c fix(bulk): accent/script/alias-aware artist search + no-results state (#1218)
## Summary
Bulk → By album returned **no artists** for "Maximo Park" and showed a
blank list with no explanation.

**Root cause:** artist search matched decide's
`mb_artists_normalized.name_normalized`, which maps every char outside
`[a-z0-9 ]` to a space with no accent folding. "Maxïmo Park" is stored
as `max mo park`, so `maximo park` can never match. ~179k artists (6%,
Cyrillic/CJK/etc.) normalize to `""` and were unfindable.

## Changes
- `musicbrainz_service.search_artists`: one SQL `search_key()` (NFKD,
drop combining marks, fold ø/ł/đ/ß/æ/œ/þ, keep letters/digits of any
script) applied in BigQuery to the artist name, every
`musicbrainz.artist_alias`, and the query. Ranking is unchanged in
spirit (exact → prefix → substring, name before alias, then popularity).
New `alias` field when only an alias matched.
- Verified live: `maximo park`→Maxïmo Park, `bjork`→Björk,
`beyonce`→Beyoncé, `motorhead`→Motörhead, `royksopp`→Röyksopp, `hikaru
utada`→宇多田ヒカル, `mumiy troll`→Мумий Тролль, `宇多田`, `Мумий`.
- Cost: ~0.4 GB scanned/search (~$0.002), ~2 s, 10-min cache; well under
the 2 GiB byte cap.
- UI: no-results card ("No artists match "X"" + hint + **Add songs by
name** → switches to By text); alias rows show "Also known as …". 33
locales.
- `isArtistMatch` (frontend) is now Unicode-aware, matching the
backend's `_norm_artist` (follow-up from #1214: CJK/Cyrillic names never
matched).

## Testing
- Backend unit: 6681 passed. New `search_key` + `search_artists` tests.
- Frontend jest: 1776 passed. New no-results/loading/error/alias/switch
tests + isArtistMatch parity tests.
- Local CodeRabbit CLI: no findings.

## Follow-up (not in this PR)
decide `/api/catalog/artists` has the same flaw ("bjork" doesn't find
Björk); a proper fix is a folded column in decide's
`mb_artists_normalized` ETL.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_018teGmBJj3EVmCfpfMFG4WU

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 22:44:58 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 c4c4f9265c feat(bulk): album batches search the album once; one flacfetch search in flight per batch (#1214)
## Why
A real Bulk "By album" batch (Chase & Status, *No More Idols*, 12
tracks):
- It took about 5 minutes to reach audio selection.
- 4 of the 12 searches timed out.
- All 12 were parked for manual choice, each showing the same album
torrents.

Causes:
- The album was dropped at submit, so each track was searched
separately.
- 4 parallel searches ran against flacfetch's single search worker, and
the queued ones ran out the 95s budget.
- The auto-pick bar was 50 seeders (lowered to 10 in #1211).
- A title-only file search misses the album for common titles ("Time").

## What
- **Submit carries the album:** `POST /api/bulk/submit` takes `album`
{artist, title, release_group_mbid, release_mbid} plus each song's
`position`. These are stamped as `state_data.bulk_album` /
`bulk_track_position`. The frontend album mode sends them, using the
release-group title.
- **One album search per album:** the worker makes one flacfetch `POST
/search/album` call (nomadkaraoke/flacfetch#56).
  - Each track's file is resolved inside the album's torrents.
- The whole album comes from **one torrent**: the first BEST CHOICE
(lossless, non-vinyl, 10+ seeders).
- A track is auto-selected when that torrent's file names it and the
artist matches.
- A track the batch torrent doesn't cover gets the normal per-track pick
among the album torrents that do.
  - Parked jobs show the album torrents in the selection modal.
- **Fallbacks are one search at a time** (`SEARCH_CONCURRENCY = 1`):
uncovered tracks, text batches, and failed or unavailable album searches
(404 from an older flacfetch). A batch search deadline (3000s) parks any
leftovers before the 3600s job timeout could strand them.
- **Auto-pick checks the artist** on the backend too: a Unicode-aware
port of `isArtistMatch`, used by bulk and KaraokeHunt. A 10+-seeder
cover or tribute torrent is no longer auto-picked.
- Fake album search for the local stack; docs in API.md and
LESSONS-LEARNED.

The money path is unchanged. Album results have no duration, so nothing
is charged at select time. The real duration charge or refund still
happens after download (`reconcile_and_maybe_pause`).

## Testing
- Backend: 6677 passed. The only failures are the 8
`test_emulator_integration` tests, which hit a stale local Firestore
emulator from another session; they passed in CI on #1211.
- Frontend: Jest 1765/1765.
- New tests cover:
- album worker path: one search, all auto-selected, uncovered →
fallback, search failure → fallback, low seeders → park, auto disabled
  - `process_batch` routing, with at most one search in flight
  - the deadline
- album helpers, artist match (incl. CJK and &), client `search_album`
(incl. 404 and 502), submit stamping, the fake album response, and the
frontend submit payload
- Code review: CodeRabbit CLI unavailable (invalid token / org
mismatch), so a local Superpowers review ran instead. Its fixes are in
the second commit.

## Rollout
flacfetch v0.34.0 (#56) deploys to flacup first. Then: one album search,
12 auto-selected jobs, one torrent; this is verified in prod after
merge.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_019MixBuvzxB1c5AYc3mbDyW

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Album-mode bulk submissions now include album details and track
positions, enabling one album-wide search for matching tracks.
* Search results are checked against both track title and artist before
automatic selection, with tolerance for minor name variations.
* **Improvements**
* Tracks not covered by an album result, or affected by an unavailable
or failed album search, fall back to individual searches.
* Searches still awaiting selection after the batch deadline are parked
for manual review.
* **Documentation**
* Updated bulk submission guidance to describe album searches and
fallback behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 17:06:57 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 25888ca003 fix: Stripe made-for-you emails + bulk_submit job creation off the event loop (v0.282.2) (#1212)
## Summary
Two event-loop stalls from the 2026-10-22 #alerts triage. Each froze
every request on the instance.

- **22s stall, 11 requests (15:57 UTC).** `stripe_webhook` (async) →
`_handle_made_for_you_order` sent the customer order-confirmation email
through sync Postmark, with retries and SMTP fallback, on the loop. It
was the NOMAD-1800 order. The confirmation, the admin notification and
`_prepare_theme_for_job` (GCS) now run via `asyncio.to_thread`.
- **26s stall, 9 requests (15:30 UTC).** `bulk_submit` (async) ran the
credit check, default-theme lookup and one sync `create_job` +
`update_job` per song on the loop (an album batch). That block is now a
sync helper run via `asyncio.to_thread`. Behaviour is unchanged: the
same `HTTPException`s propagate from the thread.

## Guard (`test_no_blocking_async_routes.py`)
- `BLOCKING_CALLS` now includes `send_made_for_you_order_confirmation`
and `send_made_for_you_admin_notification`.
- New `test_no_known_blocking_calls_in_awaited_route_helpers` also scans
`_handle_made_for_you_order`. It is an awaited helper, not a decorated
route, so the route-only scan missed it, including its
`_prepare_theme_for_job`, which was already on the denylist.
- New `test_bulk_submit_creates_jobs_off_the_loop`.
- Both new tests fail on `origin/main` (checked against the old files)
and pass here.

Out of scope: sync `create_job`/`update_job` still runs directly in
about 20 other async routes (audio_search, file_upload, jobs.retry_job,
…). Each call is usually fast. It's worth a broader sweep if the
watchdog keeps catching them.

## Testing
- `make test` backend: 6607 passed, 8 failed. All 8 are in
`test_emulator_integration.py`, because another session's Firestore
emulator holds :8080 locally (same as noted on #1210). CI's emulator job
is authoritative.
- Targeted: bulk submit, made-for-you, Stripe webhook and the blocking
guard: 114 passed.
- The local CodeRabbit CLI failed with "Invalid token", so the bot
review stays on.

Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562858789785309195
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562858791446388862

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_016F41fLoBotUw2wUjGCuAok

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Improvements**
* Bulk song submissions and made-for-you order processing are less
likely to delay other activity while work is underway.
* Existing credit checks and handling of jobs created before credits run
out remain unchanged.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 19:55:36 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 f6d846234d fix: alert noise + loop stalls + kn-data-sync OOM from #alerts triage (v0.282.1) (#1210)
## Summary
Fixes for the open and noise groups from the 2026-10-22 #alerts triage.

**Noise that should not page:**
- `stripe_service.py`: "Invalid webhook signature" now logs WARNING. All
42 hits on 10-21 came from `facebookexternalhit` via Cloudflare with a
malformed header. Every real Stripe delivery since 10-14 got a 200.
(#alerts 1562375740685815953)
- `error_monitor/known_issues.py`: `startup_probe` also matches Cloud
Run's "Default STARTUP **TCP** probe failed" wording.
(1562224513276706933)
- `frontend/lib/crash-reporter.ts`: a `removeChild`/`insertBefore`
NotFoundError is benign **only while the page is machine-translated**,
i.e. `<html>` has `translated-ltr`/`translated-rtl`. In the 10-21 crash,
Chrome had translated `/en/app` into Spanish ("Crea un video karaoke" is
not our string), and the customer's job completed and was downloaded.
(1562541569238638725)

**Event-loop stalls** (continues the WIP freeze item in BACKLOG):
- `review.py` audio-editor routes: `apply_edit`, `get_metadata_from_gcs`
and the GCS `download_file`/`upload_file` calls now go through
`asyncio.to_thread`. They were stalling the loop about 10s per edit (job
330d796c, 10-19). (1561861805695967363)
- `internal.py`: `process_conversion_emails` is imported at module top.
The lazy import inside the hourly cron's `async def` wrote a `.pyc` on
the loop, stalling it up to 15.3s, 4× on 10-20/21. (1562224514933334047)
- `backend/Dockerfile`: `python -m compileall` over `/app/backend` and
`/app/karaoke_gen`, so no lazy import compiles at runtime.
- `test_no_blocking_async_routes.py`: `BLOCKING_CALLS` now includes
`apply_edit`, `get_metadata_from_gcs`, `upload_file` and
`download_file`. Two more routes in `review.py` were caught and wrapped:
the preview ASS upload and the join-audio upload.

**kn-data-sync** (`infrastructure/modules/kn_data_sync.py`):
- Memory goes from 512M to 1Gi. The 10-22 full sync OOMed at 489 MiB on
a retry. 1Gi also gives gen2 more CPU, and the cost is about $0 within
the free tier.
- `attempt_deadline="320s"` on both schedulers. The default 180s gave up
on a run that took 184.8s and succeeded, then retried it, which meant 3
full KN pulls that day. (1562752965629190217, 1562752969395535933)

## Testing
- `make test`: 6603 passed. 8 failed in `test_emulator_integration.py`,
the known local-only failure (another session's stack holds :8080); they
fail the same way on unmodified main, and CI's emulator job is
authoritative.
- Targeted runs: error monitor, audio-edit routes, Stripe/webhook,
conversion emails and the blocking guard all pass (710 tests). Frontend
crash-reporter jest: 17 passed, with new translated / not-translated
cases.
- The local CodeRabbit CLI failed with "Invalid token", so the bot
review is left on for this PR.

## Deploy note
The `infrastructure/` change (kn-data-sync) needs `pulumi up` before
merge, per the repo rule. That needs Andrew's `/gcp-admin` grant.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_015AvGQEptp3Pg9RHhKGZ5KB

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved responsiveness during audio editing and review submissions,
reducing the chance that file and audio processing delays other
requests.
* Reduced false crash reports on pages affected by machine translation.
  * Improved recognition of startup probe failures in error monitoring.
  * Clarified logging for malformed payment webhook signatures.
* **Reliability**
* Increased resources and allowed more time for scheduled data sync
jobs.
  * Reduced delays to API requests during background processing.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

## Alerts
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562375740685815953
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562224513276706933
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562541569238638725
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1561861805695967363
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562224514933334047
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562752965629190217
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562752969395535933

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 19:26:43 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 e4dad6e3c5 fix(delivery): hold completion email when Dropbox delivery failed (NOMAD-1800) (#1213)
## Summary

Job `dd892383` (NOMAD-1800) finished while the business Dropbox was at
its 6 TiB quota. Both Dropbox upload passes failed with
`insufficient_space`, but the job still went COMPLETE and emailed the
customer a delivery email whose link was the literal `[Dropbox URL not
available]`. The only #alerts signal was a raw traceback.

Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562874010939949110

## Changes

- **New `backend/services/delivery_guard.py`**
- `completion_email_hold_reason(job)`: if the job's *effective*
distribution settings deliver to Dropbox (`dropbox_path` and
`brand_prefix`, the same condition the workers use, so
private/NonPublished jobs count too) and `state_data.dropbox_link` is
missing, the email is **held**.
- `hold_completion_email(...)`: records
`state_data.completion_email_held = {reason_code, reason, recipient,
at}`, logs a WARNING, and sends **one actionable #alerts message per
job** (`send_throttled_ops_alert`, keyed per job). The message covers
the job, brand code, the masked recipient whose email was held, the
reason (`insufficient_space` is classified as "Dropbox is FULL"), the
fix steps and the runbook. The Pushbullet push is unchanged.
- `check_dropbox_space_usage()`: calls `users_get_space_usage()`, using
team allocation for the business account. It posts one #alerts message
when usage reaches 95% and records its state in Firestore
(`ops_failure_alerts/dropbox-space-usage`), so it stays quiet while
usage stays high and re-arms once usage drops below 95%. It runs on the
existing hourly `process-stale-reviews` cron, so no infra change is
needed.
- **`JobManager._schedule_completion_email`**: checks the hold before
sending. Made-for-you ownership still transfers to the customer.
- **Admin `POST /api/admin/jobs/{id}/send-completion-email`**: returns
409 when the link is missing, unless `allow_missing_dropbox_link: true`
is passed. A successful send clears `completion_email_held`.
- **`TemplateService.render_job_completion`**: no more `[Dropbox URL not
available]` / `[YouTube URL not available]` placeholders. When a link is
missing, that whole section is dropped (English default and GCS wording,
plus the localized template). Jobs with no Dropbox delivery at all
(tenant, no Dropbox configured) still get their email, just without a
Dropbox paragraph. The YouTube-queued and private paths are unchanged.
- Dropbox upload failure logs (orchestrator and native pass) go from
ERROR to WARNING, so one root cause raises one alert (the held-delivery
alert) instead of an error-monitor traceback.
- Docs: `TROUBLESHOOTING.md` § "Held completion email (Dropbox delivery
failed)" with the manual recovery steps, and `API.md` for the new 409
and override.
- Version 0.282.0.

Not done: item 5 of the handoff (an admin "redistribute Dropbox only"
action). The existing admin re-render also re-uploads YouTube, so it
isn't a clean fix for public jobs. The manual recovery steps are in
TROUBLESHOOTING.md instead.

## Testing

- New `backend/tests/test_delivery_guard.py` (31 tests): failure
classification, hold/no-hold matrix (no link, link present, no Dropbox,
tenant, private defaults, settings error), held marker and alert
content, quota threshold, alert-once and re-arm, failed send does not
latch, API error never raises.
- `test_made_for_you.py`: email held when the link is missing (ownership
still transfers, marker recorded, alert sent); email sent when the link
is present; email sent for a job with no Dropbox delivery.
- `test_admin_email_endpoints.py`: 409 without the link; override sends
and clears the marker; fixed link sends and clears the marker; job with
no Dropbox delivery sends normally.
- `test_template_service.py`: missing links never render placeholders
(en, de, unknown GCS template).
- Backend unit suite (`backend/tests`, excluding emulator tests): 6637
passed, 7 failed, 25 skipped. All 7 failures are
`test_internal_api.py::*requires_auth` / `*requires_admin`, which fail
the same way on unmodified `origin/main` locally (they get 200 instead
of 401, apparently from the local environment), and this PR doesn't
touch them.
- The local CodeRabbit CLI failed ("Invalid token"), so this PR is left
open for the bot review.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_016F41fLoBotUw2wUjGCuAok

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 17:53:08 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 6c53f0471a feat(youtube): auto-reply to song requests in YouTube comments with requests-board links (v0.280.0) (#1205)
## Summary

Backlog Frog #6 (YouTube channel → gen funnel). Andrew's spec: *"we
could probably also solve that by building some automation to detect
youtube comments asking for songs and auto-reply with a link to the
requests board"*.

An hourly bot reads new top-level comments on the Nomad Karaoke channel
and, when someone asks for a song:
- **Song already on the channel:** replies with that video's link.
- **Otherwise:** adds the song to the requests board (or finds the
existing request) and replies with a link straight to it, e.g. `Added
"Tiny Hitman" by Bear Ghost to our requests board 🥳 upvote it here and
we'll email you when it's made: <link>`.
- **Vague asks** ("more Johnnie please") get a generic board link.
- **Non-English comments** get a translated reply, which falls back to
English if a URL doesn't survive translation.

The reply promotes only the board, not gen (Andrew's call). The goal is
turning anonymous YouTubers into board users with an email address.

**Andrew signed this off on a 6-month local replay**
(`scripts/youtube-comments/replay.py`, read-only):
- 265 threads, 33 song requests.
- It flagged all 29 requests Andrew had answered by hand, and all 7
vague asks.
- It posted no replies to non-requests.
- His calls on borderline cases became prompt rules: yes to Playa Fly,
no to "Joe Cuervo Karaoke 🎤" on an unrelated video.

## Changes

- **Decision core:** `backend/services/youtube_comment_bot.py`, a pure
module with injectable ports (Gemini classifier/translator, board,
channel index), shared by the replay and the worker.
- **Worker:** `backend/workers/youtube_comment_bot_worker.py`, called
through `POST /api/internal/youtube-comments/run` (`?dry_run=true`).
- Creates a create-only action doc per comment **before** posting, so
nothing gets a double reply.
- Keeps a high-water mark with a 6h overlap. A Gemini error causes a
retry, not a permanent skip.
  - Per-run cap and quota coordination.
  - Posts to #noise per reply; failures go to #alerts, throttled.
- **Kill switch:** `YOUTUBE_COMMENT_AUTOREPLY_ENABLED` defaults to
**false**, and this PR leaves it off. A follow-up PR switches it on
after a prod dry run. `YOUTUBE_COMMENT_AUTOREPLY_START` enables a
one-time backfill of unanswered requests.
- **Requests board:**
- New `source="youtube_comment"`; bot-added requests are owned by the
placeholder `youtube-comment@nomadkaraoke.com` with no votes.
- The **first real up-voter claims the request**, inside the vote
transaction. The daily picker skips unclaimed requests, so a free credit
never goes to the placeholder.
- Frontend `?request=<id>` deep link: highlight, scroll, banner (33
locales).
- Signing in from the link uses purpose `requests_board:<id>`, so the
magic link returns to the request with `&upvote=1` and the upvote is
cast once.
- **OAuth:** the bot uses its **own** Desktop OAuth client and secret
`youtube-comment-bot-credentials` (`youtube` + `youtube.force-ssl`),
because Google refuses `force-ssl` in the device flow. The upload token
is untouched.
- **Infra (Pulumi, already applied locally):** secret container + Cloud
Scheduler `youtube-comment-autoreply-hourly`.
- **Description template:** a make-your-own line moves above YouTube's
"...more" fold (`utm_campaign=make_cta_top`). The daily backfill
re-drains the channel over about 11 days.
- **Docs:** `docs/REQUESTS-BOARD.md` § YouTube comment auto-reply,
API.md, README.md, plus the plan in
`docs/archive/2026-10-22-youtube-comment-autoreply-plan.md`.

## Testing

- `backend/tests/test_youtube_comment_bot.py` (33): skips, routing,
vague asks, translation fallback, malformed classifier output, API
helpers.
- `backend/tests/test_youtube_comment_bot_worker.py` (12): kill switch,
dry run, idempotency (second run and concurrent claim), go-live start,
cap, quota, missing scope, post failure, classifier-error retry.
- `backend/tests/emulator/test_song_request_service.py`: bot add/dedupe,
claim on first upvote, downvote doesn't claim, picker skips unclaimed,
human resubmit claims, `find_published_by_key`.
- `backend/tests/test_admin_login_link.py`: `requests_board:<id>`
redirect and purpose helper.
- `frontend/e2e/regression/requests-board-deep-link.spec.ts` (3):
highlight + sign-in purpose, upvote-once on return + param dropped, no
toggle-off when already upvoted.
- Full `make test`: backend 6584 passed; the only failures are the
local-environment `test_emulator_integration` cases, which fail
identically on `origin/main` (port 8080 is held by another session's
emulator). Frontend Jest: 1760 passed.
- Behavioural acceptance: the 6-month replay above.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_016E3ibN2EcKxnGQvEf6T4V8

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 01:51:15 -04:00
Andrew Beveridge ☄️andClaude Sonnet 5.5 83e097fa3b feat(local-stack): fake review fixtures and knobs for mobile native review (v0.279.5) (#1202)
## Summary
Local-stack (FAKE_WORKERS=1) support for nomad-mobile Phase 2, which is
the native lyrics and instrumental review. Maestro can now drive every
review state against the real review endpoints. Nothing changes in prod:
all hooks are gated on `fake_workers_enabled()`.

## Changes
- **New per-user `FakeConfig` knobs:**
- `review_fixture` (`default` / `corrections` / `duet` / `rtl` /
`large`). The fixtures are built from the real `CorrectionResult` types:
AI spans with `undo_info`, an untimed word, an evidenced vocal gap with
synced reference lines, duet singers, Hebrew/Arabic lines, and 300
lines.
  - `backing_vocals` (`silent` / `audible`)
  - `stems_delay_seconds` (stems land after review opens)
- `review_mode` and `auto_approval` (writes
`processing_metadata.auto_approval`, so the confident and skipped-step
states can be reached)
  - `review_complete_fail_once`
- **Fake media gets an amplitude envelope**, so waveforms aren't flat.
There is also an audible backing stem variant.
- **`/api/review/{id}/audio` dev-proxy branch** now uses
`_ranged_response` (Range → 206), so seeking works on iOS. The prod
signed-URL path is untouched.
- **`GET /api/test/jobs/{id}/review-result`** returns the submitted
review (selection, `is_duet`, corrected lines) for Maestro assertions.
- `docs/TESTING.md` § Local stack documents the knobs.

## Testing
- `make test`: backend 2184 + 6532 + 163 passed, frontend 1760 passed.
- 29 new tests in `backend/tests/fake_workers/test_review_fixtures.py`.
- Smoke-tested on the local stack: every fixture loads; `with_backing`
analysis; Range 206; 503 then 200 on complete; `review-result`;
`auto_approval` states; late stems.

Reviewed locally (CodeRabbit CLI, 1 finding fixed).

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01G81fc5sBB8GwG5Xhz2UMgc

---------

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-20 00:20:38 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 b67b273056 feat(kjbox): decide partner secret for catalog resolve + fake judge + error monitor for decide search (v0.278.10) (#1190)
## ⚠️ Merge prerequisite (do BEFORE merging)

The Secret Manager secret `decide-gen-partner-secret` must have a
version before this merges, otherwise the CI deploy's `--set-secrets
...DECIDE_GEN_PARTNER_SECRET=decide-gen-partner-secret:latest` fails.

```
gcloud auth login admin@nomadkaraoke.com
cd ~/Projects/nomadkaraoke/karaoke-gen-decide-partner-resolve/infrastructure
GOOGLE_OAUTH_ACCESS_TOKEN=$(gcloud auth print-access-token) pulumi up     # review: should only add the decide-gen-partner-secret Secret
openssl rand -hex 32 | tr -d '\n' | gcloud secrets versions add decide-gen-partner-secret --project=nomadkaraoke --data-file=-
```

No extra IAM grant needed: gen's API runs as `karaoke-backend@`, which
already has project-level `secretmanager.secretAccessor`.

## Summary
- `POST /api/kjbox/catalog/resolve` now also accepts karaoke-decide via
header `X-Partner-Secret` (new setting `decide_partner_secret` / env
`DECIDE_GEN_PARTNER_SECRET`). Other kjbox endpoints stay
`X-Kjbox-Secret` only. decide gets its own AI-call budget (60/min) so
kjbox nights are never starved; the verdict cache is shared.
- Fake match judge / resolve for the local E2E stack (`FAKE_WORKERS`):
`judge_kind` / `judge_canonical_*` knobs; no Gemini or catalog calls.
- Error monitor now watches decide's new `karaoke-decide-search` service
and `song-index-build` job.
- Pulumi: declares the `decide-gen-partner-secret` Secret (value added
manually); CI deploy mounts it.
- Version 0.278.10.

## Testing
- [x] Changed-module + kjbox/catalog/error-monitor/fake-worker backend
tests pass (133 passed). Full `backend/tests` suite exceeds 10 min
locally, so left to CI.

## Review
- CodeRabbit CLI failed twice (WebSocket closed / connection error), so
no local CodeRabbit review completed; I did a manual diff review instead
(no issues found). Deliberately NO `@coderabbitai ignore`, so the GitHub
bot reviews this PR.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01VsLw9cTTMBab1tiRxCsYrm

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 21:42:27 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 92fd647203 fix: preview with untimed in-browser edits fills timings or 422s, not a KeyError 500 (v0.278.15) (#1195)
@coderabbitai ignore

## Why
BACKLOG "Lyrics preview 500 `KeyError 'end_time'` on unsaved edits"
(#alerts msg 1559340473439617117). On 10-12 a paying user's review
preview failed 3× (job 16792df7). Both the GCE and the local fallback
failed with `'end_time'`. `JSON.stringify` drops `undefined`, so a word
or line edited in the browser can arrive without its timing keys, and
`CorrectionOperations.update_correction_result_with_data` indexes
`w["end_time"]` / `s["end_time"]`.

## What
- `timing_sanitizer.sanitize_corrections`: a segment with missing or
non-finite bounds now gets them from its words' timings, or else from
the neighbouring lines' gap. It used to be skipped. The existing word
clamp then fills untimed words inside that window.
- New `find_missing_timings()` returns the text of a line that still has
no usable timing.
- `POST /api/review/{id}/preview-video` now sanitizes the payload up
front, as review submit already did. If a line can't be recovered, it
returns **422** `The line "…" has a word with no timing. Sync that line,
then try the preview again.` (en/es/de) instead of a 500.
- The frontend payload is unchanged. The backend now treats a missing
key and a null value the same way, so a frontend change would add
nothing.

## Testing
- `test_timing_sanitizer.py`: bounds from words, missing `end_time`
**key** (the 16792df7 shape), untimed line filled from its neighbours'
gap, unrecoverable line reported. The old "non-finite segment bounds
left untouched" test is replaced, because that is the behaviour this PR
changes.
- `test_routes_review.py::TestPreviewVideoMissingTimings`: a 422 naming
the line, with no render attempted; fillable gaps reach the renderer
filled.
- 131 passed across review routes, sanitizer, and submit-corrections
tests. Local CodeRabbit: 0 findings.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Jw21syo4HizJJjYw2mg2Vp

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 14:57:13 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 3bbf397dc4 fix: keep auth, tenant lookup and job-log writes off the API event loop (v0.278.14) (#1194)
@coderabbitai ignore

## Why
The residual event-loop freezes BACKLOG item. Seven days of
`EVENT_LOOP_STALL in progress` stacks (2026-10-12→19, 355 samples),
grouped by their outermost app frame, ranked the app-level loop
blockers:

| Source | Samples | Max |
|---|---|---|
| in-process screens worker → job-log handler → sync Firestore write per
log line | 30 | 2.2s |
| `require_auth` → `validate_token_full` (session read on **every**
authed request) | 19 | 1.9s |
| preview encodes
(`_run_preview_encode`/`_encode_preview_in_background`) → job-log write
| 15 | 2.4s |
| `change_visibility` → sync Dropbox/GCS deletes | 6 | 1.2s |
| tenant middleware → `get_tenant_config` (GCS on cache miss) | 5 | 1.2s
|
| `require_review_auth` → `get_job` | 5 | 2.2s |
| `auto_correct_worker._run_suggest` → `get_job` (the only stall since
#1192) | 1 | 1.2s |

(The render/video in-process stalls stopped with #1192.)

## What
- **`backend/services/log_write_offload.py`**: when a job-log write is
made on a running event loop in the API process, it goes to one
background writer thread. The queue is bounded at 5000 and a drop is
logged at WARNING, rate-limited; `atexit` drains the queue. Writes stay
inline in threads and Cloud Run Jobs, so the worker code paths behave
exactly as before. Logs are read back ordered by their own `timestamp`,
so the hand-off doesn't reorder them.
- **`asyncio.to_thread`** for
`require_auth`/`require_auth_legacy`/`require_review_auth`, the tenant
middleware, `rerender_with_current_theme`'s tenant lookup,
`auto_correct._run_suggest`, and `change_visibility`'s `get_job`.
- **Visibility change:** `_delete_*_outputs` are now plain `def`, and
the sync steps run in one worker thread. Moving them off the loop
removed the implicit serialisation, so both directions now **claim
`visibility_change_in_progress` atomically** before touching anything.
They use the new `FirestoreService.try_set_job_state_flag`, a
transactional test-and-set. A lost claim returns 409. The
regenerate→private continuation passes `holds_claim=True`, because its
regenerate already set the flag. The claim is released if the reset
fails, and a raising screens trigger rolls back like a `False` one.
- **Guard (`test_no_blocking_async_routes.py`):** `BLOCKING_CALLS` gains
`validate_token(_full)`, `validate_session`, `get_tenant_config`,
`append_log_to_subcollection` and `append_worker_log`. Every `async def`
in `api/dependencies.py` and `middleware/` is now checked, not just
decorated routes. Mutation-checked: reverting `require_auth` fails the
guard.
- `docs/LESSONS-LEARNED.md` entry. About 70 direct
`get_job`/`update_job` calls in async routes remain. They're single
round-trips that rarely show in the watchdog, and the doc says to move
them when one does.

## Testing
- New: `test_log_write_offload.py` (inline vs offloaded, Cloud Run Job
inline, ordering, full-queue drop, Firestore append off the loop
thread), claim tests in `test_change_visibility.py` (lost claim, release
on failure, chained continuation, trigger exception, route 409), and
emulator `test_try_set_job_state_flag.py` (once-only, already set,
missing job, one winner among 4 concurrent callers).
- Backend: 6478 passed. The 8 failures are
`test_emulator_integration.py`, caused by my local environment: another
session's emulator setup has an `ssh` forward on :4443 instead of
fake-GCS, and they fail identically on untouched main. CI's emulator job
covers them and the new emulator test.
- Local CodeRabbit: 3 cycles. I fixed the claim race (two findings),
trigger-exception rollback, and the guarded claim release. I skipped two
minor nitpicks on the pre-existing trigger-failure rollback.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Jw21syo4HizJJjYw2mg2Vp

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 14:41:34 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 0d0605d7b5 fix: final encode waits for stem separation instead of stranding the job; pin audio-separator 0.44.2 (v0.278.11) (#1191)
## Problem
The daily E2E Stage 2 failed on 2026-10-17 (run #216) and 2026-10-18
(#217). When a customer finishes review **before** stem separation does,
`video-encoding-job` fails its prerequisites (`Missing instrumental:
instrumental_clean`) 3× within seconds. The job then sits at
`instrumental_selected` ("Starting final encode") forever: it's never
marked failed, nothing alerts, and separation finishing doesn't
re-trigger the encode.

The race became common when the GPU image's unpinned `pip install
"audio-separator[gpu]"` picked up **0.47.0** on the 10-17 rebuild. On
L4, the mel-band RoFormer **karaoke** models run 3–5× slower than on
0.44.2, so total separation went from ~5 to ~10–16 min:

| model (~3-min track) | 0.44.2 | 0.47.0 |
|---|---|---|
| karaoke_aufr33_viperx | ~25s | 75s |
| karaoke_gabox_v2 | ~26s | 149s |
| karaoke_becruily | ~27s | 157s |
| instrumental fv7z / resurrection | ~50–60s | ~62s |

The model YAMLs still say `num_overlap: 8, batch_size: 1`, so this isn't
the #299 YAML-defaults change; the cause is somewhere in 0.45–0.47's
RoFormer inference changes. That root cause is tracked separately.

## Fix
- **Pin** `audio-separator[gpu]==0.44.2` in
`backend/Dockerfile.gpu-base` (`ARG AUDIO_SEPARATOR_VERSION`). Changing
this file triggers the GPU base image rebuild in CI.
- **Video worker parks instead of failing.** When the selected stem is
missing *and* separation is still running (`audio_complete` false, valid
status), it sets `state_data.video_waiting_for_stems`, puts a
user-visible message ("Waiting for audio separation to finish; the final
video starts automatically") and exits 0, so Cloud Run doesn't retry. It
re-checks after parking, and an atomic Firestore test-and-clear
(`claim_job_state_flag`) means exactly one of the video worker and the
audio worker proceeds if the stems land mid-park.
- **Audio worker resumes it.** After `mark_audio_complete`,
`resume_video_if_waiting_for_stems` re-triggers a parked encode. The
fake-workers pipeline does the same, so the local stack matches prod.
- **Backstop in `recover_stuck_jobs`.** `instrumental_selected` for more
than 10 min with the stem present → re-trigger the video worker (covers
a lost resume or lost dispatch). More than 60 min with separation never
finishing → `mark_job_failed` (`stage=video_waiting_for_stems`), so the
job fails loudly. Jobs older than 24h are never revived. (No jobs are in
`instrumental_selected` in prod right now.)

## Testing
- New `backend/tests/test_video_waits_for_stems.py` (16 tests): pending
detection (including that a wrong status never parks), park / continue /
claim-race outcomes, and resume (flag claimed / no flag / job moved on).
- `test_recover_stuck_jobs_repark.py`: 5 new sweep tests (re-trigger,
fresh, still separating, never-ready → fail, >24h ignored).
- Full `backend/tests` (excluding emulator): 6455 passed.
- Local CodeRabbit review: 0 findings.

## Verify after deploy
- The GPU job logs `Separator version 0.44.2`, and karaoke-model
separation is back to ~20–30s.
- The next daily E2E Stage 2 passes.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_019pmG2XAgtUguEJWLFpkKHN

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 02:21:52 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 0e507852af fix(alerts): demote expected conditions off ERROR + Payment Link webhook fix (v0.278.9) (#1187)
## Summary

Alert hygiene, part of keeping Discord #alerts to actionable signal
only. ERROR is page-worthy: the error monitor turns every new ERROR
pattern into an #alerts message. This PR moves expected conditions down
to INFO/WARNING and fixes the one real bug among them, which was a
Payment Link checkout crashing the Stripe webhook. It covers the BACKLOG
items "Payment Link checkouts crash the Stripe webhook" and part of
"Alert-hygiene bundle".

## Changes

1. **Stripe webhook, non-gen checkouts** (`users.py`,
`stripe_service.py`, `stripe_admin_service.py`)
- A session that gen didn't create (a Stripe Payment Link:
`payment_link` is set, there's no gen metadata, and
`customer_email=None`) used to log ERROR "No user email in checkout
session". `store_payment` then crashed on `None.lower()`, so the payment
never reached `stripe_payments` or revenue.
- New `StripeService.is_gen_checkout()`. A non-gen checkout now logs
INFO "non-gen checkout (payment link ...)" and grants no credits. It is
marked processed for idempotency and is still stored, with
`customer_details.email`, `order_type="payment_link"`, `payment_link_id`
and the line-item description.
- `store_payment` now handles None for metadata, customer_details,
total_details and payment-method fields.
- **Not backfilled:** session `cs_live_a1ENsua3w6mDH04PPc…` (2026-10-13,
a $36 tip, no customer action needed). It could be backfilled through an
admin path that calls `store_payment` on the expanded session. GCP
writes from agents are restricted, so this PR doesn't do it.
2. **Lyrics providers** (`base_lyrics_provider._log_request_exception`):
a 404 means "no lyrics for this track" and now logs INFO. 429 moves to
WARNING alongside 5xx, connection errors and timeouts. Auth errors and
other 4xx stay ERROR. After 5 404s from one provider in a process, it
logs a single WARNING saying the provider may be dead (TODO points at
the Musixmatch backlog item).
3. **research_audio** (`audio_search.py`): no more
`AWAITING_AUDIO_SELECTION -> SEARCHING_AUDIO ->
AWAITING_AUDIO_SELECTION` transitions, which were invalid and logged 2
ERRORs per re-search. The job stays in AWAITING_AUDIO_SELECTION with no
status writes, so a crashed request can't strand it. After the search
the job is re-read. If a selection landed in the meantime, the fresh
results are discarded with a 409 instead of replacing the results that
selection indexed into.
4. **audio-download-job on a deleted job** (`audio_download_worker.py`):
if `get_job()` returns None at the start or after the download, or a
failure happens and a fresh read confirms the doc is gone, the worker
logs INFO and returns success, so it exits 0. That means no `fail_job`,
no further writes, no worker triggers and no Cloud Run retry. A GCS
NotFound on a job that still exists stays an ERROR. Seen with job
6ffe88ad on 2026-10-14.
5. **Loop watchdog**: "Failed to record loop stall (non-fatal)" now logs
WARNING with exc_info. Only that line changed, to keep the conflict with
feat/alert-noise-monitor trivial.
6. **Job-list queries** (`firestore_service.list_jobs`,
`list_jobs_summary`, `list_jobs_by_batch_id`): `query.stream()` now gets
`retry=` and `timeout=` of 30s. The client default is 300s for both,
which caused the "Error listing jobs summary: Timeout of 300.0s
exceeded" errors.

Version 0.278.9 (0.278.8 is taken by the still-open #1186).

## Testing

- New/updated unit tests:
- `test_stripe_webhook_non_gen_checkout.py`: webhook stores the payment,
logs INFO, logs nothing at ERROR, and marks the session processed.
- `test_stripe_admin_service.py`: Payment Link session and null fields.
- `tests/unit/test_base_lyrics_provider.py`: log levels for 404, 429 and
401, plus the dead-provider WARNING.
- `test_audio_search_research_attach.py`: no status writes, and a
selection made mid-search gets a 409.
- `test_audio_download_worker.py`: job deleted at start, mid-download,
or with NotFound on write.
  - `test_loop_watchdog.py`: recorder failure logs WARNING.
- `test_firestore_list_query_timeout.py`: bounded retry/timeout on all
three list queries.
- `pytest backend/tests --ignore=backend/tests/emulator`: 6434 passed.
`pytest tests/unit`: 2108 passed.
- Local CodeRabbit review: cycle 1 raised 2 findings (non-gen session
idempotency, re-search race), both fixed. Cycle 2 was clean.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_019pmG2XAgtUguEJWLFpkKHN

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 02:58:58 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 290f927175 fix(backend): move remaining event-loop blockers off the loop (v0.278.7) (#1185)
## Summary
Found while triaging Discord #alerts and 14 days of freeze telemetry
(2026-10-18). The event-loop watchdog still caught real freezes that hit
users:

- **151s freeze, 32 requests affected (2026-10-17 20:10 UTC).** `POST
/api/review/{id}/custom-lyrics/generate` called
`CustomLyricsService.generate` (a sync Gemini call) on the loop. Every
user on the instance saw "server unavailable". It now runs via
`asyncio.to_thread`.
- **14.9s freeze, 7 requests (2026-10-16).** The first `from
google.cloud import run_v2` in
`worker_service._trigger_worker_cloud_run_job` ran on the loop.
`google.cloud.run_v2` and `httpcore` (imported lazily by httpx) are now
in the startup `_LAZY_ROUTE_IMPORTS`, so they are imported off-loop
before readiness.
- **GCS on the loop in async routes.** `_prepare_theme_for_job` ran on
the loop in `upload_and_create_job`, `create_job_with_upload_urls`,
`create_job_from_url` and audio-search `search_audio`;
`submit_audio_edit` also uploaded `edit_log.json` on the loop. These now
go through `to_thread`.
- **Guard.** `upload_json`, `_prepare_theme_for_job` and `generate` are
added to the `BLOCKING_CALLS` denylist in
`test_no_blocking_async_routes.py`. The denylist caught the two extra
routes above.

## Testing
- `pytest backend/tests --ignore=backend/tests/emulator`: 6418 passed,
36 skipped.
- `test_audio_search` source check updated to accept the
`to_thread(_prepare_theme_for_job, …)` form.
- Local CodeRabbit review: 0 findings.

## Verify after deploy
No new `EVENT_LOOP_STALL ended … requests_affected>0` lines with
culprits `_call_gemini`, `run_v2` or `upload_json`.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_019pmG2XAgtUguEJWLFpkKHN

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 01:56:13 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 9a44a30368 fix(review): idempotent correction-data review transition (v0.278.4) (#1182)
## Summary
The same race as #1181, on the other door into review. `GET
/api/review/{job_id}/correction-data` moves `AWAITING_REVIEW` to
`IN_REVIEW`. When two loads arrive concurrently, both see
`AWAITING_REVIEW`, so the second transition raises `in_review ->
in_review`. That returns a 500 and the review page shows "Failed to load
lyrics". The iOS in-app browser hits this every time; mobile E2E caught
it.

**Fix:** a shared `ensure_in_review()` helper in
`backend/services/review_state.py`, now used by both `correction-data`
and `start-review`.
- An already-`IN_REVIEW` job does not transition again.
- A lost race re-reads the job and carries on.
- Any other invalid transition still errors.
- `record_review_started` runs only for the winning transition.

Version 0.278.4.

## Testing
- 4 new `TestCorrectionDataReviewTransitionIdempotent` cases.
- 493 review and correction backend tests pass.
- Local CodeRabbit found nothing.
- Verified against the local stack via the mobile Maestro review-bridge
flow.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_011yLH4kWXMqjeKfZUG3ZiWZ

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 12:07:03 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 072f85cad9 fix(review): idempotent start-review (no more in_review -> in_review error) (v0.278.3) (#1181)
## Summary
Opening a lyrics review could fail with `Invalid state transition
in_review -> in_review`.

`POST /api/jobs/{job_id}/start-review` always called
`transition_to_state(IN_REVIEW)` and never checked for `in_review`. So
any second open failed: a duplicate request, a double-mount, a second
tab, or the mobile app's in-app browser opening the review again. Mobile
E2E found this, and a screenshot captured gen's error page.

The endpoint is now idempotent:
- If the job is already `in_review`, it returns `{"status": "success",
"job_status": "in_review"}`.
- If the transition raises because a concurrent call won the race, it
re-reads the job. If the job is now `in_review`, it returns the same
success.
- Any other invalid transition still raises.
- `record_review_started` runs only for the winning `awaiting_review →
in_review` transition.

Version 0.278.3.

## Testing
- 4 new `TestStartReview` cases in `backend/tests/test_jobs_api.py`.
- The review-related backend tests pass: 452.
- Local CodeRabbit found nothing.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_011yLH4kWXMqjeKfZUG3ZiWZ

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 10:20:58 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 2838d2dd1d fix(credits): keep credits_charged through job creation; one helper for confirm/refund/select (v0.278.2) (#1180)
## Summary
Fixes `credits_charged` being wiped from `state_data`, which made
duration confirm, refunds and the web cost dialog disagree. Mobile E2E
against the local stack found it: a 15-minute track showed "1 credit in
total (1 more)", and confirming the true total was rejected with 409.

**Root cause:** `create_job` debits the credit and records
`state_data.credits_charged`. But create-from-search, audio-search
create, made-for-you orders (which also lost `payment_bypassed`) and the
admin source override then replaced the whole `state_data` map, deleting
the field. Readers filled the gap with different defaults:
- reconcile: 1
- confirm-duration: 0
- 48h expiry: 0

**Fix**
- **Writers:** per-key `state_data.<key>` updates instead of whole-map
writes. Retry-from-scratch keeps the billing keys. Reconcile records
`credits_charged` when it pauses a job.
- **Readers:** one helper, `backend/services/job_credits.py`
(`get_credits_charged`, `is_payment_bypassed`). Every reader uses it:
reconcile, confirm-duration, `/select`, the bulk worker,
`_refund_credit_for_job` and the 48h expiry.
- For legacy jobs that lack the field, it returns what `create_job`
actually debited: 0 for bypassed, admin, tenant or no-email jobs,
otherwise 1.
- **`/select` and the bulk worker** never bill payment-bypassed jobs
(made-for-you, tenant, admin-created). This includes legacy ones whose
flag was wiped.
- **API responses:** `GET /api/jobs/{id}` and the job lists return the
resolved `credits_charged` when the stored field is missing. This is
response-only and never persisted, so the web and mobile clients
acknowledge exactly `credits_charged + pending`.
- **Confirm-duration** records the full total, so cancelling after a
confirm refunds it all. The 48h expiry refunds once:
`cancel_job(refund=False)` stops a second refund.
- **Admin source override** clears the old source's duration, selection
and confirm markers and keeps `credits_charged`. The next `/select` then
charges only the difference.
- **Web cost hook:** uses the server's resolved figure.
- Version 0.278.2.

## Testing
- New `test_credits_charged_consistency.py` covers:
- reconcile, then confirm with total 2, then cancel: 2 refunded, for new
and legacy jobs
  - 409 on the old total
  - refund amounts unchanged for jobs that have the field
  - `/select` and the bulk worker never deducting on bypassed jobs
  - the override followed by `/select` charging only the difference
- Stale-expiry legacy tests, and an emulator test for the dotted-path
merge (runs in CI).
- Backend unit tests: 6411 passed. Emulator integration tests run in CI;
locally they collided with a running local stack. Frontend jest: 1754
passed.
- An independent review found one Critical issue (`/select` billing
bypassed jobs) and one Important issue (web/server default mismatch).
Both are fixed and re-reviewed clean. Local CodeRabbit raised one minor,
which is fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_011yLH4kWXMqjeKfZUG3ZiWZ

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 08:38:35 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 695b8ddf84 feat: mobile app backend — handoff token, app checkout return, Google/Apple sign-in (v0.278.0) (#1178)
## Summary
Backend support for the Nomad Karaoke mobile app (nomad-mobile Phase 1,
"Make it").

- **`POST /api/users/auth/handoff-token`**: the app opens a job's web
review already signed in. The token is a single-use, 120s
`job_review:<id>` magic link, returned as
`{FRONTEND_URL}/{locale}/auth/verify?token=…&handoff=1`.
- The verify page skips the "Complete Sign-In" tap only when the URL has
`handoff=1` **and** the server confirms the token was minted as a
handoff (`link-status` → `handoff: true`). Emailed links keep the
scanner guard.
  - The web locale is validated against the 33 web locales.
- **Checkout `return_to: "app"`**: Stripe success and cancel go to `GET
/api/users/credits/app-return`, which 302s to
`nomadkaraoke://checkout?status=success|cancelled`.
  - The client never supplies a URL, so this adds no open redirect.
  - `session_id` is never forwarded.
- **`POST /api/users/auth/google` and `/auth/apple`**: ID-token sign-in
(PyJWT + JWKS: RS256, issuer, audience, expiry, `email_verified`). It
returns the same `VerifyMagicLinkResponse` as email-code sign-in,
through a shared `finish_verified_sign_in`.
- Email-code, Google and Apple all share the same abuse checks
(`_sign_in_checks`).
  - A JWKS outage returns 503 `identity_provider_unavailable`.
- Google audiences default to the new mobile web and iOS OAuth clients,
which are recorded in ARCHITECTURE.md.
- **Job summary** now includes `credit_refunded`.
- **Local stack:**
- fake-config `search_confidence` (perfect / recommended / unsure), with
a drift guard tied to the frontend `getSearchConfidence`
  - `youtube_url_enabled`
  - `fake:<email>` social tokens, accepted only in fake mode
- PyJWT is now a direct dependency (`pyjwt[crypto]`). Before this it was
only present transitively.
- Version 0.278.0.

## Testing
- `make test` is green:
  - CLI: 2088 passed
  - backend unit: 6360 passed
  - emulator: 156 passed, plus 4 handoff lifecycle tests
  - frontend jest: 1751 passed
- New suites:
  - `test_handoff_token.py`
  - `emulator/test_handoff_token_lifecycle.py`
  - `test_checkout_app_return.py`
  - `test_social_signin.py`
  - `test_magic_link_status.py`
  - `fake-search-tiers.test.ts`
  - verify-page tests
- Reviews:
- An independent code review was approved, and its hardening suggestions
were applied.
  - A local CodeRabbit review found nothing.

Plan: `docs/archive/2026-10-17-mobile-app-auth-handoff-plan.md`. Spec:
nomad-mobile
`docs/superpowers/specs/2026-10-17-mobile-phase1-make-it-design.md`.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_011yLH4kWXMqjeKfZUG3ZiWZ

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 04:03:05 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 8ec789a9aa fix(bulk): album lookup reads our MusicBrainz mirror, never the live API (v0.277.0) (#1176)
## Summary
Bulk → By album → "Britney Spears" showed **"Lookup failed. Please try
again."** `/api/bulk/album/artists` returned 502 because every attempt
to `musicbrainz.org/ws/2` from Cloud Run failed with
`httpx.ConnectError('')`. The same request from a home IP returns 200.
It last worked on 9/29. Most likely MusicBrainz is refusing our shared
GCP egress.

Per Andrew, we should only query our own infra, with no live-API
fallback. `musicbrainz_service` now reads **only our BigQuery
MusicBrainz mirror** (`karaoke_decide.mb_*`, which karaoke-decide's
`mb-refresh` rebuilds weekly from the official dumps).

## Changes
- **`backend/services/musicbrainz_service.py`**
- The HTTP client, throttle and retries for musicbrainz.org are removed.
Queries go through the BigQuery REST `jobs.query` endpoint (httpx + ADC;
no new dependency).
- Queries are parameterized, capped with `maximumBytesBilled` = 2 GiB,
and cached in memory for 10 min.
  - Lookups by table:
- artists: `mb_artists_normalized`. Ranking is exact match (including
"The "+q), then prefix, then substring, then popularity.
    - albums: `mb_artist_credit_artists` → `mb_release_groups`.
    - editions: `mb_releases`.
- tracks: `mb_release_tracks`, the new table in karaoke-decide#158,
clustered by release group + release.
- Measured against prod data: about 0.6–1 s per query, MB to 0.4 GB
billed.
- Canonical-pressing selection now ranks an edition by its best
release-event country.
- **Route:** passes `release_group_mbid` through on edition switches. An
unknown release returns **404** instead of 502.
- **Frontend:** `switchEdition` sends the album's `release_group_mbid`
so the lookup stays cluster-pruned. Older clients still work through a
one-off lookup that scans about 0.2 GB.
- **Infra:** the `karaoke-backend` SA gets project-level
`bigquery.dataViewer` + `bigquery.jobUser`. decide's Pulumi owns the
dataset ACL, so a dataset-scoped grant would fight it.
- **Docs:** `docs/API.md` and
`docs/archive/2026-10-16-bulk-album-musicbrainz-mirror.md`.

## Rollout (order matters)
1. Merge karaoke-decide#158, then build `mb_release_tracks`: by hand
(SQL in the archive doc, about $0.08) or at Sunday's refresh.
2. Run `pulumi up` for the IAM change. The agent's credentials are
read-only, so this needs Andrew.
3. Merge this PR.

## Testing
- `make test`: all passed (backend + 1735 frontend tests).
- New and rewritten unit tests cover normalization, BigQuery row
decoding, the REST request shape (no musicbrainz.org, billing cap,
params, cache), the HTTP/timeout errors, every lookup, the release-group
lookup path, and the 404.
- Ran the service live against the prod mirror: Britney Spears → 39
albums → …Baby One More Time (11 tracks, 6 variants); also checked
"beatles", "Beyoncé", "ac/dc", "Sigur Rós".
- CodeRabbit CLI: 1 minor finding (404 vs 502), fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01C1sMXk2MaxBLbep5jgiFab

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 00:06:16 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 b2ef88a861 fix(review): unify backing-vocals recommendation on the 3-stem scorer (v0.276.2) (#1169)
## Summary
- `backing_vocals_analysis.recommended_selection` (legacy
`AudioAnalyzer` rule: keep only if >20% audible or a segment > −20 dB)
disagreed with the auto-approval `score_backing` verdict on **59/389
jobs (15%, Jul–Oct)** — every case 0.5–20% audible with no loud segment.
The finish modal followed the scorer, the instrumental screens followed
the heuristic, so the same job showed different "✓ recommended" stems
(job 58555663).
- Where they disagreed, admin/Andrew picks matched the scorer 14/18;
customer picks lean clean regardless (38% clean even when both signals
say keep). The heuristic also recommended keeping on 5 jobs the scorer
flagged as "backing stem IS the lead" (humans picked clean 4/5).
- 58555663 itself: the intro backing (16–29s) is a voiced B3–C♯4 hum
motif with the lead stem silent, pitch-matched to the "Mm-mm-mm" hum
lines — `with_backing` was right.

## Changes
- `unified_backing_recommendation` / `with_unified_recommendation` in
`auto_approval/instrumental.py`: scorer's confident verdict → `clean` if
backing-is-lead → else the heuristic (preserved as
`heuristic_recommended_selection`). Idempotent.
- Applied at write time (`screens_worker._analyze_backing_vocals`) and
read time (correction-data, `/instrumental-urls`,
`/instrumental-analysis`) so already-analysed jobs are consistent too.
- `backing_is_lead()` predicate extracted from `score_backing` and
shared.
- No change to auto-completion / enforcement (`backing_decision`
untouched); frontend unchanged.
- Docs: API.md note, LESSONS-LEARNED entry. Version 0.276.2.

## Testing
- [x] `make test` — backend 2054 + 6243 + emulator 152, frontend 1730,
all passing
- [x] New unit tests (helpers matrix), worker write-path test, route
test with the 58555663 signal profile

## Review
- [x] CodeRabbit CLI review completed locally — no findings

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_018aUxu88FZajta46BRdmpcc

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 13:08:49 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 abf7b693b4 feat(review): instrumental chooser appears when stems land after review opens (v0.276.0) (#1167)
## Summary

Recent jobs didn't show the instrumental chooser / backing-vocals
waveform in the Preview Video modal. Root cause: review opens as soon as
**lyrics** finish (~2 min after download), while audio separation lands
2-3 min later. The review page fetches `instrumental_options` once at
load, so stems that arrive mid-review never reach the modal, which
silently fell back to "Proceed to Instrumental Review".

Evidence (last 5 admin jobs): in 4/5 the review page was opened 11-23 s
after ready-for-review, and separation finished 1m46s-3m00s **after**
the page opened; the one opened after separation (6cb0f55f) was fine.

## Changes

- **Backend:** `GET /api/review/{id}/instrumental-urls` also returns
`audio_complete`, `job_status`, `backing_vocals_analysis` and a freshly
derived `auto_approval` summary (backwards-compatible).
- **Frontend:** new `useLateInstrumentalStems` hook. When the page
loaded without stems it polls every 15 s (max 10 min) until
`audio_complete`, which is set only after stems, OGGs and backing
analysis are all written, so the chooser never shows half its data.
Fresh data is overlaid as a separate modal prop, not merged into `data`,
so undo history, saved sessions and the in-flight preview render are
untouched. The getter is held in a ref because the page re-creates its
API client each render.
- **Modal:** while waiting, a spinner box ("Still separating the music
from the vocals…") and a disabled CTA ("Waiting for the instrumental…").
The chooser, ✓ badge, backing waveform and preview toggle appear in
place when ready. On failure or timeout it falls back to the old path
with a short note.
- i18n: 4 new strings, 33 locales.
- Fixed an existing Playwright test that expected "Proceed to
Instrumental Review" with both stems present (stale since v0.214; it now
picks Advanced mode).

## Testing

- `make test`: all pass (backend, plus frontend 140 suites / 1729 tests)
- New: 2 backend endpoint tests, 6 hook tests, 3 modal tests, 1
Playwright regression test (loads without stems → waiting state →
chooser appears)
- Local CodeRabbit: 1 finding (false positive: `advancedModeClean`
exists in all 33 locales)

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01QKJ7TxzFoUeNTug1PR8Cwh

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 00:21:35 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 090bbf9ada feat(tenant-theme): CD+G instrumental/closing previews + transitions; per-tab previews, wider editor (v0.275.0) (#1166)
Follow-up to #1164, from Andrew's review of the theme editor.

## CD+G tab
- **All four CD+G screens previewed** from one real CDG: up card, lyrics
page, **instrumental break** (the sample lyrics include a gap ≥ a
preview threshold; the fully drawn frame is found by colour) and
**closing screen**. ~1.3 s.
- **"How it's drawn in"** for the up card, instrumental break and
closing screen. These are the "mask" images: cdgmaker's built-in
greyscale draw-order masks (`cdgmaker/transitions/*.png`, e.g.
`centertexttoplogobottomtext`) that decide the order tiles are revealed.
They change the animation, not the finished frame. Validated server-side
against the built-ins; the editor GET returns `cdg_transitions`.
- **Instrumental gap threshold** slider (3–120 s; stored in
centiseconds, validated 300–12000).

## Theme & style modal
1. ~96vw wide (max 1800px); the preview column is sticky.
2. **Each tab previews only its own screen** (Title card → title card,
Lyrics video → lyrics frame, End screen → end screen, CD+G → the 4 CD+G
screens, Advanced → all video screens). The server renders only what's
asked (new preview `kinds`), so previews are faster too.
3. CD+G previews in a compact 2×2 grid.

## Tests
- Backend: instrumental + closing frames render (distinct colours),
one-line sample still shows the break, `kinds` limits rendering,
transitions validated (incl. non-string), threshold bounds, editor lists
transitions, route `kinds` + 422 on unknown kinds.
- Frontend: per-tab previews/requests, Advanced shows all, modal width,
4 CD+G previews in a 2-col grid, transition picks saved, threshold
seconds↔centiseconds, unset placeholder. Full jest 1720 pass; backend
suite 6126 pass; package suite 2054 pass.
- 32 locales translated. CodeRabbit: 4 minor findings fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Gp5fBmsmWJxgkEHb4JfhXR

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 16:01:56 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 c277bc8ee3 feat: local E2E stack (fake workers + test API) + email-code sign-in (v0.275.0) (#1165)
## Summary
- **Local stack:** `scripts/local-stack.sh up --frontend` runs the
**real** backend on the Firestore + GCS emulators with `FAKE_WORKERS=1`.
In-process fake workers move jobs through their real states in seconds,
using tiny real media, so web Playwright and mobile Maestro full-journey
tests can run offline against the same endpoints.
- **Email-code sign-in** (nomad-mobile spec §4.1): `POST
/api/users/auth/send-code` + `verify-code`. The code logic is shared
with kjbox via `LoginCodeStore`, and sends are throttled per email and
per IP.
- **Bug fix found by the new suite:** `confirm-duration` moved jobs to
`separating_stage1`, from which they can never reach review. It now
resumes at `downloading`. No prod job had taken this path yet.

## Changes
- `backend/services/fake_workers/`: dispatch seam in
`WorkerService.trigger_*`, plus fake search, download, checkout and
outbox, scenarios, and per-user config. The real stem upload, transcode,
backing-vocals analysis and preview render are reused. Every check is
strict (`is True`), so a MagicMock settings object never enables the
fakes, and startup **refuses FAKE_WORKERS in production**.
- `/api/test/*` router (mounted only in that mode): `status`,
`scenario`, `fake-config`, `emails`, `reset`, plus the fake Stripe
checkout page. Paying it runs the real webhook handler.
- `frontend/e2e/local-stack/` (9 specs, ~50s):
  - sign-up → make → review → instrumental → download
  - buy and cancel credits
  - duration confirm, failed, complete and review states
  - code sign-in and lockout
- New required CI job `local-stack-e2e`.
- Docs: TESTING.md § Local stack, DEVELOPMENT.md, API.md, README,
LESSONS-LEARNED.
- `MAX_SIGNUPS_PER_IP` can now be set by env. Only the local stack
raises it; production keeps the default of 2.

## Testing
- [x] Backend unit: 6175 passed
- [x] Emulator: 152 passed
- [x] Frontend Jest: 1706 passed
- [x] Local-stack Playwright: 9/9 passed (CI mode)

## Review
- [x] CodeRabbit CLI review completed locally. 1 minor finding declined:
it suggested swallowing infrastructure errors in send-code as fake
"sent" responses, which would hide real outages from users.

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01AGuxtgoscF6RJGtpjHPg3W

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 16:00:24 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 eeb08578c9 feat(tenant-theme): CD+G tab in the theme editor with real CD+G previews (v0.274.0) (#1164)
## Why
Commercial producers (Vocal Star first) care a lot about their CD+G
output, but the tenant theme editor only covered the MP4 title card /
lyrics / end screen. CD+G styling needed an admin to hand-edit the theme
JSON.

## What
**New "CD+G" tab** (only on portals with `features.enable_cdg` and a
`cdg` theme section):
- **Up card**: image, title/artist colours, title/artist sizes,
separator text between them (e.g. "MADE FAMOUS BY") + its colour, text
position from top (keeps text below a logo baked into the image),
title–artist gap.
- **Lyrics**: sung / unsung / screen / border colours, size, outline
width + colour, lyrics font ("Same as main font" or a distinct one, e.g.
condensed Oswald).
- **Instrumental breaks**: image, text, colour. **Closing screen**:
image, line colours.

**Real previews**: on the CD+G tab the preview request sets
`include_cdg`; the backend composes a short CDG with the real
`CDGGenerator` from the draft's `cdg` section and renders the up card +
a part-sung lyrics page with the new `karaoke_gen/.../cdg_decoder.py`
(single pass, multi-frame; moved out of the CDG test). 300×216 upscaled
3× with hard pixels, ~1 s. Other tabs never pay for it.

**Images**: uploads from the CD+G tab are fitted (letterboxed) to
300×216 PNG (`purpose=cdg`); the CD+G dropdowns only offer 300×216
images (`cdg_images` from the editor GET, header-only reads);
save/preview reject other sizes since the CDG renderer never resizes.
All 4 existing themes' CD+G images verified 300×216.

**Validation**: `cdg` hex colours, numeric bounds, single-line text
limits; distinct CDG font resolved for the preview the same way renders
do (`cdg_font` asset), title/outro use the main font.

## Tests
- Backend: CDG validation/fit/size-check/upload/editor list
(tenant_theme_service), real CD+G preview renders incl. colour change +
distinct font + missing asset + timeout (theme_preview_service), route
include_cdg + upload purpose. CDG title-card tests now use the shared
decoder.
- Frontend: new `TenantThemeEditorCdg.test.tsx` (10): tab gating,
includeCdg only on that tab, edits map to `cdg`, outline applies to both
strokes, font follow/distinct, fitted upload, CD+G-sized image list,
pixelated 300/216 previews. Full jest 1716 pass.
- Backend suite: all pass except `test_emulator_integration.py`, which
fails identically on main locally (a stray local Firestore emulator
enables it); CI runs emulator tests separately.
- 32 locales translated. CodeRabbit: 2 minor findings fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Gp5fBmsmWJxgkEHb4JfhXR

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 15:02:24 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 bfc8f7fb0f feat(cdg): configurable title card + per-theme CDG lyrics font (Vocal Star up card) (v0.273.0) (#1157)
## Summary
Vocal Star (and other commercial producers) need CDG files in their own
house style. Matt's 2025-02-20 email showed our CDG up card overflowing
its bordered design. He asked for his simpler card instead: logo on top,
then title, a small "MADE FAMOUS BY" line, then the artist.

Two pipeline limits blocked that:
- the CDG title card was hardcoded;
- a theme could only have **one font** for everything. The shared `font`
asset (`intro.font`) overwrites `karaoke.font_path` and `cdg.font_path`
at render time.

## Changes
- **Title card (cdgmaker composer):** new optional keys; all defaults
reproduce today's output.
- `title_separator_text` / `_color` / `_font_size`: a line between title
and artist.
  - `title_font_size`, `artist_font_size`, `title_max_height`.
- `title_block_top`: centre the block below a logo baked into the
background.
- It shrinks the title first, then the artist, so it can't loop forever.
- **Outro:** respects `title_block_top` and shrinks long text. Before,
an `assert` crashed when the text was taller than 200px.
- **Fonts:**
- The CDG title card and outro use the theme's main font
(`cdg.title_font_path`, fed by the shared `font` asset).
- A theme may keep a **distinct CDG lyrics font** via a new `cdg_font`
asset. The theme service emits it only when it differs from
`intro.font`, and `update_asset_paths` applies the shared font first so
section overrides win.
  - No visible change for existing themes, which all use one font.
- **Tenant theme editor:** keeps a distinct, available CDG font on save,
and copies it into the theme's assets if it's bundled.
- Docs: API.md.

## Testing
- New `tests/unit/lyrics_transcriber/output/test_cdg_title_card.py`
renders **real CDG files**, decodes the packets back into frames and
checks:
  - the separator sits between title and artist;
  - text stays below `title_block_top` and inside the safe area;
  - the title font is applied, and a missing one falls back;
  - long outro text shrinks instead of crashing.
- New tests for asset-mapping order, `cdg_font` emission, theme-editor
font handling, and the updated mapping assertion.
- Package unit tests: 2053 passed. Backend: 6056 passed (emulator tests
excluded locally).
- Local CodeRabbit with `--fresh`: no findings.
- Rendered Vocal Star samples (short, wrapped-long and very long
title/artist), decoded and visually checked against Matt's design.

After deploy, the Vocal Star theme gets: a new up-card background (logo
cropped from Matt's own card), separator "MADE FAMOUS BY",
`title_block_top`, black border, Avenir as the main font and Oswald kept
for CDG lyrics.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Gp5fBmsmWJxgkEHb4JfhXR

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 00:58:42 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 5047f39da2 feat(tenant): per-track catalogue codes prefix tenant outputs (Vocal Star VS codes) (v0.272.0) (#1153)
## Summary
Vocal Star (Matt, April 2025) asked for their own VS catalogue code at
the start of every output file, with all files in one folder (e.g.
`VS6588 - Gerry and the Pacemakers - Ferry Cross The Mersey`). Tenant
operators can now set a per-track **Track code**. It becomes the job's
preserved brand code (`keep_brand_code`, which both render paths already
honour), so the Dropbox folder and every delivered file use it instead
of an allocated `VSTAR-NNNN`.

## Changes
- **Backend:** `POST /api/jobs/create-with-upload-urls` accepts
`track_code`.
  - Tenant portals only; `400` elsewhere.
  - Strict filesystem-safe validation, upper-cased.
  - Maps to `JobCreate.keep_brand_code`.
- **Bulk analyze:** a leading code in filenames (`VS6588 Artist - Title
Guide.mp3`) is split off and returned as `track_code`.
  - Files pair only when code **and** song match.
- LLM pairs with conflicting codes are rejected, and the code is
stripped from LLM-extracted artists.
- **Frontend:** an optional Track code field on the tenant single-track
form and on each bulk row.
  - Client validation mirrors the backend, with a visible error.
  - Resumed bulk rows keep their code, locked.
  - 33 locales translated.
- Docs: API.md. Version 0.271.0 → 0.272.0.

## Testing
- Backend: new tests for endpoint mapping, rejection outside tenants,
unsafe codes, normalisation, filename parsing, the band-name false
positive (UB40), same-code/different-song and LLM conflicts.
- Full backend suite: 5985 passed. The only 30 failures are emulator
tests, because the local Firestore emulator wasn't running.
- Frontend: full Jest 1680/1680 passing, plus new TenantJobFlow /
TenantBulkFlow tests. `tsc` shows no new errors.
- Local CodeRabbit: 3 cycles, all real findings fixed. The one remaining
note is on a pre-existing string.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Gp5fBmsmWJxgkEHb4JfhXR

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 00:05:57 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 4ad946c3aa feat: free-to-paid conversion emails + last-chance review nudge, OFF until copy approved (v0.271.0) (#1156)
## Summary
This is BACKLOG Frog #2 ("plug the gen revenue leaks"), conversion-email
track. 44% of signups finish a free video, but only 6% ever pay, and
buyers decide fast (median about 5h). 45 users opened lyrics review and
let the job expire.

**Ships dark.** Nothing sends until Andrew approves the copy and
`CONVERSION_EMAILS_ENABLED` / `REVIEW_LAST_CHANCE_ENABLED` are added to
the ci.yml deploy env.

| Email | When | Stream |
|---|---|---|
| E1 "Your karaoke video is done. What's next?" + 25% off any pack for
48h | 2h after a never-paid user's **first** video completes | broadcast
|
| E2 "Your 25% off ends in 8 hours" | last 8h of the offer, if unpaid |
broadcast |
| R2 "6 hours left to finish your karaoke video" | 42h into review (the
existing reminder is at 24h; expiry is 48h) | transactional |

- `services/conversion_emails.py`:
- A `JobManager` hook marks the user on their first unpaid completion.
Tenant, Made For You and test accounts are excluded.
- The hourly `process-stale-reviews` cron sends the emails. It's
idempotent per user, and due times more than 24h stale are skipped, so
turning it on never emails a backlog. No infra change is needed.
- **Promo:** E1 grants `promo_discount_percent` /
`promo_discount_expires_at`. Checkout applies coupon
`conversion-offer-{pct}pct` when it beats a referral discount.
- The buy dialog shows a banner, and its prices use the effective
discount (CodeRabbit finding).
  - The email button links to `/app?buy=1`, which opens the dialog.
- **Compliance:** E1 and E2 go on Postmark's `broadcast` stream, where
Postmark manages unsubscribes (verified: it appends the link). They
include a postal-address footer (`COMMERCIAL_EMAIL_POSTAL_ADDRESS`).
`EmailProvider` gains `message_stream`, and email_log records the
stream.
- Copy is in `backend/translations` (33 locales). Docs:
`docs/CONVERSION-EMAILS.md`, covering how it works, measurement and how
to turn it on.

## Testing
- `make test`: backend 6017, emulator 150, frontend Jest 1688, package
2044. All green. Re-ran the targeted suites after rebasing onto the tip
PR.
- New tests:
- `test_conversion_emails.py` (26): eligibility, E1/E2 windows,
staleness, promo vs referral, broadcast stream, rendering.
- Stale-review R2 tests (3), `BuyCreditsDialog.test.tsx` (3), and an
AuthStatus `?buy=1` test.
- Rendered E1 and R2 HTML and checked them visually.
- Local CodeRabbit: 1 finding fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_019KuzHAU4rxkv8JfD8NmHTN

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 23:50:59 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 542ad66b65 feat: opt-in 'tip the creator' add-on at gen checkout (v0.270.0) (#1155)
## Summary
This is BACKLOG Frog #2 ("plug the gen revenue leaks"), tip track. Tips
were **27% of old order-form revenue** ($950 on 178 of 231 orders), and
gen had nothing like them.

- Both gen checkouts (credit packs and Made For You) now offer an
**opt-in** Stripe Checkout `optional_items` tip: "Add to your order →
Tip the creator (optional) $5 [+ Add]".
- It's never pre-added. The old order form put a $5 tip line in the cart
at quantity 1 (opt-out), which is banned for UK/EU consumers.
- **Price:** $1/unit, found or created by
`lookup_key=gen_checkout_tip_usd_1`. It defaults to 5 units and is
adjustable from 1 to 100.
- Live product `prod_VRY9FhfNhiS0NN` already exists, with a description
explaining "each unit is $1".
- Any Stripe error means no tip is offered, so checkout is never
blocked. Kill switch: `CHECKOUT_TIP_ENABLED=false`.
- **Accounting:** credits are untouched. `stripe_payments.tip_cents`
records the tip separately.
- Referral earnings exclude tips. If the tip can't be read from Stripe,
the base is capped at the package list price (CodeRabbit finding).
- Docs: STRIPE-SETUP.md has a new "Checkout Tip" section.

## Testing
- Verified against **live Stripe**: created a real session (since
expired, nothing charged), and the hosted page shows the tip as an
opt-in "Add" item with the total unchanged.
- `make test`: backend 5977, emulator 150, frontend Jest 1676, package
2044. All green.
- New `backend/tests/test_checkout_tip.py`: price lookup and creation,
kill switch, optional item on both checkouts, opt-in only, tip_cents,
referral base.
- Local CodeRabbit: 1 finding fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_019KuzHAU4rxkv8JfD8NmHTN

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 23:39:06 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 ff9b716435 feat: first-touch UTM/referrer signup attribution + UTM on YouTube description (v0.269.0) (#1154)
## Summary
This is BACKLOG Frog #2 ("plug the gen revenue leaks"), attribution
track. Until now we stored no acquisition source, so we couldn't tell
which channel brings the people who pay.

- **Capture (frontend):** `lib/attribution.ts` stores the first-touch
`utm_*` params, the external referrer host and the landing path for 30
days. It runs in the `[locale]` layout and in `LocaleRedirect`, before
`location.replace` wipes `document.referrer`. The data is sent with
`POST /auth/magic-link`.
- **Store (backend):** `services/signup_attribution.py` whitelists,
length-caps and sanitises the data. On account creation it sets
`signup_attribution` and `signup_source`: the `utm_source`, or a
referrer channel label such as `youtube`, `google`, `website`, `decide`
or `kjbox`. Malformed input never blocks sign-in, and the request field
is bounded (16 keys, values up to 500 chars).
- **Report:** `GET /api/users/admin/attribution-summary?days=90` returns
signups, verified, made a job, paid and revenue, per source. The admin
user detail shows the signup source and attribution.
- **YouTube:** description template links now carry
`utm_source=youtube&utm_medium=description&utm_campaign=requests|make_cta`.
They're static, so they're still derivable from the title, and the daily
backfill rolls them out. The `requests.nomadkaraoke.com` Cloudflare
redirect now has `preserve_query_string` (Pulumi `nomad-cloudflare`,
**already applied** locally).
- **Docs:** STRIPE-SETUP.md gets a "Legacy Payment Links" section. The
old order-form link `dR67sx…` was **deactivated 2026-10-14** and points
people to gen; it had 0 completed checkouts in about 30 sessions since
Dec. Also updated API.md.

Companion: nomadkaraoke/public-website#34 makes
`nomadkaraoke.com/r/CODE` pass the UTM params through.

## Testing
- `make test`: backend 5986 passed, emulator 150, frontend Jest 1683,
package 2044. All green.
- New tests: `backend/tests/test_signup_attribution.py` (cleaning,
channel labels, magic-link wiring, bounded payload, summary endpoint)
and `frontend/lib/__tests__/attribution.test.ts`.
- Local CodeRabbit: 3 findings fixed (malformed referrer, NaN expiry,
unbounded payload); re-review found 0.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_019KuzHAU4rxkv8JfD8NmHTN

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 23:31:16 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 983f6ecdcd fix(storage): keep user-uploaded inputs forever; uploads/ expires after 180d not 7d (v0.268.2) (#1137)
## Summary
`uploads/` had a bucket lifecycle **delete after 7 days**, and user
inputs were only partly copied out of it. The only copy step was the
audio worker (stem separation) copying the mix to `jobs/{id}/input/`,
best-effort. Everything else stayed in `uploads/`:
- bring-your-own-instrumental / tenant jobs, which skip separation;
- the uploaded instrumental, which was only staged at render time;
- lyrics and style uploads, which were never copied.

Result:
- randy-vild re-renders 404'd on their own input (2026-10-08). That data
was restored from Andrew's Dropbox.
- Firestore shows 64 completed Jan–Feb 2026 consumer jobs, plus
BYO-instrumental jobs, whose inputs are gone permanently.

## Changes
- **`backend/services/input_persistence.py`** (new):
- Server-side copies `uploads/{id}/**` to `jobs/{id}/input/**` and
repoints every job field that referenced the upload:
`input_media_gcs_path`, `existing_instrumental_gcs_path`,
`lyrics_file_gcs_path`, `style_params_gcs_path`, plus any matching value
under `file_urls` / `state_data` / `style_assets`.
  - Idempotent: unchanged copies are skipped.
- The audio, lyrics, screens, render-video and video workers all call it
on entry, via `asyncio.to_thread`. On failure it logs an ERROR, which
alerts, but doesn't fail the job.
- **`POST /api/admin/persist-uploads`** (dry run by default): backfill
for jobs no worker will touch again. Lookup failures are isolated per
job.
- **GCE encoder:** prefers the explicitly downloaded top-level
`existing_instrumental.*`. The recursive glob could otherwise return the
raw, unconformed copy now kept under `input/audio/`.
- **Pulumi:** `uploads/` lifecycle goes from 7 to **180 days**; `temp/`
stays at 7. **Already applied** and verified directly on the bucket.
- **Docs:** LESSONS-LEARNED, DISASTER-RECOVERY (new user-inputs row; the
stale "30d noncurrent" figure corrected to 7d), API, README.

## Verification
- `make test` all green.
- 16 new tests:
- copy mapping, repointing, idempotency, re-copy when a kept copy
differs, already-expired paths left alone;
  - the worker wrapper never raises;
- an AST guard that every pipeline worker calls the copy step via
`to_thread`;
  - backfill dry run / apply / error isolation;
  - the encoder's instrumental preference.
- Local CodeRabbit: 2 findings, both fixed (lyrics path, per-job
lookup); re-review clean.

## After merge
Run `POST /api/admin/persist-uploads` (dry run, then `?dry_run=false`)
to copy the inputs of the 51 jobs that still have objects in `uploads/`.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 17:33:06 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 f2612ac110 fix: close edit→delete refund loop, retry lost YouTube uploads, unblock event loop (v0.268.1) (#1135)
## Summary
Investigation of two 2026-10-08 Discord alerts (plus a third that
arrived mid-session) turned up four issues:

1. **Credit-refund loop.** A brand-new welcome-credit account received a
finished video (job 8a10a22d), clicked **Edit** (status back to
`awaiting_review`), then **deleted** the job and got its credit back.
`delete_job` refunds any non-complete status, and Edit makes a delivered
job look "never completed".
2. **YouTube 410 "Gone" dropped the upload.** The resumable session was
lost 1.3s into the upload. `upload_video` called `execute()` once with
no retries, and the orchestrator only queued quota errors, so any other
failure lost the upload.
3. **EVENT_LOOP_STALL 21s in `upload_json`.** `complete_review` (an
async route) wrote the corrections JSON to GCS directly on the event
loop. Ranking a week of watchdog stacks by route showed the same pattern
in `recover_stuck_jobs` (~260 stalls/week), the YouTube queue processor
(its `_process_single_upload` ran a whole upload on the loop),
`create_job_from_search` (theme prep), `_validate_audio_durations`
(15s), and the sync `run_job` / generation bump inside worker triggers.
4. **False 30s "stall" in asyncio `get_debug`.** Cloud Run CPU
throttling froze the whole instance mid-callback with no request in
flight. The watchdog only discounted freezes where the loop was parked
in `select()`.

## Changes
- **Refund:** `_refund_credit_for_job` skips jobs with `edit_count > 0`
(already delivered). This covers delete, cancel, fail and the 48h review
auto-expiry.
- **YouTube:** on 404/410 the upload restarts with a fresh session (3
attempts, with backoff). `execute(num_retries=3)` lets googleapiclient
resume chunks on 5xx/429 within a session. Any other failure is queued
in `youtube_upload_queue` (`reason="upload_error"`) for the existing
processor, which retries up to 5 times. Admins still get a Discord
warning: "queued for automatic retry".
- **Event loop:**
- New `backend/utils/loop_bridge.py`: `run_in_body_thread` runs sync
route bodies on a dedicated 16-thread pool, so they can't exhaust the
default executor the dispatched coroutines need. `run_on_loop`
dispatches coroutines back to the API loop (`run_coroutine_threadsafe`,
120s timeout).
- `complete_review`: its sync save runs via `to_thread`, and the render
trigger stays on the loop.
- `create_job_from_search`, `recover_stuck_jobs` and
`retry_pending_render_jobs`: bodies run in the route-body pool.
`_retrigger_parked_download` now does its writes in that thread.
- `_validate_audio_durations` and the YouTube queue's
`_process_single_upload` keep their async signatures but run in threads.
The queue's quota and Firestore calls go through `to_thread`.
- `worker_service`: the generation bump and `client.run_job` go through
`to_thread`.
- **Watchdog:** if the watchdog thread itself oversleeps by 2s or more
while the process uses less than 25% CPU, the whole process was frozen.
That time is subtracted from the stall and isn't sampled. Blocking I/O
(the watchdog still wakes on time) and GIL-hogging CPU work (CPU keeps
advancing) are still reported.
- Docs: `API.md` (edit/refund note) and `LESSONS-LEARNED.md`
(mostly-sync routes that await one trigger; throttling freezes).

## Testing
- Backend unit suite: **5863 passed**, 36 skipped (`pytest backend/tests
--ignore=backend/tests/emulator`).
- New tests:
- edited-track refund cases (delete in awaiting_review /
lyrics_complete, cancel);
  - YouTube restart on 410, give up after 3 attempts, no restart on 403;
  - orchestrator queues failed uploads and warns when queueing fails;
  - loop-responsiveness ticker tests for each offloaded path;
  - `run_on_loop` result, exception and timeout-with-cancel;
  - `run_in_body_thread` copies context and uses its own pool;
- watchdog: freeze ignored, I/O stall and CPU hog still reported, freeze
subtracted from a real stall.
- Local CodeRabbit: 3 cycles. Fixed the nested-executor deadlock risk,
the missing `run_on_loop` timeout, the parked-download writes on the
loop, and a coroutine leak. The last cycle had 0 findings.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 16:15:47 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 60687d6cd6 feat(theme): editable + previewed end screen; optional title/end screens (v0.268.0) (#1134)
## Summary
Randy Vild (tenant `randy-vild`) emailed: *"I have completed 3 songs but
all three songs produce the large text at the end even though I have in
my settings smaller text like in my beginning."*

Root cause: the theme editor's size/position sliders only edit
`intro.*`. The end screen has its own `end.*` layout (title 350 /
closing message 400 / yellow capitals artist in Randy's theme). The
editor neither exposed it nor previewed it ("The end screen isn't
previewed").

## Changes
- **End screen tab**: own title/artist colours, size + position sliders
(closing message, song title, artist), show/hide song title / artist
name, capitals toggles, and **Copy title card layout**. That copies the
title card's regions/colours/capitals and moves the closing message just
above the title, or below the artist if there's no room, so they never
overlap.
- **End screen preview**: `theme_preview_service.render_end_screen` uses
the same `VideoGenerator.create_end_video` as `screens_worker`. The
preview column shows the screen for the active tab plus the lyrics
frame; Advanced shows all three.
- **Leave out the title card / end screen**: optional `intro.enabled` /
`end.enabled` (absent = included; not added to the DEFAULT_*
required-field lists). The setting is honoured end to end:
- The screens worker skips the omitted screen and records
`state_data.screens_included`.
- The video worker's prerequisite check and the `/retry` ladder
(`_has_title_screen`, which also requires `screens_progress: complete`)
treat an omitted screen as done.
- The orchestrator sets the omitted screen's path to `None` and gives
the original-vocals guide an intro offset of 0.
- Encoding passes explicit `include_title_screen` /
`include_end_screen=false` to the GCE worker. Without them, its loose
`*title*.png` / `*end*.png` globs would rebuild the screen from
`style/*_background.png`.
- `LocalEncodingService.encode_lossless_mp4` builds the concat from
whichever segments exist (1–3).
  - The portrait renderer drops omitted cards.
- 11 new strings × 33 locales (`endScreenHint` removed). Docs: API.md,
README.md.

## Verification
- `make test` all green: frontend 1665 tests; backend unit, integration
and emulator suites.
- New `backend/tests/test_theme_screen_omission.py`, 28 tests:
  - screens worker for all 4 include combinations;
  - prerequisites and the retry helper;
  - orchestrator config and encoder options;
  - GCE `run_encoding` with decoy theme PNGs present;
  - concat for n = 1/2/3.
- Real-ffmpeg renders:
- end-screen preview (its own background; text size changes only the end
screen);
  - portrait durations with screens omitted.
- Frontend: editor tests for the end tab, copy layout, region toggles
and enable switches, plus unit tests for the `copyTitleCardLayoutToEnd`
util.
- Rendered Randy's real theme locally:
  - as saved, the end screen reproduces the large text;
  - after Copy title card layout, it matches his title card.
- Local CodeRabbit review: 0 findings.

## Deploy notes
- The worker paths changed (`gce_encoding/`,
`local_encoding_service.py`, `karaoke_gen/`), so CI blue-green redeploys
the encoding worker.
- Existing themes are unaffected: they have no `enabled` key.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 15:09:25 -04:00