mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
286e604b7571feba9e161ef856db2e2ee00d57d2
1997
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
286e604b75 |
feat(worker): voice-pool separate-only GPU job (v0.283.1) (#1217)
## Summary - Adds `voice-pool-separate-job`, a Cloud Run GPU (L4) job for karaoke-decide's voice session. decide writes a JSONL manifest (`voice-pool/manifests/<run>.jsonl`) and starts the job with a per-run shard; each task separates its share of songs with the `instrumental_clean` Separator and writes `voice-pool/stems/<id>/vocals.flac` (skipped if already present). Results are written back to the manifest area every 10 songs. - No Firestore or gen job state. Model-load failure or an all-songs-failed task exits non-zero with a single ERROR; partial failures exit 0. - Task timeout is 3600s (Cloud Run's GPU maximum, found by the local CodeRabbit review), so decide must size the shard count to keep each task under 1h. Re-runs skip finished songs. ## Changes - `backend/workers/voice_pool_separate_worker.py` + `backend/tests/test_voice_pool_separate_worker.py` (9 tests, 91% coverage of the worker) - `infrastructure/modules/cloud_run.py`, `infrastructure/__main__.py`: Pulumi job definition - `.github/workflows/ci.yml`: CI guard step (tolerates a missing job) - `docs/ARCHITECTURE.md`: paragraph on the job - Version 0.283.0 -> 0.283.1 ## Deploy notes **Andrew: run `pulumi up` in gen infrastructure/ BEFORE merging (creates voice-pool-separate-job; the CI guard tolerates a missing job).** decide's Pulumi (later PR) will create the voice-pool SA and grant it `roles/run.jobsExecutorWithOverrides` on this job. ## Testing - [x] `pytest backend/tests/ --ignore=backend/tests/emulator`: 6691 passed, 40 skipped - [x] Local CodeRabbit review completed (2 cycles): the critical timeout finding fixed; the remaining minor (results flush inside `finally`) is a known deferred item @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_0188txBF47GpTo7hym1FoifX --------- Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>v0.283.1 |
||
|
|
c4c4f9265c |
feat(bulk): album batches search the album once; one flacfetch search in flight per batch (#1214)
## Why A real Bulk "By album" batch (Chase & Status, *No More Idols*, 12 tracks): - It took about 5 minutes to reach audio selection. - 4 of the 12 searches timed out. - All 12 were parked for manual choice, each showing the same album torrents. Causes: - The album was dropped at submit, so each track was searched separately. - 4 parallel searches ran against flacfetch's single search worker, and the queued ones ran out the 95s budget. - The auto-pick bar was 50 seeders (lowered to 10 in #1211). - A title-only file search misses the album for common titles ("Time"). ## What - **Submit carries the album:** `POST /api/bulk/submit` takes `album` {artist, title, release_group_mbid, release_mbid} plus each song's `position`. These are stamped as `state_data.bulk_album` / `bulk_track_position`. The frontend album mode sends them, using the release-group title. - **One album search per album:** the worker makes one flacfetch `POST /search/album` call (nomadkaraoke/flacfetch#56). - Each track's file is resolved inside the album's torrents. - The whole album comes from **one torrent**: the first BEST CHOICE (lossless, non-vinyl, 10+ seeders). - A track is auto-selected when that torrent's file names it and the artist matches. - A track the batch torrent doesn't cover gets the normal per-track pick among the album torrents that do. - Parked jobs show the album torrents in the selection modal. - **Fallbacks are one search at a time** (`SEARCH_CONCURRENCY = 1`): uncovered tracks, text batches, and failed or unavailable album searches (404 from an older flacfetch). A batch search deadline (3000s) parks any leftovers before the 3600s job timeout could strand them. - **Auto-pick checks the artist** on the backend too: a Unicode-aware port of `isArtistMatch`, used by bulk and KaraokeHunt. A 10+-seeder cover or tribute torrent is no longer auto-picked. - Fake album search for the local stack; docs in API.md and LESSONS-LEARNED. The money path is unchanged. Album results have no duration, so nothing is charged at select time. The real duration charge or refund still happens after download (`reconcile_and_maybe_pause`). ## Testing - Backend: 6677 passed. The only failures are the 8 `test_emulator_integration` tests, which hit a stale local Firestore emulator from another session; they passed in CI on #1211. - Frontend: Jest 1765/1765. - New tests cover: - album worker path: one search, all auto-selected, uncovered → fallback, search failure → fallback, low seeders → park, auto disabled - `process_batch` routing, with at most one search in flight - the deadline - album helpers, artist match (incl. CJK and &), client `search_album` (incl. 404 and 502), submit stamping, the fake album response, and the frontend submit payload - Code review: CodeRabbit CLI unavailable (invalid token / org mismatch), so a local Superpowers review ran instead. Its fixes are in the second commit. ## Rollout flacfetch v0.34.0 (#56) deploys to flacup first. Then: one album search, 12 auto-selected jobs, one torrent; this is verified in prod after merge. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_019MixBuvzxB1c5AYc3mbDyW <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Album-mode bulk submissions now include album details and track positions, enabling one album-wide search for matching tracks. * Search results are checked against both track title and artist before automatic selection, with tolerance for minor name variations. * **Improvements** * Tracks not covered by an album result, or affected by an unavailable or failed album search, fall back to individual searches. * Searches still awaiting selection after the batch deadline are parked for manual review. * **Documentation** * Updated bulk submission guidance to describe album searches and fallback behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v0.283.0 |
||
|
|
76069573eb |
fix(bulk): deleted/missing batch no longer dead-ends Bulk mode (#1216)
## Bug
Bulk mode in prod showed **"Couldn't load batch progress."** and nothing
else. The page remembers the last batch ID in localStorage
(`nomad-bulk-last-batch`) so a returning user can finish its review
queue. That batch's jobs had been deleted, so `GET
/api/bulk/4fbbf0165416` returned 404, and the error state had no way
back to the form.
## Fix
- A **404** forgets the batch (the same path as "Start another batch")
and shows the form.
- Other errors keep the message and add the **Start another batch**
button.
- A transient error no longer replaces an already-loaded batch, and
polling continues.
- The parent's callback is read through a ref, so the 5s polling effect
isn't restarted on every render.
No new strings.
## Testing
- New `frontend/__tests__/bulk-batch-progress.test.tsx` covers the 404 →
back to form path, and other errors → message plus button.
- Full Jest suite: 1769/1769.
Workaround until deployed:
`localStorage.removeItem('nomad-bulk-last-batch')`, then reload.
Code review: local self-review only for this 2-file fix; CodeRabbit CLI
is unavailable (org mismatch), so the GitHub bot will review it.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_019MixBuvzxB1c5AYc3mbDyW
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
25888ca003 |
fix: Stripe made-for-you emails + bulk_submit job creation off the event loop (v0.282.2) (#1212)
## Summary Two event-loop stalls from the 2026-10-22 #alerts triage. Each froze every request on the instance. - **22s stall, 11 requests (15:57 UTC).** `stripe_webhook` (async) → `_handle_made_for_you_order` sent the customer order-confirmation email through sync Postmark, with retries and SMTP fallback, on the loop. It was the NOMAD-1800 order. The confirmation, the admin notification and `_prepare_theme_for_job` (GCS) now run via `asyncio.to_thread`. - **26s stall, 9 requests (15:30 UTC).** `bulk_submit` (async) ran the credit check, default-theme lookup and one sync `create_job` + `update_job` per song on the loop (an album batch). That block is now a sync helper run via `asyncio.to_thread`. Behaviour is unchanged: the same `HTTPException`s propagate from the thread. ## Guard (`test_no_blocking_async_routes.py`) - `BLOCKING_CALLS` now includes `send_made_for_you_order_confirmation` and `send_made_for_you_admin_notification`. - New `test_no_known_blocking_calls_in_awaited_route_helpers` also scans `_handle_made_for_you_order`. It is an awaited helper, not a decorated route, so the route-only scan missed it, including its `_prepare_theme_for_job`, which was already on the denylist. - New `test_bulk_submit_creates_jobs_off_the_loop`. - Both new tests fail on `origin/main` (checked against the old files) and pass here. Out of scope: sync `create_job`/`update_job` still runs directly in about 20 other async routes (audio_search, file_upload, jobs.retry_job, …). Each call is usually fast. It's worth a broader sweep if the watchdog keeps catching them. ## Testing - `make test` backend: 6607 passed, 8 failed. All 8 are in `test_emulator_integration.py`, because another session's Firestore emulator holds :8080 locally (same as noted on #1210). CI's emulator job is authoritative. - Targeted: bulk submit, made-for-you, Stripe webhook and the blocking guard: 114 passed. - The local CodeRabbit CLI failed with "Invalid token", so the bot review stays on. Alert: https://discord.com/channels/1198691101943541770/1265517528608669707/1562858789785309195 Alert: https://discord.com/channels/1198691101943541770/1265517528608669707/1562858791446388862 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_016F41fLoBotUw2wUjGCuAok <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Improvements** * Bulk song submissions and made-for-you order processing are less likely to delay other activity while work is underway. * Existing credit checks and handling of jobs created before credits run out remain unchanged. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v0.282.2 |
||
|
|
bc8b033cb5 |
chore(cloudflare): app-link fingerprint for kjbox.cc (applied) (#1215)
Sets Pulumi config `kjboxcc_android_sha256_fingerprints` to the Android debug keystore SHA-256 used by the Nomad Karaoke app builds (release key gets added in Phase 6). Already applied (`pulumi up`: worker content only). Google Digital Asset Links check: linked=true for kjbox.cc, www.kjbox.cc and sing.nomadkaraoke.com. Config-only change; CodeRabbit not needed. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_013Bkasn6trBadg1Zb39vSVR Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
f6d846234d |
fix: alert noise + loop stalls + kn-data-sync OOM from #alerts triage (v0.282.1) (#1210)
## Summary
Fixes for the open and noise groups from the 2026-10-22 #alerts triage.
**Noise that should not page:**
- `stripe_service.py`: "Invalid webhook signature" now logs WARNING. All
42 hits on 10-21 came from `facebookexternalhit` via Cloudflare with a
malformed header. Every real Stripe delivery since 10-14 got a 200.
(#alerts 1562375740685815953)
- `error_monitor/known_issues.py`: `startup_probe` also matches Cloud
Run's "Default STARTUP **TCP** probe failed" wording.
(1562224513276706933)
- `frontend/lib/crash-reporter.ts`: a `removeChild`/`insertBefore`
NotFoundError is benign **only while the page is machine-translated**,
i.e. `<html>` has `translated-ltr`/`translated-rtl`. In the 10-21 crash,
Chrome had translated `/en/app` into Spanish ("Crea un video karaoke" is
not our string), and the customer's job completed and was downloaded.
(1562541569238638725)
**Event-loop stalls** (continues the WIP freeze item in BACKLOG):
- `review.py` audio-editor routes: `apply_edit`, `get_metadata_from_gcs`
and the GCS `download_file`/`upload_file` calls now go through
`asyncio.to_thread`. They were stalling the loop about 10s per edit (job
330d796c, 10-19). (1561861805695967363)
- `internal.py`: `process_conversion_emails` is imported at module top.
The lazy import inside the hourly cron's `async def` wrote a `.pyc` on
the loop, stalling it up to 15.3s, 4× on 10-20/21. (1562224514933334047)
- `backend/Dockerfile`: `python -m compileall` over `/app/backend` and
`/app/karaoke_gen`, so no lazy import compiles at runtime.
- `test_no_blocking_async_routes.py`: `BLOCKING_CALLS` now includes
`apply_edit`, `get_metadata_from_gcs`, `upload_file` and
`download_file`. Two more routes in `review.py` were caught and wrapped:
the preview ASS upload and the join-audio upload.
**kn-data-sync** (`infrastructure/modules/kn_data_sync.py`):
- Memory goes from 512M to 1Gi. The 10-22 full sync OOMed at 489 MiB on
a retry. 1Gi also gives gen2 more CPU, and the cost is about $0 within
the free tier.
- `attempt_deadline="320s"` on both schedulers. The default 180s gave up
on a run that took 184.8s and succeeded, then retried it, which meant 3
full KN pulls that day. (1562752965629190217, 1562752969395535933)
## Testing
- `make test`: 6603 passed. 8 failed in `test_emulator_integration.py`,
the known local-only failure (another session's stack holds :8080); they
fail the same way on unmodified main, and CI's emulator job is
authoritative.
- Targeted runs: error monitor, audio-edit routes, Stripe/webhook,
conversion emails and the blocking guard all pass (710 tests). Frontend
crash-reporter jest: 17 passed, with new translated / not-translated
cases.
- The local CodeRabbit CLI failed with "Invalid token", so the bot
review is left on for this PR.
## Deploy note
The `infrastructure/` change (kn-data-sync) needs `pulumi up` before
merge, per the repo rule. That needs Andrew's `/gcp-admin` grant.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_015AvGQEptp3Pg9RHhKGZ5KB
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved responsiveness during audio editing and review submissions,
reducing the chance that file and audio processing delays other
requests.
* Reduced false crash reports on pages affected by machine translation.
* Improved recognition of startup probe failures in error monitoring.
* Clarified logging for malformed payment webhook signatures.
* **Reliability**
* Increased resources and allowed more time for scheduled data sync
jobs.
* Reduced delays to API requests during background processing.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Alerts
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562375740685815953
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562224513276706933
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562541569238638725
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1561861805695967363
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562224514933334047
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562752965629190217
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562752969395535933
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.282.1
|
||
|
|
e4dad6e3c5 |
fix(delivery): hold completion email when Dropbox delivery failed (NOMAD-1800) (#1213)
## Summary Job `dd892383` (NOMAD-1800) finished while the business Dropbox was at its 6 TiB quota. Both Dropbox upload passes failed with `insufficient_space`, but the job still went COMPLETE and emailed the customer a delivery email whose link was the literal `[Dropbox URL not available]`. The only #alerts signal was a raw traceback. Alert: https://discord.com/channels/1198691101943541770/1265517528608669707/1562874010939949110 ## Changes - **New `backend/services/delivery_guard.py`** - `completion_email_hold_reason(job)`: if the job's *effective* distribution settings deliver to Dropbox (`dropbox_path` and `brand_prefix`, the same condition the workers use, so private/NonPublished jobs count too) and `state_data.dropbox_link` is missing, the email is **held**. - `hold_completion_email(...)`: records `state_data.completion_email_held = {reason_code, reason, recipient, at}`, logs a WARNING, and sends **one actionable #alerts message per job** (`send_throttled_ops_alert`, keyed per job). The message covers the job, brand code, the masked recipient whose email was held, the reason (`insufficient_space` is classified as "Dropbox is FULL"), the fix steps and the runbook. The Pushbullet push is unchanged. - `check_dropbox_space_usage()`: calls `users_get_space_usage()`, using team allocation for the business account. It posts one #alerts message when usage reaches 95% and records its state in Firestore (`ops_failure_alerts/dropbox-space-usage`), so it stays quiet while usage stays high and re-arms once usage drops below 95%. It runs on the existing hourly `process-stale-reviews` cron, so no infra change is needed. - **`JobManager._schedule_completion_email`**: checks the hold before sending. Made-for-you ownership still transfers to the customer. - **Admin `POST /api/admin/jobs/{id}/send-completion-email`**: returns 409 when the link is missing, unless `allow_missing_dropbox_link: true` is passed. A successful send clears `completion_email_held`. - **`TemplateService.render_job_completion`**: no more `[Dropbox URL not available]` / `[YouTube URL not available]` placeholders. When a link is missing, that whole section is dropped (English default and GCS wording, plus the localized template). Jobs with no Dropbox delivery at all (tenant, no Dropbox configured) still get their email, just without a Dropbox paragraph. The YouTube-queued and private paths are unchanged. - Dropbox upload failure logs (orchestrator and native pass) go from ERROR to WARNING, so one root cause raises one alert (the held-delivery alert) instead of an error-monitor traceback. - Docs: `TROUBLESHOOTING.md` § "Held completion email (Dropbox delivery failed)" with the manual recovery steps, and `API.md` for the new 409 and override. - Version 0.282.0. Not done: item 5 of the handoff (an admin "redistribute Dropbox only" action). The existing admin re-render also re-uploads YouTube, so it isn't a clean fix for public jobs. The manual recovery steps are in TROUBLESHOOTING.md instead. ## Testing - New `backend/tests/test_delivery_guard.py` (31 tests): failure classification, hold/no-hold matrix (no link, link present, no Dropbox, tenant, private defaults, settings error), held marker and alert content, quota threshold, alert-once and re-arm, failed send does not latch, API error never raises. - `test_made_for_you.py`: email held when the link is missing (ownership still transfers, marker recorded, alert sent); email sent when the link is present; email sent for a job with no Dropbox delivery. - `test_admin_email_endpoints.py`: 409 without the link; override sends and clears the marker; fixed link sends and clears the marker; job with no Dropbox delivery sends normally. - `test_template_service.py`: missing links never render placeholders (en, de, unknown GCS template). - Backend unit suite (`backend/tests`, excluding emulator tests): 6637 passed, 7 failed, 25 skipped. All 7 failures are `test_internal_api.py::*requires_auth` / `*requires_admin`, which fail the same way on unmodified `origin/main` locally (they get 200 instead of 401, apparently from the local environment), and this PR doesn't touch them. - The local CodeRabbit CLI failed ("Invalid token"), so this PR is left open for the bot review. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_016F41fLoBotUw2wUjGCuAok Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v0.282.0 |
||
|
|
73fa5d7c01 |
feat(bulk): album cover thumbnails + lossless BEST CHOICE at 10+ seeders (was 50) (#1211)
## Summary
- **Album cover thumbnails** in Bulk → By album. Album rows and the
tracklist header show the release group's front cover from the Cover Art
Archive (`coverartarchive.org/release-group/{mbid}/front-250`).
- Loaded lazily in the browser, never through our backend or the
MusicBrainz API.
- The disc icon stays as a placeholder, and remains when the album has
no cover (404).
- **Lossless torrents with 10+ seeders now count as BEST CHOICE** (tier
1 / auto-pick); the bar was 50.
- Prompted by a real batch (Chase & Status, *No More Idols*): all 12
jobs were parked for manual selection even though the right lossless
album torrent (31 / 15 seeders, filename matching) was the top result.
- Shared `BEST_CHOICE_MIN_SEEDERS` constant in backend
`pick_auto_selection` and frontend `categorizeResult`, plus a test
asserting the two match. The review-fixtures e2e spec uses the constant
too.
- Availability labels (High ≥50 / Medium ≥10 / Low) are unchanged.
- **Also affects** unattended KaraokeHunt conversions, which call
`pick_auto_selection`. The title/filename check still guards them.
- Fake search `recommended` (tier 2) shape drops to 5 seeders; fixture
regenerated.
Follow-up (separate PRs): release-aware bulk album search/download. One
album-level flacfetch search instead of N per-track searches;
flacfetch's search executor is single-threaded, which is why 4/12
searches in that batch timed out at 95s.
## Testing
- Jest: all of `lib/__tests__` + bulk tests pass (355), including 3 new
cover-art tests and threshold boundary tests (9 vs 10).
- Backend: 6604 passed. The 8 failures are in
`test_emulator_integration.py`, which only runs when a local Firestore
emulator is up; it hit a stale emulator from another session. That
covers job creation, which this PR doesn't touch.
- Checked a real CAA cover (200 via archive.org redirect, ~2s cold) and
a missing cover (404).
- Code review: CodeRabbit CLI unavailable (invalid token / org
mismatch), so a local Superpowers review ran instead. Its fixes are in
the last commit.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_019MixBuvzxB1c5AYc3mbDyW
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.281.0
|
||
|
|
5f72e618cc |
feat(cloudflare): serve app-link files on kjbox.cc/.well-known (#1209)
## Summary Mobile Phase 5 "Tonight": the printed venue QR (`HTTP://KJBOX.CC/<code>`) should open the Nomad Karaoke app when installed (Android App Links now; iOS universal links once the team-signed build exists in Phase 6). - New Pulumi-managed Worker `kjbox-cc-well-known` on `kjbox.cc/.well-known/*` and `www.kjbox.cc/.well-known/*` serving `assetlinks.json` (fingerprints from config `kjboxcc_android_sha256_fingerprints`, `[]` until set) and `apple-app-site-association` (team 55B76HN73D, `com.nomadkaraoke.mobile`). - The kjbox.cc `/<code>` redirect rule now skips `/.well-known/`; everything else unchanged. **Already applied** (`pulumi up`, +3 created, ~1 ruleset). Verified: both hosts' `assetlinks.json` 200 JSON with no redirect, AASA 200 `application/json`, `http://kjbox.cc/2121` still 302 → `sing.nomadkaraoke.com/?t=2121`, `/` still 302. ## Testing `infrastructure/cloudflare/tests`: 11 passed (Worker script exercised under node). CodeRabbit CLI: no findings. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_013Bkasn6trBadg1Zb39vSVR --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
032ac11a45 |
fix(youtube): comment bot survives long runs (Broken pipe on first prod run) (v0.280.3) (#1208)
## Summary The first prod run of the YouTube comment bot (#1205/#1207) posted 2 correct replies and then died with `BrokenPipeError`. **Cause:** `commentThreads.list` pages were fetched lazily between slow per-thread work. Background tasks on Cloud Run are CPU-throttled, so Gemini classification of a 100-thread page took about 10 minutes. The shared httplib2 connection idled out and the next page fetch broke. **Fix:** - **Reads:** all comment pages are fetched up front, and reads retry (`num_retries=3`, which reconnects). If a page still fails, the threads already fetched are processed, the run is marked incomplete (high-water not advanced), and #alerts gets a throttled message. - **Replies:** each post uses a freshly built client, built **before** the comment is claimed, so a client failure leaves it unclaimed. Posts are never retried, because a lost response could mean a double reply. Re-running is safe: the 2 posted replies have action docs, and the failed run saved no state, so the one-time backfill start still applies. ## Testing - `backend/tests/test_youtube_comment_bot.py`: reads use `num_retries`, posts don't. - `backend/tests/test_youtube_comment_bot_worker.py`: a later page failure still processes the fetched threads and alerts; a posting-client failure leaves the comment unclaimed. - 55 bot tests pass. Local CodeRabbit: 2 cycles, all findings fixed. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_016E3ibN2EcKxnGQvEf6T4V8 --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v0.280.3 |
||
|
|
055a5d3f73 |
feat(youtube): enable comment auto-reply in prod + one-time backfill (v0.280.2) (#1207)
## Summary Switches on the YouTube comment auto-reply bot from #1205 in prod (Andrew authorised go-live after the 6-month replay sign-off): - `YOUTUBE_COMMENT_AUTOREPLY_ENABLED=true` - `YOUTUBE_COMMENT_AUTOREPLY_START=2026-04-22T00:00:00Z`. This is used on the first run only, as a one-time backfill (Andrew: "yes please, will be a good test of the system working once in prod"). The replay predicts 3 replies: Fightmaster "Nobody's Baby", Bear Ghost "Tiny Hitman", Playa Fly "N God We Trust". Threads already answered by hand are skipped. Already verified in prod before this PR: - The endpoint returns `disabled` while the flag is off. - A `?dry_run=true` run loads secret `youtube-comment-bot-credentials`, builds the channel index and reads comments with no errors. - The Cloud Scheduler job `youtube-comment-autoreply-hourly` and the secret were applied through Pulumi in #1205. ## Testing Config-only change; the code and tests are in #1205. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_016E3ibN2EcKxnGQvEf6T4V8 Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v0.280.2 |
||
|
|
53bed95e76 |
feat(local-stack): kjbox partner secret + fake quick version (v0.280.1) (#1206)
## Summary Local-stack only (no prod impact). For mobile Phase 5 "Tonight": the kjbox local stack points at gen's local stack for make-it. - `scripts/local-stack.sh`: `KJBOX_PARTNER_SECRET` and `DECIDE_GEN_PARTNER_SECRET` default to `local-stack-secret`, external values win. - Fake workers emit a quick version (`state_data.quick_version` ready + `file_urls.quick.video_mp4`) for kjbox-sourced jobs, using prod's `should_build_quick_version` gate and quick-version helpers. `FAKE_WORKERS` is still refused in production. - docs/TESTING.md § Local stack updated. ## Testing `test_kjbox_partner_api.py`, `test_fake_workers.py`, `test_kjbox_resolve_partner.py`, `test_fake_match_judge.py`: all pass. Reviewed by a task reviewer (1 fix round). @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_013Bkasn6trBadg1Zb39vSVR --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v0.280.1 |
||
|
|
6c53f0471a |
feat(youtube): auto-reply to song requests in YouTube comments with requests-board links (v0.280.0) (#1205)
## Summary Backlog Frog #6 (YouTube channel → gen funnel). Andrew's spec: *"we could probably also solve that by building some automation to detect youtube comments asking for songs and auto-reply with a link to the requests board"*. An hourly bot reads new top-level comments on the Nomad Karaoke channel and, when someone asks for a song: - **Song already on the channel:** replies with that video's link. - **Otherwise:** adds the song to the requests board (or finds the existing request) and replies with a link straight to it, e.g. `Added "Tiny Hitman" by Bear Ghost to our requests board 🥳 upvote it here and we'll email you when it's made: <link>`. - **Vague asks** ("more Johnnie please") get a generic board link. - **Non-English comments** get a translated reply, which falls back to English if a URL doesn't survive translation. The reply promotes only the board, not gen (Andrew's call). The goal is turning anonymous YouTubers into board users with an email address. **Andrew signed this off on a 6-month local replay** (`scripts/youtube-comments/replay.py`, read-only): - 265 threads, 33 song requests. - It flagged all 29 requests Andrew had answered by hand, and all 7 vague asks. - It posted no replies to non-requests. - His calls on borderline cases became prompt rules: yes to Playa Fly, no to "Joe Cuervo Karaoke 🎤" on an unrelated video. ## Changes - **Decision core:** `backend/services/youtube_comment_bot.py`, a pure module with injectable ports (Gemini classifier/translator, board, channel index), shared by the replay and the worker. - **Worker:** `backend/workers/youtube_comment_bot_worker.py`, called through `POST /api/internal/youtube-comments/run` (`?dry_run=true`). - Creates a create-only action doc per comment **before** posting, so nothing gets a double reply. - Keeps a high-water mark with a 6h overlap. A Gemini error causes a retry, not a permanent skip. - Per-run cap and quota coordination. - Posts to #noise per reply; failures go to #alerts, throttled. - **Kill switch:** `YOUTUBE_COMMENT_AUTOREPLY_ENABLED` defaults to **false**, and this PR leaves it off. A follow-up PR switches it on after a prod dry run. `YOUTUBE_COMMENT_AUTOREPLY_START` enables a one-time backfill of unanswered requests. - **Requests board:** - New `source="youtube_comment"`; bot-added requests are owned by the placeholder `youtube-comment@nomadkaraoke.com` with no votes. - The **first real up-voter claims the request**, inside the vote transaction. The daily picker skips unclaimed requests, so a free credit never goes to the placeholder. - Frontend `?request=<id>` deep link: highlight, scroll, banner (33 locales). - Signing in from the link uses purpose `requests_board:<id>`, so the magic link returns to the request with `&upvote=1` and the upvote is cast once. - **OAuth:** the bot uses its **own** Desktop OAuth client and secret `youtube-comment-bot-credentials` (`youtube` + `youtube.force-ssl`), because Google refuses `force-ssl` in the device flow. The upload token is untouched. - **Infra (Pulumi, already applied locally):** secret container + Cloud Scheduler `youtube-comment-autoreply-hourly`. - **Description template:** a make-your-own line moves above YouTube's "...more" fold (`utm_campaign=make_cta_top`). The daily backfill re-drains the channel over about 11 days. - **Docs:** `docs/REQUESTS-BOARD.md` § YouTube comment auto-reply, API.md, README.md, plus the plan in `docs/archive/2026-10-22-youtube-comment-autoreply-plan.md`. ## Testing - `backend/tests/test_youtube_comment_bot.py` (33): skips, routing, vague asks, translation fallback, malformed classifier output, API helpers. - `backend/tests/test_youtube_comment_bot_worker.py` (12): kill switch, dry run, idempotency (second run and concurrent claim), go-live start, cap, quota, missing scope, post failure, classifier-error retry. - `backend/tests/emulator/test_song_request_service.py`: bot add/dedupe, claim on first upvote, downvote doesn't claim, picker skips unclaimed, human resubmit claims, `find_published_by_key`. - `backend/tests/test_admin_login_link.py`: `requests_board:<id>` redirect and purpose helper. - `frontend/e2e/regression/requests-board-deep-link.spec.ts` (3): highlight + sign-in purpose, upvote-once on return + param dropped, no toggle-off when already upvoted. - Full `make test`: backend 6584 passed; the only failures are the local-environment `test_emulator_integration` cases, which fail identically on `origin/main` (port 8080 is held by another session's emulator). Frontend Jest: 1760 passed. - Behavioural acceptance: the 6-month replay above. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_016E3ibN2EcKxnGQvEf6T4V8 --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v0.280.0 |
||
|
|
714ac74324 |
fix(resolve): keep Gemini SDK deadline >= 10 s (v0.279.7) — hotfix for #1203 (#1204)
## Problem #1203 (v0.279.6) set the Gemini SDK timeout for free-text resolve to 7 s. The Gemini API rejects that: ``` resolve_free_text attempt 1/2 failed: 400 INVALID_ARGUMENT. 'Manually set deadline 7s is too short. Minimum allowed deadline is 10s.' ``` So since the 21:16 UTC deploy, every uncached "Describe the song" / kjbox resolve answers kind `none` (seen on prod for 6 new descriptions at 21:20). ## Fix - The SDK timeout goes back to `match_judge_timeout_ms` (12 s), with a floor of 10 s (`GEMINI_MIN_DEADLINE_MS`). - The per-attempt cap (now 8 s) is client-side only (`asyncio.wait_for`). Worst case is still about 20 s, and decide waits 25 s. - 4xx errors other than 429 are no longer retried. - Tests: the SDK deadline respects the API minimum, and a client error is not retried. Urgent hotfix: no local CodeRabbit run, so this PR is left open for the bot review. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01G81fc5sBB8GwG5Xhz2UMgc <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit - **Bug Fixes** - Improved AI-powered match judgments by handling timeouts and temporary service failures more reliably. - Prevented unnecessary retries for invalid requests, allowing unavailable results to be returned faster. - Improved timeout handling to reduce premature failures during AI evaluations. - **Chores** - Updated the application version to 0.279.7. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v0.279.7 |
||
|
|
2172306a21 |
fix(resolve): bound free-text resolve so a first-time description answers in time (v0.279.6) (#1203)
## Problem
Prod 2026-10-20: mobile "Describe the song" returned "we couldn't work
out…" on the first ask of every new description; repeating worked.
Evidence (Cloud Logging, `karaoke-backend`):
- `/api/kjbox/catalog/resolve` from decide: 12.1 s, 12.7 s, 17.2 s, 10.6
s.
- 20:28:41 "The song from Titanic": Gemini call at :41.6, then `Failed
to search tracks from karaoke-decide` at :53.7 — the catalog tidy waited
10 s on decide's `/api/catalog/tracks`, which took 37.6 s (cold start;
it routinely takes 35-44 s cold).
- 20:32:44: `resolve_free_text failed: 504 DEADLINE_EXCEEDED` after 10.5
s (one stalled Gemini call; a fresh call answered in ~2 s).
- decide's 12 s upstream timeout fired first → `{}`; gen finished and
cached → second ask worked.
## Fix (`match_judge/free_text.py`)
- Gemini: 7 s per attempt (SDK + `asyncio.wait_for`), one retry on
transient failures; quota/billing errors are not retried. Same
provider/model/config as before.
- Catalog tidy: 4 s cap. On timeout, answer with the AI verdict
(`reason: unavailable`, so not cached) and let the shielded catalog
search finish in the background, warming decide and the catalog proxy's
cache.
- Worst case ~18-20 s; decide now waits 25 s (karaoke-decide#163).
Tests: retry on a stalled call, retry-then-give-up, no retry on quota,
slow catalog tidy answers with the AI verdict while the search finishes
in the background.
CodeRabbit (local): 2 findings. Fixed the minor one (hold the task from
creation). Skipped the suggestion to coalesce concurrent catalog
searches: the route already rate-limits per partner and the catalog
proxy caches results, so it isn't worth the extra complexity here.
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01G81fc5sBB8GwG5Xhz2UMgc
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.279.6
|
||
|
|
83e097fa3b |
feat(local-stack): fake review fixtures and knobs for mobile native review (v0.279.5) (#1202)
## Summary
Local-stack (FAKE_WORKERS=1) support for nomad-mobile Phase 2, which is
the native lyrics and instrumental review. Maestro can now drive every
review state against the real review endpoints. Nothing changes in prod:
all hooks are gated on `fake_workers_enabled()`.
## Changes
- **New per-user `FakeConfig` knobs:**
- `review_fixture` (`default` / `corrections` / `duet` / `rtl` /
`large`). The fixtures are built from the real `CorrectionResult` types:
AI spans with `undo_info`, an untimed word, an evidenced vocal gap with
synced reference lines, duet singers, Hebrew/Arabic lines, and 300
lines.
- `backing_vocals` (`silent` / `audible`)
- `stems_delay_seconds` (stems land after review opens)
- `review_mode` and `auto_approval` (writes
`processing_metadata.auto_approval`, so the confident and skipped-step
states can be reached)
- `review_complete_fail_once`
- **Fake media gets an amplitude envelope**, so waveforms aren't flat.
There is also an audible backing stem variant.
- **`/api/review/{id}/audio` dev-proxy branch** now uses
`_ranged_response` (Range → 206), so seeking works on iOS. The prod
signed-URL path is untouched.
- **`GET /api/test/jobs/{id}/review-result`** returns the submitted
review (selection, `is_duet`, corrected lines) for Maestro assertions.
- `docs/TESTING.md` § Local stack documents the knobs.
## Testing
- `make test`: backend 2184 + 6532 + 163 passed, frontend 1760 passed.
- 29 new tests in `backend/tests/fake_workers/test_review_fixtures.py`.
- Smoke-tested on the local stack: every fixture loads; `with_backing`
analysis; Range 206; 503 then 200 on complete; `review-result`;
`auto_approval` states; late stems.
Reviewed locally (CodeRabbit CLI, 1 finding fixed).
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01G81fc5sBB8GwG5Xhz2UMgc
---------
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
v0.279.5
|
||
|
|
e050cb8b3d |
fix: rescue repetitive-song reference lyrics the relevance filter undercounts (v0.279.4) (#1201)
@coderabbitai ignore ## Why Andrew: job 5c1d24eb loaded no reference lyrics. LRCLIB and Genius both returned the correct song, but the relevance filter scored them 26% and 25%, below its 30% threshold. Anchor coverage credits a repeated phrase only once, so chorus-heavy songs undercount. The transcription was decent, with a few consistent mishearings. ## What `filter_irrelevant_sources(..., transcribed_text=)` now does a second, **additive** check. A source the anchor filter rejects is kept when its word-trigram overlap with the transcription clears four guards: - **≥ 25%** of reference trigrams matched, repeats counted; - **≥ 20** reference trigrams, so tiny references can't score by chance; - **≥ 10** matches; - **≥ 15 distinct** matched trigrams, so a shared refrain alone can't pass. Trigrams must contain three distinct words, so "na na na" and "oh oh oh" don't count. Tokens are NFC-normalized and keep combining marks. Both call sites pass the transcription: the corrector, which also serves the review UI's lyrics search and add-source flow, and the controller's uncorrected path. Without `transcribed_text`, behaviour is unchanged. The rescue **never rejects** a source the anchor filter keeps. The serialized `rejected_sources` metadata is unchanged. ## Calibration (459 recent production jobs; per-job reference txt vs `(Lyrics Uncorrected).txt`) - **Wrong-song baseline:** 491k cross-job pairs, excluding pairs with the same title. - Worst genuine pair: 9.4%; p99.99: 5.5%. - The only cross-job pairs that pass are one job whose audio is another job's song under a wrong title. Their transcriptions overlap 45%, where unrelated songs score under 10%. - A naive n-gram rule would have admitted a wrong song at 41% through shared "na na" filler. Requiring three distinct words removes it. - **Rescued:** 22 sources that are currently rejected, **all the correct song**, scoring 26–88%. Examples: the 5c1d24eb song, Deacon Blue, UB40, Atomic Kitten, Sezen Aksu, Grateful Dead, Etherwood, Sub Focus, Terry Allen. - **Left rejected by the distinct-trigram guard:** one refrain-only drum & bass track (8 distinct matches; it's the right song but has the risk profile). This was a deliberate, cautious choice. - No lyrics text is committed; the tests use invented lyrics. ## Testing - `tests/correction/test_relevance.py`, 17 passed (8 new): - a repetitive correct song is rescued; - no transcription means unchanged behaviour; - a wrong song is still rejected; - filler doesn't count; - a tiny reference isn't rescued; - a source the anchor filter keeps is never rejected; - a shared refrain alone can't rescue; - combining marks are kept. - `tests/unit`: 2184 passed. Running `tests/unit`, `tests/correction` and the backend review tests together gives the same 12 order-dependent failures on untouched main; they pass separately, as CI runs them. - Local CodeRabbit: 2 cycles. Fixed the combining-marks and distinct-trigram findings. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Jw21syo4HizJJjYw2mg2Vp --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v0.279.4 |
||
|
|
b67b273056 |
feat(kjbox): decide partner secret for catalog resolve + fake judge + error monitor for decide search (v0.278.10) (#1190)
## ⚠️ Merge prerequisite (do BEFORE merging) The Secret Manager secret `decide-gen-partner-secret` must have a version before this merges, otherwise the CI deploy's `--set-secrets ...DECIDE_GEN_PARTNER_SECRET=decide-gen-partner-secret:latest` fails. ``` gcloud auth login admin@nomadkaraoke.com cd ~/Projects/nomadkaraoke/karaoke-gen-decide-partner-resolve/infrastructure GOOGLE_OAUTH_ACCESS_TOKEN=$(gcloud auth print-access-token) pulumi up # review: should only add the decide-gen-partner-secret Secret openssl rand -hex 32 | tr -d '\n' | gcloud secrets versions add decide-gen-partner-secret --project=nomadkaraoke --data-file=- ``` No extra IAM grant needed: gen's API runs as `karaoke-backend@`, which already has project-level `secretmanager.secretAccessor`. ## Summary - `POST /api/kjbox/catalog/resolve` now also accepts karaoke-decide via header `X-Partner-Secret` (new setting `decide_partner_secret` / env `DECIDE_GEN_PARTNER_SECRET`). Other kjbox endpoints stay `X-Kjbox-Secret` only. decide gets its own AI-call budget (60/min) so kjbox nights are never starved; the verdict cache is shared. - Fake match judge / resolve for the local E2E stack (`FAKE_WORKERS`): `judge_kind` / `judge_canonical_*` knobs; no Gemini or catalog calls. - Error monitor now watches decide's new `karaoke-decide-search` service and `song-index-build` job. - Pulumi: declares the `decide-gen-partner-secret` Secret (value added manually); CI deploy mounts it. - Version 0.278.10. ## Testing - [x] Changed-module + kjbox/catalog/error-monitor/fake-worker backend tests pass (133 passed). Full `backend/tests` suite exceeds 10 min locally, so left to CI. ## Review - CodeRabbit CLI failed twice (WebSocket closed / connection error), so no local CodeRabbit review completed; I did a manual diff review instead (no issues found). Deliberately NO `@coderabbitai ignore`, so the GitHub bot reviews this PR. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01VsLw9cTTMBab1tiRxCsYrm --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v0.279.3 |
||
|
|
4286d98dbe |
fix: LRCLIB as a registered server-side app; honour Retry-After, retry 5xx (v0.279.2) (#1200)
@coderabbitai ignore ## Why LRCLIB now applies stricter rate limits to unregistered clients (https://lrclib.net/docs#implementation-requirements-rate-limiting), and Cloud Run shares egress IPs. On 10-19, prod jobs got intermittent 503s, and job 80e7ffdf lost its LRCLIB lyrics. LRCLIB is our only free source; it supplied lyrics for 165 of 220 jobs in 30 days. Andrew registered a server-side app. ## What - **`karaoke_gen/lyrics_transcriber/lyrics/lrclib_client.py`** (new): - Sends `Lrclib-App-Id` + `Lrclib-Api-Key` from `LRCLIB_APP_ID` / `LRCLIB_SECRET_KEY`. - Applies LRCLIB's rules: every 429 sets a **process-wide cooldown** of `Retry-After` plus up to 30% jitter. The wait is never shortened, and overlapping 429s keep the latest expiry. Nothing is sent until the cooldown has passed. It waits and retries once only when the wait is ≤30s. - Retries 5xx after 1s/2s, honouring a 5xx `Retry-After`. - A **401 on our credentials** logs a WARNING and drops to anonymous access for the process, so a revoked key degrades instead of losing LRCLIB. - `LRCLIBProvider` (both `get` and `search`) uses it. The `quick_version` renderer uses it with `retry=False`, so the fast path gets the auth and the cooldown but never blocks. - **Infra:** - Pulumi secrets `lrclib-app-id` / `lrclib-api-key`. - Env vars on the `lyrics-transcription-job`. - The backend service deploy's `--set-secrets` (`ci.yml`). The karaoke-backend SA already has project-wide `secretAccessor`. ## Deploy order (do NOT merge before steps 1–2) 1. `pulumi up` to create the two secrets and the job env. Needs `/gcp-admin`. 2. Add the secret versions from Andrew's `.envrc` values. 3. Merge. The CI backend deploy references the secrets. ## Testing - New `tests/unit/test_lrclib_client.py` (14 tests): headers; anonymous use; Retry-After with jitter; a long Retry-After is not shortened and blocks further calls; no-retry mode; a second 429; 5xx backoff; 5xx Retry-After; 401 fallback; overlap merge (mutation-checked); provider routes through the client. - 164 passed across the client, `quick_version` and `search_lyrics` tests. - **Live** with the real app key: Dar Williams - Iowa and Wolfmother - Where Eagles Have Been were found. One 503 was recovered by the retry. - Local CodeRabbit, 3 cycles. I fixed the Retry-After cap, the cooldown after a final 429, the `quick_version` 401 path, the overlap merge and 5xx Retry-After. I skipped one false positive about the tests' fake clock (the retry path sleeps, then sends without re-checking the clock). 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Jw21syo4HizJJjYw2mg2Vp --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v0.279.2 |
||
|
|
653801cd04 |
fix: Musixmatch provider uses the RapidAPI v2 /lyrics?artist=&title= endpoint (v0.279.1) (#1199)
@coderabbitai ignore
## Why
BACKLOG "Musixmatch lyrics provider dead for 30+ days (0/287 requests
succeed)". We're still paying for the RapidAPI listing
(`cavsn/musixmatch-song-lyrics-api`, Pro $14.99/mo, active), but the old
`GET /lyrics/{artist}/{title}/` path now 404s for every song. I checked
the listing's playground (via Andrew's RapidAPI login). The API moved to
`GET /lyrics?artist=&title=` (optional `trackId`, `translate`) and
returns a flat `data` object.
## What
- `MusixmatchProvider._fetch_data_from_source` calls the new endpoint
with query params. A miss comes back as **HTTP 200 with `data.error`**,
or with empty `lyrics`. Both are now treated as not-found and logged at
INFO, so they don't alert.
- `_convert_result_format` reads `trackName` / `artistName` / `trackId`
/ `lyrics`. Raw cache entries in the old `macro_calls` shape still
convert.
## Testing
- New `tests/unit/test_musixmatch_provider.py` uses real response shapes
captured today: endpoint and params, the 200 miss logged with no ERROR,
empty lyrics, no key, v2 conversion, legacy conversion.
- 47 passed across the provider, base provider and `search_lyrics`
tests.
- **Live check** with the production key: Queen - Bohemian Rhapsody 47
lines, Adele - Hello 48 lines. The upstream misses some songs (piri -
dog; Taylor Swift - Shake It Off), so coverage is partial but no longer
zero.
- Local CodeRabbit: 0 findings.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01Jw21syo4HizJJjYw2mg2Vp
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.279.1
|
||
|
|
867617626a |
feat: requests board picks the visitor's language + header language switcher (#1198)
## Summary Follow-up to #1193. `requests.nomadkaraoke.com` always served the English board (the old redirect was also hard-coded to `/en/requests`). - **Worker (`requests-host`, already deployed):** `/` serves `/{locale}/requests/`. The locale comes from the `nk_requests_locale` cookie, else `Accept-Language` (primary-subtag match like gen's `LocaleRedirect`, so `pt-BR` → `pt` and `nn` → `nb`), else `en`. `/{locale}/` (e.g. `/es/`) forces a language. Responses send `Vary: Accept-Language, Cookie`. - **Board header:** gen's `LanguageSwitcher` sits top-right. On the vanity host it sets the cookie and reloads `/`, so the copyable URL stays bare. On gen it behaves as before (`router.replace` to `/{locale}/requests`). - **`LanguageSwitcher`:** new optional `onSelectLocale` prop that replaces the default navigation. ## Testing - Worker: `node --test infrastructure/cloudflare/requests-host/`, 10 pass. This includes a check that the Worker's locale list matches `frontend/i18n/routing.ts`. - New `components/__tests__/LanguageSwitcher.test.tsx` (2 tests); the 703 component tests pass. - Live checks against the deployed Worker: `es`, `ja`, `pt-BR`, `en-US` each get the translated board, and the cookie beats `Accept-Language`. - Mobile (390px) screenshot of a local build: the switcher shows as a compact flag top-right and the dropdown fits the screen. - Prod E2E spec extended with a browser-language check and a switcher round-trip. It will run after deploy. Local CodeRabbit review: no findings. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_016FYpoNVXzJzqKFaowqpwxr --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v0.279.0 |
||
|
|
37b4f5126a |
fix: serialise translation-preview renders and keep them in GCS (OOM SIGKILLs) (v0.278.17) (#1197)
@coderabbitai ignore ## Why BACKLOG "translation-preview renders get the API instance SIGKILLed (OOM)" (#alerts msgs 1557981404242116608, 1560563690053439628). On 10-09 and 10-16, two concurrent `GET /api/themes/translation-preview` renders for **different** languages, each a 4K ffmpeg frame taking about 20s, were followed by `Container terminated on signal 9`. Every in-flight request got a 503, including a real user's `create-from-search` on 10-09. The existing per-key lock only prevented duplicate renders of the *same* language. ## What - `_render_slot = BoundedSemaphore(1)`: one render at a time per instance. - **Persistent cache:** rendered frames go to GCS at `themes/_translation_previews/<theme>/<lang>-<sha256(style_params + sample)[:16]>.jpg`. Each language renders once per theme change rather than once per cold instance, so very few requests ever wait on the slot. A theme edit changes the digest, so stale frames are never served. Cache reads and writes are best-effort: failures log a WARNING and fall back to rendering. They use the bucket directly, so a cache miss isn't logged as a "not found". - **Fixed an existing test blind spot:** this module's autouse `no_sleep` fixture stubs the global `time.sleep`. As a result `test_concurrent_cold_previews_render_once` passed without any threads overlapping, and so did my first version of the new concurrency test. Both now wait on `threading.Event().wait()`. ## Testing - New: a GCS hit serves without rendering; a theme change gets a new key; four languages never render concurrently. Mutation-checked: with `BoundedSemaphore(8)` it fails with `assert 4 == 1`. - `test_lyrics_translation.py`: 33 passed. Local CodeRabbit: 0 findings. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Jw21syo4HizJJjYw2mg2Vp Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v0.278.17 |
||
|
|
acb02e138a |
fix: un-pin audio-separator to 0.47.1 with RoFormer overlap=2 (v0.278.16) (#1196)
@coderabbitai ignore ## Why BACKLOG "audio-separator 0.45–0.47 regression: mel-band RoFormer karaoke models 3–5× slower on L4". The root cause is **python-audio-separator #299 (0.47.0)**, not #298. #299 made RoFormer `overlap` a *window count* that defaults to each model YAML's `num_overlap`. Before that, `overlap` was effectively an 8-second step, which meant almost no overlap. The karaoke models' YAMLs set `num_overlap` 4/8, so a 3-minute track went from 23 forward passes to 87 (aufr33) and 124 (gabox_v2, becruily). That matches the measured 25→75s, 26→149s and 27→157s. 0.45.0 and 0.46.0 still had the old step formula. python-audio-separator #313 (released as **0.47.1**) keeps #299. It adds a per-model forward-pass budget test on every PR and a GPU per-pass timing test (T4: 1.21 s/pass), plus a README note. Andrew chose overlap=2 for gen: smoother chunk seams than 0.44's no-overlap, at about 1.5× the 0.44 compute instead of 4–6×. ## What - `karaoke_gen/audio_processor.py`: `SEPARATOR_MDXC_PARAMS` (overlap=2) goes to all four `Separator(...)` constructions (stage 1, stage 2 karaoke ensemble, single-model paths) **only when audio-separator ≥ 0.47**. On older versions `overlap=2` meant a 2-second step, which is 4× slower, so they keep their defaults. Ensembles load each model through `load_model`, so the param reaches every model in the preset. - `backend/Dockerfile.gpu-base`: `AUDIO_SEPARATOR_VERSION=0.47.1`, with the comment rewritten to explain the root cause. 0.47.1 needs `torch>=2.3` on Linux, so the pre-installed `torch==2.6.0+cu126` (required by the L4 driver) is kept. - The poetry lock stays on 0.44.1. The version-aware params keep CI and CPU images on the old semantics, and there's no torch churn. ## Expected cost per ~3-min track (forward passes) | model | 0.44.2 | 0.47 YAML default | 0.47 + overlap=2 | |---|---|---|---| | karaoke_aufr33 | 23 | 87 | 44 | | karaoke_gabox_v2 | 23 | 124 | 32 | | karaoke_becruily | 23 | 124 | 32 | | instrumental fv7z / resurrection | 23 | 32 / 21 | 32 / 21 | ## Verification plan (before /backlog-finish) After the deploy, time the separation of the daily E2E or a real job on the audio-separation-job logs. I expect roughly 1.5× the 0.44.2 karaoke-stage times (~40–50s per karaoke model rather than 75–157s). If it's worse, revert the pin to 0.44.2. That's one line. ## Testing - `tests/unit/test_audio_processor_local_gpu.py`: version gating (0.47.x/0.48/1.0 → overlap 2; 0.44/0.46 → defaults; unknown → defaults; fresh copy), and both stages' constructors receive `mdxc_params.overlap == 2`. - `make test`: the backend unit suite passes. The only failures are 8 `test_emulator_integration.py` tests, caused by my local environment: another session's emulator setup is on :8080/:4443, and they fail identically on untouched main. CI is green on main. - Local CodeRabbit: 0 findings. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Jw21syo4HizJJjYw2mg2Vp --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v0.278.16 |
||
|
|
92fd647203 |
fix: preview with untimed in-browser edits fills timings or 422s, not a KeyError 500 (v0.278.15) (#1195)
@coderabbitai ignore
## Why
BACKLOG "Lyrics preview 500 `KeyError 'end_time'` on unsaved edits"
(#alerts msg 1559340473439617117). On 10-12 a paying user's review
preview failed 3× (job 16792df7). Both the GCE and the local fallback
failed with `'end_time'`. `JSON.stringify` drops `undefined`, so a word
or line edited in the browser can arrive without its timing keys, and
`CorrectionOperations.update_correction_result_with_data` indexes
`w["end_time"]` / `s["end_time"]`.
## What
- `timing_sanitizer.sanitize_corrections`: a segment with missing or
non-finite bounds now gets them from its words' timings, or else from
the neighbouring lines' gap. It used to be skipped. The existing word
clamp then fills untimed words inside that window.
- New `find_missing_timings()` returns the text of a line that still has
no usable timing.
- `POST /api/review/{id}/preview-video` now sanitizes the payload up
front, as review submit already did. If a line can't be recovered, it
returns **422** `The line "…" has a word with no timing. Sync that line,
then try the preview again.` (en/es/de) instead of a 500.
- The frontend payload is unchanged. The backend now treats a missing
key and a null value the same way, so a frontend change would add
nothing.
## Testing
- `test_timing_sanitizer.py`: bounds from words, missing `end_time`
**key** (the 16792df7 shape), untimed line filled from its neighbours'
gap, unrecoverable line reported. The old "non-finite segment bounds
left untouched" test is replaced, because that is the behaviour this PR
changes.
- `test_routes_review.py::TestPreviewVideoMissingTimings`: a 422 naming
the line, with no render attempted; fillable gaps reach the renderer
filled.
- 131 passed across review routes, sanitizer, and submit-corrections
tests. Local CodeRabbit: 0 findings.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01Jw21syo4HizJJjYw2mg2Vp
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.278.15
|
||
|
|
3bbf397dc4 |
fix: keep auth, tenant lookup and job-log writes off the API event loop (v0.278.14) (#1194)
@coderabbitai ignore ## Why The residual event-loop freezes BACKLOG item. Seven days of `EVENT_LOOP_STALL in progress` stacks (2026-10-12→19, 355 samples), grouped by their outermost app frame, ranked the app-level loop blockers: | Source | Samples | Max | |---|---|---| | in-process screens worker → job-log handler → sync Firestore write per log line | 30 | 2.2s | | `require_auth` → `validate_token_full` (session read on **every** authed request) | 19 | 1.9s | | preview encodes (`_run_preview_encode`/`_encode_preview_in_background`) → job-log write | 15 | 2.4s | | `change_visibility` → sync Dropbox/GCS deletes | 6 | 1.2s | | tenant middleware → `get_tenant_config` (GCS on cache miss) | 5 | 1.2s | | `require_review_auth` → `get_job` | 5 | 2.2s | | `auto_correct_worker._run_suggest` → `get_job` (the only stall since #1192) | 1 | 1.2s | (The render/video in-process stalls stopped with #1192.) ## What - **`backend/services/log_write_offload.py`**: when a job-log write is made on a running event loop in the API process, it goes to one background writer thread. The queue is bounded at 5000 and a drop is logged at WARNING, rate-limited; `atexit` drains the queue. Writes stay inline in threads and Cloud Run Jobs, so the worker code paths behave exactly as before. Logs are read back ordered by their own `timestamp`, so the hand-off doesn't reorder them. - **`asyncio.to_thread`** for `require_auth`/`require_auth_legacy`/`require_review_auth`, the tenant middleware, `rerender_with_current_theme`'s tenant lookup, `auto_correct._run_suggest`, and `change_visibility`'s `get_job`. - **Visibility change:** `_delete_*_outputs` are now plain `def`, and the sync steps run in one worker thread. Moving them off the loop removed the implicit serialisation, so both directions now **claim `visibility_change_in_progress` atomically** before touching anything. They use the new `FirestoreService.try_set_job_state_flag`, a transactional test-and-set. A lost claim returns 409. The regenerate→private continuation passes `holds_claim=True`, because its regenerate already set the flag. The claim is released if the reset fails, and a raising screens trigger rolls back like a `False` one. - **Guard (`test_no_blocking_async_routes.py`):** `BLOCKING_CALLS` gains `validate_token(_full)`, `validate_session`, `get_tenant_config`, `append_log_to_subcollection` and `append_worker_log`. Every `async def` in `api/dependencies.py` and `middleware/` is now checked, not just decorated routes. Mutation-checked: reverting `require_auth` fails the guard. - `docs/LESSONS-LEARNED.md` entry. About 70 direct `get_job`/`update_job` calls in async routes remain. They're single round-trips that rarely show in the watchdog, and the doc says to move them when one does. ## Testing - New: `test_log_write_offload.py` (inline vs offloaded, Cloud Run Job inline, ordering, full-queue drop, Firestore append off the loop thread), claim tests in `test_change_visibility.py` (lost claim, release on failure, chained continuation, trigger exception, route 409), and emulator `test_try_set_job_state_flag.py` (once-only, already set, missing job, one winner among 4 concurrent callers). - Backend: 6478 passed. The 8 failures are `test_emulator_integration.py`, caused by my local environment: another session's emulator setup has an `ssh` forward on :4443 instead of fake-GCS, and they fail identically on untouched main. CI's emulator job covers them and the new emulator test. - Local CodeRabbit: 3 cycles. I fixed the claim race (two findings), trigger-exception rollback, and the guarded claim release. I skipped two minor nitpicks on the pre-existing trigger-failure rollback. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Jw21syo4HizJJjYw2mg2Vp --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v0.278.14 |
||
|
|
b02cce3cc7 |
feat: serve the requests board on requests.nomadkaraoke.com itself (#1193)
## Summary `requests.nomadkaraoke.com` used to 302 to `gen.nomadkaraoke.com/en/requests`, so the short link couldn't be copied out of a mobile address bar (Andrew pastes it into YouTube comment replies). Now the board is served on the vanity host itself. - **New Worker `requests-host`** (`infrastructure/cloudflare/requests-host/`): `/` serves gen's `/en/requests/` (query kept), paths with an extension are proxied from gen, and every other path 302s to the same path on gen. - **Pulumi (already applied):** Worker route `requests.nomadkaraoke.com/*`; old redirect rule removed; DNS comment updated. - **Tenant detection:** `requests` added to the non-tenant subdomains. The browser and Pages Function now share one list (`frontend/lib/edge/nonTenantSubdomains.ts`); the browser copy had drifted and was missing `requests`. Backend list updated to match. - **Board "Make it yourself"** goes to `gen.nomadkaraoke.com/app` when on the vanity host. - **Docs:** runbook rewritten, including why a Pages custom domain didn't work. Attaching the domain bypassed zone redirect rules and served the gen homepage for ~3 min before rollback. ## Testing - `node --test infrastructure/cloudflare/requests-host/`: 4 pass - Jest tenant + new non-tenant list tests: pass. Backend `test_tenant_middleware.py`: 36 pass - `make test`: 6465 pass. 8 fail in `test_emulator_integration.py`, and the same 8 fail on a clean `origin/main` checkout (pre-existing, env-related) - New prod E2E `e2e/production/requests-vanity-host.spec.ts`: board at `/` with no redirect and the API working, other paths 302 to gen. The pass/fail status of the "not a tenant" check (still sends `X-Tenant-ID: requests` until this frontend deploys) is listed below. - Manual browser check on prod: URL stays `requests.nomadkaraoke.com/?utm_source=…`, board renders, no console errors | Prod E2E check | Before this deploy | |---|---| | serves board at `/` | ✅ | | other paths → gen | ✅ | | not treated as tenant | ❌ expected, fixed by this PR | Local CodeRabbit review done (1 minor doc finding, fixed). @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_016FYpoNVXzJzqKFaowqpwxr --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v0.278.13 |
||
|
|
1344a086da |
fix: worker Cloud Run Jobs never dispatch video/render into the API process (v0.278.12) (#1192)
## Problem: a regression from #1191, caught by #alerts triage At 07:03 UTC on 2026-10-19, two new freeze alerts fired: **52s with 29 requests affected** (culprit `firestore_service.append_log_to_subcollection`, during CDG packaging) and **81s with 33 requests** (culprit `dropbox_service.upload_file`). The stacks show `video_worker.generate_video` running **inside karaoke-backend**. Cause: in today's daily E2E run (job 5a5e5181), #1191's park/resume worked as designed (parked 06:55:04, resumed 06:57:50). But the resume runs in `audio-separation-job`, which doesn't set `USE_CLOUD_RUN_JOBS_FOR_VIDEO`. So `trigger_video_worker` fell back to Cloud Tasks, which runs it as a `BackgroundTask` on the API, putting the whole video worker on the API's event loop. The same latent bug already applied to the audio worker's auto-approval → render trigger. ## Fix - `backend/config.py`: inside a Cloud Run Job (`CLOUD_RUN_JOB` is set by the platform), `use_cloud_run_jobs_for_video` and `use_cloud_run_jobs_for_render` default to **true**. An explicit env value still wins. This covers every worker job (audio, lyrics, download, ...) with no infra change, through the normal image deploy. - Error monitor normalizer: `requests_affected=N` → `requests_affected=<N>`, so a stall culprit is one alert pattern instead of one per request count. ## Testing - New `backend/tests/test_config_cloud_run_job_dispatch.py`: inside a job → true; outside → unchanged; explicit env wins. - New normalizer test. - `backend/tests` + `tests/unit/services/error_monitor`: 6871 passed. - Local CodeRabbit review: 0 findings. ## Verify after deploy The next parked-then-resumed job logs `Started Cloud Run Job video-encoding-job` from audio-separation-job, and there are no `EVENT_LOOP_STALL` lines on karaoke-backend with `video_worker` in the stack. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_019pmG2XAgtUguEJWLFpkKHN --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v0.278.12 |
||
|
|
0d0605d7b5 |
fix: final encode waits for stem separation instead of stranding the job; pin audio-separator 0.44.2 (v0.278.11) (#1191)
## Problem The daily E2E Stage 2 failed on 2026-10-17 (run #216) and 2026-10-18 (#217). When a customer finishes review **before** stem separation does, `video-encoding-job` fails its prerequisites (`Missing instrumental: instrumental_clean`) 3× within seconds. The job then sits at `instrumental_selected` ("Starting final encode") forever: it's never marked failed, nothing alerts, and separation finishing doesn't re-trigger the encode. The race became common when the GPU image's unpinned `pip install "audio-separator[gpu]"` picked up **0.47.0** on the 10-17 rebuild. On L4, the mel-band RoFormer **karaoke** models run 3–5× slower than on 0.44.2, so total separation went from ~5 to ~10–16 min: | model (~3-min track) | 0.44.2 | 0.47.0 | |---|---|---| | karaoke_aufr33_viperx | ~25s | 75s | | karaoke_gabox_v2 | ~26s | 149s | | karaoke_becruily | ~27s | 157s | | instrumental fv7z / resurrection | ~50–60s | ~62s | The model YAMLs still say `num_overlap: 8, batch_size: 1`, so this isn't the #299 YAML-defaults change; the cause is somewhere in 0.45–0.47's RoFormer inference changes. That root cause is tracked separately. ## Fix - **Pin** `audio-separator[gpu]==0.44.2` in `backend/Dockerfile.gpu-base` (`ARG AUDIO_SEPARATOR_VERSION`). Changing this file triggers the GPU base image rebuild in CI. - **Video worker parks instead of failing.** When the selected stem is missing *and* separation is still running (`audio_complete` false, valid status), it sets `state_data.video_waiting_for_stems`, puts a user-visible message ("Waiting for audio separation to finish; the final video starts automatically") and exits 0, so Cloud Run doesn't retry. It re-checks after parking, and an atomic Firestore test-and-clear (`claim_job_state_flag`) means exactly one of the video worker and the audio worker proceeds if the stems land mid-park. - **Audio worker resumes it.** After `mark_audio_complete`, `resume_video_if_waiting_for_stems` re-triggers a parked encode. The fake-workers pipeline does the same, so the local stack matches prod. - **Backstop in `recover_stuck_jobs`.** `instrumental_selected` for more than 10 min with the stem present → re-trigger the video worker (covers a lost resume or lost dispatch). More than 60 min with separation never finishing → `mark_job_failed` (`stage=video_waiting_for_stems`), so the job fails loudly. Jobs older than 24h are never revived. (No jobs are in `instrumental_selected` in prod right now.) ## Testing - New `backend/tests/test_video_waits_for_stems.py` (16 tests): pending detection (including that a wrong status never parks), park / continue / claim-race outcomes, and resume (flag claimed / no flag / job moved on). - `test_recover_stuck_jobs_repark.py`: 5 new sweep tests (re-trigger, fresh, still separating, never-ready → fail, >24h ignored). - Full `backend/tests` (excluding emulator): 6455 passed. - Local CodeRabbit review: 0 findings. ## Verify after deploy - The GPU job logs `Separator version 0.44.2`, and karaoke-model separation is back to ~20–30s. - The next daily E2E Stage 2 passes. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_019pmG2XAgtUguEJWLFpkKHN Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v0.278.11 |
||
|
|
70b2f4279a |
feat(alerts): collapse error bursts, suppress link previews, de-noise #alerts (v0.278.9) (#1188)
Andrew's goal: "I'd like to make sure that our alerts channel only has alerts in it that are actionable and important, or are just signal and not noise." This follows #1186 (#noise routing). ## Changes 1. **Burst collapse** (`error_monitor/monitor.py`, `discord.py`, `config.py`). When a run has more than 3 new patterns for one service, including leftover un-alerted ones, the monitor sends ONE #alerts message. It has the count, total occurrences, the top 5 by count, first/last seen, and a `scripts/query-error-patterns.py --service X --status new` pointer. All of those patterns are marked alerted. The grouping is deterministic, since LLM grouping is off in prod. The 10-messages-per-run cap still applies, and unsent patterns are picked up again next run. Evidence: on 2026-10-07/08, one mb-refresh run produced 73 separate alerts. 2. **No link previews.** Every error-monitor post (`_post`) and every `ops_alerts` post now sets `flags: 4` (SUPPRESS_EMBEDS), through a new `suppress_embeds` kwarg on `DiscordNotificationService.post_message`. The YouTube "New upload" post keeps its embed. 3. **Loop watchdog.** `EVENT_LOOP_STALL in progress` is now always WARNING. The `ended` line keeps the existing ERROR rule. In the last two weeks there were 24 "in progress" pattern alerts on top of 15 "ended" ones. I checked `infrastructure/modules/monitoring.py` `EVENT_LOOP_STALL_LOG_FILTER`: it keys on `"EVENT_LOOP_STALL ended"` + `severity>=ERROR`, so this needs no Pulumi change. 4. **Frontend chunk-load errors** (`frontend_ingestion.py`, `firestore_adapter.py`). Reports matching `isChunkLoadError` collapse into one pattern per build with `alert_min_hourly_count=10`. The monitor holds that pattern un-alerted until more than 10 arrive within an hour, which means a broken deploy. Separately, Safari/Firefox stacks now get the message prepended before hashing (a pattern was named `@https://gen.nomadkaraoke.com`). Chrome hashes are unchanged. Existing Safari/Firefox patterns will re-key once, and burst collapse groups them. 5. **known_issues.** Added a service-scoped ignore for `github-runner-manager` "no available instance" aborts during scale-up (seen 09-20, 09-23, 09-27, 10-08). The same message from other services still alerts. 6. **divebar-mirror schedulers.** Added `attempt_deadline="600s"` on both `divebar-mirror-daily` and `divebar-mirror-refresh`; the function timeout is 540s. With the 180s default, Cloud Scheduler retried syncs that were still running, and the overlapping runs hit OOM (4x in 30 days). ## ⚠️ Pulumi `infrastructure/modules/divebar_mirror.py` changed, so **this needs `pulumi up` by Andrew before merge**. Auto-merge is NOT enabled. ## Testing - New and updated unit tests in `tests/unit/services/error_monitor/{test_monitor,test_discord,test_frontend_ingestion,test_firestore_adapter,test_known_issues}.py`, `backend/tests/{test_loop_watchdog,test_discord_service,test_ops_alerts}.py`, and `tests/unit/test_divebar_mirror_scheduler.py` (an AST guard: deadline > function timeout). - `pytest backend/tests --ignore=backend/tests/emulator`: 6421 passed, 36 skipped. - `tests/unit/services/error_monitor`: all pass. - Local CodeRabbit review (`--committed`) completed with 0 findings. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_019pmG2XAgtUguEJWLFpkKHN --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v0.278.10 |
||
|
|
9af5c8224e |
docs(medley): recommend backing_vocals_db -8, not -14 (#1189)
The customer reported that -14 dB made the backing singers inaudible (about 25 dB under the band). -8 dB leaves them about 18 dB under: quiet but present. This updates the MEDLEYS.md spec-table example and adds the guidance. Docs only, so no version bump. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01QV2TQwyC7pwVkGPCk98Bqg Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0e507852af |
fix(alerts): demote expected conditions off ERROR + Payment Link webhook fix (v0.278.9) (#1187)
## Summary Alert hygiene, part of keeping Discord #alerts to actionable signal only. ERROR is page-worthy: the error monitor turns every new ERROR pattern into an #alerts message. This PR moves expected conditions down to INFO/WARNING and fixes the one real bug among them, which was a Payment Link checkout crashing the Stripe webhook. It covers the BACKLOG items "Payment Link checkouts crash the Stripe webhook" and part of "Alert-hygiene bundle". ## Changes 1. **Stripe webhook, non-gen checkouts** (`users.py`, `stripe_service.py`, `stripe_admin_service.py`) - A session that gen didn't create (a Stripe Payment Link: `payment_link` is set, there's no gen metadata, and `customer_email=None`) used to log ERROR "No user email in checkout session". `store_payment` then crashed on `None.lower()`, so the payment never reached `stripe_payments` or revenue. - New `StripeService.is_gen_checkout()`. A non-gen checkout now logs INFO "non-gen checkout (payment link ...)" and grants no credits. It is marked processed for idempotency and is still stored, with `customer_details.email`, `order_type="payment_link"`, `payment_link_id` and the line-item description. - `store_payment` now handles None for metadata, customer_details, total_details and payment-method fields. - **Not backfilled:** session `cs_live_a1ENsua3w6mDH04PPc…` (2026-10-13, a $36 tip, no customer action needed). It could be backfilled through an admin path that calls `store_payment` on the expanded session. GCP writes from agents are restricted, so this PR doesn't do it. 2. **Lyrics providers** (`base_lyrics_provider._log_request_exception`): a 404 means "no lyrics for this track" and now logs INFO. 429 moves to WARNING alongside 5xx, connection errors and timeouts. Auth errors and other 4xx stay ERROR. After 5 404s from one provider in a process, it logs a single WARNING saying the provider may be dead (TODO points at the Musixmatch backlog item). 3. **research_audio** (`audio_search.py`): no more `AWAITING_AUDIO_SELECTION -> SEARCHING_AUDIO -> AWAITING_AUDIO_SELECTION` transitions, which were invalid and logged 2 ERRORs per re-search. The job stays in AWAITING_AUDIO_SELECTION with no status writes, so a crashed request can't strand it. After the search the job is re-read. If a selection landed in the meantime, the fresh results are discarded with a 409 instead of replacing the results that selection indexed into. 4. **audio-download-job on a deleted job** (`audio_download_worker.py`): if `get_job()` returns None at the start or after the download, or a failure happens and a fresh read confirms the doc is gone, the worker logs INFO and returns success, so it exits 0. That means no `fail_job`, no further writes, no worker triggers and no Cloud Run retry. A GCS NotFound on a job that still exists stays an ERROR. Seen with job 6ffe88ad on 2026-10-14. 5. **Loop watchdog**: "Failed to record loop stall (non-fatal)" now logs WARNING with exc_info. Only that line changed, to keep the conflict with feat/alert-noise-monitor trivial. 6. **Job-list queries** (`firestore_service.list_jobs`, `list_jobs_summary`, `list_jobs_by_batch_id`): `query.stream()` now gets `retry=` and `timeout=` of 30s. The client default is 300s for both, which caused the "Error listing jobs summary: Timeout of 300.0s exceeded" errors. Version 0.278.9 (0.278.8 is taken by the still-open #1186). ## Testing - New/updated unit tests: - `test_stripe_webhook_non_gen_checkout.py`: webhook stores the payment, logs INFO, logs nothing at ERROR, and marks the session processed. - `test_stripe_admin_service.py`: Payment Link session and null fields. - `tests/unit/test_base_lyrics_provider.py`: log levels for 404, 429 and 401, plus the dead-provider WARNING. - `test_audio_search_research_attach.py`: no status writes, and a selection made mid-search gets a 409. - `test_audio_download_worker.py`: job deleted at start, mid-download, or with NotFound on write. - `test_loop_watchdog.py`: recorder failure logs WARNING. - `test_firestore_list_query_timeout.py`: bounded retry/timeout on all three list queries. - `pytest backend/tests --ignore=backend/tests/emulator`: 6434 passed. `pytest tests/unit`: 2108 passed. - Local CodeRabbit review: cycle 1 raised 2 findings (non-gen session idempotency, re-search race), both fixed. Cycle 2 was clean. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_019pmG2XAgtUguEJWLFpkKHN --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v0.278.9 |
||
|
|
a6d4163d2e |
feat(alerts): route non-signal Discord posts to #noise (v0.278.8) (#1186)
## Summary Andrew asked for #alerts to carry only real problems. Routine and informational posts now go to the new **#noise** channel. The GitHub org PR webhook was already moved to #noise separately. | Message | Before | After | |---|---|---| | Error monitor: New Error Pattern / Spike / Incident | #alerts | #alerts | | Error monitor: ✅ Patterns Auto-Resolved | #alerts | **#noise** | | Error monitor: 📊 Daily Error Digest | #alerts | **#noise** | | Nightly Backup Report (clean run) | #alerts | **#noise** | | Nightly backup failed/partial (title is now "🚨 Nightly Backup FAILED") | #alerts | #alerts | | ✅ E2E Daily Test PASSED | #alerts | **#noise** | | 🩹 E2E self-healed Stripe (auto-merge PR opened) | #alerts | **#noise** | | 🩹 self-heal PR needs CI approval click / could not open | #alerts | #alerts | | E2E / Tenant E2E failures, DR freshness stale, Job failed, Publish completeness SHADOW, Gemini unavailable, OAuth credential alerts, quarterly DR drill reminder | #alerts | #alerts (unchanged) | ## How the noise webhook is resolved - **Backend / error monitor and the backup function** read it at runtime: env `DISCORD_NOISE_WEBHOOK_URL` first, then the Secret Manager secret `discord-noise-webhook` (the name can be overridden with `DISCORD_NOISE_WEBHOOK_SECRET_NAME`). No Pulumi change is needed, because both service accounts already have project-wide `secretAccessor`. - **GitHub Actions** uses `secrets.DISCORD_NOISE_WEBHOOK_URL || secrets.DISCORD_ALERT_WEBHOOK_URL`. - **Nothing is silently dropped.** If the noise webhook can't be resolved, or a post to it fails, the message goes to #alerts instead. In the workflow this uses `curl -fsS` with a retry on the alert webhook. - The error monitor now rate-limits each channel separately (10 per run each), so noise can't use up the alert budget during an incident. ## Deploy notes - **Backup Cloud Function:** CI doesn't deploy its source. Run `infrastructure/functions/backup_to_aws/deploy.sh` after merge. - **Error monitor job:** runs `karaoke-backend:latest`, so it picks up the change once CI pushes the image. - **No `pulumi up` needed.** ## Testing - `tests/unit/services/error_monitor`: 376 passed. New tests cover noise vs alert routing, fallback when the noise webhook is missing or fails, separate rate limits, and secret resolution. - `backend/tests` (excluding emulator): 6418 passed, 36 skipped. - `infrastructure/functions/backup_to_aws`: 60 passed, including the new `test_discord_alert.py` and routing tests in `test_main.py`. These were run in a scratch venv because `functions_framework` isn't in the conda env, which was already the case before this change. - `actionlint` on `e2e-daily.yml`: no new findings. The only findings are info-level shellcheck notes that were already there. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_019pmG2XAgtUguEJWLFpkKHN Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v0.278.8 |
||
|
|
290f927175 |
fix(backend): move remaining event-loop blockers off the loop (v0.278.7) (#1185)
## Summary
Found while triaging Discord #alerts and 14 days of freeze telemetry
(2026-10-18). The event-loop watchdog still caught real freezes that hit
users:
- **151s freeze, 32 requests affected (2026-10-17 20:10 UTC).** `POST
/api/review/{id}/custom-lyrics/generate` called
`CustomLyricsService.generate` (a sync Gemini call) on the loop. Every
user on the instance saw "server unavailable". It now runs via
`asyncio.to_thread`.
- **14.9s freeze, 7 requests (2026-10-16).** The first `from
google.cloud import run_v2` in
`worker_service._trigger_worker_cloud_run_job` ran on the loop.
`google.cloud.run_v2` and `httpcore` (imported lazily by httpx) are now
in the startup `_LAZY_ROUTE_IMPORTS`, so they are imported off-loop
before readiness.
- **GCS on the loop in async routes.** `_prepare_theme_for_job` ran on
the loop in `upload_and_create_job`, `create_job_with_upload_urls`,
`create_job_from_url` and audio-search `search_audio`;
`submit_audio_edit` also uploaded `edit_log.json` on the loop. These now
go through `to_thread`.
- **Guard.** `upload_json`, `_prepare_theme_for_job` and `generate` are
added to the `BLOCKING_CALLS` denylist in
`test_no_blocking_async_routes.py`. The denylist caught the two extra
routes above.
## Testing
- `pytest backend/tests --ignore=backend/tests/emulator`: 6418 passed,
36 skipped.
- `test_audio_search` source check updated to accept the
`to_thread(_prepare_theme_for_job, …)` form.
- Local CodeRabbit review: 0 findings.
## Verify after deploy
No new `EVENT_LOOP_STALL ended … requests_affected>0` lines with
culprits `_call_gemini`, `run_v2` or `upload_json`.
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_019pmG2XAgtUguEJWLFpkKHN
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.278.7
|
||
|
|
9d6aa3df7d |
feat(medley): per-section backing_vocals_db for the karaoke track (v0.278.6) (#1184)
## Summary The Fiverr medley customer said the backing singers in two songs were too loud to sing over in tune, and asked for them as a faint background. - **New option:** a per-section `backing_vocals_db` in the medley spec. For the karaoke track only, the section is rebuilt from the clean instrumental plus the backing-vocal stem at that level, e.g. `-14` for a faint background. The rest of the karaoke track, the clean instrumental and the guide are unchanged. - **Cached mix:** the mix is cached per job/level, written to a `.partial.flac` and renamed, so an interrupted run never leaves a truncated mix to be reused. If the job has no backing stem, it stops with a clear error. - **Docs:** `docs/MEDLEYS.md` spec table. ## Testing - `test_make_medley.py`: parsing, the ffmpeg graph, only the karaoke variant changing, the atomic write and the cache. 20 + chord-chart tests pass. - Real data: the customer's medley with `-14` on Remember When and On the Other Hand. - Only those two sections change; the others differ by ≤ -43 dB, which is crossfade overlap. - Overall section loudness is unchanged. - A local CodeRabbit review found 1 issue (stale or partial cache). It is fixed. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v0.278.6 |
||
|
|
03ee693f28 |
feat(medley): customer chord charts + static-gain master (v0.278.5) (#1183)
## Summary Two follow-ups on the medley tooling, both from the Fiverr medley customer's feedback. - **Fixes the robotic artifact at the start of the karaoke audio.** The final single-pass `loudnorm` ran in dynamic mode. On a song that opens a cappella (HSLHT sings from 0.4s, band at 3s), the instrumental's first 3s is near-silent separation residue at -70 dB, and `loudnorm` boosted it by about 50 dB into a loud warble. The master is now one static gain to -14 LUFS plus a 4×-oversampled limiter at -1 dB (`master_filter`). After the fix, the first 3s measure -61 dB (were -15 dB). Sync is sample-identical and true peak is -0.9 dB. - **Imports the customer's own chord chart.** `scripts/medley/chord_chart.py` plus a new `make_medley.py chart` step read a chords-over-lyrics chart (.txt or .docx in Courier, capo shapes, `Capo N` / `Key change Capo 1` lines). They align it word-by-word to the reviewed lyrics, so chords come out timed on the words the chart puts them on. Detected chords fill intros, breaks and spoken verses, transposed into the capo shapes. - **Shows the capo in the chord box.** New `ChordChange.capo` field, shown as "capo N" under the chord. - **Docs:** `docs/MEDLEYS.md` gets a new section on importing a customer chart, and the mastering guidance now says never to use single-pass `loudnorm`. `docs/README.md` has an entry. ## Testing - Unit tests: - New: `tests/unit/test_chord_chart.py` (9). - Added: capo tests in `test_guitar_chords.py`, plus master and capo tests in `test_make_medley.py`. - Full `tests/unit` suite: 2100 passed, 65 skipped (run before the last CodeRabbit fix; only the 9 chord-chart tests were re-run after it). - Real data: the customer's 4-song chart against the medley. The lyrics-with-chords output matched the chart line by line, and 4K frames rendered with capo labels. - A local CodeRabbit review found 1 issue (two chord groups after the same word could come out in the wrong order). It is fixed and has a test. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01QV2TQwyC7pwVkGPCk98Bqg --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v0.278.5 |
||
|
|
9a44a30368 |
fix(review): idempotent correction-data review transition (v0.278.4) (#1182)
## Summary The same race as #1181, on the other door into review. `GET /api/review/{job_id}/correction-data` moves `AWAITING_REVIEW` to `IN_REVIEW`. When two loads arrive concurrently, both see `AWAITING_REVIEW`, so the second transition raises `in_review -> in_review`. That returns a 500 and the review page shows "Failed to load lyrics". The iOS in-app browser hits this every time; mobile E2E caught it. **Fix:** a shared `ensure_in_review()` helper in `backend/services/review_state.py`, now used by both `correction-data` and `start-review`. - An already-`IN_REVIEW` job does not transition again. - A lost race re-reads the job and carries on. - Any other invalid transition still errors. - `record_review_started` runs only for the winning transition. Version 0.278.4. ## Testing - 4 new `TestCorrectionDataReviewTransitionIdempotent` cases. - 493 review and correction backend tests pass. - Local CodeRabbit found nothing. - Verified against the local stack via the mobile Maestro review-bridge flow. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_011yLH4kWXMqjeKfZUG3ZiWZ Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v0.278.4 |
||
|
|
072f85cad9 |
fix(review): idempotent start-review (no more in_review -> in_review error) (v0.278.3) (#1181)
## Summary
Opening a lyrics review could fail with `Invalid state transition
in_review -> in_review`.
`POST /api/jobs/{job_id}/start-review` always called
`transition_to_state(IN_REVIEW)` and never checked for `in_review`. So
any second open failed: a duplicate request, a double-mount, a second
tab, or the mobile app's in-app browser opening the review again. Mobile
E2E found this, and a screenshot captured gen's error page.
The endpoint is now idempotent:
- If the job is already `in_review`, it returns `{"status": "success",
"job_status": "in_review"}`.
- If the transition raises because a concurrent call won the race, it
re-reads the job. If the job is now `in_review`, it returns the same
success.
- Any other invalid transition still raises.
- `record_review_started` runs only for the winning `awaiting_review →
in_review` transition.
Version 0.278.3.
## Testing
- 4 new `TestStartReview` cases in `backend/tests/test_jobs_api.py`.
- The review-related backend tests pass: 452.
- Local CodeRabbit found nothing.
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_011yLH4kWXMqjeKfZUG3ZiWZ
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.278.3
|
||
|
|
2838d2dd1d |
fix(credits): keep credits_charged through job creation; one helper for confirm/refund/select (v0.278.2) (#1180)
## Summary
Fixes `credits_charged` being wiped from `state_data`, which made
duration confirm, refunds and the web cost dialog disagree. Mobile E2E
against the local stack found it: a 15-minute track showed "1 credit in
total (1 more)", and confirming the true total was rejected with 409.
**Root cause:** `create_job` debits the credit and records
`state_data.credits_charged`. But create-from-search, audio-search
create, made-for-you orders (which also lost `payment_bypassed`) and the
admin source override then replaced the whole `state_data` map, deleting
the field. Readers filled the gap with different defaults:
- reconcile: 1
- confirm-duration: 0
- 48h expiry: 0
**Fix**
- **Writers:** per-key `state_data.<key>` updates instead of whole-map
writes. Retry-from-scratch keeps the billing keys. Reconcile records
`credits_charged` when it pauses a job.
- **Readers:** one helper, `backend/services/job_credits.py`
(`get_credits_charged`, `is_payment_bypassed`). Every reader uses it:
reconcile, confirm-duration, `/select`, the bulk worker,
`_refund_credit_for_job` and the 48h expiry.
- For legacy jobs that lack the field, it returns what `create_job`
actually debited: 0 for bypassed, admin, tenant or no-email jobs,
otherwise 1.
- **`/select` and the bulk worker** never bill payment-bypassed jobs
(made-for-you, tenant, admin-created). This includes legacy ones whose
flag was wiped.
- **API responses:** `GET /api/jobs/{id}` and the job lists return the
resolved `credits_charged` when the stored field is missing. This is
response-only and never persisted, so the web and mobile clients
acknowledge exactly `credits_charged + pending`.
- **Confirm-duration** records the full total, so cancelling after a
confirm refunds it all. The 48h expiry refunds once:
`cancel_job(refund=False)` stops a second refund.
- **Admin source override** clears the old source's duration, selection
and confirm markers and keeps `credits_charged`. The next `/select` then
charges only the difference.
- **Web cost hook:** uses the server's resolved figure.
- Version 0.278.2.
## Testing
- New `test_credits_charged_consistency.py` covers:
- reconcile, then confirm with total 2, then cancel: 2 refunded, for new
and legacy jobs
- 409 on the old total
- refund amounts unchanged for jobs that have the field
- `/select` and the bulk worker never deducting on bypassed jobs
- the override followed by `/select` charging only the difference
- Stale-expiry legacy tests, and an emulator test for the dotted-path
merge (runs in CI).
- Backend unit tests: 6411 passed. Emulator integration tests run in CI;
locally they collided with a running local stack. Frontend jest: 1754
passed.
- An independent review found one Critical issue (`/select` billing
bypassed jobs) and one Important issue (web/server default mismatch).
Both are fixed and re-reviewed clean. Local CodeRabbit raised one minor,
which is fixed.
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_011yLH4kWXMqjeKfZUG3ZiWZ
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.278.2
|
||
|
|
1616bc997b |
fix(api): sign final MP4s in GET /api/jobs/{id} download_urls (v0.278.1) (#1179)
## Summary
`GET /api/jobs/{id}` has always returned an empty `download_urls` for
real jobs. `JobManager.get_output_urls` signs only `job.output_files`,
and current workers never populate that field. Final videos live in
`file_urls.finals`.
This change also signs each `file_urls.finals` entry whose key ends in
`_mp4` (for example `lossy_720p_mp4`):
- URLs expire after 120 minutes.
- When a key exists in both, the `output_files` entry wins.
- Entries that aren't strings are skipped.
- After a real signing failure, it stops signing further entries, so a
stalled signer can't add a timeout per MP4.
**Why:** the mobile app (nomad-mobile Phase 1) streams the 720p MP4 in
`expo-video` and shows download progress. Both need a URL that supports
HTTP Range requests. The authenticated `/download/` proxy doesn't
support them.
Signing still goes through the bounded executor in `storage_service.py`
with its 12s timeout. The route is a sync `def`, so the event loop is
never blocked.
Version 0.278.1.
## Testing
- 6 new `TestGetOutputUrls` cases in
`backend/tests/test_job_manager.py`. That file passes: 58 tests.
- Local CodeRabbit raised one finding (the type guard), which is fixed.
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_011yLH4kWXMqjeKfZUG3ZiWZ
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.278.1
|
||
|
|
695b8ddf84 |
feat: mobile app backend — handoff token, app checkout return, Google/Apple sign-in (v0.278.0) (#1178)
## Summary
Backend support for the Nomad Karaoke mobile app (nomad-mobile Phase 1,
"Make it").
- **`POST /api/users/auth/handoff-token`**: the app opens a job's web
review already signed in. The token is a single-use, 120s
`job_review:<id>` magic link, returned as
`{FRONTEND_URL}/{locale}/auth/verify?token=…&handoff=1`.
- The verify page skips the "Complete Sign-In" tap only when the URL has
`handoff=1` **and** the server confirms the token was minted as a
handoff (`link-status` → `handoff: true`). Emailed links keep the
scanner guard.
- The web locale is validated against the 33 web locales.
- **Checkout `return_to: "app"`**: Stripe success and cancel go to `GET
/api/users/credits/app-return`, which 302s to
`nomadkaraoke://checkout?status=success|cancelled`.
- The client never supplies a URL, so this adds no open redirect.
- `session_id` is never forwarded.
- **`POST /api/users/auth/google` and `/auth/apple`**: ID-token sign-in
(PyJWT + JWKS: RS256, issuer, audience, expiry, `email_verified`). It
returns the same `VerifyMagicLinkResponse` as email-code sign-in,
through a shared `finish_verified_sign_in`.
- Email-code, Google and Apple all share the same abuse checks
(`_sign_in_checks`).
- A JWKS outage returns 503 `identity_provider_unavailable`.
- Google audiences default to the new mobile web and iOS OAuth clients,
which are recorded in ARCHITECTURE.md.
- **Job summary** now includes `credit_refunded`.
- **Local stack:**
- fake-config `search_confidence` (perfect / recommended / unsure), with
a drift guard tied to the frontend `getSearchConfidence`
- `youtube_url_enabled`
- `fake:<email>` social tokens, accepted only in fake mode
- PyJWT is now a direct dependency (`pyjwt[crypto]`). Before this it was
only present transitively.
- Version 0.278.0.
## Testing
- `make test` is green:
- CLI: 2088 passed
- backend unit: 6360 passed
- emulator: 156 passed, plus 4 handoff lifecycle tests
- frontend jest: 1751 passed
- New suites:
- `test_handoff_token.py`
- `emulator/test_handoff_token_lifecycle.py`
- `test_checkout_app_return.py`
- `test_social_signin.py`
- `test_magic_link_status.py`
- `fake-search-tiers.test.ts`
- verify-page tests
- Reviews:
- An independent code review was approved, and its hardening suggestions
were applied.
- A local CodeRabbit review found nothing.
Plan: `docs/archive/2026-10-17-mobile-app-auth-handoff-plan.md`. Spec:
nomad-mobile
`docs/superpowers/specs/2026-10-17-mobile-phase1-make-it-design.md`.
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_011yLH4kWXMqjeKfZUG3ZiWZ
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.278.0
|
||
|
|
f8e9a8b3b6 |
feat(bulk): guided Artist → Album → Tracks steps in bulk album mode (v0.277.1) (#1177)
## Summary
Bulk album mode was hard to figure out — the one-line text hint was easy
to skip and it wasn't obvious the list was albums. This adds visual
cues:
- **Step strip** `1. Artist › 2. Album › 3. Tracks` with icons; current
step highlighted, completed steps show the pick and are clickable to
jump back (replaces back buttons)
- **Per-step headings** with icons ("Choose an album — Albums by Britney
Spears")
- **Artist/album results as clickable cards** (icon + chevron, hover
border)
- **Tracklist**: Select all / none + "N of M tracks selected"
- Icons on **By album / By text** tabs
- Album mode **hides batch settings + submit until a tracklist is
loaded** (no more "Create 0 karaoke videos" on step 1)
- Leaving the tracklist clears its rows; stale in-flight lookups are
dropped (CodeRabbit finding) so a previous album can't be submitted
Cover art deliberately not used (would hit MetaBrainz's Cover Art
Archive live).
## Testing
- New `frontend/__tests__/bulk-album-mode.test.tsx` (7 tests: step
navigation, select all/none, rows cleared on back, stale-response drop,
settings/submit gating)
- Full frontend Jest suite: 142 suites / 1741 tests passed
- Visually verified locally against prod API (desktop + 390px)
- 33 locales translated
Local CodeRabbit review completed; 1 finding fixed.
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01BrgLoS3bpnku9n21g6YGgw
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.277.1
|
||
|
|
7cf5beede8 |
feat: guitar chords in karaoke videos + ChordMini ensemble chord detection (v0.277.0) (#1175)
## Summary Adds optional guitar chords to karaoke videos, first used for a Fiverr medley customer who wants chords on their video. - **Rendering (gen library):** `OutputGenerator.generate_outputs(chords=[ChordChange(start, end, label)])`. - Each chord change during a sung line is drawn above the word it lands on, in a row reserved above every line (the same slot mechanism as translated lyrics). - Chords light up when played, using the same `\k` highlight as lyrics: gold while upcoming (theme `chord_color`), sung-lyric colour once played. Quick changes on one word never overlap. - A top-right "now / next" chord box runs for the whole song, covering intros and instrumental breaks. - With no chords supplied, output is unchanged. A test asserts the ASS is byte-identical. - **Detection (`scripts/medley/detect_chords.py`):** - Ensembles both ChordMini checkpoints (BTC + 2E1D, MIT; about 80% majmin/MIREX, the best open weights we found) over the full mix, gen's clean instrumental, and an htdemucs_6s harmonic mix (guitar + piano + bass + other). - Weighted per-frame vote, then merges short blips, snaps changes to beats and spells chords for the key (Ab/Db/Eb7, F#m, borrowed bIII/bVI/bVII as flats). - **Medley script:** new `chords` step plus a `"chords": true` spec option. Chords are clipped per section and hand over mid-crossfade, then shifted for the countdown. - **Docs:** `docs/MEDLEYS.md` § Guitar chords (setup, method, accuracy caveats, theme keys) and a docs/README.md entry. There's no job or UI option yet. The medley pipeline uses this for now; a product version would need a chords toggle plus detection in a worker. ## Testing - New: `test_guitar_chords.py` (12: chord timeline helpers, word placement measured against the font, layout geometry, highlight timing, now/next box, theme colour, no-chords path byte-identical), `test_detect_chords.py` (7), and 2 medley chord-shifting tests. - `tests/unit`: 2087 passed, 65 skipped. - End to end on the 5 real medley songs: models/inputs agree on 91–99% of major/minor labels, and the output matches the expected I–IV–V progressions. Rendered 1080p and 4K medley videos with 146 chord changes; spot-checked frames (verse, instrumental screen, chorus). - Local CodeRabbit: 1 finding (doc section reference), fixed. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01QV2TQwyC7pwVkGPCk98Bqg --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
8ec789a9aa |
fix(bulk): album lookup reads our MusicBrainz mirror, never the live API (v0.277.0) (#1176)
## Summary
Bulk → By album → "Britney Spears" showed **"Lookup failed. Please try
again."** `/api/bulk/album/artists` returned 502 because every attempt
to `musicbrainz.org/ws/2` from Cloud Run failed with
`httpx.ConnectError('')`. The same request from a home IP returns 200.
It last worked on 9/29. Most likely MusicBrainz is refusing our shared
GCP egress.
Per Andrew, we should only query our own infra, with no live-API
fallback. `musicbrainz_service` now reads **only our BigQuery
MusicBrainz mirror** (`karaoke_decide.mb_*`, which karaoke-decide's
`mb-refresh` rebuilds weekly from the official dumps).
## Changes
- **`backend/services/musicbrainz_service.py`**
- The HTTP client, throttle and retries for musicbrainz.org are removed.
Queries go through the BigQuery REST `jobs.query` endpoint (httpx + ADC;
no new dependency).
- Queries are parameterized, capped with `maximumBytesBilled` = 2 GiB,
and cached in memory for 10 min.
- Lookups by table:
- artists: `mb_artists_normalized`. Ranking is exact match (including
"The "+q), then prefix, then substring, then popularity.
- albums: `mb_artist_credit_artists` → `mb_release_groups`.
- editions: `mb_releases`.
- tracks: `mb_release_tracks`, the new table in karaoke-decide#158,
clustered by release group + release.
- Measured against prod data: about 0.6–1 s per query, MB to 0.4 GB
billed.
- Canonical-pressing selection now ranks an edition by its best
release-event country.
- **Route:** passes `release_group_mbid` through on edition switches. An
unknown release returns **404** instead of 502.
- **Frontend:** `switchEdition` sends the album's `release_group_mbid`
so the lookup stays cluster-pruned. Older clients still work through a
one-off lookup that scans about 0.2 GB.
- **Infra:** the `karaoke-backend` SA gets project-level
`bigquery.dataViewer` + `bigquery.jobUser`. decide's Pulumi owns the
dataset ACL, so a dataset-scoped grant would fight it.
- **Docs:** `docs/API.md` and
`docs/archive/2026-10-16-bulk-album-musicbrainz-mirror.md`.
## Rollout (order matters)
1. Merge karaoke-decide#158, then build `mb_release_tracks`: by hand
(SQL in the archive doc, about $0.08) or at Sunday's refresh.
2. Run `pulumi up` for the IAM change. The agent's credentials are
read-only, so this needs Andrew.
3. Merge this PR.
## Testing
- `make test`: all passed (backend + 1735 frontend tests).
- New and rewritten unit tests cover normalization, BigQuery row
decoding, the REST request shape (no musicbrainz.org, billing cap,
params, cache), the HTTP/timeout errors, every lookup, the release-group
lookup path, and the 404.
- Ran the service live against the prod mirror: Britney Spears → 39
albums → …Baby One More Time (11 tracks, 6 variants); also checked
"beatles", "Beyoncé", "ac/dc", "Sigur Rós".
- CodeRabbit CLI: 1 minor finding (404 vs 502), fixed.
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01C1sMXk2MaxBLbep5jgiFab
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.277.0
|
||
|
|
85390fd12b |
docs: medley-making guide + spec-driven make_medley script (#1174)
## Summary Documents the multi-song medley process used for the first classic-country medley, and turns it into a reusable script so the next one only needs a spec file. - `docs/MEDLEYS.md`: the full process. One gen job per original song, then identical cuts on every audio variant, and the reviewed lyrics shifted onto the medley timeline and rendered with gen's own `OutputGenerator`. Also covers how to choose cut points (crossfades, pot-out/pot-in, keys/tempos, checking the brief against the recording) and gotchas. - `scripts/medley/make_medley.py`: `fetch` (Firestore + gcloud CLI) → `audio` (per-section LUFS gain, crossfade/gap/pot fades, `loudnorm` to -14 LUFS / -1 dBTP, three variants) → `video` (shifted reviewed lyrics + countdown + themed title/end screens + karaoke and guide MP4s). - `scripts/medley/example-spec.json`: the country medley as a worked example. - Linked from `CLAUDE.md` Quick Reference and `docs/README.md`. Ops tooling only. No backend or frontend change, so no version bump. ## Testing - `tests/unit/test_make_medley.py`: 12 unit tests covering spec validation, the timeline offset maths, lyric-segment shifting (including the same song used twice) and the ffmpeg filtergraph. - End-to-end run against the real prod jobs: `fetch` → `audio` reproduces the hand-built v1 medley **bit-for-bit** (max sample diff 0.0 on both the instrumental and guide WAVs). `video --resolution 360p` renders both MP4s and the LRC. - Local CodeRabbit review: 1 finding (unchecked ffmpeg exit code in `measure_lufs`), fixed. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01QV2TQwyC7pwVkGPCk98Bqg --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
bb55efd3c2 |
fix(cloudflare): pin email catch-all name to stop refresh drift (#1173)
Every `pulumi preview --refresh` on the nomad-cloudflare stack planned a no-op update on the 3 `EmailRoutingCatchAll` resources. The live rules matched the code, but the provider reads back `name: ""`. Setting `name=""` explicitly fixes it (same pattern as `support_subaddress=False`). Already applied locally. `pulumi preview --refresh` now reports **132 unchanged** (zero drift). 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01QuVxnKWqXrdn3ux2VWLxS8 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Updated the configuration for catch-all email routing to specify an empty name. This is an infrastructure-level change; no changes to end-user email features or behavior are described. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
6b7e057e50 |
feat(cloudflare): Google login for Cloudflare Access (#1172)
Adds a Google identity provider to the Nomad Karaoke Zero Trust org, so KJ Controller / FiverrBot login pages offer **Sign in with Google** alongside One-time PIN and Cloudflare. - OAuth web client `Cloudflare Access (nomadkaraoke.cloudflareaccess.com)` created in GCP project `nomadkaraoke` via the console (no API exists for web OAuth clients). Redirect URI: `https://nomadkaraoke.cloudflareaccess.com/cdn-cgi/access/callback`. - Client ID in Pulumi config `googleOauthClientId`, secret encrypted as `googleOauthClientSecret`. - **Already applied locally** (`pulumi up`: 1 created, 131 unchanged), so CI's run will be a no-op. Verified a full Google sign-in to kjbox.nomadkaraoke.com. - Access policies are unchanged; they still allow only the listed emails/domains. Infra-only change (no app code, no version bump). No local CodeRabbit run, so the bot can review this one. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01QuVxnKWqXrdn3ux2VWLxS8 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Google sign-in is now available as an identity provider for Cloudflare Zero Trust access, alongside the existing one-time PIN option. * Production access now supports the Google OAuth credentials needed to enable this sign-in option. Users can choose Google authentication where it is configured, while existing one-time PIN access remains available. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
8a16e33a34 |
fix(review): show duet singer chips in Waveforms view (v0.276.3) (#1171)
## Summary In duet mode, the Simple/Advanced lyrics views show a per-line singer chip (● 1 / ● 2 / Both) to see and toggle which singer each line belongs to. The Waveforms view was missing it, because `WaveformSegmentRow` never got the duet props. ## Changes - `WaveformSegmentRow`: new optional `singer` / `hasSingerOverrides` / `onSingerChange` props render the same `SingerChip` after the index/trash/play controls, plus `onFocus`/`onBlur`/`className` props. - `TranscriptionView`: in duet mode, passes the singer, the singer row tint and focus tracking to each waveform row, so the singer keyboard shortcuts work there too. Moved the tint logic into a shared `singerTintClass` helper. ## Testing - New `TranscriptionView.waveformsDuet.test.tsx`: chips render per line in duet mode, clicking cycles the singer, and no chips appear when duet mode is off. - `WaveformSegmentRow.test.tsx`: chip renders and cycles; focus and blur get reported. - Full frontend jest: 141 suites / 1735 tests pass. - Local CodeRabbit review: no findings. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_015cvzKpvvadUsJGZFMv9DXy Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
3d4c9c4e6e |
fix(gdrive): retry Drive 5xx/429, check publish completeness after fallback pass (v0.276.3) (#1170)
## Summary The Discord "Publish completeness SHADOW" alert for **NOMAD-1783** (job `dd885ea9`) was a false alarm. Drive returned a one-off `HttpError 500 internalError` on resumable-upload init, and the worker's fallback Drive pass re-uploaded all three files 43s later. Investigating it turned up four gaps: 1. **No retry on Drive 5xx.** `gdrive_service` only retried connection errors, so one 500 dropped the whole orchestrator upload. 2. **Check ran too early.** The shadow check ran inside the orchestrator, before `_handle_native_distribution`'s fallback pass. 3. **Fallback dropped the original-vocals guide.** kjbox's "Original Vocals" slider had no guide for NOMAD-1783. 4. **Recovered state was overwritten.** The main path persisted the orchestrator's empty `gdrive_files` and the stale "Google Drive upload failed" warning over the recovered state (`state_data.gdrive_files` was `None`). ## Changes - `is_transient_error`: Drive 429/500/502/503/504, plus 403 with `rateLimitExceeded`/`userRateLimitExceeded`, now retry like connection errors. Uploads get 4 attempts (2/4/8s backoff). Retries stay idempotent because each attempt re-runs the same-name delete. With `replace_existing=False`, only connection errors retry. - `VideoWorkerOrchestrator.defer_publish_completeness_check`: `generate_video_orchestrated` sets it and runs `check_publish_completeness()` after the fallback pass. `redistribute_video` is unchanged. - `_adopt_fallback_gdrive_upload`: carries the fallback's file IDs onto the result. The Drive-failure warning is dropped only when the recovered set is complete. - `OrchestratorResult.original_vocals_guide`: the fallback pass now ships the guide too. ## Prod remediation (already done) - Built the NOMAD-1783 vocals guide with the orchestrator's own `_emit_original_vocals_guide` (5s intro, silent intro verified) and copied it to `gs://nomadkaraoke-divebar-files/files/Nomad Karaoke/vocals-padded/`. - Restored `state_data.gdrive_files` from the actual Drive IDs (one file per folder, no duplicate) and cleared the stale warning. - Logs show no other `Google Drive upload failed` in the last 30 days. ## Testing - New `backend/tests/test_gdrive_fallback_recovery.py` (26 tests): the retry predicate, upload retry after a 500, no retry without cleanup, deferred and default check, guide kept on a failed upload, adopt/partial/no-recovery cases, and the fallback shipping the guide. - Full backend suite: 6268 passed, 36 skipped. - Local CodeRabbit: 2 cycles. Fixed partial-recovery warning handling, 403 rate-limit reasons, and the non-replace retry guard. Skipped folder-create idempotency, since the MP4/, MP4-720p/ and CDG/ folders always exist. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_0199crntgXrDkR93dQeqvnGY Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v0.276.3 |
||
|
|
b2ef88a861 |
fix(review): unify backing-vocals recommendation on the 3-stem scorer (v0.276.2) (#1169)
## Summary - `backing_vocals_analysis.recommended_selection` (legacy `AudioAnalyzer` rule: keep only if >20% audible or a segment > −20 dB) disagreed with the auto-approval `score_backing` verdict on **59/389 jobs (15%, Jul–Oct)** — every case 0.5–20% audible with no loud segment. The finish modal followed the scorer, the instrumental screens followed the heuristic, so the same job showed different "✓ recommended" stems (job 58555663). - Where they disagreed, admin/Andrew picks matched the scorer 14/18; customer picks lean clean regardless (38% clean even when both signals say keep). The heuristic also recommended keeping on 5 jobs the scorer flagged as "backing stem IS the lead" (humans picked clean 4/5). - 58555663 itself: the intro backing (16–29s) is a voiced B3–C♯4 hum motif with the lead stem silent, pitch-matched to the "Mm-mm-mm" hum lines — `with_backing` was right. ## Changes - `unified_backing_recommendation` / `with_unified_recommendation` in `auto_approval/instrumental.py`: scorer's confident verdict → `clean` if backing-is-lead → else the heuristic (preserved as `heuristic_recommended_selection`). Idempotent. - Applied at write time (`screens_worker._analyze_backing_vocals`) and read time (correction-data, `/instrumental-urls`, `/instrumental-analysis`) so already-analysed jobs are consistent too. - `backing_is_lead()` predicate extracted from `score_backing` and shared. - No change to auto-completion / enforcement (`backing_decision` untouched); frontend unchanged. - Docs: API.md note, LESSONS-LEARNED entry. Version 0.276.2. ## Testing - [x] `make test` — backend 2054 + 6243 + emulator 152, frontend 1730, all passing - [x] New unit tests (helpers matrix), worker write-path test, route test with the 58555663 signal profile ## Review - [x] CodeRabbit CLI review completed locally — no findings @coderabbitai ignore --- 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_018aUxu88FZajta46BRdmpcc Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v0.276.2 |
||
|
|
82a332dbe1 |
fix(review): proceed to instrumental review while stems separate; tolerate old backend (v0.276.1) (#1168)
Follow-up to #1167, from Andrew trying it on job 58555663. ## Changes - **Don't block the CTA while stems are separating.** #1167 disabled "Proceed to Instrumental Review" until the stems landed, but the full instrumental screen waits for them itself, and reviewers in a hurry (happy with clean) used to go straight there. The button is enabled again, and the waiting box adds "In a hurry? You can continue to the full instrumental review now instead." - **Tolerate a backend without `audio_complete`.** The frontend deploys well before Cloud Run, so the new page polled the 0.275.0 endpoint, never saw `audio_complete`, and spun until the 10-min cap even though separation had finished. The hook now treats "both stems present" as ready when the field is absent. - i18n: dropped `waitingForStems`, added `stemsWaitingProceed` (33 locales). ## Testing - Jest: hook + modal 31/31 (new: legacy-backend fallback; waiting-state CTA enabled + submits) - Playwright `cloud-mode-review.spec.ts`: 13/13 - Local CodeRabbit: no findings @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01QKJ7TxzFoUeNTug1PR8Cwh Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v0.276.1 |