206 Commits
Author SHA1 Message Date
Andrew Beveridge ☄️andClaude Opus 5.5 13619b7cc9 fix(review): preview shows the countdown intro like the final render (v0.296.0) (#1264)
## Summary
Songs whose vocals start within 3s get a "3... 2... 1..." intro in the
final render (`CountdownProcessor`: lyrics +3s, 3s leading silence), but
the **review preview never applied it**. So the preview opened straight
on vocals, and a customer (Randy Vild, "Caught Up Alive" / "Faithful and
True") asked us to add a countdown that was already coming.

## Changes
- `CountdownProcessor.add_countdown_to_lyrics_if_needed` (lyrics-only
half of `process`) + `padding_for_segments_data` (same rule on the
serialized segments the client sends).
- `CorrectionOperations.generate_preview_video` applies the countdown;
full local/CLI renders pad the audio too. Returns
`countdown_padding_seconds`. Countdown previews get a salted hash so
pre-fix cached previews aren't reused.
- GCE preview path: pads the already-downloaded audio on the API side
and uploads it to `previews/{hash}_audio.flac` for the encoder, so **no
encoding-worker change / version skew**.
- `POST /preview-video` responses carry `countdown_padding_seconds`.
- `PreviewVideoSection`: video time = song time + offset, so the
overlaid instrumental stem, waveform playhead and audition seeks stay
aligned (stem parked at 0 during the intro).
- Corrections that already contain the countdown aren't shifted again
but still get padded audio (CodeRabbit finding).

## Testing
- New:
`tests/unit/lyrics_transcriber/output/test_countdown_preview_helpers.py`,
`tests/unit/lyrics_transcriber/correction/test_operations_preview_countdown.py`,
GCE-path endpoint test in `backend/tests/test_routes_review.py`, 4
offset tests in `PreviewVideoSection.test.tsx`.
- Frontend jest (lyrics-review + lib): 892 passed.
- Real render check: local preview of job 508b38b2 (Faithful and True)
from prod corrections + audio with real ffmpeg shows the countdown, and
silence runs to 3.80s, matching the delivered final video's first word
(3.795s).
- Local CodeRabbit review: 2 findings fixed, re-review clean.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVVUdF2J1bNeVa2uLXA

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 13:56:20 -05:00
Andrew Beveridge ☄️andClaude Opus 5.5 da204bd52e fix(ass): split lyric lines wider than the frame (overlapping lyrics, NOMAD-1779) v0.295.1 (#1263)
## Summary

NOMAD-1779 (job `58555663`) shipped with two lyric lines drawn on top of
each other near the end of the song (YouTube comment: "the lyrics get
kinda messed up towards the end").

**Cause:** lines are split by character count (`max_line_length`, 40 for
Nomad). Wide-glyph lines like "Mm- mm- mm, mm- …" pass that but measured
~3890px on the 3840px frame. libass wraps an over-wide event by itself;
rows are exactly one line-height apart, so the wrapped half landed on
the next row's lyrics.

**Fix:** `SubtitlesGenerator.generate_ass` now measures each line in the
style font, scaled the way libass sizes it (existing
`LyricsLine._measure_text` / `ass_font_scale`), and splits any line
wider than 94% of the frame at a word boundary
(`SegmentResizer.split_to_width`, prefers phrase-ending punctuation,
keeps word timing, shares translations). Applies to every ASS render
(main, portrait, theme previews). Lines that fit are untouched.

**Scope:** of 309 completed branded jobs since 2026-09-10, only 1779 had
an overlap; 4 more (1759, 1738, 1766, NOMADNP-0264) had a line wrapping
into an empty row.

## Testing

- New `tests/unit/lyrics_transcriber/output/test_line_width_split.py`:
split logic (fits / splits / phrase break / balance / single long word /
timing / translation / singer), ASS generation, and a **real libass
render** asserting each lyric event draws as exactly one row. Verified
the render test fails with the fix disabled.
- Regenerated 1779's ASS from its `corrections_updated.json` with the
real theme font and rendered 2:58 with ffmpeg/libass: old ASS reproduces
the overlap exactly, new one renders 4 clean lines (widest 3502px).
- Locally: 1968 unit + 7231 backend tests pass. Failures were
environmental only (shared conda env lacks `flacfetch`: 6 unit + 2
collection errors; `test_emulator_integration.py` fails identically on
`origin/main` locally).
- CodeRabbit CLI (local, committed): no findings.

After deploy: admin re-render of 58555663.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVW1AnhNWu2qGLaNvxu

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 15:41:00 -05:00
Andrew Beveridge ☄️andClaude Opus 5.5 47c6691823 fix(admin): delete-outputs dialog shows success as green; YouTube 404 = already deleted (v0.294.3) (#1260)
## Summary
When Andrew deleted NOMAD-1875's outputs (2026-11-09), the result dialog
showed **YouTube, Dropbox and Google Drive all red** ("Partial
Success"). In reality:
- **Dropbox and Drive were deleted fine.** The backend logged `Dropbox:
success, GDrive: success`, and the brand code was recycled, which only
happens when both succeed. The dialog only treats `"deleted"` as green,
but the shared cleanup helpers (`published_outputs_cleanup.py`) return
`"success"`. The regression E2E mocks also used `"deleted"`, which is
why they never caught it.
- **The YouTube video was already gone.** The API returned 404 and
yt-dlp says "removed by the uploader", so it was removed outside gen.
The delete reported this as `failed`, with no message in the dialog.

## Changes
- `frontend/app/admin/jobs/page.tsx`: green for `success`.
`failed`/`partial` now show a reason, and a partial Drive delete lists
the files left behind ("Not deleted: cdg").
- `KaraokeFinalise.delete_youtube_video`: an HTTP 404 counts as already
deleted (warning log, returns True). Other errors (e.g. 403) still fail.
- `frontend/e2e/regression/admin-job-detail.spec.ts`: mocks now use the
real backend statuses, plus a partial-Drive assertion.
- Version 0.294.3.

## Testing
- `tests/unit/test_karaoke_finalise/test_youtube_integration.py -k
delete`: 6 passed (2 new: 404 = success, 403 = still a failure).
- Playwright regression spec: left to CI, since frontend deps aren't
installed in this worktree.
- Local CodeRabbit: 1 minor finding (show per-file Drive outcomes),
fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVUwKapcjF8czVQrPD5

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 11:57:38 -05:00
Andrew Beveridge ☄️andClaude Opus 5.5 5c41be1d3b fix(error-monitor): ignore runner-manager's other Cloud Run abort wording (#1252)
## Summary
On 2026-11-04 at 10:00Z a PR opened on python-audio-separator queued a
burst of `workflow_job` webhooks. 22 of 54 deliveries got a 500 from
Cloud Run while `github-runner-manager` scaled up. The error monitor
alerted because Cloud Run used a second wording, "aborted after waiting
too long for an available instance", which the existing ignore pattern
didn't match.

There was no impact: all self-hosted jobs in the affected runs got
runners within about 3.5 minutes and passed.

## Changes
- **Error monitor:** match both Cloud Run abort wordings for
`github-runner-manager`, but only on **webhook** requests (the bare
function URL). Cloud Scheduler ticks (`?action=check_idle`) and entries
with no URL still alert. `_SERVICE_URL_IGNORE_PATTERNS` now accepts a
predicate on the split URL as well as an exact path.
- **Runner manager (Cloud Function):** the ignore relies on the 5-minute
stalled-job tick catching dropped webhooks, so that tick's own failures
now log as structured `severity: ERROR` and page. That covers a
redispatch crash, a repo skipped in the scan, a job given up at 60 min,
and a failed re-dispatch. Before this, they were severity-less prints
that nothing alerted on. Both changes came from CodeRabbit feedback.
- Corrected the pattern's reason: GitHub did not redeliver these
(`redelivery: false`).
- Version 0.292.1; merged latest main.

## ⚠️ Deploy
`infrastructure/functions/runner_manager/` only goes live with a local
`pulumi up` (CI never runs Pulumi). Apply it **before** merging, so the
safety net's alerting is live before the webhook-abort ignore ships with
the backend deploy.

## Testing
- `tests/unit/services/error_monitor/`: 452 passed (webhook,
scheduler-tick, no-URL and other-service cases for both wordings).
- `infrastructure/functions/runner_manager`: 71 passed. The 4 updated
tests fail against main's `ephemeral.py`.
- Full `make test` not run locally; relying on CI.

Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567479055887114363

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVVTKcWCcDArTQrjS82

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* Runner-manager webhook errors are filtered more precisely. Recognized
abort messages are ignored only for matching webhook requests;
scheduler-tick errors, requests with other URLs, and requests without a
URL continue to trigger alerts.
* Runner-manager job-processing failures are recorded as structured
error logs, including stalled-job scans and dispatch failures. Give-up
logs now report the configured limit rather than the job’s current time
in the queue.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 19:58:46 -05:00
Andrew Beveridge ☄️andClaude Opus 5.5 a1cbe1c0d4 feat(encoding): retire the GCE c4d pair; run on AWS + c2df only (0.293.0) (#1255)
## Why
Since 2026-10-31 the AWS EC2 worker (`aws-encoding-worker-1`) has
handled every encode and distribution job. 27 jobs went to it, with no
Cloud Run fallbacks, no `encoding_stuck`, and nothing encoding-related
in #alerts. The c4d Spot pair (`encoding-worker-a`/`-b`) now only costs
~$8/mo of Hyperdisk. `encoding-worker-fallback-c2df` stays as the GCP
fallback.

## What
Deleting the pair needs code changes first, because several paths
assumed it exists:
- **`encoding_worker_manager`**
  - The primary is optional: an empty `primary_vm` means none.
- `ranked_candidates` skips a missing primary and always sets
`is_primary`, so with no primary a successful start records the AWS/c2df
override. Previously the legacy index-0 rule would have cleared it.
- `ensure_primary_running` and `ensure_preferred_running` start a ranked
candidate. If there are no candidates they raise `NO_CANDIDATES`.
- `demote_active_worker` looks up the override's `machine_type`, so the
AWS box can now be cooled down. It then clears the override so the retry
goes elsewhere.
- **`encoding_service`**
  - When `active_url` is empty, it targets the top-ranked candidate.
- It also invalidates its cached URL after a single-candidate fallback.
- **CI deploy (`ci.yml` + `deploy_promote.py`)**
- While AWS serves, a c2df green is validate-only: it writes
`validated_version`, keeps the AWS override, and stops the green. Before
this, it would have replaced the AWS override with c2df.
- The worker-change gate now diffs against `validated_version`, because
`primary_version` stops moving.
- An in-place restart of a serving override first waits for it to drain
(15-min cap). If it is still busy, the restart is skipped.
- **Idle function:** a missing VM is logged as a WARNING returning
`NOT_FOUND`, not an ERROR.
- **Error monitor:** watches c2df instead of a/b.
- **Pulumi:** drops the a/b VMs and static IPs. The prod preview shows
exactly 4 deletes. A separate, unrelated
`youtube-comment-autoreply-scheduler` drift shows up too; it will not be
applied.
- **Docs:** CLAUDE.md, ARCHITECTURE, DEVELOPMENT and TROUBLESHOOTING.

## Rollout (after merge + deploy)
1. Back up `config/encoding-worker`, then set `primary_vm`,
`primary_ip`, `secondary_vm` and `secondary_ip` to `""`.
2. Apply `pulumi destroy --target` on the 4 a/b URNs only.
3. Redeploy the idle function with `deploy.sh`.
4. Verify: one prod encode on AWS, warmup, and the logs.

**Rollback:** in Firestore, delete `speed_rank_overrides` and set
`aws_video_worker=false`. Restoring the pair would need a revert plus
`pulumi up`, which gives new static IPs.

## Testing
- New `backend/tests/test_encoding_worker_no_primary.py` (13 tests),
plus new deploy_promote, idle-function and error-monitor tests.
- Full backend suite: 7188 passed. The 8 `test_emulator_integration`
failures also fail on main, because of a local emulator from another
session.
- CodeRabbit CLI: no findings. An independent review found 2 issues,
both fixed: the demote retry path, and the in-place restart drain.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVTkatGnrYFhCzq4PTe

@coderabbitai ignore

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 18:50:54 -05:00
Andrew Beveridge ☄️andClaude Opus 5.5 70bdd0e693 fix(alerts): start-review 409, per-cause crash patterns, one alert per job failure (v0.290.1) (#1243)
Section 1 of the 2026-11-03 #alerts triage handoff: a stale push on a
finished song got a 500, and the error monitor could stay silent on a
new crash.

Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566980744448843886
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566980746394996879
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566980747905073224
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1565697725280948294
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1565803187334815775
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1565818418072916110
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566225593354420226
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566256141074169987
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567207141994135654
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567252495535841386
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567252497620537415

## What changed

**1a. start-review:** a job not in review gets 409
(`jobs.jobNotInReviewState`) with no transition attempted. Losing the
auto-approver race is a 409 too. `correction-data` has the same race,
and its `except Exception` turned it into a 500; it now returns 409
(`except HTTPException: raise` added). Its status gate still returns
400, as before.

**1b. Crash patterns:** `Exception in ASGI application` and
`request_audit_error` become `Unhandled exception: <Type: message>`,
read from the entry's `exception` field. The handoff's suspicion is
confirmed in `upsert_pattern`: an existing `new` pattern returns
`is_new=False`, so a second, different crash stayed silent. Both log
lines of one crash now share one pattern. The path and method go in the
sample only.

I read the line after the last frame rather than the traceback's last
line. pydantic's last line is a docs link shared by every failure, and
group tracebacks end in a separator.

**1c. One alert per failure:**
- YouTube ids are normalized out of signatures, which also fixes the
false "first time" line in `ops_alerts`.
- `Job <id> failed` (both call sites in `job_manager`) logs WARNING,
since the status write sends the job-failed alert.
- `URL download failed:` is ignored in the monitor, not demoted, because
`file_upload.py` belongs to the section 4 PR.
- `[job:<id>] Download failed` logs WARNING for the expected
`DownloadError`/`FlacfetchServiceError` types. This wasn't named in the
handoff, but it is the same duplicate on the audio-search path.
Unexpected exceptions there stay ERROR with a traceback.

**1d. Suppressions and demotions:**
- Zalo's `zaloJSV2 is not defined`: dropped at `/api/client-errors`,
only with a Zalo UA.
- `python -c` experiments on `audio-separation-job`: ignored, using the
handoff's regex.
- Spotify 403 on `/lyrics`: WARNING with `response.text[:200]`. 401 on
`/lyrics` and 401/403 on `/search` stay ERROR.
- Dropbox token probe: WARNING. A rejected refresh used to log nothing,
so it now logs ERROR, as does having no refresh credentials.
- decide-search health aborts: this was a small change. `should_ignore`
takes an optional `request_url` and matches the exact path
`/api/search/health`.

**1e. Cold start:** `anyio._backends._asyncio` is added to
`_LAZY_ROUTE_IMPORTS`. The prod stack from 10-30 confirms the stall was
in that import. Lifespan also sends one `/api/health` request through
the app in a worker thread, bounded by the existing 30s timeout and
best-effort.

## Testing

- TDD: new tests failed first, then passed.
- Affected suites: 826 passed.
- `backend/tests` (excluding `backend/tests/emulator`): 7088 passed, 8
failed, 29 skipped. All 8 are in `test_emulator_integration.py`, and
that file also shows 8 failures at the base commit a8f9618a. I did not
compare the 8 test names one by one.
- `tests/unit` outside the listed files was not run.
`test_audio_fetcher.py` and `test_remote_cli.py` fail to collect in my
environment; I didn't change either file.
- Read-only replay of real prod entries through the new code: the 11-03
crash gave one specific pattern with `POST
/api/jobs/7f89f884/start-review` in the sample, the decide-search health
aborts and the experiment traceback were ignored.
- Local CodeRabbit review completed with 0 findings.

## Not verified

- Nothing is live, so the alert volume and the cold-start stall haven't
been measured. The warm request has only run in tests, not on a real
Cloud Run cold start.
- A request's first-use imports beyond the anyio backend aren't
individually checked.
- My test environment has no `flacfetch`, so its lazy imports only log a
warning here.
- Zalo: the trigger is one report (11-03).

## Known leftovers

- `validate_state_transition` still logs ERROR before raising, so the
auto-approver race (and the existing in_review race) still emits one
`Invalid state transition` ERROR even though the client now gets a clean
response. The stale-push case no longer reaches it. I left this because
other callers rely on that log.
- A crash that also logs its own ERROR first still gives two patterns,
the ERROR and `Unhandled exception`.
- The first bot block after deploy shows "first time" once, because the
normalized signature is new.
- Old `Exception in ASGI application` and `request_audit_error` patterns
stop recurring and will auto-resolve.
- If the section 4 PR rewords `URL download failed:`, the ignore entry
in `known_issues.py` needs the same wording.

## For Andrew (infra and cost, not changed here)

- Cloud Run startup probe and min-instances for `karaoke-backend`, if
cold-start stalls continue after this ships.
- decide-search's min-0 is why its health pings abort. The suppression
hides those, but real search paths will still alert on the same abort.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVTvqXyL4QyA2EjzQMG

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 23:04:04 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 437b34ee64 fix(ci): wheel over PyPI's 100 MiB limit — drop unused RSC payloads + size guard (v0.287.2) (#1233)
## Summary
Every `Deploy - Publish to PyPI` since v0.279.4 (about 10-22) has failed
with `400 File too large. Limit for project 'karaoke-gen' is 100 MB`.
The wheel had grown to 110 MB (105.1 MiB), so PyPI is still on 0.279.4.
Backend deploys were not affected. I found this while watching deploys
during an alerts-triage session.

In the wheel, 66 MB of the bundled Next.js export is RSC `.txt` payloads
(about 3 MB per locale × 33). The only thing that uses the export is the
CLI local-review server
(`karaoke_gen/lyrics_transcriber/review/server.py`). It mounts only
`/_next` and `/static`, and every other route returns an HTML entry
point. Next's RSC fetch paths (`/{locale}/app/.../index.txt`) already
get HTML from those routes, so the `.txt` files are never served.

- `make build-frontend` deletes `*.txt` from the copied export. On the
real CI wheel from run 40826404322 this takes it from 105.1 MiB to 41.1
MiB.
- `scripts/verify_wheel_frontend.py`, which already runs in both
package-build jobs, now fails above 95 MiB. Growth will show up on the
PR instead of as a failed publish after merge.

## Testing
- New `tests/unit/test_verify_wheel_frontend.py`: 5 tests covering the
pass case, missing entrypoint, too few files, the size guard, and
headroom below 100 MiB.
- Re-zipped the real 0.286.0 CI wheel without the `.txt` files: 41.1
MiB, and the verifier passes. The verifier rejects the original wheel
(105.1 MiB) with the new error.
- CodeRabbit: 1 finding ("deleting RSC payloads can break client
navigation"). Skipped, because the local server never serves those files
(see above), so it can't change runtime behaviour.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVTiUPKxn3sQPyp3Teh

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 03:52:39 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 0d817509fa fix(api): event-loop stalls — visibility redistribution off-loop, route warmup, watchdog/Cloud Trace noise (0.286.2) (#1230)
## Summary

Fixes the 2026-10-29 #alerts event-loop stall batch (handoff
`docs/archive/2026-10-29-gen-event-loop-stalls-handoff.md` in the
workspace).

**1. Visibility change off the loop (39s stall, 13 requests incl.
/api/health)**
- `change_to_private` now runs claim, flag write, `redistribute_video`,
public-output delete and the final commit as **one** `asyncio.to_thread`
call (`_redistribute_and_commit_private`). `redistribute_video` runs on
that thread's own loop through the new `redistribute_video_off_loop`
(`asyncio.run(with_notification_drain(...))`).
- Is anything bound to the main loop? No. `redistribute_video` builds
its own `JobManager`/`StorageService`. The Dropbox, YouTube and GDrive
clients are sync. Job-log writes go through `log_write_offload`'s writer
thread. Notification tasks scheduled via `_schedule_notification` would
land on the private loop, so `with_notification_drain` drains them
before the loop closes, as the Cloud Run Job entry points do.
- Why one thread call and not just the redistribution (from CodeRabbit):
a thread can't be cancelled. So a cancelled request can no longer leave
a committed private archive with its public outputs still live, or a
claimed flag with no redistribution.
- Guard: `redistribute_video` is added to `BLOCKING_CALLS`, and
`visibility_change_service.change_to_private`/`change_to_public` are
added to `AWAITED_ROUTE_HELPERS`. I checked that putting back `await
redistribute_video(...)` fails the test.
- **Double Dropbox upload confirmed and fixed.** GCS `finals/` holds
every output twice: short canonical keys written by `_upload_results`
(`lossy_4k_mp4.mp4`, …) and the readable encoder names (verified on job
0f211e97: 17 objects / 525 MiB). Redistribution uploaded both. The
short-named `lossless_4k_mp4.mp4` also slipped past the `" (Final
Karaoke Lossless 4k).mp4"` Dropbox exclusion.
`_dedupe_redistribution_files` now keeps one copy per final, under its
readable name:
- a short name whose readable twin exists (twin found by suffix) is
dropped;
- a short name with no twin is renamed to the readable name (CDG/TXT
zips included).

**2. First-request stalls on new instances**
- Lifespan builds FastAPI's lazy route contexts
(`fastapi.routing.iter_route_contexts` +
`effective_low_priority_routes`) in a thread before readiness, with a
timeout and best-effort error handling. The results are cached by routes
version under a lock, so the OTel `_get_route_details` walk no longer
compiles route regexes on the loop. It is a no-op on FastAPI versions
without these internals. Note: the lock pins 0.128 but prod installs
unpinned (`pip install -e .`), so prod runs a newer FastAPI. I checked
against FastAPI 0.162: the first walk builds 304 contexts, the second
walk and the OTel lookup are cached (<1ms).
- `flacfetch.core.manager`, `flacfetch.downloaders.youtube` and `yt_dlp`
are added to `_LAZY_ROUTE_IMPORTS`.
- `search_audio` auto_download: `select_best` now goes through
`asyncio.to_thread`, with `select_best` added to `BLOCKING_CALLS`.

**3. Loop watchdog false positive**
- `_finish` logs WARNING, not ERROR, when all of these hold: no non-idle
sample was taken, a process freeze was observed (`frozen_s > 0`), and no
request was in flight when the stall began. Every "affected" request
therefore arrived while the instance was thawing. An unsampled stall
with no observed freeze, or with a request in flight at its start, stays
ERROR. The freeze requirement came from CodeRabbit.

**4. Cloud Trace exporter noise**
- `cloud_trace_export` ignore pattern for `Error while writing to Cloud
Trace`. CodeRabbit suggested matching only the transient gRPC detail. I
didn't: the error monitor only sees `jsonPayload.message` (the exception
is a separate field), and a persistent loss of telemetry isn't
#alerts-worthy anyway.

Smaller change: `change_to_private`'s Firestore `job_ref.update` writes
now also run off the loop.

## Testing
- New `backend/tests/test_redistribute_video.py`:
  - dedupe unit tests;
- an end-to-end `redistribute_video` run with mocks, checking the files
the Dropbox stage sees;
  - the off-loop runner uses its calling thread;
- `change_to_private` runs the redistribution on another thread and
loop;
  - a cancelled request still deletes the public outputs.
- `test_loop_watchdog.py`: frozen + unsampled + late arrival gives
WARNING; a request in flight at the start gives ERROR; no observed
freeze gives ERROR; a sampled stall with a late arrival gives ERROR.
- `test_main_startup_warmup.py`: lazy imports list; the route-context
walk; no-op without the API; failure is non-fatal; lifespan warms
off-loop before readiness.
- `test_known_issues.py`: the Cloud Trace pattern matches, and other
trace errors are not ignored.
- Local runs: `pytest backend/tests/ tests/unit/services/error_monitor
--ignore=backend/tests/emulator`: 7369 passed, 29 skipped. Not counted
in that total: `test_emulator_integration.py` fails 8/11 the same way on
the base commit, because a stray emulator from another session is on
:8080 locally. That's environmental, not caused by this PR.
- Local CodeRabbit: 3 cycles. Cycle 1: 3 findings (2 fixed, 1 skipped
with the reason above). Cycle 2: 1 finding, fixed. Cycle 3: 0 findings,
but CodeRabbit reported it reused cached results rather than a fresh
review.

## Not verified
- No prod run of a real public→private change yet. After deploy, the
Dropbox folder of the next one should contain each final exactly once,
and no `EVENT_LOOP_STALL` should appear during it.
- Route-context warmup was checked only against FastAPI 0.162 in a
scratch venv, not the exact prod version. Watch for the `Warmed N route
contexts` log line on the next cold start.
- Existing private Dropbox folders that already hold the duplicates are
not cleaned up by this PR.

Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1563719390849929330
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1563009782363721760
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1563039889354784882
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1563221363278348514
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1564308425033252895
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1563160849336762449

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVTiUPKxn3sQPyp3Teh

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 02:49:20 -04:00
Andrew Beveridge ☄️andClaude Sonnet 5.5 5c30fde9b4 fix: reference lyrics truncated at 3000 chars (blank lines counted) (v0.285.2) (#1225)
## Summary
- `BaseLyricsProvider.MAX_LYRICS_LENGTH` (3000) was measured on raw
text. Genius RapidAPI emits `\n\n` per line, so a normal ~4 min song
(job 258ba59b, 4,256 chars) was truncated after verse 2.
Lrclib/Musixmatch hit the same cap (2,975 byte files).
- Blank lines no longer count toward the cap; cap raised to 15000
(screenplays/articles are 50k+).

## Testing
- New regressions: double-spaced 70-line song and ~8k-char rap song keep
every line; screenplay tests still truncate.
- `tests/unit/test_base_lyrics_provider.py` 17 pass. 9 failures in
test_lyrics.py / test_agentic_gemini_provider.py are pre-existing on
main (env).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* Lyrics can now contain up to 15,000 characters before truncation, up
from 3,000.
* Blank lines no longer count toward the lyrics character limit, helping
preserve longer lyrics with extra spacing.

* **Documentation**
* Added guidance about the updated lyrics limit and a caution for
rerunning lyrics actions on in-progress jobs, where existing working
data may be replaced.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-27 15:17:05 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 9014754bda feat(error-monitor): watch voice pool jobs (v0.283.2) (#1219)
## Why
The Phase 3d voice pool runs for ~1 week unattended. Failures must reach
Discord #alerts (actionable-signal rule), and the error monitor only
watches Cloud Run jobs listed in `MONITORED_CLOUD_RUN_JOBS`.

## Changes
Added to `MONITORED_CLOUD_RUN_JOBS`
(`backend/services/error_monitor/config.py`):
- `voice-pool-select`, `voice-pool-fetch`, `voice-pool-analyse`
(karaoke-decide, us-central1)
- `vocal-range-gen` (karaoke-decide, us-central1; logs one ERROR from
task 0 only when every attempted job fails)
- `voice-pool-separate-job` (gen GPU job, us-east4)

The monitor's log query filters on `resource.type="cloud_run_job"` +
`resource.labels.job_name` only, with no region/location filter, so
us-east4 is covered the same way as `audio-separation-job`. No docs list
the monitored jobs, so none needed updating.

## Testing
- Updated the pinned count/contents test
(`tests/unit/services/error_monitor/test_config.py`)
- Added a test that the built log filter includes the new jobs and has
no location scope
- error-monitor tests: 414 passed
- Local CodeRabbit review: no findings
- Version 0.283.1 -> 0.283.2

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_0188txBF47GpTo7hym1FoifX

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 16:37:49 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 f6d846234d fix: alert noise + loop stalls + kn-data-sync OOM from #alerts triage (v0.282.1) (#1210)
## Summary
Fixes for the open and noise groups from the 2026-10-22 #alerts triage.

**Noise that should not page:**
- `stripe_service.py`: "Invalid webhook signature" now logs WARNING. All
42 hits on 10-21 came from `facebookexternalhit` via Cloudflare with a
malformed header. Every real Stripe delivery since 10-14 got a 200.
(#alerts 1562375740685815953)
- `error_monitor/known_issues.py`: `startup_probe` also matches Cloud
Run's "Default STARTUP **TCP** probe failed" wording.
(1562224513276706933)
- `frontend/lib/crash-reporter.ts`: a `removeChild`/`insertBefore`
NotFoundError is benign **only while the page is machine-translated**,
i.e. `<html>` has `translated-ltr`/`translated-rtl`. In the 10-21 crash,
Chrome had translated `/en/app` into Spanish ("Crea un video karaoke" is
not our string), and the customer's job completed and was downloaded.
(1562541569238638725)

**Event-loop stalls** (continues the WIP freeze item in BACKLOG):
- `review.py` audio-editor routes: `apply_edit`, `get_metadata_from_gcs`
and the GCS `download_file`/`upload_file` calls now go through
`asyncio.to_thread`. They were stalling the loop about 10s per edit (job
330d796c, 10-19). (1561861805695967363)
- `internal.py`: `process_conversion_emails` is imported at module top.
The lazy import inside the hourly cron's `async def` wrote a `.pyc` on
the loop, stalling it up to 15.3s, 4× on 10-20/21. (1562224514933334047)
- `backend/Dockerfile`: `python -m compileall` over `/app/backend` and
`/app/karaoke_gen`, so no lazy import compiles at runtime.
- `test_no_blocking_async_routes.py`: `BLOCKING_CALLS` now includes
`apply_edit`, `get_metadata_from_gcs`, `upload_file` and
`download_file`. Two more routes in `review.py` were caught and wrapped:
the preview ASS upload and the join-audio upload.

**kn-data-sync** (`infrastructure/modules/kn_data_sync.py`):
- Memory goes from 512M to 1Gi. The 10-22 full sync OOMed at 489 MiB on
a retry. 1Gi also gives gen2 more CPU, and the cost is about $0 within
the free tier.
- `attempt_deadline="320s"` on both schedulers. The default 180s gave up
on a run that took 184.8s and succeeded, then retried it, which meant 3
full KN pulls that day. (1562752965629190217, 1562752969395535933)

## Testing
- `make test`: 6603 passed. 8 failed in `test_emulator_integration.py`,
the known local-only failure (another session's stack holds :8080); they
fail the same way on unmodified main, and CI's emulator job is
authoritative.
- Targeted runs: error monitor, audio-edit routes, Stripe/webhook,
conversion emails and the blocking guard all pass (710 tests). Frontend
crash-reporter jest: 17 passed, with new translated / not-translated
cases.
- The local CodeRabbit CLI failed with "Invalid token", so the bot
review is left on for this PR.

## Deploy note
The `infrastructure/` change (kn-data-sync) needs `pulumi up` before
merge, per the repo rule. That needs Andrew's `/gcp-admin` grant.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_015AvGQEptp3Pg9RHhKGZ5KB

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved responsiveness during audio editing and review submissions,
reducing the chance that file and audio processing delays other
requests.
* Reduced false crash reports on pages affected by machine translation.
  * Improved recognition of startup probe failures in error monitoring.
  * Clarified logging for malformed payment webhook signatures.
* **Reliability**
* Increased resources and allowed more time for scheduled data sync
jobs.
  * Reduced delays to API requests during background processing.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

## Alerts
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562375740685815953
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562224513276706933
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562541569238638725
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1561861805695967363
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562224514933334047
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562752965629190217
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562752969395535933

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 19:26:43 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 e050cb8b3d fix: rescue repetitive-song reference lyrics the relevance filter undercounts (v0.279.4) (#1201)
@coderabbitai ignore

## Why
Andrew: job 5c1d24eb loaded no reference lyrics. LRCLIB and Genius both
returned the correct song, but the relevance filter scored them 26% and
25%, below its 30% threshold. Anchor coverage credits a repeated phrase
only once, so chorus-heavy songs undercount. The transcription was
decent, with a few consistent mishearings.

## What
`filter_irrelevant_sources(..., transcribed_text=)` now does a second,
**additive** check. A source the anchor filter rejects is kept when its
word-trigram overlap with the transcription clears four guards:
- **≥ 25%** of reference trigrams matched, repeats counted;
- **≥ 20** reference trigrams, so tiny references can't score by chance;
- **≥ 10** matches;
- **≥ 15 distinct** matched trigrams, so a shared refrain alone can't
pass.

Trigrams must contain three distinct words, so "na na na" and "oh oh oh"
don't count. Tokens are NFC-normalized and keep combining marks.

Both call sites pass the transcription: the corrector, which also serves
the review UI's lyrics search and add-source flow, and the controller's
uncorrected path. Without `transcribed_text`, behaviour is unchanged.
The rescue **never rejects** a source the anchor filter keeps. The
serialized `rejected_sources` metadata is unchanged.

## Calibration (459 recent production jobs; per-job reference txt vs
`(Lyrics Uncorrected).txt`)
- **Wrong-song baseline:** 491k cross-job pairs, excluding pairs with
the same title.
  - Worst genuine pair: 9.4%; p99.99: 5.5%.
- The only cross-job pairs that pass are one job whose audio is another
job's song under a wrong title. Their transcriptions overlap 45%, where
unrelated songs score under 10%.
- A naive n-gram rule would have admitted a wrong song at 41% through
shared "na na" filler. Requiring three distinct words removes it.
- **Rescued:** 22 sources that are currently rejected, **all the correct
song**, scoring 26–88%. Examples: the 5c1d24eb song, Deacon Blue, UB40,
Atomic Kitten, Sezen Aksu, Grateful Dead, Etherwood, Sub Focus, Terry
Allen.
- **Left rejected by the distinct-trigram guard:** one refrain-only drum
& bass track (8 distinct matches; it's the right song but has the risk
profile). This was a deliberate, cautious choice.
- No lyrics text is committed; the tests use invented lyrics.

## Testing
- `tests/correction/test_relevance.py`, 17 passed (8 new):
  - a repetitive correct song is rescued;
  - no transcription means unchanged behaviour;
  - a wrong song is still rejected;
  - filler doesn't count;
  - a tiny reference isn't rescued;
  - a source the anchor filter keeps is never rejected;
  - a shared refrain alone can't rescue;
  - combining marks are kept.
- `tests/unit`: 2184 passed. Running `tests/unit`, `tests/correction`
and the backend review tests together gives the same 12 order-dependent
failures on untouched main; they pass separately, as CI runs them.
- Local CodeRabbit: 2 cycles. Fixed the combining-marks and
distinct-trigram findings.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Jw21syo4HizJJjYw2mg2Vp

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 00:08:38 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 b67b273056 feat(kjbox): decide partner secret for catalog resolve + fake judge + error monitor for decide search (v0.278.10) (#1190)
## ⚠️ Merge prerequisite (do BEFORE merging)

The Secret Manager secret `decide-gen-partner-secret` must have a
version before this merges, otherwise the CI deploy's `--set-secrets
...DECIDE_GEN_PARTNER_SECRET=decide-gen-partner-secret:latest` fails.

```
gcloud auth login admin@nomadkaraoke.com
cd ~/Projects/nomadkaraoke/karaoke-gen-decide-partner-resolve/infrastructure
GOOGLE_OAUTH_ACCESS_TOKEN=$(gcloud auth print-access-token) pulumi up     # review: should only add the decide-gen-partner-secret Secret
openssl rand -hex 32 | tr -d '\n' | gcloud secrets versions add decide-gen-partner-secret --project=nomadkaraoke --data-file=-
```

No extra IAM grant needed: gen's API runs as `karaoke-backend@`, which
already has project-level `secretmanager.secretAccessor`.

## Summary
- `POST /api/kjbox/catalog/resolve` now also accepts karaoke-decide via
header `X-Partner-Secret` (new setting `decide_partner_secret` / env
`DECIDE_GEN_PARTNER_SECRET`). Other kjbox endpoints stay
`X-Kjbox-Secret` only. decide gets its own AI-call budget (60/min) so
kjbox nights are never starved; the verdict cache is shared.
- Fake match judge / resolve for the local E2E stack (`FAKE_WORKERS`):
`judge_kind` / `judge_canonical_*` knobs; no Gemini or catalog calls.
- Error monitor now watches decide's new `karaoke-decide-search` service
and `song-index-build` job.
- Pulumi: declares the `decide-gen-partner-secret` Secret (value added
manually); CI deploy mounts it.
- Version 0.278.10.

## Testing
- [x] Changed-module + kjbox/catalog/error-monitor/fake-worker backend
tests pass (133 passed). Full `backend/tests` suite exceeds 10 min
locally, so left to CI.

## Review
- CodeRabbit CLI failed twice (WebSocket closed / connection error), so
no local CodeRabbit review completed; I did a manual diff review instead
(no issues found). Deliberately NO `@coderabbitai ignore`, so the GitHub
bot reviews this PR.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01VsLw9cTTMBab1tiRxCsYrm

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 21:42:27 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 4286d98dbe fix: LRCLIB as a registered server-side app; honour Retry-After, retry 5xx (v0.279.2) (#1200)
@coderabbitai ignore

## Why
LRCLIB now applies stricter rate limits to unregistered clients
(https://lrclib.net/docs#implementation-requirements-rate-limiting), and
Cloud Run shares egress IPs. On 10-19, prod jobs got intermittent 503s,
and job 80e7ffdf lost its LRCLIB lyrics. LRCLIB is our only free source;
it supplied lyrics for 165 of 220 jobs in 30 days. Andrew registered a
server-side app.

## What
- **`karaoke_gen/lyrics_transcriber/lyrics/lrclib_client.py`** (new):
- Sends `Lrclib-App-Id` + `Lrclib-Api-Key` from `LRCLIB_APP_ID` /
`LRCLIB_SECRET_KEY`.
- Applies LRCLIB's rules: every 429 sets a **process-wide cooldown** of
`Retry-After` plus up to 30% jitter. The wait is never shortened, and
overlapping 429s keep the latest expiry. Nothing is sent until the
cooldown has passed. It waits and retries once only when the wait is
≤30s.
  - Retries 5xx after 1s/2s, honouring a 5xx `Retry-After`.
- A **401 on our credentials** logs a WARNING and drops to anonymous
access for the process, so a revoked key degrades instead of losing
LRCLIB.
- `LRCLIBProvider` (both `get` and `search`) uses it. The
`quick_version` renderer uses it with `retry=False`, so the fast path
gets the auth and the cooldown but never blocks.
- **Infra:**
  - Pulumi secrets `lrclib-app-id` / `lrclib-api-key`.
  - Env vars on the `lyrics-transcription-job`.
- The backend service deploy's `--set-secrets` (`ci.yml`). The
karaoke-backend SA already has project-wide `secretAccessor`.

## Deploy order (do NOT merge before steps 1–2)
1. `pulumi up` to create the two secrets and the job env. Needs
`/gcp-admin`.
2. Add the secret versions from Andrew's `.envrc` values.
3. Merge. The CI backend deploy references the secrets.

## Testing
- New `tests/unit/test_lrclib_client.py` (14 tests): headers; anonymous
use; Retry-After with jitter; a long Retry-After is not shortened and
blocks further calls; no-retry mode; a second 429; 5xx backoff; 5xx
Retry-After; 401 fallback; overlap merge (mutation-checked); provider
routes through the client.
- 164 passed across the client, `quick_version` and `search_lyrics`
tests.
- **Live** with the real app key: Dar Williams - Iowa and Wolfmother -
Where Eagles Have Been were found. One 503 was recovered by the retry.
- Local CodeRabbit, 3 cycles. I fixed the Retry-After cap, the cooldown
after a final 429, the `quick_version` 401 path, the overlap merge and
5xx Retry-After. I skipped one false positive about the tests' fake
clock (the retry path sleeps, then sends without re-checking the clock).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Jw21syo4HizJJjYw2mg2Vp

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 17:33:19 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 653801cd04 fix: Musixmatch provider uses the RapidAPI v2 /lyrics?artist=&title= endpoint (v0.279.1) (#1199)
@coderabbitai ignore

## Why
BACKLOG "Musixmatch lyrics provider dead for 30+ days (0/287 requests
succeed)". We're still paying for the RapidAPI listing
(`cavsn/musixmatch-song-lyrics-api`, Pro $14.99/mo, active), but the old
`GET /lyrics/{artist}/{title}/` path now 404s for every song. I checked
the listing's playground (via Andrew's RapidAPI login). The API moved to
`GET /lyrics?artist=&title=` (optional `trackId`, `translate`) and
returns a flat `data` object.

## What
- `MusixmatchProvider._fetch_data_from_source` calls the new endpoint
with query params. A miss comes back as **HTTP 200 with `data.error`**,
or with empty `lyrics`. Both are now treated as not-found and logged at
INFO, so they don't alert.
- `_convert_result_format` reads `trackName` / `artistName` / `trackId`
/ `lyrics`. Raw cache entries in the old `macro_calls` shape still
convert.

## Testing
- New `tests/unit/test_musixmatch_provider.py` uses real response shapes
captured today: endpoint and params, the 200 miss logged with no ERROR,
empty lyrics, no key, v2 conversion, legacy conversion.
- 47 passed across the provider, base provider and `search_lyrics`
tests.
- **Live check** with the production key: Queen - Bohemian Rhapsody 47
lines, Adele - Hello 48 lines. The upstream misses some songs (piri -
dog; Taylor Swift - Shake It Off), so coverage is partial but no longer
zero.
- Local CodeRabbit: 0 findings.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Jw21syo4HizJJjYw2mg2Vp

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 16:34:13 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 acb02e138a fix: un-pin audio-separator to 0.47.1 with RoFormer overlap=2 (v0.278.16) (#1196)
@coderabbitai ignore

## Why
BACKLOG "audio-separator 0.45–0.47 regression: mel-band RoFormer karaoke
models 3–5× slower on L4". The root cause is **python-audio-separator
#299 (0.47.0)**, not #298. #299 made RoFormer `overlap` a *window count*
that defaults to each model YAML's `num_overlap`. Before that, `overlap`
was effectively an 8-second step, which meant almost no overlap. The
karaoke models' YAMLs set `num_overlap` 4/8, so a 3-minute track went
from 23 forward passes to 87 (aufr33) and 124 (gabox_v2, becruily). That
matches the measured 25→75s, 26→149s and 27→157s. 0.45.0 and 0.46.0
still had the old step formula.

python-audio-separator #313 (released as **0.47.1**) keeps #299. It adds
a per-model forward-pass budget test on every PR and a GPU per-pass
timing test (T4: 1.21 s/pass), plus a README note. Andrew chose
overlap=2 for gen: smoother chunk seams than 0.44's no-overlap, at about
1.5× the 0.44 compute instead of 4–6×.

## What
- `karaoke_gen/audio_processor.py`: `SEPARATOR_MDXC_PARAMS` (overlap=2)
goes to all four `Separator(...)` constructions (stage 1, stage 2
karaoke ensemble, single-model paths) **only when audio-separator ≥
0.47**. On older versions `overlap=2` meant a 2-second step, which is 4×
slower, so they keep their defaults. Ensembles load each model through
`load_model`, so the param reaches every model in the preset.
- `backend/Dockerfile.gpu-base`: `AUDIO_SEPARATOR_VERSION=0.47.1`, with
the comment rewritten to explain the root cause. 0.47.1 needs
`torch>=2.3` on Linux, so the pre-installed `torch==2.6.0+cu126`
(required by the L4 driver) is kept.
- The poetry lock stays on 0.44.1. The version-aware params keep CI and
CPU images on the old semantics, and there's no torch churn.

## Expected cost per ~3-min track (forward passes)
| model | 0.44.2 | 0.47 YAML default | 0.47 + overlap=2 |
|---|---|---|---|
| karaoke_aufr33 | 23 | 87 | 44 |
| karaoke_gabox_v2 | 23 | 124 | 32 |
| karaoke_becruily | 23 | 124 | 32 |
| instrumental fv7z / resurrection | 23 | 32 / 21 | 32 / 21 |

## Verification plan (before /backlog-finish)
After the deploy, time the separation of the daily E2E or a real job on
the audio-separation-job logs. I expect roughly 1.5× the 0.44.2
karaoke-stage times (~40–50s per karaoke model rather than 75–157s). If
it's worse, revert the pin to 0.44.2. That's one line.

## Testing
- `tests/unit/test_audio_processor_local_gpu.py`: version gating
(0.47.x/0.48/1.0 → overlap 2; 0.44/0.46 → defaults; unknown → defaults;
fresh copy), and both stages' constructors receive `mdxc_params.overlap
== 2`.
- `make test`: the backend unit suite passes. The only failures are 8
`test_emulator_integration.py` tests, caused by my local environment:
another session's emulator setup is on :8080/:4443, and they fail
identically on untouched main. CI is green on main.
- Local CodeRabbit: 0 findings.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Jw21syo4HizJJjYw2mg2Vp

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 15:09:41 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 1344a086da fix: worker Cloud Run Jobs never dispatch video/render into the API process (v0.278.12) (#1192)
## Problem: a regression from #1191, caught by #alerts triage
At 07:03 UTC on 2026-10-19, two new freeze alerts fired: **52s with 29
requests affected** (culprit
`firestore_service.append_log_to_subcollection`, during CDG packaging)
and **81s with 33 requests** (culprit `dropbox_service.upload_file`).
The stacks show `video_worker.generate_video` running **inside
karaoke-backend**.

Cause: in today's daily E2E run (job 5a5e5181), #1191's park/resume
worked as designed (parked 06:55:04, resumed 06:57:50). But the resume
runs in `audio-separation-job`, which doesn't set
`USE_CLOUD_RUN_JOBS_FOR_VIDEO`. So `trigger_video_worker` fell back to
Cloud Tasks, which runs it as a `BackgroundTask` on the API, putting the
whole video worker on the API's event loop. The same latent bug already
applied to the audio worker's auto-approval → render trigger.

## Fix
- `backend/config.py`: inside a Cloud Run Job (`CLOUD_RUN_JOB` is set by
the platform), `use_cloud_run_jobs_for_video` and
`use_cloud_run_jobs_for_render` default to **true**. An explicit env
value still wins. This covers every worker job (audio, lyrics, download,
...) with no infra change, through the normal image deploy.
- Error monitor normalizer: `requests_affected=N` →
`requests_affected=<N>`, so a stall culprit is one alert pattern instead
of one per request count.

## Testing
- New `backend/tests/test_config_cloud_run_job_dispatch.py`: inside a
job → true; outside → unchanged; explicit env wins.
- New normalizer test.
- `backend/tests` + `tests/unit/services/error_monitor`: 6871 passed.
- Local CodeRabbit review: 0 findings.

## Verify after deploy
The next parked-then-resumed job logs `Started Cloud Run Job
video-encoding-job` from audio-separation-job, and there are no
`EVENT_LOOP_STALL` lines on karaoke-backend with `video_worker` in the
stack.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_019pmG2XAgtUguEJWLFpkKHN

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 13:17:49 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 70b2f4279a feat(alerts): collapse error bursts, suppress link previews, de-noise #alerts (v0.278.9) (#1188)
Andrew's goal: "I'd like to make sure that our alerts channel only has
alerts in it that are actionable and important, or are just signal and
not noise." This follows #1186 (#noise routing).

## Changes
1. **Burst collapse** (`error_monitor/monitor.py`, `discord.py`,
`config.py`). When a run has more than 3 new patterns for one service,
including leftover un-alerted ones, the monitor sends ONE #alerts
message. It has the count, total occurrences, the top 5 by count,
first/last seen, and a `scripts/query-error-patterns.py --service X
--status new` pointer. All of those patterns are marked alerted. The
grouping is deterministic, since LLM grouping is off in prod. The
10-messages-per-run cap still applies, and unsent patterns are picked up
again next run. Evidence: on 2026-10-07/08, one mb-refresh run produced
73 separate alerts.
2. **No link previews.** Every error-monitor post (`_post`) and every
`ops_alerts` post now sets `flags: 4` (SUPPRESS_EMBEDS), through a new
`suppress_embeds` kwarg on `DiscordNotificationService.post_message`.
The YouTube "New upload" post keeps its embed.
3. **Loop watchdog.** `EVENT_LOOP_STALL in progress` is now always
WARNING. The `ended` line keeps the existing ERROR rule. In the last two
weeks there were 24 "in progress" pattern alerts on top of 15 "ended"
ones. I checked `infrastructure/modules/monitoring.py`
`EVENT_LOOP_STALL_LOG_FILTER`: it keys on `"EVENT_LOOP_STALL ended"` +
`severity>=ERROR`, so this needs no Pulumi change.
4. **Frontend chunk-load errors** (`frontend_ingestion.py`,
`firestore_adapter.py`). Reports matching `isChunkLoadError` collapse
into one pattern per build with `alert_min_hourly_count=10`. The monitor
holds that pattern un-alerted until more than 10 arrive within an hour,
which means a broken deploy. Separately, Safari/Firefox stacks now get
the message prepended before hashing (a pattern was named
`@https://gen.nomadkaraoke.com`). Chrome hashes are unchanged. Existing
Safari/Firefox patterns will re-key once, and burst collapse groups
them.
5. **known_issues.** Added a service-scoped ignore for
`github-runner-manager` "no available instance" aborts during scale-up
(seen 09-20, 09-23, 09-27, 10-08). The same message from other services
still alerts.
6. **divebar-mirror schedulers.** Added `attempt_deadline="600s"` on
both `divebar-mirror-daily` and `divebar-mirror-refresh`; the function
timeout is 540s. With the 180s default, Cloud Scheduler retried syncs
that were still running, and the overlapping runs hit OOM (4x in 30
days).

## ⚠️ Pulumi
`infrastructure/modules/divebar_mirror.py` changed, so **this needs
`pulumi up` by Andrew before merge**. Auto-merge is NOT enabled.

## Testing
- New and updated unit tests in
`tests/unit/services/error_monitor/{test_monitor,test_discord,test_frontend_ingestion,test_firestore_adapter,test_known_issues}.py`,
`backend/tests/{test_loop_watchdog,test_discord_service,test_ops_alerts}.py`,
and `tests/unit/test_divebar_mirror_scheduler.py` (an AST guard:
deadline > function timeout).
- `pytest backend/tests --ignore=backend/tests/emulator`: 6421 passed,
36 skipped.
- `tests/unit/services/error_monitor`: all pass.
- Local CodeRabbit review (`--committed`) completed with 0 findings.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_019pmG2XAgtUguEJWLFpkKHN

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 05:13:13 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 0e507852af fix(alerts): demote expected conditions off ERROR + Payment Link webhook fix (v0.278.9) (#1187)
## Summary

Alert hygiene, part of keeping Discord #alerts to actionable signal
only. ERROR is page-worthy: the error monitor turns every new ERROR
pattern into an #alerts message. This PR moves expected conditions down
to INFO/WARNING and fixes the one real bug among them, which was a
Payment Link checkout crashing the Stripe webhook. It covers the BACKLOG
items "Payment Link checkouts crash the Stripe webhook" and part of
"Alert-hygiene bundle".

## Changes

1. **Stripe webhook, non-gen checkouts** (`users.py`,
`stripe_service.py`, `stripe_admin_service.py`)
- A session that gen didn't create (a Stripe Payment Link:
`payment_link` is set, there's no gen metadata, and
`customer_email=None`) used to log ERROR "No user email in checkout
session". `store_payment` then crashed on `None.lower()`, so the payment
never reached `stripe_payments` or revenue.
- New `StripeService.is_gen_checkout()`. A non-gen checkout now logs
INFO "non-gen checkout (payment link ...)" and grants no credits. It is
marked processed for idempotency and is still stored, with
`customer_details.email`, `order_type="payment_link"`, `payment_link_id`
and the line-item description.
- `store_payment` now handles None for metadata, customer_details,
total_details and payment-method fields.
- **Not backfilled:** session `cs_live_a1ENsua3w6mDH04PPc…` (2026-10-13,
a $36 tip, no customer action needed). It could be backfilled through an
admin path that calls `store_payment` on the expanded session. GCP
writes from agents are restricted, so this PR doesn't do it.
2. **Lyrics providers** (`base_lyrics_provider._log_request_exception`):
a 404 means "no lyrics for this track" and now logs INFO. 429 moves to
WARNING alongside 5xx, connection errors and timeouts. Auth errors and
other 4xx stay ERROR. After 5 404s from one provider in a process, it
logs a single WARNING saying the provider may be dead (TODO points at
the Musixmatch backlog item).
3. **research_audio** (`audio_search.py`): no more
`AWAITING_AUDIO_SELECTION -> SEARCHING_AUDIO ->
AWAITING_AUDIO_SELECTION` transitions, which were invalid and logged 2
ERRORs per re-search. The job stays in AWAITING_AUDIO_SELECTION with no
status writes, so a crashed request can't strand it. After the search
the job is re-read. If a selection landed in the meantime, the fresh
results are discarded with a 409 instead of replacing the results that
selection indexed into.
4. **audio-download-job on a deleted job** (`audio_download_worker.py`):
if `get_job()` returns None at the start or after the download, or a
failure happens and a fresh read confirms the doc is gone, the worker
logs INFO and returns success, so it exits 0. That means no `fail_job`,
no further writes, no worker triggers and no Cloud Run retry. A GCS
NotFound on a job that still exists stays an ERROR. Seen with job
6ffe88ad on 2026-10-14.
5. **Loop watchdog**: "Failed to record loop stall (non-fatal)" now logs
WARNING with exc_info. Only that line changed, to keep the conflict with
feat/alert-noise-monitor trivial.
6. **Job-list queries** (`firestore_service.list_jobs`,
`list_jobs_summary`, `list_jobs_by_batch_id`): `query.stream()` now gets
`retry=` and `timeout=` of 30s. The client default is 300s for both,
which caused the "Error listing jobs summary: Timeout of 300.0s
exceeded" errors.

Version 0.278.9 (0.278.8 is taken by the still-open #1186).

## Testing

- New/updated unit tests:
- `test_stripe_webhook_non_gen_checkout.py`: webhook stores the payment,
logs INFO, logs nothing at ERROR, and marks the session processed.
- `test_stripe_admin_service.py`: Payment Link session and null fields.
- `tests/unit/test_base_lyrics_provider.py`: log levels for 404, 429 and
401, plus the dead-provider WARNING.
- `test_audio_search_research_attach.py`: no status writes, and a
selection made mid-search gets a 409.
- `test_audio_download_worker.py`: job deleted at start, mid-download,
or with NotFound on write.
  - `test_loop_watchdog.py`: recorder failure logs WARNING.
- `test_firestore_list_query_timeout.py`: bounded retry/timeout on all
three list queries.
- `pytest backend/tests --ignore=backend/tests/emulator`: 6434 passed.
`pytest tests/unit`: 2108 passed.
- Local CodeRabbit review: cycle 1 raised 2 findings (non-gen session
idempotency, re-search race), both fixed. Cycle 2 was clean.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_019pmG2XAgtUguEJWLFpkKHN

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 02:58:58 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 a6d4163d2e feat(alerts): route non-signal Discord posts to #noise (v0.278.8) (#1186)
## Summary
Andrew asked for #alerts to carry only real problems. Routine and
informational posts now go to the new **#noise** channel. The GitHub org
PR webhook was already moved to #noise separately.

| Message | Before | After |
|---|---|---|
| Error monitor: New Error Pattern / Spike / Incident | #alerts |
#alerts |
| Error monitor: ✅ Patterns Auto-Resolved | #alerts | **#noise** |
| Error monitor: 📊 Daily Error Digest | #alerts | **#noise** |
| Nightly Backup Report (clean run) | #alerts | **#noise** |
| Nightly backup failed/partial (title is now "🚨 Nightly Backup FAILED")
| #alerts | #alerts |
| ✅ E2E Daily Test PASSED | #alerts | **#noise** |
| 🩹 E2E self-healed Stripe (auto-merge PR opened) | #alerts | **#noise**
|
| 🩹 self-heal PR needs CI approval click / could not open | #alerts |
#alerts |
| E2E / Tenant E2E failures, DR freshness stale, Job failed, Publish
completeness SHADOW, Gemini unavailable, OAuth credential alerts,
quarterly DR drill reminder | #alerts | #alerts (unchanged) |

## How the noise webhook is resolved
- **Backend / error monitor and the backup function** read it at
runtime: env `DISCORD_NOISE_WEBHOOK_URL` first, then the Secret Manager
secret `discord-noise-webhook` (the name can be overridden with
`DISCORD_NOISE_WEBHOOK_SECRET_NAME`). No Pulumi change is needed,
because both service accounts already have project-wide
`secretAccessor`.
- **GitHub Actions** uses `secrets.DISCORD_NOISE_WEBHOOK_URL ||
secrets.DISCORD_ALERT_WEBHOOK_URL`.
- **Nothing is silently dropped.** If the noise webhook can't be
resolved, or a post to it fails, the message goes to #alerts instead. In
the workflow this uses `curl -fsS` with a retry on the alert webhook.
- The error monitor now rate-limits each channel separately (10 per run
each), so noise can't use up the alert budget during an incident.

## Deploy notes
- **Backup Cloud Function:** CI doesn't deploy its source. Run
`infrastructure/functions/backup_to_aws/deploy.sh` after merge.
- **Error monitor job:** runs `karaoke-backend:latest`, so it picks up
the change once CI pushes the image.
- **No `pulumi up` needed.**

## Testing
- `tests/unit/services/error_monitor`: 376 passed. New tests cover noise
vs alert routing, fallback when the noise webhook is missing or fails,
separate rate limits, and secret resolution.
- `backend/tests` (excluding emulator): 6418 passed, 36 skipped.
- `infrastructure/functions/backup_to_aws`: 60 passed, including the new
`test_discord_alert.py` and routing tests in `test_main.py`. These were
run in a scratch venv because `functions_framework` isn't in the conda
env, which was already the case before this change.
- `actionlint` on `e2e-daily.yml`: no new findings. The only findings
are info-level shellcheck notes that were already there.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_019pmG2XAgtUguEJWLFpkKHN

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 02:46:50 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 9d6aa3df7d feat(medley): per-section backing_vocals_db for the karaoke track (v0.278.6) (#1184)
## Summary
The Fiverr medley customer said the backing singers in two songs were
too loud to sing over in tune, and asked for them as a faint background.

- **New option:** a per-section `backing_vocals_db` in the medley spec.
For the karaoke track only, the section is rebuilt from the clean
instrumental plus the backing-vocal stem at that level, e.g. `-14` for a
faint background. The rest of the karaoke track, the clean instrumental
and the guide are unchanged.
- **Cached mix:** the mix is cached per job/level, written to a
`.partial.flac` and renamed, so an interrupted run never leaves a
truncated mix to be reused. If the job has no backing stem, it stops
with a clear error.
- **Docs:** `docs/MEDLEYS.md` spec table.

## Testing
- `test_make_medley.py`: parsing, the ffmpeg graph, only the karaoke
variant changing, the atomic write and the cache. 20 + chord-chart tests
pass.
- Real data: the customer's medley with `-14` on Remember When and On
the Other Hand.
- Only those two sections change; the others differ by ≤ -43 dB, which
is crossfade overlap.
  - Overall section loudness is unchanged.
- A local CodeRabbit review found 1 issue (stale or partial cache). It
is fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 15:04:43 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 03ee693f28 feat(medley): customer chord charts + static-gain master (v0.278.5) (#1183)
## Summary
Two follow-ups on the medley tooling, both from the Fiverr medley
customer's feedback.

- **Fixes the robotic artifact at the start of the karaoke audio.** The
final single-pass `loudnorm` ran in dynamic mode. On a song that opens a
cappella (HSLHT sings from 0.4s, band at 3s), the instrumental's first
3s is near-silent separation residue at -70 dB, and `loudnorm` boosted
it by about 50 dB into a loud warble. The master is now one static gain
to -14 LUFS plus a 4×-oversampled limiter at -1 dB (`master_filter`).
After the fix, the first 3s measure -61 dB (were -15 dB). Sync is
sample-identical and true peak is -0.9 dB.
- **Imports the customer's own chord chart.**
`scripts/medley/chord_chart.py` plus a new `make_medley.py chart` step
read a chords-over-lyrics chart (.txt or .docx in Courier, capo shapes,
`Capo N` / `Key change Capo 1` lines). They align it word-by-word to the
reviewed lyrics, so chords come out timed on the words the chart puts
them on. Detected chords fill intros, breaks and spoken verses,
transposed into the capo shapes.
- **Shows the capo in the chord box.** New `ChordChange.capo` field,
shown as "capo N" under the chord.
- **Docs:** `docs/MEDLEYS.md` gets a new section on importing a customer
chart, and the mastering guidance now says never to use single-pass
`loudnorm`. `docs/README.md` has an entry.

## Testing
- Unit tests:
  - New: `tests/unit/test_chord_chart.py` (9).
- Added: capo tests in `test_guitar_chords.py`, plus master and capo
tests in `test_make_medley.py`.
- Full `tests/unit` suite: 2100 passed, 65 skipped (run before the last
CodeRabbit fix; only the 9 chord-chart tests were re-run after it).
- Real data: the customer's 4-song chart against the medley. The
lyrics-with-chords output matched the chart line by line, and 4K frames
rendered with capo labels.
- A local CodeRabbit review found 1 issue (two chord groups after the
same word could come out in the wrong order). It is fixed and has a
test.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01QV2TQwyC7pwVkGPCk98Bqg

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 14:24:45 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 7cf5beede8 feat: guitar chords in karaoke videos + ChordMini ensemble chord detection (v0.277.0) (#1175)
## Summary
Adds optional guitar chords to karaoke videos, first used for a Fiverr
medley customer who wants chords on their video.

- **Rendering (gen library):**
`OutputGenerator.generate_outputs(chords=[ChordChange(start, end,
label)])`.
- Each chord change during a sung line is drawn above the word it lands
on, in a row reserved above every line (the same slot mechanism as
translated lyrics).
- Chords light up when played, using the same `\k` highlight as lyrics:
gold while upcoming (theme `chord_color`), sung-lyric colour once
played. Quick changes on one word never overlap.
- A top-right "now / next" chord box runs for the whole song, covering
intros and instrumental breaks.
- With no chords supplied, output is unchanged. A test asserts the ASS
is byte-identical.
- **Detection (`scripts/medley/detect_chords.py`):**
- Ensembles both ChordMini checkpoints (BTC + 2E1D, MIT; about 80%
majmin/MIREX, the best open weights we found) over the full mix, gen's
clean instrumental, and an htdemucs_6s harmonic mix (guitar + piano +
bass + other).
- Weighted per-frame vote, then merges short blips, snaps changes to
beats and spells chords for the key (Ab/Db/Eb7, F#m, borrowed
bIII/bVI/bVII as flats).
- **Medley script:** new `chords` step plus a `"chords": true` spec
option. Chords are clipped per section and hand over mid-crossfade, then
shifted for the countdown.
- **Docs:** `docs/MEDLEYS.md` § Guitar chords (setup, method, accuracy
caveats, theme keys) and a docs/README.md entry.

There's no job or UI option yet. The medley pipeline uses this for now;
a product version would need a chords toggle plus detection in a worker.

## Testing
- New: `test_guitar_chords.py` (12: chord timeline helpers, word
placement measured against the font, layout geometry, highlight timing,
now/next box, theme colour, no-chords path byte-identical),
`test_detect_chords.py` (7), and 2 medley chord-shifting tests.
- `tests/unit`: 2087 passed, 65 skipped.
- End to end on the 5 real medley songs: models/inputs agree on 91–99%
of major/minor labels, and the output matches the expected I–IV–V
progressions. Rendered 1080p and 4K medley videos with 146 chord
changes; spot-checked frames (verse, instrumental screen, chorus).
- Local CodeRabbit: 1 finding (doc section reference), fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01QV2TQwyC7pwVkGPCk98Bqg

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 00:24:11 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 85390fd12b docs: medley-making guide + spec-driven make_medley script (#1174)
## Summary
Documents the multi-song medley process used for the first
classic-country medley, and turns it into a reusable script so the next
one only needs a spec file.

- `docs/MEDLEYS.md`: the full process. One gen job per original song,
then identical cuts on every audio variant, and the reviewed lyrics
shifted onto the medley timeline and rendered with gen's own
`OutputGenerator`. Also covers how to choose cut points (crossfades,
pot-out/pot-in, keys/tempos, checking the brief against the recording)
and gotchas.
- `scripts/medley/make_medley.py`: `fetch` (Firestore + gcloud CLI) →
`audio` (per-section LUFS gain, crossfade/gap/pot fades, `loudnorm` to
-14 LUFS / -1 dBTP, three variants) → `video` (shifted reviewed lyrics +
countdown + themed title/end screens + karaoke and guide MP4s).
- `scripts/medley/example-spec.json`: the country medley as a worked
example.
- Linked from `CLAUDE.md` Quick Reference and `docs/README.md`.

Ops tooling only. No backend or frontend change, so no version bump.

## Testing
- `tests/unit/test_make_medley.py`: 12 unit tests covering spec
validation, the timeline offset maths, lyric-segment shifting (including
the same song used twice) and the ffmpeg filtergraph.
- End-to-end run against the real prod jobs: `fetch` → `audio`
reproduces the hand-built v1 medley **bit-for-bit** (max sample diff 0.0
on both the instrumental and guide WAVs). `video --resolution 360p`
renders both MP4s and the LRC.
- Local CodeRabbit review: 1 finding (unchecked ffmpeg exit code in
`measure_lufs`), fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01QV2TQwyC7pwVkGPCk98Bqg

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 19:44:27 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 eeb08578c9 feat(tenant-theme): CD+G tab in the theme editor with real CD+G previews (v0.274.0) (#1164)
## Why
Commercial producers (Vocal Star first) care a lot about their CD+G
output, but the tenant theme editor only covered the MP4 title card /
lyrics / end screen. CD+G styling needed an admin to hand-edit the theme
JSON.

## What
**New "CD+G" tab** (only on portals with `features.enable_cdg` and a
`cdg` theme section):
- **Up card**: image, title/artist colours, title/artist sizes,
separator text between them (e.g. "MADE FAMOUS BY") + its colour, text
position from top (keeps text below a logo baked into the image),
title–artist gap.
- **Lyrics**: sung / unsung / screen / border colours, size, outline
width + colour, lyrics font ("Same as main font" or a distinct one, e.g.
condensed Oswald).
- **Instrumental breaks**: image, text, colour. **Closing screen**:
image, line colours.

**Real previews**: on the CD+G tab the preview request sets
`include_cdg`; the backend composes a short CDG with the real
`CDGGenerator` from the draft's `cdg` section and renders the up card +
a part-sung lyrics page with the new `karaoke_gen/.../cdg_decoder.py`
(single pass, multi-frame; moved out of the CDG test). 300×216 upscaled
3× with hard pixels, ~1 s. Other tabs never pay for it.

**Images**: uploads from the CD+G tab are fitted (letterboxed) to
300×216 PNG (`purpose=cdg`); the CD+G dropdowns only offer 300×216
images (`cdg_images` from the editor GET, header-only reads);
save/preview reject other sizes since the CDG renderer never resizes.
All 4 existing themes' CD+G images verified 300×216.

**Validation**: `cdg` hex colours, numeric bounds, single-line text
limits; distinct CDG font resolved for the preview the same way renders
do (`cdg_font` asset), title/outro use the main font.

## Tests
- Backend: CDG validation/fit/size-check/upload/editor list
(tenant_theme_service), real CD+G preview renders incl. colour change +
distinct font + missing asset + timeout (theme_preview_service), route
include_cdg + upload purpose. CDG title-card tests now use the shared
decoder.
- Frontend: new `TenantThemeEditorCdg.test.tsx` (10): tab gating,
includeCdg only on that tab, edits map to `cdg`, outline applies to both
strokes, font follow/distinct, fitted upload, CD+G-sized image list,
pixelated 300/216 previews. Full jest 1716 pass.
- Backend suite: all pass except `test_emulator_integration.py`, which
fails identically on main locally (a stray local Firestore emulator
enables it); CI runs emulator tests separately.
- 32 locales translated. CodeRabbit: 2 minor findings fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Gp5fBmsmWJxgkEHb4JfhXR

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 15:02:24 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 621077474e feat(tenant): track code prefixes every delivered file + CDG title shrinks unbreakable words (v0.273.1) (#1158)
## Summary
Follow-up to #1153 and #1157, found by running a real Vocal Star job
through the portal in prod (job `8dc6945f`, E2E with track code
`E2E1531717`, which passed).

1. **Track code on every file.** #1153 put the tenant's code on the
Dropbox **folder** (`VS6588 - Artist - Title/`), but the files inside
were still `Artist - Title (…)`. Matt's April 2025 ask was the VS code
at the start of **each file**, and Andrew had confirmed "all files
prefixed with the track code".
- Tenant jobs with their own code now get `VS6588 - Artist - Title (…)`
on every Dropbox file.
- The entries inside the CDG/TXT zips are renamed too, so each CDG+MP3
pair still matches.
- Nomad's own `keep_brand_code` flows (finalise-only, re-render) are
unchanged, because this requires `tenant_id`.
2. **CDG title card:** a single word wider than the screen can't wrap,
so the card now shrinks the font instead of clipping it. Seen on the
first real prod render, where the E2E title has a 24-character timestamp
token.

## Changes
- `dropbox_service.upload_folder(filename_prefix=…)`.
- `prefix_zip_entries()` streams entries and keeps their
timestamps/compression, so a repack is deterministic.
  - `skip_unchanged` also works for repacked zips.
- `tenant_track_code_prefix(job)` is set into
`OrchestratorConfig.output_filename_prefix`. Tenant-only, and the code
is sanitized again before use.
- cdgmaker composer: width check in the title-card shrink loop.
- Version 0.273.0 → 0.273.1.

## Testing
- Prefixed upload, including zip entries and subfolders.
- No-prefix path unchanged.
- Deterministic, streamed repack.
- Tenant-only prefix, with sanitization.
- Real-render test for the unbreakable word.
- Package unit tests: 2054 passed. Backend: 6066 passed (emulator tests
excluded locally).
- CodeRabbit: 3 cycles, all findings fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Gp5fBmsmWJxgkEHb4JfhXR

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 01:52:35 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 bfc8f7fb0f feat(cdg): configurable title card + per-theme CDG lyrics font (Vocal Star up card) (v0.273.0) (#1157)
## Summary
Vocal Star (and other commercial producers) need CDG files in their own
house style. Matt's 2025-02-20 email showed our CDG up card overflowing
its bordered design. He asked for his simpler card instead: logo on top,
then title, a small "MADE FAMOUS BY" line, then the artist.

Two pipeline limits blocked that:
- the CDG title card was hardcoded;
- a theme could only have **one font** for everything. The shared `font`
asset (`intro.font`) overwrites `karaoke.font_path` and `cdg.font_path`
at render time.

## Changes
- **Title card (cdgmaker composer):** new optional keys; all defaults
reproduce today's output.
- `title_separator_text` / `_color` / `_font_size`: a line between title
and artist.
  - `title_font_size`, `artist_font_size`, `title_max_height`.
- `title_block_top`: centre the block below a logo baked into the
background.
- It shrinks the title first, then the artist, so it can't loop forever.
- **Outro:** respects `title_block_top` and shrinks long text. Before,
an `assert` crashed when the text was taller than 200px.
- **Fonts:**
- The CDG title card and outro use the theme's main font
(`cdg.title_font_path`, fed by the shared `font` asset).
- A theme may keep a **distinct CDG lyrics font** via a new `cdg_font`
asset. The theme service emits it only when it differs from
`intro.font`, and `update_asset_paths` applies the shared font first so
section overrides win.
  - No visible change for existing themes, which all use one font.
- **Tenant theme editor:** keeps a distinct, available CDG font on save,
and copies it into the theme's assets if it's bundled.
- Docs: API.md.

## Testing
- New `tests/unit/lyrics_transcriber/output/test_cdg_title_card.py`
renders **real CDG files**, decodes the packets back into frames and
checks:
  - the separator sits between title and artist;
  - text stays below `title_block_top` and inside the safe area;
  - the title font is applied, and a missing one falls back;
  - long outro text shrinks instead of crashing.
- New tests for asset-mapping order, `cdg_font` emission, theme-editor
font handling, and the updated mapping assertion.
- Package unit tests: 2053 passed. Backend: 6056 passed (emulator tests
excluded locally).
- Local CodeRabbit with `--fresh`: no findings.
- Rendered Vocal Star samples (short, wrapped-long and very long
title/artist), decoded and visually checked against Matt's design.

After deploy, the Vocal Star theme gets: a new up-card background (logo
cropped from Matt's own card), separator "MADE FAMOUS BY",
`title_block_top`, black border, Avenir as the main font and Oswald kept
for CDG lyrics.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Gp5fBmsmWJxgkEHb4JfhXR

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 00:58:42 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 39dcf2598e fix: PyPI prune snippet + force marker only in PR title (v0.268.10) (#1146)
## Summary
Two follow-ups to #1145, both found while doing today's prune.

1. **Prune console snippet silently did nothing.** PyPI's delete-version
form now requires five `acknowledge_*` checkboxes. A submit without them
gets a **200 re-render**, which the old snippet counted as "deleted".
The snippet now reads each release's real form (CSRF plus checkboxes),
submits it, and counts a version deleted only when its manage page 404s.
This is the exact approach used for today's prune (88 releases, 10.73 →
~4.1 GB).
2. **The force marker matched the commit body.** A squash commit's body
is the PR description. #1145's description just *mentioned* the marker,
which forced a publish on merge. That publish ran before the prune, so
it 400'd and uploaded nothing. The marker now only counts in the commit
subject (the PR title).

Docs: the runbook covers the new form, the per-version JSON cache lag,
and a prune log.

## Testing
- Unit tests: 30 pass across both scripts, with new tests for
checkbox/404 handling and for a marker in the body not forcing a publish
- `node --check` on the generated snippet
- Replayed #1145's real merge message through the gate: no publish
- Local CodeRabbit review: 0 findings
- Full `make test` not re-run (no app code touched); PR CI covers the
package suites

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01MSW9yjwGnzz1S8MgCGm6t5

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 13:34:57 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 5005d3cb80 ci: publish to PyPI only when CLI code changed (v0.268.9) (#1145)
## Summary
PyPI publishing failed on every deploy since v0.268.4. karaoke-gen hit
PyPI's 10 GB project cap for the third time (June, Sept 29, Oct 12).
Each wheel is ~62 MiB and we published on every merge (~100/month), so
each prune only bought a couple of weeks.

PyPI is only the public `pip install` channel for the CLIs. Cloud Run
builds from source, and the encoding worker pulls the wheel from GCS. So
we now publish only when something CLI users get has changed.

## Changes
- `scripts/pypi_publish_gate.py` (stdlib): diffs
`scripts/pypi_publish_paths.txt` (`karaoke_gen/` plus the `backend/`
modules it imports) and `pyproject.toml` (ignoring the package `version
=` line only) against **the latest version on PyPI**. A skipped or
failed publish therefore never loses a CLI change. Fails open.
- `ci.yml` `package-publish`: new "Decide PyPI publish" step gates
`poetry publish` (now `--skip-existing`). **Tags and GitHub releases
still happen on every version**, because the encoding-worker gate diffs
against tags. Override with `[publish-pypi]` in the PR title/commit, or
the new workflow_dispatch input "Publish to PyPI".
- Not gated on purpose: `frontend/**` (bundled review UI) and
`backend/config.py` (cloud settings). A test keeps the backend import
list in sync, with `config.py` as the one documented exemption.
- Docs: PYPI-STORAGE-PRUNE.md, TROUBLESHOOTING.md, prune script and
reminder workflow comments.

## Impact
Replayed over v0.200 to v0.268: **24 of 136** releases would have
published (~0.7 GB/month instead of ~4 GB/month). Run against current
main, it decides "no CLI-affecting changes since v0.268.3", which is
correct for v0.268.4 to v0.268.8.

## Testing
- 15 new unit tests in `tests/unit/test_pypi_publish_gate.py`
- `make test` all green
- actionlint clean on the new step
- Local CodeRabbit review: 1 finding (doc recovery path), fixed

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01MSW9yjwGnzz1S8MgCGm6t5

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 12:56:24 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 60687d6cd6 feat(theme): editable + previewed end screen; optional title/end screens (v0.268.0) (#1134)
## Summary
Randy Vild (tenant `randy-vild`) emailed: *"I have completed 3 songs but
all three songs produce the large text at the end even though I have in
my settings smaller text like in my beginning."*

Root cause: the theme editor's size/position sliders only edit
`intro.*`. The end screen has its own `end.*` layout (title 350 /
closing message 400 / yellow capitals artist in Randy's theme). The
editor neither exposed it nor previewed it ("The end screen isn't
previewed").

## Changes
- **End screen tab**: own title/artist colours, size + position sliders
(closing message, song title, artist), show/hide song title / artist
name, capitals toggles, and **Copy title card layout**. That copies the
title card's regions/colours/capitals and moves the closing message just
above the title, or below the artist if there's no room, so they never
overlap.
- **End screen preview**: `theme_preview_service.render_end_screen` uses
the same `VideoGenerator.create_end_video` as `screens_worker`. The
preview column shows the screen for the active tab plus the lyrics
frame; Advanced shows all three.
- **Leave out the title card / end screen**: optional `intro.enabled` /
`end.enabled` (absent = included; not added to the DEFAULT_*
required-field lists). The setting is honoured end to end:
- The screens worker skips the omitted screen and records
`state_data.screens_included`.
- The video worker's prerequisite check and the `/retry` ladder
(`_has_title_screen`, which also requires `screens_progress: complete`)
treat an omitted screen as done.
- The orchestrator sets the omitted screen's path to `None` and gives
the original-vocals guide an intro offset of 0.
- Encoding passes explicit `include_title_screen` /
`include_end_screen=false` to the GCE worker. Without them, its loose
`*title*.png` / `*end*.png` globs would rebuild the screen from
`style/*_background.png`.
- `LocalEncodingService.encode_lossless_mp4` builds the concat from
whichever segments exist (1–3).
  - The portrait renderer drops omitted cards.
- 11 new strings × 33 locales (`endScreenHint` removed). Docs: API.md,
README.md.

## Verification
- `make test` all green: frontend 1665 tests; backend unit, integration
and emulator suites.
- New `backend/tests/test_theme_screen_omission.py`, 28 tests:
  - screens worker for all 4 include combinations;
  - prerequisites and the retry helper;
  - orchestrator config and encoder options;
  - GCE `run_encoding` with decoy theme PNGs present;
  - concat for n = 1/2/3.
- Real-ffmpeg renders:
- end-screen preview (its own background; text size changes only the end
screen);
  - portrait durations with screens omitted.
- Frontend: editor tests for the end tab, copy layout, region toggles
and enable switches, plus unit tests for the `copyTitleCardLayoutToEnd`
util.
- Rendered Randy's real theme locally:
  - as saved, the end screen reproduces the large text;
  - after Copy title card layout, it matches his title card.
- Local CodeRabbit review: 0 findings.

## Deploy notes
- The worker paths changed (`gce_encoding/`,
`local_encoding_service.py`, `karaoke_gen/`), so CI blue-green redeploys
the encoding worker.
- Existing themes are unaffected: they have no `enabled` key.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 15:09:25 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 8490ab75eb chore(gemini): move every Gemini call from Vertex AI to the Gemini Developer API (#1108)
## Summary
Moves **every** Gemini call in gen off **Vertex AI** (billed to the
`nomadkaraoke` GCP project, ~$85/mo) onto the **Gemini Developer API**
using the `gemini-api-key` Secret Manager secret (an AI Studio key
billed outside this project). Once deployed and verified,
`aiplatform.googleapis.com` / `generativelanguage.googleapis.com` get
disabled in nomadkaraoke.

### Key injection
- `backend/services/gemini_client.get_genai_client(timeout_ms)` — key
from `GEMINI_API_KEY`, else Secret Manager `gemini-api-key` via
`settings.get_secret` (runtime SA already has project-level
`secretAccessor`). No `project`/`location`.
- Service: CI `--set-secrets ... GEMINI_API_KEY=gemini-api-key:latest`.
- Jobs: Pulumi secret env on `lyrics-transcription-job`,
`video-encoding-job` (renders + translations), `nomad-error-monitor` —
**already applied** (targeted `pulumi up`, only that env added). Other
jobs fall back to Secret Manager.
- Local scripts (`frontend/scripts/translate.py`,
`scripts/measure_autocorrect_cost.py`,
`scripts/test_agentic_correction.py`): env, else `gcloud secrets
versions access` (in-process only).
- **Rotation:** add a new secret version; new instances / job executions
pick it up. No code change.

### Degradation (quota / billing / invalid key; per-minute 429s stay
retryable)
| Caller | Behaviour |
|---|---|
| auto-correct | non-retryable 503 → compare mode drops the Gemini leg,
keeps Opus |
| match-judge / free-text / parse-titles / tenant bulk | existing
fallbacks (no suggestion / blanks / regex) |
| credit eval | `pending_review` (fail-closed, unchanged) |
| lyrics translation | no retry; render proceeds without translation,
status recorded |
| custom lyrics | clean 503 "temporarily unavailable…" (+ 5-min timeout
so it can't hang) |
| error-monitor LLM | no retry; alerts sent without AI summary |
| translate.py | exit 2 "Gemini quota/credit exhausted — top up AI
Studio or rotate gemini-api-key" |

`note_gemini_failure()` also sends one Discord ops alert per 6h
(Firestore-throttled across instances via new
`ops_alerts.send_throttled_ops_alert`).

### Infra
- Pulumi: removes `roles/aiplatform.user` from `karaoke-backend` —
**will be applied after this deploys and Vertex traffic is confirmed at
zero**.
- Agentic correction default model `gemini/gemini-3.8-flash`
(`vertexai/` prefix still accepted, routed to the Developer API).
- Version 0.261.1 → 0.262.0. Docs: CLAUDE.md, LESSONS-LEARNED, README,
ARCHITECTURE, API, etc.

## Testing
- New: `backend/tests/services/test_backend_gemini_client.py`
(classifier, key resolution, no-Vertex client kwargs, alert throttle,
per-caller degradation),
`tests/unit/lyrics_transcriber/test_agentic_gemini_provider.py`,
`frontend/scripts/test_gemini_client.py`, lyrics-translation quota test.
- `pytest backend/tests --ignore=backend/tests/emulator` — all pass;
`pytest tests/unit` — 2021 passed.
- Verified `gemini-3.8-flash` is listed on the Developer API and
answered a 5-token test call with the key.
- `translate.py --dry-run` on `frontend/messages` and
`backend/translations` OK.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Updates**
* AI-powered lyric translation, custom lyrics, auto-correction, title
parsing, match judging, and related features now use the Gemini
Developer API.
* AI features use the configured Gemini API key. Quota, billing, or key
issues may show an unavailable message or use an existing fallback
instead of repeated attempts.
* **Documentation**
* Updated product, API, setup, and recovery guidance to reflect Gemini
API key configuration and usage.
* Updated guidance for handling AI service availability and quota
issues.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 21:58:53 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 d4fcab521c Translated lyrics follow-ups + review Waveforms 'separating' spinner (v0.261.0) (#1103)
## Summary
Follow-ups from Andrew's test of translated lyrics (job 948b6760), plus
a review-page UX gap found while investigating that job.

- **Translation row is now secondary:** it renders at 70% of the unsung
colour's opacity, with outline and shadow faded to match. A theme's
`karaoke.translation_color` still wins.
(`subtitles.py:TRANSLATION_OPACITY`)
- **YouTube label:** published videos with translations get "(With
Translation into X)" in the title, before "(Karaoke)". Only the
artist/title get truncated, never the label. The description gets an
up-front notice. This covers the live upload and the quota queue, and
applies only when `state_data.lyrics_translation.status ==
"translated"`. Turning translation off and re-rendering clears the
outcome.
- **Review preview modal:** an always-visible note explains the preview
is a quick black-background sync check, not the final look. Jobs with a
translation language also get a note that translations are added after
review.
- **Waveforms toggle:** shows a spinner and tooltip while audio
separation is still producing the vocal stem (peaks endpoint returns
202). Previously the rows just stayed empty.

## Testing
- Jest: lyrics-review suite (245+) incl. new
TranscriptionView/Preview/loader tests
- pytest: youtube description/queue/orchestrator, lyrics_translation,
translated-lyrics unit tests
- Linux libass render tests (Debian + prod static ffmpeg) pass
- Independent review agent: 2 findings fixed (Tooltip data-state
clobbering toggle state; stale translation outcome)

No local CodeRabbit run (CLI has no seat on this org), so leaving this
open to the CodeRabbit bot.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* YouTube titles and descriptions now identify the language of
translated lyrics. Titles are kept within the platform’s length limit.
  * Translated lyrics appear with softer opacity by default.
* The lyric-review screen explains how the quick preview differs from
the final video, including that translated lyrics are added after
review.
* A spinner and notice indicate when vocal waveforms are still being
prepared.

* **Bug Fixes**
* Translation status now reflects when translation is turned off,
preventing stale translated-language labels.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 16:39:56 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 bc1fc774eb Translated lyrics: don't split Japanese/Chinese translations mid-word (v0.260.1) (#1102)
## Summary
Follow-up to #1101, found while verifying it on prod (job ca2e881a,
English song → Japanese): when a long lyric line is split across rows,
its translation is shared between the pieces, and unspaced scripts were
cut at an arbitrary character — e.g. 無|駄 ("waste") across two rows,
worse in portrait.

`split_text_proportionally` now scores each split point for unspaced
scripts: free after punctuation (、。!?…), cheap after a Japanese particle
(を は が に…) or before a kanji/katakana word start following hiragana, a
full "part" of cost for a mid-word cut. It also never leaves a piece
empty when there are enough characters.

Before → after (real prod line): `ねえ、おかしいでしょ、無` / `駄な時間を省いてあげる` →
`ねえ、おかしいでしょ、` / `無駄な時間を省いてあげる`.

Thai can still split inside a word (no word-boundary data without a
dictionary).

## Testing
- New unit tests for the prod case and a particle-boundary case; all 201
lyrics output tests pass.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Improved how translated lyrics are split across rows for unspaced
text. Splits now favor punctuation and natural language boundaries,
helping avoid cutting through words where possible.
  * Thai text may still be split within words.
* **Chores**
  * Updated the project version to 0.260.1.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 14:39:13 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 f8d0febcf5 Translated lyrics beneath the karaoke lyrics for language learners (v0.260.0) (#1101)
## Summary
New job-submission option: **Add translated lyrics** → pick a language
(33 UI locales) → the karaoke videos show an AI translation of each
lyric line in smaller text beneath the synced lyrics, so language
learners can sing songs in another language while learning what the
words mean.

- **Job creation**: toggle + language picker (default = UI language) on
the Customize step, with a **live preview** of a real rendered frame
(`GET /api/themes/translation-preview?language=xx`; pre-generated sample
translations, no LLM call at preview time). Any job, public or private,
free for now. `translation_language` accepted by every creation route;
admin can edit it.
- **Post-review translation**: right after the job enters
RENDERING_VIDEO, the render worker translates the *final reviewed*
lyrics line by line with Gemini
(`backend/services/lyrics_translation.py`, `LYRICS_TRANSLATION_MODEL`,
GCS-cached so theme/admin re-renders are free) into
`jobs/{id}/lyrics/translations.json`. Not editable. A failure never
blocks the render (renders without translations, recorded in
`state_data.lyrics_translation`); a song already in the target language
gets no row.
- **Rendering** (landscape + portrait, GCE worker + local fallback):
`LyricsSegment.translation`; when present, slots get taller and screens
show 3 lines; a `Karaoke.Translation` style (unsung colour, ~½ size)
draws one static event beneath each line, sanitized, wrapped/scaled to
fit. Jobs without translations render identically.
- **RTL + LTR together**: the translation is its own ASS event with auto
paragraph direction, so e.g. a Hebrew song keeps its right-to-left
highlight with an English translation beneath (and vice versa). Long
lines split by the resizer share their translation across the pieces
(grapheme-safe for Thai/CJK).

Design: `docs/archive/2026-10-01-translated-lyrics-plan.md`.

## Testing
- `make test` all green (root 2014, backend 5416, emulator 150, frontend
1583).
- New: `tests/unit/lyrics_transcriber/output/test_translated_lyrics.py`
(layout/positions/style/sanitizing/fit/mixed RTL-LTR/Thai/portrait),
`test_translated_lyrics_render.py` (libass pixel tests incl.
Hebrew/Arabic lyrics with English/Arabic translations and CJK),
`backend/tests/test_lyrics_translation.py`,
`backend/tests/test_render_video_worker_translations.py`, GCE endpoint
test, frontend CustomizeStep/GuidedJobFlow/api tests, prod E2E
`e2e/production/translated-lyrics.spec.ts`.
- `scripts/run-render-tests-linux.sh`: 135 passed on Debian ffmpeg AND
prod's static ffmpeg (no skips).
- Real-data check: job 5710831e (Hebrew) translated to English by Gemini
and rendered with the nomad theme — RTL highlight intact, English
beneath, split line shares translation.

Local CodeRabbit CLI unavailable in this workspace (no seat), so leaving
this PR for the CodeRabbit bot; an independent local review was done and
its findings fixed in the second commit.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added an opt-in translated-lyrics option to job setup, with language
selection and a preview.
* Translations are generated after lyric review and appear beneath
lyrics in landscape and portrait videos, with text layout for multiple
languages and writing directions.
  * Translation settings can be changed or cleared by admins.
* **Bug Fixes**
* If translation fails or is unavailable, video rendering continues
without translated lyrics.
* **Documentation**
* Updated API and product documentation with translated-lyrics setup and
behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 13:41:54 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 354e43bab2 Render Hebrew/Arabic karaoke right-to-left; fix title-card tofu (v0.255.0) (#1094)
## Summary

A Hebrew KaraokeHunt job (5710831e, Omer Adam – "שני משוגעים") shipped
with its lyrics **laid out and highlighted left-to-right** and a **title
card of "?" boxes**. Diagnosed by measuring rendered pixels (not by
eye), reproduced locally and with prod's static ffmpeg 7.0.2 in Docker.

**Root causes**
- ASS style `Encoding=0` makes libass force LTR paragraphs; the `{\kf}`
before each word splits the line into runs, so each word is shaped
correctly but the words go left→right.
- `\kf` always wipes left→right within a word
([libass#406](https://github.com/libass/libass/issues/406)).
- PIL (title/end cards) has no per-glyph font fallback; the theme font
(Avenir Next) has no Hebrew glyphs.

**Fixes**
- `build_karaoke_styles` forces `Encoding=-1` (libass auto paragraph
direction; LTR unchanged).
- RTL lines get `{\frz(180+angle)\frx180\fry180}` — a visual no-op that
makes libass ≥0.15 sweep fills right→left without splitting words
(Arabic still joins). Theme angle preserved; re-emitted after duet
`{\r}` resets.
- Lead-in indicator mirrored for RTL lines.
- Text measurement mirrors libass: per-glyph fontconfig fallback, size
scaled from each font's metrics (old fixed 0.70 was 20%+ off for
fallback fonts).
- New `karaoke_gen/utils/font_fallback.py` (fontTools cmap +
fc-match/fc-list) used by title/end cards, portrait headers and ASS
measurement.
- Found by the new tests: inter-word gaps ≤0.1s were dropped, so the
highlight ran up to ~0.2s early by line end (all languages). Karaoke
tags now derive from one absolute centisecond timeline.
- Fixed pre-existing `test_render_video_worker_gce` failure on main
(MagicMock generation → `_g<mock>` id).

## Testing
- **Pixel-level render tests** (`test_ass_render_highlight.py` +
`ass_render_harness.py`): real `SubtitlesGenerator` output rendered with
ffmpeg/libass in a pure-colour palette, asserting on frames for English,
Hebrew, Arabic, Chinese, Japanese, Korean + an RTL duet line: highlight
starts at the reading-start edge, sung pixels always precede unsung in
reading order (word order + in-word fill direction), monotonic and in
time with the words, lead-in on the correct side, no missing glyphs, RTL
tags don't move the line (incl. rotated themes).
- 32 fail on pre-fix main; mutation checks ("Encoding only", "no re-emit
after `\r`", "ignore theme angle") each fail exactly the expected cases.
- 123/123 pass on Linux against **both** Debian ffmpeg (CI-equivalent)
and **prod's johnvansickle static ffmpeg** via
`scripts/run-render-tests-linux.sh`.
- CI now installs `fonts-noto-core fonts-noto-cjk`; under `CI=true`
these tests fail rather than skip on missing preconditions. (macOS
libass uses CoreText → font-sensitive checks skip locally.)
- Title-card tests (`test_font_fallback_render.py`): no `.notdef` tofu
for Hebrew/Arabic/CJK titles; Hebrew title reads right-to-left.
- Unit tests: `test_rtl_ass_output.py`, `test_font_fallback.py`.
- `make test`: unit 1972 passed; emulator 150 passed; frontend 1491
passed.
- Full local re-render of job 5710831e reviewed by Andrew: RTL highlight
smooth and correct, title card correct.

## Follow-ups (not in this PR)
- CDG output still renders RTL left-to-right / tofu (separate renderer).
- Separate investigation: transcription dropped 3 sung lines in this
job, which the section detector then labelled INSTRUMENTAL.

No local CodeRabbit review (CLI has no seat for this org), so leaving
this open for the CodeRabbit bot. A local code review ran; its findings
are addressed in the second commit.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Karaoke lyrics now render more reliably for Hebrew, Arabic, Chinese,
Japanese, and Korean, with improved reading-direction highlighting,
timing, and lead-in placement.
* Titles and portrait text can use fallback fonts when the selected font
lacks the necessary characters.

* **Bug Fixes**
* Improved lyric centering, word highlighting, and glyph rendering
across supported scripts.

* **Documentation**
  * Added guidance on lyric rendering and multilingual test coverage.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 21:02:53 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 fc05571779 fix(theme-preview): skip ffprobe for no-audio ASS generation (v0.249.1) (#1084)
## Summary
Error-monitor alert: `Failed to get audio duration: ...
/tmp/theme-preview-*/no-audio.flac`. The tenant theme editor (#1080)
passed a nonexistent audio path to `SubtitlesGenerator.generate_ass`, so
every uncached preview ran ffprobe on a missing file and logged at ERROR
before falling back to the segment-derived duration. Previews rendered
fine (200); this was pure log noise.

## Changes
- `generate_ass` / `_get_audio_duration` accept `audio_filepath=None` →
skip ffprobe, use `last segment end + 30s` silently. Real-audio callers
unchanged (a failing probe still logs ERROR).
- `theme_preview_service` passes `None`.
- Version 0.249.0 → 0.249.1.

## Testing
- New
`tests/unit/lyrics_transcriber/output/test_subtitles_audio_duration.py`
(None path: no probe, no error log; failed probe still logs;
`generate_ass(..., None)` end-to-end).
- `make test-unit test-backend-unit`: 1846 + 5016 passed.
- Local CodeRabbit review: no findings.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 16:34:46 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 13ecce4f79 feat: monitor karaoke-decide lb-refresh job errors (v0.245.4) (#1073)
## Summary
Adds `lb-refresh` (karaoke-decide's new ListenBrainz statistics →
BigQuery Cloud Run Job) to `MONITORED_CLOUD_RUN_JOBS`, so a failed run
or its stale-data ERROR reaches the error monitor and Discord. This
matches what #1067 did for `mb-refresh`.

## Changes
- `backend/services/error_monitor/config.py`: add `lb-refresh`
- `tests/unit/services/error_monitor/test_config.py`: expect 6 jobs and
include `lb-refresh`
- Version 0.245.4

## Testing
- `pytest tests/unit/services/error_monitor/`: 361 passed

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 00:54:15 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 5b2a0f8c3e feat(error-monitor): watch decide's mb-refresh job (#1067)
## Summary
- Adds `mb-refresh` (karaoke-decide's new weekly MusicBrainz → BigQuery
Cloud Run Job, nomadkaraoke/karaoke-decide PR to follow) to
`MONITORED_CLOUD_RUN_JOBS`, so failed refreshes and staleness warnings
raise error-monitor patterns / Discord alerts.
- Version bump 0.245.0 → 0.245.1.

## Testing
- `pytest tests/unit/services/error_monitor` — 361 passed (count +
membership assertions updated).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Reliability**
* The weekly MusicBrainz data refresh is now included in service
monitoring. Issues with the refresh can be detected alongside problems
with other monitored services, improving visibility into the
availability of refreshed music data.
* **Release**
  * This update is version 0.245.1.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-27 23:07:47 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 c138c9d1ac feat(review): vocal waveforms for own-instrumental jobs (v0.244.1) (#1062)
## Summary
Jobs with a user-supplied instrumental (`existing_instrumental`) skip
GPU separation. That left the lyrics review **Waveforms** mode with no
vocals stem, so `/vocals-peaks` returned 404 and no strips were drawn.
The waveform only needs a peak envelope, so a CPU approximation is
enough.

- **`backend/services/derived_vocals.py`**: decodes the mix and the
instrumental to mono at 22.05 kHz (ffmpeg). It aligns them by FFT
cross-correlation (numpy, ±2s search), fits a least-squares gain, and
subtracts, leaving roughly the vocals. It writes a FLAC and logs offset,
gain and residual ratio. A few seconds of CPU.
- **Lyrics worker**: for existing-instrumental jobs, runs this in a
thread alongside transcription. It uploads
`jobs/{id}/stems/vocals_derived.flac` and registers
`stems.vocals_derived`, and it's awaited before advancing to screens so
the screens step pre-computes peaks as usual. Best-effort: failures log
and the job continues.
- **`vocals_stem_path`**: adds `vocals_derived` as the last-resort key.
It's waveform-only: deliverable stems, the original-vocals guide and the
auto-approval timing gate don't read it.

## Testing
`backend/tests/test_derived_vocals.py` covers:
- offset recovery for positive, negative and large lags
- gain and vocals recovery on synthetic signals
- an unrelated instrumental leaving the mix intact
- a real ffmpeg round trip (24-bit stereo WAVs → mono 22.05 kHz FLAC)
- stem precedence (never overrides real separated vocals)
- worker upload and registration, plus the non-fatal failure path

The full backend unit suite passes in an isolated venv (4812 passed).
The only exceptions are 6 tests needing the spaCy `en_core_web_sm`
model, which isn't installed in that venv; CI has it.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-26 23:04:15 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 8cbcc03099 infra: GCP cost cuts (VPC connector, min-instances, Autoclass, weekly incremental git backup, hourly error monitor, PITR off) (#1063)
## Summary
These are the cost cuts Andrew approved to get the whole GCP project
under $300/mo. Performance compromises were accepted.

- **VPC connector removed.** `cloud-run-connector` only existed to reach
the old flacfetch GCE VM's private IP. flacfetch now runs on Netcup
behind a public HTTPS URL. I checked every URL secret the backend uses
and none are private IPs. The backend is detached with
`--clear-vpc-connector`: just dropping the `--vpc-connector` flag from
`gcloud run deploy` would keep the old setting. `audio-download-job` and
`bulk-search-job` no longer set `vpc_access`. Saves ~$12/mo plus the
Network Intelligence resource-hours.
- **`karaoke-backend` min-instances 2 → 1** (ci.yml).
- **Autoclass** on `karaoke-gen-storage-nomadkaraoke` (terminal
NEARLINE, because customers re-download old jobs) and on
`nomadkaraoke-divebar-files` (terminal ARCHIVE). Autoclass charges no
retrieval or early-deletion fees.
- **backup-to-aws**
- **Git bundles:** now weekly (Sundays) and incremental. A repo is
re-bundled only when its GitHub `pushed_at` is newer than its bundle's
S3 `LastModified` minus a 1h lag, so S3 itself is the state store. Every
repo is re-bundled on the first Sunday of each month.
`?mode=git_repos[&full=1]` runs the git step alone.
- **Bug fix:** since #997 every S3 **overwrite** has failed. The
`backup-writer` IAM user has no `s3:GetObject`, so HEAD on an existing
key returns 403. Git bundles and the manifest were stuck at 09-15, and
the DR freshness monitor has been red every day. The existence check now
uses `ListObjectsV2`.
- **`gcs_sync`:** detects changes by `time_created` instead of
`updated`. Autoclass transitions bump `updated`, which would have made
the sync re-copy every object and pull it back to Standard.
- **DR freshness monitor:** `git-repos/` is now judged on
`manifest.json` (192h) plus a check of the manifest's contents.
- **Error monitor:** runs hourly instead of every 15 min. Lookback is 60
min, equal to the interval, so there are no gaps or double counts.
`MAX_LOG_ENTRIES` goes from 500 to 2000.
- **Firestore PITR disabled.** The nightly export to GCS staging
remains, plus the weekly copy to S3.
- **`recover-stuck-downloads` / `retry-pending-render-jobs`:** every 5 →
10 min. Both are recovery sweeps with ~10-min stall thresholds.
- Docs updated: DISASTER-RECOVERY (PITR restore replaced, git RPO,
incident write-up), ARCHITECTURE, LESSONS-LEARNED,
GCP-COST-OPTIMIZATION, TROUBLESHOOTING.

**Kept:** the `audio-separator` GPU service. Its request logs show ~40
external `/separate` calls/month from remote-CLI users. It's already
min-instances 0.

## Already applied to prod (before merge)
- `gcloud run services update karaoke-backend --clear-vpc-connector
--min-instances 1`, same image. Health is OK, and
`/api/health/flacfetch` and `/api/health/encoding-worker` are both OK
without the connector.
- Targeted `pulumi up` on the Firestore DB, both buckets, both jobs and
both 5-min schedulers.
- backup-to-aws redeployed with `deploy.sh`. A manual `?mode=git_repos`
run bundled 13 of 121 repos (106 unchanged, 2 skipped) and uploaded 120
files with 0 errors.
- Deleted `gs://…/backups/flacfetch-torrent-backup-20260122-1931.tar.gz`
(1 GB).

## After merge
Targeted `pulumi up` for `error-monitor-scheduler`, which must wait for
the image with LOOKBACK=60, then delete the connector and drop the
`vpcaccess` API service from state (`disable_on_destroy=False`, so the
API itself stays enabled).

## Tests
`make test` passes locally. The backup_to_aws function tests (43,
including new incremental, list-based-existence and gcs_sync tests) were
run separately because CI doesn't run them.

Related: nomadkaraoke/karaoke-decide#140 (ARCHIVE lifecycle on
`nomadkaraoke-data`, already applied).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Updates**
* Pending-render retries and stuck-download recovery now run every 10
minutes instead of every 5 minutes.
  * Error monitoring now runs hourly and checks a 60-minute window.
* Cloud Run candidate deployments use one minimum instance instead of
two and no longer use a VPC connector.

* **Backups and Recovery**
* Git repository backups now run weekly, with monthly full refreshes and
incremental updates between them.
* Firestore point-in-time recovery is disabled; recovery uses nightly
exports.
  * Storage buckets now use automatic storage-class management.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-26 22:59:06 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 5d56ee9069 fix(frontend): stop reporting opaque cross-origin "Script error." (v0.238.6) (#1050)
## Summary
The error monitor keeps alerting on a frontend pattern
`@https://gen.nomadkaraoke.com/` with sample `Error: Script error.` (iOS
Safari, 4 occurrences since June). That's the browser's opaque
cross-origin error: a script from another origin (a third-party tag,
in-app browser or extension injection) threw. The browser hides
everything, so there's no error object, filename or line. Our
`window.onerror` handler wrapped it in `new Error(event.message)`, so
the only "stack" frame (`63bfc60a…js:6:874`) is our own handler. There's
nothing actionable in these reports.

## Changes
- `frontend/lib/client-error-setup.ts`: new exported
`isOpaqueCrossOriginError(event)`. The window `error` listener returns
early when `event.error == null` and the message is `Script error.`.
- `backend/services/error_monitor/frontend_ingestion.py` +
`backend/api/routes/client_errors.py`: `is_opaque_script_error()` drops
bare `Script error.` / `Error: Script error.` reports at ingestion.
They're accepted (202) but not persisted, the same way the bot filter
works. This also stops older cached bundles from alerting.
- `docs/ARCHITECTURE.md`: a note on the crash-report noise filters.
- Version 0.238.5 → 0.238.6.

## Testing
- New Jest `lib/__tests__/client-error-setup.test.ts`: tests the
predicate; an opaque event is not reported; a real error is still
reported. Full Jest: 1342 passed.
- Backend: parametrized API tests (dropped vs "Script error in …" still
persisted) and unit tests for the predicate. `tests/unit` passed (1834),
and `backend/tests` without the emulator passed (4591). Emulator tests
are left to CI.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Opaque cross-origin “Script error” events without an error object and
location details are no longer recorded. Other client errors, including
script errors with location details or different messages, continue
through the reporting flow.
* Matching is case- and whitespace-tolerant and allows an optional
trailing period.
* **Documentation**
* Updated Error Monitor guidance to describe which browser and
server-side errors are ignored.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-25 13:58:03 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 bcdfd6b6bb fix(jobs): unstick audio-edit jobs — waveform OOM, retry skip, screens handoff (#1049)
## Summary
Job `2c1b922e` (Dave Matthews Band – Why I Am, 24-bit/96kHz FLAC, audio
edit requested) stalled twice in prod. Three bugs:

1. **Waveform OOM** — `WaveformGenerator.generate_data_only` decoded the
full source at native resolution through pydub. For a 6-min hi-res FLAC
that peaked at ~1 GB RSS and OOM-killed the 1 GiB `audio-download-job`
while pre-generating editor assets. It now decodes straight to 16 kHz
mono PCM via ffmpeg and computes the envelope with numpy: **156 MB vs
1.07 GB peak** on the real file, amplitudes within 0.003 of the old
output. The same function backs the editor's on-demand waveform in the
API.
2. **Retry skipped unfinished audio-edit prep** — the Cloud Run
auto-retry saw `downloading`, treated the download as done and exited 0,
leaving the job at `downloading` forever. A retry that finds an
audio-edit job still at `downloading` now resumes the editor prep. That
path triggers nothing downstream, so it's safe. Non-edit jobs keep the
existing skip (downstream may already be triggered).
3. **Screens handoff ignored `audio_edit_complete`** —
`advance_to_screens_if_ready` (added in #1002) and the
`recover-stuck-jobs` lost-screens sweep only accepted `downloading`.
Audio-edited jobs prep at `audio_edit_complete`, so **every audio-edited
job since Sept 15 stalled** after lyrics and separation finished (log:
`screens not (re)triggered — status=audio_edit_complete`). Both now use
a shared `PREP_PHASE_STATUSES`.

2c1b922e was unstuck manually (admin reset → `awaiting_audio_edit`, then
a direct `/api/internal/workers/screens` dispatch). Added a
TROUBLESHOOTING entry covering both signatures and the manual fixes.

## Testing
- New tests: hi-res 96k/24-bit decode path (low-rate mono args, envelope
matches pydub reference, silent → 0, undecodable → error);
download-worker resume vs skip; handoff at `audio_edit_complete`;
watchdog recovers `audio_edit_complete` and queries both statuses.
- `pytest tests/unit/` → 1824 passed, 8 skipped
- `pytest backend/tests/ --ignore=backend/tests/emulator` → 4586 passed,
36 skipped
- Version 0.238.4 → 0.238.5

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Audio-edit jobs can resume preparation after an interruption without
redownloading existing audio.
* Screen generation can proceed after audio editing is complete, and
stalled screen-generation triggers can be recovered.
* Waveform previews are more reliable for high-resolution and silent
audio. Invalid audio and decoding timeouts now produce clear errors.
* **Documentation**
* Added troubleshooting guidance for jobs stuck during audio editing or
before screen generation.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-24 23:39:25 -04:00
Andrew Beveridge ☄️andClaude Fable 5 d85b6e271c feat: precomputed vocals peaks — Waveforms strips paint sub-second (#1031)
## Summary

Final piece of the review-page speed work (#1024 → #1028 → #1029): the
Waveforms-mode strips no longer require downloading + WebAudio-decoding
the entire vocals stem (~7.5MB, ~4–6s) — the backend now serves a
**precomputed 400-peaks/s envelope as ~150KB of uint8/base64 JSON**,
cached in GCS and pre-warmed at prep time, so strips paint sub-second.

## How

- **`WaveformGenerator.generate_peaks()`** — numpy max-abs envelope,
bucket-for-bucket identical to the frontend's `computePeaks` (same
drift-free fractional-stride bucketing; exact match vs a naive reference
in tests).
- **`AudioAnalysisService.get_vocals_peaks()`** — GCS cache at
`review-audio/vocals_peaks_400.json`, validated against the stem path;
miss decodes the small transcoded OGG; upload best-effort.
- **`GET /api/review/{job_id}/vocals-peaks`** — off the event loop under
the shared waveform semaphore; `202 Retry-After: 15` while separation
runs (mirrors `/audio/vocals`); `Cache-Control: private, max-age=600`;
`peaks_per_second` clamped 50–1000.
- **`screens_worker`** pre-computes the envelope next to the existing
OGG + waveform pre-warm, so it's a pure cache read by the time a job
reaches review.
- **Frontend** — `VocalsAudioDataLoader` prefers the peaks endpoint (one
quick retry on transient errors) and **falls back to the original full
download+decode** on any terminal peaks failure, so a peaks-side problem
can never cost a user their waveforms. Local mode (no peaks URL) uses
the old path unchanged. Vocal stem selection factored into
`backend/utils/stems.py` (shared by routes + worker).

## Testing

- Package: 4 tests incl. exact naive-reference parity (**106 passed**)
- Backend: 4 service + 4 endpoint tests (**4518 passed**)
- Frontend: `fetchVocalsPeaks` units + 5 loader-behavior tests (prefers
peaks / falls back / one-retry-then-fallback / 202 polling / audio-only
mode) (**1307 passed**)
- `review-fast-load.spec.ts` prod E2E extended with a cached
vocals-peaks latency/shape check
- Post-deploy: will verify the endpoint on a real in-review job and
confirm payload size + latency

## Notes

- Existing in-review jobs have no cache yet — their first Waveforms open
computes + caches lazily (bounded); every open after that is sub-second.
New jobs are pre-warmed.
- 8-bit amplitude quantization is visually lossless at strip heights
(14px).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-20 13:08:19 -04:00
Andrew Beveridge ☄️andClaude Fable 5 527d55ef63 fix: preserve source audio extension for temp downloads (ffmpeg extension-fallback failure) (#1023)
## Summary

Root-cause fix for the 2026-09-17 19:47 ET error-pattern cluster (job
`e026e727`): a paying user's uploaded MP3 had a ~1.2 MB ID3 tag
(embedded cover art from a ytmp3 rip). The backend downloads job audio
to temp files hardcoded as `*.flac` — ffmpeg's content probe is
inconclusive inside the huge tag, so it falls back to the **file
extension**, commits to the flac demuxer, and fails with `Could not find
codec parameters for stream 0 (Audio: flac, 0 channels)`. That single
file broke every consumer of the input audio:

- `GET /api/review/{id}/waveform-data` → 500 ("Decoding failed. ffmpeg
returned error code: 8")
- Review OGG transcode → "Cannot determine format of input 0:0 after
EOF" (fell back to raw FLAC serving)
- Preview video encode → failed on **both** GCE worker and local
fallback ("Invalid data found when processing input")
- Duration probe → `Failed to get audio duration: 'duration'` (KeyError)

**Repro:** the file is a perfectly valid 227 s MP3 — `ffprobe` succeeds
when the copy is named `.mp3` or has no extension, and fails when named
`.flac` (verified on ffmpeg 8.0 locally and 7.0.2-static in prod). The
`majorbrand: dash` metadata is a copied ID3 tag, not a mislabeled MP4.

## Changes

- New `backend/utils/audio_filenames.py` —
`local_audio_filename(gcs_path, stem)` names local copies with the
source object's real (lowercased) extension; unknown/missing extension →
bare name, forcing pure content probing.
- Applied at every site that downloads job audio to a temp file:
- `backend/services/audio_analysis_service.py` (waveform data/image,
analysis, mute-region waveform, stem comparison)
- `backend/services/audio_transcoding_service.py` (review OGG transcode)
- `backend/api/routes/review.py` `_prepare_preview_inputs` (preview
local encode)
- `backend/services/gce_encoding/main.py` (GCE preview + render audio
download)
- `backend/workers/render_video_worker.py`,
`backend/workers/audio_worker.py`, `backend/workers/lyrics_worker.py`
- `docs/TROUBLESHOOTING.md`: new runbook entry for this failure class.
- Version bump 0.229.3 → 0.229.4.

## Testing

- New `tests/unit/test_audio_filenames.py` — 9 tests: helper behavior
(extension preservation, lowercasing, unknown/missing-extension
fallback) + service-level assertions that the transcoding and analysis
services download to source-matching temp names.
- Full `make test` suite passes locally (backend + frontend, 105 suites
/ 1286 frontend tests).

## Impact / rollout

- No data repair needed for job `e026e727` — its stems/lyrics are
intact; the review page starts working once this deploys.
- `gce_encoding/main.py` ships in the wheel; the encoding worker's
per-job wheel version check picks it up (restart `encoding-worker`
service to force immediate pickup).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **Bug Fixes**
- Improved audio processing for files with non-FLAC extensions,
including MP3 and OGG files.
- Preview rendering, waveform generation, transcription, and audio
conversion now preserve source file extensions, reducing decoding
failures.
  - Audio conversion fallbacks now use the correct source content type.

- **Documentation**
- Added troubleshooting guidance for waveform and preview failures
related to audio metadata or file extensions.

- **Tests**
- Added coverage for extension handling across audio download and
processing workflows.

- **Release**
  - Updated the application version to 0.229.4.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-17 21:40:36 -04:00
Andrew Beveridge ☄️andClaude Opus 4.8 303ff7e728 fix(lyrics-cache): resolve cache_dir at instantiation — stop burning AudioShake credits on every job (#1017)
## Problem

The cross-container AudioShake/lyrics transcription cache has been
**completely non-functional in production**. Every lyrics job logs
`Cache sync from GCS: 0 downloaded, 14 not found` and the GCS
`lyrics-transcriber-cache/` prefix is **empty** — so every job
(including repeat songs and every daily/canary E2E run of the same
track) makes a fresh, credit-spending AudioShake alignment call.

## Root cause

`OutputConfig.cache_dir` used a bare
`os.getenv("LYRICS_TRANSCRIBER_CACHE_DIR", ...)` as its dataclass field
default. Dataclass defaults are evaluated **once at import time**. The
cloud lyrics worker sets `LYRICS_TRANSCRIBER_CACHE_DIR` to a per-job
temp dir at **runtime** (`backend/workers/lyrics_worker.py:316`), long
after the module imported — so it was ignored.

Result: the transcriber wrote its cache to the frozen default
`~/lyrics-transcriber-cache`, while the GCS-sync agent read/wrote
`{temp_dir}/lyrics-cache`. The upload step scanned the sync dir, found
nothing the transcriber had written, and uploaded nothing → GCS stayed
empty forever → every future job missed.

## Fix

Use `field(default_factory=...)` for `cache_dir` (and `output_dir`,
which had the same latent `os.getcwd()`-at-import bug) so both resolve
when an `OutputConfig` is **constructed**, honoring the worker's runtime
env. This mirrors the existing `default_factory` pattern already used
for the local-whisper fields in `TranscriberConfig`.

After this deploys, the first job for a given audio hash populates GCS,
and subsequent jobs with the same audio (all the daily/canary E2E runs,
plus real repeat songs) hit the cache and spend **zero** AudioShake
credits.

## Testing

- New regression tests
(`tests/unit/lyrics_transcriber/test_output_config_cache_dir.py`) assert
`cache_dir` tracks the env set *after* import, is re-evaluated per
instance, respects an explicit override, and that `output_dir` tracks
cwd at construction.
- Existing `tests/unit/test_lyrics_cache_service.py` still passes (21
passed).

## Verification plan (post-deploy)

Watch a lyrics job's logs for `Cache sync to GCS: N uploaded` on the
first run, then `Cache sync from GCS: >0 downloaded` on the next run of
the same audio, and confirm objects appear under
`gs://karaoke-gen-storage-nomadkaraoke/lyrics-transcriber-cache/`.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Cache and output directories now reflect the environment and current
working directory when each job starts.
* Runtime cache directory changes are honored, improving reuse of cached
transcription results.

* **Tests**
* Added coverage for cache path resolution, fallback behavior,
overrides, cache reuse, and transcription pipeline outcomes.
* Added validation for malformed countdown settings and local GPU
processing handoffs.

* **Chores**
  * Updated the project version to 0.229.1.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-09-17 15:14:15 -04:00
Andrew Beveridge ☄️andClaude Opus 4.8 37b9e6088b ci(pypi): recurring storage-cap prune reminder (fix failing PyPI publish) (#960)
## Problem

The `Deploy - Publish to PyPI` step on `main` has been failing:

```
HTTP Error 400: Project size too large. Limit for project 'karaoke-gen'
total size is 10 GB.
```

This is a recurrence of the known 10 GB PyPI project cap. The project is
at
**10.71 GB / 479 releases**; each release is a ~62 MiB universal wheel
(bundled
33-locale frontend) auto-published on every merge (~3 GB/month), so it
refills
the cap every few months. **Only the public `pip install` publish step
fails** —
Cloud Run and the GCE encoding worker (pulls the wheel from GCS) are
unaffected.

The prior fix (v0.187.5) was wheel-only publishing + a one-time manual
deletion,
which only bought ~2 months. This makes the cleanup **recurring and
durable**.

## Why not just automate deletion in CI?

PyPI has **no deletion API** (upload tokens / Trusted Publishing are
upload-only,
and 2FA is mandatory). Deletion is only possible from a logged-in
browser
session, which can't be safely/durably stored in CI. So we automate the
*decision + reminder* and keep the *deletion* a ~1-minute human step.

## What this adds

- **`scripts/prune_pypi_releases.py`** (stdlib-only) — reads the public
JSON API
and computes a retention plan: **keep last 60 days + newest release per
older
calendar month + latest overall**. Renders a table / JSON / Markdown
report and
a **paste-into-browser-console** deletion snippet. Never deletes
anything
  itself. Today's plan: keep 82, delete 397, **10.71 GB → 4.17 GB**.
- **`.github/workflows/pypi-prune-reminder.yml`** — monthly cron (+
manual
dispatch) that runs the planner and, once the project crosses a 7 GB
watermark,
opens/updates a single tracking issue with the plan + snippet. No
secrets.
- **`docs/PYPI-STORAGE-PRUNE.md`** runbook + a `TROUBLESHOOTING.md`
entry.
- **`tests/unit/test_prune_pypi_releases.py`** — 13 tests covering the
policy
(recency window, per-month keeper, latest-overall, same-day timestamp
ranking,
  empty/undated releases, sizes, rendering).

## Testing

- `pytest tests/unit/test_prune_pypi_releases.py` → 13 passed.
- Ran the planner against live PyPI data; generated JS validated with
`node --check`.
- `black` clean; workflow YAML validated; `should_nag` gate verified.

## Notes

- **No version bump** — tooling/CI/docs only, nothing in the wheel
changes.
Merging therefore **skips** the publish step (tag `v0.209.1` already
exists),
so this PR does not itself add another fat wheel while we're over the
cap.
- **To unblock publishing now:** after merge, run the prune (locally via
the
script or trigger the workflow with `workflow_dispatch`), paste the
snippet on
the logged-in PyPI releases page, then re-run the publish. The monthly
cron
  keeps it under the cap going forward.
- Related: filed
[pypi/support#12050](https://github.com/pypi/support/issues/12050)
requesting a 50 GB limit as a one-off buffer; the recurring prune is the
  durable fix.

@coderabbitai ignore

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-29 21:46:10 -04:00
Andrew Beveridge ☄️andClaude Opus 4.8 1e99f9e0d5 feat(admin): per-job/user locale badge + "emails sent to user" history (#930)
## Summary

Two admin-only features so support/intervention work can (a) know what
language/country a user is using the product in, and (b) see the exact
emails ever sent to a user, rendered as they appeared in the inbox with
full delivery metadata.

### A — Language / country visibility (admin-only)
- New `get_full_locale_from_request()` captures the real UI language
(any of 33, not the narrowed en/es/de email locale) on **every**
job-creation path (URL, upload, audio-search, bulk, made-for-you via
Stripe metadata). Stored as `Job.locale` (+ added to the dashboard
summary projection) and `User.ui_locale` (kept separate from
`User.locale`).
- `LocaleBadge` (flag + language) shown in the admin jobs table, admin
job detail (+ country via the existing `IpInfo` on the creation IP), the
user-detail header, and the customer `JobCard` **gated to admins only**
— regular users never see it.
- Country is derived from IP (existing `ip_geolocation_service`); no new
capture.

### B — "Emails sent to this user"
- Every send routes through one `EmailService._log_and_send` chokepoint
that persists each email to a new Firestore `email_log` collection
(rendered HTML + Postmark MessageID), best-effort so it can never break
or delay a send. Single-use auth/capability tokens (magic-link, review,
referral) are **redacted** before persistence.
- `PostmarkAdminService` merges the Postmark Messages API (last ~45
days: exact HTML + opens/clicks/bounce/delivery status, paginated) with
the Firestore log (permanent), de-duplicated by message id.
- Endpoints `GET /api/admin/users/{email}/emails` and `GET
/api/admin/emails/{id}`; the user-detail "Emails Sent" card opens a
modal rendering the email in a sandboxed `<iframe srcDoc>` with a
metadata side panel.
- New Firestore composite index (`email_log` recipient + created_at)
added to Pulumi.

## Testing
- **41 new backend tests** (locale capture incl. q=0, email logging +
token redaction, PostmarkAdminService incl. pagination + 45-day
fallback, job-locale persistence, admin endpoints) — all pass.
- **10 new frontend tests** (LocaleBadge, UserEmailsCard list +
sandboxed-iframe modal) — all pass.
- Regression sweep on related existing suites: green. Preview-email flow
smoke-tested through the new chokepoint.

## Notes
- `pulumi up` for the new `email_log` index should be applied before
merge (done locally).
- `Job.locale` populates for new jobs going forward; historical jobs
fall back to the submitting user's locale where shown.

Local CodeRabbit review completed and addressed (token redaction,
pagination, row a11y, locale-overwrite guard, q=0 handling).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

@coderabbitai ignore

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-23 01:51:42 -04:00
Andrew Beveridge ☄️andClaude Opus 4.8 51d9233cb8 fix(fallbacks): T4 — fail loud on missing configured render asset (font dir, background) (#807)
## Summary
Theme 4 of the silent-fallback audit
(`docs/archive/2026-06-09-fallback-audit-plan.md`, in #805). A
*configured* branded render asset that is missing must not silently
substitute a different font/background on a paid deliverable. Mirrors
the existing background-image check in
`output/video.VideoGenerator.__init__` (which already raises).

## Changes
- **`output/video._build_ass_filter`** (4.1): when the theme's karaoke
`font_path` is set but the file is missing, the final MP4 used to
silently drop `:fontsdir` → libass resolves a system fallback font for
the whole song. Now raises `FileNotFoundError`. *(Unset `font_path` is
still fine — no fontsdir, no raise.)*
- **`video_generator._create_background`** (4.3): a set-but-missing
title/end-screen `background_image` used to silently fall back to a flat
color. Now raises `FileNotFoundError`. *(No background configured is
still fine — flat color.)*

## Scope (per your decision)
Only these two clean "configured-but-missing → raise" cases are changed.
The font **substitution** items (4.2 title/end font, 4.4 CDG→arial, 4.8
measurement font) and duet/colour items (4.5–4.7) are **left as-is** —
font substitution overlaps with legitimate CJK font fallback
(`test_video_generator_cjk_font.py`) and warrants per-item treatment.

## Testing
- [x] New `tests/unit/test_fallback_render_assets.py` (raise +
legitimate-unset cases)
- [x] 105 render/video tests pass

## Review
- [x] CodeRabbit CLI review completed locally (0 findings)

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-08-22 18:50:16 -04:00
Andrew Beveridge ☄️andClaude Opus 4.8 5fff6d67b0 fix(fallbacks): T1 — flacfetch sole downloader, no silent YouTube downgrade (#806)
## Summary
Theme 1 of the silent-fallback audit
(`docs/archive/2026-06-09-fallback-audit-plan.md`, added in #805).
flacfetch is the sole/lossless downloader — a configured-remote failure
must not silently downgrade to a local YouTube-only rip. This is the
exact class as the 2026-06-08 Facebook-URL incident.

## Changes
- **`audio_search_service._search_remote`**: a `FlacfetchServiceError`
from the remote service was swallowed and silently re-run against the
local YouTube-only fetcher. Now fails loudly (`AudioSearchError`) and
clears cached state. **No silent downgrade.**
- **`create_audio_fetcher`** (local CLI factory): a half-configured
remote (exactly one of `FLACFETCH_API_URL`/`FLACFETCH_API_KEY`) used to
warn and silently use local mode, masking a deploy/secret typo. Now
raises `AudioFetcherError`.

## Scope / deferred (needs your call)
Findings **1.2** (tracker configured-but-unusable → silently
YouTube-only) and **1.4** (`select_best` → first result on ranking
failure) live in the **shared local `FlacFetcher`** and have dedicated
tests asserting graceful local-CLI behaviour
(`test_select_best_handles_exception`). Your "fail loud" directive was
framed around the *cloud* silent-downgrade; hard-failing the local CLI
when a user simply has no usable tracker would change intentional local
behaviour. **Deferred for a per-item decision** rather than steamrolled
— see the audit plan doc.

## Testing
- [x] `backend/tests/test_audio_search.py` +
`tests/unit/test_audio_fetcher.py` pass (209)
- Added: remote-error-does-not-fall-back-to-local; updated the two
half-config factory tests to expect a raise.

## Review
- [x] CodeRabbit CLI review completed locally (1 major finding: clear
cached results on failure — fixed)

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-08-22 18:35:12 -04:00
Andrew Beveridge ☄️andClaude Opus 4.8 c9c019aaf7 perf(encoding): faster, smaller, portable "With Vocals" mp4 (yuv420p + veryfast + AAC) (#915)
## Summary
- The **"(With Vocals).mp4"** sing-along deliverable was the *single
longest finalization stage* (~59s on a 32-vCPU worker).
`convert_mov_to_mp4` encoded it accidentally-badly: no `-pix_fmt`
(inherited the renderer's `High 4:4:4`/`yuv444p` — the only 4:4:4 file
in the set, poorly supported by browsers/TVs/phones), default `medium`
preset (slow), and `-c:a copy` → **FLAC-in-mp4** (non-standard) despite
the documented spec ("4k H264/**AAC**").
- Verified the file is a **pure leaf** — nothing downstream reads its
video or audio. Every later output (Karaoke.mp4, Lossless/Lossy/720p,
MKV) derives from the *source* `with_vocals` file + the *instrumental*
audio; YouTube uploads the lossless MKV. So the encode is chosen purely
for the deliverable.

## Change
`convert_mov_to_mp4` (software/CPU path — what runs on prod workers):
```
-c:v libx264 -pix_fmt yuv420p -preset veryfast -c:a aac -ar 48000 -b:a 320k -movflags +faststart
```
Applied to both implementations, kept in sync:
- `LocalEncodingService.convert_mov_to_mp4` (cloud worker — prod). Its
`MP4_FLAGS` has no pix_fmt, so `-pix_fmt yuv420p` is explicit on both
GPU and CPU commands.
- `KaraokeFinalise.convert_mov_to_mp4` (local CLI). Already emitted AAC
and already forces yuv420p via its `mp4_flags`; adds the `-preset
veryfast` speed win + `-b:a 320k`.

## Measured result (real 138s 4K source: Arctic Monkeys – Riot Van)
| Metric | Before (prod) | After |
|---|---|---|
| Encode time | 56 s | **29 s** (~2×) |
| File size | 22 MB | **14 MB** (~36% smaller) |
| Video | h264 High 4:4:4 / yuv444p | **h264 High / yuv420p** |
| Audio | FLAC-in-mp4 (~639 kbps) | **AAC-LC 48 kHz / 320 kbps** |

This is the longest finalization stage, so ~halving it cuts **~20-25%
off total finalization wall-time**. The file also gets **smaller** (AAC
320k ≪ FLAC ~639k, 4:2:0 < 4:4:4) — no size/CPU tradeoff; every axis
improves, plus universal playback + spec alignment. Verified end-to-end
by running the exact shipped command on the source and re-probing the
output.

## Testing
- [x] `backend/tests/test_local_encoding_service.py` +
`tests/unit/test_karaoke_finalise/test_ffmpeg_commands.py` updated to
assert the new flags (pix_fmt/preset/AAC/320k + regression guard: no
`-c:a copy`)
- [x] 236 passed / 8 skipped across encoding + karaoke_finalise suites
- [x] End-to-end: exact command on real source → 29s, 14MB, yuv420p, AAC
320k

## Review
- [x] Local CodeRabbit review completed (2 cycles); findings addressed
(forced yuv420p on NVENC path; removed duplicate pix_fmt on CLI path).
Pre-existing skipped-NVENC-test fixture gap noted, out of scope.

## Non-goals
User-uploaded (finalise-only) non-H.264 videos still re-encode correctly
(codec-agnostic). Whether a "reference" file needs to be 4K at all is a
separate product decision.

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-16 16:49:21 -04:00