Commit Graph
165 Commits
Author SHA1 Message Date
Andrew Beveridge ☄️andClaude Opus 5.5 bcdfd6b6bb fix(jobs): unstick audio-edit jobs — waveform OOM, retry skip, screens handoff (#1049)
## Summary
Job `2c1b922e` (Dave Matthews Band – Why I Am, 24-bit/96kHz FLAC, audio
edit requested) stalled twice in prod. Three bugs:

1. **Waveform OOM** — `WaveformGenerator.generate_data_only` decoded the
full source at native resolution through pydub. For a 6-min hi-res FLAC
that peaked at ~1 GB RSS and OOM-killed the 1 GiB `audio-download-job`
while pre-generating editor assets. It now decodes straight to 16 kHz
mono PCM via ffmpeg and computes the envelope with numpy: **156 MB vs
1.07 GB peak** on the real file, amplitudes within 0.003 of the old
output. The same function backs the editor's on-demand waveform in the
API.
2. **Retry skipped unfinished audio-edit prep** — the Cloud Run
auto-retry saw `downloading`, treated the download as done and exited 0,
leaving the job at `downloading` forever. A retry that finds an
audio-edit job still at `downloading` now resumes the editor prep. That
path triggers nothing downstream, so it's safe. Non-edit jobs keep the
existing skip (downstream may already be triggered).
3. **Screens handoff ignored `audio_edit_complete`** —
`advance_to_screens_if_ready` (added in #1002) and the
`recover-stuck-jobs` lost-screens sweep only accepted `downloading`.
Audio-edited jobs prep at `audio_edit_complete`, so **every audio-edited
job since Sept 15 stalled** after lyrics and separation finished (log:
`screens not (re)triggered — status=audio_edit_complete`). Both now use
a shared `PREP_PHASE_STATUSES`.

2c1b922e was unstuck manually (admin reset → `awaiting_audio_edit`, then
a direct `/api/internal/workers/screens` dispatch). Added a
TROUBLESHOOTING entry covering both signatures and the manual fixes.

## Testing
- New tests: hi-res 96k/24-bit decode path (low-rate mono args, envelope
matches pydub reference, silent → 0, undecodable → error);
download-worker resume vs skip; handoff at `audio_edit_complete`;
watchdog recovers `audio_edit_complete` and queries both statuses.
- `pytest tests/unit/` → 1824 passed, 8 skipped
- `pytest backend/tests/ --ignore=backend/tests/emulator` → 4586 passed,
36 skipped
- Version 0.238.4 → 0.238.5

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Audio-edit jobs can resume preparation after an interruption without
redownloading existing audio.
* Screen generation can proceed after audio editing is complete, and
stalled screen-generation triggers can be recovered.
* Waveform previews are more reliable for high-resolution and silent
audio. Invalid audio and decoding timeouts now produce clear errors.
* **Documentation**
* Added troubleshooting guidance for jobs stuck during audio editing or
before screen generation.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-24 23:39:25 -04:00
Andrew Beveridge ☄️andClaude Fable 5 d85b6e271c feat: precomputed vocals peaks — Waveforms strips paint sub-second (#1031)
## Summary

Final piece of the review-page speed work (#1024 → #1028 → #1029): the
Waveforms-mode strips no longer require downloading + WebAudio-decoding
the entire vocals stem (~7.5MB, ~4–6s) — the backend now serves a
**precomputed 400-peaks/s envelope as ~150KB of uint8/base64 JSON**,
cached in GCS and pre-warmed at prep time, so strips paint sub-second.

## How

- **`WaveformGenerator.generate_peaks()`** — numpy max-abs envelope,
bucket-for-bucket identical to the frontend's `computePeaks` (same
drift-free fractional-stride bucketing; exact match vs a naive reference
in tests).
- **`AudioAnalysisService.get_vocals_peaks()`** — GCS cache at
`review-audio/vocals_peaks_400.json`, validated against the stem path;
miss decodes the small transcoded OGG; upload best-effort.
- **`GET /api/review/{job_id}/vocals-peaks`** — off the event loop under
the shared waveform semaphore; `202 Retry-After: 15` while separation
runs (mirrors `/audio/vocals`); `Cache-Control: private, max-age=600`;
`peaks_per_second` clamped 50–1000.
- **`screens_worker`** pre-computes the envelope next to the existing
OGG + waveform pre-warm, so it's a pure cache read by the time a job
reaches review.
- **Frontend** — `VocalsAudioDataLoader` prefers the peaks endpoint (one
quick retry on transient errors) and **falls back to the original full
download+decode** on any terminal peaks failure, so a peaks-side problem
can never cost a user their waveforms. Local mode (no peaks URL) uses
the old path unchanged. Vocal stem selection factored into
`backend/utils/stems.py` (shared by routes + worker).

## Testing

- Package: 4 tests incl. exact naive-reference parity (**106 passed**)
- Backend: 4 service + 4 endpoint tests (**4518 passed**)
- Frontend: `fetchVocalsPeaks` units + 5 loader-behavior tests (prefers
peaks / falls back / one-retry-then-fallback / 202 polling / audio-only
mode) (**1307 passed**)
- `review-fast-load.spec.ts` prod E2E extended with a cached
vocals-peaks latency/shape check
- Post-deploy: will verify the endpoint on a real in-review job and
confirm payload size + latency

## Notes

- Existing in-review jobs have no cache yet — their first Waveforms open
computes + caches lazily (bounded); every open after that is sub-second.
New jobs are pre-warmed.
- 8-bit amplitude quantization is visually lossless at strip heights
(14px).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-20 13:08:19 -04:00
Andrew Beveridge ☄️andClaude Fable 5 527d55ef63 fix: preserve source audio extension for temp downloads (ffmpeg extension-fallback failure) (#1023)
## Summary

Root-cause fix for the 2026-09-17 19:47 ET error-pattern cluster (job
`e026e727`): a paying user's uploaded MP3 had a ~1.2 MB ID3 tag
(embedded cover art from a ytmp3 rip). The backend downloads job audio
to temp files hardcoded as `*.flac` — ffmpeg's content probe is
inconclusive inside the huge tag, so it falls back to the **file
extension**, commits to the flac demuxer, and fails with `Could not find
codec parameters for stream 0 (Audio: flac, 0 channels)`. That single
file broke every consumer of the input audio:

- `GET /api/review/{id}/waveform-data` → 500 ("Decoding failed. ffmpeg
returned error code: 8")
- Review OGG transcode → "Cannot determine format of input 0:0 after
EOF" (fell back to raw FLAC serving)
- Preview video encode → failed on **both** GCE worker and local
fallback ("Invalid data found when processing input")
- Duration probe → `Failed to get audio duration: 'duration'` (KeyError)

**Repro:** the file is a perfectly valid 227 s MP3 — `ffprobe` succeeds
when the copy is named `.mp3` or has no extension, and fails when named
`.flac` (verified on ffmpeg 8.0 locally and 7.0.2-static in prod). The
`majorbrand: dash` metadata is a copied ID3 tag, not a mislabeled MP4.

## Changes

- New `backend/utils/audio_filenames.py` —
`local_audio_filename(gcs_path, stem)` names local copies with the
source object's real (lowercased) extension; unknown/missing extension →
bare name, forcing pure content probing.
- Applied at every site that downloads job audio to a temp file:
- `backend/services/audio_analysis_service.py` (waveform data/image,
analysis, mute-region waveform, stem comparison)
- `backend/services/audio_transcoding_service.py` (review OGG transcode)
- `backend/api/routes/review.py` `_prepare_preview_inputs` (preview
local encode)
- `backend/services/gce_encoding/main.py` (GCE preview + render audio
download)
- `backend/workers/render_video_worker.py`,
`backend/workers/audio_worker.py`, `backend/workers/lyrics_worker.py`
- `docs/TROUBLESHOOTING.md`: new runbook entry for this failure class.
- Version bump 0.229.3 → 0.229.4.

## Testing

- New `tests/unit/test_audio_filenames.py` — 9 tests: helper behavior
(extension preservation, lowercasing, unknown/missing-extension
fallback) + service-level assertions that the transcoding and analysis
services download to source-matching temp names.
- Full `make test` suite passes locally (backend + frontend, 105 suites
/ 1286 frontend tests).

## Impact / rollout

- No data repair needed for job `e026e727` — its stems/lyrics are
intact; the review page starts working once this deploys.
- `gce_encoding/main.py` ships in the wheel; the encoding worker's
per-job wheel version check picks it up (restart `encoding-worker`
service to force immediate pickup).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **Bug Fixes**
- Improved audio processing for files with non-FLAC extensions,
including MP3 and OGG files.
- Preview rendering, waveform generation, transcription, and audio
conversion now preserve source file extensions, reducing decoding
failures.
  - Audio conversion fallbacks now use the correct source content type.

- **Documentation**
- Added troubleshooting guidance for waveform and preview failures
related to audio metadata or file extensions.

- **Tests**
- Added coverage for extension handling across audio download and
processing workflows.

- **Release**
  - Updated the application version to 0.229.4.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-17 21:40:36 -04:00
Andrew Beveridge ☄️andClaude Opus 4.8 303ff7e728 fix(lyrics-cache): resolve cache_dir at instantiation — stop burning AudioShake credits on every job (#1017)
## Problem

The cross-container AudioShake/lyrics transcription cache has been
**completely non-functional in production**. Every lyrics job logs
`Cache sync from GCS: 0 downloaded, 14 not found` and the GCS
`lyrics-transcriber-cache/` prefix is **empty** — so every job
(including repeat songs and every daily/canary E2E run of the same
track) makes a fresh, credit-spending AudioShake alignment call.

## Root cause

`OutputConfig.cache_dir` used a bare
`os.getenv("LYRICS_TRANSCRIBER_CACHE_DIR", ...)` as its dataclass field
default. Dataclass defaults are evaluated **once at import time**. The
cloud lyrics worker sets `LYRICS_TRANSCRIBER_CACHE_DIR` to a per-job
temp dir at **runtime** (`backend/workers/lyrics_worker.py:316`), long
after the module imported — so it was ignored.

Result: the transcriber wrote its cache to the frozen default
`~/lyrics-transcriber-cache`, while the GCS-sync agent read/wrote
`{temp_dir}/lyrics-cache`. The upload step scanned the sync dir, found
nothing the transcriber had written, and uploaded nothing → GCS stayed
empty forever → every future job missed.

## Fix

Use `field(default_factory=...)` for `cache_dir` (and `output_dir`,
which had the same latent `os.getcwd()`-at-import bug) so both resolve
when an `OutputConfig` is **constructed**, honoring the worker's runtime
env. This mirrors the existing `default_factory` pattern already used
for the local-whisper fields in `TranscriberConfig`.

After this deploys, the first job for a given audio hash populates GCS,
and subsequent jobs with the same audio (all the daily/canary E2E runs,
plus real repeat songs) hit the cache and spend **zero** AudioShake
credits.

## Testing

- New regression tests
(`tests/unit/lyrics_transcriber/test_output_config_cache_dir.py`) assert
`cache_dir` tracks the env set *after* import, is re-evaluated per
instance, respects an explicit override, and that `output_dir` tracks
cwd at construction.
- Existing `tests/unit/test_lyrics_cache_service.py` still passes (21
passed).

## Verification plan (post-deploy)

Watch a lyrics job's logs for `Cache sync to GCS: N uploaded` on the
first run, then `Cache sync from GCS: >0 downloaded` on the next run of
the same audio, and confirm objects appear under
`gs://karaoke-gen-storage-nomadkaraoke/lyrics-transcriber-cache/`.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Cache and output directories now reflect the environment and current
working directory when each job starts.
* Runtime cache directory changes are honored, improving reuse of cached
transcription results.

* **Tests**
* Added coverage for cache path resolution, fallback behavior,
overrides, cache reuse, and transcription pipeline outcomes.
* Added validation for malformed countdown settings and local GPU
processing handoffs.

* **Chores**
  * Updated the project version to 0.229.1.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-09-17 15:14:15 -04:00
Andrew Beveridge ☄️andClaude Opus 4.8 37b9e6088b ci(pypi): recurring storage-cap prune reminder (fix failing PyPI publish) (#960)
## Problem

The `Deploy - Publish to PyPI` step on `main` has been failing:

```
HTTP Error 400: Project size too large. Limit for project 'karaoke-gen'
total size is 10 GB.
```

This is a recurrence of the known 10 GB PyPI project cap. The project is
at
**10.71 GB / 479 releases**; each release is a ~62 MiB universal wheel
(bundled
33-locale frontend) auto-published on every merge (~3 GB/month), so it
refills
the cap every few months. **Only the public `pip install` publish step
fails** —
Cloud Run and the GCE encoding worker (pulls the wheel from GCS) are
unaffected.

The prior fix (v0.187.5) was wheel-only publishing + a one-time manual
deletion,
which only bought ~2 months. This makes the cleanup **recurring and
durable**.

## Why not just automate deletion in CI?

PyPI has **no deletion API** (upload tokens / Trusted Publishing are
upload-only,
and 2FA is mandatory). Deletion is only possible from a logged-in
browser
session, which can't be safely/durably stored in CI. So we automate the
*decision + reminder* and keep the *deletion* a ~1-minute human step.

## What this adds

- **`scripts/prune_pypi_releases.py`** (stdlib-only) — reads the public
JSON API
and computes a retention plan: **keep last 60 days + newest release per
older
calendar month + latest overall**. Renders a table / JSON / Markdown
report and
a **paste-into-browser-console** deletion snippet. Never deletes
anything
  itself. Today's plan: keep 82, delete 397, **10.71 GB → 4.17 GB**.
- **`.github/workflows/pypi-prune-reminder.yml`** — monthly cron (+
manual
dispatch) that runs the planner and, once the project crosses a 7 GB
watermark,
opens/updates a single tracking issue with the plan + snippet. No
secrets.
- **`docs/PYPI-STORAGE-PRUNE.md`** runbook + a `TROUBLESHOOTING.md`
entry.
- **`tests/unit/test_prune_pypi_releases.py`** — 13 tests covering the
policy
(recency window, per-month keeper, latest-overall, same-day timestamp
ranking,
  empty/undated releases, sizes, rendering).

## Testing

- `pytest tests/unit/test_prune_pypi_releases.py` → 13 passed.
- Ran the planner against live PyPI data; generated JS validated with
`node --check`.
- `black` clean; workflow YAML validated; `should_nag` gate verified.

## Notes

- **No version bump** — tooling/CI/docs only, nothing in the wheel
changes.
Merging therefore **skips** the publish step (tag `v0.209.1` already
exists),
so this PR does not itself add another fat wheel while we're over the
cap.
- **To unblock publishing now:** after merge, run the prune (locally via
the
script or trigger the workflow with `workflow_dispatch`), paste the
snippet on
the logged-in PyPI releases page, then re-run the publish. The monthly
cron
  keeps it under the cap going forward.
- Related: filed
[pypi/support#12050](https://github.com/pypi/support/issues/12050)
requesting a 50 GB limit as a one-off buffer; the recurring prune is the
  durable fix.

@coderabbitai ignore

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-29 21:46:10 -04:00
Andrew Beveridge ☄️andClaude Opus 4.8 1e99f9e0d5 feat(admin): per-job/user locale badge + "emails sent to user" history (#930)
## Summary

Two admin-only features so support/intervention work can (a) know what
language/country a user is using the product in, and (b) see the exact
emails ever sent to a user, rendered as they appeared in the inbox with
full delivery metadata.

### A — Language / country visibility (admin-only)
- New `get_full_locale_from_request()` captures the real UI language
(any of 33, not the narrowed en/es/de email locale) on **every**
job-creation path (URL, upload, audio-search, bulk, made-for-you via
Stripe metadata). Stored as `Job.locale` (+ added to the dashboard
summary projection) and `User.ui_locale` (kept separate from
`User.locale`).
- `LocaleBadge` (flag + language) shown in the admin jobs table, admin
job detail (+ country via the existing `IpInfo` on the creation IP), the
user-detail header, and the customer `JobCard` **gated to admins only**
— regular users never see it.
- Country is derived from IP (existing `ip_geolocation_service`); no new
capture.

### B — "Emails sent to this user"
- Every send routes through one `EmailService._log_and_send` chokepoint
that persists each email to a new Firestore `email_log` collection
(rendered HTML + Postmark MessageID), best-effort so it can never break
or delay a send. Single-use auth/capability tokens (magic-link, review,
referral) are **redacted** before persistence.
- `PostmarkAdminService` merges the Postmark Messages API (last ~45
days: exact HTML + opens/clicks/bounce/delivery status, paginated) with
the Firestore log (permanent), de-duplicated by message id.
- Endpoints `GET /api/admin/users/{email}/emails` and `GET
/api/admin/emails/{id}`; the user-detail "Emails Sent" card opens a
modal rendering the email in a sandboxed `<iframe srcDoc>` with a
metadata side panel.
- New Firestore composite index (`email_log` recipient + created_at)
added to Pulumi.

## Testing
- **41 new backend tests** (locale capture incl. q=0, email logging +
token redaction, PostmarkAdminService incl. pagination + 45-day
fallback, job-locale persistence, admin endpoints) — all pass.
- **10 new frontend tests** (LocaleBadge, UserEmailsCard list +
sandboxed-iframe modal) — all pass.
- Regression sweep on related existing suites: green. Preview-email flow
smoke-tested through the new chokepoint.

## Notes
- `pulumi up` for the new `email_log` index should be applied before
merge (done locally).
- `Job.locale` populates for new jobs going forward; historical jobs
fall back to the submitting user's locale where shown.

Local CodeRabbit review completed and addressed (token redaction,
pagination, row a11y, locale-overwrite guard, q=0 handling).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

@coderabbitai ignore

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-23 01:51:42 -04:00
Andrew Beveridge ☄️andClaude Opus 4.8 51d9233cb8 fix(fallbacks): T4 — fail loud on missing configured render asset (font dir, background) (#807)
## Summary
Theme 4 of the silent-fallback audit
(`docs/archive/2026-06-09-fallback-audit-plan.md`, in #805). A
*configured* branded render asset that is missing must not silently
substitute a different font/background on a paid deliverable. Mirrors
the existing background-image check in
`output/video.VideoGenerator.__init__` (which already raises).

## Changes
- **`output/video._build_ass_filter`** (4.1): when the theme's karaoke
`font_path` is set but the file is missing, the final MP4 used to
silently drop `:fontsdir` → libass resolves a system fallback font for
the whole song. Now raises `FileNotFoundError`. *(Unset `font_path` is
still fine — no fontsdir, no raise.)*
- **`video_generator._create_background`** (4.3): a set-but-missing
title/end-screen `background_image` used to silently fall back to a flat
color. Now raises `FileNotFoundError`. *(No background configured is
still fine — flat color.)*

## Scope (per your decision)
Only these two clean "configured-but-missing → raise" cases are changed.
The font **substitution** items (4.2 title/end font, 4.4 CDG→arial, 4.8
measurement font) and duet/colour items (4.5–4.7) are **left as-is** —
font substitution overlaps with legitimate CJK font fallback
(`test_video_generator_cjk_font.py`) and warrants per-item treatment.

## Testing
- [x] New `tests/unit/test_fallback_render_assets.py` (raise +
legitimate-unset cases)
- [x] 105 render/video tests pass

## Review
- [x] CodeRabbit CLI review completed locally (0 findings)

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-08-22 18:50:16 -04:00
Andrew Beveridge ☄️andClaude Opus 4.8 5fff6d67b0 fix(fallbacks): T1 — flacfetch sole downloader, no silent YouTube downgrade (#806)
## Summary
Theme 1 of the silent-fallback audit
(`docs/archive/2026-06-09-fallback-audit-plan.md`, added in #805).
flacfetch is the sole/lossless downloader — a configured-remote failure
must not silently downgrade to a local YouTube-only rip. This is the
exact class as the 2026-06-08 Facebook-URL incident.

## Changes
- **`audio_search_service._search_remote`**: a `FlacfetchServiceError`
from the remote service was swallowed and silently re-run against the
local YouTube-only fetcher. Now fails loudly (`AudioSearchError`) and
clears cached state. **No silent downgrade.**
- **`create_audio_fetcher`** (local CLI factory): a half-configured
remote (exactly one of `FLACFETCH_API_URL`/`FLACFETCH_API_KEY`) used to
warn and silently use local mode, masking a deploy/secret typo. Now
raises `AudioFetcherError`.

## Scope / deferred (needs your call)
Findings **1.2** (tracker configured-but-unusable → silently
YouTube-only) and **1.4** (`select_best` → first result on ranking
failure) live in the **shared local `FlacFetcher`** and have dedicated
tests asserting graceful local-CLI behaviour
(`test_select_best_handles_exception`). Your "fail loud" directive was
framed around the *cloud* silent-downgrade; hard-failing the local CLI
when a user simply has no usable tracker would change intentional local
behaviour. **Deferred for a per-item decision** rather than steamrolled
— see the audit plan doc.

## Testing
- [x] `backend/tests/test_audio_search.py` +
`tests/unit/test_audio_fetcher.py` pass (209)
- Added: remote-error-does-not-fall-back-to-local; updated the two
half-config factory tests to expect a raise.

## Review
- [x] CodeRabbit CLI review completed locally (1 major finding: clear
cached results on failure — fixed)

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-08-22 18:35:12 -04:00
Andrew Beveridge ☄️andClaude Opus 4.8 c9c019aaf7 perf(encoding): faster, smaller, portable "With Vocals" mp4 (yuv420p + veryfast + AAC) (#915)
## Summary
- The **"(With Vocals).mp4"** sing-along deliverable was the *single
longest finalization stage* (~59s on a 32-vCPU worker).
`convert_mov_to_mp4` encoded it accidentally-badly: no `-pix_fmt`
(inherited the renderer's `High 4:4:4`/`yuv444p` — the only 4:4:4 file
in the set, poorly supported by browsers/TVs/phones), default `medium`
preset (slow), and `-c:a copy` → **FLAC-in-mp4** (non-standard) despite
the documented spec ("4k H264/**AAC**").
- Verified the file is a **pure leaf** — nothing downstream reads its
video or audio. Every later output (Karaoke.mp4, Lossless/Lossy/720p,
MKV) derives from the *source* `with_vocals` file + the *instrumental*
audio; YouTube uploads the lossless MKV. So the encode is chosen purely
for the deliverable.

## Change
`convert_mov_to_mp4` (software/CPU path — what runs on prod workers):
```
-c:v libx264 -pix_fmt yuv420p -preset veryfast -c:a aac -ar 48000 -b:a 320k -movflags +faststart
```
Applied to both implementations, kept in sync:
- `LocalEncodingService.convert_mov_to_mp4` (cloud worker — prod). Its
`MP4_FLAGS` has no pix_fmt, so `-pix_fmt yuv420p` is explicit on both
GPU and CPU commands.
- `KaraokeFinalise.convert_mov_to_mp4` (local CLI). Already emitted AAC
and already forces yuv420p via its `mp4_flags`; adds the `-preset
veryfast` speed win + `-b:a 320k`.

## Measured result (real 138s 4K source: Arctic Monkeys – Riot Van)
| Metric | Before (prod) | After |
|---|---|---|
| Encode time | 56 s | **29 s** (~2×) |
| File size | 22 MB | **14 MB** (~36% smaller) |
| Video | h264 High 4:4:4 / yuv444p | **h264 High / yuv420p** |
| Audio | FLAC-in-mp4 (~639 kbps) | **AAC-LC 48 kHz / 320 kbps** |

This is the longest finalization stage, so ~halving it cuts **~20-25%
off total finalization wall-time**. The file also gets **smaller** (AAC
320k ≪ FLAC ~639k, 4:2:0 < 4:4:4) — no size/CPU tradeoff; every axis
improves, plus universal playback + spec alignment. Verified end-to-end
by running the exact shipped command on the source and re-probing the
output.

## Testing
- [x] `backend/tests/test_local_encoding_service.py` +
`tests/unit/test_karaoke_finalise/test_ffmpeg_commands.py` updated to
assert the new flags (pix_fmt/preset/AAC/320k + regression guard: no
`-c:a copy`)
- [x] 236 passed / 8 skipped across encoding + karaoke_finalise suites
- [x] End-to-end: exact command on real source → 29s, 14MB, yuv420p, AAC
320k

## Review
- [x] Local CodeRabbit review completed (2 cycles); findings addressed
(forced yuv420p on NVENC path; removed duplicate pix_fmt on CLI path).
Pre-existing skipped-NVENC-test fixture gap noted, out of scope.

## Non-goals
User-uploaded (finalise-only) non-H.264 videos still re-encode correctly
(codec-agnostic). Whether a "reference" file needs to be 4K at all is a
separate product decision.

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-16 16:49:21 -04:00
Andrew Beveridge ☄️andClaude Opus 4.8 3f4b7550ab feat(portrait): 9:16 karaoke video output (GCS + Dropbox) (#890)
## Summary

Adds a **portrait 1080×1920 karaoke video** as an additional output of
every finalised job — for phone viewing / social sharing — distributed
to **GCS + the Dropbox brand-code folder only** (never Google Drive or
YouTube). Default-on, gated by `PORTRAIT_RENDER_ENABLED`, and fully
**non-fatal**.

It is a genuine **re-render** at portrait resolution with re-wrapped
lyrics — *not* a transform of the finished landscape video. The design
experiments proved transforms are unusable: letterboxing leaves the
lyrics tiny (~32% of the frame); centre-cropping slices the start/end
off **every** line. See the comparison in
`docs/archive/2026-06-14-portrait-video-experiments/comparison.png`.

## What's included

**New `karaoke_gen/portrait/` module** (reuses the existing
`SubtitlesGenerator` engine + the job's corrected-lyrics data, so
word-by-word timing is identical to the landscape video):
- `wrap.py` — orphan-merging balanced wrapper (no stranded "No," lines),
timing-preserving
- `background.py` — branded portrait background (wordmark/header +
song/artist + footer) + "thank you" end card
- `renderer.py` — portrait ASS + title/end cards + single-pass ffmpeg
assembly (title + lyrics body + end)

**Backend integration** (minimal, low-risk):
- `gce_encoding/main.py::render_portrait_into_outputs` — renders on the
GCE encoder inside `run_encoding`, using the un-padded instrumental +
un-countdown-processed segments (inherently in sync). Writes into
`output_dir`, so it rides the existing `finals/` GCS upload + Dropbox
folder upload.
- `EncodingOutput.portrait_mp4_path` + filename classification +
orchestrator download-allowlist mapping + admin files-manifest entry.
YouTube/Drive use their own allowlists, so the portrait is naturally
excluded from them.

## Testing

- 17 new tests in `tests/unit/portrait/` (balanced wrapping,
background/layout, guard + non-fatal wiring, and a real end-to-end
ffmpeg render on a synthetic fixture).
- Existing GCE/encoding/orchestration/lyrics-output suites green (116
tests total across touched areas).
- **Verified end-to-end on real prod job piri – dog (`052b94ab`)** via
the backend integration function: produced a valid, on-brand 1080×1920
portrait with live word-highlighting, correct title/end cards, correct
duration.

## Safety / rollout

- Additive + `PORTRAIT_RENDER_ENABLED` (default on) + wrapped in
try/except: a portrait failure logs and returns `None` — it never fails
the job or blocks landscape outputs/distribution.
- Subprocess timeouts on ffprobe/ffmpeg so a stalled render can't hang
the worker.
- ~15–30s + ~10 MB per job.

Design: `docs/archive/2026-06-14-portrait-video-design.md` · Plan:
`docs/archive/2026-07-13-portrait-video-plan.md`

🤖 Generated with [Claude Code](https://claude.com/claude-code)

@coderabbitai ignore

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-08-14 21:01:08 -04:00
Andrew Beveridge ☄️andClaude Opus 4.8 d6361ca7ad fix(error-monitor): stop transient-capacity + bot crashes from paging (#897)
## Why

Two alert-channel noise sources surfaced during the flacfetch download
incident. Neither was a real production problem, but both page — so we
fix the **monitor's signal quality** rather than mute them.

### 1. Transient encoding-worker capacity retries
`[job:…] Encoding worker start failed, aborting retries: VM … 503 —
SERVICE UNAVAILABLE` is a transient GCP zonal-capacity event.
`render_video_worker` catches `EncodingWorkerStartError` and **parks the
job for auto-retry** (`_park_job_for_capacity_retry`) — the job is *not*
lost, and a genuine give-up fails the job via a separate ERROR. Added a
`known_issues` suppression **scoped to the transient signal only** (503
/ `SERVICE UNAVAILABLE` or a capacity code:
`ZONE_RESOURCE_POOL_EXHAUSTED` / `STOCKOUT` / `QUOTA_EXCEEDED`). A
non-transient `EncodingWorkerStartError` (other/unknown code) shares the
prefix but **still pages**.

### 2. One incident → N alerts (normalizer gap)
The same failure produced *two* alerts (same job, two preview-render
IDs). `preview_<jobid>_<renderid>` is underscore-joined, and the generic
hex normalizer is `\b`-anchored (`_` is a word char), so every preview
render hashed to a distinct "new pattern." Added a normalizer rule that
collapses `preview_<hex>_<hex>` → `preview_<ID>`.

### 3. Bot/crawler browser crashes
`NotFoundError: removeChild` on `/en/r/` came from **bingbot** running
headless Chrome — a React DOM-reconciliation error no real user hits (0
real-user occurrences in 7d). Reports from bot/crawler user-agents are
now dropped at the `/client-errors` ingestion edge (still `202` so the
client won't retry), instead of being persisted/alerted.

## Tests
- `known_issues`: transient 503 **and** capacity-code variants
suppressed; non-transient start failure + genuine job failure still
alert.
- `normalizer`: preview-ID collapse; two renders of one job → same hash.
- `is_bot_user_agent`: crawler matrix true, real-browser/empty/None
false.
- API: a bingbot report is accepted (`202`, `is_new=false`) but **not
upserted**.

`133 passed`. Local CodeRabbit review completed and its two findings
(scope the capacity regex; sanitize the logged bot URL) were addressed;
a confirming re-review was blocked by the free-tier rate limit, so
leaving the bot review on.

## Notes
The actual outage in this incident (daily E2E download failure) was a
**flacfetch** bug, fixed + deployed separately (flacfetch #36). This PR
only removes the alert noise those investigations surfaced.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **Bug Fixes**
- Bot and crawler error reports are acknowledged without being persisted
or triggering alerts.
- Transient encoding-worker capacity failures are filtered as retryable
infrastructure noise.
- Preview render job identifiers are normalized consistently for
improved error grouping.

- **Maintenance**
  - Updated the application version to 0.192.2.
- Expanded validation for bot detection, error filtering, and preview
error normalization.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-07 10:20:47 +02:00
Andrew Beveridge ☄️andClaude Opus 4.8 6b8ce07188 fix: split CJK (Japanese) lyric lines by display width to prevent overlap (#880)
## Summary

Japanese (and other CJK) karaoke videos rendered with **overlapping
lyric lines** that ran off both screen edges — reported for job
`4dcca2d6` (レッツゴー!陰陽師).

**Root cause:** `SegmentResizer` split lines using raw **character
count** (`len(text) <= max_line_length`), a proxy for visual width
calibrated for Latin. East-Asian Wide/Fullwidth glyphs render ~2.5× a
Latin character and CJK has no inter-word spaces, so a Japanese line
well under the *character* budget was ~2× too wide for the 4K frame.
With no `WrapStyle` set, libass smart-wraps the too-wide line onto a
second physical row that lands on the next slot's fixed Y position →
overlap.

## Changes

- **Display-width metric** (`display_width`,
`unicodedata.east_asian_width` W/F → `WIDE_RATIO=2.7` units, else 1).
For pure ASCII/Latin `display_width == len`, so **English/European
output is byte-for-byte unchanged** (the Latin natural-break splitter is
untouched).
- Only the **oversize gate** uses display width; wide-script segments
are then split by **word-token packing** (`_pack_words_into_lines`) —
boundary-safe (never cuts inside a multi-char CJK token like `立ち向かう`)
and counts the inter-token space the renderer emits.
- **Preserve wordless oversized segments** instead of dropping them
(guards a latent pre-existing bug).
- `WIDE_RATIO=2.7` calibrated against the real render (libass ~0.70×
effective size; Montserrat/Avenir vs Noto/Hiragino): ~14 CJK
glyphs/line, ~83% of 3840px.

## Testing

- New `test_segment_resizer_cjk.py` (13 tests): JP/ZH/KO splitting,
mixed CJK+Latin, `display_width` units, **exact** English byte-for-byte
output, multi-char CJK token non-fragmentation, wordless-segment
preservation.
- Full backend unit suite: **1701 passed**, 8 skipped.
- **End-to-end verified:** re-rendered job `4dcca2d6` through the
production `generate_outputs` path with the fix — over-budget lines
**21/36 → 0**, and rendered 4K frames confirm no overlap (same
timestamps as the original bug report).

## Review

Reviewed locally with CodeRabbit (3 cycles → no findings). Findings
fixed: CJK-token fragmentation, wordless-segment lyric drop, exact-ASSII
test assertion.

@coderabbitai ignore

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-18 16:28:16 -04:00
Andrew Beveridge ☄️andClaude Opus 4.8 37b405b9c6 fix: reject untimed lyrics before render + retry transient GDrive drops (#859)
## Summary

Triage + fix for a batch of Discord prod error alerts. Grouped into 3
root causes; fixed the two actionable ones.

### Group A — render/preview crash on untimed lyrics (PRIMARY,
job-breaking)
All the `unsupported format string passed to NoneType.__format__` and
`'<' not supported between instances of 'NoneType' and 'float'` alerts
were **one root cause**. Job `231806a4` ("Nelly Furtado - Maneater") had
Danish custom parody lyrics with **all 50 segments + 364 words
`start_time`/`end_time` = null** — the user pasted lyrics into the
Synchronizer's *Edit Lyrics* modal (which creates untimed segments as an
intermediate state) but never tap-synced, then triggered preview +
render. The output pipeline crashed cryptically at ~6 None-unsafe timing
sites instead of rejecting cleanly.

**Fix — fail loudly with an actionable message instead of crashing:**
- New shared gate `validate_segment_timing()` (`LyricsTimingError`) at
the top of `OutputGenerator.generate_outputs()` — covers preview + local
render + encoder.
- `render_video_worker` gates right after loading corrections,
**before** any expensive work or encoder dispatch — stops the
GCE-encoder crash *now*, without an encoder-wheel redeploy.
- Preview endpoint maps `LyricsTimingError` → **422** "synchronize
first" (was a cryptic 500).
- Defense-in-depth: `segment_resizer` debug logging is now None-safe (a
log line must never crash the pipeline).
- Frontend: block **Submit** when untimed words remain, pointing the
user to the synchronizer (`countUntimedWords` mirrors the backend gate).

### Group B — GDrive "brand_code search incomplete: N/3 subfolder(s)
failed"
Transient connection drops (`SSL: UNEXPECTED_EOF_WHILE_READING`, `Broken
pipe`) against idle Cloud Run containers. Wrapped the per-subfolder
Drive lookups in the existing retry-with-backoff + service-reset so
single drops self-heal.

### Group C — investigated, transient, no code change
- `request_audit_error` + "Exception in ASGI application" = one
Firestore *transaction-expiry* on
`/api/admin/rate-limits/blocklists/allowlisted-domains` (count 1,
admin-only).
- `Default STARTUP TCP probe failed … DEADLINE_EXCEEDED` = one slow cold
start that **succeeded on retry** at 12:10.

## Testing
- `test_timing_validation.py` (incl. all-null regression fixture
matching job 231806a4)
- None-safe resizer logging test
- GDrive transient-recovery + retry-exhaustion tests
- Frontend `countUntimedWords` Jest tests
- Re-ran output/, gdrive, render-worker, review-custom-lyrics,
generate_outputs-consumer suites — all green. CodeRabbit: 0 findings.

## Notes
- Frontend guard toast uses a plain string to match the existing
hardcoded toasts in `handleSubmitToServer` (no new i18n key → no locale
churn).
- Detail: `docs/archive/2026-07-01-render-untimed-lyrics-crash-plan.md`.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 16:54:10 -04:00
Andrew Beveridge ☄️andClaude Opus 4.8 eb711b32a8 fix(divebar): folder-brand-aware index parser + symmetric xref normalization (#857)
## Summary
- Rewrites the Divebar Drive→BigQuery **filename parser** to be
folder-brand-aware, fixing mis-split artists and brand-polluted titles
that prevented mirror files from surfacing in kjbox.
- Fixes a **normalization asymmetry** in the KN↔Divebar xref join (KN
side was only `LOWER(TRIM())` vs the db's fully-normalized columns) and
removes a buggy `brand_match` branch that produced wrong-song links.
- Switches both divebar Cloud Function modules to `pulumi.FileArchive`
so `pulumi up` actually redeploys changed code.

## Changes
- `divebar_mirror/filename_parser.py` — folder-brand-aware rewrite:
strip the brand token wherever it sits (disc-id prefix/suffix,
`(CODE)`/`CODE -` prefix, `[Brand]`/`(Brand)`/` - Brand` suffix, generic
karaoke tags), each strip gated on a folder-brand match so real song
text (`(Live)`, `(G)I-DLE`, bands like `UB40`/`311`/`ACDC`) survives.
Curated folder→KN-code map fills `brand_code`. En-dash separator +
leading-dash cleanup.
- `divebar_lookup/main.py` — new `_norm_sql(col)` routes **both** join
sides through one identical BigQuery expression replicating
`normalize_for_search`; `brand_match` branch removed (it matched
brand+artist with no title → Cartesian wrong-song links; 107,485/107,485
pairs were title mismatches in live data).
- `modules/divebar_mirror.py` + `modules/divebar_lookup.py` —
`FileArchive` + `BucketObject` with `generation` pinning (matches
`runner_manager`), so code changes redeploy.
- New `tests/unit/test_divebar_filename_parser.py` (48 cases) + 3
`_norm_sql` guard tests; version bump 0.188.3→0.188.4.

## Impact (validated read-only over all 50,046 live rows)
- null `brand_code`: 18,437 (37%) → **1,619 (3%)**
- Divebar files cross-referenced to a KN song: production **23,318** →
**33,449** on current data (+10,131 from the symmetry fix alone),
lifting toward ~38k after the parser re-index.

## Testing
- [x] 64 divebar unit tests pass locally (parser 48, index_builder 5,
divebar_lookup 11)
- [x] Offline validation harness over all 50k live rows (read-only
BigQuery); `_norm_sql` confirmed to reproduce `normalize_for_search`
exactly
- [ ] CI full suite

## Deploy / re-index (post-merge)
`pulumi up` redeploys both functions; then trigger scheduler
`divebar-mirror-daily` (re-parse → MERGE, preserves `gcs_path`) **then**
`divebar-xref-rebuild-daily`. Order matters. (`kn_data_sync` still uses
a static source object — future-edit redeploy trap, out of scope here.)

## Review
- [x] CodeRabbit CLI review completed locally (1 finding → fixed:
brand_match removal; cycle 2 clean)

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-28 14:16:35 -04:00
Andrew Beveridge ☄️andClaude Opus 4.8 113197293b fix: restore missing (Title/End).mov + original audio in Dropbox folders (#842)
## Summary
- Organised Dropbox track folders stopped containing the 5-second screen
videos (`(Title).mov`/`(End).mov`) and the original input audio
(`(flacfetch).flac` / `(uploaded).mp3`). Root cause:
`_upload_to_dropbox` uploads the whole `output_dir`, and the
monolith→distributed-GCE migration quietly narrowed what lands there.
- Restores both file types for new jobs, adds an audio-only backfill for
past jobs, and adds a daily-E2E guard so this regression class is caught
automatically.
- Bundles a CLI fix for an ffmpeg interactive-prompt hang in
`--finalise-only` (found while regenerating the missing files locally).

## Changes
**Screen `.mov`s (forward):** GCE encoder copies its generated
`title/end.mov` into `outputs/` with canonical names → plumbed via
`EncodingOutput.title_mov_path`/`end_mov_path`, the `GCEEncodingBackend`
filename mapping, and `_download_gce_encoded_files` back into
`output_dir`.

**Original audio (forward):** new `backend/services/original_audio.py`
(source tag `(uploaded)`/`(flacfetch)`/`(YouTube)` + GCS path +
filename); orchestrator `_stage_original_audio_for_upload()` fetches it
into `output_dir` before folder uploads (non-fatal, only when a folder
upload will occur).

**Backfill (audio only):** `backend/scripts/backfill_original_audio.py`
— dry-run by default, `--live` to upload, skips when audio is gone from
GCS or already present. `.mov` can't be backfilled (needs re-encode).
Adds `DropboxService.file_exists` (re-raises non-not-found `ApiError`).

**Daily E2E guard:** `audio_download_worker` records original audio in
`file_urls['input']['audio']`; `_upload_results` registers
`title_mov`/`end_mov` under `file_urls['finals']`;
`happy-path-real-user.spec.ts` STEP 10.5 asserts the admin files
manifest contains `finals/title_mov`, `finals/end_mov`, `input/audio`,
`finals/lossless_4k_mp4`.

**CLI ffmpeg-hang fix:** `karaoke_finalise` only added `-y` in
non-interactive mode, so a pre-existing output made ffmpeg block on an
invisible `Overwrite? [y/N]` prompt until the 600s timeout (silently
aborting at step 2/6). `-y` is now unconditional +
`stdin=subprocess.DEVNULL` on all ffmpeg `subprocess.run` calls.

## Testing
- [x] 38 new/updated unit tests; all affected suites green
(`encoding_interface`, `video_worker_orchestrator`,
`gce_encoding_worker`, `original_audio`, `backfill_original_audio`,
`dropbox_service`, `upload_results_screen_movs`,
`audio_download_worker`, full `karaoke_finalise` suite)
- [x] Full backend suite: +38 passing vs main, 0 new failures (12
pre-existing `asyncio.get_event_loop()` ordering failures on main,
unrelated)
- [x] Output-file completeness now verified by the daily GHA E2E (STEP
10.5)

## Review
- [x] Local CodeRabbit review completed (scoped to this branch) — 1
major finding fixed (`file_exists` masking non-not-found errors),
re-review clean (0 findings)

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-17 17:11:24 -04:00
Andrew Beveridge ☄️andClaude Opus 4.8 1b7b106c21 fix(lyrics): sanitize invalid word timings to prevent overlapping screens (#804)
## Summary
- Fixes overlapping rendered lyrics on job `17f7c313` ("Atmosphere -
Scalp") — two full screens of lyrics displayed on the same 4 vertical
slots for 7-12s.
- **Distinct root cause** from the earlier libass-wrap overlap (PR
#794), which is why it recurred. This one is a genuine same-Y /
same-time overlap caused by invalid word timestamps.

## Root cause
The transcription left the 4 leading words of a segment ("A whiskey and
a") with invalid timestamps (`start=0.0`, `end=-0.005`) while the
segment itself correctly started at 15.18s.
`SegmentResizer._create_segment_from_words` derives a split line's
`start_time` from `words[0].start_time`, so the split line `"A whiskey
and a beer,"` inherited `start_time=0.0`. A `LyricsScreen` sorts by
`min(line.segment.start_time)`, so the screen jumped ahead of the INTRO
section screen; the INTRO then wiped the active-line bookkeeping and the
next screen faded in (post-instrumental) over the still-displayed
previous screen.

## Changes
- Add `SegmentResizer._sanitize_segment_timings`, run at the top of
`resize_segments` (covers all output paths via `generator.py`).
- Clamps **only invalid** word timings into the segment's `[start, end]`
window and keeps them non-decreasing; valid words are left untouched so
the karaoke highlight sweep is preserved.
- New regression tests in
`tests/unit/lyrics_transcriber/output/test_segment_resizer_timing.py`.
- Bump version `0.176.1` → `0.176.2`.

## Testing
- [x] New timing regression tests + prior line-length regression tests
pass
- [x] Full `lyrics_transcriber` unit suite: 110 passed
- [x] End-to-end against job `17f7c313`: **4 same-Y overlaps before → 0
after**

## Review
- [x] Local CodeRabbit review completed (0 findings)

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-09 19:24:43 -04:00
Andrew Beveridge ☄️andClaude Opus 4.8 70bbabc661 fix(lyrics): prevent over-long lines overlapping the slot below in rendered video (#794)
## Summary
- Fixes overlapping lyric lines in rendered karaoke videos (preview and
final 4K).
- Root cause: `SegmentResizer` could emit a line longer than
`max_line_length`, which **libass word-wraps onto a second physical row
at render time**, colliding with the lyric line in the slot below (two
lines drawn on top of each other).
- This lives in the shared `lyrics_transcriber` package, so it fixes the
overlap for **all jobs**, not just the one reported.

## Root cause
In `_create_segments_from_lines`, word objects are re-assigned to the
splitter's lines via fragile substring matching. A `;` break was
positioned *before* the semicolon, so the line ended with `around` while
the word token was `around;`; the match failed and every remaining word
fell into an **unbounded catch-all** that joined them into one over-long
line. Space-dash-space (` - `) breaks orphaned the dash the same way.

## Changes
- `;` break now splits **after** the semicolon, keeping it attached to
its word token (also reads better).
- The catch-all now **greedily packs leftover words into ≤
`max_line_length` lines** instead of one unbounded segment — a length
guarantee that holds regardless of which punctuation desyncs
word-to-line matching.
- Added regression tests (`test_segment_resizer_line_length.py`):
semicolon case, dash case, single-long-word edge case, word-order
preservation.

## Testing
- [x] New regression tests pass; full backend unit suite green (the only
failures are pre-existing `audio-separator`-not-installed env errors,
unrelated to this change).
- [x] Verified against job `5c631108` + all 22 saved review sessions:
over-length output lines → **0**, no word loss/reorder.
- [x] Re-rendered the 4K overlap moment locally — confirmed the
collision is gone.

## Review
- [x] Local CodeRabbit CLI review completed; 1 minor finding addressed.

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 18:33:08 -04:00
Andrew Beveridge ☄️andClaude Opus 4.8 fe8430d990 fix(admin): allow regen-screens from in_review state + surface errors via toast (#793)
## Summary
- Fixes a 400 when regenerating title/end screens for a job the user has
opened in the review UI (`in_review` state).
- Fixes the root cause behind "click Regenerate Screens → modal just
closes, no error": `<Toaster />` was never mounted, so **all** `toast()`
calls (success and error) silently no-op'd.

## Changes
- **Backend:** add `in_review` to `REGENERATE_SCREENS_ALLOWED_STATES` in
`backend/api/routes/admin.py`. The sibling `/restart` endpoint already
allows `in_review`; this restores parity. Updated `docs/API.md`.
- **Frontend:** mount the shadcn `<Toaster />` (same `@/hooks/use-toast`
store) in the root `app/layout.tsx` so admin toasts render app-wide.
- **Frontend:** give the regen-screens handlers specific, human-readable
failure titles/messages (admin dashboard + compact `AdminJobActions`
toolbar).
- Version bump `0.174.12` → `0.174.13`.

## Testing
- [x] Backend: `tests/unit/test_regenerate_screens.py` — allow-list
membership, `in_review` → 200, invalid state → 400, missing job → 404 (4
pass).
- [x] Frontend: `AdminJobActions.test.tsx` — error-toast path now
covered (5 pass).
- [x] CodeRabbit CLI review: 0 findings.
- [x] Typecheck: no new errors from these changes (pre-existing errors
unrelated).

## Review
- [x] CodeRabbit CLI review completed locally
- [x] Issues addressed

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 12:31:19 -04:00
Andrew Beveridge ☄️andClaude Opus 4.7 7268dfd830 feat(duet): multi-singer karaoke support + local CLI review fixes (#730)
## Summary

- **New feature:** Multi-singer / duet support. Review UI gains a "Duet
Mode" toggle and a per-segment chip to assign Singer 1 / Singer 2 /
Both. Rendered MP4 shows each singer in their signature color (blue /
pink / yellow), sweeping to a near-white tint of the same hue as each
word is sung.
- **Unplanned but necessary:** ~15 local CLI regression fixes that
landed alongside. The i18n work on `main` (LocaleRedirect, static-export
locale prefixes) silently broke the local `karaoke-gen` CLI review flow;
shipped-with-package `ReviewServer` was also missing several
`/api/review/{jobId}/*` endpoint aliases that the unified frontend
expects. This branch closes all of that so the local flow works
end-to-end again.

Full design spec:
[`docs/archive/2026-04-21-multi-singer-duets-design.md`](./docs/archive/2026-04-21-multi-singer-duets-design.md).
Implementation plan:
[`docs/archive/2026-04-21-multi-singer-duets-plan.md`](./docs/archive/2026-04-21-multi-singer-duets-plan.md).

## Changes — Duet Feature

**Backend**
- `Word.singer` / `LyricsSegment.singer` optional fields (`SingerId =
Literal[0,1,2]`, 0=Both)
- `DEFAULT_KARAOKE_STYLE` gains a `singers` block with per-singer colors
(unhighlighted = signature color, highlighted = near-white hue tint)
- `CDG_DUET_SINGERS` palette constant (+ `build_cdg_lyrics` helper —
wired into the render flow but not yet fed into `CDGGenerator`'s TOML
builder; follow-up tracked in `docs/LESSONS-LEARNED.md`)
- `ass/style.py` gains `build_karaoke_styles` factory that emits one ASS
style per singer in use
- `LyricsLine` / `LyricsScreen` / `SubtitlesGenerator` /
`SegmentResizer` all forward segment + word singer fields through the
render chain (several bugs where rebuilds silently dropped the field
were fixed)
- `is_duet` flag persisted in review-session `state_data` (cloud) and
captured by the local `ReviewServer` with a latch-up-only semantic on
`complete_review` so InstrumentalSelector can't accidentally downgrade
it
- `CorrectionOperations.generate_preview_video` threads `is_duet` into
the inner `preview_config`
- `render_video_worker` reads `is_duet` from state_data and propagates
through `OutputGenerator` → `SubtitlesGenerator` (both local + GCE
paths)

**Frontend (`frontend/`, sources only — bundle at
`karaoke_gen/nextjs_frontend/out/` also rebuilt)**
- `SingerId` type + optional `singer` on `Word` / `LyricsSegment` /
`CorrectionData.is_duet`
- `duet.ts` helpers: `cycleSinger`, `resolveSegmentSinger`,
`collectSingersInUse`, `hasMultipleSingers`, `hasWordOverrides`
- `SingerChip` component (click to cycle 1 → 2 → Both, shows `*` when
words have overrides)
- `Header` "Duet Mode" toggle (opt-in, hides chip column when off)
- `TranscriptionView` renders chip + subtle row-tint gradient when duet
is on
- `LyricsAnalyzer` manages `isDuet` state, includes it in save payloads
- `segmentOperations.ts` extends split/merge/addBefore/delete with
singer-inheritance rules (including the tricky implicit-vs-explicit
singer case)
- Keyboard shortcuts `1` / `2` / `B` on a focused segment
- `InstrumentalSelector` falls back to `hasMultipleSingers(segments)`
when `correctionData.is_duet` is missing
- Auto-enable duet on session-restore when restored data shows
multi-singer evidence
- `lyricsReview.duet.*` namespace added to `frontend/messages/en.json`
and auto-translated to all 32 other locales

**i18n**
- Every new user-visible string goes through `next-intl`; the
translation pipeline ran for each en.json change
- Backend new error strings added to
`backend/translations/{en,es,de}.json` and tracked via
`.en-snapshot.json`

## Changes — Local CLI Review Regression Fixes

These were uncovered while trying to manually validate the duet feature
end-to-end. They land in this PR because blocking validation would have
blocked the feature.

- `ReviewServer` now serves the `/{locale}/app/jobs/local/review` (and
broader `/{locale}/app/...`) paths that `LocaleRedirect` unconditionally
bounces to — discovers valid locales from the frontend build
- Added missing `/api/review/{jobId}/...` aliases: `search-lyrics` (new
handler using `CorrectionOperations.search_lyrics_sources`),
`add-lyrics`, `handlers` (POST, matching cloud), `audio/{stem-or-hash}`
(dispatches stems vs audio-by-hash), `instrumental-analysis`,
`waveform-data`, `v1/annotations`, `sessions` stubs
- Review-session history persistence added (external session)
- `job-router-routes.ts` now parses locale-prefixed pathnames so
`/en/app/jobs/local/instrumental` renders the InstrumentalSelector
instead of the lyrics review fallback
- Test-mock signatures updated for the `run_combined_review` 3-tuple
return

## Testing

- [x] Unit tests pass: **1606 package tests + 3002 backend tests + 848
frontend tests**, all green. Coverage: 69.29% (≥69% gate)
- [x] Integration tests pass (including new end-to-end multi-singer
fixture tests)
- [x] Playwright production E2E spec compiles
(`frontend/e2e/production/duet-review.spec.ts`)
- [x] Manual validation done end-to-end on Lady Gaga × Bradley Cooper —
"Shallow (145s duet clip)" — full pipeline produces a proper
duet-rendered MP4 with three per-singer ASS styles tagged correctly
against dialogue lines (6 Singer 1, 16 Singer 2, 4 Both across this
clip). Checked-in fixture at `tests/data/Shallow-Duet-145.ogg`.
- [ ] Manual validation against a second track with dense
call-and-response (spec listed Eminem × Rihanna "Love the Way You Lie")
— not done this PR; can follow up after merge if anything surprising
surfaces
- [ ] End-to-end CDG duet output — still deferred (see Known Follow-Ups)

## Review

- [x] Local code-review done via `feature-dev:code-reviewer` agent
(CodeRabbit CLI fell over on the 3605-file bundle rebuild commit)
- [x] Two findings addressed in commit `da3004ea`: handlers HTTP method,
mergeSegment implicit-singer case

@coderabbitai ignore

## Known Follow-Ups

- `build_cdg_lyrics` helper is implemented and unit-tested but
`CDGGenerator` still constructs its TOML config directly rather than via
`SettingsLyric`. CDG output from duet jobs renders in solo colors today.
Documented in `docs/LESSONS-LEARNED.md`; not required for the MP4/ASS
path which is the feature's primary surface.
- Cloud `complete_review` overwrites `is_duet` in state_data
unconditionally (local server has latch-only semantics). In practice the
frontend's `hasMultipleSingers` fallback in `InstrumentalSelector` keeps
this consistent, but the cloud could gain the same latch for parity.

## Version

Bumped `pyproject.toml` from `0.171.0` → `0.171.1`.

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-22 23:27:21 -04:00
Andrew Beveridge ☄️andClaude Opus 4.7 26026edea1 fix(error-monitor): alert on out-of-band patterns (e.g. frontend crashes) (#727)
@coderabbitai ignore

Follow-up to #725 — found during post-deploy verification. The
error-monitor Cloud Run Job only alerted on patterns its own Cloud
Logging scan produced. Frontend crash patterns inserted via the new
\`POST /api/client-errors\` endpoint landed in Firestore with
\`status=new\`/\`alerted_at=None\` but never fired a Discord alert.

## Fix

- New adapter method \`get_unalerted_new_patterns()\` — queries
Firestore for \`status=new\` AND \`alerted_at=None\`.
- New Step 3b in the monitor cycle merges those out-of-band patterns
into \`new_pattern_ids\` + \`groups\` before the LLM/dedup/alert steps.
- Early-exit moved from "no log entries" to "no patterns AND no
unalerted" so an empty-log cycle still flushes pending frontend alerts.
- 3 new unit tests for the adapter method.

## Verified reproduction (pre-fix)

Three frontend patterns from #725's live-verification are currently
sitting in \`error_patterns\` with \`alerted_at=None\`:
- \`fad2f5811881700e...\` (curl probe)
- \`f84b386bc152102e...\` (async throw via Firefox)
- \`13a3b42284f7e66f...\` (unhandled rejection via Firefox)

The 01:31 UTC monitor execution ran and processed 1 backend log entry —
our 3 frontend patterns were ignored because they weren't in
\`new_pattern_ids\`.

## Test plan

- [ ] Merge + deploy.
- [ ] Manually trigger the Cloud Run Job (\`gcloud run jobs execute
nomad-error-monitor --region=us-central1\`).
- [ ] Confirm Discord receives 3 alerts for the pending frontend
patterns.
- [ ] Confirm those patterns now have \`alerted_at\` set in Firestore.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-20 01:48:49 +00:00
Andrew Beveridge ☄️andClaude Opus 4.7 6445864534 feat: frontend crash reporting → Discord alerts + auto cache bust (#725)
@coderabbitai ignore

## Summary

Gives us long-term peace of mind that frontend errors never reach users
silently. Every client-side crash is captured, shown to the user with
useful debug info, reported to the backend, alerted via the existing
Discord error channel, and — critically — after a fix ships, affected
users pick it up automatically.

## What this does

- **Backend endpoint** `POST /api/client-errors` — normalises inbound
reports and upserts into the same `error_patterns` Firestore collection
the backend already uses. No new Discord wiring needed:
`service="frontend"` patterns flow through the existing error-monitor
Cloud Run Job → Discord pipeline.
- **Three capture layers on the frontend**:
- Global `window.onerror` + `unhandledrejection` listeners (mounted once
per tab via `ClientErrorInit` in the locale layout)
- Next.js App Router `error.tsx` (per-locale) + `global-error.tsx`
(last-resort root)
- React `CrashReportBoundary` wrapping `LyricsReviewWrapper` +
`InstrumentalReviewWrapper` so a crash there renders in place rather
than white-screening the route
- **User-facing `CrashReport` card** — shows error name/message +
collapsible stack + debug info + build SHA; auto-sends on mount (catches
bugs in the wild without user action); has Copy / Send again / Try again
/ Back buttons.
- **Version-aware cache busting** so the next bug I fix actually reaches
affected users:
- `public/version.json` emitted at build time via a new `prebuild` hook.
- `isStale()` compares the compiled `NEXT_PUBLIC_COMMIT_SHA` against the
live `/version.json`.
- `ChunkLoadError` in the global handlers triggers a hard reload
(unregisters SWs, clears caches, navigates with `?_v=<sha>`
query-string) behind a 60 s circuit breaker to prevent reload loops.
- `CrashReport` shows a prominent "A newer version is available — Update
now" banner when stale; demotes "Try again" to secondary in that case.
- Ambient 10-minute poll dispatches a `karaoke:stale-version`
CustomEvent (no auto-reload ambient — only on crash).
- **Admin `/admin/_crash-test` page** — three buttons that fire async
throw / unhandled rejection / render-time throw, for post-deploy
pipeline verification.
- **Rate limiting** on the endpoint: 60/min per IP sliding window, plus
pattern-hash dedup collapsing viral bugs to a single pattern.
- **Privacy**: URL query strings + fragments stripped; user email only
sent if authenticated; no cookies / localStorage / form-value capture.
- **i18n**: `crashReport` namespace in `en.json` translated into all 32
locales.

Reuses existing env var `NEXT_PUBLIC_COMMIT_SHA` — no new build config.

## Review & verification

- 49 Jest suites / 800 tests pass (`make test`).
- New unit tests: `crash-reporter.test.ts`, `CrashReport.test.tsx`,
`CrashReportBoundary.test.tsx`, `version-check.test.ts`,
`test_frontend_ingestion.py`, `test_client_errors.py` — all green.
- Implemented task-by-task via subagent-driven workflow with spec + code
quality review per task (see
`docs/archive/2026-04-19-frontend-crash-reporting-plan.md`).
- E2E Playwright test deferred — the admin crash-test page covers manual
post-deploy verification.

## Test plan

- [ ] Deploy to prod.
- [ ] Visit `https://gen.nomadkaraoke.com/admin/_crash-test` as admin;
click each of the three buttons.
- [ ] Confirm a Discord alert arrives in the error channel (may take up
to ~15 min per the monitor Cloud Run schedule).
- [ ] Verify Firestore `error_patterns` contains 3 new docs with
`service="frontend"`.
- [ ] In a separate browser session already open to the app, simulate a
stale-bundle scenario (edit `public/version.json` on the CDN, or use an
older tab after a new deploy) — confirm `CrashReport` shows the "Update
now" banner if an error is thrown.
- [ ] Test the original Firefox Android bug (if it reproduces) and
verify the crash now lands in Discord with a usable stack trace.

## Follow-ups

- Source-map upload for readable production stacks (`Task 15` in the
plan).
- Consider forwarding the `source` payload field into
`PatternData.sample_message` if finer-grained Discord triage
(`window.onerror` vs `unhandledrejection` vs boundary) becomes valuable.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-19 21:14:12 -04:00
Andrew Beveridge ☄️andClaude Opus 4.7 c36683f065 fix(lyrics): log upstream 5xx/timeout as WARNING, not ERROR (#720)
## Summary
- Third-party lyrics providers (Musixmatch, Genius, LRCLIB, Spotify)
returning 5xx/Timeout/ConnectionError are transient upstream failures,
not bugs in our code. We already fall back to other providers and jobs
complete successfully.
- Logging those as ERROR was triggering production error-monitor Discord
alerts for non-actionable noise (see pattern `f9f95f01…` — Musixmatch
502 on 2026-04-19 that a job recovered from using Genius + Spotify).
- Added a shared `BaseLyricsProvider._log_request_exception()` helper
that classifies 5xx / `ConnectionError` / `Timeout` as WARNING and keeps
4xx / other `RequestException` as ERROR (those still indicate a bug in
our request).
- All 5 call sites across the 4 providers updated to use the helper.
Per-provider log labels made more specific (`"Genius RapidAPI"` /
`"Spotify RapidAPI"` instead of the generic `"RapidAPI"`) for easier
debugging.

## Test plan
- [x] New unit tests cover 502, 503, 404, Timeout, ConnectionError, and
generic RequestException severity classification
(`tests/unit/test_base_lyrics_provider.py::TestLogRequestException`, 6
cases)
- [x] Existing lyrics tests continue to pass (35 total)
- [ ] After deploy: confirm the Musixmatch 502 error-pattern stops
regenerating in Firestore (should now log at WARNING and be ignored by
the monitor)

@coderabbitai ignore

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-19 18:24:25 -04:00
Andrew Beveridge ☄️andClaude Opus 4.7 8600b5d674 test(error-monitor): fix pre-existing broken tests (#721)
## Summary
Two pre-existing test breakages surfaced when the `Package Unit Tests`
CI gate re-ran on a newer PR (it had been skipped on recent
frontend-only changes):

1. **`test_config.py`** — assumed the old `MONITORED_CLOUD_FUNCTIONS`
list, but PR #716 moved Gen2 Cloud Functions into
`MONITORED_CLOUD_RUN_SERVICES` (since they log as `cloud_run_revision`).
Tests updated to assert the new structure with the current kebab-case
names.

2. **`test_firestore_adapter.py`** — three tests used hardcoded
datetimes (`2026-04-10`) that are now more than 7 days stale relative to
the real clock, so `_prune_rolling_counts` discarded all fixture
entries. The three affected tests now freeze `_utcnow` via `patch`,
keeping them stable over time.

No production code changes.

## Test plan
- [x] `python -m pytest tests/unit/services/error_monitor/` — all 310
pass locally
- [x] Verified the 6 previously-failing tests now pass
- [x] No changes outside `tests/unit/services/error_monitor/`

@coderabbitai ignore

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-19 18:15:09 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 9a5c9d497a feat: production error monitor — log collection, LLM analysis, Discord alerts (#713)
## Summary

- Add a production error monitoring service that runs as a Cloud Run Job
every 15 minutes
- Queries Cloud Logging across all NK services (Cloud Run, Jobs,
Functions, GCE VMs), normalizes/deduplicates errors, uses Gemini Flash
for incident grouping, sends Discord alerts, and auto-resolves silent
patterns
- Updates all `/prod-*` slash commands to use pre-computed Firestore
patterns as primary data source

## Architecture

Based on the proven Aquarius error bot system, adapted for Nomad
Karaoke's multi-service architecture (20+ resources across 4 resource
types).

**New backend modules** (`backend/services/error_monitor/`):
- `config.py` — constants, monitored services, env var overrides
- `normalizer.py` — regex normalization + SHA-256 pattern hashing
- `known_issues.py` — 8 ignore patterns for infrastructure noise
- `firestore_adapter.py` — CRUD for `error_patterns`, `error_incidents`,
`discord_alerts`
- `discord.py` — webhook client with 5 alert formatters (new pattern,
incident, spike, auto-resolved, daily digest)
- `llm_analysis.py` — Gemini Flash incident grouping + near-duplicate
detection
- `monitor.py` — orchestrator entry point (`python -m
backend.services.error_monitor.monitor`)

**New infrastructure** (`infrastructure/modules/error_monitor.py`):
- Cloud Run Job (`nomad-error-monitor`) sharing karaoke-backend image
- Cloud Scheduler: every 15 min monitor + 08:00 UTC daily digest
- Scheduler service account with `roles/run.invoker`
- Backend SA gets `roles/logging.viewer`

**New CLI scripts:**
- `scripts/query-error-patterns.py` — query patterns from Firestore
- `scripts/resolve-error-pattern.py` — mark patterns as fixed with PR
reference

## Testing

- 310 unit tests across 7 test modules (all passing)
- Covers: config, normalizer, known issues, Firestore adapter, Discord
formatting, LLM analysis, monitor orchestrator
- P0/P1 caller-callee contract tests for monitor integration points
- All tests use mocked external dependencies (no GCP calls)

## Review

- [x] CodeRabbit CLI review completed locally (1 cycle, 3 issues fixed)
- [x] Test review completed (P0/P1 gaps addressed)
- [x] Documentation updated (README, ARCHITECTURE)
- [x] Version bumped (0.168.2 → 0.169.0)

## Design Spec

`docs/archive/2026-04-12-error-monitor-design.md`

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 01:02:08 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 1c846ebae7 fix: preserve gcs_path in divebar catalog sync (#649)
## Summary

- **index_builder.py**: replaced `WRITE_TRUNCATE` with staging table +
`MERGE` SQL to preserve `gcs_path` values during daily catalog index
updates
- **divebar_file_sync.py**: added `blob.exists()` check before
downloading from Drive — if a file is already in GCS, just updates
`gcs_path` in BigQuery without re-downloading
- Added unit tests verifying the MERGE approach preserves `gcs_path`
correctly

**Root cause**: The daily 2 AM index sync was atomically replacing the
entire `divebar_catalog` BigQuery table, wiping all `gcs_path` values.
The 3 AM file sync then saw all ~48K files as unsynced and re-downloaded
the entire ~779 GB catalog from Google Drive every night.

## Test plan

- [x] Unit tests for staging+MERGE behavior (5 tests pass)
- [ ] Verify staging table auto-creates on first run (BigQuery creates
on WRITE_TRUNCATE with schema)
- [ ] Monitor next nightly sync — file_sync should report "recovered
from GCS" instead of downloading
- [ ] Confirm gcs_path values persist across consecutive index runs

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 22:31:42 +00:00
Andrew Beveridge ☄️andClaude Sonnet 4.6 c02fdc8d9f Fix GCE cache invalidation and empty corrections handling (#648)
## Summary
- **GCE style asset disk cache** now validates file size against GCS on
cache hit, re-downloading when content has changed at the same URI.
Transient GCS errors gracefully fall back to cached version.
- **Review complete endpoint** no longer writes
`corrections_updated.json` when no corrections data is present (root
cause fix for KeyError in render workers).
- **Both GCE and Cloud Run render workers** validate
`corrections_updated.json` has a `corrections` key before processing,
falling back to base result gracefully (defensive fix).

## Test plan
- [x] Added `test_cache_hit_redownloads_on_size_mismatch` — verifies
stale cache replaced
- [x] Updated `test_cache_hit_skips_download` for size validation flow
- [x] Added `test_gce_worker_skips_empty_corrections` — verifies empty
dict doesn't crash
- [x] Added `TestCompleteReviewSkipsEmptyCorrections` — 4 tests covering
guard conditions
- [x] All 2875 tests pass (8 pre-existing emulator failures unrelated)

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-31 18:03:26 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 3bc1fcb250 fix: move screen video generation to GCE encoding worker (#647)
## Summary

- Screen MOV generation (4K H.264) was silently failing on Cloud Run
after right-sizing to 2Gi (PR #640), producing 36-byte empty shells that
broke all downstream encoding
- `screens_worker` now only generates PNG/JPG images (Pillow, no FFmpeg)
- GCE encoding worker generates MOV from PNG before encoding, with
proper `subprocess.run()` error handling
- Fixed `os.system()` in `video_generator.py` to use `subprocess.run()`
so failures are no longer silent

## Root Cause

PR #640 reduced Cloud Run from 8CPU/16Gi to 2CPU/2Gi. The
`VideoGenerator._create_video_from_image()` used `os.system(ffmpeg ...)`
which OOMs silently for 4K H.264 encoding, producing 36-byte empty MOV
shells. All jobs since March 31 are affected.

## Test plan

- [x] 2875 tests pass (8 pre-existing emulator failures unrelated)
- [x] CodeRabbit: no findings
- [ ] Re-run failed job f0d51a0a after deploy to verify fix
- [ ] Verify admin "Regen screens" button works (tested flow: deletes
old screens → worker generates PNG/JPG only → GCE generates MOV from
PNG)

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 21:44:53 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 0464d3edb3 Fix render_video GCE endpoint import paths (#643)
@coderabbitai ignore

## Summary

- Fix broken import paths in GCE worker's `/render-video` endpoint that
caused `ImportError: No module named 'backend.output_generator'` in
production
- Fix incorrect API signatures for OutputGenerator, CountdownProcessor,
CorrectionResult, load_styles_from_gcs
- Update tests to mock correct module paths

## Root cause

The subagent that implemented Task 2 (PR #641) used invented module
paths (`backend.output_generator`,
`backend.services.countdown_processor`, etc.) instead of the actual
karaoke-gen wheel paths
(`karaoke_gen.lyrics_transcriber.output.generator`, etc.). The tests
also mocked the wrong paths, so they passed locally but the code failed
in production.

## Changes

- `backend/services/gce_encoding/main.py` — Rewrote `run_render_video()`
to use correct imports and API calls matching `render_video_worker.py`
- `tests/unit/test_gce_render_video_endpoint.py` — Rewrote
TestRunRenderVideo to mock correct module paths

## Test plan

- [ ] 18 unit tests pass (verified locally with
`GOOGLE_CLOUD_PROJECT=""` to simulate CI)
- [ ] E2E happy path test after merge validates full pipeline

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 14:01:36 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 1cf6b62398 Route render_video to GCE encoding worker (#641)
@coderabbitai ignore

## Summary

- Route `render_video_worker` (OutputGenerator/ffmpeg lyrics rendering)
to the GCE encoding worker instead of running on Cloud Run
- Fixes OOM crashes after Cloud Run memory was right-sized to 2Gi
- Adds style asset disk cache on GCE worker to avoid repeated downloads
of the same default theme
- Preserves local fallback when GCE is disabled

## Changes

### GCE Worker (`backend/services/gce_encoding/main.py`)
- New `/render-video` endpoint with `RenderVideoRequest` model
- `run_render_video()` — downloads corrections/audio/styles from GCS,
runs OutputGenerator, uploads outputs
- `download_with_cache()` — SHA-256 keyed disk cache for style assets
(fonts, backgrounds)
- `metadata` field added to `JobStatus` model for countdown padding info

### EncodingService (`backend/services/encoding_service.py`)
- `submit_render_video_job()` — POST to `/render-video` with retry logic
- `render_video_on_gce()` — submit + wait convenience method

### render_video_worker (`backend/workers/render_video_worker.py`)
- Conditional GCE routing when `encoding_service.is_enabled`
- Cloud Run does zero downloads in GCE path — constructs GCS paths from
Firestore metadata
- Existing local path preserved unchanged as fallback

## Test plan

- [ ] 18 new unit tests covering cache, endpoint, service methods, and
routing logic (all pass)
- [ ] Full test suite: 1164 passed, 0 failures
- [ ] CodeRabbit review: 1 bug caught and fixed (missing metadata in
cached response)
- [ ] E2E happy path CI validates full pipeline after merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 16:58:00 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 022b4e343a fix: backing vocals recommendation showing 0% with contradictory message (#636)
## Summary

- Fixed API defaulting `recommended_selection` to `with_backing` when no
analysis exists, creating contradictory "Significant backing vocals
(0%)" message — now defaults to `clean`
- Fixed analyzer returning `review_needed` (unknown to frontend) instead
of `with_backing` when backing vocals detected
- Fixed race condition where backing vocals analysis ran in
screens_worker before audio separation completed — moved to audio_worker

## Test plan

- [x] All 25 analyzer unit tests pass (3 updated to match new
`WITH_BACKING` return value)
- [x] All 21 frontend instrumental review component tests pass
- [x] Full test suite: 2863 passed (8 pre-existing emulator failures)
- [x] CodeRabbit: no findings
- [ ] Verify fix in production: create new job and check instrumental
review shows correct recommendation

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-30 00:39:59 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 16fcb77d0f fix: relevance filtering in correction_disabled path + guidance message (#635)
## Summary

Two fixes found during post-deploy testing of #632:

1. **Relevance filter bypassed when SKIP_CORRECTION=true** — The
controller's `create_uncorrected_result()` path builds CorrectionResult
directly without going through LyricsCorrector, so wrong-song references
were never filtered. Now applies `filter_irrelevant_sources()` in this
path too.

2. **Misleading "All lyrics matched" message** — When no reference
sources exist (all filtered or none found), the guidance panel showed
"All lyrics matched the reference — no corrections needed." Now shows
"No reference lyrics were found" with guidance to use the search panel.

## Testing

- 9/9 relevance tests passing
- 22/22 GuidancePanel tests passing (updated + new test for empty state)

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-30 04:21:58 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 591c050545 feat: Reference lyrics search with relevance filtering (#632)
## Summary

- Add relevance filtering to the correction pipeline — automatically
rejects wrong-song reference lyrics (below 30% word match threshold,
validated against 168 prod source pairs)
- New `POST /api/review/{job_id}/search-lyrics` endpoint lets users
search with alternate artist/title from the review UI, with force-add
override for edge cases
- Redesigned Add Reference Lyrics modal with Search (default) and Paste
tabs, plus inline empty state when no valid sources exist

## Changes

### Backend
- **Relevance filtering** (`correction/relevance.py`): Per-source
scoring based on anchor sequence word coverage. Integrated into
`LyricsCorrector.run()` after anchor finding — sources below threshold
removed before corrections computed
- **Search endpoint** (`routes/review.py`): Queries all providers
(Genius, Spotify, Musixmatch, LRCLIB) with user-supplied artist/title,
auto-filters results, returns added/rejected/not-found breakdown
- **Provider factory** (`operations.py`): `create_lyrics_providers()`
reads env vars, `search_lyrics_sources()` orchestrates
fetch+filter+correct
- **Force override**: `add-lyrics` and `search-lyrics` support `force`
flag to bypass threshold

### Frontend
- **Tabbed modal**: `AddLyricsModal` rewritten with Search (default) and
Paste tabs
- **SearchLyricsTab**: Artist/title inputs, provider search, rejected
source cards with force-add checkboxes
- **ReferenceEmptyState**: Inline search/paste UI when no valid
reference sources exist
- **API client**: New `searchLyrics()` method with
`SearchLyricsResponse` types

### Analysis
- Empirical threshold analysis script
(`scripts/analyze_reference_relevance.py`) + visual review tool
- 168 source pairs from 81 prod jobs: wrong-song max 3.5%, correct-song
min 26.8% → 30% threshold

## Testing
- 35 new backend tests (relevance scoring, filtering, corrector
integration, search operation, API endpoint)
- 717 existing frontend tests passing (no regressions)
- Edge case: all sources filtered out handled gracefully

## Review
- [x] Local CodeRabbit review completed (3 issues fixed: response
contract, force param coercion, null checks)

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-30 00:25:47 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 78e4cc7010 fix: bounds check ref_words in anchor gap extension (IndexError b7e8c96d) (#623)
## Summary

- Fix IndexError in `_extend_anchors_into_single_word_gaps` when
`ref_texts_clean` and `ref_words` have different word counts due to
`clean_text()` normalization
- Adds production fixture regression test from crashing job `b7e8c96d`
(Bằng Kiều - Dẫu có lỗi lầm)

## Root Cause

`ref_texts_clean` is built by joining words → `clean_text()` →
`split()`, which can produce a **different word count** than the raw
`ref_words` Word objects list. For Spotify in the crashing job:
`ref_texts_clean` had 599 words, `ref_words` had 528. Anchor
`reference_positions` computed against the 599-word list were used to
index the 528-word list.

## Changes

- `anchor_sequence.py`: Add `< len(ref_words[source])` bounds check in
both `_apply_forward_extension` and `_apply_backward_extension`
- `test_anchor_gap_extension_bounds.py`: 4 regression tests (production
fixture + synthetic bounds tests)
- `anchor_crash_b7e8c96d.json`: Sanitized production fixture (49
anchors, 526 words)
- `LESSONS-LEARNED.md`: Document text normalization word count mismatch
gotcha
- Version bump: 0.153.4 → 0.153.5

## Testing

- [x] 33 anchor gap extension tests pass (29 existing + 4 new)
- [x] Full unit suite: 1146 tests pass
- [ ] After deploy: retry job `b7e8c96d`

## Review

- [x] CodeRabbit CLI review completed locally (1 finding fixed:
sanitized copyrighted lyrics)

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-28 14:23:13 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 e3c55672e5 fix: bounds check in anchor sequence gap extension (#621)
## Summary

- Fix IndexError in `_extend_anchors_into_single_word_gaps` when
`gap_position >= len(all_words)`
- This crashes lyrics transcription for songs where the last anchor ends
at the transcription boundary
- Affected job: `b7e8c96d` (Bằng Kiều - Dẫu có lỗi lầm)

## Fix

Add `if gap_position >= len(all_words): continue` before accessing
`all_words[gap_position]`.

## Test plan

- [x] 2 new boundary regression tests in test_anchor_gap_extension.py
- [x] Full unit suite: 1142 passed (29 anchor gap tests)
- [ ] After deploy: retry job b7e8c96d

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-27 19:51:42 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 0c996d1c8e fix: rename Stage 1 vocals before Stage 2 to avoid stem tag regex collision (#618)
## Summary

Follow-up to #617 — the stem tag parsing fix alone wasn't enough. The
audio-separator library's ensemble code uses
`re.search(r'_\(([^)]+)\)')` to classify intermediate stems, matching
the **first** `_(Tag)_` pattern. When Stage 2 receives Stage 1 output
like `Song_(Vocals)_preset_instrumental_clean.flac`, the regex matches
the Stage 1 tag instead of the Stage 2 tag, classifying ALL stems as
"Vocals" and producing only 1 output file instead of 2.

## Fix

Copy Stage 1 vocals to a clean filename (`Artist - Song (Vocals).flac`)
before passing to Stage 2, avoiding the regex collision.

## Test plan

- [x] Unit tests updated to verify clean filename is used for Stage 2
input
- [x] Full test suite: 1140 passed, 0 failed
- [ ] After deploy: trigger test job and verify backing vocals +
combined instrumental stems appear

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-27 14:18:07 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 f58c74e67e fix: ensemble preset stem tag parsing — restore backing vocals separation (#617)
## Summary

- Fix stem filename parsing bug where preset name `instrumental_clean`
caused the Vocals file to be misclassified as instrumental, preventing
Stage 2 (backing vocals separation) from running
- All new GPU-processed jobs were only producing `instrumental_clean` —
no backing vocals, lead vocals, or combined instrumentals
- The instrumental review page showed "No backing vocals detected" for
all jobs

## Root Cause

Ensemble preset output filenames follow
`{base}_(StemTag)_preset_{name}.flac`. The old code did `"instrumental"
in basename` which matched the **preset name** `instrumental_clean` in
the Vocals file, so `result["vocals"]` was never set, and Stage 2 never
ran.

## Fix

Extract the stem tag from between `_(` and `)_` delimiters and match
only that, not the full filename.

## Test plan

- [x] 5 new regression tests for stem tag parsing (both stages)
- [x] 2 pre-existing tests fixed to use realistic ensemble filenames
- [x] Full unit suite: 1140 passed, 0 failed
- [x] CodeRabbit: no findings
- [ ] After deploy: trigger a test job and verify all stems appear on
instrumental review page

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-27 12:14:32 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 92ae3fb28f feat: run audio separation directly on GPU in audio worker (#608)
## Summary

- **Eliminates HTTP hop** between the audio worker Cloud Run Job and the
audio-separator Cloud Run Service by running the `Separator` class
directly with an L4 GPU attached
- **Adds GPU Docker image** (`karaoke-backend-gpu`) with CUDA 12.6,
PyTorch, and pre-baked ensemble preset models
- **Moves audio worker to us-east4** with L4 GPU, 4 CPU, 16Gi memory —
old separator service kept for rollback

## Changes

### Code
- `audio_processor.py` — Ensemble presets wired into local Separator
path (2-stage: `instrumental_clean` + `karaoke`)
- `audio_worker.py` — Dual-mode support: `MODEL_DIR` env var → local
GPU, `AUDIO_SEPARATOR_API_URL` → remote API
- `create_audio_processor()` — New `model_file_dir` parameter to enable
local GPU mode

### Infrastructure (Pulumi)
- `gpu_artifact_registry.py` — New Artifact Registry in us-east4 for GPU
images
- `cloud_run.py` — Audio worker job: GPU (nvidia-l4), us-east4, 16Gi,
`MODEL_DIR=/models`
- `__main__.py` — Wire in GPU registry + Cloud Build IAM

### Docker
- `Dockerfile.gpu-base` — CUDA 12.6 + Python 3.11 + audio-separator[gpu]
+ baked models
- `cloudbuild-gpu-base.yaml` — Cloud Build config (E2_HIGHCPU_32, 100GB
disk, 1h timeout)
- `download_models.py` — Bakes `instrumental_clean` + `karaoke` preset
models at build time

### Tests
- 8 new unit tests covering factory config, mode selection, and preset
wiring
- Updated existing test patch targets for module-level Separator import

## Testing
- [x] 22/22 unit tests pass (8 new + 14 existing)
- [x] Code review completed (subagent-driven + final review)
- [x] CodeRabbit CLI review completed locally
- [ ] GPU image build (requires `gcloud builds submit`)
- [ ] Production verification with `under_pressure_harmonies.flac`

## Rollback Plan
Set `AUDIO_SEPARATOR_API_URL` back on the audio worker job → falls
through to remote API path. Old separator service is still running
(scale-to-zero).

## Design & Plan
- Design spec:
`docs/archive/2026-03-26-direct-gpu-audio-separation-design.md`
- Implementation plan:
`docs/archive/2026-03-26-direct-gpu-audio-separation-plan.md`

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-26 15:13:51 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 7d887a11b0 Fix FLAC uploads to AudioShake by stripping metadata (#593)
## Summary
- Strip metadata from FLAC files before uploading to AudioShake to
prevent 502 errors caused by non-standard tags (MP4/M4A container fields
embedded in FLAC)
- Uses `ffmpeg -map 0:a -c copy -map_metadata -1` to remove metadata
while keeping audio unchanged
- Added unit + integration tests for the new metadata stripping path

## Context
AudioShake confirmed that non-standard metadata tags in FLAC files cause
their processing pipeline to fail during task creation. The audio itself
is valid — only the metadata is problematic. This fix proactively strips
all metadata before upload.

## Test plan
- [x] Unit test: FLAC files route through `_strip_metadata` instead of
direct upload
- [x] Integration test: metadata is actually removed from output FLAC
(verified via ffprobe)
- [x] All 30 AudioShake tests pass
- [x] Full test suite: 1117 passed (3 pre-existing failures unrelated to
this change)

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-24 23:18:56 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 7294db2f6d fix: ASS subtitle preview/render divergence — embedded newlines drop lyrics (#578)
## Summary

- **Bug:** Lyrics that rendered correctly in preview videos were missing
words in final (with_vocals/final) video output. Discovered on job
`0d4b550c` (The Ataris - All Souls Day).
- **Root cause:** Word objects from the transcription pipeline contain
embedded newline characters (e.g., `"spell\n"`) at segment boundaries.
These corrupt ASS subtitle dialogue events by splitting them across file
lines, causing the video renderer to silently drop words after the
newline.
- **Why preview worked but final didn't:** Two divergent
CorrectionResult construction paths — preview used
`update_correction_result_with_data()` which `.strip()`s word text,
while the render worker used manual dict merge + `Word.from_dict()`
which did not strip.

## Changes

- **`Word.__post_init__`** strips whitespace/newlines from text — single
source of truth for Word invariants
- **Defensive strip in `_create_ass_text()`** — belt-and-suspenders
safety net at the ASS output boundary
- **Render worker refactored** to use the same
`CorrectionOperations.update_correction_result_with_data()` method as
preview generation (DRY, eliminates divergence)
- **Removed redundant `.strip()`** from
`update_correction_result_with_data()` Word text (now handled by
`__post_init__`)
- **Documented lesson** in `docs/LESSONS-LEARNED.md`: "Data Types Must
Enforce Their Own Invariants"
- **8 unit tests** covering Word stripping and ASS output sanitization

## Testing

- [x] 1126 unit tests pass, 0 failures
- [x] Sanity check with real bug data from job `0d4b550c` confirms fix
- [x] CodeRabbit CLI review completed, feedback addressed

## Review

- [x] Local CodeRabbit review completed
- [x] Review feedback addressed (test improved to bypass `__post_init__`
and exercise defensive strip independently)

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-22 23:20:12 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 38b966f30c Modal → GCP audio separation migration (#571)
@coderabbitai ignore

## Summary
- Comprehensive migration plan: Modal → Cloud Run GPU (L4) for audio
separation
- Replaces old stub plan with full implementation details
- **Architecture**: Cloud Run GPU service (scale-to-zero, ~$0.13/job)
- **Model upgrade**: Ensemble presets as default (instrumental_clean +
karaoke) for +35% quality
- **Pipeline optimization**: Decouple separation from lyrics review
critical path
- Implementation will be added to this PR

## Plan highlights
- Phase 1: Cloud Run GPU server (python-audio-separator repo)
- Phase 2: GCP infrastructure via Pulumi (this repo)
- Phase 3: Pipeline decoupling + preset-based model config
- Phase 4: Cutover, monitor, decommission Modal

See `docs/archive/2026-03-22-modal-to-gcp-migration-plan.md` for full
details.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-22 22:20:55 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 25aa2ef8ef Fix three production job failure modes with robust error handling (#565)
@coderabbitai ignore

## Summary

Fixes three distinct production job failures identified from
investigating jobs 503a4ad3, 7be8d1e5, and 1868f323:

- **0-segment lyrics guard**: Block review submission when transcription
produces no lyrics (e.g. karaoke input with no vocals). Backend
validation + frontend warning with guidance. CDG orchestrator gracefully
skips when LRC has no content.
- **Audio processor stage 2 validation**: Validate backing vocals
separation produced all expected stems (mirrors existing stage 1
validation). Defensive null checks prevent confusing NoneType crashes.
Full tracebacks in error logs.
- **AudioShake retry with exponential backoff**: 5 attempts via
tenacity, ~40 min spread (60s base, 3x multiplier), retries on
5xx/429/connection errors. Applied to all HTTP call points. Fixed file
handle exhaustion bug on upload retry.

Also includes:
- Troubleshooting runbooks for all 3 failure modes
- Lessons learned entry on pipeline robustness patterns
- Seed planning doc for Modal-to-GCP audio separation migration
- Version bump to 0.147.0

## Test plan

- [x] 2696 backend tests passing (54 new/modified)
- [x] 712 frontend tests passing (5 new for ReviewChangesModal)
- [x] HTTP endpoint test verifies 400 response for 0-segment review
submission
- [x] AudioShake retry tests verify 502 retry, 400 no-retry, connection
error retry
- [x] Audio processor tests verify stage 2 validation, null safety,
combined instrumentals
- [x] CDG orchestrator tests verify graceful skip on empty LRC, fail on
valid LRC errors
- [x] LRC timestamp regex tests for millisecond, centisecond, and
no-decimal formats

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-21 17:58:31 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 1c0eb8bf52 fix: prevent style_params.json lifecycle deletion, CDG font fallback, video worker state guard (#530)
## Summary
- Fix GCS lifecycle rule silently deleting `style_params.json` after 7
days by moving style files from `uploads/` to `jobs/` prefix
- Improve CDG font fallback chain to try bundled fonts by filename
before failing, preventing opaque `KeyError: 'font'` errors
- Add video worker state guard (matching audio/lyrics/render worker
pattern) to prevent cascading retry errors on failed jobs

## Changes
- **Style path prefix**: Changed all `uploads/{job_id}/style/`
references to `jobs/{job_id}/style/` in `theme_service.py`,
`file_upload.py`, `audio_search.py` and 7 test files
- **CDG font fallback**: Enhanced `_validate_and_setup_font()` with
5-step resolution (exact → bundled full path → bundled by filename →
arial.ttf → None), added None guard in `_create_toml_data()`,
absolutized all resolved paths
- **Video worker guard**: Added `validate_worker_can_run()` to both
orchestrated and legacy video pipelines, added terminal state check
(`failed`, `cancelled`, `complete`, `prep_complete`, `error`) to
`_check_worker_idempotency()`
- **Docs**: Added GCS lifecycle gotcha to LESSONS-LEARNED.md

## Context
Triggered by investigation of two production job failures:
- Job `2d4a7bab` (Mar 5): `style_params.json` 404 after 10 days — GCS
lifecycle rule deleted it
- Job `27054772` (Mar 14): CDG generation failed with `KeyError: 'font'`
when font couldn't be resolved, then Cloud Tasks retries produced
cascading `Invalid state transition: failed -> GENERATING_VIDEO` errors

## Testing
- [x] 2551 backend tests pass
- [x] 139 remote CLI tests pass
- [x] 10 new tests added (5 font fallback, 5 state guard/idempotency)

## Review
- [x] Local CodeRabbit review completed (3 findings, 2 fixed, 1 skipped
as nitpick)

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-15 16:19:59 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 e04337fa5d Fix broken notification and CLI review URLs (#507)
## Summary
- Push notifications were sending users to `/review/{id}` and
`/instrumental/{id}` which 404 on Cloudflare Pages static hosting
- Remote CLI was opening browser to path-based URLs missing the `#/`
hash fragment
- Documentation referenced incorrect URL formats

All URLs now correctly use hash-based routing: `/app/jobs#/{id}/review`
and `/app/jobs#/{id}/instrumental`

## Changes
- **backend/services/push_notification_service.py** — Fixed notification
click URLs
- **backend/api/routes/jobs.py** — Fixed edit track response
`review_url`
- **karaoke_gen/utils/remote_cli.py** — Fixed CLI browser open URLs
- **karaoke_gen/utils/cli_args.py** — Fixed help text
- **Tests** — Updated assertions in push notification and remote CLI
tests
- **Docs** — Fixed URL examples in API.md, README.md, TESTING.md
- **pyproject.toml** — Bump version 0.137.0 → 0.137.1

## Test plan
- [x] Backend push notification tests pass (35 tests)
- [x] Remote CLI tests pass (174 tests)
- [x] Full backend test suite passes (103 tests)
- [x] Frontend unit tests pass (581 tests)

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 21:22:28 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 19314fff6e fix: admin endpoints crash with AuthResult unpacking error (#491)
## Summary

- Five admin endpoints in `users.py` crashed with `ValueError: too many
values to unpack (expected 3)` because `require_admin` now returns
`AuthResult` (4-value `__iter__`) but these endpoints still used 3-tuple
unpacking
- Fixed by using `AuthResult` properties directly
(`auth_data.user_email`) instead of tuple unpacking
- Audit trail improved: now logs actual admin email instead of truncated
token prefix

**Affected endpoints:** `POST /admin/credits`, `POST
/admin/users/{email}/disable`, `POST /admin/users/{email}/enable`,
`DELETE /admin/users/{email}`, `POST /admin/users/{email}/role`

## Test plan

- [x] 10 new unit tests covering all 5 fixed endpoints (success,
failure, unicode regression)
- [x] All 1118 existing tests still pass
- [x] Reproduced 500 error via curl before fix, confirmed fix resolves
it

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 04:53:40 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 ac12f2cb19 fix: CJK character support for title cards and downloads (#482)
## Summary

- **Title card renders question marks for CJK characters** → Added font
fallback to Noto Sans CJK (auto-detected via fontconfig) when text
contains Chinese/Japanese/Korean characters. Per-element font selection
so mixed CJK/ASCII titles work correctly.
- **Download links fail with latin-1 encoding error** → Use RFC 5987
`filename*=UTF-8''` encoding in Content-Disposition header for
non-latin-1 filenames. ASCII fallback for older clients.

Fixes job `f8a36b60`.

## Test plan

- [x] 22 unit tests for CJK font detection and fallback logic
- [x] 9 unit tests for Content-Disposition header construction (ASCII,
latin-1, CJK, Cyrillic, RFC 5987 roundtrip)
- [x] 4 integration tests for `_render_all_text` with per-element font
selection
- [x] 1 new test in existing unicode sanitization suite for CJK RFC 5987
encoding
- [x] All 125 related tests passing
- [ ] Deploy and re-test job f8a36b60 download links in production
- [ ] Regenerate title/end screens for job f8a36b60 to verify CJK font
rendering

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 22:14:22 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 4fe960481b feat: show all incomplete jobs first + hide-completed toggle (#393)
## Summary
- **Priority sorting**: Blocking + processing jobs always appear first
in Recent Jobs, regardless of creation date
- **Hide completed toggle**: Eye/EyeOff button filters out completed and
failed jobs, showing only incomplete ones
- **Persistent preferences**: Both the display limit dropdown and
hide-completed toggle persist in localStorage across page reloads
- **Oversized lyrics truncation fix**: Corrected screen generation for
songs with unusually long lyric lines

## Changes
- `frontend/lib/job-status.ts`: Added `hideCompleted` parameter to
`getDisplayJobs()` — filters by job priority before applying display
limit
- `frontend/app/app/page.tsx`: Added Eye/EyeOff toggle with tooltip,
localStorage persistence for both `jobLimit` and `hideCompleted`
- `frontend/__tests__/job-status.test.ts`: 5 new tests covering
hideCompleted filtering behavior

## Test plan
- [x] All 168 tests pass (`make test`)
- [x] 85 job-status unit tests pass including 5 new hideCompleted tests
- [ ] Visual verification: toggle button appears next to limit dropdown,
state persists on reload

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-21 16:22:37 -05:00
Andrew Beveridge ☄️andClaude Sonnet 4.5 d9bb8b3181 fix: sanitize Unicode in temp file paths (FFmpeg exit 183) (#375)
## Summary
Fixes job 1eab1172 which failed with FFmpeg exit code 183 due to
Cyrillic characters in temp subtitle file paths. Python's
`str.isalnum()` returns True for ALL Unicode alphanumeric characters
(Cyrillic, Arabic, Chinese, emoji, etc.), not just ASCII.

## Root Cause
- Job `1eab1172` with Russian title "я куплю тебе дом" failed
- Temp subtitle path contained Cyrillic:
`temp_subtitles_zivert___я_куплю_тебе_дом_1770150285852.ass`
- FFmpeg cannot handle Unicode in filter argument paths → exit code 183

## Changes
- **karaoke_gen/lyrics_transcriber/output/video.py** (lines 222, 282):
- Changed `c.isalnum()` to `c.isascii() and c.isalnum()` for temp
subtitle paths
  - Affects both `generate_video()` and `generate_preview_video()`

- **karaoke_gen/utils/remote_cli.py** (line 1970):
- Changed `c.isalnum()` to `c.isascii() and c.isalnum()` for output
folder names
- Added fallback to `job_{job_id}` if folder name is empty or contains
no alphanumeric chars

- **Tests added** (21 comprehensive tests):
- `test_video_generator_unicode.py` (10 tests): Cyrillic, Chinese,
Arabic, emoji, accented chars
- `test_remote_cli_unicode_folders.py` (11 tests): Folder sanitization,
edge cases, real failure case

- **Documentation**:
- Updated `docs/LESSONS-LEARNED.md` with "Unicode in File Paths (Feb
2026)" section
  - Updated `docs/README.md` with recent changes
  - Bumped version 0.115.1 → 0.115.2

## Testing
- [x] All 1088 unit tests pass (68.91% coverage, target 69%)
- [x] 21 new comprehensive Unicode tests covering:
  - Cyrillic, Chinese, Arabic, emoji, accented characters
  - Empty folder name edge case
  - Separator-only folder name edge case
  - Exact real-world failure case from job 1eab1172
- [x] Verified temp file paths are ASCII-only
- [x] Verified folder names safe for all filesystems

## Review
- [x] CodeRabbit CLI review completed locally (2 cycles)
- [x] All identified issues addressed:
  - Empty folder name fallback added
  - Separator-only folder name guard enhanced
- [x] Test coverage maintained at target

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-05 09:13:33 -08:00
Andrew Beveridge ☄️andClaude Opus 4.5 29be83cb56 Combined review flow: lyrics + instrumental selection in one session (#351)
## Summary

Merges the two-step human review process (lyrics review → instrumental
selection) into a single unified flow. Users now review lyrics AND
select instrumental track in one session.

**Benefits:**
- Better UX (one interaction vs two)
- Reduced drop-off between steps
- Single email notification instead of two
- Simpler workflow

## Changes

### Backend
- `complete_review` endpoint now accepts optional
`instrumental_selection` in request body
- Backing vocals analysis moved from `render_video_worker` to
`screens_worker` (runs before review)
- Removed standalone instrumental selection endpoints (now part of
combined review)
- Added `CompleteReviewRequest` model with validation

### Frontend
- New `InstrumentalSelectorEmbedded` component for combined review UI
- `LyricsAnalyzer` updated to proceed to instrumental review
- `ReviewChangesModal` shows "Proceed to Instrumental Review" button
- Mock data updated in E2E tests

### Local CLI
- `gen_cli.py` refactored to use `run_combined_review()` instead of
separate `run_instrumental_review()`
- `ReviewServer` extended with instrumental options and analysis
- Deleted standalone `InstrumentalReviewServer`

### Documentation
- Updated `docs/README.md` with recent changes entry
- Updated `docs/ARCHITECTURE.md` with new pipeline diagram
- Updated `docs/API.md` with endpoint changes
- Added lesson to `docs/LESSONS-LEARNED.md`

## Test plan

- [x] Backend unit tests: 1742 passed
- [x] Frontend unit tests: 234 passed
- [x] New integration tests: 5 tests for combined flow
- [x] New backend tests: 7 tests for CompleteReviewRequest and
state_data
- [x] New frontend E2E tests: 6 tests for combined flow UI

🤖 Generated with [Claude Code](https://claude.com/claude-code)

@coderabbitai ignore

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-25 11:49:51 -08:00
Andrew Beveridge ☄️andClaude Opus 4.5 5da050e063 fix: resolve phantom gaps in anchor sequence finder (#345)
## Summary

- Fix two bugs causing phantom gaps in lyrics correction
- Add post-filtering gap extension to absorb single-word gaps into
adjacent anchors
- Add comprehensive test fixtures from 14 production jobs

## Bugs Fixed

1. **Initial gap creation bug** (line 986): The condition `if
transcription_position > 0` was always False for position 0, causing the
first word to be completely uncovered. Fixed to `if
transcribed_word_ids`.

2. **Text normalization mismatch** (line 455): Transcription words used
simple `lower().strip()` while references used `clean_text()` which
removes apostrophes. This caused "Train's" != "trains" mismatches. Fixed
to use consistent `clean_text()` for both.

## Results

| Metric | Before | After |
|--------|--------|-------|
| Coverage (Nana Grizol) | 79.6% | 87.0% |
| Phantom gaps at boundaries | Common | Eliminated |
| Words with apostrophes | Mismatched | Correctly matched |

## Test plan

- [x] All 1736 existing tests pass
- [x] New test suite with 14 real production job fixtures
- [x] Visual review of all test fixtures confirms improvement
- [x] Coverage tests verify no regression

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 22:22:58 +00:00
Andrew Beveridge ☄️andClaude b90ff2cc3a feat: fill brand code gaps for numbers >= 1001 (#339)
## Summary
- Brand code calculation now fills gaps for numbers >= 1001
- Legacy gaps (< 1000) are preserved to maintain historical numbering
- Enables efficient brand code reuse after admin delete operations

## Test plan
- [x] Unit tests for backend Dropbox service (8 tests)
- [x] Unit tests for CLI local mode (5 tests)
- [x] All 100 brand-code related tests pass
- [x] CodeRabbit review - no issues

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-24 09:33:34 +00:00