mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
0e507852aff8be9a79c3c1cfb0eec3a27d3a9607
103
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
9d6aa3df7d |
feat(medley): per-section backing_vocals_db for the karaoke track (v0.278.6) (#1184)
## Summary The Fiverr medley customer said the backing singers in two songs were too loud to sing over in tune, and asked for them as a faint background. - **New option:** a per-section `backing_vocals_db` in the medley spec. For the karaoke track only, the section is rebuilt from the clean instrumental plus the backing-vocal stem at that level, e.g. `-14` for a faint background. The rest of the karaoke track, the clean instrumental and the guide are unchanged. - **Cached mix:** the mix is cached per job/level, written to a `.partial.flac` and renamed, so an interrupted run never leaves a truncated mix to be reused. If the job has no backing stem, it stops with a clear error. - **Docs:** `docs/MEDLEYS.md` spec table. ## Testing - `test_make_medley.py`: parsing, the ffmpeg graph, only the karaoke variant changing, the atomic write and the cache. 20 + chord-chart tests pass. - Real data: the customer's medley with `-14` on Remember When and On the Other Hand. - Only those two sections change; the others differ by ≤ -43 dB, which is crossfade overlap. - Overall section loudness is unchanged. - A local CodeRabbit review found 1 issue (stale or partial cache). It is fixed. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
03ee693f28 |
feat(medley): customer chord charts + static-gain master (v0.278.5) (#1183)
## Summary Two follow-ups on the medley tooling, both from the Fiverr medley customer's feedback. - **Fixes the robotic artifact at the start of the karaoke audio.** The final single-pass `loudnorm` ran in dynamic mode. On a song that opens a cappella (HSLHT sings from 0.4s, band at 3s), the instrumental's first 3s is near-silent separation residue at -70 dB, and `loudnorm` boosted it by about 50 dB into a loud warble. The master is now one static gain to -14 LUFS plus a 4×-oversampled limiter at -1 dB (`master_filter`). After the fix, the first 3s measure -61 dB (were -15 dB). Sync is sample-identical and true peak is -0.9 dB. - **Imports the customer's own chord chart.** `scripts/medley/chord_chart.py` plus a new `make_medley.py chart` step read a chords-over-lyrics chart (.txt or .docx in Courier, capo shapes, `Capo N` / `Key change Capo 1` lines). They align it word-by-word to the reviewed lyrics, so chords come out timed on the words the chart puts them on. Detected chords fill intros, breaks and spoken verses, transposed into the capo shapes. - **Shows the capo in the chord box.** New `ChordChange.capo` field, shown as "capo N" under the chord. - **Docs:** `docs/MEDLEYS.md` gets a new section on importing a customer chart, and the mastering guidance now says never to use single-pass `loudnorm`. `docs/README.md` has an entry. ## Testing - Unit tests: - New: `tests/unit/test_chord_chart.py` (9). - Added: capo tests in `test_guitar_chords.py`, plus master and capo tests in `test_make_medley.py`. - Full `tests/unit` suite: 2100 passed, 65 skipped (run before the last CodeRabbit fix; only the 9 chord-chart tests were re-run after it). - Real data: the customer's 4-song chart against the medley. The lyrics-with-chords output matched the chart line by line, and 4K frames rendered with capo labels. - A local CodeRabbit review found 1 issue (two chord groups after the same word could come out in the wrong order). It is fixed and has a test. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01QV2TQwyC7pwVkGPCk98Bqg --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
7cf5beede8 |
feat: guitar chords in karaoke videos + ChordMini ensemble chord detection (v0.277.0) (#1175)
## Summary Adds optional guitar chords to karaoke videos, first used for a Fiverr medley customer who wants chords on their video. - **Rendering (gen library):** `OutputGenerator.generate_outputs(chords=[ChordChange(start, end, label)])`. - Each chord change during a sung line is drawn above the word it lands on, in a row reserved above every line (the same slot mechanism as translated lyrics). - Chords light up when played, using the same `\k` highlight as lyrics: gold while upcoming (theme `chord_color`), sung-lyric colour once played. Quick changes on one word never overlap. - A top-right "now / next" chord box runs for the whole song, covering intros and instrumental breaks. - With no chords supplied, output is unchanged. A test asserts the ASS is byte-identical. - **Detection (`scripts/medley/detect_chords.py`):** - Ensembles both ChordMini checkpoints (BTC + 2E1D, MIT; about 80% majmin/MIREX, the best open weights we found) over the full mix, gen's clean instrumental, and an htdemucs_6s harmonic mix (guitar + piano + bass + other). - Weighted per-frame vote, then merges short blips, snaps changes to beats and spells chords for the key (Ab/Db/Eb7, F#m, borrowed bIII/bVI/bVII as flats). - **Medley script:** new `chords` step plus a `"chords": true` spec option. Chords are clipped per section and hand over mid-crossfade, then shifted for the countdown. - **Docs:** `docs/MEDLEYS.md` § Guitar chords (setup, method, accuracy caveats, theme keys) and a docs/README.md entry. There's no job or UI option yet. The medley pipeline uses this for now; a product version would need a chords toggle plus detection in a worker. ## Testing - New: `test_guitar_chords.py` (12: chord timeline helpers, word placement measured against the font, layout geometry, highlight timing, now/next box, theme colour, no-chords path byte-identical), `test_detect_chords.py` (7), and 2 medley chord-shifting tests. - `tests/unit`: 2087 passed, 65 skipped. - End to end on the 5 real medley songs: models/inputs agree on 91–99% of major/minor labels, and the output matches the expected I–IV–V progressions. Rendered 1080p and 4K medley videos with 146 chord changes; spot-checked frames (verse, instrumental screen, chorus). - Local CodeRabbit: 1 finding (doc section reference), fixed. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01QV2TQwyC7pwVkGPCk98Bqg --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
85390fd12b |
docs: medley-making guide + spec-driven make_medley script (#1174)
## Summary Documents the multi-song medley process used for the first classic-country medley, and turns it into a reusable script so the next one only needs a spec file. - `docs/MEDLEYS.md`: the full process. One gen job per original song, then identical cuts on every audio variant, and the reviewed lyrics shifted onto the medley timeline and rendered with gen's own `OutputGenerator`. Also covers how to choose cut points (crossfades, pot-out/pot-in, keys/tempos, checking the brief against the recording) and gotchas. - `scripts/medley/make_medley.py`: `fetch` (Firestore + gcloud CLI) → `audio` (per-section LUFS gain, crossfade/gap/pot fades, `loudnorm` to -14 LUFS / -1 dBTP, three variants) → `video` (shifted reviewed lyrics + countdown + themed title/end screens + karaoke and guide MP4s). - `scripts/medley/example-spec.json`: the country medley as a worked example. - Linked from `CLAUDE.md` Quick Reference and `docs/README.md`. Ops tooling only. No backend or frontend change, so no version bump. ## Testing - `tests/unit/test_make_medley.py`: 12 unit tests covering spec validation, the timeline offset maths, lyric-segment shifting (including the same song used twice) and the ffmpeg filtergraph. - End-to-end run against the real prod jobs: `fetch` → `audio` reproduces the hand-built v1 medley **bit-for-bit** (max sample diff 0.0 on both the instrumental and guide WAVs). `video --resolution 360p` renders both MP4s and the LRC. - Local CodeRabbit review: 1 finding (unchecked ffmpeg exit code in `measure_lufs`), fixed. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01QV2TQwyC7pwVkGPCk98Bqg --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
c277bc8ee3 |
feat: local E2E stack (fake workers + test API) + email-code sign-in (v0.275.0) (#1165)
## Summary - **Local stack:** `scripts/local-stack.sh up --frontend` runs the **real** backend on the Firestore + GCS emulators with `FAKE_WORKERS=1`. In-process fake workers move jobs through their real states in seconds, using tiny real media, so web Playwright and mobile Maestro full-journey tests can run offline against the same endpoints. - **Email-code sign-in** (nomad-mobile spec §4.1): `POST /api/users/auth/send-code` + `verify-code`. The code logic is shared with kjbox via `LoginCodeStore`, and sends are throttled per email and per IP. - **Bug fix found by the new suite:** `confirm-duration` moved jobs to `separating_stage1`, from which they can never reach review. It now resumes at `downloading`. No prod job had taken this path yet. ## Changes - `backend/services/fake_workers/`: dispatch seam in `WorkerService.trigger_*`, plus fake search, download, checkout and outbox, scenarios, and per-user config. The real stem upload, transcode, backing-vocals analysis and preview render are reused. Every check is strict (`is True`), so a MagicMock settings object never enables the fakes, and startup **refuses FAKE_WORKERS in production**. - `/api/test/*` router (mounted only in that mode): `status`, `scenario`, `fake-config`, `emails`, `reset`, plus the fake Stripe checkout page. Paying it runs the real webhook handler. - `frontend/e2e/local-stack/` (9 specs, ~50s): - sign-up → make → review → instrumental → download - buy and cancel credits - duration confirm, failed, complete and review states - code sign-in and lockout - New required CI job `local-stack-e2e`. - Docs: TESTING.md § Local stack, DEVELOPMENT.md, API.md, README, LESSONS-LEARNED. - `MAX_SIGNUPS_PER_IP` can now be set by env. Only the local stack raises it; production keeps the default of 2. ## Testing - [x] Backend unit: 6175 passed - [x] Emulator: 152 passed - [x] Frontend Jest: 1706 passed - [x] Local-stack Playwright: 9/9 passed (CI mode) ## Review - [x] CodeRabbit CLI review completed locally. 1 minor finding declined: it suggested swallowing infrastructure errors in send-code as fake "sent" responses, which would hide real outages from users. @coderabbitai ignore --- 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01AGuxtgoscF6RJGtpjHPg3W --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
39dcf2598e |
fix: PyPI prune snippet + force marker only in PR title (v0.268.10) (#1146)
## Summary Two follow-ups to #1145, both found while doing today's prune. 1. **Prune console snippet silently did nothing.** PyPI's delete-version form now requires five `acknowledge_*` checkboxes. A submit without them gets a **200 re-render**, which the old snippet counted as "deleted". The snippet now reads each release's real form (CSRF plus checkboxes), submits it, and counts a version deleted only when its manage page 404s. This is the exact approach used for today's prune (88 releases, 10.73 → ~4.1 GB). 2. **The force marker matched the commit body.** A squash commit's body is the PR description. #1145's description just *mentioned* the marker, which forced a publish on merge. That publish ran before the prune, so it 400'd and uploaded nothing. The marker now only counts in the commit subject (the PR title). Docs: the runbook covers the new form, the per-version JSON cache lag, and a prune log. ## Testing - Unit tests: 30 pass across both scripts, with new tests for checkbox/404 handling and for a marker in the body not forcing a publish - `node --check` on the generated snippet - Replayed #1145's real merge message through the gate: no publish - Local CodeRabbit review: 0 findings - Full `make test` not re-run (no app code touched); PR CI covers the package suites @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01MSW9yjwGnzz1S8MgCGm6t5 --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
5005d3cb80 |
ci: publish to PyPI only when CLI code changed (v0.268.9) (#1145)
## Summary PyPI publishing failed on every deploy since v0.268.4. karaoke-gen hit PyPI's 10 GB project cap for the third time (June, Sept 29, Oct 12). Each wheel is ~62 MiB and we published on every merge (~100/month), so each prune only bought a couple of weeks. PyPI is only the public `pip install` channel for the CLIs. Cloud Run builds from source, and the encoding worker pulls the wheel from GCS. So we now publish only when something CLI users get has changed. ## Changes - `scripts/pypi_publish_gate.py` (stdlib): diffs `scripts/pypi_publish_paths.txt` (`karaoke_gen/` plus the `backend/` modules it imports) and `pyproject.toml` (ignoring the package `version =` line only) against **the latest version on PyPI**. A skipped or failed publish therefore never loses a CLI change. Fails open. - `ci.yml` `package-publish`: new "Decide PyPI publish" step gates `poetry publish` (now `--skip-existing`). **Tags and GitHub releases still happen on every version**, because the encoding-worker gate diffs against tags. Override with `[publish-pypi]` in the PR title/commit, or the new workflow_dispatch input "Publish to PyPI". - Not gated on purpose: `frontend/**` (bundled review UI) and `backend/config.py` (cloud settings). A test keeps the backend import list in sync, with `config.py` as the one documented exemption. - Docs: PYPI-STORAGE-PRUNE.md, TROUBLESHOOTING.md, prune script and reminder workflow comments. ## Impact Replayed over v0.200 to v0.268: **24 of 136** releases would have published (~0.7 GB/month instead of ~4 GB/month). Run against current main, it decides "no CLI-affecting changes since v0.268.3", which is correct for v0.268.4 to v0.268.8. ## Testing - 15 new unit tests in `tests/unit/test_pypi_publish_gate.py` - `make test` all green - actionlint clean on the new step - Local CodeRabbit review: 1 finding (doc recovery path), fixed @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01MSW9yjwGnzz1S8MgCGm6t5 --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0329d47e38 |
fix: YouTube description backfill reverted translated-lyrics notices (v0.268.9) (#1144)
## Summary
The daily YouTube description backfill worker was undoing the
translated-lyrics notice on videos with translated lyrics.
**Root cause:** `classify()` parsed `Artist - Title (With Translation
into Spanish) (Karaoke)` with the translation label left inside the song
title, and `render_for()` never passed a `translation_language_name`.
Every translated upload therefore looked "pending". The worker re-checks
the whole channel every day, even after a cycle completes, so it rewrote
each new translated video. The rewrite dropped the `This karaoke video
includes translated lyrics (…)` notice, put the label into `{title}`,
and added it to the tags as well.
## Changes
- `backend/services/youtube_backfill.py`: `classify()` strips the `(With
Translation into X)` label from the parsed song title and records
`translation_language_name`. `render_for()` passes it to
`render_youtube_description`, which is the same thing the upload
pipeline does. Tags now use the clean song title.
- On its next run the worker restores the notice on the translated
videos it already reverted, because their titles still carry the label.
- README: added a safety note that any per-video description variant has
to be recoverable from the live title.
- Version 0.268.9
## Testing
- 5 new tests in `backend/tests/test_youtube_backfill.py`: parsing,
untranslated case, matching upload left alone, reverted description
restored, title preserved + clean tags.
- `make test-backend-unit`: 5961 passed.
- Local CodeRabbit review: no findings.
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_013RzGSdPUXLn1nK5eMEZZTW
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
3ecea6f2e7 |
feat(storage): enable real purge with deletion audit log and orphan cleanup mode (v0.266.0) (#1127)
## Summary Andrew approved this follow-up to #1125 and #1126. It turns on real deletion, adds an audit log of every deleted object, and adds an orphan-folder cleanup mode. ### Real deletion on `STORAGE_RETENTION_DRY_RUN=false` and `STORAGE_RETENTION_MAX_JOBS_PER_RUN=50` are added to the backend deploy's `--set-env-vars`. The API service is the only thing that reads them: it passes `--dry-run` to `storage-retention-job`. That keeps the setting in one place, so a later deploy can't silently revert it. ### Deletion audit log - `storage-retention/deletion-logs/<ts>-<mode>.jsonl` has one line per deleted or failed object: timestamp, run_id, mode, job_id, path, size_bytes, generation, category and result. - The log is rewritten after every job or folder, so a crash still leaves a complete record. Each run also writes `<ts>-<mode>.summary.json`. - Deletes are pinned to the logged generation (`if_generation_match`). - `assert_deletable` refuses any path outside `jobs/{id}/`, so `storage-retention/` is never touched. No lifecycle rule matches live objects there. - The Firestore manifest on each job doc now also records generations and the log path. ### Orphan cleanup - `POST /api/internal/storage-retention/orphans` runs on the same Cloud Run job with `--mode orphans`. - It targets `jobs/{id}/` folders with no doc in `jobs`, `jobs-dev` or `youtube_upload_queue`, re-checked live per folder against a fresh listing. - It skips any folder with an object modified in the last 30 days, keeps `input/`, and deletes the rest. - `encoding_worker_jobs` doesn't count as a record: it's the encoder's expiring result cache, which deleted jobs leave behind. - It is a dry-run unless called with `dry_run=false&confirm=delete-orphan-job-folders`, and the scheduler never runs it. ### Docs - `docs/STORAGE-RETENTION.md`: run modes, log locations, jq review commands and restore one-liners (noncurrent `cp #gen` and soft-delete `restore`). - `scripts/storage_retention_report.py`: list runs, summarise a run, or find which run deleted a path or job. ## Testing New tests cover the log contents, generation-pinned deletes, per-job flushing across a crash, the path guard, orphan planning (input kept, 30-day skip, live record re-check, batch limit), and the endpoint and worker modes. Full backend suite: 5774 passed. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added admin-triggered orphan-folder cleanup, with dry-run enabled by default, optional folder limits, and explicit confirmation required for deletion. * Added deletion reports and audit logs covering removed objects, reclaimed space, and failures, plus a read-only tool to review runs and search records. * **Bug Fixes** * Added safeguards to prevent deleting changed objects or files outside their intended job folders. * **Documentation** * Expanded storage-retention guidance with recovery details, review commands, and cleanup instructions. * **Chores** * Set the production cleanup limit to 50 jobs per run and updated the package version. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
4e43449559 |
fix(backend): stop API event-loop freezes + identify users in degradation telemetry (v0.264.0) (#1120)
## Summary The "Reconnecting" pill and the "servers unavailable" banner (164 / 49 events, 2026-09-19 → 10-02) were mostly caused by the single API instance's **event loop freezing for 20–45s**. Every request on the instance, `/api/health` included, completed at the same moment. The full evidence and plan are in `docs/archive/2026-10-03-backend-loop-freezes-telemetry-plan.md`. ### What froze the loop, and the fix for each - **Screens worker:** it ran as a post-response `async def` BackgroundTask doing 4K PIL renders, GCS transfers and ffmpeg transcodes on the loop. Under `--cpu-throttling` it was also CPU-starved. **Fix:** it now runs inline in its Cloud Tasks request (CPU stays allocated, failures still return 200 so Cloud Tasks doesn't double-run it), with the heavy steps in `asyncio.to_thread`. Also offloaded: style load, timing signals, pre-apply. - **Matplotlib font cache:** it was rebuilt on every new instance (17/28 episodes). **Fix:** it's pre-built in `Dockerfile.base` (`MPLCONFIGDIR=/opt/mplconfig`), and pyplot is imported lazily. - **Hourly stale-review cron:** an `async def` with zero awaits doing ≤1500 reads and SMTP sends. **Fix:** it runs in a thread. - **166 `async def` routes that never awaited** (sync Firestore/GCS/email/Gemini on the loop, e.g. magic-link verify). **Fix:** they're now plain `def`, so they run in FastAPI's threadpool. The new AST guard `test_no_blocking_async_routes.py` fails CI on new ones. It also flags sync routes that call `asyncio.create_task`/`get_running_loop`; that caught `confirm_duration` and `trigger_youtube_queue_processing`, which stay async. ### Safety net - `backend/services/loop_watchdog.py`: a heartbeat plus a daemon thread. Any stall over 1s logs `EVENT_LOOP_STALL` with the loop thread's stack and a `culprit=file:line`. Stalls of 5s or more are written to `client_events` as `server_loop_stall`. - Pulumi: the log metric `backend/event_loop_stalls` plus an alert on ≥2 stalls of ≥10s in 30 min. **This still needs a local `pulumi up`.** Pulumi is never applied by CI, and the admin gcloud login has expired. ### Telemetry gaps fixed - **Identity:** `user_email` used to be hard-coded to null. The frontend now sends its session token, and the server resolves `user_email`, `is_admin`, `is_internal` and `is_test`. A client-sent email is ignored. - **New fields:** `device_fingerprint` (the existing FingerprintJS visitorId), `tab_id`, `episode_id`, and `tenant`. - **New event:** `banner_recovered`, carrying `duration_ms` and `peak_status`. - **Report:** `scripts/client_events_report.py` summarises the data, excluding admin/internal/test accounts by default. ## Testing - `make test` passes: backend 5529, plus the emulator suite, karaoke_gen unit tests, and frontend Jest 1623. - New tests: - Watchdog: a deterministic state machine plus detection of a real blocking call. - Event-loop responsiveness: a ticker keeps ticking while the heavy steps run. - Screens inline and failure semantics. - `client_events` identity and fields. - The AST guard. - The report summary. - Frontend reporter and banner episode tests. - Local CodeRabbit CLI review: 0 findings. 🤖 Generated with [Claude Code](https://claude.com/claude-code) @coderabbitai ignore --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
8490ab75eb |
chore(gemini): move every Gemini call from Vertex AI to the Gemini Developer API (#1108)
## Summary Moves **every** Gemini call in gen off **Vertex AI** (billed to the `nomadkaraoke` GCP project, ~$85/mo) onto the **Gemini Developer API** using the `gemini-api-key` Secret Manager secret (an AI Studio key billed outside this project). Once deployed and verified, `aiplatform.googleapis.com` / `generativelanguage.googleapis.com` get disabled in nomadkaraoke. ### Key injection - `backend/services/gemini_client.get_genai_client(timeout_ms)` — key from `GEMINI_API_KEY`, else Secret Manager `gemini-api-key` via `settings.get_secret` (runtime SA already has project-level `secretAccessor`). No `project`/`location`. - Service: CI `--set-secrets ... GEMINI_API_KEY=gemini-api-key:latest`. - Jobs: Pulumi secret env on `lyrics-transcription-job`, `video-encoding-job` (renders + translations), `nomad-error-monitor` — **already applied** (targeted `pulumi up`, only that env added). Other jobs fall back to Secret Manager. - Local scripts (`frontend/scripts/translate.py`, `scripts/measure_autocorrect_cost.py`, `scripts/test_agentic_correction.py`): env, else `gcloud secrets versions access` (in-process only). - **Rotation:** add a new secret version; new instances / job executions pick it up. No code change. ### Degradation (quota / billing / invalid key; per-minute 429s stay retryable) | Caller | Behaviour | |---|---| | auto-correct | non-retryable 503 → compare mode drops the Gemini leg, keeps Opus | | match-judge / free-text / parse-titles / tenant bulk | existing fallbacks (no suggestion / blanks / regex) | | credit eval | `pending_review` (fail-closed, unchanged) | | lyrics translation | no retry; render proceeds without translation, status recorded | | custom lyrics | clean 503 "temporarily unavailable…" (+ 5-min timeout so it can't hang) | | error-monitor LLM | no retry; alerts sent without AI summary | | translate.py | exit 2 "Gemini quota/credit exhausted — top up AI Studio or rotate gemini-api-key" | `note_gemini_failure()` also sends one Discord ops alert per 6h (Firestore-throttled across instances via new `ops_alerts.send_throttled_ops_alert`). ### Infra - Pulumi: removes `roles/aiplatform.user` from `karaoke-backend` — **will be applied after this deploys and Vertex traffic is confirmed at zero**. - Agentic correction default model `gemini/gemini-3.8-flash` (`vertexai/` prefix still accepted, routed to the Developer API). - Version 0.261.1 → 0.262.0. Docs: CLAUDE.md, LESSONS-LEARNED, README, ARCHITECTURE, API, etc. ## Testing - New: `backend/tests/services/test_backend_gemini_client.py` (classifier, key resolution, no-Vertex client kwargs, alert throttle, per-caller degradation), `tests/unit/lyrics_transcriber/test_agentic_gemini_provider.py`, `frontend/scripts/test_gemini_client.py`, lyrics-translation quota test. - `pytest backend/tests --ignore=backend/tests/emulator` — all pass; `pytest tests/unit` — 2021 passed. - Verified `gemini-3.8-flash` is listed on the Developer API and answered a 5-token test call with the key. - `translate.py --dry-run` on `frontend/messages` and `backend/translations` OK. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Updates** * AI-powered lyric translation, custom lyrics, auto-correction, title parsing, match judging, and related features now use the Gemini Developer API. * AI features use the configured Gemini API key. Quota, billing, or key issues may show an unavailable message or use an existing fallback instead of repeated attempts. * **Documentation** * Updated product, API, setup, and recovery guidance to reflect Gemini API key configuration and usage. * Updated guidance for handling AI service availability and quota issues. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
f8d0febcf5 |
Translated lyrics beneath the karaoke lyrics for language learners (v0.260.0) (#1101)
## Summary
New job-submission option: **Add translated lyrics** → pick a language
(33 UI locales) → the karaoke videos show an AI translation of each
lyric line in smaller text beneath the synced lyrics, so language
learners can sing songs in another language while learning what the
words mean.
- **Job creation**: toggle + language picker (default = UI language) on
the Customize step, with a **live preview** of a real rendered frame
(`GET /api/themes/translation-preview?language=xx`; pre-generated sample
translations, no LLM call at preview time). Any job, public or private,
free for now. `translation_language` accepted by every creation route;
admin can edit it.
- **Post-review translation**: right after the job enters
RENDERING_VIDEO, the render worker translates the *final reviewed*
lyrics line by line with Gemini
(`backend/services/lyrics_translation.py`, `LYRICS_TRANSLATION_MODEL`,
GCS-cached so theme/admin re-renders are free) into
`jobs/{id}/lyrics/translations.json`. Not editable. A failure never
blocks the render (renders without translations, recorded in
`state_data.lyrics_translation`); a song already in the target language
gets no row.
- **Rendering** (landscape + portrait, GCE worker + local fallback):
`LyricsSegment.translation`; when present, slots get taller and screens
show 3 lines; a `Karaoke.Translation` style (unsung colour, ~½ size)
draws one static event beneath each line, sanitized, wrapped/scaled to
fit. Jobs without translations render identically.
- **RTL + LTR together**: the translation is its own ASS event with auto
paragraph direction, so e.g. a Hebrew song keeps its right-to-left
highlight with an English translation beneath (and vice versa). Long
lines split by the resizer share their translation across the pieces
(grapheme-safe for Thai/CJK).
Design: `docs/archive/2026-10-01-translated-lyrics-plan.md`.
## Testing
- `make test` all green (root 2014, backend 5416, emulator 150, frontend
1583).
- New: `tests/unit/lyrics_transcriber/output/test_translated_lyrics.py`
(layout/positions/style/sanitizing/fit/mixed RTL-LTR/Thai/portrait),
`test_translated_lyrics_render.py` (libass pixel tests incl.
Hebrew/Arabic lyrics with English/Arabic translations and CJK),
`backend/tests/test_lyrics_translation.py`,
`backend/tests/test_render_video_worker_translations.py`, GCE endpoint
test, frontend CustomizeStep/GuidedJobFlow/api tests, prod E2E
`e2e/production/translated-lyrics.spec.ts`.
- `scripts/run-render-tests-linux.sh`: 135 passed on Debian ffmpeg AND
prod's static ffmpeg (no skips).
- Real-data check: job 5710831e (Hebrew) translated to English by Gemini
and rendered with the nomad theme — RTL highlight intact, English
beneath, split line shares translation.
Local CodeRabbit CLI unavailable in this workspace (no seat), so leaving
this PR for the CodeRabbit bot; an independent local review was done and
its findings fixed in the second commit.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added an opt-in translated-lyrics option to job setup, with language
selection and a preview.
* Translations are generated after lyric review and appear beneath
lyrics in landscape and portrait videos, with text layout for multiple
languages and writing directions.
* Translation settings can be changed or cleared by admins.
* **Bug Fixes**
* If translation fails or is unavailable, video rendering continues
without translated lyrics.
* **Documentation**
* Updated API and product documentation with translated-lyrics setup and
behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
||
|
|
777daaf25f |
Missing-lyrics gate: block auto-approval when singing has no lyrics and the reference has lines there (v0.258.0) (#1097)
## Summary Phase 2a of the missing-lyrics work (phase 1: #1095, shadow detector + backfill). **Audit** (262 completed jobs, 60 days, server-side `scripts/audit_vocal_gaps.py`): - Longest unlyricked vocal run per job: p50 0.7s, p90 5.1s, p99 22.4s. 47 jobs (18%) have a run of 3s or more, far too many to gate on audio alone. - Cross-checked against **synced** reference lyrics (LRCLIB line timestamps): of 57 suspect gaps that have a synced reference, only 15 contain a reference line. Audio-only is ~25% precise. The rest are vocal chops, samples, ad-libs and synth bleed, mostly in drum & bass tracks. - **Evidence-based rule:** a gap is reliable when the reference lyrics place lines in it. **Changes** - `vocal_gaps` 0.2.0: records the synced reference lines inside each gap. `evidenced` = suspect vocal run AND reference lines, either synced or between the neighbouring anchors. - Scorer 0.5.0: new never-auto `missing-lyrics-gate` with a readable reason, e.g. *"possible missing lyrics: ~10s of singing at 0:20-0:32 has no transcribed words, and the reference lyrics have 3 line(s) there"*. Audio-only gaps don't gate. - The executor analyzes vocal gaps *before* scoring (off the event loop, fingerprinted cache) and passes them to both scoring passes. It fails open on pending audio or analysis errors. - `GET /api/review/{id}/correction-data` now includes `vocal_gaps`, for the upcoming review-UI marker (phase 2b). On job 5710831e, the synced LRCLIB lines inside the gap are exactly the three dropped lines. The genuine instrumentals have none. ## Testing - `test_vocal_gaps.py`: synced lines with edge tolerance; section headers ignored; evidence requires both singing and reference (instrumental with misplaced reference lines doesn't gate; singing without reference doesn't gate). - `test_scorer.py`: reasons only for evidenced gaps; the gate overrides confident lyrics; non-evidenced gaps don't gate. - `test_executor.py`: evidenced gaps block auto-approval with the reason and skip the timing check; cached evidenced gaps still gate. - `test_combined_review_flow.py`: `vocal_gaps` is included in correction-data. - Full backend suite: 5211 passed. Versioned 0.258.0 to follow #1096 (0.257.0). No local CodeRabbit review (the CLI has no seat for this org), so this is left open for the CodeRabbit bot. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
e2f7dbd13c |
Detect sung stretches with no transcribed lyrics (shadow) + backfill endpoint (v0.256.0) (#1095)
## Summary
Transcription can silently **drop whole sung lines**. On job 5710831e
(Hebrew), the words stop at 20.66s and resume at 32.40s while the lead
vocal is ~96% active, and both reference sources have three pre-chorus
lines there. `SectionDetector` (gap ≥ 10s, no audio check) then rendered
"♪ INSTRUMENTAL ♪" over the singing. Nothing caught it:
- the scorer's anchor coverage measures how much of the *transcribed*
text matches the reference, so omissions don't lower it (98.8% here);
- `timing_check` G3 (`max_unclaimed_run_s`) does fire on this job but is
shadow-only and only computed for AUTO-bound jobs;
- the review UI shows nothing at the gap.
This PR is **phase 1: a shadow signal plus server-side backfill for
calibration**. It changes no verdicts or rendering.
- `backend/services/auto_approval/vocal_gaps.py`: for every
transcription gap ≥ 3s, records the lead-vocal active fraction, the
longest breath-bridged vocal run (with a 1.5s allowance for a held note
carried over from the previous word), and the reference lines between
the neighbouring words' reference positions. Section headers are dropped
and ±1-word anchor drift at line boundaries is snapped. `suspect` = run
≥ 3s.
- The executor analyzes every job once stems exist (any verdict), off
the event loop, and stores `state_data.vocal_gaps` keyed by an
`input_key` fingerprint so resets/edits recompute. A summary goes in
`processing_metadata.auto_approval.vocal_gaps`.
- `POST /api/internal/jobs/{id}/vocal-gaps[?dry_run=true]` (admin) plus
`scripts/audit_vocal_gaps.py` for backfill/calibration. Cloud Run does
the stem work.
**On the real job:** one suspect gap, 20.66–32.40s (10.26s run after the
allowance), whose reference lines are exactly the three missing Genius
lines. The three genuine instrumentals have ≤ 0.16s runs.
## Testing
- `test_vocal_gaps.py` (25 tests, synthesized audio + synthetic
correction data):
- dropped lines vs instrumental vs ad-lib vs held note
- a short dropped line diluted by a long gap
- breath bridging
- silent stem = error
- malformed reference data
- reference snapping (off-by-one, single-word line, headers, reversed
chorus mapping)
- Executor tests: stored without changing the outcome; analyzed for
review-bound jobs; cache hit only for the same `input_key`, stale
recompute; pending audio / no stem / error / exception all non-fatal;
backfill uses the reviewed lyrics + original references; dry run.
- Internal endpoint tests (store / dry_run / admin).
- Full backend suite: 5199 passed.
- A local code review's findings were addressed in the second commit:
event loop, stale cache, silent stem, held notes, reference robustness,
audit ordering.
## Next (after deploy)
Run the audit → hand-check suspects → choose thresholds → phase 2:
scorer REVIEW gate with an explicit reason, no fake INSTRUMENTAL over
vocals, review-UI marker with one-click insert of the reference lines.
Also noted: anchor reference ids look shifted by one word at `[Section]`
header boundaries. Separate issue, documented in the plan doc.
No local CodeRabbit review (the CLI has no seat for this org), so this
is left open for the CodeRabbit bot.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added vocal-gap analysis to identify stretches of missing lyrics that
overlap with lead vocals. Results are saved with job data and can be
requested through an admin-only internal API.
* Analysis also runs automatically during auto-approval when vocal stems
are available. It does not change job status or approval decisions.
* **Documentation**
* Documented the internal API, its dry-run option, and the analysis
result statuses.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
||
|
|
354e43bab2 |
Render Hebrew/Arabic karaoke right-to-left; fix title-card tofu (v0.255.0) (#1094)
## Summary
A Hebrew KaraokeHunt job (5710831e, Omer Adam – "שני משוגעים") shipped
with its lyrics **laid out and highlighted left-to-right** and a **title
card of "?" boxes**. Diagnosed by measuring rendered pixels (not by
eye), reproduced locally and with prod's static ffmpeg 7.0.2 in Docker.
**Root causes**
- ASS style `Encoding=0` makes libass force LTR paragraphs; the `{\kf}`
before each word splits the line into runs, so each word is shaped
correctly but the words go left→right.
- `\kf` always wipes left→right within a word
([libass#406](https://github.com/libass/libass/issues/406)).
- PIL (title/end cards) has no per-glyph font fallback; the theme font
(Avenir Next) has no Hebrew glyphs.
**Fixes**
- `build_karaoke_styles` forces `Encoding=-1` (libass auto paragraph
direction; LTR unchanged).
- RTL lines get `{\frz(180+angle)\frx180\fry180}` — a visual no-op that
makes libass ≥0.15 sweep fills right→left without splitting words
(Arabic still joins). Theme angle preserved; re-emitted after duet
`{\r}` resets.
- Lead-in indicator mirrored for RTL lines.
- Text measurement mirrors libass: per-glyph fontconfig fallback, size
scaled from each font's metrics (old fixed 0.70 was 20%+ off for
fallback fonts).
- New `karaoke_gen/utils/font_fallback.py` (fontTools cmap +
fc-match/fc-list) used by title/end cards, portrait headers and ASS
measurement.
- Found by the new tests: inter-word gaps ≤0.1s were dropped, so the
highlight ran up to ~0.2s early by line end (all languages). Karaoke
tags now derive from one absolute centisecond timeline.
- Fixed pre-existing `test_render_video_worker_gce` failure on main
(MagicMock generation → `_g<mock>` id).
## Testing
- **Pixel-level render tests** (`test_ass_render_highlight.py` +
`ass_render_harness.py`): real `SubtitlesGenerator` output rendered with
ffmpeg/libass in a pure-colour palette, asserting on frames for English,
Hebrew, Arabic, Chinese, Japanese, Korean + an RTL duet line: highlight
starts at the reading-start edge, sung pixels always precede unsung in
reading order (word order + in-word fill direction), monotonic and in
time with the words, lead-in on the correct side, no missing glyphs, RTL
tags don't move the line (incl. rotated themes).
- 32 fail on pre-fix main; mutation checks ("Encoding only", "no re-emit
after `\r`", "ignore theme angle") each fail exactly the expected cases.
- 123/123 pass on Linux against **both** Debian ffmpeg (CI-equivalent)
and **prod's johnvansickle static ffmpeg** via
`scripts/run-render-tests-linux.sh`.
- CI now installs `fonts-noto-core fonts-noto-cjk`; under `CI=true`
these tests fail rather than skip on missing preconditions. (macOS
libass uses CoreText → font-sensitive checks skip locally.)
- Title-card tests (`test_font_fallback_render.py`): no `.notdef` tofu
for Hebrew/Arabic/CJK titles; Hebrew title reads right-to-left.
- Unit tests: `test_rtl_ass_output.py`, `test_font_fallback.py`.
- `make test`: unit 1972 passed; emulator 150 passed; frontend 1491
passed.
- Full local re-render of job 5710831e reviewed by Andrew: RTL highlight
smooth and correct, title card correct.
## Follow-ups (not in this PR)
- CDG output still renders RTL left-to-right / tofu (separate renderer).
- Separate investigation: transcription dropped 3 sung lines in this
job, which the section detector then labelled INSTRUMENTAL.
No local CodeRabbit review (CLI has no seat for this org), so leaving
this open for the CodeRabbit bot. A local code review ran; its findings
are addressed in the second commit.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Karaoke lyrics now render more reliably for Hebrew, Arabic, Chinese,
Japanese, and Korean, with improved reading-direction highlighting,
timing, and lead-in placement.
* Titles and portrait text can use fallback fonts when the selected font
lacks the necessary characters.
* **Bug Fixes**
* Improved lyric centering, word highlighting, and glyph rendering
across supported scripts.
* **Documentation**
* Added guidance on lyric rendering and multilingual test coverage.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
||
|
|
badd845a1d |
perf(jobs): move video-encoding-job (video + render) to us-east4 (v0.249.1) (#1083)
## Summary Follow-up to #1077. `video-encoding-job` runs both the post-review render (`render_video_worker`) and the final video (`video_worker`). In us-central1 each run waited **2–5 min** for Cloud Run Jobs to start it, so every customer job paid that twice. This moves it to us-east4 alongside the download, lyrics and bulk-search jobs. ## Cost (measured, not estimated) - Over the last 30 days the job read **156 GB** from Google services, mostly the us-central1 bucket, across 494 runs (Cloud Monitoring `container/network/received_bytes_count`, kind=google). That's about 0.32 GB per run. - Pricing from the Billing Catalog: GCS inter-region within North America is **free for the first 100 GB/month, then $0.02/GB**. Cloud Run writes to Google APIs are **$0**. - Internet egress (224 GB/month, Dropbox etc.) and Cloud Run CPU cost the same in either region. - Existing cross-region reads are about 24 GB/month (GPU job and lyrics), which the billing export shows at about $0. - **Net: about $1.60/month**, or about $3/month worst case. Time saved: roughly 4–8 min per customer job. ## Changes - **backend**: `trigger_video_worker` (via `_trigger_cloud_run_job`) and `trigger_render_video_worker` now use `cpu_jobs_region`. `GCP_REGION` stays us-central1; encoding VM and zone logic are unchanged. - **infra (Pulumi, already applied)**: new us-east4 `video-encoding-job`, with the legacy us-central1 copy kept for rollback and its deletion protection turned off. A preview after applying showed 290 unchanged. - **ci**: the video job image update now targets us-east4, using the us-east4 CPU image copy. - **Also updated**: rollback.sh, tests (video and render routing), TROUBLESHOOTING/ARCHITECTURE/LESSONS-LEARNED. ## Testing - `make test` passes: backend tests plus 1443 frontend tests. - Prod smoke test of the new us-east4 job: it imports `video_worker` and `render_video_worker` and calls the encoding VM's `/health` on its public IP (port 8080 is open to 0.0.0.0/0 with API-key auth). **Started in 25 s, exit 0.** 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Updates** - Video encoding and render jobs now run in the `us-east4` region, alongside other CPU jobs. - Video encoding continues to access storage and encoding resources in `us-central1`; cross-region transfer may incur costs. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
cb5ce7f162 |
chore(pypi): prune keep-window default 60→30 days (#1082)
Ran the PyPI prune runbook today (2026-09-29): publishing was blocked (`Project size too large`). The 60-day policy would only have freed 1.08 GB (→9.64 GB), refilling within ~a week, so pruned with `--keep-days 30` instead (69 releases deleted, 0.188.5–0.207.0; newest release of each older month retained) → ~6.8 GB. Re-ran the failed publish; 0.249.0 is live on PyPI. This PR makes 30 days the default in the planner script, the monthly reminder workflow, and the runbook. Unit tests pass (13). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
3e855ff65d |
perf(jobs): run download/lyrics/bulk-search Cloud Run Jobs in us-east4 — cut 2-5 min job start latency (v0.247.1) (#1077)
## Summary Every gen job waited **2–5 min** before `audio-download-job` (and then `lyrics-transcription-job`) even started. That blocked kjbox's goal of singing within about 4 minutes of making a song. **Root cause: Cloud Run Jobs in us-central1 are slow to start on Google's side.** Our image, CPU and code aren't the cause. - The `Started` condition fires about 10 s before the first Python log, so our process startup is fine. - decide's 203 MB image on 4 CPU is just as slow in us-central1. - A throwaway job running Google's tiny sample image, 4 runs per region: **us-central1 4m–5m07s**, **us-east4 8–19 s**, **us-east1 8–25 s**. All four us-central1 runs started in the same second, as if released from a queue. Quotas are far from their limits. ## Changes - **backend**: new `CPU_JOBS_REGION` setting (default `us-east4`) for triggering `audio-download-job`, `lyrics-transcription-job` and `bulk-search-job`. `GCP_REGION` is unchanged. `video-encoding-job` stays in us-central1, because it pulls multi-GB outputs from the us-central1 bucket. - **infra (Pulumi, already applied)**: the three jobs are created in us-east4 as new resources. The legacy us-central1 copies are kept as the rollback path, with deletion protection turned off so a follow-up can remove them. The preview after applying showed 289 unchanged, so there's no drift. - **ci**: the CPU app image is also pushed to `us-east4-docker.pkg.dev/nomadkaraoke/karaoke-backend-gpu/karaoke-backend-cpu`, so jobs pull from their own region. The lyrics job image update now targets us-east4. After each deploy, CI starts a no-op `audio-download-job` execution so the one-time ~2.5 min image import doesn't land on the first customer. - `scripts/rollback.sh`: lyrics job moves to the us-east4 image, and jobs whose target tag is missing are skipped instead of aborting the loop. - Docs: TROUBLESHOOTING runbook (measure + rollback), ARCHITECTURE, LESSONS-LEARNED. ## Testing - `make test`: all backend tests pass, plus 1422 frontend tests. - New tests in `backend/tests/test_worker_service_cpu_jobs_region.py`: routing of each moved job, video job stays in `GCP_REGION`, env override for rollback, and the Pulumi/backend region constants must match. - Prod smoke test of the new us-east4 `audio-download-job` with the real image (`import backend.workers.audio_download_worker`): **Started in 7.0 s, 8.6 s, 10.0 s** (first run of a new digest adds a one-time ~2.5 min import, which the CI step covers). - Local review via a reviewer agent: no must-fix findings; the rollback-path suggestions are addressed. ## Rollout One-time image seed (Cloud Build gcrane copy) and `pulumi up` are done. After merge, CI deploys the backend, which starts triggering the us-east4 jobs. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Performance** * Audio downloads, bulk searches, and lyric transcriptions now run in a region with faster job startup times, reducing delays before processing begins. Video encoding continues to run in its existing region. * **Reliability** * Rollbacks now skip jobs when the target image is unavailable and continue with the remaining jobs. Other image lookup errors still stop the rollback. * **Documentation** * Updated architecture and troubleshooting guides with regional job details, startup-delay guidance, and rollback instructions. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
e9465c73d4 |
chore: gemini-3.8-flash everywhere + auto-correct on Opus 5.5 (cost) (#1054)
## Why GCP credits expired 2026-09-19; Vertex Gemini 3.1 Pro spend had grown $9 → $37 → $105/30d. `gemini-3.8-flash` is $0.75/$3.75 per 1M (vs $2/$12 for 3.1 Pro; rises to $1.50/$7.50 on 2027-01-01). Opus 5.5 is $4/$20 (vs $5/$25 for 4.8). ## Changes - All Gemini defaults → `gemini-3.8-flash`: credit eval, custom lyrics, auto-correct (default + compare leg), match judge, parse titles, tenant bulk analyze, agentic AI router, error-monitor LLM analysis (also `us-central1` → `global`, required for Gemini 3.x), frontend `translate.py`. - Auto-correct Claude leg `claude-opus-4-8` → `claude-opus-5-5` (ci.yml env), with `effort: "high"` pinned (5.5 defaults to medium). Pricing table gains both new models. - Docs updated; version 0.241.1. ## Evidence (12 recent prod jobs, pre-review lyrics scored against the human-reviewed final) | Model | Precision | Word-level recall | $/job | |---|---|---|---| | gemini-3.1-pro-preview | 91% | 74% | $0.067 | | gemini-3.8-flash | 98% | 38–41% | $0.020 | | gemini-3.8-flash @ high thinking | 97% | 60% (4/12 timed out, 504) | $0.041 | | claude-opus-4-8 | 94% | 33% | $0.080 | | claude-opus-5-5 (effort high) | 93% | 58% | $0.048 | Flash for the Gemini leg is a deliberate cost/recall trade-off (Andrew chose it); Opus 5.5 more than compensates. Also verified: `gemini-3.8-flash` works on Vertex `global` (rejects `thinking_level=minimal`; nothing here uses it), match-judge latency median 3.3s / max 6.2s (< 12s timeout), Japanese translation smoke test preserves ICU placeholders. Translation cache is keyed on English text + locale, so no re-translation churn. Tests: auto_correct, credit eval, custom lyrics, parse_titles, error_monitor, agentic router suites pass locally. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * AI-powered correction, translation, matching, title parsing, and evaluation now use updated models, including Gemini 3.8 Flash and Claude Opus 5.5. * **Updates** * AI service pricing reflects scheduled rate changes, including a Gemini 3.8 Flash rate change beginning January 1, 2027. * Application version updated to 0.241.1. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
30cf41e7d6 |
feat: concurrent-reliability phases 2-5 — Cloud Run shape, degradation telemetry, banner tuning, load-test guard (#1028)
## Summary Phases 2–5 of the concurrent-load reliability initiative (Phase 1 = #1024, verified live: the 10-tab meltdown went from 9/30 failed requests to 30/30 OK). Verbatim spec: `docs/archive/2026-09-17-concurrent-review-reliability-verbatim-record.md`; plan: `docs/archive/2026-09-18-concurrent-review-reliability-investigation-and-plan.md`. ## P2 — Cloud Run shape - **`--concurrency 80 → 16`** for karaoke-backend: the service is one uvicorn process per 2-vCPU instance, and review loads are heavy; at 80 a burst never triggers scale-out and all lands on the single min-instance. 16 makes bursts fan out while staying well above normal per-instance load. (Deploy flags live in ci.yml — the service is not Pulumi-managed.) - **Local preview ffmpeg fallback bounded** (`LOCAL_PREVIEW_MAX_CONCURRENCY=1`, 900s queue timeout, fast RuntimeError after that): the GCE-outage fallback runs ffmpeg ON the API instance; unbounded, a few concurrent Preview clicks during an encoder outage pegged both vCPUs, stalled the health probe, and showed everyone the orange banner (Andrew's banner-during-preview screenshot). ## P3 — degradation telemetry (the explicit ask) - New **`POST /api/client-events`** (unauthenticated so review-token users report too; 30/min/IP rate limit; bot-filtered; detail capped; URLs query-stripped). Each event → Firestore `client_events` collection **and** a structured `client_event type=…` log line, so "how often are users seeing this?" is reviewable via Firestore aggregation or Cloud Logging correlation. - Frontend reporter `lib/degradation-events.ts` (60s per-type throttle, localhost no-op, never throws), wired into: - `BackendStatusBanner` → `banner_reconnecting` / `banner_unavailable` with stall age, in-flight count, probe state - review lyrics-load failure → `lyrics_load_failed` with message/status - `VocalsAudioDataLoader` → `waveform_slow` (>20s to first strips) and `waveform_failed` (gave up) with poll/retry counts ## P4 — banner tuning - Escalating to the full "We're having trouble reaching our servers" card now requires **two consecutive failed health probes** (`UNAVAILABLE_PROBE_FAILURES=2`). One failed 4s probe against a briefly-pegged instance shows only the calm "Reconnecting…" pill. Single-failure behavior for the pill is unchanged, as is instant clearing on recovery. ## P5 — regression guard - **`scripts/load-test-review.py`** checked in: replays the 10-tab incident (Firestore-ADC review tokens, curl request trios, pass/fail thresholds, exit code). Documented in `docs/TESTING.md`. Validated against prod post-Phase-1: 9/9 requests OK, tabs < 3s. ## Testing - Backend: 6 new tests for client-events (persist/validate/rate-limit/bot/fail-soft/detail-cap) + local-preview bound test — full suite **4517 passed**. - Frontend: new degradation-events suite, two-probe escalation + debug-snapshot tests; one existing api-fetch test updated to the new escalation contract — full suite **1298 passed**. - Live validation of the load-test tool against prod (small 3-tab run): PASS. ## Notes for review - Cloud Run concurrency change means bursts now spin up extra instances (billable during requests only; max-instances unchanged at 20). - Firestore `client_events` has no TTL policy yet (needs a gcloud/console action our read-only ADC can't do); volume is tiny (throttled per client) — can add later if it ever matters. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added automatic reporting for degraded experiences, including reconnecting or unavailable backend status, failed lyrics loads, and slow or failed waveform loads. - Added concurrent review-page load testing to help verify performance under multiple open tabs. - **Improvements** - Backend status now remains in “reconnecting” until multiple health checks fail, reducing premature unavailable messages. - Local preview processing is limited during heavy demand to improve stability. - **Documentation** - Added API documentation for client telemetry and guidance for running concurrent-load regression tests. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
912ca37a01 |
Lyrics review: Waveforms mode + always-show backing vocals in preview (#1010)
## Summary
Two review-UI improvements, both driven from real reviewing pain:
### 1. New "Waveforms" mode in Synced Lyrics
A third view (alongside Simple/Advanced) that renders a **compact inline
copy of the Edit Segment timeline for every segment** — so you can spot
mis-timed words (e.g. an over-long trailing word whose vocal energy died
earlier) at a glance without opening a modal per line.
- Reuses the modal's `TimelineEditor`: resizable/draggable word bars
over each segment's vocal waveform, with the 1s buffer padding,
start/end boundary lines, and greyed read-only neighbour words.
- **Drag** a bar body moves timing; **drag** an edge resizes; both
commit on release (a 4px threshold distinguishes drag from click). A
plain **click** on a word — or the index/✎ — opens the Edit modal.
**Trash** deletes the segment (matches Advanced).
- Bars colour-code with the **exact `HIGHLIGHT_CLASSES` tints** the
Advanced pills use (anchor/gap/corrected/AI/edited) and show the
struck-through **AI-correction ghost text** above corrected bars.
- Tight vertical density (~39px/segment: 20px bars + 14px waveform + 5px
gap) to stay close to Advanced.
- View mode persists to `localStorage` (`lyricsReviewViewMode`,
migrating the old `lyricsReviewAdvancedMode` boolean).
- Advanced + Waveforms now size the **Reference column to its longest
line** so Synced gets the freed width (no more ~25% wasted space).
### 2. Preview modal always shows the backing-vocals waveform
Previously the backing-vocals waveform only appeared *after* switching
away from the recommended "Clean instrumental", making it easy to accept
clean without ever seeing/hearing the backing vocals you'd lose. Now
it's **always shown** whenever a playable backing stem exists, with
selection-aware wording ("kept" vs "removed — click to hear what you'd
be leaving out"). Clicking still auditions the backing track without
changing the final choice.
### Also
- Fixes a pre-existing `Header` `FORMATTING_ERROR` (`autoCorrectedDesc`
rendered without its `count` param on jobs with ≥1 auto-correction).
- `scripts/list-review-jobs.py` — lists in-review jobs with
ready-to-open local review URLs.
## Testing
- New unit tests: `contextWords`, `segmentTiming` (recompute + view-mode
migration), `wordDecorations` (classifier + palette),
`WaveformSegmentRow` (control wiring + untimed fallback).
- Updated `ReviewChangesModal` tests for the always-show behavior +
reworded copy.
- Full frontend Jest suite green (104 suites / 1283 tests).
- i18n: all 33 locales translated; stale key pruned; parity validated.
- Verified live against a real in-review prod job (Playwright):
coloring, ghost text, click-to-open, compact layout, dynamic columns.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added a Waveforms view to Synced Lyrics for inline word-timing
adjustments.
- Added controls to edit segments, play timed segments, delete segments,
and handle untimed lyrics.
- Waveform bars now visually distinguish corrections, edits, gaps, and
related word states.
- Backing-vocal waveforms remain available for review even when the
clean instrumental is selected.
- **Improvements**
- Auto-correction messaging now displays the number of applied
corrections.
- Clarified backing-vocal and clean-instrumental guidance across
supported languages.
- Updated the application version to 0.227.0.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
|
462f246768 |
internal: admin endpoint to re-run auto-approval eval on parked review jobs (#1005)
## Summary
Adds `POST /api/internal/jobs/{job_id}/auto-approval-eval` (admin-only):
re-runs the auto-approval evaluator on a job parked at AWAITING_REVIEW.
The KaraokeHunt outreach batch runs with `made_for_you: true`
(stale-review exemption), which is also an enforcement blocker at
scoring time — this endpoint lets us lift the flag and re-evaluate so
confident jobs full-auto through review.
## Testing
- `test_internal_api.py::TestAutoApprovalEval` — outcome passthrough +
auth required; file green in full-collection run.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Administrators can manually re-run auto-approval evaluation for jobs
awaiting review.
- Jobs remain in review unless the evaluation is fully confident,
providing a fail-safe review process.
- **Bug Fixes**
- Improved batch job processing so an issue updating an optional job
setting no longer stops the remaining workflow.
- Prevented missing cached results from causing failures during
recanonicalization tasks.
- **Tests**
- Added coverage for authorized evaluation requests and unauthorized
access.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
|
6800c14e57 |
admin: make made_for_you editable on jobs (stale-review exemption) + KaraokeHunt outreach tooling (#1004)
## Summary - Add `made_for_you` to `EDITABLE_JOB_FIELDS` (with a boolean guard) so admin batch flows can exempt jobs from the 48h stale-review auto-expiry (`stale_review_processor` skips MFY jobs). Needed for the KaraokeHunt outreach batch: ~90 jobs held at review under a holding account until coordinated emails go out. - KaraokeHunt outreach tooling under `scripts/karaokehunt_outreach/` (phase1b/1c analysis, phase2 executor, batch generator, torrent review) — standalone scripts, no runtime impact. - Session handoff docs in `docs/archive/`. ## Testing - `backend/tests/test_admin_job_update.py`: new `TestUpdateJobMadeForYou` (editable + non-bool rejected) — 20/20 pass. - Full backend suite: 4417 passed (unit) + emulator suite green via `make test-e2e`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Administrators can mark jobs as “made for you,” keeping them available for review indefinitely instead of automatically expiring after 48 hours. - Invalid values for this setting are rejected with a clear validation error. - **Documentation** - Added operational documentation for the KaraokeHunt outreach and migration workflow, including request review, song matching, account credits, and generation stages. - **Chores** - Updated the application version to 0.225.4. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
|
5b5264dc54 |
Incident hardening: deploy/CI/runner resilience (R1/R2/R3/R4/C1/C2) (#987)
Part 3 of 3 of the **Incident hardening (post-NOMAD-1632)** initiative (plan: `docs/archive/2026-09-12-incident-hardening-plan.md`). Closes the deploy/CI/runner gaps: no functional gate, no codified rollback, bad revision serves 100% immediately, fire-and-forget runner stranding, and a flaky smoke. **No backend app code changed → no version bump.** ## ⚠️ Reviewer note — read first **R3/R4 rewrites the live `deploy-backend` step.** It's designed to be *auto-revert-safe* (a failed candidate smoke means we simply never migrate traffic, so the prior revision keeps serving 100% — a bad revision serves **0%**, not an outage). But it changes how **every** backend deploy works and **can't be fully exercised in CI**. This is the one to watch on first merge. If you'd rather de-risk, R3/R4 is the only item here that's safe to split out — the rest (R1/R2/C1/C2) are independent and lower-risk. ## R2 — codified fast rollback (`scripts/rollback.sh` + runbook) Dry-run-able, explicit-target-required rollback for the Cloud Run **service** (traffic-shift to a prior revision — instant, no rebuild) and the three **jobs** (image re-pin, correct per-job region/repo — verified against ci.yml). `--list` shows available revisions/tags; `--dry-run` prints every gcloud command. `docs/TROUBLESHOOTING.md` "Fast rollback" runbook added as the first section. ## R3/R4 — service traffic-canary `deploy-backend` now deploys `--no-traffic --tag candidate`, smokes the tagged candidate URL (health + expected version, with retries), then `update-traffic --to-latest` **only on pass**. Cloud Run Jobs can't traffic-split (noted inline) → they rely on R1 + R2. ## R1 — post-deploy functional canary, version-change-gated New `deploy-verify` job runs the real production happy-path (`happy-path-real-user.spec.ts`, admin-credit variant — reuses e2e-daily's impersonate token refresh, secret names verified identical) after a backend deploy, **only when the backend version actually changed** (new "Detect backend version change" step compares `poetry version` to the currently-served `/api/health/detailed`). Frontend/docs/infra-only merges short-circuit. On failure → Discord alert + loud fail printing the exact `scripts/rollback.sh` invocation. Per the plan's caveat, a fresh happy-path won't reproduce every incident class (e.g. Failure B's stale-cache) — an admin reset+retry variant is a noted follow-up. ## C1 — ephemeral-runner stranding fix `infrastructure/functions/runner_manager/ephemeral.py` now confirms the VM insert **operation** succeeded instead of fire-and-forget: a late stockout/quota/transient error raises → existing zone-fallback / webhook redelivery kicks in instead of silently stranding a queued job for ~15 min. An insert-confirm *timeout* is treated as "still provisioning" (not a failure); guarded for older clients. **+4 tests (46 pass).** ## C2 — de-flake the E2E smoke `networkidle` → `domcontentloaded` + explicit element waits (the marketing page's background requests made `networkidle` time out and redden real deploys). ## Validation (local only — no deploy) - `ci.yml` + `e2e-daily.yml` parse; **actionlint clean** on new code. - `rollback.sh`: `bash -n` + `shellcheck` clean; constants cross-checked against ci.yml. - `test_ephemeral.py`: **46 passed**. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
|
a344783376 |
gen: unify YouTube descriptions + automated back-catalogue rewrite (#978)
## Summary
Unifies the descriptions across all ~1.6k videos on
youtube.com/@nomadkaraoke to one consistent, conversion-focused
template, makes new uploads use it automatically, and adds a
**reusable** automated system to re-apply any future template change
across the whole channel.
Old descriptions were inconsistent: pre-2024 ones linked a dead $10
Fiverr gig; newer ones led with "AI-powered" (bad rep among viewers) and
linked nowhere useful.
## New template
Single source of truth: `Settings.default_youtube_description` rendered
by `backend/services/youtube_description.py` (placeholders `{title}
{artist} {artist_hashtag} {brand_code}`). Drops all "AI" framing; leads
with the craft; CTA to the requests board then the referral link
`nomadkaraoke.com/r/youtube` (50% off, tracked via referrals); Nomad
Discord + decide + KaraokeNerds; fair-use line; preserves the brand
code.
## Live pipeline (new uploads)
Both upload sites (`video_worker_orchestrator.py`,
`youtube_queue_processor.py`) now render via the shared module +
enriched tags — new uploads use the new template automatically.
## Bulk rewrite of the back catalogue
- Shared logic `backend/services/youtube_backfill.py` (classify / target
/ render / update-body / enumeration). **High-confidence-only**
eligibility (title ends in `(Karaoke)`) + curated `skip_ids.txt` /
`include_ids.txt` lists shipped in the package.
- **Manual CLI** `scripts/youtube-descriptions/` (`analyze` / `apply` /
`status`) — read-only report + full old→new diffs before any write.
- **Automated daily drain**
`backend/workers/youtube_description_backfill_worker.py` + `POST
/api/internal/youtube-backfill/run` + Cloud Scheduler (Pulumi). Progress
in Firestore keyed by a **template fingerprint**, so any future template
change auto-re-drains the whole channel with no manual reset.
Quota-coordinated with uploads (reserves headroom via
`YouTubeQuotaService`); Postmark progress + completion emails.
## Verified
- Analyzed all 1,631 videos → **1,614 to rewrite** (1,610 clean + 4
rescued truncated titles), **17 skipped** (lyrics-with-vocals, demos,
tutorials), zero false positives.
- One video (`PmwRXIZoCpg`) updated live and verified on YouTube (new
description + enriched tags; title/category preserved).
## Tests
42 new unit tests (renderer, classifier/targeting/update-body, worker:
full drain, budget cap, quota reserve, dry-run, completion/idempotency,
template-change cycle, quotaExceeded stop). Local review done via the
local code-review fallback (CodeRabbit CLI was rate-limited) — 3
findings fixed, 2 documented trade-offs.
## Deploy notes
- Backend auto-deploys on merge (worker + endpoint + config + data
files).
- **`pulumi up` required** to create the
`youtube-description-backfill-daily` Cloud Scheduler job.
- New config env (all defaulted): `YOUTUBE_BACKFILL_ENABLED`,
`..._DAILY_MAX_UPDATES`, `..._QUOTA_RESERVE`, `..._ENRICH_TAGS`,
`..._REPORT_EMAIL`.
Plan/architecture:
`docs/archive/2026-09-03-youtube-descriptions-plan.md`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- YouTube uploads now use standardized descriptions with artist, title,
brand-code, and hashtag details.
- Tags are generated consistently from karaoke, artist, and title
information.
- Existing channel videos can be analyzed and updated in quota-limited
batches, with dry-run support and progress reporting.
- Added scheduled daily backfills and administrative controls for update
limits, exclusions, and overrides.
- Added safeguards for low-confidence titles and explicitly skipped
videos.
- **Documentation**
- Added guidance for automated and manual YouTube description management
workflows.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
|
37b9e6088b |
ci(pypi): recurring storage-cap prune reminder (fix failing PyPI publish) (#960)
## Problem The `Deploy - Publish to PyPI` step on `main` has been failing: ``` HTTP Error 400: Project size too large. Limit for project 'karaoke-gen' total size is 10 GB. ``` This is a recurrence of the known 10 GB PyPI project cap. The project is at **10.71 GB / 479 releases**; each release is a ~62 MiB universal wheel (bundled 33-locale frontend) auto-published on every merge (~3 GB/month), so it refills the cap every few months. **Only the public `pip install` publish step fails** — Cloud Run and the GCE encoding worker (pulls the wheel from GCS) are unaffected. The prior fix (v0.187.5) was wheel-only publishing + a one-time manual deletion, which only bought ~2 months. This makes the cleanup **recurring and durable**. ## Why not just automate deletion in CI? PyPI has **no deletion API** (upload tokens / Trusted Publishing are upload-only, and 2FA is mandatory). Deletion is only possible from a logged-in browser session, which can't be safely/durably stored in CI. So we automate the *decision + reminder* and keep the *deletion* a ~1-minute human step. ## What this adds - **`scripts/prune_pypi_releases.py`** (stdlib-only) — reads the public JSON API and computes a retention plan: **keep last 60 days + newest release per older calendar month + latest overall**. Renders a table / JSON / Markdown report and a **paste-into-browser-console** deletion snippet. Never deletes anything itself. Today's plan: keep 82, delete 397, **10.71 GB → 4.17 GB**. - **`.github/workflows/pypi-prune-reminder.yml`** — monthly cron (+ manual dispatch) that runs the planner and, once the project crosses a 7 GB watermark, opens/updates a single tracking issue with the plan + snippet. No secrets. - **`docs/PYPI-STORAGE-PRUNE.md`** runbook + a `TROUBLESHOOTING.md` entry. - **`tests/unit/test_prune_pypi_releases.py`** — 13 tests covering the policy (recency window, per-month keeper, latest-overall, same-day timestamp ranking, empty/undated releases, sizes, rendering). ## Testing - `pytest tests/unit/test_prune_pypi_releases.py` → 13 passed. - Ran the planner against live PyPI data; generated JS validated with `node --check`. - `black` clean; workflow YAML validated; `should_nag` gate verified. ## Notes - **No version bump** — tooling/CI/docs only, nothing in the wheel changes. Merging therefore **skips** the publish step (tag `v0.209.1` already exists), so this PR does not itself add another fat wheel while we're over the cap. - **To unblock publishing now:** after merge, run the prune (locally via the script or trigger the workflow with `workflow_dispatch`), paste the snippet on the logged-in PyPI releases page, then re-run the publish. The monthly cron keeps it under the cap going forward. - Related: filed [pypi/support#12050](https://github.com/pypi/support/issues/12050) requesting a 50 GB limit as a one-off buffer; the recurring prune is the durable fix. @coderabbitai ignore --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
|
8ce0d39faa |
fix(lyrics-review): Tap To Sync timing corruption + padded edit-segment timeline (#956)
## Summary Fixes **Tap To Sync timing corruption** in the Edit Segment modal and reworks the segment timeline for easier re-timing of the first/last words. ### Tap To Sync fixes (`useManualSync`) - **Negative durations on re-sync** — the overlap-correction effect pulled a just-tapped word's end back to the *next* word's **stale** start (it hadn't been re-tapped yet), producing `end_time < start_time`. The sanitizer then reported *"Fixed N word timing(s) that fell outside this segment"* on reopen. Now guarded + clamped so it never inverts. - **First word squished / segment start ratcheting** — `clampSyncTime` forced every tap into the tight `[segStart, segEnd]`, so a legit tap in the 3s playback lead-in got shoved up to `segStart` and the start could never be re-synced earlier. Now allows taps in `[segStart − leadIn, segEnd + leadOut]` (floored at 0). - **Last word cut off** — symmetric lead-out + auto-stop no longer fires while the spacebar is held, so the last word can extend the segment (hold past the end / default-duration overshoot) instead of being truncated. - Both keyboard **and touch** sync paths use the widened window. ### Edit Segment timeline UX - ±1s **greyed context padding** on each side (dashed boundary), **faded waveform** outside the segment, and **neighbouring segments' words** as read-only context blocks. Word bars, drag, ruler, cursor and click-to-scrub all map over the padded view; words can be dragged/synced into the padding to extend the segment. - Modal widened `960px → 95vw / max 1400px`. - Header **play button** starts 1s before and auto-stops 1s after the segment. ### Dev tooling (dev-gated, no prod impact) - `&edit=1` makes a replay session editable locally (guarded by `NODE_ENV !== 'production'`) so the real review UI can be exercised against real job data + audio. Documented in `docs/REPLAY.md`. - `review_test_fixtures.py` now serves a synthetic **click-track** so Tap To Sync is exercisable offline. ## Testing - New/expanded unit tests in `hooks/__tests__/useManualSync*.ts(x)`: negative-duration regression (fails on old code, passes on fix), lead-in/lead-out clamp cases, and segment-end extension. All 13 pass; 178 lyrics-review + hook tests green. - Typecheck clean on touched files. - Verified end-to-end in the browser against a real completed job (replay + real vocals): tap-sync produces valid monotonic timings, no sanitizer banner, first/last words extend correctly, timeline padding + fade + neighbour words render as intended. ## Notes - CodeRabbit reviewed locally; its one finding (touch path missing the lead window) is fixed. @coderabbitai ignore --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
|
9b6c4f236e |
fix: Cloud Tasks queue region from GPU workers + REVIEW_COMPLETE lost-trigger recovery (v0.202.1) (#948)
## Summary Follow-up to #947, found by live-testing auto-approval in prod (job `b8bda9c2` — the first real fully-auto-approved job). **Bug:** the GPU `audio-separation-job` sets `GCP_REGION` to its GPU region, but every Cloud Tasks queue lives in `us-central1`. Any queue enqueue from inside that worker fails with `404 Queue does not exist` — which silently dropped the auto-approval render trigger (job auto-approved perfectly, then sat at `review_complete` forever). This is also a latent pre-existing bug for any other enqueue attempted from the GPU job. **Fixes:** 1. New `settings.cloud_tasks_region` (default `us-central1`, overridable via `CLOUD_TASKS_REGION`) used for all Cloud Tasks queue paths. `gcp_region` is still used for Cloud Run Job locations (correct). 2. `recover-stuck-jobs` cron gains a sweep for `REVIEW_COMPLETE` jobs stalled >10 min with no `render_progress` — re-triggers the render worker (idempotent via worker-generation fence). This closes the lost-trigger orphan gap for BOTH the human complete-review fire-and-forget task and the auto-approval path, and will auto-rescue `b8bda9c2` on the first tick after deploy. ## Testing - New unit tests: queue path uses `us-central1` even when `GCP_REGION=us-east4`; stall detector (stalled / fresh / render-underway). - Note: `test_internal_api.py` auth tests fail identically on pristine `main` in this local env (green in CI) — pre-existing, unrelated. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
f01213f5cb |
Fully-auto review for confident jobs: review_mode framework + auto-approval scorer/executor + replay tool (v0.202.0) (#947)
## Summary Ships the **fully-automated review framework**: jobs now default to `review_mode="auto"` — when the auto-approval scorer is fully confident on BOTH axes, the combined review screens are skipped entirely and the job renders straight through. Everything else falls back to human review exactly as before. **The confident class (deliberately narrow to start, ~5% of jobs expected):** - **Lyrics**: synced reference present, ≥90-99% anchor coverage, and every gap word covered by the pre-generated AI auto-correct suggestions (which the executor applies server-side — a faithful port of the review UI's on-load auto-apply, including conflict-group resolution) — OR zero gaps outright. - **Backing vocals**: non-subjective only — no audible backing content → clean instrumental. - **Never-auto gates**: vocalization sections (da-da-dun/woo-woo — P3), phantom/absurd-duration lines (P8), too-few-words, no/unsynced reference, errored backing analysis. **Calibrated against a 20-job human-review corpus** (all recent admin reviews replayed + annotated): safety 100% (every job that truly needed a human stays in review), lyrics-auto 3/20, overall-auto 1/20. ## How it works - `screens_worker` (pre-AWAITING_REVIEW) and `audio_worker` (post-stems, for the audio-lags-lyrics ordering) both call `maybe_auto_complete_review()`. Every job gets a verdict recorded in `processing_metadata.auto_approval` (shadow data keeps accumulating for calibration); enforcement additionally requires: flag on + `review_mode="auto"` + not made-for-you + no existing/custom instrumental + audio complete. - Enforce path mirrors a human "Complete Review": apply cached AI suggestions → sanity checks → `corrections_updated.json` → `instrumental_selection="clean"` → clear progress keys → `GENERATING_SCREENS→REVIEW_COMPLETE` (new legal edge; no review token or "ready for review" notification is ever issued) → trigger render worker. - **Fail-safe everywhere**: stale suggestions, duplicate-word artifacts (P1 self-conflict signature), state changed mid-enforce (human opened editor), any exception → normal human review. Auto-approval can never fail a job. - Kill switch: `AUTO_APPROVAL_ENFORCE_ENABLED=false` (scoring continues). Per-job: `review_mode="always_review"` (admin PATCH, validated). - Excluded from enforcement (still shadow-scored): made-for-you orders (human QC is part of that product), custom-instrumental jobs. ## Also in this PR (sessions 1-3 of the initiative) - **Review replay tool** (admin-only, prod-inert): `?replay=1` reopens the real lyrics/instrumental review UIs read-only for completed jobs — queue nav, Post-AI/Final toggle, edit-log action panel, dev-only audio byte-proxy (env-gated `REVIEW_AUDIO_PROXY`, path-anchored). This is what produced the calibration corpus. - `auto_approval` service package: scorer (anchor/gap/AI-coverage signals + P3/P8 gate detectors), apply engine, executor, lyrics-diff, offline capture script. ## Testing - 100+ new unit tests across scorer/apply/executor/replay routes; full backend suite 4000+ green locally (8 pre-existing emulator failures reproduce identically on pristine `main` — local-env only, green in CI). Frontend: 1196 Jest tests green. - Scorer validated read-only against the 20 real prod jobs in the private calibration corpus. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
70ea964ca9 |
feat(dev): full local backend for external contributors (no GCP creds needed) (#946)
## Summary Motivated by external contributor Martín (netux) asking "is there an easy way to run the full backend without GCS?" — he couldn't test PR #895's backend changes locally. This PR makes the whole backend runnable locally against emulators with **zero GCP credentials and zero paid API keys**. ### What was broken for contributors - Workers migrated to Cloud Run Jobs for prod robustness (audio-download, audio-separation, lyrics-transcription, bulk-search) silently bypassed the direct-HTTP dev mode → local jobs stalled at PENDING forever - Signed URL generation called `google.auth.default()` → crashed with no credentials (review audio, downloads, theme previews, upload URLs) - Magic-link login 503'd locally (console email provider counted as "not configured") - Fresh emulator had no default theme → every job create returned 422 - `docs/DEVELOPMENT.md` told people to `cp .env.example .env` but the file didn't exist — and nothing loaded `.env` anyway - `run-backend-local.sh` defaulted to the production GCP project ### Changes - **worker_service**: CRJ-based workers run as local subprocesses (`python -m backend.workers.X --job-id Y`, exact CRJ parity) when `ENABLE_CLOUD_TASKS=false` **and** not `is_production()` (regression-tested both ways) - **storage_service**: signed URLs → plain fake-gcs-server URLs when `STORAGE_EMULATOR_HOST` set (GET via JSON API download path; PUT via unvalidated-signature path, enabled by `-public-host` on the emulator) - **users route**: magic-link 503 only in production; dev prints the link to the console - **file_upload bugfix**: a theme with style params but no asset files never got `style_params_gcs_path` written (empty-dict falsiness in `elif theme_style_assets:`) — surfaced by the minimal seed theme, latent in prod - **scripts/seed-local-data.py** (new): creates bucket + default theme; wired into `run-backend-local.sh --with-emulators`, which now uses `test-project` - **config**: `load_dotenv()` so `.env` actually works; new root `.env.example` - **CONTRIBUTING.md** (new): no-credentials quickstart; README + DEVELOPMENT.md link to it ### Verified end-to-end locally (no GCP credentials) Upload job → local audio separation (real models) → transcription → screens → `awaiting_review`; review endpoints all working including `/api/review/{job}/audio/vocals` (the PR #895 endpoint) and 302 → emulator audio URLs. Magic-link login round-trip via console link. ### Tests - 224 tests across affected files pass; full backend unit suite: 3969 passed - `backend/tests/test_emulator_integration.py` has 8 pre-existing failures that reproduce identically on unmodified `main` locally (broken theme-service mock in that file) — unrelated to this PR - New tests: dev-mode subprocess dispatch (audio + bulk-search), prod-mode never-Popen guard, emulator URL generation (GET + PUT) @coderabbitai ignore --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
7729d30937 |
build: build frontend into the wheel in CI instead of committing out/ to git (supersedes #892) (#926)
Build the Next.js frontend into the wheel in CI instead of committing the generated out/ to git (fixes #876; supersedes #892). Validated: package-build-test green (npm ci → make build-frontend → poetry build → verify_wheel_frontend guard → install). Thanks @netux for surfacing it. |
||
|
|
6f86b3d901 |
test(e2e): rewrite tenant daily E2E as a faithful frontend journey (#873)
## Why The daily tenant E2E (`scripts/e2e/tenant_e2e.py`) drove the whole flow via **direct backend API calls**, bypassing the frontend. Two problems: 1. **Not representative** — it never exercised the tenant portal UI a real operator uses. 2. **Intermittent stuck-render orphans** (Run #31 vocalstar, #32 singa) — because it never loaded the lyrics-review page, it never fired `encoding-worker/warmup`, so the encoder was cold when the render started and the job orphaned in `rendering_video` (no recovery). Root-caused this session; the fix is to make the test faithful, which fires the warmup a real user fires. ## What Replaces `tenant_e2e.py` with **`frontend/e2e/production/tenant-happy-path.spec.ts`** — a Playwright journey mirroring `happy-path-real-user.spec.ts`, run per-tenant via the existing `e2e-tenant-daily.yml` matrix (vocalstar, singa; `max-parallel: 1`): login (real magic-link UI) → submit track (tenant form + real audio upload) → wait for transcription → **open review UI (fires warmup)** → approve → wait for render → assert tenant distribution (Dropbox present, no YouTube, downloads, `is_private`) → cleanup. - **Tenant auth**: added `inbox.testmail.app` to both tenants' `allowed_email_domains` (setup scripts + live GCS config via `scripts/e2e/update_tenant_allowlist.py`) so the test signs in through the real magic-link UI. - **Test audio**: short 60s clips fetched from private GCS in CI (`scripts/e2e/fetch_test_audio.sh`) — fast, reliable uploads and quick renders. - Workflow reworked to Node/Playwright; `tenant_e2e.py` deleted; version `0.190.3 → 0.190.4`. ## Findings surfaced (validating the exercise) - **Browser upload was CORS-blocked in Playwright** by the shared config's global `x-playwright-test` header (it made the GCS signed-URL PUT preflight request a header the bucket's CORS doesn't allow). Verified the same PUT succeeds in a real browser — a **test-harness artifact, not a product bug**. Worked around by stripping the header for `storage.googleapis.com` requests. (Note: any prod E2E doing a browser signed-URL upload is affected by this global header.) ## Verification Both tenants run **green end-to-end against prod** locally (login → submit → review → render → distribution; render completed without orphaning). Reworked workflow dispatched on this branch to confirm CI wiring (Node setup + private-audio fetch + secrets). Test-only + CI + tenant-config change. No product code. 🤖 Generated with [Claude Code](https://claude.com/claude-code) @coderabbitai ignore --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
21e8a05c61 |
test(tenant): retry daily E2E once on transient encoder failure (#834)
@coderabbitai ignore
The shared GCE encoder intermittently drops a job mid-encode ("lost
contact with worker ... not found") — a pre-existing,
**non-tenant-specific** transient (it hit the daily E2E twice but not
the manual runs; both tenants complete when the encoder cooperates). The
daily health check now detects that specific error and retries the whole
job once, so it doesn't false-alarm on encoder flakiness. Other failures
still fail immediately. Test-script only.
Separately worth a look (out of scope here): the encoder "lost contact /
job not found" reliability affects consumer renders too.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
|
1218337346 |
test(tenant): daily E2E asserts portal downloads via file_urls (#831)
@coderabbitai ignore The daily tenant E2E asserted on `job.download_urls` (backed by `output_files`, which the orchestrator pipeline never populates), so it failed even on fully-successful jobs. The `OutputLinks` UI reads `job.file_urls` (finals/packages) — assert that instead. Confirmed in prod: **both vocalstar and singa pass the full E2E** — job completes, Dropbox link present, no YouTube, and all downloads available (4K / 720p / with-vocals MP4 + CDG + TXT). Doc updated with the final outcome. Test/docs only — no production code change. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
66dcfd7d19 |
fix(tenant): stage instrumental in the orchestrator encode path (#829)
@coderabbitai ignore Follow-up to #828. That PR staged the instrumental in `_encode_via_gce` (legacy `video_worker.py`), but production jobs run the **orchestrator** pipeline (`video_worker_orchestrator.py`), whose `_run_encoding` builds its own `EncodingInput` and never made the user-supplied instrumental available under `jobs/{job_id}/`. Tenant jobs still failed at the GCE encoder with "No instrumental audio found" (confirmed — the "Staged existing instrumental" log never appeared). **Fix:** in the orchestrator's `_run_encoding`, when using the GCE backend, upload the local instrumental (existing/custom — identified by the `(Instrumental User)` / `(Instrumental Custom)` filename) into `jobs/{job_id}/` so the encoder's recursive `find_file` locates it. Consumer `clean`/`backing` jobs are untouched (their stems already live under `jobs/{job_id}/`). Also: the daily E2E uses a tenant-specific title so the two matrix tenants don't share a `base_name` (the GCE encoder caches by base_name). Validated by re-running the full prod E2E to completion after deploy. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
40c7960854 |
fix(tenant): stage instrumental for the GCE encoder + daily tenant E2E (#828)
@coderabbitai ignore ## Final render blocker (bug #8) With #824/#825/#826/#827, tenant jobs now render the karaoke video — but the GCE encoder failed with **"No instrumental audio found"**. **Root cause:** the user-supplied instrumental lives under `uploads/{job}/audio/`, but the encoder only pulls input from `jobs/{job}/` (`input_gcs_path`) and locates the instrumental via `find_file("*existing_instrumental*", "*Instrumental User*")`. On top of that, the running encoder VM (`encoding-worker-b`) boots a **2026-01-23 image** that predates the encoder's own `existing_instrumental` config-download path (added 2026-03-04), so it could never fetch it from `uploads/` either. **Fix (image-agnostic — no risky shared-VM recreation, no consumer impact):** `_encode_via_gce` server-side copies the instrumental into `jobs/{job_id}/existing_instrumental.<ext>` before submitting. The encoder downloads `jobs/{job}/` and its **recursive** `find_file` then locates it on any image version. Added `StorageService.copy_blob` + unit test. > Note: the stale encoder VM image is a separate infra item — the `encoding-worker` VMs should be recreated from the image family's newest image (which has 3 months of encoder fixes). Tracked for a Pulumi pass; not required for this fix. ## Daily tenant E2E (second ask) - `scripts/e2e/tenant_e2e.py` — full per-tenant prod health check: submit → approve review → render → assert (completes, Dropbox link present, no YouTube, downloads available) → clean up job + uploads. - `.github/workflows/e2e-tenant-daily.yml` — daily matrix (vocalstar, singa), WIF auth, Discord + email on failure. - `docs/archive/2026-06-15-tenant-portals-e2e.md` — the full story (8 bugs, all fixed). ## Testing - `copy_blob` unit test; storage suite green (20). - Full tenant E2E re-run to completion after deploy. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
d6bd2f1787 |
fix(tenant): complete the render for jobs that supply their own instrumental (#827)
@coderabbitai ignore ## Two final render blockers (found by the Vocal Star E2E) With attribution (#824), the audio slot (#825), and transcription padding (#826) fixed, the first real tenant job advanced through transcription → screens → review → render, surfacing the last two never-exercised issues: ### 1. Render prerequisites — "No instrumental selected" existing-instrumental jobs (every tenant submission) have no instrumental-selection UI step, so `instrumental_selection` was never written to state_data. The video orchestrator then defaults to `'clean'` (a separated stem that was never produced for these jobs), and `_validate_prerequisites` rejects the job, leaving it stuck retrying. **Fix:** `complete-review` now defaults `instrumental_selection` to `'custom'` (use the provided instrumental) when `existing_instrumental_gcs_path` is set. An explicit request value still wins. The render path already supports `'custom'`. ### 2. Theme asset paths (reproducibility) The tenant setup scripts wrote full `gs://bucket/themes/.../assets/X` URLs, but `theme_service._build_style_assets_mapping` only resolves bare basenames (the consumer convention). With `gs://` URLs, `style_assets` came back empty → assets never downloaded → encoder failed with "Video background image not found". Scripts now emit basenames, and add the newly-required `extra_text_text_transform` field the strict theme validator expects. (The two live GCS themes — vocalstar, singa — were corrected out-of-band to match.) ## Testing - New regression tests: complete-review defaults to `custom` for existing-instrumental jobs; explicit selection still wins. - Review suites green (102). Full E2E re-run to completion after deploy. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
fea6ebec1a |
fix(tenant): make white-label B2B portals attribute jobs to the tenant (Vocal Star + Singa) (#824)
@coderabbitai ignore ## Problem The white-label tenant portals (`vocalstar.nomadkaraoke.com`, `singa.nomadkaraoke.com`) looked "done" but **had never worked end-to-end**: 0 of the last 484 production jobs carried a `tenant_id`. No tenant job had ever run. **Root cause:** the frontend API client never sent the `X-Tenant-ID` header. Tenant portals call the shared `api.nomadkaraoke.com`, whose Host isn't the tenant subdomain, and the `?tenant=` query param is disabled in prod — so the backend could never detect the tenant. Every tenant sign-in and job submission silently fell back to consumer behavior (default theme, no tenant Dropbox, etc.). The `getTenantHeaders()` helper existed and was unit-tested, but was never wired into requests. ## Changes **Frontend** (`lib/api.ts`, `lib/tenant.ts`) - Attach `X-Tenant-ID` on all authenticated requests **and** the magic-link request, sourced from `window.__TENANT_CONFIG__` (edge-injected by the Cloudflare function; mirrored from the tenant store for dev/preview). Consumer build resolves no tenant → no header → **behavior unchanged**. **Backend** (`_apply_tenant_overrides` in `file_upload.py`) - Make tenant config **authoritative** over global distribution defaults. Prod sets `DEFAULT_DROPBOX_PATH` / `DEFAULT_GDRIVE_FOLDER_ID` / `DEFAULT_BRAND_PREFIX` globally; these seed `dist`, and the old "fill only when unset" overlay let them leak — so every tenant job would have landed in the **shared consumer Dropbox folder** and uploaded to the **global Google Drive**. Now Dropbox/GDrive routing is gated by the tenant feature flags and cleared when disabled. **Tenant configs** (GCS + setup scripts, Vocal Star + Singa) - Disable Google Drive (`gdrive_upload=false`, `gdrive_folder_id=null`), keep YouTube off, enable Dropbox to each tenant's own folder. B2B = locked theme, no YouTube/GDrive, Dropbox + portal download only. ## Why it's consumer-safe - Frontend: header only attached when a tenant is resolved (never on `gen.nomadkaraoke.com`). - Backend: `_apply_tenant_overrides` early-returns when `tenant_config is None` (all consumer jobs). ## Testing - New: frontend header-attachment tests (tenant build vs consumer); backend authoritative-override + feature-flag-gating regression guards (incl. the global-default-leak case). - Full suites green: **3466 backend**, **1013 frontend**. - Verified live tenant configs via API after update. Manual prod E2E (real track through each portal) + a daily per-tenant E2E workflow follow in a second PR, once this deploys and is verified in prod. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
39fb93653f |
fix(auto-correct): switch active model from Fable 5 to Opus 4.8 (#822)
## Summary Claude **Fable 5** (`claude-fable-5`) is no longer available on the Anthropic API. Verified live against `api.anthropic.com`: ``` 404 not_found_error: "Claude Fable 5 is not available. Please use Opus 4.8. Learn more: https://www.anthropic.com/news/fable-mythos-access" ``` `claude-opus-4-8` responds normally and is the API's own prescribed replacement. Opus 4.8 was also ≈ Gemini 3.1 Pro in our 30-job offline eval, so switching to it preserves the two-model compare quality. ## Changes - **`.github/workflows/ci.yml`** (load-bearing): `AUTO_CORRECT_MODEL=claude-opus-4-8` and `AUTO_CORRECT_COMPARE_MODELS=claude-opus-4-8;gemini-3.1-pro-preview`. Compare mode now runs **Opus 4.8 + Gemini 3.1 Pro** instead of Fable + Gemini. - **`scripts/measure_autocorrect_cost.py`**: default models + docstring example → Opus, so the cost tool measures what prod runs. - **`docs/API.md`**: example auto-correct response reflects Opus. - **`pyproject.toml`**: `0.182.1` → `0.182.2`. ## Re-enable path Fable is **disabled, not deleted** — its pricing entry in `pricing.py` and all Fable unit tests are intentionally left intact. If access returns, flip it back by reverting the two env-var values in `ci.yml`. ## Testing - Auto-correct suite: 73 passed (`backend/tests/services/auto_correct/` + `test_proactive_auto_correct.py`). - Live API probe confirmed Fable 404s and Opus 4.8 works. - CodeRabbit (uncommitted): 0 findings. @coderabbitai ignore Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
|
364e4ebd5f |
feat(auto-correct): proactive multi-model suggestions + cost logging + de-AI-ified review UI (#816)
## Summary Makes AI lyrics auto-correction run **automatically on every job** (multi-model), so suggestions are cached and ready the moment the reviewer opens the lyrics UI — plus token/cost instrumentation and a UI cleanup that removes all "AI"/"model" wording. Measured cost first (`scripts/measure_autocorrect_cost.py`): **~$0.13/track** multi-model on normal songs — cheap enough to run proactively. ## Changes **Cost instrumentation** - Capture per-model `usage` from Anthropic + Gemini; log greppable `auto-correct usage` / `auto-correct cost … total_est_cost_usd=…` lines from `suggest()`. - `backend/services/auto_correct/pricing.py` (Anthropic rates authoritative; Gemini estimated) + `scripts/measure_autocorrect_cost.py` offline harness. **Proactive generation (every job)** - `lyrics_worker._run_proactive_auto_correct()` runs the multi-model path after `mark_lyrics_complete` (downstream already triggered → non-blocking), gated by `AUTO_CORRECT_PROACTIVE_ENABLED`, bounded by a 180s timeout, and **swallows every error** — it never blocks or fails the karaoke job (flaky AI / out of credits / timeout all degrade gracefully). - Cache-key aligned with the UI request, so the review page's on-load call is an instant cache hit (no double spend). - `ci.yml`: wires `AUTO_CORRECT_*` env + the `anthropic-api-key` secret onto `lyrics-transcription-job` (same SA as the API service — no IAM change). **Review UI** - Always multi-model (removed the model-selection toggle); button renamed **"Auto-correct"**; stripped model/consensus/cached attribution badges; reworded strings to drop "AI"/"model" (translated to all 32 locales). - Auto-runs once on review load so suggestions are ready without a click; the button stays for the paste-references-later flow. ## Testing - Backend: **3460 passed** (`backend/tests/`, emulator excluded) — incl. new `test_pricing.py` + `test_proactive_auto_correct.py` (gating, graceful failure, cache alignment). - Frontend: **1006 jest tests pass**; type-clean; 33-locale key parity verified. ## Notes / follow-ups - Proactive activates on the **next deploy** (the lyrics-job env/secret apply via this PR's ci.yml on merge). - Left "AI" wording in two *separate* features untouched (different subsystems): `lyricsReview.agenticMetrics` and `lyricsReview.modals.correctionAnnotation` — flagged for a separate decision. - The production E2E spec's click-to-open flow may want a tweak now that suggestions auto-run on load. @coderabbitai ignore --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
e806616408 |
fix: idempotent session revoke, error-pattern script crash, mobile header overlap (#798)
## Summary Three independent fixes bundled together: 1. **`revoke_session` idempotency** — stops the recurring `Error revoking session` prod error on `karaoke-backend`. Firestore `document.update()` raises `NotFound` (404) when a session doc is already gone (logged out / expired / cleaned up); logout against an already-absent session now succeeds instead of erroring. 2. **`query-error-patterns.py` crash** — `last_seen`/`first_seen` are stored as ISO-8601 strings in Firestore, but the sort key and `--hours` filter did `float(value)`, raising `ValueError` and breaking the `--json` path that `/prod-investigate` relies on. Added a `_to_epoch()` helper handling datetime objects, ISO strings, and numeric epochs. 3. **Mobile header overlap** — `AppHeader` was `position: fixed` and each consumer cleared it with a hardcoded top padding. On narrow viewports the header wraps to 3 rows (149px on a 412px Pixel), exceeding the 112px offset, hiding the "Create Karaoke Video" heading and the top of the form behind it (worse across locales with longer button labels). Switched `AppHeader` to `position: sticky` (matching the already-working jobs-page header) so content always clears it at any height, and removed the now-redundant magic-number offsets. ## Changes - `backend/services/user_service.py` — catch `google_exceptions.NotFound` in `revoke_session`, return `True` - `backend/tests/test_revoke_session.py` — new unit tests (existing / missing / unexpected-error paths) - `scripts/query-error-patterns.py` — `_to_epoch()` helper for sort + `--hours` filter - `frontend/components/app-header.tsx` — `fixed` → `sticky` - `frontend/app/[locale]/app/page.tsx`, `frontend/app/[locale]/app/referrals/page.tsx` — drop hardcoded top padding - `pyproject.toml` — bump to 0.174.17 ## Testing - [x] Backend tests pass locally (47 passed incl. 3 new `revoke_session` tests) - [x] `query-error-patterns.py --json` and `--hours` verified against live Firestore - [x] Mobile header fix verified in Playwright at 412px (Pixel) and 1280px — overlap eliminated (title moves from 12px behind the header to a clear gap below it) - [x] CodeRabbit CLI review: 0 findings ## Review - [x] Local CodeRabbit review completed - [x] Review feedback addressed (none) @coderabbitai ignore --- 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
a45798bb4d |
fix: remove SendGrid leftovers; consolidate on Postmark (#779)
## Summary - Fixes the failing e2e-daily notification job ([run #25983930960](https://github.com/nomadkaraoke/karaoke-gen/actions/runs/25983930960/job/76378786060)) which still used \`smtp.sendgrid.net\` — SendGrid was cancelled ~3 weeks ago - Drops the \`sendgrid\` Python dep, removes \`SendGridEmailProvider\`, simplifies provider selection in \`email_service.py\` - Removes \`SENDGRID_API_KEY\` env bindings from Pulumi (gdrive-validator Cloud Function + video Cloud Run Job) and from ci.yml's Cloud Run \`--set-secrets\` - \`sendgrid-api-key\` Secret resource intentionally kept in \`infrastructure/modules/secrets.py\` with a "delete manually" marker so an unrelated \`pulumi up\` doesn't destroy the live secret + versions - Version bumped 0.174.10 → 0.174.11 ## Pulumi applied \`pulumi up --target\` already run locally with the two affected resources (video-encoding-job + gdrive-validator-function). PR #776's runner-manager state intentionally untouched. @coderabbitai ignore ## Test plan - [x] Full backend test suite (3178 tests) passes locally - [x] CodeRabbit local review: no findings - [x] Pulumi up applied locally (clean) - [ ] CI green on this PR - [ ] After merge: e2e-daily.yml notification job should pass on next scheduled run - [ ] Verify gdrive-validator function still emails on next post-job trigger 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
e40f39f0c2 |
fix(referrals): handle TZ-aware referred_at in record_earning (#737)
## Summary - **Bug**: `record_earning()` in `referral_service.py` compared naive `datetime.utcnow()` against TZ-aware `earning_expires` (derived from Firestore's TZ-aware `referred_at`), raising `TypeError: can't compare offset-naive and offset-aware datetimes`. The exception was swallowed by a broad `except` in the Stripe webhook handler and only logged as a warning, so **every referral earning was silently lost** since the referral system shipped (~3 weeks). - **Symptom (user report)**: referral code `angel` showed 0 purchases on the admin referrals page despite `tnt.travistoo@gmail.com` (a real user referred by angel) paying $28 on 2026-04-27. Production logs at the exact webhook moment confirmed: `Referral earning recording failed: can't compare offset-naive and offset-aware datetimes`. - **Fix**: same pattern previously applied to `_build_user_public()` — promote `now` to TZ-aware before comparing. - **Regression test**: existing tests mocked `referred_at` with naive `datetime.utcnow()`, so the bug was invisible. Added two TZ-aware variants matching production reality. - **Backfill script**: `scripts/backfill_referral_earnings.py` retroactively creates the missed `ReferralEarning` records and bumps per-link `stats.purchases` / `stats.total_earned_cents`. Idempotent, dry-run by default. Skips purchases predating attribution and refunded sessions. ## Backfill dry-run output (verified) ``` Earnings to create: 20 Skipped (not credit_purchase): 7 Skipped (before attribution): 7 Skipped (refunded / unsuccessful / outside window / zero): 0 Stats updates per link: angel: +1 purchases, +$5.60 (Travis) e2etest70: +15 purchases, +$9.00 (e2e tests) loyalty50: +4 purchases, +$7.15 (Angelina + John) ``` ## Test plan - [x] Unit test `test_creates_earning_with_tz_aware_referred_at` reproduces the production `TypeError` before the fix and passes after. - [x] Unit test `test_no_earning_outside_window_with_tz_aware_referred_at` confirms expired-window path also handles TZ-aware datetimes cleanly. - [x] Full referral test suite passes (71 tests). - [x] Full backend test suite passes (2987 tests). - [ ] **Post-merge**: run `python scripts/backfill_referral_earnings.py --apply` with full Firestore write credentials (claude-readonly ADC blocks writes). Expect 20 earnings created, 4 link stats updated. - [ ] **Post-merge**: verify admin referrals page now shows `angel: 1 purchase / $5.60 earned`. @coderabbitai ignore Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
9a5c9d497a |
feat: production error monitor — log collection, LLM analysis, Discord alerts (#713)
## Summary - Add a production error monitoring service that runs as a Cloud Run Job every 15 minutes - Queries Cloud Logging across all NK services (Cloud Run, Jobs, Functions, GCE VMs), normalizes/deduplicates errors, uses Gemini Flash for incident grouping, sends Discord alerts, and auto-resolves silent patterns - Updates all `/prod-*` slash commands to use pre-computed Firestore patterns as primary data source ## Architecture Based on the proven Aquarius error bot system, adapted for Nomad Karaoke's multi-service architecture (20+ resources across 4 resource types). **New backend modules** (`backend/services/error_monitor/`): - `config.py` — constants, monitored services, env var overrides - `normalizer.py` — regex normalization + SHA-256 pattern hashing - `known_issues.py` — 8 ignore patterns for infrastructure noise - `firestore_adapter.py` — CRUD for `error_patterns`, `error_incidents`, `discord_alerts` - `discord.py` — webhook client with 5 alert formatters (new pattern, incident, spike, auto-resolved, daily digest) - `llm_analysis.py` — Gemini Flash incident grouping + near-duplicate detection - `monitor.py` — orchestrator entry point (`python -m backend.services.error_monitor.monitor`) **New infrastructure** (`infrastructure/modules/error_monitor.py`): - Cloud Run Job (`nomad-error-monitor`) sharing karaoke-backend image - Cloud Scheduler: every 15 min monitor + 08:00 UTC daily digest - Scheduler service account with `roles/run.invoker` - Backend SA gets `roles/logging.viewer` **New CLI scripts:** - `scripts/query-error-patterns.py` — query patterns from Firestore - `scripts/resolve-error-pattern.py` — mark patterns as fixed with PR reference ## Testing - 310 unit tests across 7 test modules (all passing) - Covers: config, normalizer, known issues, Firestore adapter, Discord formatting, LLM analysis, monitor orchestrator - P0/P1 caller-callee contract tests for monitor integration points - All tests use mocked external dependencies (no GCP calls) ## Review - [x] CodeRabbit CLI review completed locally (1 cycle, 3 issues fixed) - [x] Test review completed (P0/P1 gaps addressed) - [x] Documentation updated (README, ARCHITECTURE) - [x] Version bumped (0.168.2 → 0.169.0) ## Design Spec `docs/archive/2026-04-12-error-monitor-design.md` @coderabbitai ignore --- 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
|
591c050545 |
feat: Reference lyrics search with relevance filtering (#632)
## Summary
- Add relevance filtering to the correction pipeline — automatically
rejects wrong-song reference lyrics (below 30% word match threshold,
validated against 168 prod source pairs)
- New `POST /api/review/{job_id}/search-lyrics` endpoint lets users
search with alternate artist/title from the review UI, with force-add
override for edge cases
- Redesigned Add Reference Lyrics modal with Search (default) and Paste
tabs, plus inline empty state when no valid sources exist
## Changes
### Backend
- **Relevance filtering** (`correction/relevance.py`): Per-source
scoring based on anchor sequence word coverage. Integrated into
`LyricsCorrector.run()` after anchor finding — sources below threshold
removed before corrections computed
- **Search endpoint** (`routes/review.py`): Queries all providers
(Genius, Spotify, Musixmatch, LRCLIB) with user-supplied artist/title,
auto-filters results, returns added/rejected/not-found breakdown
- **Provider factory** (`operations.py`): `create_lyrics_providers()`
reads env vars, `search_lyrics_sources()` orchestrates
fetch+filter+correct
- **Force override**: `add-lyrics` and `search-lyrics` support `force`
flag to bypass threshold
### Frontend
- **Tabbed modal**: `AddLyricsModal` rewritten with Search (default) and
Paste tabs
- **SearchLyricsTab**: Artist/title inputs, provider search, rejected
source cards with force-add checkboxes
- **ReferenceEmptyState**: Inline search/paste UI when no valid
reference sources exist
- **API client**: New `searchLyrics()` method with
`SearchLyricsResponse` types
### Analysis
- Empirical threshold analysis script
(`scripts/analyze_reference_relevance.py`) + visual review tool
- 168 source pairs from 81 prod jobs: wrong-song max 3.5%, correct-song
min 26.8% → 30% threshold
## Testing
- 35 new backend tests (relevance scoring, filtering, corrector
integration, search operation, API endpoint)
- 717 existing frontend tests passing (no regressions)
- Edge case: all sources filtered out handled gracefully
## Review
- [x] Local CodeRabbit review completed (3 issues fixed: response
contract, force param coercion, null checks)
@coderabbitai ignore
---
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
||
|
|
19efa269a8 |
feat: blue-green encoding worker deployment with deep health checks (#595)
## Summary - Replaces single encoding worker VM with a blue-green pair (a/b) for zero-downtime deployments - VMs auto-start on demand (JIT warmup when user opens lyrics review) and auto-shutdown after 15 min idle - CI deploys to secondary VM, runs a real encode test, then swaps the Firestore pointer atomically - Deep health check catches issues like PR #587's pip crash before they reach users - Cloud Function checks idle VMs every 5 minutes and stops them to minimize cost (~$10-15/mo idle vs $800/mo always-on) ## Architecture - **Firestore config doc** (`config/encoding-worker`) is single source of truth for primary/secondary routing - **Backend** reads primary IP from Firestore (30s TTL cache), starts VM on demand via Compute API - **CI workflow** handles blue-green swap: deploy → health check → real encode test → atomic Firestore swap → drain & stop old - **Cloud Function** auto-stops idle VMs based on active jobs, user activity, and deploy status - **Frontend** sends warmup signal on lyrics review page load + heartbeat on interactions ## Changes ### New Files - `backend/services/encoding_worker_manager.py` — VM lifecycle + Firestore config management - `backend/api/routes/encoding_worker.py` — Warmup and heartbeat API endpoints - `infrastructure/functions/encoding_worker_idle/` — Idle shutdown Cloud Function - `infrastructure/modules/encoding_worker_manager.py` — Pulumi module for Cloud Function + Scheduler - `scripts/seed-encoding-worker-config.py` — Firestore config seeding script - `infrastructure/encoding-worker/test-assets/test-preview.ass` — Test file for deploy health check ### Modified Files - `infrastructure/compute/encoding_worker_vm.py` — Single VM → blue-green pair - `infrastructure/__main__.py` — Wire up new VMs, Cloud Function, IAM - `backend/services/encoding_service.py` — Dynamic URL from Firestore, warmup-aware retry - `.github/workflows/ci.yml` — Blue-green deploy flow - `frontend/` — Warmup signal on lyrics review, heartbeat, warming-up UI state - `docs/` — Architecture, development, troubleshooting, lessons learned ## Test plan - [x] 47 new backend tests pass (EncodingWorkerManager, routes, dynamic URL resolution) - [x] 9 Cloud Function unit tests pass - [x] All 2808 existing backend tests pass (1 pre-existing failure unrelated) - [x] All 1116 existing unit tests pass (3 pre-existing failures unrelated) - [ ] Pulumi preview shows expected resource changes - [ ] Pulumi up applied locally before merge - [ ] Firestore config seeded with VM IPs - [ ] End-to-end verification after deployment ## Migration Steps (post-merge) 1. `pulumi up` creates 2 VMs, removes old VM 2. Seed Firestore: `python scripts/seed-encoding-worker-config.py <ip-a> <ip-b>` 3. Start VM A, verify health 4. First CI deploy exercises the blue-green flow ## Design Docs - Spec: `docs/superpowers/specs/2026-03-24-blue-green-encoding-worker-design.md` - Plan: `docs/superpowers/plans/2026-03-24-blue-green-encoding-worker.md` @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
|
b4be5968c2 |
fix: payment dashboard date handling, chart rendering, and sidebar padding (#466)
@coderabbitai ignore
## Summary
- **Revenue chart showed nothing**: `store_payment()` used
`datetime.utcnow()` for `created_at` instead of the Stripe session's
actual `created` timestamp. All backfilled payments appeared on a single
date. Fixed to use `session["created"]`.
- **Single data point invisible**: Recharts AreaChart doesn't render a
visible area with only one point. Added `dot={chart.length === 1}` so
single points show as dots.
- **Payment detail sidebar had no padding**: `SheetContent` had no
padding class. Added `p-6`.
- **Backfill date fix script**: Added `scripts/fix_backfill_dates.py`
(already run against production to correct 32 records).
## Test plan
- [x] All 32 payment tests pass locally
- [x] Revenue chart verified visually via dev server — shows proper date
distribution (Dec 2025 - Mar 2026)
- [x] Payment detail sidebar verified visually — proper padding on all
sides
- [x] Production API confirmed returning correct date-bucketed data
after fix script
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
|
||
|
|
937f8f7ec1 |
feat: YouTube quota GCP monitoring + legacy beta cleanup (#460)
## Summary - Remove the deprecated beta enrollment system (endpoints, models, email templates, admin UI, rate limits) — ~20 files cleaned - Remove legacy count-based YouTube upload tracking from rate_limit_service and dead blocklist methods - Add GCP Cloud Monitoring integration to cross-reference YouTube API quota usage with Firestore tracking - Add drift detection (alerts when GCP reported vs Firestore tracked differs >10%) - Display GCP quota data in admin rate-limits dashboard ## Changes ### Backend (15 files) - **Removed:** 5 beta endpoints, `BetaTesterStatus` enum, beta User fields, beta email templates, `validate_email_for_beta()`, deprecated YouTube/beta rate limit methods, dead blocklist CRUD methods - **Added:** `get_gcp_quota_usage()` to `YouTubeQuotaService` with Cloud Monitoring API query, 5-min cache, and drift detection - **Updated:** Rate limits API response model with GCP quota fields (`gcp_quota_available`, `gcp_quota_units_consumed`, `quota_drift`, `quota_drift_alert`) ### Frontend (11 files) - **Deleted:** `frontend/app/admin/beta/page.tsx` (entire beta admin page) - **Removed:** Beta sidebar nav, layout breadcrumb, admin stats card, beta types/API methods, legacy YouTube stats cards - **Added:** GCP Cloud Monitoring section in rate-limits page showing reported usage, delay, and drift alerts ### Tests (4 files) - Removed 8 obsolete tests for deprecated methods - Created `test_gcp_quota_monitor.py` with 10 tests covering GCP quota querying, caching, fallbacks, and drift detection - Updated mock data and assertions across test files ### Docs (3 files) - Removed beta endpoint docs from API.md, added GCP quota fields - Updated ARCHITECTURE.md with Cloud Monitoring integration - Removed beta program section from PRODUCT-VISION.md ## Test plan - [x] All 2265 backend tests pass (1 pre-existing failure unrelated) - [x] All 516 frontend Jest tests pass - [x] New GCP quota monitor tests cover success, caching, fallbacks, drift alerts - [ ] Verify admin rate-limits page no longer shows beta/legacy cards - [ ] Verify GCP quota section shows "unavailable" gracefully when monitoring data not yet populated @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
|
d91c2c2b4e |
feat: Stripe admin dashboard with revenue analytics and payment management (#461)
<!-- @coderabbitai ignore --> ## Summary - Full Stripe payment admin dashboard with revenue analytics, payment history, refund management, payout tracking, dispute monitoring, and webhook event audit log - New `StripeAdminService` backend service for payment data storage and analytics - Enhanced webhook handler to store enriched payment data and handle refunds/disputes - User payment history integration on admin user detail page - Backfill script for migrating existing payment data - 32 new unit/endpoint tests covering all P0 critical paths (refund logic, payment storage, credit deduction) - Production E2E tests for payments page and API endpoints - Bug fix: `auth_data[0]` subscript error in refund endpoint (AuthResult not subscriptable) ## Changes ### Backend (7 phases) - **New:** `backend/services/stripe_admin_service.py` - Core payment admin service (store, list, detail, refund, revenue summary, payouts, disputes, webhook events) - **New:** `scripts/backfill_stripe_payments.py` - One-time migration script for existing payments - **Modified:** `backend/api/routes/admin.py` - 9 new admin endpoints with Pydantic models - **Modified:** `backend/api/routes/users.py` - Enhanced webhook handler with payment storage, refund/dispute handling, event logging ### Frontend - **New:** `frontend/app/admin/payments/page.tsx` - 4-tab payments page (Revenue, Transactions, Payouts, Webhooks) - **New:** `frontend/components/admin/payment-detail.tsx` - Payment detail slide-out sheet - **New:** `frontend/components/admin/refund-dialog.tsx` - Refund dialog with credit deduction warning - **Modified:** `frontend/lib/api.ts` - 11 new TypeScript interfaces + adminApi methods - **Modified:** `frontend/app/admin/users/detail/page.tsx` - Payment history section - **Modified:** `frontend/components/admin/admin-sidebar.tsx` - Payments nav item - **Modified:** `frontend/app/admin/page.tsx` - Revenue card on dashboard ### Tests - **New:** `backend/tests/test_stripe_admin_service.py` - 21 unit tests - **New:** `backend/tests/test_admin_payment_endpoints.py` - 11 endpoint tests - **New:** `frontend/e2e/production/admin-payments.spec.ts` - Production E2E tests ## Post-deploy steps 1. Register webhook events in Stripe Dashboard: `charge.refunded`, `charge.dispute.created`, `charge.dispute.closed` 2. Run backfill: `python scripts/backfill_stripe_payments.py` 3. Create Firestore indexes if prompted ## Test plan - [x] 32 new tests pass (service + endpoint) - [x] 181 total tests pass (full suite, 1 pre-existing skip) - [ ] Visual verification of payments page after deploy - [ ] Backfill script dry-run against production - [ ] Verify webhook events are logged after deploy 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
|
2615261672 |
feat: add Singa whitelabel tenant and fix tenant infrastructure (#453)
## Summary - Add Singa as second whitelabel tenant (green/black branding, file-upload-only, locked theme) - Fix critical Zustand getter bug where computed properties became stale after set() calls - Fix React hydration error #300 (conditional return between hooks) - Add TenantLandingPage component for minimal tenant portal - Add Cloudflare Pages cache control headers - Add gs:// URL conversion fallback in tenant-logo ## Test plan - 27 tenant unit tests pass (5 new regression tests for Zustand bug) - 181 E2E tests pass - Verified singa/vocalstar/gen domains all working correctly <!-- @coderabbitai ignore --> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |