mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
bfc8f7fb0f6429fe640c758252f5c98f1d7d8299
123
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
bfc8f7fb0f |
feat(cdg): configurable title card + per-theme CDG lyrics font (Vocal Star up card) (v0.273.0) (#1157)
## Summary Vocal Star (and other commercial producers) need CDG files in their own house style. Matt's 2025-02-20 email showed our CDG up card overflowing its bordered design. He asked for his simpler card instead: logo on top, then title, a small "MADE FAMOUS BY" line, then the artist. Two pipeline limits blocked that: - the CDG title card was hardcoded; - a theme could only have **one font** for everything. The shared `font` asset (`intro.font`) overwrites `karaoke.font_path` and `cdg.font_path` at render time. ## Changes - **Title card (cdgmaker composer):** new optional keys; all defaults reproduce today's output. - `title_separator_text` / `_color` / `_font_size`: a line between title and artist. - `title_font_size`, `artist_font_size`, `title_max_height`. - `title_block_top`: centre the block below a logo baked into the background. - It shrinks the title first, then the artist, so it can't loop forever. - **Outro:** respects `title_block_top` and shrinks long text. Before, an `assert` crashed when the text was taller than 200px. - **Fonts:** - The CDG title card and outro use the theme's main font (`cdg.title_font_path`, fed by the shared `font` asset). - A theme may keep a **distinct CDG lyrics font** via a new `cdg_font` asset. The theme service emits it only when it differs from `intro.font`, and `update_asset_paths` applies the shared font first so section overrides win. - No visible change for existing themes, which all use one font. - **Tenant theme editor:** keeps a distinct, available CDG font on save, and copies it into the theme's assets if it's bundled. - Docs: API.md. ## Testing - New `tests/unit/lyrics_transcriber/output/test_cdg_title_card.py` renders **real CDG files**, decodes the packets back into frames and checks: - the separator sits between title and artist; - text stays below `title_block_top` and inside the safe area; - the title font is applied, and a missing one falls back; - long outro text shrinks instead of crashing. - New tests for asset-mapping order, `cdg_font` emission, theme-editor font handling, and the updated mapping assertion. - Package unit tests: 2053 passed. Backend: 6056 passed (emulator tests excluded locally). - Local CodeRabbit with `--fresh`: no findings. - Rendered Vocal Star samples (short, wrapped-long and very long title/artist), decoded and visually checked against Matt's design. After deploy, the Vocal Star theme gets: a new up-card background (logo cropped from Matt's own card), separator "MADE FAMOUS BY", `title_block_top`, black border, Avenir as the main font and Oswald kept for CDG lyrics. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Gp5fBmsmWJxgkEHb4JfhXR --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
5047f39da2 |
feat(tenant): per-track catalogue codes prefix tenant outputs (Vocal Star VS codes) (v0.272.0) (#1153)
## Summary Vocal Star (Matt, April 2025) asked for their own VS catalogue code at the start of every output file, with all files in one folder (e.g. `VS6588 - Gerry and the Pacemakers - Ferry Cross The Mersey`). Tenant operators can now set a per-track **Track code**. It becomes the job's preserved brand code (`keep_brand_code`, which both render paths already honour), so the Dropbox folder and every delivered file use it instead of an allocated `VSTAR-NNNN`. ## Changes - **Backend:** `POST /api/jobs/create-with-upload-urls` accepts `track_code`. - Tenant portals only; `400` elsewhere. - Strict filesystem-safe validation, upper-cased. - Maps to `JobCreate.keep_brand_code`. - **Bulk analyze:** a leading code in filenames (`VS6588 Artist - Title Guide.mp3`) is split off and returned as `track_code`. - Files pair only when code **and** song match. - LLM pairs with conflicting codes are rejected, and the code is stripped from LLM-extracted artists. - **Frontend:** an optional Track code field on the tenant single-track form and on each bulk row. - Client validation mirrors the backend, with a visible error. - Resumed bulk rows keep their code, locked. - 33 locales translated. - Docs: API.md. Version 0.271.0 → 0.272.0. ## Testing - Backend: new tests for endpoint mapping, rejection outside tenants, unsafe codes, normalisation, filename parsing, the band-name false positive (UB40), same-code/different-song and LLM conflicts. - Full backend suite: 5985 passed. The only 30 failures are emulator tests, because the local Firestore emulator wasn't running. - Frontend: full Jest 1680/1680 passing, plus new TenantJobFlow / TenantBulkFlow tests. `tsc` shows no new errors. - Local CodeRabbit: 3 cycles, all real findings fixed. The one remaining note is on a pre-existing string. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Gp5fBmsmWJxgkEHb4JfhXR --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
ff9b716435 |
feat: first-touch UTM/referrer signup attribution + UTM on YouTube description (v0.269.0) (#1154)
## Summary This is BACKLOG Frog #2 ("plug the gen revenue leaks"), attribution track. Until now we stored no acquisition source, so we couldn't tell which channel brings the people who pay. - **Capture (frontend):** `lib/attribution.ts` stores the first-touch `utm_*` params, the external referrer host and the landing path for 30 days. It runs in the `[locale]` layout and in `LocaleRedirect`, before `location.replace` wipes `document.referrer`. The data is sent with `POST /auth/magic-link`. - **Store (backend):** `services/signup_attribution.py` whitelists, length-caps and sanitises the data. On account creation it sets `signup_attribution` and `signup_source`: the `utm_source`, or a referrer channel label such as `youtube`, `google`, `website`, `decide` or `kjbox`. Malformed input never blocks sign-in, and the request field is bounded (16 keys, values up to 500 chars). - **Report:** `GET /api/users/admin/attribution-summary?days=90` returns signups, verified, made a job, paid and revenue, per source. The admin user detail shows the signup source and attribution. - **YouTube:** description template links now carry `utm_source=youtube&utm_medium=description&utm_campaign=requests|make_cta`. They're static, so they're still derivable from the title, and the daily backfill rolls them out. The `requests.nomadkaraoke.com` Cloudflare redirect now has `preserve_query_string` (Pulumi `nomad-cloudflare`, **already applied** locally). - **Docs:** STRIPE-SETUP.md gets a "Legacy Payment Links" section. The old order-form link `dR67sx…` was **deactivated 2026-10-14** and points people to gen; it had 0 completed checkouts in about 30 sessions since Dec. Also updated API.md. Companion: nomadkaraoke/public-website#34 makes `nomadkaraoke.com/r/CODE` pass the UTM params through. ## Testing - `make test`: backend 5986 passed, emulator 150, frontend Jest 1683, package 2044. All green. - New tests: `backend/tests/test_signup_attribution.py` (cleaning, channel labels, magic-link wiring, bounded payload, summary endpoint) and `frontend/lib/__tests__/attribution.test.ts`. - Local CodeRabbit: 3 findings fixed (malformed referrer, NaN expiry, unbounded payload); re-review found 0. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_019KuzHAU4rxkv8JfD8NmHTN Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
3566fd136e |
fix: loop-stall alert flapped on idle autoscaled instances (v0.268.5) (#1140)
## Summary The `>=2 event-loop stalls of 10s+ in 30 minutes` alert flapped on 2026-10-12 and 2026-10-13. Investigation: - All 11 ERROR-level `EVENT_LOOP_STALL ended` lines in the last 4 days (10-40s each) were on instances with **zero requests** in or near the stall window. The check joined `labels.instanceId` between the stall lines and the request logs. - None of them had a culprit. A process freeze advances the 1s/5s/15s sample schedule without taking a sample. - **Pattern:** Cloud Run autoscaling starts an extra instance, it serves one request (e.g. `/api/tenant/config`), then it sits idle and CPU-throttled. The startup warmup thread competes for CPU slices meanwhile (SpaCy preload took 690s on one instance). The watchdog reads that as a loop stall. ## Changes - `RequestTracker` + pure-ASGI `RequestTrackingMiddleware` (outermost) count in-flight and started requests. - `LoopWatchdog(request_tracker=...)`: a stall logs at ERROR, the level the alert counts, only when a request was in flight at its start or arrived during it. The verdict waits 0.5s after the loop resumes so requests queued in the socket are counted. The same rule applies to `in progress` samples, so they don't show up in the error monitor either. - Idle stalls still log at WARNING with `requests_affected=0` and are still recorded to `client_events` (`detail.requests_affected`). - Without a tracker, behaviour is unchanged. - Docs: TROUBLESHOOTING, LESSONS-LEARNED, API. No infra change: the alert metric already filters on `severity>=ERROR`. ## Testing - 10 new unit tests: - idle → WARNING; - in-flight request → ERROR; - request queued during the stall and accepted after resume → ERROR; - request finished mid-stall → still ERROR; - in-progress sample levels; - no-tracker behaviour unchanged; - middleware counting, including exceptions and non-http scopes; - middleware is installed outermost. - `make test`: all passed. - Local CodeRabbit review: 2 findings, both fixed. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01DTfrTaaZQfXUUduAhZpLeq Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
983f6ecdcd |
fix(storage): keep user-uploaded inputs forever; uploads/ expires after 180d not 7d (v0.268.2) (#1137)
## Summary
`uploads/` had a bucket lifecycle **delete after 7 days**, and user
inputs were only partly copied out of it. The only copy step was the
audio worker (stem separation) copying the mix to `jobs/{id}/input/`,
best-effort. Everything else stayed in `uploads/`:
- bring-your-own-instrumental / tenant jobs, which skip separation;
- the uploaded instrumental, which was only staged at render time;
- lyrics and style uploads, which were never copied.
Result:
- randy-vild re-renders 404'd on their own input (2026-10-08). That data
was restored from Andrew's Dropbox.
- Firestore shows 64 completed Jan–Feb 2026 consumer jobs, plus
BYO-instrumental jobs, whose inputs are gone permanently.
## Changes
- **`backend/services/input_persistence.py`** (new):
- Server-side copies `uploads/{id}/**` to `jobs/{id}/input/**` and
repoints every job field that referenced the upload:
`input_media_gcs_path`, `existing_instrumental_gcs_path`,
`lyrics_file_gcs_path`, `style_params_gcs_path`, plus any matching value
under `file_urls` / `state_data` / `style_assets`.
- Idempotent: unchanged copies are skipped.
- The audio, lyrics, screens, render-video and video workers all call it
on entry, via `asyncio.to_thread`. On failure it logs an ERROR, which
alerts, but doesn't fail the job.
- **`POST /api/admin/persist-uploads`** (dry run by default): backfill
for jobs no worker will touch again. Lookup failures are isolated per
job.
- **GCE encoder:** prefers the explicitly downloaded top-level
`existing_instrumental.*`. The recursive glob could otherwise return the
raw, unconformed copy now kept under `input/audio/`.
- **Pulumi:** `uploads/` lifecycle goes from 7 to **180 days**; `temp/`
stays at 7. **Already applied** and verified directly on the bucket.
- **Docs:** LESSONS-LEARNED, DISASTER-RECOVERY (new user-inputs row; the
stale "30d noncurrent" figure corrected to 7d), API, README.
## Verification
- `make test` all green.
- 16 new tests:
- copy mapping, repointing, idempotency, re-copy when a kept copy
differs, already-expired paths left alone;
- the worker wrapper never raises;
- an AST guard that every pipeline worker calls the copy step via
`to_thread`;
- backfill dry run / apply / error isolation;
- the encoder's instrumental preference.
- Local CodeRabbit: 2 findings, both fixed (lyrics path, per-job
lookup); re-review clean.
## After merge
Run `POST /api/admin/persist-uploads` (dry run, then `?dry_run=false`)
to copy the inputs of the 51 jobs that still have objects in `uploads/`.
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
f2612ac110 |
fix: close edit→delete refund loop, retry lost YouTube uploads, unblock event loop (v0.268.1) (#1135)
## Summary Investigation of two 2026-10-08 Discord alerts (plus a third that arrived mid-session) turned up four issues: 1. **Credit-refund loop.** A brand-new welcome-credit account received a finished video (job 8a10a22d), clicked **Edit** (status back to `awaiting_review`), then **deleted** the job and got its credit back. `delete_job` refunds any non-complete status, and Edit makes a delivered job look "never completed". 2. **YouTube 410 "Gone" dropped the upload.** The resumable session was lost 1.3s into the upload. `upload_video` called `execute()` once with no retries, and the orchestrator only queued quota errors, so any other failure lost the upload. 3. **EVENT_LOOP_STALL 21s in `upload_json`.** `complete_review` (an async route) wrote the corrections JSON to GCS directly on the event loop. Ranking a week of watchdog stacks by route showed the same pattern in `recover_stuck_jobs` (~260 stalls/week), the YouTube queue processor (its `_process_single_upload` ran a whole upload on the loop), `create_job_from_search` (theme prep), `_validate_audio_durations` (15s), and the sync `run_job` / generation bump inside worker triggers. 4. **False 30s "stall" in asyncio `get_debug`.** Cloud Run CPU throttling froze the whole instance mid-callback with no request in flight. The watchdog only discounted freezes where the loop was parked in `select()`. ## Changes - **Refund:** `_refund_credit_for_job` skips jobs with `edit_count > 0` (already delivered). This covers delete, cancel, fail and the 48h review auto-expiry. - **YouTube:** on 404/410 the upload restarts with a fresh session (3 attempts, with backoff). `execute(num_retries=3)` lets googleapiclient resume chunks on 5xx/429 within a session. Any other failure is queued in `youtube_upload_queue` (`reason="upload_error"`) for the existing processor, which retries up to 5 times. Admins still get a Discord warning: "queued for automatic retry". - **Event loop:** - New `backend/utils/loop_bridge.py`: `run_in_body_thread` runs sync route bodies on a dedicated 16-thread pool, so they can't exhaust the default executor the dispatched coroutines need. `run_on_loop` dispatches coroutines back to the API loop (`run_coroutine_threadsafe`, 120s timeout). - `complete_review`: its sync save runs via `to_thread`, and the render trigger stays on the loop. - `create_job_from_search`, `recover_stuck_jobs` and `retry_pending_render_jobs`: bodies run in the route-body pool. `_retrigger_parked_download` now does its writes in that thread. - `_validate_audio_durations` and the YouTube queue's `_process_single_upload` keep their async signatures but run in threads. The queue's quota and Firestore calls go through `to_thread`. - `worker_service`: the generation bump and `client.run_job` go through `to_thread`. - **Watchdog:** if the watchdog thread itself oversleeps by 2s or more while the process uses less than 25% CPU, the whole process was frozen. That time is subtracted from the stall and isn't sampled. Blocking I/O (the watchdog still wakes on time) and GIL-hogging CPU work (CPU keeps advancing) are still reported. - Docs: `API.md` (edit/refund note) and `LESSONS-LEARNED.md` (mostly-sync routes that await one trigger; throttling freezes). ## Testing - Backend unit suite: **5863 passed**, 36 skipped (`pytest backend/tests --ignore=backend/tests/emulator`). - New tests: - edited-track refund cases (delete in awaiting_review / lyrics_complete, cancel); - YouTube restart on 410, give up after 3 attempts, no restart on 403; - orchestrator queues failed uploads and warns when queueing fails; - loop-responsiveness ticker tests for each offloaded path; - `run_on_loop` result, exception and timeout-with-cancel; - `run_in_body_thread` copies context and uses its own pool; - watchdog: freeze ignored, I/O stall and CPU hog still reported, freeze subtracted from a real stall. - Local CodeRabbit: 3 cycles. Fixed the nested-executor deadlock risk, the missing `run_on_loop` timeout, the parked-download writes on the loop, and a coroutine leak. The last cycle had 0 findings. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
60687d6cd6 |
feat(theme): editable + previewed end screen; optional title/end screens (v0.268.0) (#1134)
## Summary
Randy Vild (tenant `randy-vild`) emailed: *"I have completed 3 songs but
all three songs produce the large text at the end even though I have in
my settings smaller text like in my beginning."*
Root cause: the theme editor's size/position sliders only edit
`intro.*`. The end screen has its own `end.*` layout (title 350 /
closing message 400 / yellow capitals artist in Randy's theme). The
editor neither exposed it nor previewed it ("The end screen isn't
previewed").
## Changes
- **End screen tab**: own title/artist colours, size + position sliders
(closing message, song title, artist), show/hide song title / artist
name, capitals toggles, and **Copy title card layout**. That copies the
title card's regions/colours/capitals and moves the closing message just
above the title, or below the artist if there's no room, so they never
overlap.
- **End screen preview**: `theme_preview_service.render_end_screen` uses
the same `VideoGenerator.create_end_video` as `screens_worker`. The
preview column shows the screen for the active tab plus the lyrics
frame; Advanced shows all three.
- **Leave out the title card / end screen**: optional `intro.enabled` /
`end.enabled` (absent = included; not added to the DEFAULT_*
required-field lists). The setting is honoured end to end:
- The screens worker skips the omitted screen and records
`state_data.screens_included`.
- The video worker's prerequisite check and the `/retry` ladder
(`_has_title_screen`, which also requires `screens_progress: complete`)
treat an omitted screen as done.
- The orchestrator sets the omitted screen's path to `None` and gives
the original-vocals guide an intro offset of 0.
- Encoding passes explicit `include_title_screen` /
`include_end_screen=false` to the GCE worker. Without them, its loose
`*title*.png` / `*end*.png` globs would rebuild the screen from
`style/*_background.png`.
- `LocalEncodingService.encode_lossless_mp4` builds the concat from
whichever segments exist (1–3).
- The portrait renderer drops omitted cards.
- 11 new strings × 33 locales (`endScreenHint` removed). Docs: API.md,
README.md.
## Verification
- `make test` all green: frontend 1665 tests; backend unit, integration
and emulator suites.
- New `backend/tests/test_theme_screen_omission.py`, 28 tests:
- screens worker for all 4 include combinations;
- prerequisites and the retry helper;
- orchestrator config and encoder options;
- GCE `run_encoding` with decoy theme PNGs present;
- concat for n = 1/2/3.
- Real-ffmpeg renders:
- end-screen preview (its own background; text size changes only the end
screen);
- portrait durations with screens omitted.
- Frontend: editor tests for the end tab, copy layout, region toggles
and enable switches, plus unit tests for the `copyTitleCardLayoutToEnd`
util.
- Rendered Randy's real theme locally:
- as saved, the end screen reproduces the large text;
- after Copy title card layout, it matches his title card.
- Local CodeRabbit review: 0 findings.
## Deploy notes
- The worker paths changed (`gce_encoding/`,
`local_encoding_service.py`, `karaoke_gen/`), so CI blue-green redeploys
the encoding worker.
- Existing themes are unaffected: they have no `enabled` key.
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
9d04274de7 |
feat(tenant): theme edits reach in-progress tracks + outdated-track re-render prompts (v0.267.0) (#1132)
## Why Randy Vild edited his portal theme and his next video (RVILD-0003) still had the old title card. Each job copies the theme when it's created. This one was created 30 minutes before his edit and was waiting for encoding capacity when he saved. Edits only reached jobs created afterwards, finished tracks gave no sign they were outdated, and admins saw two identical "Re-render" buttons on portal tracks. ## What - **Theme save → in-progress tracks** (`services/theme_change_service.py`): every tenant track that hasn't started rendering gets a fresh copy of the theme. Tracks whose screens may already exist (`generating_screens`…`in_review`, `render_pending_capacity`) are flagged `state_data.theme_screens_stale`. - **Render-worker divert:** a flagged job in `review_complete` goes once through the screens worker (`LYRICS_COMPLETE` + `regen_restore_status=review_complete`, the same bounce the theme re-render uses). The screens worker regenerates the screens and re-triggers the render. Doing this at render time covers every path into render (review submit, auto-approve, retry, capacity retry) and never interrupts someone mid-review. Stale `finals/*.mov` files are deleted too, so the encoder can't splice the old intro back in. - **Edit of a finished tenant track** picks up the current theme when the track's copy is outdated. - **New endpoints:** `GET /api/tenant/theme/outdated-jobs`, plus `POST /api/tenant/theme/rerender-outdated`, which is quiet (no per-track emails), capped at 20 per call, and keeps its blocking I/O off the event loop. The save response now includes `refreshed_jobs` and `outdated_job_ids`. Outdated means the theme's `style_params.json` was written after the job's `theme_applied_at` (or `created_at` if that's unset). - **UI:** after a save, the theme editor says how many in-progress tracks were updated and offers **Re-render all N**. Outdated job cards get an "Older theme" badge on Re-render. On portal cards, admins see only the portal Re-render, not the duplicate admin one. - i18n: 12 new strings, translated into all 33 locales; 3 superseded keys removed. ## Testing - `make test` passes (backend unit, emulator and e2e; frontend Jest and Playwright). - New `backend/tests/test_theme_change.py` (36 tests, 98% coverage of the new service), plus a render-worker divert test, editor dialog tests and OutputLinks tests. - Local CodeRabbit review: 1 major finding (bulk re-render blocking the event loop / unbounded), fixed. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
6c3c70654d |
feat(storage): retention purge (dry-run) and on-demand regenerate (v0.265.0) (#1125)
## Summary
Implements the approved storage-retention plan (2026-10-03): purge
regenerable files from jobs completed more than 30 days ago, and
regenerate them on demand.
### Purge (`backend/services/storage_retention.py`)
- Daily Cloud Scheduler `storage-retention-daily` calls `POST
/api/internal/storage-retention/run` (Pulumi, already applied).
- **DRY-RUN by default** (`STORAGE_RETENTION_DRY_RUN`, default true). It
writes a report to
`gs://karaoke-gen-storage-nomadkaraoke/storage-retention/reports/<ts>-dry-run.json`.
Real deletion needs the env flipped after a human review.
- Walks completed Firestore jobs. Completion time is the newest
`complete` timeline entry.
- **Keeps:** input (incl. `edited.flac`), lyrics, style, review
sessions, audio_edit, uploads, packages, analysis, screen images, the
720p final (both copies), `stems/custom_instrumental.*`,
`stems/vocals_derived.*`, and the root custom/existing instrumentals.
Unknown files are kept too.
- **Purges:** other finals, `videos/`, `previews/`, `encoded/`,
`quick/*.mp4`, `screens/*.mov`, `review-audio/`, and separated stems
(only when the input is kept in `jobs/{id}/input/`).
- **Skips:**
- finalise-only jobs
- jobs whose input audio is gone (66 old jobs)
- jobs with an in-flight edit, re-render, visibility change or
regenerate
- pending YouTube uploads
- excluded tenants (env)
- Real runs:
- batched, with a resumable cursor
- re-check eligibility atomically via a claim
- record `renders_purged_at`, `stems_purged_at` and a `storage_purge`
manifest
- drop purged `file_urls` entries
- `?job_ids=…&dry_run=false&min_age_days=0` runs a scoped real pass, for
testing. A global real purge only happens through the setting.
### Regenerate (`backend/services/regenerate_service.py`)
- `POST /api/jobs/{id}/regenerate` (owner or admin, rate-limited per job
and per user) plus a job-page **Regenerate video** button and progress
notice. There is also `POST /api/admin/jobs/{id}/regenerate`.
- GCS-only: claim, then screens, render, encode. It uses the existing
lyrics, instrumental selection and style. No brand-code allocation, no
YouTube/Dropbox/GDrive/Discord. The owner is notified when it's done.
- **Stems gate** in the screens worker (`stems_restore.py`): if
`stems_purged_at` is set, it runs the audio-separation job in restore
mode first, and the audio worker re-triggers screens afterwards. This
covers regenerate, admin/theme re-render, private→public, Edit (which
now restores stems before review) and retry.
- Public→private on a job with archived finals regenerates first, then
redistributes (no more "No finals" failure).
- Archived downloads return **410** `output_archived` with a
`regenerate_url`.
### Fixes
- Encoder: the with-backing and clean instrumental lookups no longer
fall back to another instrumental (that was a silent wrong-instrumental
bug); a missing instrumental now fails loudly.
- `existing_instrumental_gcs_path` is persisted at upload to
`jobs/{id}/custom_instrumental.<ext>`, because `uploads/` expires after
7 days. Older jobs get a resolve-and-repoint fallback in the video
worker and orchestrator.
- Bucket noncurrent-version retention reduced from 30 to 7 days (Pulumi,
applied).
## Testing
- New: `test_storage_retention.py` (108), `test_regenerate_service.py`
(66), `test_instrumental_resolution.py` (16), plus edit-flow, upload and
visibility updates. Full backend unit suite: 5731 passed.
- Frontend: OutputLinks regenerate tests and job-status helper tests
(Jest). All 33 locales translated.
- Local read-only dry-run against prod: 538 jobs eligible, about 401 GiB
purgeable.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added an option to regenerate archived or missing video outputs from a
completed job. Existing publishing links remain unchanged, and job
progress is shown while processing.
* Added automatic restoration of archived audio stems when needed for
regeneration or job edits.
* **Improvements**
* Added scheduled cleanup of eligible files from older completed jobs
while retaining key source and review assets.
* Improved recovery when requested downloads have been archived by
offering a regeneration option.
* Improved handling of saved instrumentals so they remain available for
future processing.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
||
|
|
4e43449559 |
fix(backend): stop API event-loop freezes + identify users in degradation telemetry (v0.264.0) (#1120)
## Summary The "Reconnecting" pill and the "servers unavailable" banner (164 / 49 events, 2026-09-19 → 10-02) were mostly caused by the single API instance's **event loop freezing for 20–45s**. Every request on the instance, `/api/health` included, completed at the same moment. The full evidence and plan are in `docs/archive/2026-10-03-backend-loop-freezes-telemetry-plan.md`. ### What froze the loop, and the fix for each - **Screens worker:** it ran as a post-response `async def` BackgroundTask doing 4K PIL renders, GCS transfers and ffmpeg transcodes on the loop. Under `--cpu-throttling` it was also CPU-starved. **Fix:** it now runs inline in its Cloud Tasks request (CPU stays allocated, failures still return 200 so Cloud Tasks doesn't double-run it), with the heavy steps in `asyncio.to_thread`. Also offloaded: style load, timing signals, pre-apply. - **Matplotlib font cache:** it was rebuilt on every new instance (17/28 episodes). **Fix:** it's pre-built in `Dockerfile.base` (`MPLCONFIGDIR=/opt/mplconfig`), and pyplot is imported lazily. - **Hourly stale-review cron:** an `async def` with zero awaits doing ≤1500 reads and SMTP sends. **Fix:** it runs in a thread. - **166 `async def` routes that never awaited** (sync Firestore/GCS/email/Gemini on the loop, e.g. magic-link verify). **Fix:** they're now plain `def`, so they run in FastAPI's threadpool. The new AST guard `test_no_blocking_async_routes.py` fails CI on new ones. It also flags sync routes that call `asyncio.create_task`/`get_running_loop`; that caught `confirm_duration` and `trigger_youtube_queue_processing`, which stay async. ### Safety net - `backend/services/loop_watchdog.py`: a heartbeat plus a daemon thread. Any stall over 1s logs `EVENT_LOOP_STALL` with the loop thread's stack and a `culprit=file:line`. Stalls of 5s or more are written to `client_events` as `server_loop_stall`. - Pulumi: the log metric `backend/event_loop_stalls` plus an alert on ≥2 stalls of ≥10s in 30 min. **This still needs a local `pulumi up`.** Pulumi is never applied by CI, and the admin gcloud login has expired. ### Telemetry gaps fixed - **Identity:** `user_email` used to be hard-coded to null. The frontend now sends its session token, and the server resolves `user_email`, `is_admin`, `is_internal` and `is_test`. A client-sent email is ignored. - **New fields:** `device_fingerprint` (the existing FingerprintJS visitorId), `tab_id`, `episode_id`, and `tenant`. - **New event:** `banner_recovered`, carrying `duration_ms` and `peak_status`. - **Report:** `scripts/client_events_report.py` summarises the data, excluding admin/internal/test accounts by default. ## Testing - `make test` passes: backend 5529, plus the emulator suite, karaoke_gen unit tests, and frontend Jest 1623. - New tests: - Watchdog: a deterministic state machine plus detection of a real blocking call. - Event-loop responsiveness: a ticker keeps ticking while the heavy steps run. - Screens inline and failure semantics. - `client_events` identity and fields. - The AST guard. - The report summary. - Frontend reporter and banner episode tests. - Local CodeRabbit CLI review: 0 findings. 🤖 Generated with [Claude Code](https://claude.com/claude-code) @coderabbitai ignore --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0fa10e4330 |
feat(frontend): friendly cold-start waking state and backend pre-warm (v0.263.0) (#1110)
## Summary `karaoke-backend` now runs with min-instances 0, so a request that hits a cold backend waits ~15-20s (measured 18.4s) while Cloud Run boots an instance. Cloud Run queues requests during that window, so the health probe times out exactly as it does during a recycle, and users saw the amber "Reconnecting…" / "having trouble reaching our servers" banner. This replaces that with a calm cold-start experience and pre-warms the backend. ### Behaviour (`frontend/lib/backend-status.ts`) - New `waking` status. **Cold-start rule:** if no backend response succeeded in the `REACHABLE_FRESH_MS` (10 min) before the oldest stalled read began (fresh page load, or a tab idle long enough for scale-to-zero), a stall is treated as a cold start: - probe `/api/health` after `COLD_PROBE_AFTER_MS` (1s) instead of 10s, so a normal warm load (reads finish in <1s) never sends an extra probe; - once the read is ≥3s old **and** the probe failed (4s timeout), show the waking card (~5s into a cold start); - it clears the instant reads complete. - **Escalation:** past `WAKING_ESCALATE_MS` (60s) the normal reconnecting/unavailable logic takes over. A stall that starts within 10 min of a successful response is a recycle/outage, so the existing banner applies unchanged. - The probe still gates everything: a warm backend with a merely slow endpoint shows nothing. - `apiFetch` marks the backend reachable on every non-transient response, for any method. GET hard timeout (45s) and retry logic are unchanged and already ride out a ~20-30s cold start (the queued request simply completes). ### Pre-warm - `prewarmBackend()`: an untracked, fire-and-forget `GET /api/health` (`keepalive`, `cache: no-store`, no custom headers, so no CORS preflight). It runs on app load via the global `BackendStatusBanner` (all `[locale]` pages) and the admin layout, and again when a tab becomes visible after >10 min hidden. It's skipped if the backend answered in the last 10 min. Success marks the backend reachable. - Companion PR nomadkaraoke/public-website#31 pre-warms from the marketing homepage Hero (no-cors, idle-time). ### UI - Waking card: neutral card colours plus a tenant-primary spinner. "Starting up our karaoke servers…" / "This usually takes about 20 seconds. Thanks for your patience!". No warning colours and no dismiss button. - New `banner_waking` degradation event, added to the frontend type and the backend `client-events` allow-list. It includes `since_reachable_ms`. - Strings translated for all 33 locales (Gemini Developer API path from #1108). ## Testing - Store unit tests with fake timers. Cold start → waking (~5s) → clears. Fast cold read shows nothing. Warm slow endpoint shows nothing. Prolonged stall → `unavailable` after 60s. Previously reachable then stall → `reconnecting`. Reachable >10 min ago → waking. Pre-warm de-dupe, failure and visibility re-warm are covered. - apiFetch tests: a fresh session hitting a cold backend shows `waking` (never the warning) and the GET completes at 18s. The pre-warm request shape (no headers, keepalive) is checked. - Backend: `banner_waking` accepted by `/api/client-events`. - `make test` passes. - Real browser check: `next dev` against a local stub that holds every request for 18s from the first hit. The waking card appears by ~7s, the dashboard shows "Loading jobs…" (no error or empty state) and everything resolves at 18.3s. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a startup status banner for cold starts, with an estimate that startup usually takes about 20 seconds. The banner is available in multiple languages. * The app now checks backend availability when needed and after a tab has been hidden for an extended period, helping requests recover from cold starts. * **Documentation** * Updated API telemetry documentation to include the new startup event. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
8490ab75eb |
chore(gemini): move every Gemini call from Vertex AI to the Gemini Developer API (#1108)
## Summary Moves **every** Gemini call in gen off **Vertex AI** (billed to the `nomadkaraoke` GCP project, ~$85/mo) onto the **Gemini Developer API** using the `gemini-api-key` Secret Manager secret (an AI Studio key billed outside this project). Once deployed and verified, `aiplatform.googleapis.com` / `generativelanguage.googleapis.com` get disabled in nomadkaraoke. ### Key injection - `backend/services/gemini_client.get_genai_client(timeout_ms)` — key from `GEMINI_API_KEY`, else Secret Manager `gemini-api-key` via `settings.get_secret` (runtime SA already has project-level `secretAccessor`). No `project`/`location`. - Service: CI `--set-secrets ... GEMINI_API_KEY=gemini-api-key:latest`. - Jobs: Pulumi secret env on `lyrics-transcription-job`, `video-encoding-job` (renders + translations), `nomad-error-monitor` — **already applied** (targeted `pulumi up`, only that env added). Other jobs fall back to Secret Manager. - Local scripts (`frontend/scripts/translate.py`, `scripts/measure_autocorrect_cost.py`, `scripts/test_agentic_correction.py`): env, else `gcloud secrets versions access` (in-process only). - **Rotation:** add a new secret version; new instances / job executions pick it up. No code change. ### Degradation (quota / billing / invalid key; per-minute 429s stay retryable) | Caller | Behaviour | |---|---| | auto-correct | non-retryable 503 → compare mode drops the Gemini leg, keeps Opus | | match-judge / free-text / parse-titles / tenant bulk | existing fallbacks (no suggestion / blanks / regex) | | credit eval | `pending_review` (fail-closed, unchanged) | | lyrics translation | no retry; render proceeds without translation, status recorded | | custom lyrics | clean 503 "temporarily unavailable…" (+ 5-min timeout so it can't hang) | | error-monitor LLM | no retry; alerts sent without AI summary | | translate.py | exit 2 "Gemini quota/credit exhausted — top up AI Studio or rotate gemini-api-key" | `note_gemini_failure()` also sends one Discord ops alert per 6h (Firestore-throttled across instances via new `ops_alerts.send_throttled_ops_alert`). ### Infra - Pulumi: removes `roles/aiplatform.user` from `karaoke-backend` — **will be applied after this deploys and Vertex traffic is confirmed at zero**. - Agentic correction default model `gemini/gemini-3.8-flash` (`vertexai/` prefix still accepted, routed to the Developer API). - Version 0.261.1 → 0.262.0. Docs: CLAUDE.md, LESSONS-LEARNED, README, ARCHITECTURE, API, etc. ## Testing - New: `backend/tests/services/test_backend_gemini_client.py` (classifier, key resolution, no-Vertex client kwargs, alert throttle, per-caller degradation), `tests/unit/lyrics_transcriber/test_agentic_gemini_provider.py`, `frontend/scripts/test_gemini_client.py`, lyrics-translation quota test. - `pytest backend/tests --ignore=backend/tests/emulator` — all pass; `pytest tests/unit` — 2021 passed. - Verified `gemini-3.8-flash` is listed on the Developer API and answered a 5-token test call with the key. - `translate.py --dry-run` on `frontend/messages` and `backend/translations` OK. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Updates** * AI-powered lyric translation, custom lyrics, auto-correction, title parsing, match judging, and related features now use the Gemini Developer API. * AI features use the configured Gemini API key. Quota, billing, or key issues may show an unavailable message or use an existing fallback instead of repeated attempts. * **Documentation** * Updated product, API, setup, and recovery guidance to reflect Gemini API key configuration and usage. * Updated guidance for handling AI service availability and quota issues. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
f8d0febcf5 |
Translated lyrics beneath the karaoke lyrics for language learners (v0.260.0) (#1101)
## Summary
New job-submission option: **Add translated lyrics** → pick a language
(33 UI locales) → the karaoke videos show an AI translation of each
lyric line in smaller text beneath the synced lyrics, so language
learners can sing songs in another language while learning what the
words mean.
- **Job creation**: toggle + language picker (default = UI language) on
the Customize step, with a **live preview** of a real rendered frame
(`GET /api/themes/translation-preview?language=xx`; pre-generated sample
translations, no LLM call at preview time). Any job, public or private,
free for now. `translation_language` accepted by every creation route;
admin can edit it.
- **Post-review translation**: right after the job enters
RENDERING_VIDEO, the render worker translates the *final reviewed*
lyrics line by line with Gemini
(`backend/services/lyrics_translation.py`, `LYRICS_TRANSLATION_MODEL`,
GCS-cached so theme/admin re-renders are free) into
`jobs/{id}/lyrics/translations.json`. Not editable. A failure never
blocks the render (renders without translations, recorded in
`state_data.lyrics_translation`); a song already in the target language
gets no row.
- **Rendering** (landscape + portrait, GCE worker + local fallback):
`LyricsSegment.translation`; when present, slots get taller and screens
show 3 lines; a `Karaoke.Translation` style (unsung colour, ~½ size)
draws one static event beneath each line, sanitized, wrapped/scaled to
fit. Jobs without translations render identically.
- **RTL + LTR together**: the translation is its own ASS event with auto
paragraph direction, so e.g. a Hebrew song keeps its right-to-left
highlight with an English translation beneath (and vice versa). Long
lines split by the resizer share their translation across the pieces
(grapheme-safe for Thai/CJK).
Design: `docs/archive/2026-10-01-translated-lyrics-plan.md`.
## Testing
- `make test` all green (root 2014, backend 5416, emulator 150, frontend
1583).
- New: `tests/unit/lyrics_transcriber/output/test_translated_lyrics.py`
(layout/positions/style/sanitizing/fit/mixed RTL-LTR/Thai/portrait),
`test_translated_lyrics_render.py` (libass pixel tests incl.
Hebrew/Arabic lyrics with English/Arabic translations and CJK),
`backend/tests/test_lyrics_translation.py`,
`backend/tests/test_render_video_worker_translations.py`, GCE endpoint
test, frontend CustomizeStep/GuidedJobFlow/api tests, prod E2E
`e2e/production/translated-lyrics.spec.ts`.
- `scripts/run-render-tests-linux.sh`: 135 passed on Debian ffmpeg AND
prod's static ffmpeg (no skips).
- Real-data check: job 5710831e (Hebrew) translated to English by Gemini
and rendered with the nomad theme — RTL highlight intact, English
beneath, split line shares translation.
Local CodeRabbit CLI unavailable in this workspace (no seat), so leaving
this PR for the CodeRabbit bot; an independent local review was done and
its findings fixed in the second commit.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added an opt-in translated-lyrics option to job setup, with language
selection and a preview.
* Translations are generated after lyric review and appear beneath
lyrics in landscape and portrait videos, with text layout for multiple
languages and writing directions.
* Translation settings can be changed or cleared by admins.
* **Bug Fixes**
* If translation fails or is unavailable, video rendering continues
without translated lyrics.
* **Documentation**
* Updated API and product documentation with translated-lyrics setup and
behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
||
|
|
f89de165cc |
Admin: re-render any completed job without review (v0.257.0) (#1096)
## Summary
An admin-only **Re-render** button on completed jobs in the `/app` jobs
list. It regenerates title/end cards, the karaoke video, all encoded
formats, packaging and distribution from the job's existing reviewed
lyrics, instrumental selection and style snapshot, **with no
lyrics/instrumental review**. Motivating case: re-rendering
already-delivered videos after a renderer fix (RTL/Hebrew, v0.255.0).
`POST /api/admin/jobs/{job_id}/rerender` with body `{"notify_customer":
false}`.
**Behaviour (decided with Andrew)**
- Works for consumer and tenant jobs, public or private. Uses the job's
existing style; no theme re-snapshot.
- **Published outputs are deleted up front, like Edit (YouTube, GDrive,
Dropbox), but the brand code is kept, not recycled.** The pipeline
re-publishes under the same NOMAD-xxxx. YouTube gets a new upload and
therefore a new URL.
- Outputs are only deleted for destinations the re-render will actually
re-publish (`plan_republish` uses the same conditions as the video
worker). Anything else is kept, with a warning shown to the admin.
- **No customer email/push/Discord by default.** The confirm dialog has
an "Email the customer when done" checkbox. Already-notified
requests-board voters aren't re-emailed; voter emails still owed from
the original publish do go out.
- A failed admin re-render is set to FAILED with an in-progress marker
(scoped to `review_token`, so it can't leak into later runs). It can be
retried by **admins only**; `/retry` returns 403 to customers while the
marker is active.
- **Mutual exclusion with the YouTube upload queue:** the claim and the
queue's `mark_processing` are Firestore transactions that each read both
documents. A re-render is refused with 409 while an upload is in flight.
A queued or failed deferred upload is cancelled atomically when YouTube
will be re-published. The queue never starts an upload for a job that is
mid re-render.
- After a successful upload, the queue never re-queues an entry, so a
later error can't cause a duplicate upload.
- Admin re-render completions don't increment the customer's
completed-jobs count.
**Refactors**
- New `backend/services/published_outputs_cleanup.py`, shared by Edit,
admin delete-outputs and the re-render.
- The Dropbox folder delete now uses the uploader's sanitised name and
also tries the legacy raw name. This is a deliberate fix for titles with
special characters.
- Remaining duplicate copies (`jobs.py` ~2440,
`visibility_change_service`) are listed as follow-ups in the design doc.
## Testing
- 173 tests in `test_admin_rerender.py` +
`test_published_outputs_cleanup.py`, built through three local
code-review rounds. They cover:
- claim and validation
- retry gating
- marker scoping
- publish planning (each mismatch case)
- queue exclusion in both directions, no starvation, and that the
re-render's own later upload is never cancelled
- the no-re-queue-after-upload rule
- notification gating on both pipelines
- community voter cases plus reconcile
- restart atomicity
- legacy Dropbox naming
- tenant re-render and Edit unchanged
- Frontend Jest: button visibility (admin vs non-admin, complete vs
not), dialog, payload, and warnings toast.
- `make test`: backend 1972 + 5372 + 150, frontend 1506.
- i18n: new strings in all 33 locales (frontend) plus backend
translations.
Docs: `docs/API.md`,
`docs/archive/2026-10-01-admin-rerender-any-job.md`.
No local CodeRabbit review (the CLI has no seat for this org), so this
is left open for the CodeRabbit bot. Three local code-review rounds ran;
their findings are addressed in the fix commits.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Administrators can re-render eligible completed tracks without another
review, retaining existing track settings and brand code.
* Choose whether to notify the customer; notifications are off by
default. Published outputs are replaced when applicable, and warnings
identify outputs that remain unchanged.
* Administrators can resume eligible failed or cancelled re-renders.
Re-render controls and status messages are available in multiple
languages.
* **Bug Fixes**
* Improved cleanup and handling of published files and queued YouTube
uploads during re-renders, with safer retries and clearer failure
reporting.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
||
|
|
e2f7dbd13c |
Detect sung stretches with no transcribed lyrics (shadow) + backfill endpoint (v0.256.0) (#1095)
## Summary
Transcription can silently **drop whole sung lines**. On job 5710831e
(Hebrew), the words stop at 20.66s and resume at 32.40s while the lead
vocal is ~96% active, and both reference sources have three pre-chorus
lines there. `SectionDetector` (gap ≥ 10s, no audio check) then rendered
"♪ INSTRUMENTAL ♪" over the singing. Nothing caught it:
- the scorer's anchor coverage measures how much of the *transcribed*
text matches the reference, so omissions don't lower it (98.8% here);
- `timing_check` G3 (`max_unclaimed_run_s`) does fire on this job but is
shadow-only and only computed for AUTO-bound jobs;
- the review UI shows nothing at the gap.
This PR is **phase 1: a shadow signal plus server-side backfill for
calibration**. It changes no verdicts or rendering.
- `backend/services/auto_approval/vocal_gaps.py`: for every
transcription gap ≥ 3s, records the lead-vocal active fraction, the
longest breath-bridged vocal run (with a 1.5s allowance for a held note
carried over from the previous word), and the reference lines between
the neighbouring words' reference positions. Section headers are dropped
and ±1-word anchor drift at line boundaries is snapped. `suspect` = run
≥ 3s.
- The executor analyzes every job once stems exist (any verdict), off
the event loop, and stores `state_data.vocal_gaps` keyed by an
`input_key` fingerprint so resets/edits recompute. A summary goes in
`processing_metadata.auto_approval.vocal_gaps`.
- `POST /api/internal/jobs/{id}/vocal-gaps[?dry_run=true]` (admin) plus
`scripts/audit_vocal_gaps.py` for backfill/calibration. Cloud Run does
the stem work.
**On the real job:** one suspect gap, 20.66–32.40s (10.26s run after the
allowance), whose reference lines are exactly the three missing Genius
lines. The three genuine instrumentals have ≤ 0.16s runs.
## Testing
- `test_vocal_gaps.py` (25 tests, synthesized audio + synthetic
correction data):
- dropped lines vs instrumental vs ad-lib vs held note
- a short dropped line diluted by a long gap
- breath bridging
- silent stem = error
- malformed reference data
- reference snapping (off-by-one, single-word line, headers, reversed
chorus mapping)
- Executor tests: stored without changing the outcome; analyzed for
review-bound jobs; cache hit only for the same `input_key`, stale
recompute; pending audio / no stem / error / exception all non-fatal;
backfill uses the reviewed lyrics + original references; dry run.
- Internal endpoint tests (store / dry_run / admin).
- Full backend suite: 5199 passed.
- A local code review's findings were addressed in the second commit:
event loop, stale cache, silent stem, held notes, reference robustness,
audit ordering.
## Next (after deploy)
Run the audit → hand-check suspects → choose thresholds → phase 2:
scorer REVIEW gate with an explicit reason, no fake INSTRUMENTAL over
vocals, review-UI marker with one-click insert of the reference lines.
Also noted: anchor reference ids look shifted by one word at `[Section]`
header boundaries. Separate issue, documented in the plan doc.
No local CodeRabbit review (the CLI has no seat for this org), so this
is left open for the CodeRabbit bot.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added vocal-gap analysis to identify stretches of missing lyrics that
overlap with lead vocals. Results are saved with job data and can be
requested through an admin-only internal API.
* Analysis also runs automatically during auto-approval when vocal stems
are available. It does not change job status or approval decisions.
* **Documentation**
* Documented the internal API, its dry-run option, and the analysis
result statuses.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
||
|
|
e15a485007 |
Re-render: refresh tenant Dropbox archive in place instead of rejecting (v0.254.1) (#1093)
## Summary Follow-up to #1091, found by testing it in prod. randy-vild jobs are archived to Dropbox under a brand code (`RVILD-0001 - Randy Vild - What Goes Up`). The "never distributed" guard therefore rejected the very jobs the Re-render button is for (`POST /api/jobs/2579a1ea/rerender` → 400 "published to YouTube/Dropbox/Drive"). - Dropbox-archived tenant jobs can now be re-rendered. The re-render stores the job's `brand_code` in `state_data.theme_rerender`, and the orchestrator config and native distribution use it as `keep_brand_code`. The same folder is overwritten (Dropbox `WriteMode.overwrite`) and no new code is allocated. - The code is scoped to the re-render marker (cleared on success), not written to the job's `keep_brand_code` field. That way a later Edit, which recycles the brand code, can't reuse it. - YouTube/GDrive-published jobs are still rejected. The frontend hides the button only when a YouTube link shows. ## Testing - Backend 5158 passed, 36 skipped; `test_theme_rerender.py` 35 cases, including Dropbox-archived jobs being re-renderable, the marker recording the brand code, and the orchestrator keeping it. - Frontend `OutputLinks.test.tsx` 28 passed. - CodeRabbit CLI isn't available on this org, and the bot has been rate-limited. This is a small change on top of the reviewed #1091. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
9085a3005e |
feat(tenants): configure Dropbox output folder in tenant setup (v0.254.0) (#1092)
## Summary The Randy Vild tenant's completion email said "Here's the dropbox folder…" then `[Dropbox URL not available]`: the tenant was created with the (optional) Dropbox path left blank, so every job was download-only. The admin form already had the fields, but nothing guided, validated, or provisioned them. ### Changes - **Tenant create/update** (`tenant_admin_service`): `dropbox_path` + `brand_prefix` must be set together (the worker silently skips upload otherwise); path normalized, prefix uppercased/validated (2-12 alnum) and rejected if it's the consumer default (`NOMAD`/`NOMADNP`) or another tenant's (brand-code counters are keyed by prefix alone). The Dropbox folder is **created up front**; `features.dropbox_upload` always follows the path. - **DropboxService**: `ensure_folder()` (idempotent); `list_folders()` treats a missing path as empty — previously the *first* brand-code allocation for a new tenant would raise `not_found`. - **Admin UI** (`/admin/tenants`): pre-fills `/MediaUnsynced/Karaoke/Tracks-<Name>` + prefix from the name (Vocal Star→VSTAR, Singa→SINGA, Randy Vild→RVILD), blocks half-set delivery, warns that download-only tenants get no email link; table shows `PREFIX → path`. - Update route maps conflict→409, Dropbox provisioning failure→502. ### Prod data fix (already applied, outside this PR) - `randy-vild` tenant → `/MediaUnsynced/Karaoke/Tracks-RandyVild`, `RVILD`; `brand_code_counters/RVILD` seeded at 3. - 10 in-review Randy Vild jobs backfilled with `dropbox_path`/`brand_prefix` (jobs snapshot these at creation). - 2 completed jobs copied into `RVILD-0001 - Randy Vild - What Goes Up` / `RVILD-0002 - Randy Vild - Embers in the wind` (24 files each via Dropbox `save_url` from signed GCS URLs), `state_data.brand_code`/`dropbox_link` set. ## Testing - `make test`: backend 5121 + 1846 + 150 passed, frontend 1476 passed - New unit tests: tenant delivery validation/folder creation/prefix conflicts, DropboxService ensure_folder + missing-path listing (real SDK error types), admin page pre-fill + half-set block + download-only warning, `suggestDelivery`/`deliveryError`. - Local CodeRabbit review: 2 minor findings, both fixed. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
31a8838b07 |
Tenant: re-render tracks with the current theme; edited tenant tracks keep their downloads (v0.253.0) (#1091)
## Summary
- **Bug: finished tenant track showed no download buttons** (randy-vild
job 2579a1ea). The Edit flow sets `outputs_deleted_at`, and it was only
cleared when the re-render uploaded to YouTube/Dropbox/GDrive. Tenant
jobs never distribute, so every download stayed hidden after the
re-render. A customer's private job (6452888e) had the same problem for
a month. Every successful render now clears the flag
(`video_worker.py`). Both prod jobs were cleared by hand.
- **New: "Re-render" button** on finished tenant-portal tracks (`POST
/api/jobs/{id}/rerender`). It re-applies the tenant's **current** theme
and rebuilds the title/end screens, lyrics video, all encodes and
CDG/TXT. It reuses the reviewed lyrics and instrumental selection, so
there's no review step. Same mechanism as the private→public visibility
flow (`regen_restore_status=review_complete` → screens → render →
encode). Owner (still on the tenant allowlist) or admin. Free. The theme
editor now points to the button.
- **Encoding-worker cache:** the worker caches finished jobs by id, so a
re-run submitted while the VM was still up could get the previous render
or encode back. Render and encode now submit as
`{job_id}_g{worker_generation}`.
- **Old title/end cards:** the encoder's `**/*Title*.mov` lookup would
have picked up the old `finals/… (Title).mov`. The re-render now deletes
the MOVs in `finals/`.
- **Failed re-render:** it can be re-run. Retry re-runs it if the new
screens weren't made yet, and otherwise resumes from render/video as
usual.
## Testing
- Backend: 5128 passed, 36 skipped. New `test_theme_rerender.py` (31
cases: validation, claim/conflict, artifact cleanup incl. finals MOVs,
trigger rollback, route auth/allowlist, retry branch, generation-keyed
ids) and `test_video_worker_outputs_deleted_clear.py` (fails without the
fix).
- Frontend: `OutputLinks.test.tsx` 28 passed (button visibility, confirm
→ API → refresh). Strings translated to all 33 locales.
- Review: CodeRabbit CLI isn't available on this org. An independent
reviewer agent went over the diff: 1 must-fix (stale finals MOVs) fixed;
the failed-re-render recovery fixed.
## Not done (follow-ups)
- Re-render re-sends the completion email and push notification (same as
the visibility flow).
- No per-tenant rate limit. One re-render per job at a time is enforced
by the status claim.
- The status briefly shows "Lyrics ready" before "Generating screens".
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
||
|
|
c831bd5e89 |
feat(uploads): line up mismatched user instrumentals + tenant bulk upload modal (v0.251.0) (#1087)
## Summary
A Randy Vild tenant bulk upload lost 3 of 10 tracks to the 0.5s
mix/instrumental duration check, and the bulk UI showed only
"Submitting…" while uploading. All three rejected tracks were valid
edits of the same song:
| Track | Difference | Now |
|---|---|---|
| Keep Smiling | instrumental has 2.97s extra lead-in | start trimmed →
matches mix exactly |
| Simulation | instrumental 6.5s longer outro | cut at mix end with 2s
fade |
| Faithful and True | instrumental stops 4.45s early | end padded with
silence |
## Changes
**Backend: conform instead of cancel**
(`backend/services/instrumental_conform.py`, `uploads-complete`)
- On a >0.5s mismatch, the start offset is found by FFT
cross-correlation (reusing `derived_vocals.find_offset`) at **25% and
75%** of the song, searching ±15s. The start is then trimmed or padded,
and the end is faded and cut or padded to the mix length. All of this is
one ffmpeg pass on the original file, keeping its sample rate and bit
depth, and written as FLAC.
- Stored at `uploads/{job}/conformed/existing_instrumental.flac`, which
becomes `existing_instrumental_gcs_path`. The original upload is kept,
and details go in `state_data.instrumental_conformed`.
- Still rejected (job cancelled and refunded, 400):
- correlation < 0.3 (wrong file; real pairs score 0.78–0.90, wrong pairs
about 0.01);
- offsets at 25% and 75% differ by more than 20 ms (edited differently
mid-song).
- Conforming runs fully off the event loop, with at most 2 at a time per
instance. GCS or ffmpeg failures return a retryable 503 instead of
cancelling the job.
- The orchestrator now stages the recorded
`existing_instrumental_gcs_path` for the GCE encoder, so the conformed
copy is what gets rendered.
**Frontend**
- Tenant bulk shows `UploadProgressModal` for the whole batch: total
bytes, speed and ETA, tracks submitted and failed, and an offline or
retrying notice. The modal gained optional `title`, `detail` and
`notice` props and uses the tenant's primary colour.
- Bulk retry bug:
- **Before:** a server-cancelled job kept its ID. The retry's 400 ("not
pending") was read as "already processing", so the row showed
**Submitted** for a dead job.
- **Now:** any 400 checks the job's status. A processing job counts as
done. A cancelled or failed job is dropped, so the retry creates a fresh
job. Otherwise the row keeps the job for retry, so no duplicate is
created.
- The single-track tenant form and the own-instrumental field now check
only the size cap in the browser; the server handles length differences.
## Testing
- Backend: 5086 passed (new: synthetic lead-in, long/short outro, late
start, wrong-file and mid-song-edit cases; route and helper guards;
orchestrator staging). Frontend jest: 1459 passed (new: batch modal
progress and offline notice, failed rows, retry and no-duplicate cases).
- Ran `conform_instrumental` on Randy's real files:
- offsets at 25% and 75% agree to the sample;
- output is exactly the mix length at 24-bit / 44.1 kHz;
- after conforming, each file lines up with its mix at 0 offset;
- a wrong pairing is rejected (correlation 0.02).
- Real frontend (`next dev`, randy-vild tenant, mocked API, slowed
upload server), 8 tracks / 192 MB: the modal showed progress, track
counts, speed and ETA, then closed onto the "All tracks submitted"
screen.
- Review: an independent reviewer agent found 1 must-fix and 4
should-fix issues, all addressed in the last commit. The CodeRabbit CLI
has no seat here, so this PR is left open to the CodeRabbit bot.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Uploaded instrumentals with different durations can now be aligned and
adjusted to match the mix when the audio matches.
* Bulk uploads now display overall progress, completed and failed track
counts, connection updates, and retry guidance.
* Failed bulk-upload tracks can be retried, with clearer handling for
submissions already in progress.
* **Bug Fixes**
* Instrumental duration checks now occur during upload processing
instead of blocking the upload in the browser. Files that cannot be
matched are reported as a mismatch.
* **Documentation**
* Updated upload instructions and related messages across supported
languages.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
||
|
|
f4eca80551 |
feat(audio-download): stalled torrent → 'Keep trying?' prompt (v0.250.0) (#1085)
## Why Job 64be8a87 (a Fiverr order): the only seeder of a 1-seeder RED single was offline. The torrent sat at **0 peers through 5 re-announces for 10 min**, flacfetch aborted, the job showed a raw error, and a Discord "Job failed" alert fired even though a Cloud Run auto-retry was already queued. That retry got the file in 17s. Andrew's ask: 10 min is too short, and a stalled download should explain itself and offer "Keep trying?" for up to another hour. ## What - **Longer first wait:** 20 min with no progress (was 10), sent to flacfetch per request as `max_stall_seconds` (needs flacfetch v0.31.0, nomadkaraoke/flacfetch#51, already deployed to flacup). The worker's wait and the Cloud Run **per-execution timeout override** come from the same budget (`backend/services/audio_download_limits.py`), so the job's Pulumi config is untouched. - **Stall → user prompt, no silent retry:** flacfetch now reports `error_code: torrent_stalled` (older versions are detected from the error text). The worker fails the job with `error_details = {stage: audio_download, code: audio_download_stalled, stall_minutes, keep_trying}` and exits 0, so Cloud Run doesn't retry the same torrent. Other download failures keep their auto-retries. - **Job card:** `StalledDownloadNotice` replaces the raw error with an amber explanation and two buttons: - **Keep trying (up to 1 hour)**: `POST /api/jobs/{id}/retry {"keep_trying": true}` (the body is optional and backward compatible). This gives a 3600s stall budget and a 1h25m execution timeout. - **Choose different audio**: new `POST /api/jobs/{id}/choose-different-audio` (owner-checked). It moves FAILED to `awaiting_audio_selection` (new transition) using the saved search results and opens the Select Audio dialog. - The generic Retry button is hidden for stalled jobs. `error_details.{code,stage,stall_minutes,keep_trying}` were added to the summary fields. Strings are translated into all 33 locales. - **Heartbeat:** while waiting on a torrent, the worker writes `state_data.audio_download_progress` (progress, peers) every 2 min. Without this, recover-stuck-jobs (which parks a job with no update for 10 min) would park a live 20–60 min wait and start a duplicate download. - **Ops alert:** the alert is skipped while a Cloud Run auto-retry is pending (the final attempt clears the marker and alerts then). Stall alerts get a 🟠 "Audio download stalled" headline. - Docs updated: API.md, TROUBLESHOOTING.md, ARCHITECTURE.md. ## Tests - `make test`: backend 1846 + 5078 + 150 passed; frontend 1451 passed. - New tests: - `test_audio_download_limits.py`, `test_routes_jobs_stalled_download.py` (real HTTP via TestClient), `test_worker_service_audio_download_timeout.py`. - Stall, keep-trying, heartbeat and `main()` exit-code cases in `test_audio_download_worker.py`. - Stall detection and payload in `test_flacfetch_client.py`; retry-pending suppression in `test_ops_alerts.py`. - Frontend: `StalledDownloadNotice.test.tsx`, plus JobCard/JobActions cases. ## Notes - Credits are unchanged: a failed job is still auto-refunded and retries don't re-charge. That's existing behaviour for every retry. - A local CodeRabbit review couldn't run (the CLI seat is on another org), so this PR has no ignore line and the GitHub bot will review it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Failed audio downloads now offer options to keep trying for up to an hour or choose a different audio source. * Retry progress and clearer stalled-download guidance are available in the job view, with messages translated across supported languages. * **Bug Fixes** * Stalled downloads can be recovered without triggering unnecessary automatic retries or duplicate failure alerts. * Choosing different audio returns eligible failed jobs to audio selection. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
80f6d34007 |
feat(tenant): self-service theme editor with exact server-rendered previews (#1080)
## Summary Tenant portal users (anyone on the tenant's allowlist) can now edit their tenant's theme from **Theme & style** in the user menu. - **Guided form:** - backgrounds (upload or solid colour) for the title card, lyrics video and end screen - font (bundled or uploaded TTF/OTF) - title and artist colours, positions and sizes - sung, upcoming and outline lyric colours - lyric size, top position and max line length - the end-screen message - **Advanced:** validated full-theme JSON. - **Exact previews:** the title card and a lyrics-video frame (first line half-sung) are rendered **by the production renderers** from the unsaved draft, in about 2 s, debounced. The plan is in `docs/archive/2026-09-29-tenant-theme-editor-plan.md`. - **Save** applies to jobs created afterwards. ## Backend - New router: `GET/PUT /api/tenant/theme`, `POST /api/tenant/theme/assets`, `POST /api/tenant/theme/preview`. - **Membership guard:** `X-Tenant-ID` is client-controlled, so every call re-checks the tenant allowlist (admins always pass). Non-members and spoofed headers get 403. The tenant bulk router only checks that the tenant exists, so this pattern wasn't copied from there. - `theme_preview_service`: - title card via `VideoGenerator.create_title_video(duration=0)` - karaoke frame via `SegmentResizer` → `SubtitlesGenerator.generate_ass` → one ffmpeg frame, using `setpts` to jump straight to time T and scaling inside the same pass (11.8 s → 0.66 s) - bounded concurrency, and a cache for identical drafts - `tenant_theme_service`, stricter than the admin console: - asset fields must be basenames that exist in the theme (no paths or `gs://`) - bounds, colours and regions are validated - **font consistency with the render pipeline:** the single `font` asset is applied everywhere, `karaoke.font` is set to the TTF family name libass matches, and a chosen bundled font is copied into the theme's assets - required ASS keys are filled in - uploads are content-addressed, so they never overwrite an asset that live jobs use - tenants can't change their config or allowlist through this API ## Frontend - `components/tenant-theme/TenantThemeEditorDialog` and `theme-utils`. - `tenantThemeApi` in `lib/api.ts`. - A menu item in `AuthStatus`, shown only on tenant portals. - `tenantTheme.*` strings in all 33 locales. ## Tests - **pytest:** - real renders (the background colour appears in the frames; only the karaoke frame changes when the sung colour changes; caching) - sanitisation (paths, un-uploaded assets, bounds, colours, regions, sections) - uploads (content addressing, `no-store`, rejecting non-images and bad fonts) - save (font family, bundled font copy, no config changes) - the route guard for spoofed, other-tenant, inactive and no-tenant cases, on every endpoint - **Jest:** load, debounced preview, hex→rgba conversion, upload, region sliders, Advanced JSON validation, save, discard, the unsaved-changes guard, and preview errors. - **Prod E2E:** `e2e/production/tenant-theme-editor.spec.ts`. - Results: backend 4998 passed; frontend 1434 passed. No local CodeRabbit review was run (no CLI seat in this org), so this PR is left open for the GitHub bot. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a theme editor to the signed-in menu, with controls for title cards, karaoke videos, end screens, and advanced settings. * Preview theme drafts, upload images and fonts, and save or discard changes. Saved themes apply to future videos. * Added translations for the editor across supported languages. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
518126727b |
feat(tenant): tenant users never see or use consumer credits (#1076)
## Problem A tenant user's first login on `randy-vild.nomadkaraoke.com` showed the consumer "We weren't able to grant free credits" screen with **Buy Credits**. The backend also sent a credits-denied email. Tenant jobs were being charged consumer credits too, so the client's first submit would fail with "out of credits". Tenants are companies or bulk clients on a separate commercial agreement. ## Changes - **Sign-in:** - Tenant magic links skip the whole welcome-credit flow: no AI eval, no denial or welcome email, no referral attribution. `credit_status` comes back as `not_applicable`. - Sending a tenant magic link no longer precomputes a credit evaluation. - **Jobs:** `create_job` never credit-checks or charges jobs with a `tenant_id`. They're marked `payment_bypassed`, so duration reconciliation never pauses them for credits and refunds never mint credits. - **Access:** an empty tenant allowlist now means **admins only**. Free jobs must never be open to arbitrary sign-ups. All 3 live tenants already have allowlists. The admin UI copy is updated. - **Frontend:** on tenant portals, the credits badge and menu, Buy Credits, and Referrals are hidden. The verify page never shows credit screens for tenant logins. ## Tests - `test_tenant_no_consumer_credits.py`: a tenant job isn't charged and is marked `payment_bypassed`; consumer jobs are still charged and still rejected without credits. - Verify route: a tenant login returns `not_applicable` and the grant is never called. - Model: an empty allowlist means admins only. - `AuthStatus`: tenant mode hides credits. - Results: backend 4953 passed; frontend 1423 passed. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Access** * Tenant portals with no configured email addresses or domains are restricted to administrator sign-ins. Configured allowlists continue to limit access to listed users and administrators. * **Tenant Portal** * Tenant sign-ins no longer receive consumer welcome-credit evaluations or related credit emails. * Tenant jobs bypass consumer credit checks and charges. * Credit balances, credit-purchase options, and referral links are hidden in tenant portals. * Successful tenant sign-ins proceed to the app regardless of credit status. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
fe217fda1f |
kjbox singer search: Gemini fallback with the job flow's catalog tidy + descriptive queries (v0.246.2) (#1065)
## Summary The Gemini fallback for kjbox's singer song identification (kjbox `docs/SONG-IDENTIFICATION.md`). kjbox's on-device matcher answers most searches; this endpoint handles what it can't. - `POST /api/kjbox/catalog/resolve` now runs `resolve_and_tidy`: 1. One Gemini call (`gemini-3.8-flash`) splits the singer's one-line query into artist/title and fixes typos. 2. It now also resolves **descriptions**: "that song from titanic" → Céline Dion — My Heart Will Go On; "is this the real life" → Queen — Bohemian Rhapsody; "song from top gun" → ambiguous: Danger Zone / Take My Breath Away / Hold My Hand (live-checked). 3. The chosen song goes through the job flow's own `judge_match(stage="fast")` catalogue tidy, so formatting matches gen's job form. - AI kind `none` keeps its typed split, so kjbox can pre-fill its make-it form. A failed catalogue pass is returned uncached. - kjbox calls it only when on-device identification and karaoke search both found nothing, or when the singer uses "Describe it". It's cached and rate-limited as before. - Docs: `docs/API.md` notes that the match judge's corrections are Gemini in practice (its catalogue pass is exact-match only and decide's catalogue is prefix-only). ## Tests - `test_match_judge_free_text.py`: resolve_and_tidy cases, descriptive prompt coverage - `TestCatalogResolve` - Local CodeRabbit: findings fixed, re-review clean. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
3a6e175fbf |
feat(tenant): fully automatic tenant lifecycle — subdomain provisioning, delete, per-email access (#1072)
## Summary
The admin tenant console now creates and deletes **fully working**
tenants.
- **Create:**
- Provisions `{id}.nomadkaraoke.com` through the Cloudflare API: a Pages
custom domain on `karaoke-gen-tenant` plus a proxied CNAME.
- If a non-tenant DNS record already owns the hostname, create returns
`409` before anything is written.
- If Cloudflare fails, the GCS tenant is rolled back and create returns
`502`.
- The subdomain must be `{id}.nomadkaraoke.com`, because the edge
function and middleware derive the tenant from the first hostname label.
- **Delete:** new `DELETE /api/admin/tenants/{id}` removes the subdomain
(only records that point at the tenant Pages project), the theme (unless
it's the default or shared), the registry entry and the config. Jobs are
kept.
- **Retry/backfill:** new `POST /api/admin/tenants/{id}/domain` is
idempotent.
- **Per-email access:** new `auth.allowed_emails`, alongside domains. If
either list has entries, only listed emails/domains can sign in; both
empty = open. `@nomadkaraoke.com` admins can always sign in; the
docstring already claimed this but the code didn't do it.
- **Admin UI:**
- domain status badge with set-up/retry
- access editor with an "anyone can sign in" warning
- delete that requires typing the tenant id
- after create, polls until the certificate is live
- **Infra:** secret `cloudflare-tenant-domains-token`. The Pulumi
container is already applied; the value is an account-owned Cloudflare
token scoped to Pages R/W on the account plus DNS R/W on the
nomadkaraoke.com zone only.
## Testing
- New `test_tenant_domain_service.py` runs against a stateful fake
Cloudflare (httpx MockTransport). It covers provisioning, idempotency,
foreign-record conflicts, deprovisioning only our records, API errors
and a missing token.
- Service tests cover:
- create provisions the domain
- a DNS conflict or unusable Cloudflare means no writes
- a provisioning failure means a full rollback
- delete removes everything, aborts if Cloudflare fails, and keeps a
shared theme
- access-list normalisation and enforcement
- Route tests cover delete (204/404/502), the domain endpoint (200/409),
502 on create, allowed_emails passthrough, and admin-only access.
- Model tests cover individual emails, combined lists, the legacy flag,
and the admin bypass.
- Frontend (Jest) covers access editing, the open-portal warning, domain
set-up, and type-to-confirm delete.
- Results: `pytest backend/tests --ignore=backend/tests/emulator` gives
4927 passed; `jest` gives 1421 passed.
No local CodeRabbit review was run, so this PR is left open for the
GitHub bot. A separate agent review was done before merge.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Tenant creation provisions a portal subdomain, with its setup status
shown in tenant management.
* Admins can retry domain setup and delete tenants after confirming the
tenant ID.
* Tenant access can be restricted to specific email addresses, allowed
email domains, or both. Addresses are normalized, and admin access is
retained.
* **Bug Fixes**
* Email restrictions are enforced for magic-link sign-in even when no
domain allowlist is configured.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
||
|
|
7ebe929d71 |
feat: quick scrolling-lyrics version for kjbox make-it jobs (v0.244.0) (#1064)
## Summary
kjbox singer "make it" jobs take ~30 min. For kjbox jobs (`X-Client-Id:
kjbox*`) the audio worker now also produces a rough **scrolling-lyrics
draft video** within a few minutes of the audio landing, so the singer
can choose to sing right away (kjbox side: nomadkaraoke/kjbox PR to
follow).
- **Quick separation** — one fast single-model pass
(`mel_band_roformer_instrumental_fv7z_gabox.ckpt`, already baked in the
GPU image, MDXC overlap 2; `QUICK_VERSION_MODEL` override) on the warm
L4 **before** the ensemble.
- **Background render** (`backend/services/quick_version/`, ported from
kjbox `fastgen/`) while the ensemble runs: LRCLIB lyrics (synced →
time-anchored scroll; plain → constant crawl; none → title card), PIL
crawl PNG + one ffmpeg overlay pass, 854×480, vocals mixed back at 30%
as a guide.
- Uploads `jobs/{id}/quick/quick.mp4` → `file_urls.quick.video_mp4`;
`state_data.quick_version = {status, lyrics_tier, ready_at, timings…}`.
- The worker joins the render (≤300 s) before temp-dir cleanup, even if
the ensemble fails. Quick-version failures never fail the job. Skipped
on retries once `ready`, in remote-API mode, or with
`KJBOX_QUICK_VERSION_ENABLED=false`.
- No new endpoint: kjbox polls `GET /api/jobs/{id}` and downloads
`/api/jobs/{id}/download/quick/video_mp4`.
Plan: workspace
`docs/archive/2026-09-26-make-it-quick-fastgen-version-plan.md`.
## Testing
- `backend/tests/test_quick_version.py` (37): lyrics parsing/LRCLIB
fallbacks, layout + scroll expression, ffmpeg cmd, real PIL+ffmpeg
render (skips without ffmpeg), gating, separator params/output
detection, render thread success/failure, audio-worker wiring (join on
ensemble failure, timeout → failed, remote mode skips).
- Full backend suite: 4831 passed.
- Local real runs: Fv7z single-model separation produced both stems with
the custom names; ABBA "Waterloo" rendered in ~9 s (LRCLIB synced).
- Not yet measured in prod: GPU job cold start + L4 quick-separation
time — first thing to check after deploy.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
|
||
|
|
455fb11a14 |
feat(upload): unified browser upload flow, bring-your-own instrumental, stranded-upload cleanup (v0.243.0) (#1058)
## Summary Two invisible upload failures on the same song led to this: - **Job 8729f19a**: large audio upload (signed-URL flow) abandoned mid-upload. The only UI signal was the "Creating..." button text and the card already said "Setting up", so the job sat `pending` forever with the credit spent. - **Job 2d0c1ae0**: 44 MB instrumental upload on the review screen was a multipart POST through Cloud Run (32 MiB cap) and failed silently. Only the CORS preflight was logged. ## Changes **Shared upload layer (frontend)**: all user uploads now go through one path: - `lib/upload.ts`: signed-URL PUT with progress, sequential multi-file upload with aggregate progress, client-side duration check. - `hooks/useUploadTask.ts`: progress state plus a `beforeunload` guard. - `components/upload/UploadProgressModal.tsx`: a modal the user can't close, showing progress, speed, ETA, "file N of M" and a keep-tab-open warning. - Used by guided job creation, tenant single job and instrumental review. Tenant bulk keeps its per-row table and uses the shared guard. **Guided flow** - Always uses the signed-URL path, which also fixes `requires_audio_edit` being silently dropped for uploads of 25MB or more. - Private upload jobs can attach their own instrumental (`existing_instrumental`). This skips separation and instrumental review. The duration is checked before upload, and the backing-vocals option is hidden. **Instrumental review** - New `instrumental-upload-url` and `instrumental-upload-complete` endpoints: direct-to-GCS upload, then a finalize step. - One shared processing helper with the multipart route: duration check, FLAC conversion, `stems.custom_instrumental`. - Returns a signed `audio_url`. The old post-upload preview URL had no backend route. - Inline error plus toast for mismatches and failures. **Backend robustness** - `state_data.awaiting_upload`: the dashboard shows "Waiting for upload" instead of "Setting up". - The hourly stale-review run also cancels and refunds non-tenant jobs still awaiting upload after 2h with nothing uploaded. - `uploads-complete` now checks ownership (previously any signed-in user could finalize any pending job). On an instrumental duration mismatch it cancels and refunds the job instead of stranding it. ## Testing - `make test`: backend 1839 + 4788 + 150 emulator passed; frontend Jest 1391 passed across 117 suites. - New tests: - backend: stale-upload processor; signed instrumental endpoints (15 cases); `requires_audio_edit` persistence; uploads-complete ownership and cancel-on-mismatch - frontend: upload lib, `useUploadTask`, modal (speed/ETA, can't be dismissed), `OwnInstrumentalField`, `TenantJobFlow`, API flows - Manual check against the real frontend (local dev, prod read-only; create, upload and finalize endpoints mocked, uploads sent to a throttled local PUT server, so no prod data was changed): - guided flow with and without an instrumental - instrumental review upload on 2d0c1ae0, including a mismatch caught before upload - Escape and outside clicks don't close the modal; navigating away triggers the `beforeunload` prompt Plan: `docs/archive/2026-09-26-unified-upload-flow-plan.md` 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
e9465c73d4 |
chore: gemini-3.8-flash everywhere + auto-correct on Opus 5.5 (cost) (#1054)
## Why GCP credits expired 2026-09-19; Vertex Gemini 3.1 Pro spend had grown $9 → $37 → $105/30d. `gemini-3.8-flash` is $0.75/$3.75 per 1M (vs $2/$12 for 3.1 Pro; rises to $1.50/$7.50 on 2027-01-01). Opus 5.5 is $4/$20 (vs $5/$25 for 4.8). ## Changes - All Gemini defaults → `gemini-3.8-flash`: credit eval, custom lyrics, auto-correct (default + compare leg), match judge, parse titles, tenant bulk analyze, agentic AI router, error-monitor LLM analysis (also `us-central1` → `global`, required for Gemini 3.x), frontend `translate.py`. - Auto-correct Claude leg `claude-opus-4-8` → `claude-opus-5-5` (ci.yml env), with `effort: "high"` pinned (5.5 defaults to medium). Pricing table gains both new models. - Docs updated; version 0.241.1. ## Evidence (12 recent prod jobs, pre-review lyrics scored against the human-reviewed final) | Model | Precision | Word-level recall | $/job | |---|---|---|---| | gemini-3.1-pro-preview | 91% | 74% | $0.067 | | gemini-3.8-flash | 98% | 38–41% | $0.020 | | gemini-3.8-flash @ high thinking | 97% | 60% (4/12 timed out, 504) | $0.041 | | claude-opus-4-8 | 94% | 33% | $0.080 | | claude-opus-5-5 (effort high) | 93% | 58% | $0.048 | Flash for the Gemini leg is a deliberate cost/recall trade-off (Andrew chose it); Opus 5.5 more than compensates. Also verified: `gemini-3.8-flash` works on Vertex `global` (rejects `thinking_level=minimal`; nothing here uses it), match-judge latency median 3.3s / max 6.2s (< 12s timeout), Japanese translation smoke test preserves ICU placeholders. Translation cache is keyed on English text + locale, so no re-translation churn. Tests: auto_correct, credit eval, custom lyrics, parse_titles, error_monitor, agentic router suites pass locally. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * AI-powered correction, translation, matching, title parsing, and evaluation now use updated models, including Gemini 3.8 Flash and Claude Opus 5.5. * **Updates** * AI service pricing reflects scheduled rate changes, including a Gemini 3.8 Flash rate change beginning January 1, 2027. * Application version updated to 0.241.1. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
b177c36641 |
feat(kjbox): free-text song resolve for singer search auto-correct (v0.241.0) (#1053)
## Summary
Andrew wants the kjbox singer song search to auto-correct like gen's job
creation does, e.g. "the strokes max picu" → The Strokes — Machu Picchu,
with Undo. gen's match-judge only takes a separate artist and title, and
its catalog stage only fixes formatting, not typos. So this adds:
- **`POST /api/kjbox/catalog/resolve`** (`X-Kjbox-Secret`) `{query}` →
match-judge verdict shape (`kind` cosmetic/content/ambiguous/none,
`confident`, `canonical_*`, `alternatives`) plus
`typed_artist`/`typed_title`.
- **`backend/services/match_judge/free_text.py`**: one small Gemini call
(the match judge's model and settings) that splits and corrects
together. Bad or incomplete model output, or any model failure, degrades
to `kind: "none"`, never a 5xx.
- **Cost and abuse controls:**
- Results are cached per case-folded query (in-process, 2000 entries);
transient failures are not cached.
- A partner-wide cap of 120 AI calls/min per instance (429
`rate_limited`); cache hits are free.
- kjbox only calls this after its own catalogue search came back empty.
- Docs: `docs/API.md` § kjbox Partner API.
## Testing
- `test_match_judge_free_text.py` (8)
- `TestCatalogResolve` (4)
- Full backend suite passes (emulator tests excluded)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added free-text song search resolution that identifies artist and
title, corrects typos, and can suggest alternatives when a match is
ambiguous.
* Repeated searches can use cached results. If resolution is
unavailable, the request returns without an error; uncached requests may
be rate-limited.
* **Documentation**
* Documented the song-resolution API, including its response behavior
and request limits.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
||
|
|
36e95c7fe9 |
feat(kjbox): on-demand review sign-in link + record who started the review (v0.240.0) (#1052)
## Summary
Backs Andrew's feedback on the kjbox singer make-it flow (kjbox
companion PR to follow):
- **`POST /api/kjbox/jobs/{job_id}/review-link`** (partner secret +
singer Bearer session): mints the same one-click `job_review:<id>`
sign-in link the kjbox review email carries, on demand. This powers
"**Tap here** to review it yourself" on the singer's My songs. It is
owner-only (404 otherwise, including unknown ids) and only while
`awaiting_review`/`in_review` (409 `not_in_review`). It also returns
`review_started_by`.
- **`state_data.review_started_by` / `review_started_at`** are recorded
on the first `AWAITING_REVIEW → IN_REVIEW` transition. The value is
`"admin"` when an admin other than the owner (the KJ) opened it, and
`"owner"` otherwise (review-token links count as owner). It is wired
into `GET /api/review/{id}/correction-data` (`require_review_auth` now
stashes the reviewer on `request.state`) and into the legacy
`start-review` / `corrections` routes. kjbox reads it through its admin
job poll to show "Andrew has already started reviewing".
- Docs: `docs/API.md` § kjbox Partner API.
## Testing
- New `TestReviewLink` (6 tests) in `test_kjbox_partner_api.py`
- New `test_review_started_by.py` (7 tests)
- New route-level `TestCorrectionDataRecordsReviewStarter` (2 tests)
- Full backend suite: 4671 passed (excluding emulator tests)
CodeRabbit CLI has no seat on this org, so this PR is opened without the
ignore line and the GitHub bot will review it.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added a partner API endpoint for job owners to request a localized
sign-in link for jobs awaiting or undergoing review.
* Review details now indicate whether the owner or an administrator
started the review.
* **Documentation**
* Updated the API documentation with the required partner secret header
for review-link requests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
||
|
|
d570b1f139 |
feat(kjbox): partner API for singer email-code sign-in + show credits (v0.239.0) (#1051)
## Summary
Partner API so the kjbox singer UI (live karaoke nights) can turn
patrons into real karaoke-gen customers: the singer verifies their email
with a 6-digit code typed into the kjbox page, and kjbox then searches
audio / creates the job **as that user** (normal gen delivery emails).
All under `/api/kjbox`, gated by `X-Kjbox-Secret` (Secret Manager
`kjbox-partner-secret`, wired in ci.yml; 503 when unset):
- `POST /auth/send-code` — emails a 6-digit code (hashed at rest, 10-min
expiry, new code supersedes old). Exempt from the 2-signups-per-IP cap
(a venue shares one IP) — replaced by a partner-wide 100 new accounts /
24h cap + 5 codes / email / hour. New users tagged `signup_source=kjbox`
(+ venue).
- `POST /auth/verify-code` — 5 attempts then burned, single-use; shares
the post-login path with magic-link verify (refactored into
`complete_verified_login`: welcome credit w/ AI eval, locale, referral,
session, welcome email).
- `POST /credits/show-credit` (+ user Bearer) — +1 credit, no email
("free at Andrew's shows"), idempotent per key, 5 / user / 24h.
- Review-needed + 24h reminder emails for `X-Client-Id: kjbox*` jobs
carry a one-click sign-in link to the review page (singers never signed
in on gen's site).
Paired kjbox PR: nomadkaraoke/kjbox (singer make-it flow, v0.116.0).
Plan: kjbox `docs/archive/2026-09-24-singer-make-it-auto-flow-plan.md`.
## Testing
- New: `backend/tests/test_kjbox_partner_api.py`,
`backend/tests/emulator/test_kjbox_partner_service.py` (+ additions to
existing tests for the verify refactor, review-email links, X-Client-Id
on create-from-search)
- Backend unit suite: 4644 passed; emulator suite passed.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added partner sign-in with one-time email codes and show-credit
claiming, with limits on code requests and credit grants to prevent
repeated or duplicate claims.
* Added localized sign-in emails and single-use review links for
eligible reminders.
* Sign-in supports welcome-credit evaluation and referral attribution.
* **Improvements**
* Review reminders can use custom links, and new accounts can record
signup source and venue.
* Credential-like request headers are redacted from stored metadata.
* **Documentation**
* Added partner API usage and behavior details.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
||
|
|
a9c6de0617 |
feat: KaraokeHunt interceptor — one freebie per email + community-version short-circuit (v0.236.0) (#1039)
## Summary Follow-ups from Andrew's live test of the v0.235.0 interceptor (#1037): 1. **Community short-circuit** — his "abba - Waterloo" test generated a real job despite a community karaoke version existing. Now `check_community_versions` gates the whole routing: existing version → no job, no board; the email's main button is the YouTube link (outcome `community_existing`), and new users keep their conversion credit for a custom song. 2. **One freebie EVER per email** (Andrew's spec): the goal is converting installs into direct gen.nomadkaraoke.com users, not keeping the dead app alive. First-time emails (all variants) now state the app is retired and they must uninstall it and use Gen directly. A second request from a converted address gets no job/board/credit — just the "please uninstall" email (`repeat_request`), throttled to one per address per 7 days. Accidental same-song double-taps stay silent (`duplicate` dedup runs first). ## Implementation - `karaokehunt_conversion.py`: `CONVERTED_OUTCOMES` + `_freebie_already_used` (fails open) + `_uninstall_email_recently_sent` (fails closed — anti-spam) + community short-circuit before routing; queried fields re-verified in Python (defense in depth). - `send_karaokehunt_conversion`: new `community` and `uninstall` variants; rewritten intro (retired app, one-time freebie); bold uninstall note on every first-time variant; new keys translated to all 32 locales. - Docs: API.md + interceptor README routing updated. ## Testing 42 karaokehunt tests (6 new: community short-circuit ×2, repeat/uninstall ×2, email variants ×2 more); full `make test` green. Post-merge: live POSTs through create.karaokehunt.com for (a) a song with a community version and (b) a repeat request from a converted address. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * KaraokeHunt requests now receive a one-time conversion per email address. Repeat requests receive an uninstall reminder instead of a new conversion, with reminders limited to once every seven days. * When a community version is available, the email links directly to it; new users keep their free credit. * Conversion emails explain credit status and include a Nomad Karaoke sign-in link. * **Documentation** * Updated guidance on request outcomes and the one-time conversion process. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
3d1ee6a621 |
feat: KaraokeHunt retired-app request interceptor — auto-convert to Gen users (v0.235.0) (#1037)
## Summary The retired KaraokeHunt mobile app still has installs submitting ~5-10 song requests/month. Each request POSTs to the dead `create.karaokehunt.com`, fires a client-side Pushbullet push to Andrew, and shows the user a hardcoded *"you should receive an email in 5-10 minutes with a link to your new karaoke video"* — a promise broken for years. This PR makes it true and converts those users into Gen users (Andrew's spec, BACKLOG Frog #2 extension). ## How - **Cloudflare Worker** (`infrastructure/cloudflare/karaokehunt-interceptor/`, deployed via `deploy.sh` — the karaokehunt.com zone is not IaC-managed): answers the app instantly, `waitUntil`-forwards the payload to the new backend intake with a shared secret. - **`POST /api/karaokehunt/request`**: unauthenticated intake, gated by `X-KH-Forwarder-Secret` (Secret Manager `karaokehunt-forwarder-secret`, already created; wired in ci.yml `--set-secrets`), 10/min/IP rate limit, lenient payload parsing, Firestore `karaokehunt_requests` audit trail. - **Conversion** (`backend/workers/karaokehunt_conversion.py`, mirrors `community_daily_pick`): - new email → account + 1 credit (`karaokehunt_app_conversion`) → job created **as them** (consumes it); first sign-in still grants the welcome credit (deliberate — next song covered) - existing user with credits → job on their credit (email offers to restore it on reply) - no credits, or over `KARAOKEHUNT_DAILY_JOB_CAP` (default 3/day, bounds unattended spend) → free community requests board (dedup → upvote) - conservative `pick_auto_selection` (not `select_best`): unconfident matches park in `AWAITING_AUDIO_SELECTION` and the email asks them to pick — garbage app inputs must never auto-publish to YouTube - 14-day per-email+song dedup; failures retryable via admin `POST /api/karaokehunt/internal/reprocess/{doc_id}` - **Email** `send_karaokehunt_conversion`: 3 variants (job started / pick the recording / on the board), 7-day one-click sign-in link, KaraokeNerds community-version bonus link when one exists. EN + all 32 locales via translate.py. ## Safety Deploys **dark**: endpoint 503s until the secret env lands, and receives no traffic until the Worker + DNS flip (done post-merge via `deploy.sh`). Kill switch = delete the Worker route; backend-side = remove the secret mapping or `KARAOKEHUNT_DAILY_JOB_CAP=0`. ## Testing - 36 new tests (`test_karaokehunt_intake.py`, `test_karaokehunt_conversion.py`): secret/rate-limit/JSON gates, all conversion routings, credit-grant guards, park-on-unconfident-search, email variants + HTML escaping, reprocess auth. - Full `make test` green (backend + frontend). - Post-deploy: live end-to-end POST through `create.karaokehunt.com` mimicking the app payload, verifying account/job/email/Firestore. Plan doc: `docs/archive/2026-09-22-karaokehunt-app-interceptor-plan.md` 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Requests submitted through the retired KaraokeHunt app are now received and converted into Nomad Karaoke jobs, accounts, or community-board requests. - Users receive localized email updates with job status, credit usage, audio-selection prompts, and tracking links. - Requests can be retried by administrators when processing fails. - Duplicate protection, request limits, and daily processing caps help prevent repeated or excessive submissions. - **Documentation** - Added API and deployment documentation for the request-forwarding workflow. - **Chores** - Released version 0.235.0. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
d85b6e271c |
feat: precomputed vocals peaks — Waveforms strips paint sub-second (#1031)
## Summary Final piece of the review-page speed work (#1024 → #1028 → #1029): the Waveforms-mode strips no longer require downloading + WebAudio-decoding the entire vocals stem (~7.5MB, ~4–6s) — the backend now serves a **precomputed 400-peaks/s envelope as ~150KB of uint8/base64 JSON**, cached in GCS and pre-warmed at prep time, so strips paint sub-second. ## How - **`WaveformGenerator.generate_peaks()`** — numpy max-abs envelope, bucket-for-bucket identical to the frontend's `computePeaks` (same drift-free fractional-stride bucketing; exact match vs a naive reference in tests). - **`AudioAnalysisService.get_vocals_peaks()`** — GCS cache at `review-audio/vocals_peaks_400.json`, validated against the stem path; miss decodes the small transcoded OGG; upload best-effort. - **`GET /api/review/{job_id}/vocals-peaks`** — off the event loop under the shared waveform semaphore; `202 Retry-After: 15` while separation runs (mirrors `/audio/vocals`); `Cache-Control: private, max-age=600`; `peaks_per_second` clamped 50–1000. - **`screens_worker`** pre-computes the envelope next to the existing OGG + waveform pre-warm, so it's a pure cache read by the time a job reaches review. - **Frontend** — `VocalsAudioDataLoader` prefers the peaks endpoint (one quick retry on transient errors) and **falls back to the original full download+decode** on any terminal peaks failure, so a peaks-side problem can never cost a user their waveforms. Local mode (no peaks URL) uses the old path unchanged. Vocal stem selection factored into `backend/utils/stems.py` (shared by routes + worker). ## Testing - Package: 4 tests incl. exact naive-reference parity (**106 passed**) - Backend: 4 service + 4 endpoint tests (**4518 passed**) - Frontend: `fetchVocalsPeaks` units + 5 loader-behavior tests (prefers peaks / falls back / one-retry-then-fallback / 202 polling / audio-only mode) (**1307 passed**) - `review-fast-load.spec.ts` prod E2E extended with a cached vocals-peaks latency/shape check - Post-deploy: will verify the endpoint on a real in-review job and confirm payload size + latency ## Notes - Existing in-review jobs have no cache yet — their first Waveforms open computes + caches lazily (bounded); every open after that is sub-second. New jobs are pre-warmed. - 8-bit amplitude quantization is visually lossless at strip heights (14px). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
30cf41e7d6 |
feat: concurrent-reliability phases 2-5 — Cloud Run shape, degradation telemetry, banner tuning, load-test guard (#1028)
## Summary Phases 2–5 of the concurrent-load reliability initiative (Phase 1 = #1024, verified live: the 10-tab meltdown went from 9/30 failed requests to 30/30 OK). Verbatim spec: `docs/archive/2026-09-17-concurrent-review-reliability-verbatim-record.md`; plan: `docs/archive/2026-09-18-concurrent-review-reliability-investigation-and-plan.md`. ## P2 — Cloud Run shape - **`--concurrency 80 → 16`** for karaoke-backend: the service is one uvicorn process per 2-vCPU instance, and review loads are heavy; at 80 a burst never triggers scale-out and all lands on the single min-instance. 16 makes bursts fan out while staying well above normal per-instance load. (Deploy flags live in ci.yml — the service is not Pulumi-managed.) - **Local preview ffmpeg fallback bounded** (`LOCAL_PREVIEW_MAX_CONCURRENCY=1`, 900s queue timeout, fast RuntimeError after that): the GCE-outage fallback runs ffmpeg ON the API instance; unbounded, a few concurrent Preview clicks during an encoder outage pegged both vCPUs, stalled the health probe, and showed everyone the orange banner (Andrew's banner-during-preview screenshot). ## P3 — degradation telemetry (the explicit ask) - New **`POST /api/client-events`** (unauthenticated so review-token users report too; 30/min/IP rate limit; bot-filtered; detail capped; URLs query-stripped). Each event → Firestore `client_events` collection **and** a structured `client_event type=…` log line, so "how often are users seeing this?" is reviewable via Firestore aggregation or Cloud Logging correlation. - Frontend reporter `lib/degradation-events.ts` (60s per-type throttle, localhost no-op, never throws), wired into: - `BackendStatusBanner` → `banner_reconnecting` / `banner_unavailable` with stall age, in-flight count, probe state - review lyrics-load failure → `lyrics_load_failed` with message/status - `VocalsAudioDataLoader` → `waveform_slow` (>20s to first strips) and `waveform_failed` (gave up) with poll/retry counts ## P4 — banner tuning - Escalating to the full "We're having trouble reaching our servers" card now requires **two consecutive failed health probes** (`UNAVAILABLE_PROBE_FAILURES=2`). One failed 4s probe against a briefly-pegged instance shows only the calm "Reconnecting…" pill. Single-failure behavior for the pill is unchanged, as is instant clearing on recovery. ## P5 — regression guard - **`scripts/load-test-review.py`** checked in: replays the 10-tab incident (Firestore-ADC review tokens, curl request trios, pass/fail thresholds, exit code). Documented in `docs/TESTING.md`. Validated against prod post-Phase-1: 9/9 requests OK, tabs < 3s. ## Testing - Backend: 6 new tests for client-events (persist/validate/rate-limit/bot/fail-soft/detail-cap) + local-preview bound test — full suite **4517 passed**. - Frontend: new degradation-events suite, two-probe escalation + debug-snapshot tests; one existing api-fetch test updated to the new escalation contract — full suite **1298 passed**. - Live validation of the load-test tool against prod (small 3-tab run): PASS. ## Notes for review - Cloud Run concurrency change means bursts now spin up extra instances (billable during requests only; max-instances unchanged at 20). - Firestore `client_events` has no TTL policy yet (needs a gcloud/console action our read-only ADC can't do); volume is tiny (throttled per client) — can add later if it ever matters. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added automatic reporting for degraded experiences, including reconnecting or unavailable backend status, failed lyrics loads, and slow or failed waveform loads. - Added concurrent review-page load testing to help verify performance under multiple open tabs. - **Improvements** - Backend status now remains in “reconnecting” until multiple health checks fail, reducing premature unavailable messages. - Local preview processing is limited during heavy demand to improve stability. - **Documentation** - Added API documentation for client telemetry and guidance for running concurrent-load regression tests. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
b336f36f4e |
perf: survive concurrent review-page loads (10-tab meltdown fix, Phase 1) (#1024)
## Summary
Andrew can reliably take the backend down by opening ~10 lyrics-review
tabs at once: 1–2 load, the rest show "We're having trouble reaching our
servers" / "The server is temporarily unavailable". This PR is Phase 1
of the systematic fix (investigation + full phased plan in
`docs/archive/2026-09-18-concurrent-review-reliability-investigation-and-plan.md`;
Andrew's verbatim spec in
`docs/archive/2026-09-17-concurrent-review-reliability-verbatim-record.md`).
**Reproduced before the fix (prod, 2026-09-18 03:35Z):** 10 simulated
tabs → 8/30 requests HTTP 500, survivors took 44–72s (vs 0.8–1.3s solo),
Cloud Run logged 9× "request aborted — no available instance". Root
cause: `/waveform-data` decodes a whole stem (pydub→ffmpeg + Python RMS
loop) **synchronously on the single uvicorn event loop, uncached**,
while every other review endpoint also does blocking Firestore/GCS on
that loop — one wedged instance takes all traffic (concurrency=80 → no
scale-out at 10 tabs).
## Changes
**Backend**
- `/waveform-data`: persistent GCS cache
(`jobs/{id}/review-audio/waveform_review_{n}.json`, validated against
source path + num_points), computed via `asyncio.to_thread` under a
bounded semaphore (`REVIEW_WAVEFORM_MAX_CONCURRENCY`, default 4). Cache
miss decodes the ~3MB transcoded OGG instead of the ~35MB raw FLAC.
Pre-computed in `screens_worker` (next to the existing OGG pre-warm) so
review opens are pure cache reads.
- Blocking Firestore/GCS calls (`get_job`, `file_exists`,
`download_json`, `transition_to_state`) moved off the event loop via
`asyncio.to_thread` in the review hot-path endpoints (correction-data,
waveform-data, audio proxies, instrumental-urls/-analysis).
- `/audio/vocals`: now served from the bounded LRU byte cache
(previously a fresh multi-MB GCS download per request), honors HTTP
Range (206), and ranged review-audio responses carry `Cache-Control:
private, max-age=600`.
**Frontend**
- `VocalsAudioDataLoader`: transient failures (5xx/429/network) retry
with backoff `[5s, 15s, 30s, 60s]` instead of silently giving up — the
"Waveforms mode shows no waveforms until manual reload" symptom. 404
stays terminal via new typed `AudioFetchError`; 202 polling unchanged.
## Testing
- 12 new backend tests (waveform cache
hit/miss/stale-source/num_points/OGG-preference/upload-failure; vocals
LRU + Range 206 + Cache-Control; waveform-data endpoint routing) — full
backend suite **4500 passed**.
- 3 new frontend tests (transient retry → load, retry exhaustion, 404
terminal) — full Jest suite **1289 passed** across 105 suites.
- Post-deploy: will re-run the 10-tab burst repro against prod and
record before/after numbers in the plan doc.
## Follow-ups (later phases, tracked in the plan doc + BACKLOG)
Cloud Run shape (lower concurrency so bursts scale out),
`/api/client-events` degradation telemetry (Andrew's explicit ask),
banner threshold tuning, `instrumental-analysis` still signs 4 URLs
(13.7s baseline), load-test regression guard.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Review waveforms load faster with cached waveform data and improved
concurrent-load handling.
* Audio playback supports partial-range requests and short-term caching
for smoother streaming.
* Transient audio failures are automatically retried up to four times.
* **Bug Fixes**
* Reduced review-page failures and delays during simultaneous loads.
* Waveform generation falls back to available audio when backing vocals
are unavailable.
* **Documentation**
* Added API documentation for waveform data and updated audio streaming
behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
||
|
|
bb743b49e4 |
Review fast full load: remove IAM signBlob from the review hot path (v0.229.0) (#1018)
## Why
Root-cause fix for the **2026-09-17 signing-hang incident**. `GET
/api/review/{id}/correction-data` signed 2-3 GCS URLs per load via the
IAM **signBlob** API — a network round-trip that made review loads slow
and, when it stalled (~120s), exhausted the request thread pool and took
gen down (`Service Unavailable`, "Loading lyrics data…" hang).
`v0.228.1` bounded/timed signing and made audio + waveform
**best-effort** (the degraded fail-soft path). Andrew's ask was to fix
the root cause so the review page **always loads fast with all features
— audio + waveform — present on first paint**, not the degraded path.
## What (Option B — same-origin byte proxy)
Removes IAM signBlob from the review hot path **entirely** —
`correction-data` now performs **zero** signing calls.
**Backend** (`backend/api/routes/review.py`)
- New `GET /{job_id}/instrumental-audio/{option_id}`: streams the
transcoded OGG bytes with HTTP **Range (206)** support (so `<audio>` can
seek), authenticated by `require_review_auth` (token in the query string
for raw-media-src use, exactly like `/audio/vocals`). The stem GCS path
is resolved **server-side** from the option id — clients can't request
arbitrary blobs. Bounded **LRU byte cache** (prod-safe, unlike the
unbounded dev cache).
- `_build_instrumental_options`: returns **relative same-origin proxy
paths** as `audio_url`, no signing. Local-dev byte proxy path unchanged.
- Dropped the `backing_vocals_waveform_url` signing — it was a **dead
field** (0 consumers repo-wide; the waveform is drawn from the
Bearer-auth JSON `/waveform-data` endpoint).
- Also removed a now-unused `storage` param + refreshed stale
docstrings.
**Frontend** (`frontend/lib/api.ts`)
- Resolves each option's relative proxy path into an absolute,
`?token=`-authenticated `<audio>` src in the API layer
(`getCorrectionData` + `refreshInstrumentalUrls`) — **zero component
changes**, mirrors the proven `getVocalsAudioUrl` pattern. Also fixes
the CORS `Origin: null` issue for good, and the proxy URLs never expire
(the whole signed-URL expiry/refresh dance is now moot).
The `v0.228.1` signing timeout + bounded admission stays as
**defense-in-depth** for the remaining, non-review-hot-path signing
(e.g. preview-video).
## Tests
- Backend: new endpoint (bytes / Range 206 / auth / 404s), plus
**positive assertions that the hot path signs nothing**
(`generate_signed_url` / `get_review_audio_url_async` not called).
- Frontend: URL-resolution unit tests (relative→absolute+token; dev
absolute + null left untouched; refresh path).
- Prod E2E: `frontend/e2e/production/review-fast-load.spec.ts` —
env-gated contract (proxy `audio_url`s present, endpoint serves
`audio/ogg` + Range, loads fast).
- Full suites green: **backend 4469 passed**, **frontend jest 1286
passed**.
## Deploy note
Minor, self-healing skew: during the backend↔frontend deploy gap, only
the **instrumental preview player** for a job opened in that window may
not play (lyrics + everything else fine); a reload after both deploy
fixes it. Far less severe than the signBlob-stall outage this removes.
Plan + rationale:
`docs/archive/2026-09-17-review-fast-full-load-plan.md`. Runbook updated
in `docs/TROUBLESHOOTING.md`.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Review pages now load instrumental audio through same-origin,
authenticated audio URLs.
- Audio playback supports streaming and range requests for faster, more
reliable access.
- Instrumental audio options automatically include usable
token-authenticated URLs.
- **Bug Fixes**
- Review data no longer depends on URL-signing round trips, preventing
loading delays caused by signing timeouts.
- Removed the unused backing-vocals waveform URL from review responses.
- **Documentation**
- Updated API and troubleshooting documentation for the new audio
streaming behavior.
- **Chores**
- Version updated to 0.229.0.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
|
36b331f3d7 |
feat(requests-board): admin history view + fix community publish transition (#1008)
## Why Andrew asked whether we can see past community-requests-board submissions and whether our first real user got a video. Investigation surfaced: - **First real user was fully served.** `tas041964@gmail.com` requested *Sam Cooke and the Soul Stirrers – Wonderful* (Sep 15); it was auto-picked, generated, published to YouTube (NOMAD-1650), and he got the completion email + emailed thanks. ✅ - **No admin view existed** to see this — you had to query Firestore directly. - **A real bug:** the community publish fan-out (`mark_published` + voter emails) fired **only** on the `youtube_upload_queue` path. Jobs published via the direct video-worker distribution path (like Sam Cooke's) bypassed it, so the request stayed `in_progress`, never lit up the board's public "Recently made" list, and other up-voters would silently miss their "track is live" email. ## What 1. **Fix the publish transition** — extract the fan-out into `backend/services/community_publish.py` and call it from **both** publish paths (`youtube_queue_processor` + `video_worker`). Also fixes a latent `len(int)` crash in the log line. 2. **Backfill / safety net** — `POST /api/internal/community-requests/reconcile`: idempotent, publishes any `in_progress` pick whose job is already live on YouTube. (Will be run in prod post-deploy to fix the Sam Cooke request.) 3. **Admin history view** — `GET /api/admin/community-requests` + a `/admin/community-requests` page: every request, all statuses, with requester, votes, and job + YouTube links. The public board's "Recently made" section already existed in the UI — it was empty only because of bug #1; fixing the transition (+ reconcile backfill) lights it up. ## Testing - `backend/tests/test_community_voter_notify.py` — fan-out from the shared module, both-path idempotency, voter dedupe/retry, YouTube-url resolution, and the reconcile scan. - `backend/tests/test_community_review_admin.py` — the new admin history endpoint maps all statuses. - All 31 requests-board + 158 worker tests pass. The one unrelated failure in `test_audio_processor_local_gpu.py` is pre-existing (confirmed with changes stashed). ## Notes - Admin surface is English-only (no i18n changes); no `en.json` strings added. - Local CodeRabbit review couldn't run (CLI requires life360-org SSO; no seat on this repo) — leaving to the GitHub bot. Plan: `docs/archive/2026-09-17-requests-board-history-and-publish-transition-plan.md` 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added an admin Community Requests history page covering all statuses, votes, submitters, review details, linked jobs, and YouTube videos. - Added admin reconciliation for eligible requests with live videos. - Added automatic voter email notifications after community-request publication. - **Bug Fixes** - Improved publication handling across direct and queued video workflows. - Added recovery for incomplete voter notifications and missed publication transitions. - **Documentation** - Documented the community-request history and reconciliation API endpoints. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
|
bedbcdc3bd |
Requests board: skip songs with existing community karaoke versions (submit + pick time) (#979)
## Summary Stops the requests board from spending a free track — or wasting voters' time — on a song that **already has a good community karaoke version online**, reusing the same KaraokeNerds check the normal job-submission flow uses, at two points. Follow-up to the requests-board Phase 2 automation (#977). Addresses Andrew's two asks: **A) Submission time (soft, non-blocking).** As a signed-in user types an artist/title on the board, we debounce-check KaraokeNerds and surface any existing community versions in a dismissible banner with YouTube links. They can still request it — but many will just watch the existing one. Reuses the existing `POST /api/catalog/community-check` endpoint + `CommunityVersionBanner` component (added an optional board-appropriate subtitle). **B) Pick time (review).** A song may have had no version when requested but gained one by the time it's upvoted to the top. So the daily picker now KaraokeNerds-checks each candidate **in rank order**: - A candidate that already has a community version is **flagged for review** (`review_state=pending`) instead of auto-made, and the picker **skips to the next clean request** — so a fresh, non-duplicate free track still ships that day. - Andrew is emailed a summary and resolves held picks at **`/admin/community-reviews`**: **Make ours anyway** (creates the job like a normal pick), **Reject** (declines and emails every up-voter a link to the existing version), or **Keep on board** (leaves it votable, snoozing re-review for 30 days). - Held requests stay visible/votable on the board; they just aren't auto-made. Flag/email is idempotent (no daily re-nag). ## Andrew's decisions (via AskUserQuestion) Held pick → **skip to next clean request**. Review surface → **admin queue page + email**. "Keep" → **return to board, snooze re-review (30 days)**. ## Changes - **Model:** `SongRequest.review_state` (pending|snoozed), `community_versions`, `community_checked_at`, `review_snoozed_until`. - **Service:** `list_pick_candidates` (skips review-blocked), `set_review_pending` (transactional/idempotent), `snooze_review`, `clear_review`, `reject_request`, `list_pending_reviews`. - **Picker:** candidate loop + community check; extracted `_provision_and_start` shared by the picker and the admin "make" action. - **Admin:** `GET/POST /api/admin/community-reviews`, `/admin/community-reviews` page + sidebar item. New existing-version email to up-voters (33 locales). - **Frontend:** board submit form debounced check + banner. ## Testing - Picker candidate loop (flag dup → make next clean; all-dup → empty + flagged; resume skips re-check), admin actions (make/reject/keep, 404, 500), emulator coverage for the new service methods. 86 passing across touched suites; frontend typechecks clean; 32/32 locale parity. ## Deploy No `infrastructure/` changes — merge deploys it. The daily picker is already enabled in prod, so the pick-time check goes live with the backend deploy; the submission banner ships with the frontend. ## Review CodeRabbit reviewed locally (one minor grammar nit, fixed). @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
|
05f024bdd4 |
Requests voting board Phase 2: daily free-community-track automation (#977)
## Summary
Phase 2 of the requests voting board (`requests.nomadkaraoke.com`) — the
daily automation that turns the top-voted request into a free,
auto-generated, published karaoke video. **Deploys dark** behind a
kill-switch (`COMMUNITY_DAILY_PICK_ENABLED`, default off); the daily
Scheduler only shadow-logs until it's flipped on.
Picks up
`docs/archive/2026-09-03-requests-voting-board-phase2-handoff.md`.
Design + Andrew's decisions:
`docs/archive/2026-09-03-requests-board-phase2-plan.md`.
## What's included
- **Daily picker** (`community_daily_pick.py`, Scheduler noon US
Eastern, DST-aware): claims the UTC day with a create-only lock
(`daily_community_pick/{YYYY-MM-DD}` → **one free track/day, total**),
picks the top open request with net votes ≥ 0 (source-agnostic; skips
community-rejected), grants the requester a free credit, and submits the
job **as that user** (owned by them, credit consumed) via the same
search → auto-select → download path the web flow uses. Fully idempotent
(resumable per-day lock phase + per-request guard flags).
Shadow/`?dry_run=true` mode.
- **24h ownership handoff** (`community_handoff.py`, hourly): reassigns
a stale review to the next up-voter (compare-and-set guarded), up to **5
voters**, then parks the track (`stalled`). Community jobs excluded from
the normal stale-review auto-cancel.
- **Voter publish fan-out**: YouTube upload-queue hook marks the request
`published` and emails every up-voter (except the owner) a new "track
you voted for is live" email (33 locales). Retry-safe (per-voter success
tracking).
- **Two internal endpoints** + two Cloud Scheduler jobs (Pulumi).
## Andrew's decisions (via AskUserQuestion)
- Trigger: **noon US Eastern**. Trending-agent fallback source:
**deferred** to a follow-up. Net-negative requests: **skipped**. Handoff
cap: **5 voters then park**.
## Deferred / follow-ups (not in this PR)
- **Trending-agent fallback source** (auto-submit a candidate when the
board is empty) — deliberately deferred; must generate candidates the
careful way (karaoke-decide `candidates` + KaraokeNerds).
- **YouTube description board link** — left untouched to avoid
clobbering Andrew's parallel descriptions work; add
`requests.nomadkaraoke.com` to `default_youtube_description` once that
lands.
## Testing
- Worker unit tests (kill-switch shadow, empty board, day-lock
idempotency, credit-grant/job-create guards, handoff cap/stale/CAS,
voter fan-out retry-safety) + emulator tests for the new service methods
against real Firestore transactions.
- All touched existing suites (youtube queue, stale-review,
requests-board routes/logic) pass. 79 passed locally.
## Review
CodeRabbit reviewed locally (2 cycles completed; 2 major + 1 minor
findings all addressed — retry-safe fan-out, CAS owner assignment,
up-voter ordering). 3rd cycle hit the free-OSS rate limit; it would only
re-verify the applied fixes.
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
|
d41854f787 |
feat(requests-board): public song-request voting board (Phase 1) (#975)
## What & why
Phase 1 of the **daily-free-community-track** initiative (BACKLOG:
*"Daily free community track via public voting board"*). A public,
Hacker-News-style voting board at **requests.nomadkaraoke.com** where
anyone signs in with an email magic link, submits a song request
(artist/title auto-corrected + deduped), and casts **one vote per
calendar day**. Publicly viewable so it can be linked from YouTube
descriptions to drive top-of-funnel awareness.
Ships the board first (this PR) so the linkable URL exists; the
automation (daily auto-picker, free-credit grant on pick, 24h ownership
handoff, voter publish-emails, and the trending-karaoke agent as a
fallback request source — one free track/day total) is **Phase 2**.
Design doc: `docs/archive/2026-09-02-requests-voting-board-plan.md`.
## Backend
- New `song_request` models + service (Firestore `song_requests` /
`song_request_votes`) with transactional vote arithmetic (create / move
/ flip / toggle-off), dedupe, and ranking. One-vote-per-day is enforced
structurally via the vote doc id (`{email}__{YYYY-MM-DD}`).
- New `/api/requests-board` router: `GET /requests` (public, annotates
the caller's vote), `POST /requests` (submit — auto-corrects via the
existing `match_judge`), `POST /requests/{id}/vote`, `GET /me`.
- Board sign-in (`purpose="requests_board"`): passwordless identity only
— **no welcome credit**, a **higher per-IP signup cap**
(`BOARD_MAX_SIGNUPS_PER_IP`, default 30) so a shared venue WiFi can
register many voters while the per-fingerprint cap stays strict, and
`redirect_path="/requests"`.
- New `POST /users/claim-welcome-credit` powers the "Make it yourself
now" conversion (grants the standard credit the normal way, idempotent).
## Frontend
- `app/[locale]/requests` board: submit form with "we tidied that to…"
feedback, ranked vote list with daily-vote UX, convert-to-gen CTA,
recently-made list with YouTube links.
- API client methods + types; verify page honors `redirect_path`; new
`requests` i18n namespace translated to **all 33 locales**.
- `requests` added to `nonTenantSubdomains`.
## Testing
- 20 backend unit tests (routes + pure logic + sign-in + board signup
limit) and 8 emulator tests exercising **real Firestore transactions**
(vote arithmetic, dedupe, idempotent resubmit, one-vote-per-day). Full
backend unit suite green (4247 passed); frontend typecheck clean.
- Reviewed locally with CodeRabbit CLI (3 cycles); all findings
addressed (auth headers, idempotent resubmit, signup-limit hardening,
blank-input validation, i18n particle, refresh race).
## Deploy note
The board is reachable at `gen.nomadkaraoke.com/en/requests` on deploy.
Wiring the vanity `requests.nomadkaraoke.com` is a **post-deploy
Cloudflare step** — see
`docs/runbooks/requests-subdomain-cloudflare.md`.
@coderabbitai ignore
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
|
af36975db7 |
Admin create/manage tenants: album batch via tenant bulk flow + full theme customisation (#971)
## Why
Enables the "album for one client" use case: a batch of tracks with
audio provided up front, a client-supplied instrumental per track, and
one shared theme. That pipeline already exists as the **tenant bulk
flow** (folder pick → LLM pairs mixed↔instrumental → resumable uploads →
skip separation → locked theme on every job) — the only missing piece
was an easy way to **mint a tenant** (previously a hand-run
`scripts/setup-*-tenant.py`). This adds admin create **and** manage, so
a client's feedback can be iterated on (edit theme, re-render).
## What
**Backend** (`backend/services/tenant_admin_service.py` +
`backend/api/routes/tenant_admin.py`):
- `POST /api/admin/tenants` (multipart) — mint a tenant: derives a
**self-contained theme** from the default Nomad theme (copies its assets
so every `background_image` basename resolves), applies colour
overrides, overlays uploaded backgrounds, registers the theme, writes
`tenants/{id}/config.json` with B2B defaults (private, no
YouTube/GDrive, locked theme). Reserves the config **create-only before
any theme write**, with rollback on failure.
- `GET /api/admin/tenants` / `GET /api/admin/tenants/{id}` / `GET
/api/admin/tenants/_template` — list, full detail (config + theme
`style_params` + assets), and the default theme JSON as a template.
- `PUT /api/admin/tenants/{id}` (multipart) — **full theme
customisation**: merge config (one level, `id` immutable), replace the
entire `style_params.json`, upload/replace assets (images **or fonts**),
update logo.
- Full `style_params` override accepted on create too. Theme docs
validated to `intro/karaoke/end/cdg`. Guards against a tenant id
colliding with an existing theme.
**Frontend** (`/admin/tenants`):
- Tenants list + **Create** dialog (name→auto-slug, colours,
backgrounds, Dropbox/delivery, logo, + an "Advanced: full theme JSON"
editor seeded from the Nomad template).
- **Manage** dialog: editable config fields + a validated full **theme
JSON editor** + asset list & upload.
- New tenants are driveable immediately via **`?preview_tenant=<id>`** —
no DNS needed.
- Sidebar "Tenants" entry;
`adminApi.listTenants/createTenant/getTenant/updateTenant/getThemeTemplate`.
## Testing
- 36 backend tests (`test_tenant_admin_service.py`,
`test_tenant_admin_route.py`) + 4 frontend tests
(`admin-tenants-page.test.tsx`), incl. atomic-reservation /
conflict-no-clobber / rollback / theme-collision / full-theme-edit.
- Full `make test` green locally (backend + 1245 frontend); typecheck
clean.
- Reviewed locally with CodeRabbit (2 cycles, all findings addressed).
## Notes
- Admin is English-only → no i18n locale files touched.
- Follow-up (out of scope): saved **user** themes in the Custom Video
Style dialog (reuses this theme-derivation core); and an optional
"re-apply theme + re-render existing job" action.
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
|
5d6a6d4f1b |
fix(review): async preview video generation — survive encoder cold-starts (v0.207.0) (#954)
## Summary
Preview videos in the lyrics review UI got stuck on "Generating preview
video..." whenever the GCE encoding worker needed a cold start (job
`326e1aef`, 2026-08-29): the synchronous `POST /preview-video` held the
connection for 2m36s while three zones were stocked out and a fallback
VM booted — Cloudflare cuts browser connections at ~100s, so the UI
never saw the (successful!) response.
### Backend
- `POST /api/review/{job_id}/preview-video` now returns `{"status":
"generating", "preview_hash"}` immediately after the ASS subtitles are
uploaded; encoding runs in a background asyncio task (established
pattern in this file): GCE encode → local render fallback → GCS error
marker (`previews/{hash}.error.json`) if both fail
- Cache short-circuit: the preview hash is deterministic over the
correction data, so an identical retry returns the existing mp4
instantly as `{"status": "success"}`
- New `GET /api/review/{job_id}/preview-video/{hash}/status` poll
endpoint — GCS-backed (mp4/error-marker existence), so polls work across
Cloud Run instances
- Blocking prep work (GCS downloads, ASS generation) moved to
`asyncio.to_thread` so it no longer stalls the event loop
### Frontend
- `PreviewVideoSection` polls the status endpoint (3s interval, 5-min
cap), escalating to a "starting up an encoding server — this can take a
couple of minutes" message after 30s
- Fixed the Retry button: it previously reset state without
re-triggering generation (spun forever) and was hardcoded English; now
works via a retry nonce and is translated
- Removed the dead `worker_starting` branch (a status the backend never
returned)
- New i18n keys (`encoding`, `timedOut`, `retry`) + updated
`startingEncoder` translated for all 33 locales
## Testing
- 9 new backend tests: cache-hit, generating + background spawn +
stale-marker cleanup, status endpoint (ready/error/generating),
background task (GCE success / local fallback / error marker)
- 7 new Jest tests for `PreviewVideoSection`: cache hit,
poll-until-ready, transient poll failure, terminal error stops polling,
timeout, Retry re-runs, missing hash
- Full suites green: backend 4093 ✓, karaoke_gen unit 1779 ✓, frontend
Jest 1203 ✓, `tsc --noEmit` no new errors vs main, translation
`--keys-only` validation ✓
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
||
|
|
f01213f5cb |
Fully-auto review for confident jobs: review_mode framework + auto-approval scorer/executor + replay tool (v0.202.0) (#947)
## Summary Ships the **fully-automated review framework**: jobs now default to `review_mode="auto"` — when the auto-approval scorer is fully confident on BOTH axes, the combined review screens are skipped entirely and the job renders straight through. Everything else falls back to human review exactly as before. **The confident class (deliberately narrow to start, ~5% of jobs expected):** - **Lyrics**: synced reference present, ≥90-99% anchor coverage, and every gap word covered by the pre-generated AI auto-correct suggestions (which the executor applies server-side — a faithful port of the review UI's on-load auto-apply, including conflict-group resolution) — OR zero gaps outright. - **Backing vocals**: non-subjective only — no audible backing content → clean instrumental. - **Never-auto gates**: vocalization sections (da-da-dun/woo-woo — P3), phantom/absurd-duration lines (P8), too-few-words, no/unsynced reference, errored backing analysis. **Calibrated against a 20-job human-review corpus** (all recent admin reviews replayed + annotated): safety 100% (every job that truly needed a human stays in review), lyrics-auto 3/20, overall-auto 1/20. ## How it works - `screens_worker` (pre-AWAITING_REVIEW) and `audio_worker` (post-stems, for the audio-lags-lyrics ordering) both call `maybe_auto_complete_review()`. Every job gets a verdict recorded in `processing_metadata.auto_approval` (shadow data keeps accumulating for calibration); enforcement additionally requires: flag on + `review_mode="auto"` + not made-for-you + no existing/custom instrumental + audio complete. - Enforce path mirrors a human "Complete Review": apply cached AI suggestions → sanity checks → `corrections_updated.json` → `instrumental_selection="clean"` → clear progress keys → `GENERATING_SCREENS→REVIEW_COMPLETE` (new legal edge; no review token or "ready for review" notification is ever issued) → trigger render worker. - **Fail-safe everywhere**: stale suggestions, duplicate-word artifacts (P1 self-conflict signature), state changed mid-enforce (human opened editor), any exception → normal human review. Auto-approval can never fail a job. - Kill switch: `AUTO_APPROVAL_ENFORCE_ENABLED=false` (scoring continues). Per-job: `review_mode="always_review"` (admin PATCH, validated). - Excluded from enforcement (still shadow-scored): made-for-you orders (human QC is part of that product), custom-instrumental jobs. ## Also in this PR (sessions 1-3 of the initiative) - **Review replay tool** (admin-only, prod-inert): `?replay=1` reopens the real lyrics/instrumental review UIs read-only for completed jobs — queue nav, Post-AI/Final toggle, edit-log action panel, dev-only audio byte-proxy (env-gated `REVIEW_AUDIO_PROXY`, path-anchored). This is what produced the calibration corpus. - `auto_approval` service package: scorer (anchor/gap/AI-coverage signals + P3/P8 gate detectors), apply engine, executor, lyrics-diff, offline capture script. ## Testing - 100+ new unit tests across scorer/apply/executor/replay routes; full backend suite 4000+ green locally (8 pre-existing emulator failures reproduce identically on pristine `main` — local-env only, green in CI). Frontend: 1196 Jest tests green. - Scorer validated read-only against the 20 real prod jobs in the private calibration corpus. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
156d4369f9 |
Tenant bulk upload: folder pick + LLM pairing + review table + resumable uploads (v0.200.0) (#943)
## Summary Adds a **Bulk** mode to tenant portals (frog #3, Vocal Star ask): drop a folder of tracks, automation pairs Mixed vs Instrumental audio and extracts Artist/Title from filenames, the operator reviews/fixes an editable table, then submits all as jobs — with uploads that survive slow/flaky connections. ### Bulk mode - `POST /api/tenant/bulk/analyze` — authed, gated on new `TenantFeatures.bulk_upload` (default on), cap 100 audio files. Regex fast path for the `S####-1/-2 … Guide/BV/Instru` convention + Vertex-Gemini pass for leftovers (`backend/services/tenant_bulk/analyze.py`), regex-only fallback if the model errors. Pure analysis — filenames only, no state written. - Label parsing: STRONG labels (guide/instru/bv/backing vocals) override the `-1`/`-2` slot (surfaces the real-world `S1102-2 … Guide` anomaly as mixed-only unpaired); WEAK words (Off/Lead/Vocals/Karaoke) stay in the title unless bracketed. - `TenantBulkFlow.tsx`: folder/file pick → editable review table (reassignable Mixed/Instrumental dropdowns, warnings strip, low-confidence cautions) → bounded-concurrency submit reusing the tenant single-submit sequence. Mixed-only rows blocked + flagged. Jobs stamped `state_data.batch_id` + `created_from='tenant-bulk'`. ### Resilient uploads (opt-in `upload_mode: "resumable"`) - Backend returns **GCS resumable session URIs** (created server-side with the caller's Origin → GCS answers session CORS itself; **no bucket CORS changes**). Signed PUT stays the default for all existing clients. - `frontend/lib/resumable-upload.ts` (dependency-free): 256KiB-aligned adaptive chunks, always-query-offset resume (no persisted byte ever re-sent), stall detection instead of hard timeouts, backoff+jitter cut short by online/visibility events, waiting-offline pause, EWMA throughput + per-row ETA, non-advancing-308 loop guard. - `frontend/lib/upload-recovery.ts`: IndexedDB persistence → after a reload/crash a "Resume unfinished batch?" banner lets the operator re-pick the folder and every upload resumes from its server-side offset against the same jobs (never duplicates). ### Tests - Backend: 26 new (real Bulk-Batch-1 fixture incl. S1102 anomaly, label-in-title regressions, endpoint auth/feature/cap gates, upload_mode contract) - Frontend: 22 new (engine chunk choreography/offset recovery/permanent errors/retry caps; component resumable routing, re-pick recovery, retry-no-duplicate) - Full suite green: backend (incl. emulator) + 1194 frontend - Prod E2E extension of `tenant-happy-path` is a documented follow-up (needs this deployed first) ### Docs - `docs/API.md`: new endpoint + `upload_mode`/`batch_id` fields - Design + decision record: workspace `docs/archive/2026-08-25-tenant-bulk-upload-plan.md` Local CodeRabbit review: 3 completed rounds on this branch, all 9 findings fixed. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
e0e5959edc |
fix(review): surface relevance-filter rejection when pasting lyrics + Add Anyway force option (#937)
## Summary
Pasting reference lyrics in the review UI could silently do nothing: if
the relevance filter (30% anchor-match threshold) rejected the pasted
source during the correction rerun, the `add-lyrics` endpoint still
returned a generic success, so the paste tab cleared the box, closed the
modal, and no reference lyrics appeared — with zero feedback.
Observed in prod on job `0eebb248` ("Jonny Hill & Linda Feller –
Darling"): two "successful" add-lyrics calls, both sources rejected at
1.3% relevance (3/230 words), `reference_lyrics` left empty.
## Changes
- **Backend (`/api/review/{job_id}/add-lyrics`)**: when the source is
dropped by the relevance filter, return `{status: "rejected", rejection:
{relevance, matched_words, total_words}}` and skip persisting the no-op
rerun to GCS. Same change in the local `ReviewServer` (wheel/CLI review
mode), including `force` support it previously lacked.
- **Frontend**: `addLyrics` API client sends the `force` flag (as a
string — the endpoint body is `Dict[str, str]`, which also makes it
deploy-skew-safe). `PasteLyricsTab` keeps the pasted text on rejection,
shows a warning with the match %, and offers an **Add Anyway** button
that retries with `force=true`. Covers both the Add Lyrics modal and the
inline empty-state, which share the tab.
- **i18n**: new `pasteLyrics.rejectedWarning` / `addAnyway` strings
translated to all 33 locales.
- **Docs**: `docs/API.md` add-lyrics section documents the rejected
response.
- **Tests**: backend endpoint tests (success/rejected/force passthrough,
no-upload-on-reject), API client tests (force string, rejected
envelope), and `PasteLyricsTab` component tests (rejection UI, Add
Anyway flow, warning cleared on edit, legacy void result).
## Testing
- `make test`: full backend + frontend suite green (92 suites / 1158
frontend tests; backend 56 review-route tests incl. 3 new).
- Local CodeRabbit review completed; single finding was a pre-existing
translation nitpick (skipped).
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
||
|
|
b5461132d0 |
feat(admin): create users from admin UI + auto-create account on job user_email reassignment (#936)
## Summary
Lets admins provision user accounts before the user has ever logged in —
so we can grant free credits, submit jobs on their behalf, and
impersonate them — and fixes the gap where reassigning a job's
`user_email` to an unknown email left the job orphaned (no account, no
credits, impersonation 404s).
### Changes
- **`POST /api/users/admin/users`** — create a user account directly:
`email` (required), `display_name`, `initial_credits` (0-1000),
`credit_reason`. Returns 409 on duplicate. Initial credits set
`welcome_credits_granted` so the user doesn't also get the automatic
welcome credit on first login.
- **`PATCH /api/admin/jobs/{id}`** — `user_email` is now trimmed +
lowercased (account doc IDs are lowercased emails; mixed case would
silently orphan the job), blank/invalid values are rejected with 400,
and a `gen_users` account is auto-created when none exists. Response
message notes the creation.
- **Admin users page** — "Create User" button + dialog; on success the
list filters to the new user so Impersonate / Add Credits work
immediately.
- **Admin jobs page** — field-save toast surfaces the account-created
message; local state mirrors the backend's email normalization.
### Testing
- New `backend/tests/test_admin_create_user.py` (9 tests) +
`TestUpdateJobUserEmailAutoCreate` (5 tests) in
`test_admin_job_update.py`
- Full backend suite: 3917 passed. Frontend: 1151 Jest tests passed;
eslint/tsc clean on touched files
- CodeRabbit local review: 2 cycles, all findings addressed (PII
redaction, blank-email 400, masked audit log, normalized local state)
Version bump: 0.198.3 → 0.199.0
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
||
|
|
1e99f9e0d5 |
feat(admin): per-job/user locale badge + "emails sent to user" history (#930)
## Summary
Two admin-only features so support/intervention work can (a) know what
language/country a user is using the product in, and (b) see the exact
emails ever sent to a user, rendered as they appeared in the inbox with
full delivery metadata.
### A — Language / country visibility (admin-only)
- New `get_full_locale_from_request()` captures the real UI language
(any of 33, not the narrowed en/es/de email locale) on **every**
job-creation path (URL, upload, audio-search, bulk, made-for-you via
Stripe metadata). Stored as `Job.locale` (+ added to the dashboard
summary projection) and `User.ui_locale` (kept separate from
`User.locale`).
- `LocaleBadge` (flag + language) shown in the admin jobs table, admin
job detail (+ country via the existing `IpInfo` on the creation IP), the
user-detail header, and the customer `JobCard` **gated to admins only**
— regular users never see it.
- Country is derived from IP (existing `ip_geolocation_service`); no new
capture.
### B — "Emails sent to this user"
- Every send routes through one `EmailService._log_and_send` chokepoint
that persists each email to a new Firestore `email_log` collection
(rendered HTML + Postmark MessageID), best-effort so it can never break
or delay a send. Single-use auth/capability tokens (magic-link, review,
referral) are **redacted** before persistence.
- `PostmarkAdminService` merges the Postmark Messages API (last ~45
days: exact HTML + opens/clicks/bounce/delivery status, paginated) with
the Firestore log (permanent), de-duplicated by message id.
- Endpoints `GET /api/admin/users/{email}/emails` and `GET
/api/admin/emails/{id}`; the user-detail "Emails Sent" card opens a
modal rendering the email in a sandboxed `<iframe srcDoc>` with a
metadata side panel.
- New Firestore composite index (`email_log` recipient + created_at)
added to Pulumi.
## Testing
- **41 new backend tests** (locale capture incl. q=0, email logging +
token redaction, PostmarkAdminService incl. pagination + 45-day
fallback, job-locale persistence, admin endpoints) — all pass.
- **10 new frontend tests** (LocaleBadge, UserEmailsCard list +
sandboxed-iframe modal) — all pass.
- Regression sweep on related existing suites: green. Preview-email flow
smoke-tested through the new chokepoint.
## Notes
- `pulumi up` for the new `email_log` index should be applied before
merge (done locally).
- `Job.locale` populates for new jobs going forward; historical jobs
fall back to the submitting user's locale where shown.
Local CodeRabbit review completed and addressed (token redaction,
pagination, row a11y, locale-overwrite guard, q=0 handling).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
@coderabbitai ignore
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
|
84b0683cba |
fix(audio-search): recover parked jobs — edit + re-search + own-audio fallback (#906)
## Problem
In Bulk Mode (and single jobs), the "Choose audio" dialog was a **hard
dead-end**. When a search returned nothing usable — 0 results, a
transient flacfetch failure, or an auto-picked source that's unusable —
the bulk worker parks the job in `AWAITING_AUDIO_SELECTION` with
**empty** cached results. The dialog then showed "No audio sources
found" with **no retry, no way to edit the title, and no manual
source**, so the user was trapped.
Real-world trigger (batch `4af58968a64a`): two Arctic Monkeys tracks
parked with 0 results (one a transient miss on a common track, one an
over-long title), and three more stuck downloading the **same dead RED
torrent** — all with no path forward.
## What this does
Adds a full recovery path to the shared `AudioSearchDialog` (used by
bulk **and** single jobs):
- **Edit + Search again** — always-available editable artist/title with
a re-search button. Edits apply to the job's search terms *and* display
metadata.
- **Provide your own audio** — paste a YouTube/yt-dlp URL, or upload a
file.
- **Graceful empty state** — a 400 (no cached results) renders as the
empty state with the refine bar, not a red error; real network/5xx
errors still surface.
### Backend (`audio_search.py`) — all guarded to
`AWAITING_AUDIO_SELECTION`, tenant-gated, no new job, no double-charge
(final duration billing reconciles post-download)
- `POST /{job_id}/research` — retry / edit-and-re-search. Empty results
are a normal `200`.
- `POST /{job_id}/provide-url` — YouTube URLs go through the **async**
audio-download worker; generic yt-dlp URLs download inline, bounded by a
timeout (never strands a job).
- `POST /{job_id}/attach-upload-url` + `/attach-upload-complete` —
signed-URL upload of the user's own file (extension allow-list +
traversal-safe; the complete step pins the exact uploaded object).
- Factored the triplicated result serialization into
`_build_result_response`.
## Testing
- 20 backend endpoint tests + 5 frontend dialog tests (all green);
broader audio/jobs sweeps pass.
- i18n: 15 new `audioSearch` keys across all 33 locales.
## Review
Local CodeRabbit review completed (2 cycles, all 6 findings addressed,
final pass 0 findings).
@coderabbitai ignore
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
|
c94abbf7f4 |
docs: Cloudflare edge security (WAF + origin lock) — persistent-doc coverage (#889)
Documents the 2026-07-20 edge cutover (shipped in #886 + #888) in the persistent docs. ## Changes - **ARCHITECTURE.md**: Cloudflare edge added to the System Overview request path + new **§ Edge Security (Cloudflare)** (WAF, rate limit, origin lock, SSL "full" rationale). - **LESSONS-LEARNED.md**: the SSL "dragon" resolution (managed cert can't provision behind CF, but "full" mode makes it work + renewal-safe → never use "full (strict)") + Free-plan WAF/rate-limit constraints + origin-lock-without-breaking-internal-callers. - **README.md**: Recent Changes entry + Current State row. - **API.md**: origin-lock note (direct-to-`*.run.app` → 403 without `X-Edge-Auth`). Docs-only; no version bump. @coderabbitai ignore --- 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
|
ad3c0bffc0 |
fix: harden 4 prod error paths (seek non-finite, auto-correct 429, download race + timeout) (#877)
## Summary Investigated and fixed four production error patterns flagged by the error monitor, each with tests. ## Changes **Frontend — `TypeError: currentTime non-finite`** (`/en/app/jobs` review UI) - Guard lyrics-review/audio-editor seek sinks (`AudioPlayer.seekAndPlay`/`handleSeek`, `AudioEditor.seekTo`) and the `handlePlaySegment` dispatcher with `Number.isFinite`; harden `TimelineEditor.handleTimelineClick` against a zero-width container. Restores parity with instrumental-review, which already had these guards. **Backend — auto-correct Vertex `429 RESOURCE_EXHAUSTED` surfaced as generic 502** - Retry model calls with exponential backoff on transient (rate-limit/overload) errors via `_is_transient_model_error`; surface a distinct **retryable 429** instead of 502. Retry env knobs parsed defensively (clamp + warn, never crash at import). **Backend — `audio-download-job` `InvalidStateTransitionError: downloading -> downloading`** - Root cause: a manual `/retry` raced the Cloud Run auto-retry; the losing run's post-download transition raised and the generic `except` `fail_job`'d a job the winner had already advanced. Fix: idempotent transition — re-read status first, treat an already-advanced job as success, and use `raise_on_invalid=False` so a race never fails a live job. **Backend/infra — `audio-download-job` 600s task-timeout SIGKILL** - Internal `wait_for_download` timeout equalled the Cloud Run task timeout, so slow torrents were SIGKILLed before failing gracefully (leaving the orphaned retry that fed the race above). New `DOWNLOAD_WAIT_TIMEOUT_SECONDS=1080` kept strictly below the task timeout, raised `600s → 1200s`. **Infra already applied to prod via `pulumi up` (targeted); stack is clean.** ## Testing - [x] Backend: 122 tests across touched suites pass (auto_correct incl. 10 retry/env, audio_download_worker incl. 3 race/timeout, review API, proactive) - [x] Frontend: full jest suite 1073/1073 pass (incl. new AudioPlayer non-finite guard test); `tsc` clean for changed files - [x] Infra: `pulumi preview` clean (273 unchanged) after applying timeout + reconciling pre-existing divebar drift ## Review - [x] Local CodeRabbit review completed (1 major finding — retry env validation — fixed + tested) - [x] Version bumped 0.190.5 → 0.191.0 @coderabbitai ignore --- 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |