mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
main
2052
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
f4f0f6af77 |
feat(review): only show the review button once the job is fully prepared (v0.298.0) (#1273)
## Summary The review button now appears only once a job is fully prepared: stems separated, auto-approval re-scored with them, AI lyric corrections generated and applied on the server, and review audio + waveforms cached. The browser no longer applies AI corrections by itself, so there is one code path. Prompted by job ae995697 (a Made For You order): the review button showed at 18:29, but the stems, backing verdict and vocal waveform only existed from 18:32, and the server pre-apply had aborted on a duplicate-word check, leaving the browser to apply corrections on load. ## Changes - **Hold every job until stems land** (`review_gate.route_after_scoring`). Before, only jobs whose lyrics would auto-ship were held, so Made For You and lyrics-gated jobs opened review early. Same kill switch (`AUTO_APPROVAL_AUDIO_HOLD_ENABLED`) and 20-minute backstop. - **Generate AI suggestions on the API service before parking.** The hold is released inside the audio-separation job, which has no LLM credentials, so that job only applies an existing cache. - **`enter_human_review` prepares everything before AWAITING_REVIEW**: pre-apply (generating on a cache miss), then `review_assets.prepare_review_assets` (OGG transcodes, backing waveform, vocal peak envelope), then the transition. - **Non-strict pre-apply for human review** (`build_applied_segments(strict=False)`): what "Accept All" produces. Stale/duplicate signals are recorded as `warnings`. The auto-ship path stays strict. - **Removed the browser's on-load auto-run/auto-apply** from `useAutoCorrect`. - **Job card shows "[5/10] Preparing review"** while held (new summary field + 33 locales). - Docs: ARCHITECTURE review-gate step, `docs/archive/2026-11-19-review-ready-gate.md`. Trade-off: Made For You and lyrics-gated jobs reach review (and send the review-ready notification) ~3 minutes later. ## Testing - Backend unit suite: 7294 passed. Locally failing: 8 emulator-integration tests (something unrelated listens on :8080 so they don't skip; CI runs them against real emulators) and 1 locale test that tripped on a gitignored translation snapshot (passes once removed). - New/updated tests: review gate (review-verdict jobs held, kill switch, prepare-before-transition order, suggestions generated before parking, no double wait, missing job), non-strict apply, duplicate-word pre-apply (ae995697 case), review_assets. - Frontend Jest: 1834 passed; no type errors in touched files. - Local CodeRabbit: 2 minor findings fixed, re-review clean. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVVdAxGitbw1FhYP4e4 --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v0.298.0 |
||
|
|
0449d0c107 |
fix(infra): build Cloud Function source archives without __pycache__ (v0.297.4) (#1272)
## Summary
A full `pulumi preview` from main showed runner-manager drift ("replace
source, update function"), last applied from the
`karaoke-gen-runner-scan-retry` worktree. I checked it:
- #1262 (that worktree's branch) is fully on main. The squash merge is
byte-identical to the branch tip. The live scheduler job (`retryCount
2`, `maxRetryDuration 60s`, `attemptDeadline 300s`) also matches the
code exactly.
- The drift comes from the **deployed zip containing
`__pycache__/*.pyc`** (including a pytest pyc) from the worktree's test
run. `pulumi.FileArchive` hashes every file, so every clean checkout saw
a phantom diff. Separately, the 7-day lifecycle rule had already deleted
the bucket object.
## Changes
- New `infrastructure/modules/source_archive.clean_source_archive()`
builds an `AssetArchive` of source files only. It skips `__pycache__`,
`*.pyc`, `.pytest_cache`, other tool caches and `.DS_Store`.
- Used for runner-manager, divebar-lookup and divebar-mirror (the three
`FileArchive` users).
- Unit tests, added to the "Infra - Cloud Function tests" CI job.
- LESSONS-LEARNED entry.
## Applied (before merge, per repo convention)
- I ran a targeted preview with `__pycache__` present in the worktree.
Runner-manager showed source replace + function update (the one-time
correction). Divebar-lookup and divebar-mirror showed **no change**.
- Targeted `pulumi up` ran as claude-admin (Andrew's grant): 2 changes,
289 unchanged.
- The new revision is `github-runner-manager-00031-sam`, and the
deployed zip now holds only the 4 source files.
- The re-preview shows 291 unchanged.
## Testing
- `pytest infrastructure/test_source_archive.py`: 4 passed
- `pytest infrastructure/functions/runner_manager`: 75 passed
- Local CodeRabbit review: no findings
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_012RYLVSfwoV9WyncjqMPHrq
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.297.4
|
||
|
|
2971145c85 |
feat(iam): let claude-readonly read secret values (#1271)
## Why Claude sessions couldn't read secrets such as the backend admin API token. ADC is `claude-readonly@`, which had `secretmanager.viewer` (metadata only). The gcloud CLI's `admin@nomadkaraoke.com` login lapses and needs an interactive re-login, so every documented `gcloud secrets versions access …` command failed in sessions. For example, retrying jobs ae995697/e2f60e2d on 2026-11-19 had to go through a browser session. Andrew asked for `claude-readonly@` to be able to read all secrets. ## Change - Project-level `roles/secretmanager.secretAccessor` for `claude-readonly@`. - The token commands in CLAUDE.md, TESTING.md, TROUBLESHOOTING.md and STORAGE-RETENTION.md now pass `--account=claude-readonly@nomadkaraoke.iam.gserviceaccount.com`, so they work without a fresh gcloud login. ## Trade-off The SA's key is now secret-equivalent: whoever holds it can read every secret and act as backend admin through the API. The key stays on Andrew's Mac only. ## Applied `pulumi up --target <claude-readonly-secretmanager-accessor>`: **+1 created, 290 unchanged**, 2026-11-19. Verified: `claude-readonly@` reads `admin-tokens` (and the API accepts the token) and other secrets. The full `pulumi preview` also shows unrelated drift, not touched here: runner_manager function source (last applied from the `karaoke-gen-runner-scan-retry` worktree) and its scheduler retry config. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVVVVLQSWvtZqBGxRFb --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
1840645faa |
feat(youtube): Firestore catalogue of every channel video with change history (v0.297.3) (#1270)
## Summary Follows #1269. The daily description backfill already fetches every channel video's snippet, status and content details, then throws them away. This saves them to Firestore as a lasting record of the channel, so we can investigate YouTube video activity later. It uses no extra YouTube quota. ## Data model - `youtube_videos/{video_id}`, one doc per video: - from YouTube: title, description, tags, published_at, privacy_status, upload_status, category_id, duration, default_language, made_for_kids, license, embeddable; - parsed from the title: artist, song_title, brand_code, kind, backfill_target; - bookkeeping: first_seen_at, last_seen_at, last_changed_at, in_uploads_playlist, deleted, deleted_at. - `youtube_videos/{id}/history/{auto}`: `{at, event, changes: {field: {old, new}}, source}`. - `event`: first_seen, changed, deleted or restored. - `source`: `backfill` for this job's own rewrites, otherwise `observed`. - If a Studio edit and a backfill rewrite land in the same run, they're logged as two separate entries. - **Soft-delete only.** A known video YouTube stops returning gets `deleted=True` and a `deleted_at`. It stays in the known-ID list and keeps being checked, so if it comes back that's logged as `restored`. - **Writes only on change.** `last_seen_at` is refreshed at most once every 20 hours. A video's doc and its history entries always go in one Firestore batch. - **Best effort:** if the catalogue sync fails, the run logs a WARNING and still succeeds. A dry run skips it. ## Changes - New `backend/services/youtube_video_catalog.py` (100% covered). - `fetch_snippets` also requests `contentDetails`, at no extra quota cost. - The worker passes the metadata from before each rewrite (for attribution), keeps gone IDs in `known_video_ids`, and adds `catalog` stats to the run summary. - README `scripts/youtube-descriptions` documents the catalogue. Version 0.297.3. ## Testing - `pytest backend/tests -k "youtube_comment or backfill or video_catalog"`: 129 passed. - 12 new tests use an in-memory Firestore fake. They cover first seen, unchanged same day and next day, Studio edits, backfill attribution, both kinds of change in one run, soft-delete once, a stub for a video never catalogued, restore, playlist-only changes, and batches never splitting a doc from its history. - After deploy: one manual run on prod to fill the catalogue (expect about 1.9k new docs), then check it. Local CodeRabbit: 2 rounds, 3 minor issues fixed (source attribution, doc and history in one batch, a video first seen and rewritten in the same run). @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVVbErmEFhmiW48uuFK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v0.297.3 |
||
|
|
4a23398c04 |
fix(youtube-backfill): check known video IDs the uploads playlist drops (v0.297.2) (#1269)
## Summary Andrew got "[YouTube Descriptions] 5 updated, 0 remaining" on 2026-11-14. The cause: the channel's uploads playlist (`playlistItems.list` on `UU…`) isn't a complete list of the channel. - Its pages overlap: 1914 items, but 11 are duplicates. - It leaves out public videos for weeks at a time. Five 2023 This Is Me Smiling videos were missing for the whole Oct 22 template cycle, so the backfill reported "complete" while they still had the old description. They came back on 11-14 and were rewritten. - `hSPiNfrqf5w` (Alive in the Chase) is still on the pre-10-22 template, and `QP2lKxkfeqE` is missing from the playlist again right now. ## Changes - `youtube_backfill.iter_all_video_ids` removes duplicate IDs. - New `youtube_backfill.enumerate_channel(youtube, known_ids)`: lists the playlist, then looks up known IDs it left out directly with `videos.list` (about 1 unit per 50 IDs). Returns `recovered_ids` and `gone_ids`. - The worker keeps every ID it has seen in Firestore `youtube_backfill/known_video_ids`. - The first run seeds the set from the quota ledger's `desc-backfill:<id>` entries (1800 IDs on prod, read-only check). - If seeding fails, nothing is saved, so the next run retries. - Deleted videos are dropped from the set. - A dry run saves nothing. - The run summary and progress email report `recovered_from_known`. A WARNING log names the IDs the playlist omitted. - Docs: `scripts/youtube-descriptions/README.md`, `docs/LESSONS-LEARNED.md`. Version 0.297.2. ## Testing - `pytest backend/tests -k "youtube_comment or backfill"`: 114 passed. - New tests cover duplicate removal, recovering videos the playlist omits, gone IDs, seeding from the ledger, a failed seed being retried, no saves on a dry run, and the email count. - Seeding checked against prod Firestore (read-only): 1800 IDs, including all 5 affected videos and Alive in the Chase. - The first prod run after deploy should log "uploads playlist omitted N known videos" and rewrite Alive in the Chase. Local CodeRabbit: 2 rounds. Fixed 1 major (failed seed was never retried) and 1 minor (loose test assertion). @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVVbErmEFhmiW48uuFK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v0.297.2 |
||
|
|
5bf17d520f |
fix(youtube): upload with the current channel description template (#1267)
## Why On 2026-11-10 the daily description backfill sent "[YouTube Descriptions] 1 updated, 0 remaining". The video was `0uM7NClsA_E` (NOMAD-1779, job `58555663`). The NOMAD-1779 admin re-render on 11-09 uploaded it with the description template saved on the job at creation (2026-10-16). The channel template changed on 10-22, so that copy was out of date, and the backfill fixed the video the next morning. ## Change - `get_effective_distribution_for_job` now gives non-private jobs, which all upload to the shared Nomad channel, the **current** `Settings.default_youtube_description`. The job's own `youtube_description_template` is used only if no channel template is configured. This covers first uploads, admin re-renders and redistributes (the orchestrated path). - The legacy `KaraokeFinalise` path in `video_worker.py` now resolves the description through the same function instead of reading the job field directly. - `docs/API.md`: added a note that an admin-edited per-job template no longer reaches YouTube while a channel template is set. The backfill was already overwriting those edits. - Version bumped to 0.297.1. ## Testing - Added tests: the current template wins over an old copy on the job, and the job's copy is used when there is no channel template. Updated the existing non-private test. - Backend suite (non-emulator): 7,255 passed. The only failures need the Firestore emulator (`test_emulator_integration.py`), which wasn't running locally; CI runs that suite. - Local CodeRabbit review: no findings. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVSJbAsXHW4NxxzusWX Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v0.297.1 |
||
|
|
ad64286004 |
feat(tenant): tenant portals are shared workspaces (v0.297.0) (#1266)
## Summary Andrew's expectation: "all jobs in the tenant portal are accessible to everyone who has permission to access that portal." Until now a non-admin only saw/opened jobs whose `user_email` was their own, even on a tenant portal. A client logging in with their own email never saw the 10 jobs bulk-uploaded for them under a separate `@nomadkaraoke.com` address, and re-submitted every song. ## Rule (`backend/services/job_access.py`) - Admins: every job. - Owner: their jobs. - **Tenant portal member** (currently allowed to sign in: `allowed_emails` / `allowed_email_domains`, tenant active): every job of that tenant. Re-checked per request against the live tenant config (~5 min cache), so removing someone from the allowlist removes access. Fails closed on lookup errors. - Consumer jobs (no tenant): owner-only, unchanged. ## Wired into - `_check_job_ownership` (all job routes: view, edit, rerender, delete, …) - full-auth review access (`require_review_auth`: lyrics/instrumental/audio-edit) - file-upload guards (finalize, style uploads; style uploads keep denying ownerless legacy jobs as before) - `GET /api/jobs`: members on a tenant portal list all of that tenant's jobs (still tenant-scoped) - tenant theme refresh/outdated/rerender-outdated scope now covers the whole portal (routes were already member-gated) Unchanged on purpose: kjbox singer review links and the consumer visibility toggle (consumer-only). ## Testing - New `backend/tests/test_tenant_shared_jobs.py` (20): member/domain member/case, outsider, consumer privacy, cross-tenant, inactive tenant, no allowlist, removed member, fail-closed, token-without-email, ownerless per-route, list scoping (member/outsider/consumer portal), review auth 200/403, ownership wrapper. - Updated theme tests that asserted the old owner-only scope; added teammate-can-rerender. - Local: 2,710 job/review/upload/tenant tests + 656 theme/tenant tests pass (only `test_emulator_integration.py` fails locally, which needs the Firestore emulator; same on main). - Local CodeRabbit: 1 finding (ownerless legacy jobs on style uploads) fixed; re-review clean. ## Prod cleanup done alongside (admin API) Deleted (archived) the 10 stale Sept bulk-upload jobs for the randy-vild tenant that duplicated his completed RVILD tracks. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVVUdF2J1bNeVa2uLXA --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v0.297.0 |
||
|
|
13619b7cc9 |
fix(review): preview shows the countdown intro like the final render (v0.296.0) (#1264)
## Summary
Songs whose vocals start within 3s get a "3... 2... 1..." intro in the
final render (`CountdownProcessor`: lyrics +3s, 3s leading silence), but
the **review preview never applied it**. So the preview opened straight
on vocals, and a customer (Randy Vild, "Caught Up Alive" / "Faithful and
True") asked us to add a countdown that was already coming.
## Changes
- `CountdownProcessor.add_countdown_to_lyrics_if_needed` (lyrics-only
half of `process`) + `padding_for_segments_data` (same rule on the
serialized segments the client sends).
- `CorrectionOperations.generate_preview_video` applies the countdown;
full local/CLI renders pad the audio too. Returns
`countdown_padding_seconds`. Countdown previews get a salted hash so
pre-fix cached previews aren't reused.
- GCE preview path: pads the already-downloaded audio on the API side
and uploads it to `previews/{hash}_audio.flac` for the encoder, so **no
encoding-worker change / version skew**.
- `POST /preview-video` responses carry `countdown_padding_seconds`.
- `PreviewVideoSection`: video time = song time + offset, so the
overlaid instrumental stem, waveform playhead and audition seeks stay
aligned (stem parked at 0 during the intro).
- Corrections that already contain the countdown aren't shifted again
but still get padded audio (CodeRabbit finding).
## Testing
- New:
`tests/unit/lyrics_transcriber/output/test_countdown_preview_helpers.py`,
`tests/unit/lyrics_transcriber/correction/test_operations_preview_countdown.py`,
GCE-path endpoint test in `backend/tests/test_routes_review.py`, 4
offset tests in `PreviewVideoSection.test.tsx`.
- Frontend jest (lyrics-review + lib): 892 passed.
- Real render check: local preview of job 508b38b2 (Faithful and True)
from prod corrections + audio with real ffmpeg shows the countdown, and
silence runs to 3.80s, matching the delivered final video's first word
(3.795s).
- Local CodeRabbit review: 2 findings fixed, re-review clean.
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_012RYLVVUdF2J1bNeVa2uLXA
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.296.0
|
||
|
|
ac844b3933 |
ci: pin Playwright jobs to ubuntu-24.04 (chromium unsupported on 26.04) (#1265)
## Summary GitHub's `ubuntu-latest` now resolves to Ubuntu 26.04, where our Playwright (1.57) fails at `npx playwright install --with-deps chromium`: > Error: ERROR: Playwright does not support chromium on ubuntu26.04-x64 That breaks every PR's E2E Smoke / Full / Local-Stack jobs (seen on #1264) and both daily E2E workflows on main (Payment + Happy Path, Tenant Portals failed today, 2026-11-11). ## Change Pins only the 7 jobs that install Playwright browsers to `ubuntu-24.04` (ci.yml ×4, e2e-daily.yml ×2, e2e-tenant-daily.yml ×1). Everything else stays on `ubuntu-latest`. Workflow-only, so no version bump. A Playwright upgrade that supports 26.04 can follow later and drop the pin. ## Testing This PR's own CI runs the pinned E2E jobs. Workflow-only change, no local CodeRabbit review run, so the bot review is left on. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVVUdF2J1bNeVa2uLXA <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Chores** * Updated the operating system used by automated smoke, end-to-end, and daily test runs to Ubuntu 24.04. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
da204bd52e |
fix(ass): split lyric lines wider than the frame (overlapping lyrics, NOMAD-1779) v0.295.1 (#1263)
## Summary NOMAD-1779 (job `58555663`) shipped with two lyric lines drawn on top of each other near the end of the song (YouTube comment: "the lyrics get kinda messed up towards the end"). **Cause:** lines are split by character count (`max_line_length`, 40 for Nomad). Wide-glyph lines like "Mm- mm- mm, mm- …" pass that but measured ~3890px on the 3840px frame. libass wraps an over-wide event by itself; rows are exactly one line-height apart, so the wrapped half landed on the next row's lyrics. **Fix:** `SubtitlesGenerator.generate_ass` now measures each line in the style font, scaled the way libass sizes it (existing `LyricsLine._measure_text` / `ass_font_scale`), and splits any line wider than 94% of the frame at a word boundary (`SegmentResizer.split_to_width`, prefers phrase-ending punctuation, keeps word timing, shares translations). Applies to every ASS render (main, portrait, theme previews). Lines that fit are untouched. **Scope:** of 309 completed branded jobs since 2026-09-10, only 1779 had an overlap; 4 more (1759, 1738, 1766, NOMADNP-0264) had a line wrapping into an empty row. ## Testing - New `tests/unit/lyrics_transcriber/output/test_line_width_split.py`: split logic (fits / splits / phrase break / balance / single long word / timing / translation / singer), ASS generation, and a **real libass render** asserting each lyric event draws as exactly one row. Verified the render test fails with the fix disabled. - Regenerated 1779's ASS from its `corrections_updated.json` with the real theme font and rendered 2:58 with ffmpeg/libass: old ASS reproduces the overlap exactly, new one renders 4 clean lines (widest 3502px). - Locally: 1968 unit + 7231 backend tests pass. Failures were environmental only (shared conda env lacks `flacfetch`: 6 unit + 2 collection errors; `test_emulator_integration.py` fails identically on `origin/main` locally). - CodeRabbit CLI (local, committed): no findings. After deploy: admin re-render of 58555663. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVW1AnhNWu2qGLaNvxu --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
d98e0d6a5e |
fix(runner-manager): retry a repo once on transient GitHub errors before paging (v0.295.1) (#1262)
## Summary The stalled-job scan logged one `RemoteDisconnected` from the GitHub API at ERROR, and the error monitor posted it to #alerts. The next scheduler tick rescans every repo anyway, so one dropped connection doesn't mean the safety net failed. - Network failures (`URLError`, `http.client` errors, `OSError`/timeouts) and GitHub 5xx/429 now get **one immediate retry** for that repo (logged as plain INFO). - If the retry also fails, or the error is not transient (403/404 auth or permission errors), it still logs `Skipping <repo> in stalled-job scan` at **ERROR**, so a broken safety net still pages. #1252 depends on that. - Pulled the per-repo body into `_queued_self_hosted_jobs_in_repo` so it can be retried. ## Testing - `pytest infrastructure/functions/runner_manager -q`: 75 passed, 4 of them new (a retried disconnect doesn't page, two transient failures in a row page, a 403 pages with no retry, 5xx/429 classification). - Local CodeRabbit review: no findings. ## Deploy This is a Cloud Function deployed by Pulumi, so it isn't live after merge until a targeted `pulumi up` runs on the runner-manager function (needs `/gcp-admin`). Alert: https://discord.com/channels/1198691101943541770/1265517528608669707/1569441998656635012 @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVWJ7FwupZXGBp7cE5X Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v0.295.1 |
||
|
|
2db5407a5d |
feat: archive deleted jobs (doc + logs) instead of losing them (v0.295.0) (#1261)
## Why
When NOMAD-1875 was deleted (2026-11-09), its Firestore record and logs
were gone for good, so we couldn't use them to debug the delete-outputs
result afterwards. Andrew's call: job records are **system data**. Keep
them forever, and on account deletion redact the user's identity rather
than deleting the record.
## What
- `FirestoreService.archive_job()` copies `jobs/{id}` to
`jobs_archive/{id}`, adding `archived_at`, `archived_by` and
`archive_reason`. It also copies the `logs` subcollection to
`archived_logs` without `ttl_expiry` (the TTL policy is on the `logs`
collection group, so otherwise they'd expire after 30 days), and the
`audio_edit_sessions` subcollection.
- `JobManager.delete_job()` archives first and **aborts the delete if
the archive fails**.
- Callers pass who deleted the job and why: `admin_delete` /
`owner_delete` from `DELETE /api/jobs/{id}`, and
`cleanup_distribution_delete`.
- **Account deletion** now also redacts archived jobs: owner/customer
email becomes `deleted:<uuid>`, and IPs, notes and contact fields are
removed. These are single-field queries, so no new composite index is
needed. Archived jobs are never deleted.
- Bulk test cleanup (`delete_jobs_by_filter`) still hard-deletes.
- `scripts/get_job.py` falls back to the archive. Documented in
`docs/ARCHITECTURE.md`.
No infra change: Firestore creates the collection on first write, and no
TTL policy is set because archived jobs are kept forever.
## Testing
- Unit: archive happens before delete; the delete is aborted when the
archive fails; archived jobs are redacted on account deletion (including
made-for-you customer jobs).
- Emulator: `backend/tests/emulator/test_job_archive_emulator.py` checks
the doc, logs (no `ttl_expiry`) and audio-edit sessions are copied, and
that a missing job returns False.
- `test_account_deletion`, `test_job_manager`, `test_jobs_api`,
`test_services`, `test_api_integration`: 195 passed.
- Local CodeRabbit: no findings.
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_012RYLVUwKapcjF8czVQrPD5
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.295.0
|
||
|
|
47c6691823 |
fix(admin): delete-outputs dialog shows success as green; YouTube 404 = already deleted (v0.294.3) (#1260)
## Summary
When Andrew deleted NOMAD-1875's outputs (2026-11-09), the result dialog
showed **YouTube, Dropbox and Google Drive all red** ("Partial
Success"). In reality:
- **Dropbox and Drive were deleted fine.** The backend logged `Dropbox:
success, GDrive: success`, and the brand code was recycled, which only
happens when both succeed. The dialog only treats `"deleted"` as green,
but the shared cleanup helpers (`published_outputs_cleanup.py`) return
`"success"`. The regression E2E mocks also used `"deleted"`, which is
why they never caught it.
- **The YouTube video was already gone.** The API returned 404 and
yt-dlp says "removed by the uploader", so it was removed outside gen.
The delete reported this as `failed`, with no message in the dialog.
## Changes
- `frontend/app/admin/jobs/page.tsx`: green for `success`.
`failed`/`partial` now show a reason, and a partial Drive delete lists
the files left behind ("Not deleted: cdg").
- `KaraokeFinalise.delete_youtube_video`: an HTTP 404 counts as already
deleted (warning log, returns True). Other errors (e.g. 403) still fail.
- `frontend/e2e/regression/admin-job-detail.spec.ts`: mocks now use the
real backend statuses, plus a partial-Drive assertion.
- Version 0.294.3.
## Testing
- `tests/unit/test_karaoke_finalise/test_youtube_integration.py -k
delete`: 6 passed (2 new: 404 = success, 403 = still a failure).
- Playwright regression spec: left to CI, since frontend deps aren't
installed in this worktree.
- Local CodeRabbit: 1 minor finding (show per-file Drive outcomes),
fixed.
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_012RYLVUwKapcjF8czVQrPD5
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.294.3
|
||
|
|
d59b5bf3bd |
fix(gdrive-validator): don't report gaps for tracks still uploading (#1259)
## Summary On 2026-11-07 the validator emailed **"SEQUENCE GAPS: CDG missing 1872, MP4-720p missing 1872"**. That was a false alarm, caused by a race: - NOMAD-1871 completed at 12:20:23 UTC. Its delayed post-job check ran at **12:24:47**. - NOMAD-1872 was in `packaging` from 12:23:39 to 12:25:40. Its 4K MP4 was already in Drive, but the 720p MP4 and the CDG were not. - The global-max gap logic saw 1872 in MP4 but not in the other two folders. All three files (`gdrive_files` in Firestore) landed about a minute later, and the later post-job runs (12:29 and 12:36) found no issues. ## Change - `list_folder_contents` can now also collect each file's Drive `createdTime`. - New helper `recently_uploaded_sequences()`: a track with any file created within `IN_FLIGHT_GRACE_MINUTES` (default 30) counts as still uploading, and `validate_files(..., in_flight=...)` leaves it out of the gap check. - These tracks are still covered by their own post-job trigger (the targeted `brand_code` completeness check, which this change doesn't touch) and by the daily 9 PM run. - Docs: `docs/GDRIVE-VALIDATOR.md` (new env var, plus a note on the race in the gap runbook). ## Testing - `infrastructure/functions/gdrive_validator/test_main.py`: 40 passed. 3 new tests cover a track still uploading, an old missing track (still reported), and bad timestamps. - `backend/tests/test_gdrive_validation_delay.py`: 13 passed. - Local CodeRabbit review: 2 minor findings, both fixed (env var parse fallback, offset-less timestamps). ## Deploy This only affects the Cloud Function, so no backend version bump. Deploying it takes `infrastructure/functions/gdrive_validator/deploy.sh`, which needs `/gcp-admin`. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVUwKapcjF8czVQrPD5 --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
bb01e9e36a |
feat(kn-data-sync): live Spotify songs in the song-id export (v0.294.2) (#1258)
## Summary Companion to the karaoke-decide PR that keeps every live Spotify search result forever (`karaoke_decide.spotify_live_tracks`). With this change, those songs join the daily **song-id export**, which feeds: - the kjbox on-device matcher (`song_id.db`); - decide's song-search service; - the mobile app's offline index. So a song we look up once, like "Cole Phillips – Roll of Dice" (a 2025 release missing from the July 2025 Spotify snapshot and from MusicBrainz), is known everywhere from the next build on, with no Spotify or Gemini call. ## Changes - `kn_data_sync/main.py`: `SPOTIFY_LIVE_UNION` is appended to `SONG_ID_SQL` when the table exists. Before decide's first live search creates it, the union is skipped with an INFO log. - Grouped by lowercased artist/title. - Popularity uses the D5a estimate: 0.8 × the artist's popularity (via `art`), capped at 50, else 30. The API no longer returns track popularity. - Docs: API.md. Version 0.294.2. ## Testing - `test_main.py`: union added when the table exists, skipped when it doesn't, four columns and the estimate (9 passed). - **BigQuery dry run** of the full export with the union (inline row in place of the table): valid, ~8.2 GB scanned (unchanged). - Local CodeRabbit review: 0 findings. ## Deploy The function is deployed by hand: `infrastructure/functions/kn_data_sync/deploy.sh` (`gcloud functions deploy`). CI doesn't deploy it, and it needs `/gcp-admin`. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVUQ68KFYtZF7vEfhAq Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v0.294.2 |
||
|
|
c133b64c7d |
fix(quick-version): retry LRCLIB with a cleaned title + honour lyrics overrides (v0.294.1) (#1257)
## Why At a kjbox show on 2026-11-06, the make-it quick version for Sombr "My Body Isn't Ready (Orchestral Version Live From Sound City)" (job `f9764a9f`) rendered with only "(lyrics unavailable)" on screen. The singer heard the backing track with no lyrics. LRCLIB only has this song under the plain title. ## What - `fetch_lrclib_lyrics`: if the exact `/get` and `/search` find nothing, retry `/search` once with the title cleaned (`clean_title`: trailing `(…)`/`[…]` groups and `- Live …`/`- Remaster` style suffixes removed). - A synced hit from the retry keeps its timing only when its length is within **4 s** of the audio, because a different length usually means a different recording (studio vs live). Any other hit is used as plain lyrics (constant crawl). - Plain lyrics found under the exact title still win, and titles with nothing to clean make no extra request. - The quick render now searches with the job's `lyrics_artist`/`lyrics_title` overrides, else the title without a tempo label, the same names the review page uses. The displayed artist/title are unchanged. - Companion: kjbox v0.150.0 (nomadkaraoke/kjbox#314, deployed) lets the KJ add lyrics on the box when a draft still comes back without any. ## Testing - `backend/tests/test_quick_version.py`: 56 pass, including 17 new tests (cleaned-title retry: matching sync, wrong-length sync downgraded to plain, no duration, exact plain wins, failure paths, no retry for plain titles; `clean_title` cases; lyrics-search names incl. tempo label and overrides). - Live LRCLIB check: `fetch_lrclib_lyrics('Sombr', "My Body Isn't Ready (Orchestral Version Live From Sound City)", 224.5)` returns **synced, 37 lines**. Before this change it returned nothing. - Local CodeRabbit review: 0 findings. Full suite left to CI. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVWXLyvTmHNnqQu3rzA Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v0.294.1 |
||
|
|
cd11088be5 |
feat(match-judge): ground AI corrections in live Spotify (v0.294.0) (#1256)
## Summary
At a live kjbox night (2026-11-05, job `f01c9997`) a singer typed "Cole
Phillips – Roll of Dice", a 2025 release missing from our BigQuery
catalogue. Gemini "corrected" it to "Roll of the Dice" twice:
1. in the free-text resolver;
2. in the job form's full judge, after the singer had tapped "keep what
I typed".
Both times the catalogue check found nothing, and the guess still went
out as `confident` ("No catalog match → AI verdict unchanged").
Meanwhile the audio search had found the real track on Spotify at score
1.0.
This PR asks **live Spotify** before trusting Gemini, and **grounds**
every AI correction in real data.
Depends on karaoke-decide's new `GET /api/partner/gen/spotify-search`.
Until that's deployed, the call returns `None` ("no evidence"). That is
safe, but it makes AI corrections non-confident in the meantime, so
**merge and deploy decide first**.
## Changes
- `catalog_proxy_service.spotify_search`: calls decide's partner route
with `X-Partner-Secret` (the existing `decide_partner_secret`). Cached
10 min. Returns `None` when there's no evidence. Never raises.
- `match_judge/service.py`, fast and full passes:
- When the catalogue can't confirm the typed song, an exact live-Spotify
hit is a confident catalog verdict. No Gemini call: "Roll of dice" →
"Roll of Dice".
- Spotify results go to Gemini as candidates.
- **Grounding rule:** a confident AI cosmetic/content verdict must name
a song in the candidates, or found by one Spotify lookup of it.
Otherwise `confident=False`, which both clients already show as "Did you
mean…?" and never apply.
- `match_judge/free_text.py` (`resolve_and_tidy`):
- Live Spotify first, capped at 3 s so the worst case stays inside
decide's 25 s wait.
- A result that *is* the query (either order) answers it directly.
- Otherwise the results go into the Gemini prompt.
- An AI answer the tidy can't confirm is not confident. A timed-out tidy
keeps today's behaviour (`unavailable`, not cached).
- Docs: API.md (judge + resolver), LESSONS-LEARNED entry. Version
0.294.0.
## Testing
- New and updated unit tests:
`tests/services/match_judge/test_service.py` (Spotify confirm in
fast/full, catalogue hit skips Spotify, unconfirmed downgrade with the
Cole Phillips case, second-lookup confirm, any credited artist, Spotify
failure, ambiguous untouched, merge), `test_match_judge_free_text.py`
(exact Spotify answer without Gemini, both word orders, downgrade, AI
pick from live results kept, results in prompt, Spotify timeout/failure
fallback), `tests/services/test_catalog_proxy_spotify.py` (secret
header, cache, no-evidence cases, network error, no secret).
- `make test-backend-unit`: 7212 passed. The 8 failures in
`test_emulator_integration.py` also fail on `main` without the emulator
stack.
- Local CodeRabbit review: no findings.
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_012RYLVUQ68KFYtZF7vEfhAq
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.294.0
|
||
|
|
5c41be1d3b |
fix(error-monitor): ignore runner-manager's other Cloud Run abort wording (#1252)
## Summary On 2026-11-04 at 10:00Z a PR opened on python-audio-separator queued a burst of `workflow_job` webhooks. 22 of 54 deliveries got a 500 from Cloud Run while `github-runner-manager` scaled up. The error monitor alerted because Cloud Run used a second wording, "aborted after waiting too long for an available instance", which the existing ignore pattern didn't match. There was no impact: all self-hosted jobs in the affected runs got runners within about 3.5 minutes and passed. ## Changes - **Error monitor:** match both Cloud Run abort wordings for `github-runner-manager`, but only on **webhook** requests (the bare function URL). Cloud Scheduler ticks (`?action=check_idle`) and entries with no URL still alert. `_SERVICE_URL_IGNORE_PATTERNS` now accepts a predicate on the split URL as well as an exact path. - **Runner manager (Cloud Function):** the ignore relies on the 5-minute stalled-job tick catching dropped webhooks, so that tick's own failures now log as structured `severity: ERROR` and page. That covers a redispatch crash, a repo skipped in the scan, a job given up at 60 min, and a failed re-dispatch. Before this, they were severity-less prints that nothing alerted on. Both changes came from CodeRabbit feedback. - Corrected the pattern's reason: GitHub did not redeliver these (`redelivery: false`). - Version 0.292.1; merged latest main. ## ⚠️ Deploy `infrastructure/functions/runner_manager/` only goes live with a local `pulumi up` (CI never runs Pulumi). Apply it **before** merging, so the safety net's alerting is live before the webhook-abort ignore ships with the backend deploy. ## Testing - `tests/unit/services/error_monitor/`: 452 passed (webhook, scheduler-tick, no-URL and other-service cases for both wordings). - `infrastructure/functions/runner_manager`: 71 passed. The 4 updated tests fail against main's `ephemeral.py`. - Full `make test` not run locally; relying on CI. Alert: https://discord.com/channels/1198691101943541770/1265517528608669707/1567479055887114363 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVVTKcWCcDArTQrjS82 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Bug Fixes** * Runner-manager webhook errors are filtered more precisely. Recognized abort messages are ignored only for matching webhook requests; scheduler-tick errors, requests with other URLs, and requests without a URL continue to trigger alerts. * Runner-manager job-processing failures are recorded as structured error logs, including stalled-job scans and dispatch failures. Give-up logs now report the configured limit rather than the job’s current time in the queue. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v0.293.1 |
||
|
|
a1cbe1c0d4 |
feat(encoding): retire the GCE c4d pair; run on AWS + c2df only (0.293.0) (#1255)
## Why Since 2026-10-31 the AWS EC2 worker (`aws-encoding-worker-1`) has handled every encode and distribution job. 27 jobs went to it, with no Cloud Run fallbacks, no `encoding_stuck`, and nothing encoding-related in #alerts. The c4d Spot pair (`encoding-worker-a`/`-b`) now only costs ~$8/mo of Hyperdisk. `encoding-worker-fallback-c2df` stays as the GCP fallback. ## What Deleting the pair needs code changes first, because several paths assumed it exists: - **`encoding_worker_manager`** - The primary is optional: an empty `primary_vm` means none. - `ranked_candidates` skips a missing primary and always sets `is_primary`, so with no primary a successful start records the AWS/c2df override. Previously the legacy index-0 rule would have cleared it. - `ensure_primary_running` and `ensure_preferred_running` start a ranked candidate. If there are no candidates they raise `NO_CANDIDATES`. - `demote_active_worker` looks up the override's `machine_type`, so the AWS box can now be cooled down. It then clears the override so the retry goes elsewhere. - **`encoding_service`** - When `active_url` is empty, it targets the top-ranked candidate. - It also invalidates its cached URL after a single-candidate fallback. - **CI deploy (`ci.yml` + `deploy_promote.py`)** - While AWS serves, a c2df green is validate-only: it writes `validated_version`, keeps the AWS override, and stops the green. Before this, it would have replaced the AWS override with c2df. - The worker-change gate now diffs against `validated_version`, because `primary_version` stops moving. - An in-place restart of a serving override first waits for it to drain (15-min cap). If it is still busy, the restart is skipped. - **Idle function:** a missing VM is logged as a WARNING returning `NOT_FOUND`, not an ERROR. - **Error monitor:** watches c2df instead of a/b. - **Pulumi:** drops the a/b VMs and static IPs. The prod preview shows exactly 4 deletes. A separate, unrelated `youtube-comment-autoreply-scheduler` drift shows up too; it will not be applied. - **Docs:** CLAUDE.md, ARCHITECTURE, DEVELOPMENT and TROUBLESHOOTING. ## Rollout (after merge + deploy) 1. Back up `config/encoding-worker`, then set `primary_vm`, `primary_ip`, `secondary_vm` and `secondary_ip` to `""`. 2. Apply `pulumi destroy --target` on the 4 a/b URNs only. 3. Redeploy the idle function with `deploy.sh`. 4. Verify: one prod encode on AWS, warmup, and the logs. **Rollback:** in Firestore, delete `speed_rank_overrides` and set `aws_video_worker=false`. Restoring the pair would need a revert plus `pulumi up`, which gives new static IPs. ## Testing - New `backend/tests/test_encoding_worker_no_primary.py` (13 tests), plus new deploy_promote, idle-function and error-monitor tests. - Full backend suite: 7188 passed. The 8 `test_emulator_integration` failures also fail on main, because of a local emulator from another session. - CodeRabbit CLI: no findings. An independent review found 2 issues, both fixed: the demote retry path, and the in-place restart drain. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVTkatGnrYFhCzq4PTe @coderabbitai ignore Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v0.293.0 |
||
|
|
d63c9781e1 |
feat(replay): sync-triage verdict bar + vocal waveforms for stems-purged jobs (v0.292.0) (#1254)
## Summary
Andrew found the standalone triage page unintuitive, with no way to scan
visually. So the **real review UI in replay mode** becomes the triage
tool for the word-timing reference set (full-auto v2): anchors, gaps,
waveforms and audio he already knows.
## Changes
- **`ReplaySyncVerdict` bar** (replay only, admin): Perfectly synced /
Not perfect, reason chips, notes, Save / **Save & next job** through the
existing `queue` URL param.
- It shows the saved verdict and flags unsaved changes.
- It saves via `POST /api/review/{id}/sync-reference`, the same store as
the review-modal checkbox from #1242, which now also accepts `reasons`
(trimmed, max 10).
- **Vocal waveform for old jobs:** storage retention purges stems after
30 days, so replaying old jobs showed no waveform. `_review_vocals_path`
now falls back to the backfilled `training/lead_vocals/{job_id}.flac`,
but only for stems-purged jobs. It's used by the vocals stream and
vocals-peaks.
- Docs: REPLAY.md § Sync triage.
## Testing
- Backend: reasons stored trimmed and capped; waveform fallback
(registered stem preferred, purged → training copy, unpurged never
checks storage). `test_routes_review` + `test_derived_vocals` +
sync-reference tests: 143 pass.
- Frontend: `ReplaySyncVerdict.test.tsx` (save & next, reasons + notes,
saved state, unsaved flag, disabled until chosen): 5 pass. tsc shows no
new errors.
- **Manual:** local replay (read-only against prod data) on a Feb-2026
stems-purged job (83414c76). The verdict bar renders, and Waveforms mode
shows the vocal waveform via the fallback (vocals-peaks 200).
- CodeRabbit ran locally; its one finding (unsaved state) is fixed.
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_012RYLVVVPsvPbXFhRYY5VSJ
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.292.0
|
||
|
|
394def0b53 |
fix(youtube): get source Opus from Tornado reliably; keep signed URLs out of logs (0.291.2) (#1253)
The first full pipeline run with Tornado (job 586644dd) delivered 128k AAC rather than Opus. `audio_only` + `format: opus`, which #1247 relied on, is unreliable on our plan: `Tk3ty5ansqs` returned Opus on 11-03 and AAC on 11-05, and 4 of 5 videos returned AAC with a `reencoding_notice`. Alert: https://discord.com/channels/1198691101943541770/1265517528608669707/1567241547739570298 ## Fix Request `format: webm, max_resolution: 360` and strip the video with `ffmpeg -c:a copy`. All 3 videos tried this way carried Opus, and billing was identical to audio-only on the videos compared (e.g. 21,026,831 bytes for `dQw4w9WgXcQ`). Checked against a real flacfetch job input (`Tk3ty5ansqs`): same audio stream MD5, same `.webm` container, 5,531,092 vs 5,529,792 bytes. Also stops httpx logging signed delivery URLs at INFO. They were appearing in job logs for 586644dd. ## Testing - 83 mocked tests pass, including a real ffmpeg demux test (audio packets unchanged, video dropped) and a failure test. - Not yet run through the deployed pipeline with this format; that test follows this merge. ## Left as is Local CodeRabbit flagged that cancelling during demux doesn't stop the ffmpeg thread. Skipped: the copy takes under a second, both files are in a temp directory that gets discarded, and nothing else reads them. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVTvqXyL4QyA2EjzQMG Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v0.291.2 |
||
|
|
688ca162f8 |
infra: give audio-download-job the Tornado key (0.291.1) (#1251)
`audio-download-job` does most YouTube downloads, and it gets secrets from Pulumi rather than CI's API deploy, so #1247's wiring alone would have left Tornado off for them. Already applied to prod with `gcloud run jobs update --update-secrets` (env diff checked: one variable added). This records it in Pulumi so a later apply doesn't remove it. Pulumi preview couldn't run from Andrew's Mac: its GCP provider fails with `socket is not connected` on IPv6 token requests, while `gcloud` works. Run a targeted preview later to confirm no diff: `scripts/gcp-admin pulumi preview --stack prod --diff --target 'urn:pulumi:prod::karaoke-gen-infrastructure::gcp:cloudrunv2/job:Job::audio-download-job-us-east4'` Merging also triggers the API deploy, which now finds secret `tornado-api-key` and wires it there. No local CodeRabbit review on this small config change, so the bot may review it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVTvqXyL4QyA2EjzQMG <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Bug Fixes** * Updated credential availability for audio download processing. * **Documentation** * Clarified how the Tornado API credential is provided to the API service and audio download processing. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v0.291.1 |
||
|
|
ca05245469 |
feat(youtube): TornadoAPI first for YouTube downloads, flacfetch as fallback (v0.291.0) (#1247)
YouTube downloads depended on one logged-in cookie session on one VM. This tries TornadoAPI first and keeps flacfetch as the fallback. Plan: workspace `docs/archive/2026-11-03-tornado-youtube-download-plan.md`. Alert: https://discord.com/channels/1198691101943541770/1265517528608669707/1567241547739570298 Alert: https://discord.com/channels/1198691101943541770/1265517528608669707/1567252495535841386 Alert: https://discord.com/channels/1198691101943541770/1265517528608669707/1567252497620537415 ## Inactive until the secret exists Nothing changes in prod when this merges. The deploy adds `TORNADO_API_KEY` only once secret `tornado-api-key` has a version, and creating it needs `/gcp-admin`. `YOUTUBE_DOWNLOAD_PROVIDER=flacfetch` switches it off. ## What changed - **`tornado_client.py`:** create job, poll with a 180s deadline (cancels on timeout), stream the delivered file, delete it from Tornado. Retries with one idempotency key per download; honours `Retry-After`. - **`YouTubeDownloadService`:** Tornado first for YouTube, then flacfetch. WARNING per fallback, ERROR on 401/402/403, so an outage or lapsed billing is visible. Non-YouTube URLs stay on flacfetch. - **Availability check:** Tornado metadata first, because flacfetch's check rejects valid videos while its cookies are bot-blocked. If the lookup fails, flacfetch decides as before. ## Spike results (real API, 11 jobs) - `audio_only` + `format: opus` gives **packet-identical audio** to flacfetch's real job input (`Tk3ty5ansqs`, stream MD5 match). `audio_codec: copy` or no `format` gives 128k AAC instead, so the request omits it. - `mBJivIsN0n0`, which bot-blocked job 526bf46c, completed in about 5s of processing. - An age-restricted video completed without cookies; a missing video ends in `Warning`. - Billed about 25.6 MB for a 4-minute song. ## Testing - 79 mocked tests pass (`test_tornado_client.py`, `test_youtube_download_service.py`). - Live run through `YouTubeDownloadService` with the GCS upload stubbed, using a URL with a playlist parameter: only the single video was sent. - Local CodeRabbit: all findings fixed; the last small fix wasn't re-reviewed. ## Not verified - **A full pipeline job with this input.** It can only run once the secret exists. Files are Ogg Opus named `.ogg` (flacfetch's were WebM Opus), so the codec is the same but the container is new to this path. Run the daily E2E or one admin job first. - **Deviation from the plan:** the Tornado job id is logged but not saved on the gen job, because a restart loses the in-request download on the flacfetch path too. A restart can orphan one job, which costs a fraction of a cent. - **CI's secret check:** it depends on the deploy service account being able to read the secret; the log line `tornado-api-key wired:` shows the result. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVTvqXyL4QyA2EjzQMG Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v0.291.0 |
||
|
|
3a563ab3e6 |
fix(jobs): main is failing: restore the start-review guard removed twice (0.290.4) (#1250)
#1248 and #1249 were opened at the same time to resolve the same duplicate guard. Each removed a different one, so `start_review` on main now has none: 6 cases in `test_jobs_api.py` fail, no deploy can finish, and a stale push on a finished song would 500 again. Restores #1243's guard, as #1248 intended, and the test assertion #1249 loosened. `backend/tests/test_jobs_api.py` passes locally. **Other sessions: please leave this route alone until this merges.** No local CodeRabbit review on this small unblocking fix, so the bot may review it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVTvqXyL4QyA2EjzQMG Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v0.290.4 |
||
|
|
dd09aab941 |
fix(jobs): main is failing: resolve duplicate start-review guards (0.290.3) (#1249)
#1243 and #1246 both added a 409 guard to `start_review` and merged minutes apart. #1243's test expects its own message, but #1246's guard now runs first, so main's Backend Unit Tests fail (3 cases) and no deploy can finish. Keeps #1246's guard, because #1243's would refuse statuses that have a valid transition to `in_review`. The race handler is unchanged. The test still checks 409, no transition, and no enum text in the message. `backend/tests/test_jobs_api.py` passes locally. No local CodeRabbit review on this small unblocking fix, so the bot may review it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVTvqXyL4QyA2EjzQMG <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Bug Fixes** * Review requests for jobs that cannot enter review continue to receive a conflict response. Eligible jobs can proceed into review, with review-start actions recorded only when a job transitions from awaiting review. * Updated the conflict-response test to check that internal status naming is not exposed. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
5ae9d407b2 |
chore: remove duplicate start-review 409 guard (v0.290.3) (#1248)
#1243 and #1246 merged within minutes of each other and both added a 409 to `start-review`. This removes the one from #1246. The guard from #1243 stays, since it also handles losing the race to auto-approval. No behaviour change for a finished job, and the #1246 test for it stays. The two guards differ for statuses like `lyrics_complete`, which could technically move to `in_review`: under the #1243 guard those already get a 409. Docs-free, small diff; no local CodeRabbit run, so the bot can review it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVTajJiZnzPUt9qsVEJ <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Bug Fixes** * Starting a review is now limited to jobs awaiting review or already in review; other job statuses receive a conflict response. * Repeated requests to open an eligible review remain safe, and concurrent state changes are handled using the job’s current status. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v0.290.3 |
||
|
|
a10e548606 |
fix(url-jobs): retry a YouTube bot block, then point to Audio Search (v0.290.1) (#1244)
Implements fix C from section 4 of the 2026-11-03 alerts triage handoff. Job 526bf46c, from a paying customer, failed within 3s of a YouTube bot wall and showed the raw yt-dlp error. The credit was refunded, and the customer then uploaded an MP3. ## Changes - **Download:** `create-from-url` retries a bot-blocked download once after 30s. Other download errors behave as before. - **Still blocked:** the job fails with a translated message pointing to Audio Search. The raw error goes in `error_details`. The response stays a 500, because the frontend treats 503 as a transient backend blip. - **Pre-check:** a bot-blocked availability check now waits 30s and checks again before rejecting. No job or credit exists yet at that point. A repeat block returns the same translated message instead of the old hardcoded English. CodeRabbit raised this: without it, the retry never ran when the pre-check caught the block. - **Logging:** the bot-block paths log WARNING rather than ERROR, because `fail_job` already raises the alert for the failure. `audio_search.py` has its own bot-block handling, which this PR doesn't change. The monitor and start-review changes are in a separate PR. ## Depends on flacfetch The retries only help much with https://github.com/nomadkaraoke/flacfetch/pull/61 deployed, since that's what requests fresh cookies straight away. Without it, new cookies only arrive from the keeper's 30-minute probe, so a 30s wait rarely helps. This PR is safe to deploy first, but it adds up to 30s before the same failure. ## Testing - New `test_create_from_url_bot_block.py`, written test-first: retries, final failure, pre-check, non-bot errors and all 33 locales. - Backend tests selected with `-k "url or youtube or audio_search or i18n or translation"`: 785 passed, 5 skipped. Emulator and integration tests excluded. I did not run the full `make test`, so CI is the full check. - Backend translation key parity passes. - Local CodeRabbit review: 1 finding, fixed. The fresh re-review had no new findings. **Not verified:** - Nothing has run in production, and I created no test jobs. - **The 32 new translations were written by hand, not by `translate.py`.** gcloud auth had expired, so the Gemini key wasn't available. To replace them, run `translate.py` for that key with a valid login. - ruff reports F823 for `asyncio` at about line 647 of `file_upload.py`. It's also on main, in a different function, and I left it alone. Alert: https://discord.com/channels/1198691101943541770/1265517528608669707/1567241547739570298 Alert: https://discord.com/channels/1198691101943541770/1265517528608669707/1567252495535841386 Alert: https://discord.com/channels/1198691101943541770/1265517528608669707/1567252497620537415 @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVTvqXyL4QyA2EjzQMG --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v0.290.2 |
||
|
|
08b146fce0 |
fix: no review push before AI suggestions are scored; start-review 409 after review ends (v0.290.1) (#1246)
Andrew's job `7f89f884` (2026-11-03) got a "review your lyrics" push, then auto-approved about 3.7 minutes later. Tapping the push afterwards left the app on its loader. **False review push.** The screens worker's first score ran at 00:15:40, before the proactive auto-correct finished at 00:15:46. Without suggestions the lyrics scored `needs-review`, so `hold_for_audio` was false and review opened with its push. The audio worker's re-score then saw the suggestions (`auto/ai-resolved`) and auto-approved. In the last 14 days 5 jobs auto-approved, and 4 of them opened review first (typically about 3.7 minutes before). `route_after_scoring` now waits for the suggestions and scores again when the first pass ran without them, then holds or auto-completes. If suggestions can't be had, or the re-score reaches no verdict, the first verdict stands and review opens as before. **start-review on a finished job.** `complete -> in_review` raised an invalid transition: a 500, an ERROR log (4 in prod for this job, which the error monitor can turn into an #alerts message), and the app retrying as if it were transient. It now returns 409 without attempting the transition. The app's handling is in nomad-mobile #44. **Tests:** `test_review_gate.py` and `test_jobs_api.py` pass with new cases. `test_internal_api.py` has 7 auth test failures locally that also fail on the main clone, so they aren't from this change. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVTajJiZnzPUt9qsVEJ Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
70bdd0e693 |
fix(alerts): start-review 409, per-cause crash patterns, one alert per job failure (v0.290.1) (#1243)
Section 1 of the 2026-11-03 #alerts triage handoff: a stale push on a
finished song got a 500, and the error monitor could stay silent on a
new crash.
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566980744448843886
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566980746394996879
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566980747905073224
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1565697725280948294
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1565803187334815775
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1565818418072916110
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566225593354420226
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566256141074169987
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567207141994135654
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567252495535841386
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567252497620537415
## What changed
**1a. start-review:** a job not in review gets 409
(`jobs.jobNotInReviewState`) with no transition attempted. Losing the
auto-approver race is a 409 too. `correction-data` has the same race,
and its `except Exception` turned it into a 500; it now returns 409
(`except HTTPException: raise` added). Its status gate still returns
400, as before.
**1b. Crash patterns:** `Exception in ASGI application` and
`request_audit_error` become `Unhandled exception: <Type: message>`,
read from the entry's `exception` field. The handoff's suspicion is
confirmed in `upsert_pattern`: an existing `new` pattern returns
`is_new=False`, so a second, different crash stayed silent. Both log
lines of one crash now share one pattern. The path and method go in the
sample only.
I read the line after the last frame rather than the traceback's last
line. pydantic's last line is a docs link shared by every failure, and
group tracebacks end in a separator.
**1c. One alert per failure:**
- YouTube ids are normalized out of signatures, which also fixes the
false "first time" line in `ops_alerts`.
- `Job <id> failed` (both call sites in `job_manager`) logs WARNING,
since the status write sends the job-failed alert.
- `URL download failed:` is ignored in the monitor, not demoted, because
`file_upload.py` belongs to the section 4 PR.
- `[job:<id>] Download failed` logs WARNING for the expected
`DownloadError`/`FlacfetchServiceError` types. This wasn't named in the
handoff, but it is the same duplicate on the audio-search path.
Unexpected exceptions there stay ERROR with a traceback.
**1d. Suppressions and demotions:**
- Zalo's `zaloJSV2 is not defined`: dropped at `/api/client-errors`,
only with a Zalo UA.
- `python -c` experiments on `audio-separation-job`: ignored, using the
handoff's regex.
- Spotify 403 on `/lyrics`: WARNING with `response.text[:200]`. 401 on
`/lyrics` and 401/403 on `/search` stay ERROR.
- Dropbox token probe: WARNING. A rejected refresh used to log nothing,
so it now logs ERROR, as does having no refresh credentials.
- decide-search health aborts: this was a small change. `should_ignore`
takes an optional `request_url` and matches the exact path
`/api/search/health`.
**1e. Cold start:** `anyio._backends._asyncio` is added to
`_LAZY_ROUTE_IMPORTS`. The prod stack from 10-30 confirms the stall was
in that import. Lifespan also sends one `/api/health` request through
the app in a worker thread, bounded by the existing 30s timeout and
best-effort.
## Testing
- TDD: new tests failed first, then passed.
- Affected suites: 826 passed.
- `backend/tests` (excluding `backend/tests/emulator`): 7088 passed, 8
failed, 29 skipped. All 8 are in `test_emulator_integration.py`, and
that file also shows 8 failures at the base commit
v0.290.1
|
||
|
|
ea69386e50 |
fix(ci): Java 25 for the Firestore emulator (#1245)
`google-cloud-cli-firestore-emulator` 592.0.0 is compiled for Java 25 (class file 69), and the package is installed unpinned from apt, so CI's Java 21 fails with `UnsupportedClassVersionError`. This blocks every PR: Backend Emulator Integration and Local Stack E2E both failed on #1243 and #1244 (runs 41705112128, 41706068761). Main last passed 11-02. CI-only change; this PR's own run is the test. @coderabbitai ignore not added: no local review was run on this one-line CI change, so the bot may review it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVTvqXyL4QyA2EjzQMG Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
a8f9618a07 |
feat(review): admin "perfectly synced" mark for the word-timing reference set (v0.290.0) (#1242)
## Summary
Andrew's review depth varies: Thursday live-event jobs often get only a
quick review. Treating every admin review as perfect sync truth
therefore poisons aligner training and evaluation (full-auto v2). This
adds an explicit, admin-only **"perfectly synced"** mark in the lyrics
review. Ticked jobs join the word-timing reference set; unticked means
unverified. Andrew's definition: a confident spot-check (anchors +
waveforms), not necessarily a full listen.
## Changes
- **Backend:** new `POST /api/review/{job_id}/sync-reference`
(`require_admin`).
- Stores `state_data.sync_reference = {verified, by, at, notes}`.
- `verified: true` needs saved review output: the `file_urls` pointer,
or the direct GCS path for older jobs. Otherwise it returns 409.
Revoking is always allowed.
- **`correction-data`** returns `sync_reference`, so a re-review shows
the existing mark and can revoke it.
- **Frontend:**
- The review modal shows the checkbox plus an optional notes field, to
admins only.
- The modal passes the mark to `onSubmit`. `LyricsAnalyzer` saves it
**after** `submitCorrections` succeeds, so a mark can't sit on unsaved
timing. A failure shows a toast and never blocks the review.
- Once the box is touched, its current state is always sent, so it can
be revoked.
- **i18n:** 4 new strings × 33 locales.
- **Docs:** API.md. Version 0.290.0.
## Testing
- Backend: `test_review_sync_reference.py` (admin store, notes trim,
404, 422, 403 non-admin, 409 without review, direct-path fallback).
Related review suites pass (29).
- Frontend: `ReviewChangesModal.syncReference.test.tsx` (hidden for
non-admin, tick passes the mark, untick sends nothing, saved mark
pre-ticks and revokes, touched-then-unticked sends `false`). The modal
suites pass (29). tsc shows no new errors (21 pre-existing test-file
errors, the same on main).
- **CodeRabbit:** ran locally for 3 cycles and fixed 6 findings. One was
left by design: a pre-ticked mark on an edited re-review is visible in
the modal Andrew confirms, so submitting re-affirms it. The only user is
Andrew.
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_012RYLVVVPsvPbXFhRYY5VSJ
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.290.0
|
||
|
|
452a8a3e28 |
fix(encoding): stems-ready resumes reach the AWS video worker (v0.289.1) (#1241)
## Summary Found by the prod E2E on 2026-10-31. - With `aws_video_worker` on, the render worker dispatched the video worker to AWS, and it parked waiting for stems. - The **stems-ready resume** came from `audio-separation-job`, which has no `ENCODING_WORKER_FALLBACK_VMS` env. It never saw the AWS worker, so it silently started the Cloud Run job. - The job completed fine, because the Cloud Run job is the fallback, but its distribution ran from GCP. ## Changes - `EncodingWorkerManager.fallback_entries()`: in production, when the env var is missing, read the `encoding-worker-fallback-vms` secret (cached per process by `get_secret`). This affects the audio, lyrics and download jobs, which also trigger and warm the encoder. Outside production it never fetches. - `_trigger_video_worker_on_aws`: logs when the AWS dispatch is skipped because AWS isn't the top-ranked worker, so it can't fail silently again. ## Testing - 3 new tests: secret fallback in prod, no fetch outside prod, env preferred over the secret. All 171 related tests pass. - Local CodeRabbit: no findings. - After deploy: another prod E2E to confirm the resume reaches AWS. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v0.289.1 |
||
|
|
78264fa23a |
feat(encoding): run the video worker (distribution) on the AWS box (v0.289.0) (#1240)
## Summary Phase 2 of moving encoding to AWS; Phase 1 was #1236 and #1239. Design and rationale are in `docs/archive/2026-10-29-aws-encoding-worker-plan.md` § Phase 2. When Firestore `config/encoding-worker.aws_video_worker` is true **and** the AWS worker ranks first, `trigger_video_worker` runs the video worker **on the AWS box** instead of starting the Cloud Run job `video-encoding-job`. - **Uploads move to AWS:** the Dropbox/YouTube/Drive uploads (~187 GB/mo, ~$18/mo of Cloud Run internet egress) leave from AWS, on credits. - **No Cloud Run CPU** for the job (~$11/mo). - **Off by default.** Any failure falls back to the Cloud Run job, so this path can't drop a job. ## Changes - **`POST /run-worker` (worker):** - Spawns `python -m backend.workers.video_worker --job-id …` with the box's video-worker env. - Status is reported via `/status/{id}`. - A run counts in `/health` `active_jobs` from the moment it's accepted, so the idle watchdog won't stop the box mid-run. - Returns 503 unless the env file exists. - `job_id` is pattern-validated. - Any startup failure marks the run failed. - **`backend/services/local_encoded_outputs.py`:** a same-box handoff. - When the co-located video worker asks (`stash_local_outputs`, set only where `LOCAL_ENCODED_OUTPUTS_DIR` exists), the encoder keeps local copies of the finals, keyed by blob path. - The video worker takes those instead of downloading them back out of GCS. Without this, the move would *add* ~400 MB/job of GCS egress. - Copies are made off the event loop and purged after 6 h. - No change on GCE or Cloud Run. - **`video-worker-env.py` (box, every boot):** - Mirrors the **live** `video-encoding-job` env, with secrets resolved, so nothing is hand-copied and nothing drifts. - Runs as the job's identity, `karaoke-backend@`, via a second WIF binding (Pulumi `aws-encoding-worker-wif-backend-user`). - On failure it writes no env file, so `/run-worker` answers 503 and the Cloud Run job is used. - **`WorkerService._trigger_video_worker_on_aws`:** checks the flag and that the AWS worker ranks first, ensures it's running, waits for health, then POSTs `/run-worker`. Returns False on any doubt. - **Also:** `startup-aws.sh` creates the backend credential config and runs the env helper; CI uploads `video-worker-env.py`; `worker_code_paths.txt` now gates `local_encoded_outputs.py`. ## Rollout (needs `/gcp-admin`) 1. **Before merge:** targeted `pulumi up` of the backend WIF binding. It's additive. 2. **After stage B of Phase 1** (AWS worker preferred, timings verified): stop/start the box once so the new `startup-aws.sh` writes the env, then set `aws_video_worker: true`. Set it to `false` to roll back. ## Testing - **27 new tests:** - handoff stash/take/purge, path-escape guard, and the per-request stash flag; - `/run-worker`: disabled, bad module, dedupe, active immediately, spawn args/env, failure log tail, wheel-not-ready, safe start, unsafe ids; - env builder; - the trigger's AWS branch: off, not first, happy path, didn't start, refusals, errors. - **Full backend + unit suites:** no new failures (the 13 local failures are identical on main). - **Local CodeRabbit:** 5 findings, all fixed. The code-paths gate test also caught a missing worker path, now fixed. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v0.289.0 |
||
|
|
20b957958c |
infra(aws): encoding worker on-demand c7a.4xlarge; stage A status (#1239)
Follow-up to #1236. - **Instance type:** the AWS encoding worker is now **on-demand c7a.4xlarge** instead of Spot c7a.8xlarge. - The Spot create failed with `InsufficientInstanceCapacity`, and Spot placement scores were 1-3/10 for c7a/c7i/c6a/c8a 4-8xlarge across us-east-1/2. - A worker that stops and starts on demand has to win capacity on every start. - Cost is ~$82/mo of credits. The 4xlarge matched or beat GCE in the replay benchmark. - **Already applied** (instance `i-0c21505d7017356bb`, EIP 3.128.70.236). - **Docs and plan:** stage A status, plus the live end-to-end test through the worker HTTP API (render 117s; all 9 outputs; portrait SSIM 0.993, the only difference being anti-aliasing of the header text). The only code-file change is a comment in `encoding_worker_preference.py`, so there is no version bump. No local CodeRabbit run (IaC and docs only), so the bot review stays on. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Operational Updates** * The AWS encoding worker now runs on an on-demand `c7a.4xlarge` instance rather than a Spot `c7a.8xlarge`. * The worker stops when the operating system shuts down, while its root volume is preserved. * If AWS cannot start the worker because capacity is insufficient, it is temporarily deprioritized for 15 minutes, allowing processing to fall back to GCE. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
4c648bd16c |
infra: grant Google Play Pub/Sub Publisher on the RTDN topic (#1238)
## Summary Google Play real-time developer notifications (RTDN) for the mobile app (`com.nomadkaraoke.mobile`) publish to the Pub/Sub topic `Play-Store-Notifications`. RevenueCat pulls them through its own subscription (`RevenueCat-Subscriber-appc2bb515ecd`) and calls gen's `/webhooks/revenuecat`. The topic was set in Play Console > Monetization setup on 2026-10-31, but **Send test notification** failed. The topic has no `roles/pubsub.publisher` grant for `google-play-developer-notifications@system.gserviceaccount.com`. ## Changes - New `infrastructure/modules/play_rtdn.py`: one non-authoritative `gcp.pubsub.TopicIAMMember` (Publisher for Play's service account). The topic and the RevenueCat subscription were created by RevenueCat's "Connect to Google" flow and stay unmanaged. - Wired into `infrastructure/__main__.py`. ## Testing - `pulumi preview --stack prod --diff --target …play-rtdn-publisher` (read-only ADC): `+ 1 to create, 292 unchanged`, with exactly the member, role and topic above. - CodeRabbit CLI: no findings. - Infra-only change, so no version bump. ## Apply CI never runs Pulumi. Apply **before merging**, through `/gcp-admin`: `scripts/gcp-admin pulumi up --stack prod --target 'urn:pulumi:prod::karaoke-gen-infrastructure::gcp:pubsub/topicIAMMember:TopicIAMMember::play-rtdn-publisher'` Then resend Play's test notification. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVUH7B2p5YotvzcFsAR Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
ce57b4b598 |
fix(review): Tap To Sync partial pass keeps word order; local stack never reaches prod (v0.288.1) (#1237)
## Summary
### 1. Tap To Sync: a pass that stops before the last word
(customer-facing)
If a pass stopped before the last word (auto-stop about a second after
the line end, or the user pressing Stop), the untapped words after the
last tapped one kept their **old** timings, and those could now start
*before* the last tapped word. On gen's fake fixture line ("Nothing here
was ever transcribed", words 15.5 to 19.5, next line at 20.0), taps at
16.0, 17.2, 18.4 and 19.6 followed by auto-stop left "ever" at 19.6 and
"transcribed" at a stale 18.74 to 19.5. The two words were out of order,
and the video rendered them out of order.
- `frontend/hooks/useManualSync.ts`: new pure
`commitPartialSyncPass(words, synced)` runs in `cleanupManualSync`,
which is both the auto-stop path and the Stop toggle. It clears (sets
start/end to null) every untapped word after the last tapped one that
starts before it, then caps the last tapped word's end before the next
still-timed word in the line. The existing untimed-lyrics guard then
blocks finishing and points the user at those words. This is the same
rule as nomad-mobile's review-core `commitTapSync`
(`packages/review-core/src/tapSync.ts`, branch
`feat/sess-20261030-1930-e2e-health`). It uses the hook's existing
`syncedIndicesRef`.
- Backend backstop, `backend/services/timing_sanitizer.py`
`sanitize_corrections`: within a line, a word that starts before the
previous word's **start** is now treated like any other invalid start.
Small overlaps (a start a few ms before the previous word's end, common
in AI transcriptions) are left exactly as on main, so they get no move,
no WARNING and no edit-modal toast. It is **moved** to the previous
word's end, not cleared. I chose moving because that is how the module
already repairs bad or missing starts, because a cleared word would just
be refilled at prev_end on the next sanitize, and because nulls would
crash the render (`find_missing_timings` only guards the preview). The
module logs one aggregated `WARNING` per submission (never ERROR, since
this is an expected and repaired condition).
- `frontend/lib/lyrics-review/sanitizeWordTimings.ts` (the frontend
mirror, run when the edit modal opens) applies the same ordering rule,
with the same exemption for small overlaps.
### 2. The local stack must not reach production
nomad-mobile's E2E runs gen web inside an Android emulator's Chrome.
- `components/google-analytics.tsx`: GA (`G-D0E63LB9FY`) is no longer
loaded when the page host is localhost, 127.0.0.1 or 10.0.2.2. The check
runs at runtime after mount, so it also covers CI, and the server render
and first client render match.
- `lib/api/autoCorrect.ts` and `lib/api/customLyrics.ts` now use
`API_BASE_URL` from `lib/api.ts` instead of `NEXT_PUBLIC_API_BASE ??
'https://api.nomadkaraoke.com'`. That env var is set nowhere. A local
page now calls its own backend through relative URLs, so it never sends
its bearer token to production.
- New `lib/local-host.ts` is the single list of local hostnames, shared
by `lib/api.ts` and GA. `lib/api.ts` now also treats 10.0.2.2 as local.
- `scripts/local-stack.sh` backend_env: blank `FLACFETCH_API_URL=` and
`FLACFETCH_API_KEY=`, so direnv can't point the local stack at the real
flacfetch.
Version is bumped to 0.288.1.
## Testing
Each change had a failing test written first.
- Frontend jest (hook, sanitizer, edit modals, GA, API clients,
api/local-mode, auto-correct, custom lyrics): 13 suites, **150 passed**.
- New tests: 3 partial-pass hook tests (auto-stop, Stop, and the
keep-and-cap case), 2 sanitizer tests (the stale-word move, and a 20 ms
overlap left untouched), 4 GA tests, and 8 API-base tests (local hosts
use relative URLs, the production host uses the production API).
- Backend pytest: `test_timing_sanitizer.py` +
`test_jobs_corrections.py` **20 passed** (3 new, including a 20 ms
overlap that is left untouched), and `test_routes_review.py` **114
passed**.
- `customLyrics.test.ts`'s AbortSignal test now asserts that the
caller's abort reaches fetch, not that the signal object is identical. A
relative URL makes `apiFetch` manage the request and wrap the signal.
## Notes
- The web hook has no view of the next line, so the cap only uses words
in the same line. The backend sanitizer and segment bounds handle the
rest.
- The render-time `SegmentResizer._sanitize_segment_timings` is
unchanged (it clamps to the window only).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_012RYLVSLS3eVDqPctfYBMTt
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved lyric timing cleanup when recording or editing stops before
the final word. Out-of-order timings are corrected while small overlaps
are preserved, helping keep lyrics synchronized without changing valid
timings.
* Corrected out-of-order word timings during lyric cleanup and editing.
* **Improvements**
* Local app use now works more consistently across supported addresses,
including Android emulators, with API requests routed appropriately.
* Analytics scripts are now omitted on local pages.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.288.1
|
||
|
|
de5dfcd50d |
feat(encoding): AWS EC2 encoding worker as a pool candidate (v0.288.0) (#1236)
## Summary
Phase 1 of moving encoding to AWS. EC2 is paid for by the AWS Activate
credits (acct 790139457751, expiring 2028-10-31).
- **Plan:** `docs/archive/2026-10-29-aws-encoding-worker-plan.md`.
- **Benchmark** (3 real prod jobs replayed): an EC2 c7a.4xlarge matched
or beat GCE c4d-highcpu-16.
The EC2 box runs the **unchanged** `gce_encoding` worker service and
joins the existing candidate pool as `provider: aws`. The existing
failover, active override, lost-job detection and stockout cooldowns all
apply to it as they are.
## Changes
- **`backend/services/aws_encoding_compute.py` (new):**
`AwsComputeAdapter` looks to the manager like the
`compute_v1.InstancesClient` methods it uses.
- EC2 states map to GCE statuses.
- EC2 capacity errors are added to `CAPACITY_ERROR_CODES`.
- A missing instance raises `NotFound`.
- Controller credentials come from Secret Manager
`aws-encoding-controller-credentials`.
- **`EncodingWorkerManager`:**
- `_compute_for(vm)` sends AWS VMs to the adapter; the GCE paths are
untouched.
- A shared `parse_fallback_vms` passes `provider`/`instance_id`/`region`
through.
- `ranked_candidates()` is used by both the encode path and the warmups.
- `ensure_preferred_running()` behaves **exactly like
`ensure_primary_running` whenever the GCE primary ranks first**, which
is today's case.
- An AWS control-plane fault fails over to the GCE candidates.
- **Ranking:** `SPEED_RANK["c7a"] = 90`, so AWS is the last resort by
default. Firestore `config/encoding-worker.speed_rank_overrides =
{"c7a": 5}` makes it preferred, with no deploy.
- **Isolation from GCE-only tooling:**
- The override now records `active_override_provider`.
- The CI blue-green deploy, `deploy_promote` and the idle Cloud Function
all ignore the AWS worker; it hot-updates its wheel per job.
- A GCE promotion clears any stale provider tag.
- **Worker:**
- Uses `gcloud storage` on AWS, because gsutil can't use WIF credential
files. GCE keeps gsutil.
- `infrastructure/encoding-worker/aws/` adds:
- `provision.sh`, as EC2 user-data;
- `startup-aws.sh`, uploaded by CI;
- `idle-watchdog.py`, an idle self-stop that runs as its own systemd
timer.
- **Infra (GCP), already applied with `/gcp-admin`:**
- The WIF pool `aws-encoding`, plus an AWS provider that only admits the
`nomad-encoding-worker` role, federated to `encoding-worker@`.
- The secret `aws-encoding-controller-credentials`.
- Preview and apply: 4 creates, 288 unchanged.
- **Infra (AWS):** a new Pulumi project `infrastructure/aws` (stack
`nomadkaraoke/prod`):
- the c7a.8xlarge **persistent Spot** worker, which stops on
interruption or shutdown;
- an Elastic IP;
- a security group on 8080 (API key, same as the GCE firewall);
- the worker role with SSM;
- a tag-scoped `nomad-encoding-controller` user.
- Preview: 10 creates. **I apply it after merge**, because the box
provisions from the deployed wheel and boot scripts.
- `boto3` is now an explicit dependency.
- CI also runs the `encoding_worker_idle` function tests, in the job
renamed "Infra - Cloud Function tests".
## Rollout (after merge)
1. **Stage A:** `pulumi up` the AWS stack. Store the controller key in
the secret, then add the worker's entry to
`encoding-worker-fallback-vms`. It's inert unless both c4d and c2d stock
out.
2. **Stage B:** a supervised `speed_rank_overrides {"c7a": 5}` window.
Run real jobs, then compare timings and outputs, including a visual
check of the portrait video.
3. **Stage C:** keep it preferred. After a clean week, delete the c4d
pair.
Phase 2, running distribution (Dropbox/YouTube/Drive uploads) from AWS,
follows in a separate PR.
## Testing
- **New tests:**
- adapter: botocore `Stubber`, covering the state map, capacity and
NotFound handling;
- manager routing and ranking with overrides;
- `ensure_preferred_running` branches;
- failover on an AWS control-plane fault;
- provider tagging;
- deploy_promote exclusions;
- the idle function skipping AWS;
- watchdog decisions;
- the gcs CLI choice.
- **Updated** the candidate and warmup-route tests to the new seams.
- **Full backend + unit suites** have no new failures. The local
emulator and cross-suite failures are identical on main.
- **Local CodeRabbit:** 2 findings, both fixed (failover on AWS faults;
stale provider tag).
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.288.0
|
||
|
|
65940c6f57 |
fix(local-stack): decide partner calls go to local decide, never production (#1235)
The local stack's backend env had no `KARAOKE_DECIDE_API_URL`, so gen's decide partner client (account deletion) used the production default and sent `DELETE /api/partner/gen/users` to decide.nomadkaraoke.com. Production answered 401, because the local secret differs, so nothing was touched. It surfaced in nomad-mobile's E2E account-1 flow. The fix defaults it to decide's local stack on :8030, and callers can still override it. Script only, no runtime code. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Configuration** * The local environment can now use a custom karaoke decision service address, with a default address provided when none is configured. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
437b34ee64 |
fix(ci): wheel over PyPI's 100 MiB limit — drop unused RSC payloads + size guard (v0.287.2) (#1233)
## Summary
Every `Deploy - Publish to PyPI` since v0.279.4 (about 10-22) has failed
with `400 File too large. Limit for project 'karaoke-gen' is 100 MB`.
The wheel had grown to 110 MB (105.1 MiB), so PyPI is still on 0.279.4.
Backend deploys were not affected. I found this while watching deploys
during an alerts-triage session.
In the wheel, 66 MB of the bundled Next.js export is RSC `.txt` payloads
(about 3 MB per locale × 33). The only thing that uses the export is the
CLI local-review server
(`karaoke_gen/lyrics_transcriber/review/server.py`). It mounts only
`/_next` and `/static`, and every other route returns an HTML entry
point. Next's RSC fetch paths (`/{locale}/app/.../index.txt`) already
get HTML from those routes, so the `.txt` files are never served.
- `make build-frontend` deletes `*.txt` from the copied export. On the
real CI wheel from run 40826404322 this takes it from 105.1 MiB to 41.1
MiB.
- `scripts/verify_wheel_frontend.py`, which already runs in both
package-build jobs, now fails above 95 MiB. Growth will show up on the
PR instead of as a failed publish after merge.
## Testing
- New `tests/unit/test_verify_wheel_frontend.py`: 5 tests covering the
pass case, missing entrypoint, too few files, the size guard, and
headroom below 100 MiB.
- Re-zipped the real 0.286.0 CI wheel without the `.txt` files: 41.1
MiB, and the verifier passes. The verifier rejects the original wheel
(105.1 MiB) with the new error.
- CodeRabbit: 1 finding ("deleting RSC payloads can break client
navigation"). Skipped, because the local server never serves those files
(see above), so it can't change runtime behaviour.
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_012RYLVTiUPKxn3sQPyp3Teh
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.287.2
|
||
|
|
1878ad3c1d |
feat(retention): keep lead-vocal stems as word-timing training data (v0.287.1) (#1234)
## Summary Storage retention purged `stems/lead_vocals.*` 30 days after completion. As a result, 660 of 865 completed jobs have no audio left to pair with their human-reviewed word timings (`lyrics/corrections_updated.json`, which is kept). The full-auto v2 timing work (an aligner fine-tuned on our data, plus a per-word timing verifier as the auto-ship gate) needs exactly that pair. Today's first bake-off had to make do with the 41 reviewed jobs that still had stems, and data volume was the limiting factor. The cost is about 15 MB per job; every other separated stem is still purged. ## Changes - `storage_retention.classify_file`: `stems/lead_vocals.*` (exact basename) is now KEEP with category `training_lead_vocals`. - Tests: the stem moves from the purge list to the keep list; a near-match (`lead_vocals_backup`) is still purged. - Docs: ARCHITECTURE retention list. ## Testing `backend/tests/test_storage_retention.py`: 133 passed. CodeRabbit ran locally; its one finding (exact-match the basename) is fixed. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVVVPsvPbXFhRYY5VSJ --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
bfcf1fd951 |
feat(auto-approval): hold confident-lyrics jobs out of review until stems land (v0.287.0) (#1232)
## Summary
In prod (2026-08-31 to 10-29), about a third of jobs were last scored
**before stems existed**. People opened the review link 15–40 s after
AWAITING_REVIEW. The audio worker's "second chance" re-score then hit
IN_REVIEW (`wrong_status`), so:
- 134 jobs never got a backing-vocals verdict;
- 7 jobs whose lyrics were in the auto tier went to a human anyway (e.g.
6c4be7d8, ed0b0c33, c4d8a256).
This is the first quick win from the full-auto v2 re-evaluation, which
targets >50% zero-touch jobs.
## Changes
- **`backend/services/auto_approval/review_gate.py` (new).**
- When the text verdict is AUTO and the only blocker is
`audio_incomplete`, the screens worker parks the job in
`GENERATING_SCREENS` with `state_data.auto_review_waiting_for_stems`. No
review link or notification goes out.
- The hold is claimed atomically (`claim_job_state_flag`) by one of:
- the audio worker once stems land (normal path);
- the screens worker itself, if stems landed while it was parking;
- `recover_stuck_jobs`, for holds older than 20 min.
- The claimant re-scores (`trigger="audio_hold"`), then either
auto-completes or opens review exactly like before, with server-side
pre-apply.
- If the post-claim re-score fails, the job falls back to human review.
If that also fails, the hold marker is restored so the sweep retries. A
job is never stranded.
- **Executor:** `hold_for_audio` is recorded in the verdict. It never
applies with `always_review`, `made_for_you`,
`backing_preference=review`, unconfident lyrics, or the kill switch on.
- **Kill switch:** `AUTO_APPROVAL_AUDIO_HOLD_ENABLED` (code default
true, because the executor runs in both the service and the
audio-separation job).
- Docs: LESSONS-LEARNED, README status row.
## Testing
- New: `test_review_gate.py` (11), `test_screens_worker_review_hold.py`
(2), executor hold cases (7), and recover-stuck-jobs hold sweep (3).
- Full backend suite locally: 6962 passed. The only failures are
`test_emulator_integration`, which needs the Firestore emulator and also
fails on pristine main.
- CodeRabbit CLI ran locally. Its one finding (stranded job on
post-claim failure) is fixed; the re-review is clean.
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_012RYLVVVPsvPbXFhRYY5VSJ
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
1f22091bf9 |
docs(CLAUDE.md): CI does not run Pulumi (#1231)
The Infrastructure rule said that changes in `infrastructure/` deploy automatically on merge. That is wrong: ci.yml says Pulumi deploys are done locally, and CI never runs `pulumi up`. In the 2026-10-29 cost session, a merged infra fix (#1228) sat un-applied because of this. Docs only, so no version bump. No local CodeRabbit run, since this is a one-line docs change; leaving bot review on. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Documentation** * Clarified that CI validates infrastructure changes but does not apply them; merged changes require a local `pulumi up` to take effect. * Added guidance that Claude sessions need Andrew’s `/gcp-admin` grant to apply infrastructure changes. Removed outdated claims that GCP changes deploy automatically on merge or that CI deployments can be interrupted on spot VMs. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0d817509fa |
fix(api): event-loop stalls — visibility redistribution off-loop, route warmup, watchdog/Cloud Trace noise (0.286.2) (#1230)
## Summary Fixes the 2026-10-29 #alerts event-loop stall batch (handoff `docs/archive/2026-10-29-gen-event-loop-stalls-handoff.md` in the workspace). **1. Visibility change off the loop (39s stall, 13 requests incl. /api/health)** - `change_to_private` now runs claim, flag write, `redistribute_video`, public-output delete and the final commit as **one** `asyncio.to_thread` call (`_redistribute_and_commit_private`). `redistribute_video` runs on that thread's own loop through the new `redistribute_video_off_loop` (`asyncio.run(with_notification_drain(...))`). - Is anything bound to the main loop? No. `redistribute_video` builds its own `JobManager`/`StorageService`. The Dropbox, YouTube and GDrive clients are sync. Job-log writes go through `log_write_offload`'s writer thread. Notification tasks scheduled via `_schedule_notification` would land on the private loop, so `with_notification_drain` drains them before the loop closes, as the Cloud Run Job entry points do. - Why one thread call and not just the redistribution (from CodeRabbit): a thread can't be cancelled. So a cancelled request can no longer leave a committed private archive with its public outputs still live, or a claimed flag with no redistribution. - Guard: `redistribute_video` is added to `BLOCKING_CALLS`, and `visibility_change_service.change_to_private`/`change_to_public` are added to `AWAITED_ROUTE_HELPERS`. I checked that putting back `await redistribute_video(...)` fails the test. - **Double Dropbox upload confirmed and fixed.** GCS `finals/` holds every output twice: short canonical keys written by `_upload_results` (`lossy_4k_mp4.mp4`, …) and the readable encoder names (verified on job 0f211e97: 17 objects / 525 MiB). Redistribution uploaded both. The short-named `lossless_4k_mp4.mp4` also slipped past the `" (Final Karaoke Lossless 4k).mp4"` Dropbox exclusion. `_dedupe_redistribution_files` now keeps one copy per final, under its readable name: - a short name whose readable twin exists (twin found by suffix) is dropped; - a short name with no twin is renamed to the readable name (CDG/TXT zips included). **2. First-request stalls on new instances** - Lifespan builds FastAPI's lazy route contexts (`fastapi.routing.iter_route_contexts` + `effective_low_priority_routes`) in a thread before readiness, with a timeout and best-effort error handling. The results are cached by routes version under a lock, so the OTel `_get_route_details` walk no longer compiles route regexes on the loop. It is a no-op on FastAPI versions without these internals. Note: the lock pins 0.128 but prod installs unpinned (`pip install -e .`), so prod runs a newer FastAPI. I checked against FastAPI 0.162: the first walk builds 304 contexts, the second walk and the OTel lookup are cached (<1ms). - `flacfetch.core.manager`, `flacfetch.downloaders.youtube` and `yt_dlp` are added to `_LAZY_ROUTE_IMPORTS`. - `search_audio` auto_download: `select_best` now goes through `asyncio.to_thread`, with `select_best` added to `BLOCKING_CALLS`. **3. Loop watchdog false positive** - `_finish` logs WARNING, not ERROR, when all of these hold: no non-idle sample was taken, a process freeze was observed (`frozen_s > 0`), and no request was in flight when the stall began. Every "affected" request therefore arrived while the instance was thawing. An unsampled stall with no observed freeze, or with a request in flight at its start, stays ERROR. The freeze requirement came from CodeRabbit. **4. Cloud Trace exporter noise** - `cloud_trace_export` ignore pattern for `Error while writing to Cloud Trace`. CodeRabbit suggested matching only the transient gRPC detail. I didn't: the error monitor only sees `jsonPayload.message` (the exception is a separate field), and a persistent loss of telemetry isn't #alerts-worthy anyway. Smaller change: `change_to_private`'s Firestore `job_ref.update` writes now also run off the loop. ## Testing - New `backend/tests/test_redistribute_video.py`: - dedupe unit tests; - an end-to-end `redistribute_video` run with mocks, checking the files the Dropbox stage sees; - the off-loop runner uses its calling thread; - `change_to_private` runs the redistribution on another thread and loop; - a cancelled request still deletes the public outputs. - `test_loop_watchdog.py`: frozen + unsampled + late arrival gives WARNING; a request in flight at the start gives ERROR; no observed freeze gives ERROR; a sampled stall with a late arrival gives ERROR. - `test_main_startup_warmup.py`: lazy imports list; the route-context walk; no-op without the API; failure is non-fatal; lifespan warms off-loop before readiness. - `test_known_issues.py`: the Cloud Trace pattern matches, and other trace errors are not ignored. - Local runs: `pytest backend/tests/ tests/unit/services/error_monitor --ignore=backend/tests/emulator`: 7369 passed, 29 skipped. Not counted in that total: `test_emulator_integration.py` fails 8/11 the same way on the base commit, because a stray emulator from another session is on :8080 locally. That's environmental, not caused by this PR. - Local CodeRabbit: 3 cycles. Cycle 1: 3 findings (2 fixed, 1 skipped with the reason above). Cycle 2: 1 finding, fixed. Cycle 3: 0 findings, but CodeRabbit reported it reused cached results rather than a fresh review. ## Not verified - No prod run of a real public→private change yet. After deploy, the Dropbox folder of the next one should contain each final exactly once, and no `EVENT_LOOP_STALL` should appear during it. - Route-context warmup was checked only against FastAPI 0.162 in a scratch venv, not the exact prod version. Watch for the `Warmed N route contexts` log line on the next cold start. - Existing private Dropbox folders that already hold the duplicates are not cleaned up by this PR. Alert: https://discord.com/channels/1198691101943541770/1265517528608669707/1563719390849929330 Alert: https://discord.com/channels/1198691101943541770/1265517528608669707/1563009782363721760 Alert: https://discord.com/channels/1198691101943541770/1265517528608669707/1563039889354784882 Alert: https://discord.com/channels/1198691101943541770/1265517528608669707/1563221363278348514 Alert: https://discord.com/channels/1198691101943541770/1265517528608669707/1564308425033252895 Alert: https://discord.com/channels/1198691101943541770/1265517528608669707/1563160849336762449 @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVTiUPKxn3sQPyp3Teh --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
1abdd8c860 |
fix(cost): runner-manager only dispatches VMs for jobs it can serve; enforce AR cleanup (v0.286.1) (#1228)
## Summary On 2026-10-27, **238 GCP `gha-general` VMs (~$67)** ran in one day. This pushed October's GCP bill past the $300 budget. **Cause:** - nomad-mobile's E2E jobs run on the utility MacBook (`[self-hosted, macOS, nomad-mbp]`). - The runner manager acted on *any* job labelled `self-hosted`, so each macOS job got a Linux VM that could never take it. - While the job waited for an MBP lane, `redispatch_stalled_jobs` launched another VM every 5 min for up to 60 min. - Each of those VMs then idled until the 2 h reaper deleted it. ## Changes - `ephemeral.can_serve(labels)`: GitHub assigns a job only to a runner that advertises every label the job requests, so a job is ours only if its labels ⊆ the resolved family's `runner_labels_for()`. - The new check gates the webhook (`main.py`), `list_queued_self_hosted_jobs` and `redispatch_stalled_jobs`. - I checked every `runs-on: [self-hosted, …]` label set in the org's workflows. All of them except the macOS lanes are subsets of our families' labels, so no existing job loses its VM. - CI: these function tests never ran in CI before. A new `runner-manager-tests` job runs them on infra or ci.yml changes and is enforced by CI Gate. - Docs: `docs/EPHEMERAL-GHA-RUNNERS.md` § "Only jobs our runners can serve". ## Testing - 6 new tests: macOS jobs, unknown labels, our families, the redispatch skip, queued-scan exclusion, and the webhook no-op. All 71 runner_manager tests pass locally. - Local CodeRabbit review: 2 cycles, both CI-wiring findings fixed. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
19f3c731fb |
test(e2e): sanitized page-activity log on tenant E2E failure (#1229)
## Summary Tenant E2E #155 (singa, 2026-10-28) failed with the portal stuck on the `/en/app/` loading spinner for 30s. The backend served tenant config in 168ms, 8 live re-runs were fine, and run #156 got past the same step. The only evidence was a screenshot: CI has no Playwright traces, because traces hold `X-Admin-Token` and the repo's artifacts are public. This adds a public-safe stand-in, `frontend/e2e/helpers/page-activity.ts`. It records: - main-frame navigations; - each request's method and origin/path (no query strings), with status and duration or the failure reason; - console errors/warnings and uncaught page errors, with URLs and token-like values in their text redacted. On failure it writes `page-activity.log` into test-results (the scrubber still checks it) and prints the requests still in flight to the job log. It is wired into `tenant-happy-path.spec.ts`. The approach is documented in docs/TESTING.md, which also records #155 as a one-off flake. ## Testing - New offline spec `e2e/self-healing/page-activity.spec.ts` (3 tests): a fixture page with one hung, one failed and one token-bearing request, plus console and page errors. The self-heal suite runs in CI's frontend-e2e-smoke job. The whole suite passes locally (36 tests). - `playwright test --list` on the production config compiles the tenant spec. - Local CodeRabbit review: 2 findings (redact console text; race on the REQFAIL assertion), both fixed. - Test-only change, so no version bump. Alert: https://discord.com/channels/1198691101943541770/1265517528608669707/1564876714532798465 @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVTiUPKxn3sQPyp3Teh --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
53bc8aa708 |
infra(cloudflare): remove stale www.app.karaokehunt.com CNAME (#1227)
Removes `www.app.karaokehunt.com` → `karaokay-karaoke-song-choice-helper-eiqxrt.custom-domain-proxy.flutterflow.app`. That hostname is the old KaraokeHunt FlutterFlow web app. It no longer responds, and leaving a CNAME that points at an unclaimed third-party proxy is a subdomain-takeover risk. Part of the final decommission of GCP project `projectbread-karaokay`. `karaokehunt.com`, `www.karaokehunt.com` and `app.karaokehunt.com` still 301-redirect to nomadkaraoke.com. **Already applied:** `pulumi preview` showed exactly one delete, then `pulumi up` reported 1 deleted, 135 unchanged. CI doesn't run Pulumi, so this PR just brings `main` in line with what's live. Tests: the `infrastructure/cloudflare` tests pass (11). 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVU32bqJHRH3psJ5Udm <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Configuration** * Removed the DNS record for `www.app.karaokehunt.com`. * The separate proxied IPv6 record for `www.karaokehunt.com` remains configured. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
331d18ddf8 |
kjbox: one-tap Google/Apple sign-in for singers; sessions last 30 idle days / 365 max (v0.286.0) (#1226)
## Summary
Backend for one-tap sign-in in the kjbox singer page
(sing.nomadkaraoke.com). At the 2026-10-22 show, typing an email and a
6-digit code was the biggest friction point for singers making a karaoke
video on demand. Plan: kjbox
`docs/archive/2026-10-28-singer-make-it-one-screen-plan.md`.
- **`POST /api/kjbox/auth/google` and `/api/kjbox/auth/apple`** take
`{id_token, locale?, venue?}` and return the same body as
`/api/kjbox/auth/verify-code`, with no code needed.
- The token is checked by `login_code._verify_identity`, so the
audiences match the app's (and `fake:` tokens work only on the local
stack).
- Venue rules are the same as verify-code: no per-IP signup cap,
`KJBOX_SIGNUP_CAP_PER_24H` on new accounts, `signup_source="kjbox"` +
`signup_venue`, no `signup_ip`.
- Send-code's eligibility checks are applied explicitly. A disposable
domain gets 422. A blocked email or IP gets `422 sign_in_not_allowed`.
Apple relay addresses are allowed.
- Code sign-in and social sign-in now share one `_finish_sign_in`
helper.
- **`SESSION_EXPIRY_DAYS` (inactivity) 7 → 30, globally** (Andrew's
call). With weekly shows, every returning singer was signed out by the
next show. **`SESSION_ABSOLUTE_EXPIRY_DAYS` 30 → 365** (Andrew,
2026-10-28), so a weekly regular isn't signed out once a month.
- Version 0.286.0. API.md updated.
This stays inert until kjbox calls the endpoints. Before the singer page
can use them, Andrew needs to:
- add `https://sing.nomadkaraoke.com` as an Authorized JavaScript origin
on the "Nomad Karaoke Web" OAuth client;
- create an Apple web Services ID and add it to
`APPLE_SIGNIN_AUDIENCES`.
## Testing
`backend/tests/test_kjbox_partner_api.py` covers:
- a new singer via Google and via Apple;
- an Apple relay address, which is never treated as disposable;
- a code user who later signs in with Google;
- the signup cap applying to new accounts only;
- an invalid token returning 401;
- the partner secret being required;
- disposable and blocked email/IP refusals;
- 20 days idle still valid and 31 days idle expired; a new session
expires a year from sign-in; a weekly regular is still signed in two
months on.
259 auth/session/kjbox tests pass locally. Local CodeRabbit: 2 cycles.
The first found the missing eligibility checks (fixed) and the
absolute-expiry question (now 365 days). The second found nothing new.
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
5c30fde9b4 |
fix: reference lyrics truncated at 3000 chars (blank lines counted) (v0.285.2) (#1225)
## Summary - `BaseLyricsProvider.MAX_LYRICS_LENGTH` (3000) was measured on raw text. Genius RapidAPI emits `\n\n` per line, so a normal ~4 min song (job 258ba59b, 4,256 chars) was truncated after verse 2. Lrclib/Musixmatch hit the same cap (2,975 byte files). - Blank lines no longer count toward the cap; cap raised to 15000 (screenplays/articles are 50k+). ## Testing - New regressions: double-spaced 70-line song and ~8k-char rap song keep every line; screenplay tests still truncate. - `tests/unit/test_base_lyrics_provider.py` 17 pass. 9 failures in test_lyrics.py / test_agentic_gemini_provider.py are pre-existing on main (env). 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Bug Fixes** * Lyrics can now contain up to 15,000 characters before truncation, up from 3,000. * Blank lines no longer count toward the lyrics character limit, helping preserve longer lyrics with extra spacing. * **Documentation** * Added guidance about the updated lyrics limit and a caution for rerunning lyrics actions on in-progress jobs, where existing working data may be replaced. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
|
cd1fa3feaf |
fix: provide-url YouTube downloads failing with 'No audio source selection found' (v0.285.1) (#1224)
Pasting a YouTube URL in the audio modal (job 258ba59b) failed: provide-url set source_name/download_url but not source_id, and the worker fallback required source_id. Now sets source_id and the worker accepts download_url. The retry loop could never succeed before. Tests: worker regression test for the provide-url job shape; provide-url test asserts source_id. Emulator test_e2e_cli_backend needs Firestore emulator, left to CI. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Bug Fixes** * YouTube audio jobs submitted with a pasted URL can now proceed when no video ID is available. * YouTube video IDs are saved with provided-URL jobs when available, and stale search selections no longer override the pasted URL. * **Chores** * Updated the application version to 0.285.1. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>v0.285.1 |
||
|
|
3230bfbe0b |
feat(requests-board): real-user-only requests, 1 request + 1 vote/day, delete on last vote, confident unattended audio (v0.285.0) (#1223)
## Why Investigating two bad community picks for one user (jobs 8695c32e, 94c83a7e) found: - a YouTube-bot request was "claimed" by a vote that was later moved, leaving a 0-vote request with a stale owner that was still pickable (`vote_count >= 0`); - the daily pick's `select_best` raised on remote (dict) results and silently fell back to result #0 (logged as a WARNING on 7 days since 10 Oct). ## What - **Real users only:** every request must belong to a real user email (`require_real_email`, enforced in the service); the YouTube comment bot no longer creates requests, it replies with how to request on the board. Legacy placeholder rows are never listed/picked/credited. - **Limits:** 1 request + 1 vote per day (submitting spends the vote → `409 vote_already_used`; `429 submission_limit`). The cap uses a create-only per-day marker, so deleting a request doesn't give the submission back (purged on account deletion). - **Last up-vote deletes the request** after confirmation (`409 would_delete_request` + `request_id` until `confirm_delete=true`), freeing remaining voters' votes. Replaces the claim logic. - **Picker:** needs `vote_count >= 1` and an owner who still up-votes it and hasn't owned it already; deleting a community job reopens its request (or deletes it if nobody backs it). - **Audio:** unattended pick only on an exact artist+title match (torrent ≥3 seeders, else Spotify; ambiguity → none). Otherwise the job parks in `AWAITING_AUDIO_SELECTION` and the owner is emailed. - **Email:** community-pick review email says the song came from the requests board. - **UI:** confirm before spending the daily vote / submitting, thank-you panel with credit-pack CTA (`/app?buy=1`), form disabled once the vote is spent, delete-request warning. Strings in all 33 frontend locales + en/es/de backend. ## Testing - Emulator tests for all service rules (real Firestore transactions), unit tests for routes, audio pick, daily pick parking, notifications; Jest tests for the board client. - Local CodeRabbit review run; all 7 findings fixed. - Note: a handful of existing job-creation emulator tests and `test_internal_api` auth tests fail identically on clean `origin/main` in my local env; CI is the judge. Plan: docs/archive/2026-10-26-requests-board-overhaul-plan.md. Docs: REQUESTS-BOARD.md, API.md, LESSONS-LEARNED.md. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>v0.285.0 |