100 Commits
Author SHA1 Message Date
Andrew Beveridge ☄️andClaude Opus 5.5 f4f0f6af77 feat(review): only show the review button once the job is fully prepared (v0.298.0) (#1273)
## Summary

The review button now appears only once a job is fully prepared: stems
separated, auto-approval re-scored with them, AI lyric corrections
generated and applied on the server, and review audio + waveforms
cached. The browser no longer applies AI corrections by itself, so there
is one code path.

Prompted by job ae995697 (a Made For You order): the review button
showed at 18:29, but the stems, backing verdict and vocal waveform only
existed from 18:32, and the server pre-apply had aborted on a
duplicate-word check, leaving the browser to apply corrections on load.

## Changes

- **Hold every job until stems land**
(`review_gate.route_after_scoring`). Before, only jobs whose lyrics
would auto-ship were held, so Made For You and lyrics-gated jobs opened
review early. Same kill switch (`AUTO_APPROVAL_AUDIO_HOLD_ENABLED`) and
20-minute backstop.
- **Generate AI suggestions on the API service before parking.** The
hold is released inside the audio-separation job, which has no LLM
credentials, so that job only applies an existing cache.
- **`enter_human_review` prepares everything before AWAITING_REVIEW**:
pre-apply (generating on a cache miss), then
`review_assets.prepare_review_assets` (OGG transcodes, backing waveform,
vocal peak envelope), then the transition.
- **Non-strict pre-apply for human review**
(`build_applied_segments(strict=False)`): what "Accept All" produces.
Stale/duplicate signals are recorded as `warnings`. The auto-ship path
stays strict.
- **Removed the browser's on-load auto-run/auto-apply** from
`useAutoCorrect`.
- **Job card shows "[5/10] Preparing review"** while held (new summary
field + 33 locales).
- Docs: ARCHITECTURE review-gate step,
`docs/archive/2026-11-19-review-ready-gate.md`.

Trade-off: Made For You and lyrics-gated jobs reach review (and send the
review-ready notification) ~3 minutes later.

## Testing

- Backend unit suite: 7294 passed. Locally failing: 8
emulator-integration tests (something unrelated listens on :8080 so they
don't skip; CI runs them against real emulators) and 1 locale test that
tripped on a gitignored translation snapshot (passes once removed).
- New/updated tests: review gate (review-verdict jobs held, kill switch,
prepare-before-transition order, suggestions generated before parking,
no double wait, missing job), non-strict apply, duplicate-word pre-apply
(ae995697 case), review_assets.
- Frontend Jest: 1834 passed; no type errors in touched files.
- Local CodeRabbit: 2 minor findings fixed, re-review clean.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVVdAxGitbw1FhYP4e4

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 16:12:58 -05:00
Andrew Beveridge ☄️andClaude Opus 5.5 0449d0c107 fix(infra): build Cloud Function source archives without __pycache__ (v0.297.4) (#1272)
## Summary
A full `pulumi preview` from main showed runner-manager drift ("replace
source, update function"), last applied from the
`karaoke-gen-runner-scan-retry` worktree. I checked it:

- #1262 (that worktree's branch) is fully on main. The squash merge is
byte-identical to the branch tip. The live scheduler job (`retryCount
2`, `maxRetryDuration 60s`, `attemptDeadline 300s`) also matches the
code exactly.
- The drift comes from the **deployed zip containing
`__pycache__/*.pyc`** (including a pytest pyc) from the worktree's test
run. `pulumi.FileArchive` hashes every file, so every clean checkout saw
a phantom diff. Separately, the 7-day lifecycle rule had already deleted
the bucket object.

## Changes
- New `infrastructure/modules/source_archive.clean_source_archive()`
builds an `AssetArchive` of source files only. It skips `__pycache__`,
`*.pyc`, `.pytest_cache`, other tool caches and `.DS_Store`.
- Used for runner-manager, divebar-lookup and divebar-mirror (the three
`FileArchive` users).
- Unit tests, added to the "Infra - Cloud Function tests" CI job.
- LESSONS-LEARNED entry.

## Applied (before merge, per repo convention)
- I ran a targeted preview with `__pycache__` present in the worktree.
Runner-manager showed source replace + function update (the one-time
correction). Divebar-lookup and divebar-mirror showed **no change**.
- Targeted `pulumi up` ran as claude-admin (Andrew's grant): 2 changes,
289 unchanged.
- The new revision is `github-runner-manager-00031-sam`, and the
deployed zip now holds only the 4 source files.
- The re-preview shows 291 unchanged.

## Testing
- `pytest infrastructure/test_source_archive.py`: 4 passed
- `pytest infrastructure/functions/runner_manager`: 75 passed
- Local CodeRabbit review: no findings

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVSfwoV9WyncjqMPHrq

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 15:16:34 -05:00
Andrew Beveridge ☄️andClaude Opus 5.5 2971145c85 feat(iam): let claude-readonly read secret values (#1271)
## Why
Claude sessions couldn't read secrets such as the backend admin API
token. ADC is `claude-readonly@`, which had `secretmanager.viewer`
(metadata only). The gcloud CLI's `admin@nomadkaraoke.com` login lapses
and needs an interactive re-login, so every documented `gcloud secrets
versions access …` command failed in sessions. For example, retrying
jobs ae995697/e2f60e2d on 2026-11-19 had to go through a browser
session. Andrew asked for `claude-readonly@` to be able to read all
secrets.

## Change
- Project-level `roles/secretmanager.secretAccessor` for
`claude-readonly@`.
- The token commands in CLAUDE.md, TESTING.md, TROUBLESHOOTING.md and
STORAGE-RETENTION.md now pass
`--account=claude-readonly@nomadkaraoke.iam.gserviceaccount.com`, so
they work without a fresh gcloud login.

## Trade-off
The SA's key is now secret-equivalent: whoever holds it can read every
secret and act as backend admin through the API. The key stays on
Andrew's Mac only.

## Applied
`pulumi up --target <claude-readonly-secretmanager-accessor>`: **+1
created, 290 unchanged**, 2026-11-19. Verified: `claude-readonly@` reads
`admin-tokens` (and the API accepts the token) and other secrets.

The full `pulumi preview` also shows unrelated drift, not touched here:
runner_manager function source (last applied from the
`karaoke-gen-runner-scan-retry` worktree) and its scheduler retry
config.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVVVVLQSWvtZqBGxRFb

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 13:32:46 -05:00
Andrew Beveridge ☄️andClaude Opus 5.5 1840645faa feat(youtube): Firestore catalogue of every channel video with change history (v0.297.3) (#1270)
## Summary
Follows #1269. The daily description backfill already fetches every
channel video's snippet, status and content details, then throws them
away. This saves them to Firestore as a lasting record of the channel,
so we can investigate YouTube video activity later. It uses no extra
YouTube quota.

## Data model
- `youtube_videos/{video_id}`, one doc per video:
- from YouTube: title, description, tags, published_at, privacy_status,
upload_status, category_id, duration, default_language, made_for_kids,
license, embeddable;
- parsed from the title: artist, song_title, brand_code, kind,
backfill_target;
- bookkeeping: first_seen_at, last_seen_at, last_changed_at,
in_uploads_playlist, deleted, deleted_at.
- `youtube_videos/{id}/history/{auto}`: `{at, event, changes: {field:
{old, new}}, source}`.
  - `event`: first_seen, changed, deleted or restored.
- `source`: `backfill` for this job's own rewrites, otherwise
`observed`.
- If a Studio edit and a backfill rewrite land in the same run, they're
logged as two separate entries.
- **Soft-delete only.** A known video YouTube stops returning gets
`deleted=True` and a `deleted_at`. It stays in the known-ID list and
keeps being checked, so if it comes back that's logged as `restored`.
- **Writes only on change.** `last_seen_at` is refreshed at most once
every 20 hours. A video's doc and its history entries always go in one
Firestore batch.
- **Best effort:** if the catalogue sync fails, the run logs a WARNING
and still succeeds. A dry run skips it.

## Changes
- New `backend/services/youtube_video_catalog.py` (100% covered).
- `fetch_snippets` also requests `contentDetails`, at no extra quota
cost.
- The worker passes the metadata from before each rewrite (for
attribution), keeps gone IDs in `known_video_ids`, and adds `catalog`
stats to the run summary.
- README `scripts/youtube-descriptions` documents the catalogue. Version
0.297.3.

## Testing
- `pytest backend/tests -k "youtube_comment or backfill or
video_catalog"`: 129 passed.
- 12 new tests use an in-memory Firestore fake. They cover first seen,
unchanged same day and next day, Studio edits, backfill attribution,
both kinds of change in one run, soft-delete once, a stub for a video
never catalogued, restore, playlist-only changes, and batches never
splitting a doc from its history.
- After deploy: one manual run on prod to fill the catalogue (expect
about 1.9k new docs), then check it.

Local CodeRabbit: 2 rounds, 3 minor issues fixed (source attribution,
doc and history in one batch, a video first seen and rewritten in the
same run).

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVVbErmEFhmiW48uuFK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 18:41:04 -05:00
Andrew Beveridge ☄️andClaude Opus 5.5 4a23398c04 fix(youtube-backfill): check known video IDs the uploads playlist drops (v0.297.2) (#1269)
## Summary
Andrew got "[YouTube Descriptions] 5 updated, 0 remaining" on
2026-11-14. The cause: the channel's uploads playlist
(`playlistItems.list` on `UU…`) isn't a complete list of the channel.
- Its pages overlap: 1914 items, but 11 are duplicates.
- It leaves out public videos for weeks at a time. Five 2023 This Is Me
Smiling videos were missing for the whole Oct 22 template cycle, so the
backfill reported "complete" while they still had the old description.
They came back on 11-14 and were rewritten.
- `hSPiNfrqf5w` (Alive in the Chase) is still on the pre-10-22 template,
and `QP2lKxkfeqE` is missing from the playlist again right now.

## Changes
- `youtube_backfill.iter_all_video_ids` removes duplicate IDs.
- New `youtube_backfill.enumerate_channel(youtube, known_ids)`: lists
the playlist, then looks up known IDs it left out directly with
`videos.list` (about 1 unit per 50 IDs). Returns `recovered_ids` and
`gone_ids`.
- The worker keeps every ID it has seen in Firestore
`youtube_backfill/known_video_ids`.
- The first run seeds the set from the quota ledger's
`desc-backfill:<id>` entries (1800 IDs on prod, read-only check).
  - If seeding fails, nothing is saved, so the next run retries.
  - Deleted videos are dropped from the set.
  - A dry run saves nothing.
- The run summary and progress email report `recovered_from_known`. A
WARNING log names the IDs the playlist omitted.
- Docs: `scripts/youtube-descriptions/README.md`,
`docs/LESSONS-LEARNED.md`. Version 0.297.2.

## Testing
- `pytest backend/tests -k "youtube_comment or backfill"`: 114 passed.
- New tests cover duplicate removal, recovering videos the playlist
omits, gone IDs, seeding from the ledger, a failed seed being retried,
no saves on a dry run, and the email count.
- Seeding checked against prod Firestore (read-only): 1800 IDs,
including all 5 affected videos and Alive in the Chase.
- The first prod run after deploy should log "uploads playlist omitted N
known videos" and rewrite Alive in the Chase.

Local CodeRabbit: 2 rounds. Fixed 1 major (failed seed was never
retried) and 1 minor (loose test assertion).

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVVbErmEFhmiW48uuFK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 14:03:13 -05:00
Andrew Beveridge ☄️andClaude Opus 5.5 5bf17d520f fix(youtube): upload with the current channel description template (#1267)
## Why
On 2026-11-10 the daily description backfill sent "[YouTube
Descriptions] 1 updated, 0 remaining". The video was `0uM7NClsA_E`
(NOMAD-1779, job `58555663`). The NOMAD-1779 admin re-render on 11-09
uploaded it with the description template saved on the job at creation
(2026-10-16). The channel template changed on 10-22, so that copy was
out of date, and the backfill fixed the video the next morning.

## Change
- `get_effective_distribution_for_job` now gives non-private jobs, which
all upload to the shared Nomad channel, the **current**
`Settings.default_youtube_description`. The job's own
`youtube_description_template` is used only if no channel template is
configured. This covers first uploads, admin re-renders and
redistributes (the orchestrated path).
- The legacy `KaraokeFinalise` path in `video_worker.py` now resolves
the description through the same function instead of reading the job
field directly.
- `docs/API.md`: added a note that an admin-edited per-job template no
longer reaches YouTube while a channel template is set. The backfill was
already overwriting those edits.
- Version bumped to 0.297.1.

## Testing
- Added tests: the current template wins over an old copy on the job,
and the job's copy is used when there is no channel template. Updated
the existing non-private test.
- Backend suite (non-emulator): 7,255 passed. The only failures need the
Firestore emulator (`test_emulator_integration.py`), which wasn't
running locally; CI runs that suite.
- Local CodeRabbit review: no findings.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVSJbAsXHW4NxxzusWX

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 22:24:47 -05:00
Andrew Beveridge ☄️andClaude Opus 5.5 ad64286004 feat(tenant): tenant portals are shared workspaces (v0.297.0) (#1266)
## Summary
Andrew's expectation: "all jobs in the tenant portal are accessible to
everyone who has permission to access that portal." Until now a
non-admin only saw/opened jobs whose `user_email` was their own, even on
a tenant portal. A client logging in with their own email never saw the
10 jobs bulk-uploaded for them under a separate `@nomadkaraoke.com`
address, and re-submitted every song.

## Rule (`backend/services/job_access.py`)
- Admins: every job.
- Owner: their jobs.
- **Tenant portal member** (currently allowed to sign in:
`allowed_emails` / `allowed_email_domains`, tenant active): every job of
that tenant. Re-checked per request against the live tenant config (~5
min cache), so removing someone from the allowlist removes access. Fails
closed on lookup errors.
- Consumer jobs (no tenant): owner-only, unchanged.

## Wired into
- `_check_job_ownership` (all job routes: view, edit, rerender, delete,
…)
- full-auth review access (`require_review_auth`:
lyrics/instrumental/audio-edit)
- file-upload guards (finalize, style uploads; style uploads keep
denying ownerless legacy jobs as before)
- `GET /api/jobs`: members on a tenant portal list all of that tenant's
jobs (still tenant-scoped)
- tenant theme refresh/outdated/rerender-outdated scope now covers the
whole portal (routes were already member-gated)

Unchanged on purpose: kjbox singer review links and the consumer
visibility toggle (consumer-only).

## Testing
- New `backend/tests/test_tenant_shared_jobs.py` (20): member/domain
member/case, outsider, consumer privacy, cross-tenant, inactive tenant,
no allowlist, removed member, fail-closed, token-without-email,
ownerless per-route, list scoping (member/outsider/consumer portal),
review auth 200/403, ownership wrapper.
- Updated theme tests that asserted the old owner-only scope; added
teammate-can-rerender.
- Local: 2,710 job/review/upload/tenant tests + 656 theme/tenant tests
pass (only `test_emulator_integration.py` fails locally, which needs the
Firestore emulator; same on main).
- Local CodeRabbit: 1 finding (ownerless legacy jobs on style uploads)
fixed; re-review clean.

## Prod cleanup done alongside (admin API)
Deleted (archived) the 10 stale Sept bulk-upload jobs for the randy-vild
tenant that duplicated his completed RVILD tracks.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVVUdF2J1bNeVa2uLXA

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 14:49:27 -05:00
Andrew Beveridge ☄️andClaude Opus 5.5 13619b7cc9 fix(review): preview shows the countdown intro like the final render (v0.296.0) (#1264)
## Summary
Songs whose vocals start within 3s get a "3... 2... 1..." intro in the
final render (`CountdownProcessor`: lyrics +3s, 3s leading silence), but
the **review preview never applied it**. So the preview opened straight
on vocals, and a customer (Randy Vild, "Caught Up Alive" / "Faithful and
True") asked us to add a countdown that was already coming.

## Changes
- `CountdownProcessor.add_countdown_to_lyrics_if_needed` (lyrics-only
half of `process`) + `padding_for_segments_data` (same rule on the
serialized segments the client sends).
- `CorrectionOperations.generate_preview_video` applies the countdown;
full local/CLI renders pad the audio too. Returns
`countdown_padding_seconds`. Countdown previews get a salted hash so
pre-fix cached previews aren't reused.
- GCE preview path: pads the already-downloaded audio on the API side
and uploads it to `previews/{hash}_audio.flac` for the encoder, so **no
encoding-worker change / version skew**.
- `POST /preview-video` responses carry `countdown_padding_seconds`.
- `PreviewVideoSection`: video time = song time + offset, so the
overlaid instrumental stem, waveform playhead and audition seeks stay
aligned (stem parked at 0 during the intro).
- Corrections that already contain the countdown aren't shifted again
but still get padded audio (CodeRabbit finding).

## Testing
- New:
`tests/unit/lyrics_transcriber/output/test_countdown_preview_helpers.py`,
`tests/unit/lyrics_transcriber/correction/test_operations_preview_countdown.py`,
GCE-path endpoint test in `backend/tests/test_routes_review.py`, 4
offset tests in `PreviewVideoSection.test.tsx`.
- Frontend jest (lyrics-review + lib): 892 passed.
- Real render check: local preview of job 508b38b2 (Faithful and True)
from prod corrections + audio with real ffmpeg shows the countdown, and
silence runs to 3.80s, matching the delivered final video's first word
(3.795s).
- Local CodeRabbit review: 2 findings fixed, re-review clean.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVVUdF2J1bNeVa2uLXA

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 13:56:20 -05:00
Andrew Beveridge ☄️andClaude Opus 5.5 ac844b3933 ci: pin Playwright jobs to ubuntu-24.04 (chromium unsupported on 26.04) (#1265)
## Summary
GitHub's `ubuntu-latest` now resolves to Ubuntu 26.04, where our
Playwright (1.57) fails at `npx playwright install --with-deps
chromium`:

> Error: ERROR: Playwright does not support chromium on ubuntu26.04-x64

That breaks every PR's E2E Smoke / Full / Local-Stack jobs (seen on
#1264) and both daily E2E workflows on main (Payment + Happy Path,
Tenant Portals failed today, 2026-11-11).

## Change
Pins only the 7 jobs that install Playwright browsers to `ubuntu-24.04`
(ci.yml ×4, e2e-daily.yml ×2, e2e-tenant-daily.yml ×1). Everything else
stays on `ubuntu-latest`. Workflow-only, so no version bump. A
Playwright upgrade that supports 26.04 can follow later and drop the
pin.

## Testing
This PR's own CI runs the pinned E2E jobs.

Workflow-only change, no local CodeRabbit review run, so the bot review
is left on.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVVUdF2J1bNeVa2uLXA

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Chores**
* Updated the operating system used by automated smoke, end-to-end, and
daily test runs to Ubuntu 24.04.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 13:43:31 -05:00
Andrew Beveridge ☄️andClaude Opus 5.5 da204bd52e fix(ass): split lyric lines wider than the frame (overlapping lyrics, NOMAD-1779) v0.295.1 (#1263)
## Summary

NOMAD-1779 (job `58555663`) shipped with two lyric lines drawn on top of
each other near the end of the song (YouTube comment: "the lyrics get
kinda messed up towards the end").

**Cause:** lines are split by character count (`max_line_length`, 40 for
Nomad). Wide-glyph lines like "Mm- mm- mm, mm- …" pass that but measured
~3890px on the 3840px frame. libass wraps an over-wide event by itself;
rows are exactly one line-height apart, so the wrapped half landed on
the next row's lyrics.

**Fix:** `SubtitlesGenerator.generate_ass` now measures each line in the
style font, scaled the way libass sizes it (existing
`LyricsLine._measure_text` / `ass_font_scale`), and splits any line
wider than 94% of the frame at a word boundary
(`SegmentResizer.split_to_width`, prefers phrase-ending punctuation,
keeps word timing, shares translations). Applies to every ASS render
(main, portrait, theme previews). Lines that fit are untouched.

**Scope:** of 309 completed branded jobs since 2026-09-10, only 1779 had
an overlap; 4 more (1759, 1738, 1766, NOMADNP-0264) had a line wrapping
into an empty row.

## Testing

- New `tests/unit/lyrics_transcriber/output/test_line_width_split.py`:
split logic (fits / splits / phrase break / balance / single long word /
timing / translation / singer), ASS generation, and a **real libass
render** asserting each lyric event draws as exactly one row. Verified
the render test fails with the fix disabled.
- Regenerated 1779's ASS from its `corrections_updated.json` with the
real theme font and rendered 2:58 with ffmpeg/libass: old ASS reproduces
the overlap exactly, new one renders 4 clean lines (widest 3502px).
- Locally: 1968 unit + 7231 backend tests pass. Failures were
environmental only (shared conda env lacks `flacfetch`: 6 unit + 2
collection errors; `test_emulator_integration.py` fails identically on
`origin/main` locally).
- CodeRabbit CLI (local, committed): no findings.

After deploy: admin re-render of 58555663.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVW1AnhNWu2qGLaNvxu

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 15:41:00 -05:00
Andrew Beveridge ☄️andClaude Opus 5.5 d98e0d6a5e fix(runner-manager): retry a repo once on transient GitHub errors before paging (v0.295.1) (#1262)
## Summary
The stalled-job scan logged one `RemoteDisconnected` from the GitHub API
at ERROR, and the error monitor posted it to #alerts. The next scheduler
tick rescans every repo anyway, so one dropped connection doesn't mean
the safety net failed.

- Network failures (`URLError`, `http.client` errors,
`OSError`/timeouts) and GitHub 5xx/429 now get **one immediate retry**
for that repo (logged as plain INFO).
- If the retry also fails, or the error is not transient (403/404 auth
or permission errors), it still logs `Skipping <repo> in stalled-job
scan` at **ERROR**, so a broken safety net still pages. #1252 depends on
that.
- Pulled the per-repo body into `_queued_self_hosted_jobs_in_repo` so it
can be retried.

## Testing
- `pytest infrastructure/functions/runner_manager -q`: 75 passed, 4 of
them new (a retried disconnect doesn't page, two transient failures in a
row page, a 403 pages with no retry, 5xx/429 classification).
- Local CodeRabbit review: no findings.

## Deploy
This is a Cloud Function deployed by Pulumi, so it isn't live after
merge until a targeted `pulumi up` runs on the runner-manager function
(needs `/gcp-admin`).

Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1569441998656635012

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVWJ7FwupZXGBp7cE5X

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 15:35:08 -05:00
Andrew Beveridge ☄️andClaude Opus 5.5 2db5407a5d feat: archive deleted jobs (doc + logs) instead of losing them (v0.295.0) (#1261)
## Why
When NOMAD-1875 was deleted (2026-11-09), its Firestore record and logs
were gone for good, so we couldn't use them to debug the delete-outputs
result afterwards. Andrew's call: job records are **system data**. Keep
them forever, and on account deletion redact the user's identity rather
than deleting the record.

## What
- `FirestoreService.archive_job()` copies `jobs/{id}` to
`jobs_archive/{id}`, adding `archived_at`, `archived_by` and
`archive_reason`. It also copies the `logs` subcollection to
`archived_logs` without `ttl_expiry` (the TTL policy is on the `logs`
collection group, so otherwise they'd expire after 30 days), and the
`audio_edit_sessions` subcollection.
- `JobManager.delete_job()` archives first and **aborts the delete if
the archive fails**.
- Callers pass who deleted the job and why: `admin_delete` /
`owner_delete` from `DELETE /api/jobs/{id}`, and
`cleanup_distribution_delete`.
- **Account deletion** now also redacts archived jobs: owner/customer
email becomes `deleted:<uuid>`, and IPs, notes and contact fields are
removed. These are single-field queries, so no new composite index is
needed. Archived jobs are never deleted.
- Bulk test cleanup (`delete_jobs_by_filter`) still hard-deletes.
- `scripts/get_job.py` falls back to the archive. Documented in
`docs/ARCHITECTURE.md`.

No infra change: Firestore creates the collection on first write, and no
TTL policy is set because archived jobs are kept forever.

## Testing
- Unit: archive happens before delete; the delete is aborted when the
archive fails; archived jobs are redacted on account deletion (including
made-for-you customer jobs).
- Emulator: `backend/tests/emulator/test_job_archive_emulator.py` checks
the doc, logs (no `ttl_expiry`) and audio-edit sessions are copied, and
that a missing job returns False.
- `test_account_deletion`, `test_job_manager`, `test_jobs_api`,
`test_services`, `test_api_integration`: 195 passed.
- Local CodeRabbit: no findings.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVUwKapcjF8czVQrPD5

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 12:06:51 -05:00
Andrew Beveridge ☄️andClaude Opus 5.5 47c6691823 fix(admin): delete-outputs dialog shows success as green; YouTube 404 = already deleted (v0.294.3) (#1260)
## Summary
When Andrew deleted NOMAD-1875's outputs (2026-11-09), the result dialog
showed **YouTube, Dropbox and Google Drive all red** ("Partial
Success"). In reality:
- **Dropbox and Drive were deleted fine.** The backend logged `Dropbox:
success, GDrive: success`, and the brand code was recycled, which only
happens when both succeed. The dialog only treats `"deleted"` as green,
but the shared cleanup helpers (`published_outputs_cleanup.py`) return
`"success"`. The regression E2E mocks also used `"deleted"`, which is
why they never caught it.
- **The YouTube video was already gone.** The API returned 404 and
yt-dlp says "removed by the uploader", so it was removed outside gen.
The delete reported this as `failed`, with no message in the dialog.

## Changes
- `frontend/app/admin/jobs/page.tsx`: green for `success`.
`failed`/`partial` now show a reason, and a partial Drive delete lists
the files left behind ("Not deleted: cdg").
- `KaraokeFinalise.delete_youtube_video`: an HTTP 404 counts as already
deleted (warning log, returns True). Other errors (e.g. 403) still fail.
- `frontend/e2e/regression/admin-job-detail.spec.ts`: mocks now use the
real backend statuses, plus a partial-Drive assertion.
- Version 0.294.3.

## Testing
- `tests/unit/test_karaoke_finalise/test_youtube_integration.py -k
delete`: 6 passed (2 new: 404 = success, 403 = still a failure).
- Playwright regression spec: left to CI, since frontend deps aren't
installed in this worktree.
- Local CodeRabbit: 1 minor finding (show per-file Drive outcomes),
fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVUwKapcjF8czVQrPD5

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 11:57:38 -05:00
Andrew Beveridge ☄️andClaude Opus 5.5 d59b5bf3bd fix(gdrive-validator): don't report gaps for tracks still uploading (#1259)
## Summary
On 2026-11-07 the validator emailed **"SEQUENCE GAPS: CDG missing 1872,
MP4-720p missing 1872"**. That was a false alarm, caused by a race:

- NOMAD-1871 completed at 12:20:23 UTC. Its delayed post-job check ran
at **12:24:47**.
- NOMAD-1872 was in `packaging` from 12:23:39 to 12:25:40. Its 4K MP4
was already in Drive, but the 720p MP4 and the CDG were not.
- The global-max gap logic saw 1872 in MP4 but not in the other two
folders. All three files (`gdrive_files` in Firestore) landed about a
minute later, and the later post-job runs (12:29 and 12:36) found no
issues.

## Change
- `list_folder_contents` can now also collect each file's Drive
`createdTime`.
- New helper `recently_uploaded_sequences()`: a track with any file
created within `IN_FLIGHT_GRACE_MINUTES` (default 30) counts as still
uploading, and `validate_files(..., in_flight=...)` leaves it out of the
gap check.
- These tracks are still covered by their own post-job trigger (the
targeted `brand_code` completeness check, which this change doesn't
touch) and by the daily 9 PM run.
- Docs: `docs/GDRIVE-VALIDATOR.md` (new env var, plus a note on the race
in the gap runbook).

## Testing
- `infrastructure/functions/gdrive_validator/test_main.py`: 40 passed. 3
new tests cover a track still uploading, an old missing track (still
reported), and bad timestamps.
- `backend/tests/test_gdrive_validation_delay.py`: 13 passed.
- Local CodeRabbit review: 2 minor findings, both fixed (env var parse
fallback, offset-less timestamps).

## Deploy
This only affects the Cloud Function, so no backend version bump.
Deploying it takes
`infrastructure/functions/gdrive_validator/deploy.sh`, which needs
`/gcp-admin`.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVUwKapcjF8czVQrPD5

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 22:23:25 -05:00
Andrew Beveridge ☄️andClaude Opus 5.5 bb01e9e36a feat(kn-data-sync): live Spotify songs in the song-id export (v0.294.2) (#1258)
## Summary
Companion to the karaoke-decide PR that keeps every live Spotify search
result forever (`karaoke_decide.spotify_live_tracks`). With this change,
those songs join the daily **song-id export**, which feeds:
- the kjbox on-device matcher (`song_id.db`);
- decide's song-search service;
- the mobile app's offline index.

So a song we look up once, like "Cole Phillips – Roll of Dice" (a 2025
release missing from the July 2025 Spotify snapshot and from
MusicBrainz), is known everywhere from the next build on, with no
Spotify or Gemini call.

## Changes
- `kn_data_sync/main.py`: `SPOTIFY_LIVE_UNION` is appended to
`SONG_ID_SQL` when the table exists. Before decide's first live search
creates it, the union is skipped with an INFO log.
  - Grouped by lowercased artist/title.
- Popularity uses the D5a estimate: 0.8 × the artist's popularity (via
`art`), capped at 50, else 30. The API no longer returns track
popularity.
- Docs: API.md. Version 0.294.2.

## Testing
- `test_main.py`: union added when the table exists, skipped when it
doesn't, four columns and the estimate (9 passed).
- **BigQuery dry run** of the full export with the union (inline row in
place of the table): valid, ~8.2 GB scanned (unchanged).
- Local CodeRabbit review: 0 findings.

## Deploy
The function is deployed by hand:
`infrastructure/functions/kn_data_sync/deploy.sh` (`gcloud functions
deploy`). CI doesn't deploy it, and it needs `/gcp-admin`.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVUQ68KFYtZF7vEfhAq

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 19:57:47 -05:00
Andrew Beveridge ☄️andClaude Opus 5.5 c133b64c7d fix(quick-version): retry LRCLIB with a cleaned title + honour lyrics overrides (v0.294.1) (#1257)
## Why
At a kjbox show on 2026-11-06, the make-it quick version for Sombr "My
Body Isn't Ready (Orchestral Version Live From Sound City)" (job
`f9764a9f`) rendered with only "(lyrics unavailable)" on screen. The
singer heard the backing track with no lyrics. LRCLIB only has this song
under the plain title.

## What
- `fetch_lrclib_lyrics`: if the exact `/get` and `/search` find nothing,
retry `/search` once with the title cleaned (`clean_title`: trailing
`(…)`/`[…]` groups and `- Live …`/`- Remaster` style suffixes removed).
- A synced hit from the retry keeps its timing only when its length is
within **4 s** of the audio, because a different length usually means a
different recording (studio vs live). Any other hit is used as plain
lyrics (constant crawl).
- Plain lyrics found under the exact title still win, and titles with
nothing to clean make no extra request.
- The quick render now searches with the job's
`lyrics_artist`/`lyrics_title` overrides, else the title without a tempo
label, the same names the review page uses. The displayed artist/title
are unchanged.
- Companion: kjbox v0.150.0 (nomadkaraoke/kjbox#314, deployed) lets the
KJ add lyrics on the box when a draft still comes back without any.

## Testing
- `backend/tests/test_quick_version.py`: 56 pass, including 17 new tests
(cleaned-title retry: matching sync, wrong-length sync downgraded to
plain, no duration, exact plain wins, failure paths, no retry for plain
titles; `clean_title` cases; lyrics-search names incl. tempo label and
overrides).
- Live LRCLIB check: `fetch_lrclib_lyrics('Sombr', "My Body Isn't Ready
(Orchestral Version Live From Sound City)", 224.5)` returns **synced, 37
lines**. Before this change it returned nothing.
- Local CodeRabbit review: 0 findings. Full suite left to CI.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVWXLyvTmHNnqQu3rzA

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 18:56:59 -05:00
Andrew Beveridge ☄️andClaude Opus 5.5 cd11088be5 feat(match-judge): ground AI corrections in live Spotify (v0.294.0) (#1256)
## Summary
At a live kjbox night (2026-11-05, job `f01c9997`) a singer typed "Cole
Phillips – Roll of Dice", a 2025 release missing from our BigQuery
catalogue. Gemini "corrected" it to "Roll of the Dice" twice:
1. in the free-text resolver;
2. in the job form's full judge, after the singer had tapped "keep what
I typed".

Both times the catalogue check found nothing, and the guess still went
out as `confident` ("No catalog match → AI verdict unchanged").
Meanwhile the audio search had found the real track on Spotify at score
1.0.

This PR asks **live Spotify** before trusting Gemini, and **grounds**
every AI correction in real data.

Depends on karaoke-decide's new `GET /api/partner/gen/spotify-search`.
Until that's deployed, the call returns `None` ("no evidence"). That is
safe, but it makes AI corrections non-confident in the meantime, so
**merge and deploy decide first**.

## Changes
- `catalog_proxy_service.spotify_search`: calls decide's partner route
with `X-Partner-Secret` (the existing `decide_partner_secret`). Cached
10 min. Returns `None` when there's no evidence. Never raises.
- `match_judge/service.py`, fast and full passes:
- When the catalogue can't confirm the typed song, an exact live-Spotify
hit is a confident catalog verdict. No Gemini call: "Roll of dice" →
"Roll of Dice".
  - Spotify results go to Gemini as candidates.
- **Grounding rule:** a confident AI cosmetic/content verdict must name
a song in the candidates, or found by one Spotify lookup of it.
Otherwise `confident=False`, which both clients already show as "Did you
mean…?" and never apply.
- `match_judge/free_text.py` (`resolve_and_tidy`):
- Live Spotify first, capped at 3 s so the worst case stays inside
decide's 25 s wait.
  - A result that *is* the query (either order) answers it directly.
  - Otherwise the results go into the Gemini prompt.
- An AI answer the tidy can't confirm is not confident. A timed-out tidy
keeps today's behaviour (`unavailable`, not cached).
- Docs: API.md (judge + resolver), LESSONS-LEARNED entry. Version
0.294.0.

## Testing
- New and updated unit tests:
`tests/services/match_judge/test_service.py` (Spotify confirm in
fast/full, catalogue hit skips Spotify, unconfirmed downgrade with the
Cole Phillips case, second-lookup confirm, any credited artist, Spotify
failure, ambiguous untouched, merge), `test_match_judge_free_text.py`
(exact Spotify answer without Gemini, both word orders, downgrade, AI
pick from live results kept, results in prompt, Spotify timeout/failure
fallback), `tests/services/test_catalog_proxy_spotify.py` (secret
header, cache, no-evidence cases, network error, no secret).
- `make test-backend-unit`: 7212 passed. The 8 failures in
`test_emulator_integration.py` also fail on `main` without the emulator
stack.
- Local CodeRabbit review: no findings.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVUQ68KFYtZF7vEfhAq

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 20:00:50 -05:00
Andrew Beveridge ☄️andClaude Opus 5.5 5c41be1d3b fix(error-monitor): ignore runner-manager's other Cloud Run abort wording (#1252)
## Summary
On 2026-11-04 at 10:00Z a PR opened on python-audio-separator queued a
burst of `workflow_job` webhooks. 22 of 54 deliveries got a 500 from
Cloud Run while `github-runner-manager` scaled up. The error monitor
alerted because Cloud Run used a second wording, "aborted after waiting
too long for an available instance", which the existing ignore pattern
didn't match.

There was no impact: all self-hosted jobs in the affected runs got
runners within about 3.5 minutes and passed.

## Changes
- **Error monitor:** match both Cloud Run abort wordings for
`github-runner-manager`, but only on **webhook** requests (the bare
function URL). Cloud Scheduler ticks (`?action=check_idle`) and entries
with no URL still alert. `_SERVICE_URL_IGNORE_PATTERNS` now accepts a
predicate on the split URL as well as an exact path.
- **Runner manager (Cloud Function):** the ignore relies on the 5-minute
stalled-job tick catching dropped webhooks, so that tick's own failures
now log as structured `severity: ERROR` and page. That covers a
redispatch crash, a repo skipped in the scan, a job given up at 60 min,
and a failed re-dispatch. Before this, they were severity-less prints
that nothing alerted on. Both changes came from CodeRabbit feedback.
- Corrected the pattern's reason: GitHub did not redeliver these
(`redelivery: false`).
- Version 0.292.1; merged latest main.

## ⚠️ Deploy
`infrastructure/functions/runner_manager/` only goes live with a local
`pulumi up` (CI never runs Pulumi). Apply it **before** merging, so the
safety net's alerting is live before the webhook-abort ignore ships with
the backend deploy.

## Testing
- `tests/unit/services/error_monitor/`: 452 passed (webhook,
scheduler-tick, no-URL and other-service cases for both wordings).
- `infrastructure/functions/runner_manager`: 71 passed. The 4 updated
tests fail against main's `ephemeral.py`.
- Full `make test` not run locally; relying on CI.

Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567479055887114363

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVVTKcWCcDArTQrjS82

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* Runner-manager webhook errors are filtered more precisely. Recognized
abort messages are ignored only for matching webhook requests;
scheduler-tick errors, requests with other URLs, and requests without a
URL continue to trigger alerts.
* Runner-manager job-processing failures are recorded as structured
error logs, including stalled-job scans and dispatch failures. Give-up
logs now report the configured limit rather than the job’s current time
in the queue.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 19:58:46 -05:00
Andrew Beveridge ☄️andClaude Opus 5.5 a1cbe1c0d4 feat(encoding): retire the GCE c4d pair; run on AWS + c2df only (0.293.0) (#1255)
## Why
Since 2026-10-31 the AWS EC2 worker (`aws-encoding-worker-1`) has
handled every encode and distribution job. 27 jobs went to it, with no
Cloud Run fallbacks, no `encoding_stuck`, and nothing encoding-related
in #alerts. The c4d Spot pair (`encoding-worker-a`/`-b`) now only costs
~$8/mo of Hyperdisk. `encoding-worker-fallback-c2df` stays as the GCP
fallback.

## What
Deleting the pair needs code changes first, because several paths
assumed it exists:
- **`encoding_worker_manager`**
  - The primary is optional: an empty `primary_vm` means none.
- `ranked_candidates` skips a missing primary and always sets
`is_primary`, so with no primary a successful start records the AWS/c2df
override. Previously the legacy index-0 rule would have cleared it.
- `ensure_primary_running` and `ensure_preferred_running` start a ranked
candidate. If there are no candidates they raise `NO_CANDIDATES`.
- `demote_active_worker` looks up the override's `machine_type`, so the
AWS box can now be cooled down. It then clears the override so the retry
goes elsewhere.
- **`encoding_service`**
  - When `active_url` is empty, it targets the top-ranked candidate.
- It also invalidates its cached URL after a single-candidate fallback.
- **CI deploy (`ci.yml` + `deploy_promote.py`)**
- While AWS serves, a c2df green is validate-only: it writes
`validated_version`, keeps the AWS override, and stops the green. Before
this, it would have replaced the AWS override with c2df.
- The worker-change gate now diffs against `validated_version`, because
`primary_version` stops moving.
- An in-place restart of a serving override first waits for it to drain
(15-min cap). If it is still busy, the restart is skipped.
- **Idle function:** a missing VM is logged as a WARNING returning
`NOT_FOUND`, not an ERROR.
- **Error monitor:** watches c2df instead of a/b.
- **Pulumi:** drops the a/b VMs and static IPs. The prod preview shows
exactly 4 deletes. A separate, unrelated
`youtube-comment-autoreply-scheduler` drift shows up too; it will not be
applied.
- **Docs:** CLAUDE.md, ARCHITECTURE, DEVELOPMENT and TROUBLESHOOTING.

## Rollout (after merge + deploy)
1. Back up `config/encoding-worker`, then set `primary_vm`,
`primary_ip`, `secondary_vm` and `secondary_ip` to `""`.
2. Apply `pulumi destroy --target` on the 4 a/b URNs only.
3. Redeploy the idle function with `deploy.sh`.
4. Verify: one prod encode on AWS, warmup, and the logs.

**Rollback:** in Firestore, delete `speed_rank_overrides` and set
`aws_video_worker=false`. Restoring the pair would need a revert plus
`pulumi up`, which gives new static IPs.

## Testing
- New `backend/tests/test_encoding_worker_no_primary.py` (13 tests),
plus new deploy_promote, idle-function and error-monitor tests.
- Full backend suite: 7188 passed. The 8 `test_emulator_integration`
failures also fail on main, because of a local emulator from another
session.
- CodeRabbit CLI: no findings. An independent review found 2 issues,
both fixed: the demote retry path, and the in-place restart drain.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVTkatGnrYFhCzq4PTe

@coderabbitai ignore

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 18:50:54 -05:00
Andrew Beveridge ☄️andClaude Opus 5.5 d63c9781e1 feat(replay): sync-triage verdict bar + vocal waveforms for stems-purged jobs (v0.292.0) (#1254)
## Summary
Andrew found the standalone triage page unintuitive, with no way to scan
visually. So the **real review UI in replay mode** becomes the triage
tool for the word-timing reference set (full-auto v2): anchors, gaps,
waveforms and audio he already knows.

## Changes
- **`ReplaySyncVerdict` bar** (replay only, admin): Perfectly synced /
Not perfect, reason chips, notes, Save / **Save & next job** through the
existing `queue` URL param.
  - It shows the saved verdict and flags unsaved changes.
- It saves via `POST /api/review/{id}/sync-reference`, the same store as
the review-modal checkbox from #1242, which now also accepts `reasons`
(trimmed, max 10).
- **Vocal waveform for old jobs:** storage retention purges stems after
30 days, so replaying old jobs showed no waveform. `_review_vocals_path`
now falls back to the backfilled `training/lead_vocals/{job_id}.flac`,
but only for stems-purged jobs. It's used by the vocals stream and
vocals-peaks.
- Docs: REPLAY.md § Sync triage.

## Testing
- Backend: reasons stored trimmed and capped; waveform fallback
(registered stem preferred, purged → training copy, unpurged never
checks storage). `test_routes_review` + `test_derived_vocals` +
sync-reference tests: 143 pass.
- Frontend: `ReplaySyncVerdict.test.tsx` (save & next, reasons + notes,
saved state, unsaved flag, disabled until chosen): 5 pass. tsc shows no
new errors.
- **Manual:** local replay (read-only against prod data) on a Feb-2026
stems-purged job (83414c76). The verdict bar renders, and Waveforms mode
shows the vocal waveform via the fallback (vocals-peaks 200).
- CodeRabbit ran locally; its one finding (unsaved state) is fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVVVPsvPbXFhRYY5VSJ

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 21:33:16 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 394def0b53 fix(youtube): get source Opus from Tornado reliably; keep signed URLs out of logs (0.291.2) (#1253)
The first full pipeline run with Tornado (job 586644dd) delivered 128k
AAC rather than Opus. `audio_only` + `format: opus`, which #1247 relied
on, is unreliable on our plan: `Tk3ty5ansqs` returned Opus on 11-03 and
AAC on 11-05, and 4 of 5 videos returned AAC with a `reencoding_notice`.

Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567241547739570298

## Fix
Request `format: webm, max_resolution: 360` and strip the video with
`ffmpeg -c:a copy`. All 3 videos tried this way carried Opus, and
billing was identical to audio-only on the videos compared (e.g.
21,026,831 bytes for `dQw4w9WgXcQ`).

Checked against a real flacfetch job input (`Tk3ty5ansqs`): same audio
stream MD5, same `.webm` container, 5,531,092 vs 5,529,792 bytes.

Also stops httpx logging signed delivery URLs at INFO. They were
appearing in job logs for 586644dd.

## Testing
- 83 mocked tests pass, including a real ffmpeg demux test (audio
packets unchanged, video dropped) and a failure test.
- Not yet run through the deployed pipeline with this format; that test
follows this merge.

## Left as is
Local CodeRabbit flagged that cancelling during demux doesn't stop the
ffmpeg thread. Skipped: the copy takes under a second, both files are in
a temp directory that gets discarded, and nothing else reads them.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVTvqXyL4QyA2EjzQMG

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 05:23:33 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 688ca162f8 infra: give audio-download-job the Tornado key (0.291.1) (#1251)
`audio-download-job` does most YouTube downloads, and it gets secrets
from Pulumi rather than CI's API deploy, so #1247's wiring alone would
have left Tornado off for them.

Already applied to prod with `gcloud run jobs update --update-secrets`
(env diff checked: one variable added). This records it in Pulumi so a
later apply doesn't remove it. Pulumi preview couldn't run from Andrew's
Mac: its GCP provider fails with `socket is not connected` on IPv6 token
requests, while `gcloud` works. Run a targeted preview later to confirm
no diff:

`scripts/gcp-admin pulumi preview --stack prod --diff --target
'urn:pulumi:prod::karaoke-gen-infrastructure::gcp:cloudrunv2/job:Job::audio-download-job-us-east4'`

Merging also triggers the API deploy, which now finds secret
`tornado-api-key` and wires it there.

No local CodeRabbit review on this small config change, so the bot may
review it.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVTvqXyL4QyA2EjzQMG


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
  * Updated credential availability for audio download processing.

* **Documentation**
* Clarified how the Tornado API credential is provided to the API
service and audio download processing.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 23:36:33 -05:00
Andrew Beveridge ☄️andClaude Opus 5.5 ca05245469 feat(youtube): TornadoAPI first for YouTube downloads, flacfetch as fallback (v0.291.0) (#1247)
YouTube downloads depended on one logged-in cookie session on one VM.
This tries TornadoAPI first and keeps flacfetch as the fallback. Plan:
workspace `docs/archive/2026-11-03-tornado-youtube-download-plan.md`.

Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567241547739570298
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567252495535841386
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567252497620537415

## Inactive until the secret exists
Nothing changes in prod when this merges. The deploy adds
`TORNADO_API_KEY` only once secret `tornado-api-key` has a version, and
creating it needs `/gcp-admin`. `YOUTUBE_DOWNLOAD_PROVIDER=flacfetch`
switches it off.

## What changed
- **`tornado_client.py`:** create job, poll with a 180s deadline
(cancels on timeout), stream the delivered file, delete it from Tornado.
Retries with one idempotency key per download; honours `Retry-After`.
- **`YouTubeDownloadService`:** Tornado first for YouTube, then
flacfetch. WARNING per fallback, ERROR on 401/402/403, so an outage or
lapsed billing is visible. Non-YouTube URLs stay on flacfetch.
- **Availability check:** Tornado metadata first, because flacfetch's
check rejects valid videos while its cookies are bot-blocked. If the
lookup fails, flacfetch decides as before.

## Spike results (real API, 11 jobs)
- `audio_only` + `format: opus` gives **packet-identical audio** to
flacfetch's real job input (`Tk3ty5ansqs`, stream MD5 match).
`audio_codec: copy` or no `format` gives 128k AAC instead, so the
request omits it.
- `mBJivIsN0n0`, which bot-blocked job 526bf46c, completed in about 5s
of processing.
- An age-restricted video completed without cookies; a missing video
ends in `Warning`.
- Billed about 25.6 MB for a 4-minute song.

## Testing
- 79 mocked tests pass (`test_tornado_client.py`,
`test_youtube_download_service.py`).
- Live run through `YouTubeDownloadService` with the GCS upload stubbed,
using a URL with a playlist parameter: only the single video was sent.
- Local CodeRabbit: all findings fixed; the last small fix wasn't
re-reviewed.

## Not verified
- **A full pipeline job with this input.** It can only run once the
secret exists. Files are Ogg Opus named `.ogg` (flacfetch's were WebM
Opus), so the codec is the same but the container is new to this path.
Run the daily E2E or one admin job first.
- **Deviation from the plan:** the Tornado job id is logged but not
saved on the gen job, because a restart loses the in-request download on
the flacfetch path too. A restart can orphan one job, which costs a
fraction of a cent.
- **CI's secret check:** it depends on the deploy service account being
able to read the secret; the log line `tornado-api-key wired:` shows the
result.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVTvqXyL4QyA2EjzQMG

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 00:00:40 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 3a563ab3e6 fix(jobs): main is failing: restore the start-review guard removed twice (0.290.4) (#1250)
#1248 and #1249 were opened at the same time to resolve the same
duplicate guard. Each removed a different one, so `start_review` on main
now has none: 6 cases in `test_jobs_api.py` fail, no deploy can finish,
and a stale push on a finished song would 500 again.

Restores #1243's guard, as #1248 intended, and the test assertion #1249
loosened. `backend/tests/test_jobs_api.py` passes locally.

**Other sessions: please leave this route alone until this merges.**

No local CodeRabbit review on this small unblocking fix, so the bot may
review it.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVTvqXyL4QyA2EjzQMG

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 23:47:45 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 dd09aab941 fix(jobs): main is failing: resolve duplicate start-review guards (0.290.3) (#1249)
#1243 and #1246 both added a 409 guard to `start_review` and merged
minutes apart. #1243's test expects its own message, but #1246's guard
now runs first, so main's Backend Unit Tests fail (3 cases) and no
deploy can finish.

Keeps #1246's guard, because #1243's would refuse statuses that have a
valid transition to `in_review`. The race handler is unchanged. The test
still checks 409, no transition, and no enum text in the message.

`backend/tests/test_jobs_api.py` passes locally. No local CodeRabbit
review on this small unblocking fix, so the bot may review it.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVTvqXyL4QyA2EjzQMG

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* Review requests for jobs that cannot enter review continue to receive
a conflict response. Eligible jobs can proceed into review, with
review-start actions recorded only when a job transitions from awaiting
review.
* Updated the conflict-response test to check that internal status
naming is not exposed.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 23:35:29 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 5ae9d407b2 chore: remove duplicate start-review 409 guard (v0.290.3) (#1248)
#1243 and #1246 merged within minutes of each other and both added a 409
to `start-review`. This removes the one from #1246. The guard from #1243
stays, since it also handles losing the race to auto-approval. No
behaviour change for a finished job, and the #1246 test for it stays.

The two guards differ for statuses like `lyrics_complete`, which could
technically move to `in_review`: under the #1243 guard those already get
a 409.

Docs-free, small diff; no local CodeRabbit run, so the bot can review
it.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVTajJiZnzPUt9qsVEJ

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* Starting a review is now limited to jobs awaiting review or already in
review; other job statuses receive a conflict response.
* Repeated requests to open an eligible review remain safe, and
concurrent state changes are handled using the job’s current status.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 18:30:59 -05:00
Andrew Beveridge ☄️andClaude Opus 5.5 a10e548606 fix(url-jobs): retry a YouTube bot block, then point to Audio Search (v0.290.1) (#1244)
Implements fix C from section 4 of the 2026-11-03 alerts triage handoff.
Job 526bf46c, from a paying customer, failed within 3s of a YouTube bot
wall and showed the raw yt-dlp error. The credit was refunded, and the
customer then uploaded an MP3.

## Changes

- **Download:** `create-from-url` retries a bot-blocked download once
after 30s. Other download errors behave as before.
- **Still blocked:** the job fails with a translated message pointing to
Audio Search. The raw error goes in `error_details`. The response stays
a 500, because the frontend treats 503 as a transient backend blip.
- **Pre-check:** a bot-blocked availability check now waits 30s and
checks again before rejecting. No job or credit exists yet at that
point. A repeat block returns the same translated message instead of the
old hardcoded English. CodeRabbit raised this: without it, the retry
never ran when the pre-check caught the block.
- **Logging:** the bot-block paths log WARNING rather than ERROR,
because `fail_job` already raises the alert for the failure.

`audio_search.py` has its own bot-block handling, which this PR doesn't
change. The monitor and start-review changes are in a separate PR.

## Depends on flacfetch

The retries only help much with
https://github.com/nomadkaraoke/flacfetch/pull/61 deployed, since that's
what requests fresh cookies straight away. Without it, new cookies only
arrive from the keeper's 30-minute probe, so a 30s wait rarely helps.
This PR is safe to deploy first, but it adds up to 30s before the same
failure.

## Testing

- New `test_create_from_url_bot_block.py`, written test-first: retries,
final failure, pre-check, non-bot errors and all 33 locales.
- Backend tests selected with `-k "url or youtube or audio_search or
i18n or translation"`: 785 passed, 5 skipped. Emulator and integration
tests excluded. I did not run the full `make test`, so CI is the full
check.
- Backend translation key parity passes.
- Local CodeRabbit review: 1 finding, fixed. The fresh re-review had no
new findings.

**Not verified:**
- Nothing has run in production, and I created no test jobs.
- **The 32 new translations were written by hand, not by
`translate.py`.** gcloud auth had expired, so the Gemini key wasn't
available. To replace them, run `translate.py` for that key with a valid
login.
- ruff reports F823 for `asyncio` at about line 647 of `file_upload.py`.
It's also on main, in a different function, and I left it alone.

Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567241547739570298
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567252495535841386
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567252497620537415

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVTvqXyL4QyA2EjzQMG

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 23:18:13 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 08b146fce0 fix: no review push before AI suggestions are scored; start-review 409 after review ends (v0.290.1) (#1246)
Andrew's job `7f89f884` (2026-11-03) got a "review your lyrics" push,
then auto-approved about 3.7 minutes later. Tapping the push afterwards
left the app on its loader.

**False review push.** The screens worker's first score ran at 00:15:40,
before the proactive auto-correct finished at 00:15:46. Without
suggestions the lyrics scored `needs-review`, so `hold_for_audio` was
false and review opened with its push. The audio worker's re-score then
saw the suggestions (`auto/ai-resolved`) and auto-approved. In the last
14 days 5 jobs auto-approved, and 4 of them opened review first
(typically about 3.7 minutes before).

`route_after_scoring` now waits for the suggestions and scores again
when the first pass ran without them, then holds or auto-completes. If
suggestions can't be had, or the re-score reaches no verdict, the first
verdict stands and review opens as before.

**start-review on a finished job.** `complete -> in_review` raised an
invalid transition: a 500, an ERROR log (4 in prod for this job, which
the error monitor can turn into an #alerts message), and the app
retrying as if it were transient. It now returns 409 without attempting
the transition. The app's handling is in nomad-mobile #44.

**Tests:** `test_review_gate.py` and `test_jobs_api.py` pass with new
cases. `test_internal_api.py` has 7 auth test failures locally that also
fail on the main clone, so they aren't from this change.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVTajJiZnzPUt9qsVEJ

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 18:13:25 -05:00
Andrew Beveridge ☄️andClaude Opus 5.5 70bdd0e693 fix(alerts): start-review 409, per-cause crash patterns, one alert per job failure (v0.290.1) (#1243)
Section 1 of the 2026-11-03 #alerts triage handoff: a stale push on a
finished song got a 500, and the error monitor could stay silent on a
new crash.

Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566980744448843886
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566980746394996879
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566980747905073224
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1565697725280948294
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1565803187334815775
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1565818418072916110
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566225593354420226
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566256141074169987
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567207141994135654
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567252495535841386
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567252497620537415

## What changed

**1a. start-review:** a job not in review gets 409
(`jobs.jobNotInReviewState`) with no transition attempted. Losing the
auto-approver race is a 409 too. `correction-data` has the same race,
and its `except Exception` turned it into a 500; it now returns 409
(`except HTTPException: raise` added). Its status gate still returns
400, as before.

**1b. Crash patterns:** `Exception in ASGI application` and
`request_audit_error` become `Unhandled exception: <Type: message>`,
read from the entry's `exception` field. The handoff's suspicion is
confirmed in `upsert_pattern`: an existing `new` pattern returns
`is_new=False`, so a second, different crash stayed silent. Both log
lines of one crash now share one pattern. The path and method go in the
sample only.

I read the line after the last frame rather than the traceback's last
line. pydantic's last line is a docs link shared by every failure, and
group tracebacks end in a separator.

**1c. One alert per failure:**
- YouTube ids are normalized out of signatures, which also fixes the
false "first time" line in `ops_alerts`.
- `Job <id> failed` (both call sites in `job_manager`) logs WARNING,
since the status write sends the job-failed alert.
- `URL download failed:` is ignored in the monitor, not demoted, because
`file_upload.py` belongs to the section 4 PR.
- `[job:<id>] Download failed` logs WARNING for the expected
`DownloadError`/`FlacfetchServiceError` types. This wasn't named in the
handoff, but it is the same duplicate on the audio-search path.
Unexpected exceptions there stay ERROR with a traceback.

**1d. Suppressions and demotions:**
- Zalo's `zaloJSV2 is not defined`: dropped at `/api/client-errors`,
only with a Zalo UA.
- `python -c` experiments on `audio-separation-job`: ignored, using the
handoff's regex.
- Spotify 403 on `/lyrics`: WARNING with `response.text[:200]`. 401 on
`/lyrics` and 401/403 on `/search` stay ERROR.
- Dropbox token probe: WARNING. A rejected refresh used to log nothing,
so it now logs ERROR, as does having no refresh credentials.
- decide-search health aborts: this was a small change. `should_ignore`
takes an optional `request_url` and matches the exact path
`/api/search/health`.

**1e. Cold start:** `anyio._backends._asyncio` is added to
`_LAZY_ROUTE_IMPORTS`. The prod stack from 10-30 confirms the stall was
in that import. Lifespan also sends one `/api/health` request through
the app in a worker thread, bounded by the existing 30s timeout and
best-effort.

## Testing

- TDD: new tests failed first, then passed.
- Affected suites: 826 passed.
- `backend/tests` (excluding `backend/tests/emulator`): 7088 passed, 8
failed, 29 skipped. All 8 are in `test_emulator_integration.py`, and
that file also shows 8 failures at the base commit a8f9618a. I did not
compare the 8 test names one by one.
- `tests/unit` outside the listed files was not run.
`test_audio_fetcher.py` and `test_remote_cli.py` fail to collect in my
environment; I didn't change either file.
- Read-only replay of real prod entries through the new code: the 11-03
crash gave one specific pattern with `POST
/api/jobs/7f89f884/start-review` in the sample, the decide-search health
aborts and the experiment traceback were ignored.
- Local CodeRabbit review completed with 0 findings.

## Not verified

- Nothing is live, so the alert volume and the cold-start stall haven't
been measured. The warm request has only run in tests, not on a real
Cloud Run cold start.
- A request's first-use imports beyond the anyio backend aren't
individually checked.
- My test environment has no `flacfetch`, so its lazy imports only log a
warning here.
- Zalo: the trigger is one report (11-03).

## Known leftovers

- `validate_state_transition` still logs ERROR before raising, so the
auto-approver race (and the existing in_review race) still emits one
`Invalid state transition` ERROR even though the client now gets a clean
response. The stale-push case no longer reaches it. I left this because
other callers rely on that log.
- A crash that also logs its own ERROR first still gives two patterns,
the ERROR and `Unhandled exception`.
- The first bot block after deploy shows "first time" once, because the
normalized signature is new.
- Old `Exception in ASGI application` and `request_audit_error` patterns
stop recurring and will auto-resolve.
- If the section 4 PR rewords `URL download failed:`, the ignore entry
in `known_issues.py` needs the same wording.

## For Andrew (infra and cost, not changed here)

- Cloud Run startup probe and min-instances for `karaoke-backend`, if
cold-start stalls continue after this ships.
- decide-search's min-0 is why its health pings abort. The suppression
hides those, but real search paths will still alert on the same abort.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVTvqXyL4QyA2EjzQMG

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 23:04:04 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 ea69386e50 fix(ci): Java 25 for the Firestore emulator (#1245)
`google-cloud-cli-firestore-emulator` 592.0.0 is compiled for Java 25
(class file 69), and the package is installed unpinned from apt, so CI's
Java 21 fails with `UnsupportedClassVersionError`. This blocks every PR:
Backend Emulator Integration and Local Stack E2E both failed on #1243
and #1244 (runs 41705112128, 41706068761). Main last passed 11-02.
CI-only change; this PR's own run is the test.

@coderabbitai ignore not added: no local review was run on this one-line
CI change, so the bot may review it.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVTvqXyL4QyA2EjzQMG

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 17:53:53 -05:00
Andrew Beveridge ☄️andClaude Opus 5.5 a8f9618a07 feat(review): admin "perfectly synced" mark for the word-timing reference set (v0.290.0) (#1242)
## Summary
Andrew's review depth varies: Thursday live-event jobs often get only a
quick review. Treating every admin review as perfect sync truth
therefore poisons aligner training and evaluation (full-auto v2). This
adds an explicit, admin-only **"perfectly synced"** mark in the lyrics
review. Ticked jobs join the word-timing reference set; unticked means
unverified. Andrew's definition: a confident spot-check (anchors +
waveforms), not necessarily a full listen.

## Changes
- **Backend:** new `POST /api/review/{job_id}/sync-reference`
(`require_admin`).
  - Stores `state_data.sync_reference = {verified, by, at, notes}`.
- `verified: true` needs saved review output: the `file_urls` pointer,
or the direct GCS path for older jobs. Otherwise it returns 409.
Revoking is always allowed.
- **`correction-data`** returns `sync_reference`, so a re-review shows
the existing mark and can revoke it.
- **Frontend:**
- The review modal shows the checkbox plus an optional notes field, to
admins only.
- The modal passes the mark to `onSubmit`. `LyricsAnalyzer` saves it
**after** `submitCorrections` succeeds, so a mark can't sit on unsaved
timing. A failure shows a toast and never blocks the review.
- Once the box is touched, its current state is always sent, so it can
be revoked.
- **i18n:** 4 new strings × 33 locales.
- **Docs:** API.md. Version 0.290.0.

## Testing
- Backend: `test_review_sync_reference.py` (admin store, notes trim,
404, 422, 403 non-admin, 409 without review, direct-path fallback).
Related review suites pass (29).
- Frontend: `ReviewChangesModal.syncReference.test.tsx` (hidden for
non-admin, tick passes the mark, untick sends nothing, saved mark
pre-ticks and revokes, touched-then-unticked sends `false`). The modal
suites pass (29). tsc shows no new errors (21 pre-existing test-file
errors, the same on main).
- **CodeRabbit:** ran locally for 3 cycles and fixed 6 findings. One was
left by design: a pre-ticked mark on an edited re-review is visible in
the modal Andrew confirms, so submitting re-affirms it. The only user is
Andrew.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVVVPsvPbXFhRYY5VSJ

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 20:17:41 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 452a8a3e28 fix(encoding): stems-ready resumes reach the AWS video worker (v0.289.1) (#1241)
## Summary
Found by the prod E2E on 2026-10-31.
- With `aws_video_worker` on, the render worker dispatched the video
worker to AWS, and it parked waiting for stems.
- The **stems-ready resume** came from `audio-separation-job`, which has
no `ENCODING_WORKER_FALLBACK_VMS` env. It never saw the AWS worker, so
it silently started the Cloud Run job.
- The job completed fine, because the Cloud Run job is the fallback, but
its distribution ran from GCP.

## Changes
- `EncodingWorkerManager.fallback_entries()`: in production, when the
env var is missing, read the `encoding-worker-fallback-vms` secret
(cached per process by `get_secret`). This affects the audio, lyrics and
download jobs, which also trigger and warm the encoder. Outside
production it never fetches.
- `_trigger_video_worker_on_aws`: logs when the AWS dispatch is skipped
because AWS isn't the top-ranked worker, so it can't fail silently
again.

## Testing
- 3 new tests: secret fallback in prod, no fetch outside prod, env
preferred over the secret. All 171 related tests pass.
- Local CodeRabbit: no findings.
- After deploy: another prod E2E to confirm the resume reaches AWS.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 16:07:15 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 78264fa23a feat(encoding): run the video worker (distribution) on the AWS box (v0.289.0) (#1240)
## Summary
Phase 2 of moving encoding to AWS; Phase 1 was #1236 and #1239. Design
and rationale are in
`docs/archive/2026-10-29-aws-encoding-worker-plan.md` § Phase 2.

When Firestore `config/encoding-worker.aws_video_worker` is true **and**
the AWS worker ranks first, `trigger_video_worker` runs the video worker
**on the AWS box** instead of starting the Cloud Run job
`video-encoding-job`.
- **Uploads move to AWS:** the Dropbox/YouTube/Drive uploads (~187
GB/mo, ~$18/mo of Cloud Run internet egress) leave from AWS, on credits.
- **No Cloud Run CPU** for the job (~$11/mo).
- **Off by default.** Any failure falls back to the Cloud Run job, so
this path can't drop a job.

## Changes
- **`POST /run-worker` (worker):**
- Spawns `python -m backend.workers.video_worker --job-id …` with the
box's video-worker env.
  - Status is reported via `/status/{id}`.
- A run counts in `/health` `active_jobs` from the moment it's accepted,
so the idle watchdog won't stop the box mid-run.
  - Returns 503 unless the env file exists.
  - `job_id` is pattern-validated.
  - Any startup failure marks the run failed.
- **`backend/services/local_encoded_outputs.py`:** a same-box handoff.
- When the co-located video worker asks (`stash_local_outputs`, set only
where `LOCAL_ENCODED_OUTPUTS_DIR` exists), the encoder keeps local
copies of the finals, keyed by blob path.
- The video worker takes those instead of downloading them back out of
GCS. Without this, the move would *add* ~400 MB/job of GCS egress.
  - Copies are made off the event loop and purged after 6 h.
  - No change on GCE or Cloud Run.
- **`video-worker-env.py` (box, every boot):**
- Mirrors the **live** `video-encoding-job` env, with secrets resolved,
so nothing is hand-copied and nothing drifts.
- Runs as the job's identity, `karaoke-backend@`, via a second WIF
binding (Pulumi `aws-encoding-worker-wif-backend-user`).
- On failure it writes no env file, so `/run-worker` answers 503 and the
Cloud Run job is used.
- **`WorkerService._trigger_video_worker_on_aws`:** checks the flag and
that the AWS worker ranks first, ensures it's running, waits for health,
then POSTs `/run-worker`. Returns False on any doubt.
- **Also:** `startup-aws.sh` creates the backend credential config and
runs the env helper; CI uploads `video-worker-env.py`;
`worker_code_paths.txt` now gates `local_encoded_outputs.py`.

## Rollout (needs `/gcp-admin`)
1. **Before merge:** targeted `pulumi up` of the backend WIF binding.
It's additive.
2. **After stage B of Phase 1** (AWS worker preferred, timings
verified): stop/start the box once so the new `startup-aws.sh` writes
the env, then set `aws_video_worker: true`. Set it to `false` to roll
back.

## Testing
- **27 new tests:**
- handoff stash/take/purge, path-escape guard, and the per-request stash
flag;
- `/run-worker`: disabled, bad module, dedupe, active immediately, spawn
args/env, failure log tail, wheel-not-ready, safe start, unsafe ids;
  - env builder;
- the trigger's AWS branch: off, not first, happy path, didn't start,
refusals, errors.
- **Full backend + unit suites:** no new failures (the 13 local failures
are identical on main).
- **Local CodeRabbit:** 5 findings, all fixed. The code-paths gate test
also caught a missing worker path, now fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 15:02:42 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 20b957958c infra(aws): encoding worker on-demand c7a.4xlarge; stage A status (#1239)
Follow-up to #1236.

- **Instance type:** the AWS encoding worker is now **on-demand
c7a.4xlarge** instead of Spot c7a.8xlarge.
- The Spot create failed with `InsufficientInstanceCapacity`, and Spot
placement scores were 1-3/10 for c7a/c7i/c6a/c8a 4-8xlarge across
us-east-1/2.
- A worker that stops and starts on demand has to win capacity on every
start.
- Cost is ~$82/mo of credits. The 4xlarge matched or beat GCE in the
replay benchmark.
- **Already applied** (instance `i-0c21505d7017356bb`, EIP
3.128.70.236).
- **Docs and plan:** stage A status, plus the live end-to-end test
through the worker HTTP API (render 117s; all 9 outputs; portrait SSIM
0.993, the only difference being anti-aliasing of the header text).

The only code-file change is a comment in
`encoding_worker_preference.py`, so there is no version bump.

No local CodeRabbit run (IaC and docs only), so the bot review stays on.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Operational Updates**
* The AWS encoding worker now runs on an on-demand `c7a.4xlarge`
instance rather than a Spot `c7a.8xlarge`.
* The worker stops when the operating system shuts down, while its root
volume is preserved.
* If AWS cannot start the worker because capacity is insufficient, it is
temporarily deprioritized for 15 minutes, allowing processing to fall
back to GCE.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 12:16:19 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 4c648bd16c infra: grant Google Play Pub/Sub Publisher on the RTDN topic (#1238)
## Summary
Google Play real-time developer notifications (RTDN) for the mobile app
(`com.nomadkaraoke.mobile`) publish to the Pub/Sub topic
`Play-Store-Notifications`. RevenueCat pulls them through its own
subscription (`RevenueCat-Subscriber-appc2bb515ecd`) and calls gen's
`/webhooks/revenuecat`.

The topic was set in Play Console > Monetization setup on 2026-10-31,
but **Send test notification** failed. The topic has no
`roles/pubsub.publisher` grant for
`google-play-developer-notifications@system.gserviceaccount.com`.

## Changes
- New `infrastructure/modules/play_rtdn.py`: one non-authoritative
`gcp.pubsub.TopicIAMMember` (Publisher for Play's service account). The
topic and the RevenueCat subscription were created by RevenueCat's
"Connect to Google" flow and stay unmanaged.
- Wired into `infrastructure/__main__.py`.

## Testing
- `pulumi preview --stack prod --diff --target …play-rtdn-publisher`
(read-only ADC): `+ 1 to create, 292 unchanged`, with exactly the
member, role and topic above.
- CodeRabbit CLI: no findings.
- Infra-only change, so no version bump.

## Apply
CI never runs Pulumi. Apply **before merging**, through `/gcp-admin`:
`scripts/gcp-admin pulumi up --stack prod --target
'urn:pulumi:prod::karaoke-gen-infrastructure::gcp:pubsub/topicIAMMember:TopicIAMMember::play-rtdn-publisher'`
Then resend Play's test notification.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVUH7B2p5YotvzcFsAR

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 11:38:58 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 ce57b4b598 fix(review): Tap To Sync partial pass keeps word order; local stack never reaches prod (v0.288.1) (#1237)
## Summary

### 1. Tap To Sync: a pass that stops before the last word
(customer-facing)
If a pass stopped before the last word (auto-stop about a second after
the line end, or the user pressing Stop), the untapped words after the
last tapped one kept their **old** timings, and those could now start
*before* the last tapped word. On gen's fake fixture line ("Nothing here
was ever transcribed", words 15.5 to 19.5, next line at 20.0), taps at
16.0, 17.2, 18.4 and 19.6 followed by auto-stop left "ever" at 19.6 and
"transcribed" at a stale 18.74 to 19.5. The two words were out of order,
and the video rendered them out of order.

- `frontend/hooks/useManualSync.ts`: new pure
`commitPartialSyncPass(words, synced)` runs in `cleanupManualSync`,
which is both the auto-stop path and the Stop toggle. It clears (sets
start/end to null) every untapped word after the last tapped one that
starts before it, then caps the last tapped word's end before the next
still-timed word in the line. The existing untimed-lyrics guard then
blocks finishing and points the user at those words. This is the same
rule as nomad-mobile's review-core `commitTapSync`
(`packages/review-core/src/tapSync.ts`, branch
`feat/sess-20261030-1930-e2e-health`). It uses the hook's existing
`syncedIndicesRef`.
- Backend backstop, `backend/services/timing_sanitizer.py`
`sanitize_corrections`: within a line, a word that starts before the
previous word's **start** is now treated like any other invalid start.
Small overlaps (a start a few ms before the previous word's end, common
in AI transcriptions) are left exactly as on main, so they get no move,
no WARNING and no edit-modal toast. It is **moved** to the previous
word's end, not cleared. I chose moving because that is how the module
already repairs bad or missing starts, because a cleared word would just
be refilled at prev_end on the next sanitize, and because nulls would
crash the render (`find_missing_timings` only guards the preview). The
module logs one aggregated `WARNING` per submission (never ERROR, since
this is an expected and repaired condition).
- `frontend/lib/lyrics-review/sanitizeWordTimings.ts` (the frontend
mirror, run when the edit modal opens) applies the same ordering rule,
with the same exemption for small overlaps.

### 2. The local stack must not reach production
nomad-mobile's E2E runs gen web inside an Android emulator's Chrome.
- `components/google-analytics.tsx`: GA (`G-D0E63LB9FY`) is no longer
loaded when the page host is localhost, 127.0.0.1 or 10.0.2.2. The check
runs at runtime after mount, so it also covers CI, and the server render
and first client render match.
- `lib/api/autoCorrect.ts` and `lib/api/customLyrics.ts` now use
`API_BASE_URL` from `lib/api.ts` instead of `NEXT_PUBLIC_API_BASE ??
'https://api.nomadkaraoke.com'`. That env var is set nowhere. A local
page now calls its own backend through relative URLs, so it never sends
its bearer token to production.
- New `lib/local-host.ts` is the single list of local hostnames, shared
by `lib/api.ts` and GA. `lib/api.ts` now also treats 10.0.2.2 as local.
- `scripts/local-stack.sh` backend_env: blank `FLACFETCH_API_URL=` and
`FLACFETCH_API_KEY=`, so direnv can't point the local stack at the real
flacfetch.

Version is bumped to 0.288.1.

## Testing
Each change had a failing test written first.
- Frontend jest (hook, sanitizer, edit modals, GA, API clients,
api/local-mode, auto-correct, custom lyrics): 13 suites, **150 passed**.
- New tests: 3 partial-pass hook tests (auto-stop, Stop, and the
keep-and-cap case), 2 sanitizer tests (the stale-word move, and a 20 ms
overlap left untouched), 4 GA tests, and 8 API-base tests (local hosts
use relative URLs, the production host uses the production API).
- Backend pytest: `test_timing_sanitizer.py` +
`test_jobs_corrections.py` **20 passed** (3 new, including a 20 ms
overlap that is left untouched), and `test_routes_review.py` **114
passed**.
- `customLyrics.test.ts`'s AbortSignal test now asserts that the
caller's abort reaches fetch, not that the signal object is identical. A
relative URL makes `apiFetch` manage the request and wrap the signal.

## Notes
- The web hook has no view of the next line, so the cap only uses words
in the same line. The backend sanitizer and segment bounds handle the
rest.
- The render-time `SegmentResizer._sanitize_segment_timings` is
unchanged (it clamps to the window only).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVSLS3eVDqPctfYBMTt



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved lyric timing cleanup when recording or editing stops before
the final word. Out-of-order timings are corrected while small overlaps
are preserved, helping keep lyrics synchronized without changing valid
timings.
* Corrected out-of-order word timings during lyric cleanup and editing.
* **Improvements**
* Local app use now works more consistently across supported addresses,
including Android emulators, with API requests routed appropriately.
  * Analytics scripts are now omitted on local pages.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-30 20:28:19 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 de5dfcd50d feat(encoding): AWS EC2 encoding worker as a pool candidate (v0.288.0) (#1236)
## Summary
Phase 1 of moving encoding to AWS. EC2 is paid for by the AWS Activate
credits (acct 790139457751, expiring 2028-10-31).
- **Plan:** `docs/archive/2026-10-29-aws-encoding-worker-plan.md`.
- **Benchmark** (3 real prod jobs replayed): an EC2 c7a.4xlarge matched
or beat GCE c4d-highcpu-16.

The EC2 box runs the **unchanged** `gce_encoding` worker service and
joins the existing candidate pool as `provider: aws`. The existing
failover, active override, lost-job detection and stockout cooldowns all
apply to it as they are.

## Changes
- **`backend/services/aws_encoding_compute.py` (new):**
`AwsComputeAdapter` looks to the manager like the
`compute_v1.InstancesClient` methods it uses.
  - EC2 states map to GCE statuses.
  - EC2 capacity errors are added to `CAPACITY_ERROR_CODES`.
  - A missing instance raises `NotFound`.
- Controller credentials come from Secret Manager
`aws-encoding-controller-credentials`.
- **`EncodingWorkerManager`:**
- `_compute_for(vm)` sends AWS VMs to the adapter; the GCE paths are
untouched.
- A shared `parse_fallback_vms` passes `provider`/`instance_id`/`region`
through.
- `ranked_candidates()` is used by both the encode path and the warmups.
- `ensure_preferred_running()` behaves **exactly like
`ensure_primary_running` whenever the GCE primary ranks first**, which
is today's case.
  - An AWS control-plane fault fails over to the GCE candidates.
- **Ranking:** `SPEED_RANK["c7a"] = 90`, so AWS is the last resort by
default. Firestore `config/encoding-worker.speed_rank_overrides =
{"c7a": 5}` makes it preferred, with no deploy.
- **Isolation from GCE-only tooling:**
  - The override now records `active_override_provider`.
- The CI blue-green deploy, `deploy_promote` and the idle Cloud Function
all ignore the AWS worker; it hot-updates its wheel per job.
  - A GCE promotion clears any stale provider tag.
- **Worker:**
- Uses `gcloud storage` on AWS, because gsutil can't use WIF credential
files. GCE keeps gsutil.
  - `infrastructure/encoding-worker/aws/` adds:
    - `provision.sh`, as EC2 user-data;
    - `startup-aws.sh`, uploaded by CI;
- `idle-watchdog.py`, an idle self-stop that runs as its own systemd
timer.
- **Infra (GCP), already applied with `/gcp-admin`:**
- The WIF pool `aws-encoding`, plus an AWS provider that only admits the
`nomad-encoding-worker` role, federated to `encoding-worker@`.
  - The secret `aws-encoding-controller-credentials`.
  - Preview and apply: 4 creates, 288 unchanged.
- **Infra (AWS):** a new Pulumi project `infrastructure/aws` (stack
`nomadkaraoke/prod`):
- the c7a.8xlarge **persistent Spot** worker, which stops on
interruption or shutdown;
  - an Elastic IP;
  - a security group on 8080 (API key, same as the GCE firewall);
  - the worker role with SSM;
  - a tag-scoped `nomad-encoding-controller` user.
- Preview: 10 creates. **I apply it after merge**, because the box
provisions from the deployed wheel and boot scripts.
- `boto3` is now an explicit dependency.
- CI also runs the `encoding_worker_idle` function tests, in the job
renamed "Infra - Cloud Function tests".

## Rollout (after merge)
1. **Stage A:** `pulumi up` the AWS stack. Store the controller key in
the secret, then add the worker's entry to
`encoding-worker-fallback-vms`. It's inert unless both c4d and c2d stock
out.
2. **Stage B:** a supervised `speed_rank_overrides {"c7a": 5}` window.
Run real jobs, then compare timings and outputs, including a visual
check of the portrait video.
3. **Stage C:** keep it preferred. After a clean week, delete the c4d
pair.

Phase 2, running distribution (Dropbox/YouTube/Drive uploads) from AWS,
follows in a separate PR.

## Testing
- **New tests:**
- adapter: botocore `Stubber`, covering the state map, capacity and
NotFound handling;
  - manager routing and ranking with overrides;
  - `ensure_preferred_running` branches;
  - failover on an AWS control-plane fault;
  - provider tagging;
  - deploy_promote exclusions;
  - the idle function skipping AWS;
  - watchdog decisions;
  - the gcs CLI choice.
- **Updated** the candidate and warmup-route tests to the new seams.
- **Full backend + unit suites** have no new failures. The local
emulator and cross-suite failures are identical on main.
- **Local CodeRabbit:** 2 findings, both fixed (failover on AWS faults;
stale provider tag).

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 17:42:35 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 65940c6f57 fix(local-stack): decide partner calls go to local decide, never production (#1235)
The local stack's backend env had no `KARAOKE_DECIDE_API_URL`, so gen's
decide partner client (account deletion) used the production default and
sent `DELETE /api/partner/gen/users` to decide.nomadkaraoke.com.
Production answered 401, because the local secret differs, so nothing
was touched. It surfaced in nomad-mobile's E2E account-1 flow. The fix
defaults it to decide's local stack on :8030, and callers can still
override it. Script only, no runtime code.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Configuration**
* The local environment can now use a custom karaoke decision service
address, with a default address provided when none is configured.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 10:34:54 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 437b34ee64 fix(ci): wheel over PyPI's 100 MiB limit — drop unused RSC payloads + size guard (v0.287.2) (#1233)
## Summary
Every `Deploy - Publish to PyPI` since v0.279.4 (about 10-22) has failed
with `400 File too large. Limit for project 'karaoke-gen' is 100 MB`.
The wheel had grown to 110 MB (105.1 MiB), so PyPI is still on 0.279.4.
Backend deploys were not affected. I found this while watching deploys
during an alerts-triage session.

In the wheel, 66 MB of the bundled Next.js export is RSC `.txt` payloads
(about 3 MB per locale × 33). The only thing that uses the export is the
CLI local-review server
(`karaoke_gen/lyrics_transcriber/review/server.py`). It mounts only
`/_next` and `/static`, and every other route returns an HTML entry
point. Next's RSC fetch paths (`/{locale}/app/.../index.txt`) already
get HTML from those routes, so the `.txt` files are never served.

- `make build-frontend` deletes `*.txt` from the copied export. On the
real CI wheel from run 40826404322 this takes it from 105.1 MiB to 41.1
MiB.
- `scripts/verify_wheel_frontend.py`, which already runs in both
package-build jobs, now fails above 95 MiB. Growth will show up on the
PR instead of as a failed publish after merge.

## Testing
- New `tests/unit/test_verify_wheel_frontend.py`: 5 tests covering the
pass case, missing entrypoint, too few files, the size guard, and
headroom below 100 MiB.
- Re-zipped the real 0.286.0 CI wheel without the `.txt` files: 41.1
MiB, and the verifier passes. The verifier rejects the original wheel
(105.1 MiB) with the new error.
- CodeRabbit: 1 finding ("deleting RSC payloads can break client
navigation"). Skipped, because the local server never serves those files
(see above), so it can't change runtime behaviour.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVTiUPKxn3sQPyp3Teh

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 03:52:39 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 1878ad3c1d feat(retention): keep lead-vocal stems as word-timing training data (v0.287.1) (#1234)
## Summary
Storage retention purged `stems/lead_vocals.*` 30 days after completion.
As a result, 660 of 865 completed jobs have no audio left to pair with
their human-reviewed word timings (`lyrics/corrections_updated.json`,
which is kept).

The full-auto v2 timing work (an aligner fine-tuned on our data, plus a
per-word timing verifier as the auto-ship gate) needs exactly that pair.
Today's first bake-off had to make do with the 41 reviewed jobs that
still had stems, and data volume was the limiting factor. The cost is
about 15 MB per job; every other separated stem is still purged.

## Changes
- `storage_retention.classify_file`: `stems/lead_vocals.*` (exact
basename) is now KEEP with category `training_lead_vocals`.
- Tests: the stem moves from the purge list to the keep list; a
near-match (`lead_vocals_backup`) is still purged.
- Docs: ARCHITECTURE retention list.

## Testing
`backend/tests/test_storage_retention.py`: 133 passed. CodeRabbit ran
locally; its one finding (exact-match the basename) is fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVVVPsvPbXFhRYY5VSJ

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 07:25:11 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 bfcf1fd951 feat(auto-approval): hold confident-lyrics jobs out of review until stems land (v0.287.0) (#1232)
## Summary
In prod (2026-08-31 to 10-29), about a third of jobs were last scored
**before stems existed**. People opened the review link 15–40 s after
AWAITING_REVIEW. The audio worker's "second chance" re-score then hit
IN_REVIEW (`wrong_status`), so:
- 134 jobs never got a backing-vocals verdict;
- 7 jobs whose lyrics were in the auto tier went to a human anyway (e.g.
6c4be7d8, ed0b0c33, c4d8a256).

This is the first quick win from the full-auto v2 re-evaluation, which
targets >50% zero-touch jobs.

## Changes
- **`backend/services/auto_approval/review_gate.py` (new).**
- When the text verdict is AUTO and the only blocker is
`audio_incomplete`, the screens worker parks the job in
`GENERATING_SCREENS` with `state_data.auto_review_waiting_for_stems`. No
review link or notification goes out.
  - The hold is claimed atomically (`claim_job_state_flag`) by one of:
    - the audio worker once stems land (normal path);
    - the screens worker itself, if stems landed while it was parking;
    - `recover_stuck_jobs`, for holds older than 20 min.
- The claimant re-scores (`trigger="audio_hold"`), then either
auto-completes or opens review exactly like before, with server-side
pre-apply.
- If the post-claim re-score fails, the job falls back to human review.
If that also fails, the hold marker is restored so the sweep retries. A
job is never stranded.
- **Executor:** `hold_for_audio` is recorded in the verdict. It never
applies with `always_review`, `made_for_you`,
`backing_preference=review`, unconfident lyrics, or the kill switch on.
- **Kill switch:** `AUTO_APPROVAL_AUDIO_HOLD_ENABLED` (code default
true, because the executor runs in both the service and the
audio-separation job).
- Docs: LESSONS-LEARNED, README status row.

## Testing
- New: `test_review_gate.py` (11), `test_screens_worker_review_hold.py`
(2), executor hold cases (7), and recover-stuck-jobs hold sweep (3).
- Full backend suite locally: 6962 passed. The only failures are
`test_emulator_integration`, which needs the Firestore emulator and also
fails on pristine main.
- CodeRabbit CLI ran locally. Its one finding (stranded job on
post-claim failure) is fixed; the re-review is clean.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVVVPsvPbXFhRYY5VSJ

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 07:08:18 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 1f22091bf9 docs(CLAUDE.md): CI does not run Pulumi (#1231)
The Infrastructure rule said that changes in `infrastructure/` deploy
automatically on merge. That is wrong: ci.yml says Pulumi deploys are
done locally, and CI never runs `pulumi up`. In the 2026-10-29 cost
session, a merged infra fix (#1228) sat un-applied because of this. Docs
only, so no version bump.

No local CodeRabbit run, since this is a one-line docs change; leaving
bot review on.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Documentation**
* Clarified that CI validates infrastructure changes but does not apply
them; merged changes require a local `pulumi up` to take effect.
* Added guidance that Claude sessions need Andrew’s `/gcp-admin` grant
to apply infrastructure changes. Removed outdated claims that GCP
changes deploy automatically on merge or that CI deployments can be
interrupted on spot VMs.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 02:58:28 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 0d817509fa fix(api): event-loop stalls — visibility redistribution off-loop, route warmup, watchdog/Cloud Trace noise (0.286.2) (#1230)
## Summary

Fixes the 2026-10-29 #alerts event-loop stall batch (handoff
`docs/archive/2026-10-29-gen-event-loop-stalls-handoff.md` in the
workspace).

**1. Visibility change off the loop (39s stall, 13 requests incl.
/api/health)**
- `change_to_private` now runs claim, flag write, `redistribute_video`,
public-output delete and the final commit as **one** `asyncio.to_thread`
call (`_redistribute_and_commit_private`). `redistribute_video` runs on
that thread's own loop through the new `redistribute_video_off_loop`
(`asyncio.run(with_notification_drain(...))`).
- Is anything bound to the main loop? No. `redistribute_video` builds
its own `JobManager`/`StorageService`. The Dropbox, YouTube and GDrive
clients are sync. Job-log writes go through `log_write_offload`'s writer
thread. Notification tasks scheduled via `_schedule_notification` would
land on the private loop, so `with_notification_drain` drains them
before the loop closes, as the Cloud Run Job entry points do.
- Why one thread call and not just the redistribution (from CodeRabbit):
a thread can't be cancelled. So a cancelled request can no longer leave
a committed private archive with its public outputs still live, or a
claimed flag with no redistribution.
- Guard: `redistribute_video` is added to `BLOCKING_CALLS`, and
`visibility_change_service.change_to_private`/`change_to_public` are
added to `AWAITED_ROUTE_HELPERS`. I checked that putting back `await
redistribute_video(...)` fails the test.
- **Double Dropbox upload confirmed and fixed.** GCS `finals/` holds
every output twice: short canonical keys written by `_upload_results`
(`lossy_4k_mp4.mp4`, …) and the readable encoder names (verified on job
0f211e97: 17 objects / 525 MiB). Redistribution uploaded both. The
short-named `lossless_4k_mp4.mp4` also slipped past the `" (Final
Karaoke Lossless 4k).mp4"` Dropbox exclusion.
`_dedupe_redistribution_files` now keeps one copy per final, under its
readable name:
- a short name whose readable twin exists (twin found by suffix) is
dropped;
- a short name with no twin is renamed to the readable name (CDG/TXT
zips included).

**2. First-request stalls on new instances**
- Lifespan builds FastAPI's lazy route contexts
(`fastapi.routing.iter_route_contexts` +
`effective_low_priority_routes`) in a thread before readiness, with a
timeout and best-effort error handling. The results are cached by routes
version under a lock, so the OTel `_get_route_details` walk no longer
compiles route regexes on the loop. It is a no-op on FastAPI versions
without these internals. Note: the lock pins 0.128 but prod installs
unpinned (`pip install -e .`), so prod runs a newer FastAPI. I checked
against FastAPI 0.162: the first walk builds 304 contexts, the second
walk and the OTel lookup are cached (<1ms).
- `flacfetch.core.manager`, `flacfetch.downloaders.youtube` and `yt_dlp`
are added to `_LAZY_ROUTE_IMPORTS`.
- `search_audio` auto_download: `select_best` now goes through
`asyncio.to_thread`, with `select_best` added to `BLOCKING_CALLS`.

**3. Loop watchdog false positive**
- `_finish` logs WARNING, not ERROR, when all of these hold: no non-idle
sample was taken, a process freeze was observed (`frozen_s > 0`), and no
request was in flight when the stall began. Every "affected" request
therefore arrived while the instance was thawing. An unsampled stall
with no observed freeze, or with a request in flight at its start, stays
ERROR. The freeze requirement came from CodeRabbit.

**4. Cloud Trace exporter noise**
- `cloud_trace_export` ignore pattern for `Error while writing to Cloud
Trace`. CodeRabbit suggested matching only the transient gRPC detail. I
didn't: the error monitor only sees `jsonPayload.message` (the exception
is a separate field), and a persistent loss of telemetry isn't
#alerts-worthy anyway.

Smaller change: `change_to_private`'s Firestore `job_ref.update` writes
now also run off the loop.

## Testing
- New `backend/tests/test_redistribute_video.py`:
  - dedupe unit tests;
- an end-to-end `redistribute_video` run with mocks, checking the files
the Dropbox stage sees;
  - the off-loop runner uses its calling thread;
- `change_to_private` runs the redistribution on another thread and
loop;
  - a cancelled request still deletes the public outputs.
- `test_loop_watchdog.py`: frozen + unsampled + late arrival gives
WARNING; a request in flight at the start gives ERROR; no observed
freeze gives ERROR; a sampled stall with a late arrival gives ERROR.
- `test_main_startup_warmup.py`: lazy imports list; the route-context
walk; no-op without the API; failure is non-fatal; lifespan warms
off-loop before readiness.
- `test_known_issues.py`: the Cloud Trace pattern matches, and other
trace errors are not ignored.
- Local runs: `pytest backend/tests/ tests/unit/services/error_monitor
--ignore=backend/tests/emulator`: 7369 passed, 29 skipped. Not counted
in that total: `test_emulator_integration.py` fails 8/11 the same way on
the base commit, because a stray emulator from another session is on
:8080 locally. That's environmental, not caused by this PR.
- Local CodeRabbit: 3 cycles. Cycle 1: 3 findings (2 fixed, 1 skipped
with the reason above). Cycle 2: 1 finding, fixed. Cycle 3: 0 findings,
but CodeRabbit reported it reused cached results rather than a fresh
review.

## Not verified
- No prod run of a real public→private change yet. After deploy, the
Dropbox folder of the next one should contain each final exactly once,
and no `EVENT_LOOP_STALL` should appear during it.
- Route-context warmup was checked only against FastAPI 0.162 in a
scratch venv, not the exact prod version. Watch for the `Warmed N route
contexts` log line on the next cold start.
- Existing private Dropbox folders that already hold the duplicates are
not cleaned up by this PR.

Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1563719390849929330
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1563009782363721760
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1563039889354784882
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1563221363278348514
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1564308425033252895
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1563160849336762449

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVTiUPKxn3sQPyp3Teh

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 02:49:20 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 1abdd8c860 fix(cost): runner-manager only dispatches VMs for jobs it can serve; enforce AR cleanup (v0.286.1) (#1228)
## Summary
On 2026-10-27, **238 GCP `gha-general` VMs (~$67)** ran in one day. This
pushed October's GCP bill past the $300 budget.

**Cause:**
- nomad-mobile's E2E jobs run on the utility MacBook (`[self-hosted,
macOS, nomad-mbp]`).
- The runner manager acted on *any* job labelled `self-hosted`, so each
macOS job got a Linux VM that could never take it.
- While the job waited for an MBP lane, `redispatch_stalled_jobs`
launched another VM every 5 min for up to 60 min.
- Each of those VMs then idled until the 2 h reaper deleted it.

## Changes
- `ephemeral.can_serve(labels)`: GitHub assigns a job only to a runner
that advertises every label the job requests, so a job is ours only if
its labels ⊆ the resolved family's `runner_labels_for()`.
- The new check gates the webhook (`main.py`),
`list_queued_self_hosted_jobs` and `redispatch_stalled_jobs`.
- I checked every `runs-on: [self-hosted, …]` label set in the org's
workflows. All of them except the macOS lanes are subsets of our
families' labels, so no existing job loses its VM.
- CI: these function tests never ran in CI before. A new
`runner-manager-tests` job runs them on infra or ci.yml changes and is
enforced by CI Gate.
- Docs: `docs/EPHEMERAL-GHA-RUNNERS.md` § "Only jobs our runners can
serve".

## Testing
- 6 new tests: macOS jobs, unknown labels, our families, the redispatch
skip, queued-scan exclusion, and the webhook no-op. All 71
runner_manager tests pass locally.
- Local CodeRabbit review: 2 cycles, both CI-wiring findings fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 02:35:44 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 19f3c731fb test(e2e): sanitized page-activity log on tenant E2E failure (#1229)
## Summary
Tenant E2E #155 (singa, 2026-10-28) failed with the portal stuck on the
`/en/app/` loading spinner for 30s. The backend served tenant config in
168ms, 8 live re-runs were fine, and run #156 got past the same step.
The only evidence was a screenshot: CI has no Playwright traces, because
traces hold `X-Admin-Token` and the repo's artifacts are public.

This adds a public-safe stand-in,
`frontend/e2e/helpers/page-activity.ts`. It records:
- main-frame navigations;
- each request's method and origin/path (no query strings), with status
and duration or the failure reason;
- console errors/warnings and uncaught page errors, with URLs and
token-like values in their text redacted.

On failure it writes `page-activity.log` into test-results (the scrubber
still checks it) and prints the requests still in flight to the job log.
It is wired into `tenant-happy-path.spec.ts`. The approach is documented
in docs/TESTING.md, which also records #155 as a one-off flake.

## Testing
- New offline spec `e2e/self-healing/page-activity.spec.ts` (3 tests): a
fixture page with one hung, one failed and one token-bearing request,
plus console and page errors. The self-heal suite runs in CI's
frontend-e2e-smoke job. The whole suite passes locally (36 tests).
- `playwright test --list` on the production config compiles the tenant
spec.
- Local CodeRabbit review: 2 findings (redact console text; race on the
REQFAIL assertion), both fixed.
- Test-only change, so no version bump.

Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1564876714532798465

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVTiUPKxn3sQPyp3Teh

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 02:22:52 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 53bc8aa708 infra(cloudflare): remove stale www.app.karaokehunt.com CNAME (#1227)
Removes `www.app.karaokehunt.com` →
`karaokay-karaoke-song-choice-helper-eiqxrt.custom-domain-proxy.flutterflow.app`.
That hostname is the old KaraokeHunt FlutterFlow web app. It no longer
responds, and leaving a CNAME that points at an unclaimed third-party
proxy is a subdomain-takeover risk.

Part of the final decommission of GCP project `projectbread-karaokay`.
`karaokehunt.com`, `www.karaokehunt.com` and `app.karaokehunt.com` still
301-redirect to nomadkaraoke.com.

**Already applied:** `pulumi preview` showed exactly one delete, then
`pulumi up` reported 1 deleted, 135 unchanged. CI doesn't run Pulumi, so
this PR just brings `main` in line with what's live.

Tests: the `infrastructure/cloudflare` tests pass (11).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVU32bqJHRH3psJ5Udm

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Configuration**
  * Removed the DNS record for `www.app.karaokehunt.com`.
* The separate proxied IPv6 record for `www.karaokehunt.com` remains
configured.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 01:58:05 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 331d18ddf8 kjbox: one-tap Google/Apple sign-in for singers; sessions last 30 idle days / 365 max (v0.286.0) (#1226)
## Summary

Backend for one-tap sign-in in the kjbox singer page
(sing.nomadkaraoke.com). At the 2026-10-22 show, typing an email and a
6-digit code was the biggest friction point for singers making a karaoke
video on demand. Plan: kjbox
`docs/archive/2026-10-28-singer-make-it-one-screen-plan.md`.

- **`POST /api/kjbox/auth/google` and `/api/kjbox/auth/apple`** take
`{id_token, locale?, venue?}` and return the same body as
`/api/kjbox/auth/verify-code`, with no code needed.
- The token is checked by `login_code._verify_identity`, so the
audiences match the app's (and `fake:` tokens work only on the local
stack).
- Venue rules are the same as verify-code: no per-IP signup cap,
`KJBOX_SIGNUP_CAP_PER_24H` on new accounts, `signup_source="kjbox"` +
`signup_venue`, no `signup_ip`.
- Send-code's eligibility checks are applied explicitly. A disposable
domain gets 422. A blocked email or IP gets `422 sign_in_not_allowed`.
Apple relay addresses are allowed.
- Code sign-in and social sign-in now share one `_finish_sign_in`
helper.
- **`SESSION_EXPIRY_DAYS` (inactivity) 7 → 30, globally** (Andrew's
call). With weekly shows, every returning singer was signed out by the
next show. **`SESSION_ABSOLUTE_EXPIRY_DAYS` 30 → 365** (Andrew,
2026-10-28), so a weekly regular isn't signed out once a month.
- Version 0.286.0. API.md updated.

This stays inert until kjbox calls the endpoints. Before the singer page
can use them, Andrew needs to:
- add `https://sing.nomadkaraoke.com` as an Authorized JavaScript origin
on the "Nomad Karaoke Web" OAuth client;
- create an Apple web Services ID and add it to
`APPLE_SIGNIN_AUDIENCES`.

## Testing

`backend/tests/test_kjbox_partner_api.py` covers:
- a new singer via Google and via Apple;
- an Apple relay address, which is never treated as disposable;
- a code user who later signs in with Google;
- the signup cap applying to new accounts only;
- an invalid token returning 401;
- the partner secret being required;
- disposable and blocked email/IP refusals;
- 20 days idle still valid and 31 days idle expired; a new session
expires a year from sign-in; a weekly regular is still signed in two
months on.

259 auth/session/kjbox tests pass locally. Local CodeRabbit: 2 cycles.
The first found the missing eligibility checks (fixed) and the
absolute-expiry question (now 365 days). The second found nothing new.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 19:34:00 -04:00
Andrew Beveridge ☄️andClaude Sonnet 5.5 5c30fde9b4 fix: reference lyrics truncated at 3000 chars (blank lines counted) (v0.285.2) (#1225)
## Summary
- `BaseLyricsProvider.MAX_LYRICS_LENGTH` (3000) was measured on raw
text. Genius RapidAPI emits `\n\n` per line, so a normal ~4 min song
(job 258ba59b, 4,256 chars) was truncated after verse 2.
Lrclib/Musixmatch hit the same cap (2,975 byte files).
- Blank lines no longer count toward the cap; cap raised to 15000
(screenplays/articles are 50k+).

## Testing
- New regressions: double-spaced 70-line song and ~8k-char rap song keep
every line; screenplay tests still truncate.
- `tests/unit/test_base_lyrics_provider.py` 17 pass. 9 failures in
test_lyrics.py / test_agentic_gemini_provider.py are pre-existing on
main (env).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* Lyrics can now contain up to 15,000 characters before truncation, up
from 3,000.
* Blank lines no longer count toward the lyrics character limit, helping
preserve longer lyrics with extra spacing.

* **Documentation**
* Added guidance about the updated lyrics limit and a caution for
rerunning lyrics actions on in-progress jobs, where existing working
data may be replaced.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-27 15:17:05 -04:00
Andrew Beveridge ☄️andClaude Sonnet 5.5 cd1fa3feaf fix: provide-url YouTube downloads failing with 'No audio source selection found' (v0.285.1) (#1224)
Pasting a YouTube URL in the audio modal (job 258ba59b) failed:
provide-url set source_name/download_url but not source_id, and the
worker fallback required source_id. Now sets source_id and the worker
accepts download_url. The retry loop could never succeed before.

Tests: worker regression test for the provide-url job shape; provide-url
test asserts source_id. Emulator test_e2e_cli_backend needs Firestore
emulator, left to CI.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* YouTube audio jobs submitted with a pasted URL can now proceed when no
video ID is available.
* YouTube video IDs are saved with provided-URL jobs when available, and
stale search selections no longer override the pasted URL.

* **Chores**
  * Updated the application version to 0.285.1.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-26 20:07:38 -04:00
Andrew Beveridge ☄️andClaude Sonnet 5.5 3230bfbe0b feat(requests-board): real-user-only requests, 1 request + 1 vote/day, delete on last vote, confident unattended audio (v0.285.0) (#1223)
## Why
Investigating two bad community picks for one user (jobs 8695c32e,
94c83a7e) found:
- a YouTube-bot request was "claimed" by a vote that was later moved,
leaving a 0-vote request with a stale owner that was still pickable
(`vote_count >= 0`);
- the daily pick's `select_best` raised on remote (dict) results and
silently fell back to result #0 (logged as a WARNING on 7 days since 10
Oct).

## What
- **Real users only:** every request must belong to a real user email
(`require_real_email`, enforced in the service); the YouTube comment bot
no longer creates requests, it replies with how to request on the board.
Legacy placeholder rows are never listed/picked/credited.
- **Limits:** 1 request + 1 vote per day (submitting spends the vote →
`409 vote_already_used`; `429 submission_limit`). The cap uses a
create-only per-day marker, so deleting a request doesn't give the
submission back (purged on account deletion).
- **Last up-vote deletes the request** after confirmation (`409
would_delete_request` + `request_id` until `confirm_delete=true`),
freeing remaining voters' votes. Replaces the claim logic.
- **Picker:** needs `vote_count >= 1` and an owner who still up-votes it
and hasn't owned it already; deleting a community job reopens its
request (or deletes it if nobody backs it).
- **Audio:** unattended pick only on an exact artist+title match
(torrent ≥3 seeders, else Spotify; ambiguity → none). Otherwise the job
parks in `AWAITING_AUDIO_SELECTION` and the owner is emailed.
- **Email:** community-pick review email says the song came from the
requests board.
- **UI:** confirm before spending the daily vote / submitting, thank-you
panel with credit-pack CTA (`/app?buy=1`), form disabled once the vote
is spent, delete-request warning. Strings in all 33 frontend locales +
en/es/de backend.

## Testing
- Emulator tests for all service rules (real Firestore transactions),
unit tests for routes, audio pick, daily pick parking, notifications;
Jest tests for the board client.
- Local CodeRabbit review run; all 7 findings fixed.
- Note: a handful of existing job-creation emulator tests and
`test_internal_api` auth tests fail identically on clean `origin/main`
in my local env; CI is the judge.

Plan: docs/archive/2026-10-26-requests-board-overhaul-plan.md. Docs:
REQUESTS-BOARD.md, API.md, LESSONS-LEARNED.md.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-26 18:42:10 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 1715a07010 local stack: env overrides for parallel E2E lanes (defaults unchanged) (#1222)
Lets several local stacks run on one machine for nomad-mobile's parallel
Maestro lanes (the spare MacBook Pro runner): \`LOCAL_STACK_PROJECT\`
(per-lane Firestore project on the shared emulator, so global
fake-config does not leak between lanes) and \`LOCAL_STACK_STATE_DIR\`,
plus \`LOCAL_STACK_KEEP_EMULATORS=1\` for \`down\`. Every default is
unchanged, so existing local and CI runs behave exactly as before. Docs:
docs/TESTING.md § Local stack.

Context: nomadkaraoke/nomad-mobile#21 (E2E off Andrew's Mac).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Local development stacks can now use configurable state directories
and Firestore project IDs, making it easier to run isolated stacks side
by side.
* You can keep shared emulators running when stopping a stack, and
backend log locations reflect the selected state directory.
* **Documentation**
* Updated local setup guidance with recommendations for ports, projects,
state directories, and sharing emulators across stacks.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-26 21:55:52 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 fe89795aec fix: deliver worker-sent pushes/emails before Cloud Run Job exit; Web Push from workers (v0.284.1) (#1221)
Follow-up to #1220 (mobile Phase 6), from its whole-branch review.

## Problem
Cloud Run Job workers run inside `asyncio.run(...)`.
`JobManager._send_push_notification` (and the completion email / idle
reminder) used `loop.create_task(...)` and the worker returned
immediately, so `asyncio.run` cancelled the send. Prod logs: all 62
"Push notifications sent" lines in 14 days came from the API service,
none from worker jobs. Separately, worker jobs never had
`ENABLE_PUSH_NOTIFICATIONS` / VAPID keys, so even a delivered Web Push
would have been skipped.


## Customer impact check (emails)
Checked 2026-09-25 → 2026-10-25 (full log retention): **no customer
emails were lost.** 289 COMPLETE transitions → 11 deliberately
suppressed re-renders, 278 completion emails scheduled and 278 sent
(email_log has 297 completion records, all with Postmark MessageIDs;
Postmark outbound ≥ email_log). The email coroutines happen to do all
their I/O synchronously, so they finished in the task's first step
before `asyncio.run` cancelled; pushes hand off to a thread and were
cancelled. The drain removes that fragility for emails too. **Only
pushes were lost** (0 worker pushes in 14 days).

## Fix
- Notification tasks are tracked; every worker `main()` wraps its job in
`with_notification_drain` (bounded 10 s wait on our coroutines; provider
calls keep their own timeouts). The API server path stays
fire-and-forget.
- `failure_push_sent` is written only after the push was attempted;
in-flight id cleared via done-callback.
- Worker jobs (audio-download, audio-separation, bulk-search,
lyrics-transcription, video-encoding) get
`ENABLE_PUSH_NOTIFICATIONS=true` +
`VAPID_PUBLIC_KEY`/`VAPID_PRIVATE_KEY` (same secrets as the API).
CI-pinned jobs now `ignore_changes` their image (CI owns it), so a
targeted `pulumi up` changes env only.
- Expo messages carry `channelId: "jobs"` (the app's Android channel).

## Deploy
After merge, a targeted `pulumi up` from main on the 5 jobs. Preview
(read-only, from this branch): 5 updates, each only
+`ENABLE_PUSH_NOTIFICATIONS`/+`VAPID_*`, no removals, no image change,
283 unchanged.

**Behaviour change:** Web Push subscribers will start receiving the
worker-triggered pushes (video ready, needs review, failed) they were
always meant to get.

## Tests
6901 backend tests pass (emulator-integration file excluded: local env).
Tests reproduce the cancellation through each worker's real `main()`.
Per-task review + local CodeRabbit (no findings).

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01CZf1cM9vefnNw5djARRhyA

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 18:12:37 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 d76a136ee6 feat: mobile Phase 6 — RevenueCat IAP webhook, Expo push, self-serve account deletion (flagged off), App Review demo login (v0.284.0) (#1220)
Phase 6 (mobile release readiness) — gen side. Spec: nomad-mobile
`docs/superpowers/specs/2026-10-25-mobile-phase6-release-readiness-design.md`
(§4 payments, §5 push, §6 deletion, §9 demo account).

## What
- **`/me` app fields:** opaque `app_user_id` (UUID, used for
RevenueCat/Sentry — no email to third parties),
`capabilities.account_deletion`, `push_prefs`.
- **In-app purchases:** `POST /api/users/webhooks/revenuecat` (shared
Authorization header). Exactly-once grants keyed by
`{store}:{transaction_id}` inside the same Firestore transaction as the
balance change; refunds (REFUND / CANCELLATION with CUSTOMER_SUPPORT or
negative price) debit the granted account, refund tombstones block late
grants; other cancellations ignored. `store_payments` for accounting +
revenue sums; referral earnings on **net** proceeds; sandbox grants but
excluded from revenue/referrals. Credit packages gain `store_product_id`
+ server-driven `web_link` (iOS US only for now).
- **Push:** `POST/DELETE /api/push/expo-token` (token moves between
users atomically), `PATCH /api/users/me/push-prefs` (ready / review /
problems, gates Expo **and** Web Push), Expo sender (batched,
DeviceNotRegistered cleanup), new FAILED / AWAITING_AUDIO_SELECTION
triggers.
- **Account deletion** (`ACCOUNT_DELETION_ENABLED`, **off in prod**
until Andrew's manual pass): `GET /api/users/me/deletion-preview`,
`DELETE /api/users/me`. Refuses admin / unpaid referral balance / Stripe
Connect / made-for-you in progress (409). Cancels active jobs, deletes
decide data (partner call, twice), RevenueCat subscriber, revokes Sign
in with Apple (when a code is given), queues kjbox
(`/api/kjbox/account-deletions` pull + ack), deletes **inputs only**
(uploads, input/, audio_edit/, style/, staged instrumentals) — **never
published outputs** — strips personal fields, anonymises
jobs/referrals/song requests/show-credit records, keeps payment records.
Every step idempotent; failures → 503 retry; sessions revoked last.
- **Web:** `/account/delete` page (Google Play web deletion path),
hidden while the flag is off.
- **App Review demo login:** fixed code for one configured email — inert
until `APP_REVIEW_DEMO_EMAIL` is set (not set in prod).
- **CI:** mounts `REVENUECAT_WEBHOOK_AUTH`, `REVENUECAT_SECRET_API_KEY`,
`APP_REVIEW_DEMO_CODE` (secrets exist; runtime SA already has accessor).

## Tests
6838 backend unit tests pass (emulator suite needs emulators;
iap/deletion emulator tests pass locally), frontend jest 1785/1785.
Per-task reviews + local CodeRabbit review (5 minor findings fixed).

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01CZf1cM9vefnNw5djARRhyA

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 17:31:01 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 9014754bda feat(error-monitor): watch voice pool jobs (v0.283.2) (#1219)
## Why
The Phase 3d voice pool runs for ~1 week unattended. Failures must reach
Discord #alerts (actionable-signal rule), and the error monitor only
watches Cloud Run jobs listed in `MONITORED_CLOUD_RUN_JOBS`.

## Changes
Added to `MONITORED_CLOUD_RUN_JOBS`
(`backend/services/error_monitor/config.py`):
- `voice-pool-select`, `voice-pool-fetch`, `voice-pool-analyse`
(karaoke-decide, us-central1)
- `vocal-range-gen` (karaoke-decide, us-central1; logs one ERROR from
task 0 only when every attempted job fails)
- `voice-pool-separate-job` (gen GPU job, us-east4)

The monitor's log query filters on `resource.type="cloud_run_job"` +
`resource.labels.job_name` only, with no region/location filter, so
us-east4 is covered the same way as `audio-separation-job`. No docs list
the monitored jobs, so none needed updating.

## Testing
- Updated the pinned count/contents test
(`tests/unit/services/error_monitor/test_config.py`)
- Added a test that the built log filter includes the new jobs and has
no location scope
- error-monitor tests: 414 passed
- Local CodeRabbit review: no findings
- Version 0.283.1 -> 0.283.2

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_0188txBF47GpTo7hym1FoifX

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 16:37:49 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 eab20db79c fix(bulk): accent/script/alias-aware artist search + no-results state (#1218)
## Summary
Bulk → By album returned **no artists** for "Maximo Park" and showed a
blank list with no explanation.

**Root cause:** artist search matched decide's
`mb_artists_normalized.name_normalized`, which maps every char outside
`[a-z0-9 ]` to a space with no accent folding. "Maxïmo Park" is stored
as `max mo park`, so `maximo park` can never match. ~179k artists (6%,
Cyrillic/CJK/etc.) normalize to `""` and were unfindable.

## Changes
- `musicbrainz_service.search_artists`: one SQL `search_key()` (NFKD,
drop combining marks, fold ø/ł/đ/ß/æ/œ/þ, keep letters/digits of any
script) applied in BigQuery to the artist name, every
`musicbrainz.artist_alias`, and the query. Ranking is unchanged in
spirit (exact → prefix → substring, name before alias, then popularity).
New `alias` field when only an alias matched.
- Verified live: `maximo park`→Maxïmo Park, `bjork`→Björk,
`beyonce`→Beyoncé, `motorhead`→Motörhead, `royksopp`→Röyksopp, `hikaru
utada`→宇多田ヒカル, `mumiy troll`→Мумий Тролль, `宇多田`, `Мумий`.
- Cost: ~0.4 GB scanned/search (~$0.002), ~2 s, 10-min cache; well under
the 2 GiB byte cap.
- UI: no-results card ("No artists match "X"" + hint + **Add songs by
name** → switches to By text); alias rows show "Also known as …". 33
locales.
- `isArtistMatch` (frontend) is now Unicode-aware, matching the
backend's `_norm_artist` (follow-up from #1214: CJK/Cyrillic names never
matched).

## Testing
- Backend unit: 6681 passed. New `search_key` + `search_artists` tests.
- Frontend jest: 1776 passed. New no-results/loading/error/alias/switch
tests + isArtistMatch parity tests.
- Local CodeRabbit CLI: no findings.

## Follow-up (not in this PR)
decide `/api/catalog/artists` has the same flaw ("bjork" doesn't find
Björk); a proper fix is a folded column in decide's
`mb_artists_normalized` ETL.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_018teGmBJj3EVmCfpfMFG4WU

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 22:44:58 +00:00
Andrew Beveridge ☄️andClaude Sonnet 5.5 286e604b75 feat(worker): voice-pool separate-only GPU job (v0.283.1) (#1217)
## Summary
- Adds `voice-pool-separate-job`, a Cloud Run GPU (L4) job for
karaoke-decide's voice session. decide writes a JSONL manifest
(`voice-pool/manifests/<run>.jsonl`) and starts the job with a per-run
shard; each task separates its share of songs with the
`instrumental_clean` Separator and writes
`voice-pool/stems/<id>/vocals.flac` (skipped if already present).
Results are written back to the manifest area every 10 songs.
- No Firestore or gen job state. Model-load failure or an
all-songs-failed task exits non-zero with a single ERROR; partial
failures exit 0.
- Task timeout is 3600s (Cloud Run's GPU maximum, found by the local
CodeRabbit review), so decide must size the shard count to keep each
task under 1h. Re-runs skip finished songs.

## Changes
- `backend/workers/voice_pool_separate_worker.py` +
`backend/tests/test_voice_pool_separate_worker.py` (9 tests, 91%
coverage of the worker)
- `infrastructure/modules/cloud_run.py`, `infrastructure/__main__.py`:
Pulumi job definition
- `.github/workflows/ci.yml`: CI guard step (tolerates a missing job)
- `docs/ARCHITECTURE.md`: paragraph on the job
- Version 0.283.0 -> 0.283.1

## Deploy notes
**Andrew: run `pulumi up` in gen infrastructure/ BEFORE merging (creates
voice-pool-separate-job; the CI guard tolerates a missing job).**
decide's Pulumi (later PR) will create the voice-pool SA and grant it
`roles/run.jobsExecutorWithOverrides` on this job.

## Testing
- [x] `pytest backend/tests/ --ignore=backend/tests/emulator`: 6691
passed, 40 skipped
- [x] Local CodeRabbit review completed (2 cycles): the critical timeout
finding fixed; the remaining minor (results flush inside `finally`) is a
known deferred item

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_0188txBF47GpTo7hym1FoifX

---------

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-22 18:42:35 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 c4c4f9265c feat(bulk): album batches search the album once; one flacfetch search in flight per batch (#1214)
## Why
A real Bulk "By album" batch (Chase & Status, *No More Idols*, 12
tracks):
- It took about 5 minutes to reach audio selection.
- 4 of the 12 searches timed out.
- All 12 were parked for manual choice, each showing the same album
torrents.

Causes:
- The album was dropped at submit, so each track was searched
separately.
- 4 parallel searches ran against flacfetch's single search worker, and
the queued ones ran out the 95s budget.
- The auto-pick bar was 50 seeders (lowered to 10 in #1211).
- A title-only file search misses the album for common titles ("Time").

## What
- **Submit carries the album:** `POST /api/bulk/submit` takes `album`
{artist, title, release_group_mbid, release_mbid} plus each song's
`position`. These are stamped as `state_data.bulk_album` /
`bulk_track_position`. The frontend album mode sends them, using the
release-group title.
- **One album search per album:** the worker makes one flacfetch `POST
/search/album` call (nomadkaraoke/flacfetch#56).
  - Each track's file is resolved inside the album's torrents.
- The whole album comes from **one torrent**: the first BEST CHOICE
(lossless, non-vinyl, 10+ seeders).
- A track is auto-selected when that torrent's file names it and the
artist matches.
- A track the batch torrent doesn't cover gets the normal per-track pick
among the album torrents that do.
  - Parked jobs show the album torrents in the selection modal.
- **Fallbacks are one search at a time** (`SEARCH_CONCURRENCY = 1`):
uncovered tracks, text batches, and failed or unavailable album searches
(404 from an older flacfetch). A batch search deadline (3000s) parks any
leftovers before the 3600s job timeout could strand them.
- **Auto-pick checks the artist** on the backend too: a Unicode-aware
port of `isArtistMatch`, used by bulk and KaraokeHunt. A 10+-seeder
cover or tribute torrent is no longer auto-picked.
- Fake album search for the local stack; docs in API.md and
LESSONS-LEARNED.

The money path is unchanged. Album results have no duration, so nothing
is charged at select time. The real duration charge or refund still
happens after download (`reconcile_and_maybe_pause`).

## Testing
- Backend: 6677 passed. The only failures are the 8
`test_emulator_integration` tests, which hit a stale local Firestore
emulator from another session; they passed in CI on #1211.
- Frontend: Jest 1765/1765.
- New tests cover:
- album worker path: one search, all auto-selected, uncovered →
fallback, search failure → fallback, low seeders → park, auto disabled
  - `process_batch` routing, with at most one search in flight
  - the deadline
- album helpers, artist match (incl. CJK and &), client `search_album`
(incl. 404 and 502), submit stamping, the fake album response, and the
frontend submit payload
- Code review: CodeRabbit CLI unavailable (invalid token / org
mismatch), so a local Superpowers review ran instead. Its fixes are in
the second commit.

## Rollout
flacfetch v0.34.0 (#56) deploys to flacup first. Then: one album search,
12 auto-selected jobs, one torrent; this is verified in prod after
merge.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_019MixBuvzxB1c5AYc3mbDyW

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Album-mode bulk submissions now include album details and track
positions, enabling one album-wide search for matching tracks.
* Search results are checked against both track title and artist before
automatic selection, with tolerance for minor name variations.
* **Improvements**
* Tracks not covered by an album result, or affected by an unavailable
or failed album search, fall back to individual searches.
* Searches still awaiting selection after the batch deadline are parked
for manual review.
* **Documentation**
* Updated bulk submission guidance to describe album searches and
fallback behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 17:06:57 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 76069573eb fix(bulk): deleted/missing batch no longer dead-ends Bulk mode (#1216)
## Bug
Bulk mode in prod showed **"Couldn't load batch progress."** and nothing
else. The page remembers the last batch ID in localStorage
(`nomad-bulk-last-batch`) so a returning user can finish its review
queue. That batch's jobs had been deleted, so `GET
/api/bulk/4fbbf0165416` returned 404, and the error state had no way
back to the form.

## Fix
- A **404** forgets the batch (the same path as "Start another batch")
and shows the form.
- Other errors keep the message and add the **Start another batch**
button.
- A transient error no longer replaces an already-loaded batch, and
polling continues.
- The parent's callback is read through a ref, so the 5s polling effect
isn't restarted on every render.

No new strings.

## Testing
- New `frontend/__tests__/bulk-batch-progress.test.tsx` covers the 404 →
back to form path, and other errors → message plus button.
- Full Jest suite: 1769/1769.

Workaround until deployed:
`localStorage.removeItem('nomad-bulk-last-batch')`, then reload.

Code review: local self-review only for this 2-file fix; CodeRabbit CLI
is unavailable (org mismatch), so the GitHub bot will review it.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_019MixBuvzxB1c5AYc3mbDyW

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 16:03:49 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 25888ca003 fix: Stripe made-for-you emails + bulk_submit job creation off the event loop (v0.282.2) (#1212)
## Summary
Two event-loop stalls from the 2026-10-22 #alerts triage. Each froze
every request on the instance.

- **22s stall, 11 requests (15:57 UTC).** `stripe_webhook` (async) →
`_handle_made_for_you_order` sent the customer order-confirmation email
through sync Postmark, with retries and SMTP fallback, on the loop. It
was the NOMAD-1800 order. The confirmation, the admin notification and
`_prepare_theme_for_job` (GCS) now run via `asyncio.to_thread`.
- **26s stall, 9 requests (15:30 UTC).** `bulk_submit` (async) ran the
credit check, default-theme lookup and one sync `create_job` +
`update_job` per song on the loop (an album batch). That block is now a
sync helper run via `asyncio.to_thread`. Behaviour is unchanged: the
same `HTTPException`s propagate from the thread.

## Guard (`test_no_blocking_async_routes.py`)
- `BLOCKING_CALLS` now includes `send_made_for_you_order_confirmation`
and `send_made_for_you_admin_notification`.
- New `test_no_known_blocking_calls_in_awaited_route_helpers` also scans
`_handle_made_for_you_order`. It is an awaited helper, not a decorated
route, so the route-only scan missed it, including its
`_prepare_theme_for_job`, which was already on the denylist.
- New `test_bulk_submit_creates_jobs_off_the_loop`.
- Both new tests fail on `origin/main` (checked against the old files)
and pass here.

Out of scope: sync `create_job`/`update_job` still runs directly in
about 20 other async routes (audio_search, file_upload, jobs.retry_job,
…). Each call is usually fast. It's worth a broader sweep if the
watchdog keeps catching them.

## Testing
- `make test` backend: 6607 passed, 8 failed. All 8 are in
`test_emulator_integration.py`, because another session's Firestore
emulator holds :8080 locally (same as noted on #1210). CI's emulator job
is authoritative.
- Targeted: bulk submit, made-for-you, Stripe webhook and the blocking
guard: 114 passed.
- The local CodeRabbit CLI failed with "Invalid token", so the bot
review stays on.

Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562858789785309195
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562858791446388862

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_016F41fLoBotUw2wUjGCuAok

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Improvements**
* Bulk song submissions and made-for-you order processing are less
likely to delay other activity while work is underway.
* Existing credit checks and handling of jobs created before credits run
out remain unchanged.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 19:55:36 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 bc8b033cb5 chore(cloudflare): app-link fingerprint for kjbox.cc (applied) (#1215)
Sets Pulumi config `kjboxcc_android_sha256_fingerprints` to the Android
debug keystore SHA-256 used by the Nomad Karaoke app builds (release key
gets added in Phase 6). Already applied (`pulumi up`: worker content
only). Google Digital Asset Links check: linked=true for kjbox.cc,
www.kjbox.cc and sing.nomadkaraoke.com.

Config-only change; CodeRabbit not needed.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_013Bkasn6trBadg1Zb39vSVR

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 15:31:09 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 f6d846234d fix: alert noise + loop stalls + kn-data-sync OOM from #alerts triage (v0.282.1) (#1210)
## Summary
Fixes for the open and noise groups from the 2026-10-22 #alerts triage.

**Noise that should not page:**
- `stripe_service.py`: "Invalid webhook signature" now logs WARNING. All
42 hits on 10-21 came from `facebookexternalhit` via Cloudflare with a
malformed header. Every real Stripe delivery since 10-14 got a 200.
(#alerts 1562375740685815953)
- `error_monitor/known_issues.py`: `startup_probe` also matches Cloud
Run's "Default STARTUP **TCP** probe failed" wording.
(1562224513276706933)
- `frontend/lib/crash-reporter.ts`: a `removeChild`/`insertBefore`
NotFoundError is benign **only while the page is machine-translated**,
i.e. `<html>` has `translated-ltr`/`translated-rtl`. In the 10-21 crash,
Chrome had translated `/en/app` into Spanish ("Crea un video karaoke" is
not our string), and the customer's job completed and was downloaded.
(1562541569238638725)

**Event-loop stalls** (continues the WIP freeze item in BACKLOG):
- `review.py` audio-editor routes: `apply_edit`, `get_metadata_from_gcs`
and the GCS `download_file`/`upload_file` calls now go through
`asyncio.to_thread`. They were stalling the loop about 10s per edit (job
330d796c, 10-19). (1561861805695967363)
- `internal.py`: `process_conversion_emails` is imported at module top.
The lazy import inside the hourly cron's `async def` wrote a `.pyc` on
the loop, stalling it up to 15.3s, 4× on 10-20/21. (1562224514933334047)
- `backend/Dockerfile`: `python -m compileall` over `/app/backend` and
`/app/karaoke_gen`, so no lazy import compiles at runtime.
- `test_no_blocking_async_routes.py`: `BLOCKING_CALLS` now includes
`apply_edit`, `get_metadata_from_gcs`, `upload_file` and
`download_file`. Two more routes in `review.py` were caught and wrapped:
the preview ASS upload and the join-audio upload.

**kn-data-sync** (`infrastructure/modules/kn_data_sync.py`):
- Memory goes from 512M to 1Gi. The 10-22 full sync OOMed at 489 MiB on
a retry. 1Gi also gives gen2 more CPU, and the cost is about $0 within
the free tier.
- `attempt_deadline="320s"` on both schedulers. The default 180s gave up
on a run that took 184.8s and succeeded, then retried it, which meant 3
full KN pulls that day. (1562752965629190217, 1562752969395535933)

## Testing
- `make test`: 6603 passed. 8 failed in `test_emulator_integration.py`,
the known local-only failure (another session's stack holds :8080); they
fail the same way on unmodified main, and CI's emulator job is
authoritative.
- Targeted runs: error monitor, audio-edit routes, Stripe/webhook,
conversion emails and the blocking guard all pass (710 tests). Frontend
crash-reporter jest: 17 passed, with new translated / not-translated
cases.
- The local CodeRabbit CLI failed with "Invalid token", so the bot
review is left on for this PR.

## Deploy note
The `infrastructure/` change (kn-data-sync) needs `pulumi up` before
merge, per the repo rule. That needs Andrew's `/gcp-admin` grant.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_015AvGQEptp3Pg9RHhKGZ5KB

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved responsiveness during audio editing and review submissions,
reducing the chance that file and audio processing delays other
requests.
* Reduced false crash reports on pages affected by machine translation.
  * Improved recognition of startup probe failures in error monitoring.
  * Clarified logging for malformed payment webhook signatures.
* **Reliability**
* Increased resources and allowed more time for scheduled data sync
jobs.
  * Reduced delays to API requests during background processing.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

## Alerts
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562375740685815953
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562224513276706933
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562541569238638725
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1561861805695967363
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562224514933334047
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562752965629190217
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562752969395535933

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 19:26:43 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 e4dad6e3c5 fix(delivery): hold completion email when Dropbox delivery failed (NOMAD-1800) (#1213)
## Summary

Job `dd892383` (NOMAD-1800) finished while the business Dropbox was at
its 6 TiB quota. Both Dropbox upload passes failed with
`insufficient_space`, but the job still went COMPLETE and emailed the
customer a delivery email whose link was the literal `[Dropbox URL not
available]`. The only #alerts signal was a raw traceback.

Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562874010939949110

## Changes

- **New `backend/services/delivery_guard.py`**
- `completion_email_hold_reason(job)`: if the job's *effective*
distribution settings deliver to Dropbox (`dropbox_path` and
`brand_prefix`, the same condition the workers use, so
private/NonPublished jobs count too) and `state_data.dropbox_link` is
missing, the email is **held**.
- `hold_completion_email(...)`: records
`state_data.completion_email_held = {reason_code, reason, recipient,
at}`, logs a WARNING, and sends **one actionable #alerts message per
job** (`send_throttled_ops_alert`, keyed per job). The message covers
the job, brand code, the masked recipient whose email was held, the
reason (`insufficient_space` is classified as "Dropbox is FULL"), the
fix steps and the runbook. The Pushbullet push is unchanged.
- `check_dropbox_space_usage()`: calls `users_get_space_usage()`, using
team allocation for the business account. It posts one #alerts message
when usage reaches 95% and records its state in Firestore
(`ops_failure_alerts/dropbox-space-usage`), so it stays quiet while
usage stays high and re-arms once usage drops below 95%. It runs on the
existing hourly `process-stale-reviews` cron, so no infra change is
needed.
- **`JobManager._schedule_completion_email`**: checks the hold before
sending. Made-for-you ownership still transfers to the customer.
- **Admin `POST /api/admin/jobs/{id}/send-completion-email`**: returns
409 when the link is missing, unless `allow_missing_dropbox_link: true`
is passed. A successful send clears `completion_email_held`.
- **`TemplateService.render_job_completion`**: no more `[Dropbox URL not
available]` / `[YouTube URL not available]` placeholders. When a link is
missing, that whole section is dropped (English default and GCS wording,
plus the localized template). Jobs with no Dropbox delivery at all
(tenant, no Dropbox configured) still get their email, just without a
Dropbox paragraph. The YouTube-queued and private paths are unchanged.
- Dropbox upload failure logs (orchestrator and native pass) go from
ERROR to WARNING, so one root cause raises one alert (the held-delivery
alert) instead of an error-monitor traceback.
- Docs: `TROUBLESHOOTING.md` § "Held completion email (Dropbox delivery
failed)" with the manual recovery steps, and `API.md` for the new 409
and override.
- Version 0.282.0.

Not done: item 5 of the handoff (an admin "redistribute Dropbox only"
action). The existing admin re-render also re-uploads YouTube, so it
isn't a clean fix for public jobs. The manual recovery steps are in
TROUBLESHOOTING.md instead.

## Testing

- New `backend/tests/test_delivery_guard.py` (31 tests): failure
classification, hold/no-hold matrix (no link, link present, no Dropbox,
tenant, private defaults, settings error), held marker and alert
content, quota threshold, alert-once and re-arm, failed send does not
latch, API error never raises.
- `test_made_for_you.py`: email held when the link is missing (ownership
still transfers, marker recorded, alert sent); email sent when the link
is present; email sent for a job with no Dropbox delivery.
- `test_admin_email_endpoints.py`: 409 without the link; override sends
and clears the marker; fixed link sends and clears the marker; job with
no Dropbox delivery sends normally.
- `test_template_service.py`: missing links never render placeholders
(en, de, unknown GCS template).
- Backend unit suite (`backend/tests`, excluding emulator tests): 6637
passed, 7 failed, 25 skipped. All 7 failures are
`test_internal_api.py::*requires_auth` / `*requires_admin`, which fail
the same way on unmodified `origin/main` locally (they get 200 instead
of 401, apparently from the local environment), and this PR doesn't
touch them.
- The local CodeRabbit CLI failed ("Invalid token"), so this PR is left
open for the bot review.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_016F41fLoBotUw2wUjGCuAok

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 17:53:08 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 73fa5d7c01 feat(bulk): album cover thumbnails + lossless BEST CHOICE at 10+ seeders (was 50) (#1211)
## Summary
- **Album cover thumbnails** in Bulk → By album. Album rows and the
tracklist header show the release group's front cover from the Cover Art
Archive (`coverartarchive.org/release-group/{mbid}/front-250`).
- Loaded lazily in the browser, never through our backend or the
MusicBrainz API.
- The disc icon stays as a placeholder, and remains when the album has
no cover (404).
- **Lossless torrents with 10+ seeders now count as BEST CHOICE** (tier
1 / auto-pick); the bar was 50.
- Prompted by a real batch (Chase & Status, *No More Idols*): all 12
jobs were parked for manual selection even though the right lossless
album torrent (31 / 15 seeders, filename matching) was the top result.
- Shared `BEST_CHOICE_MIN_SEEDERS` constant in backend
`pick_auto_selection` and frontend `categorizeResult`, plus a test
asserting the two match. The review-fixtures e2e spec uses the constant
too.
  - Availability labels (High ≥50 / Medium ≥10 / Low) are unchanged.
- **Also affects** unattended KaraokeHunt conversions, which call
`pick_auto_selection`. The title/filename check still guards them.
- Fake search `recommended` (tier 2) shape drops to 5 seeders; fixture
regenerated.

Follow-up (separate PRs): release-aware bulk album search/download. One
album-level flacfetch search instead of N per-track searches;
flacfetch's search executor is single-threaded, which is why 4/12
searches in that batch timed out at 95s.

## Testing
- Jest: all of `lib/__tests__` + bulk tests pass (355), including 3 new
cover-art tests and threshold boundary tests (9 vs 10).
- Backend: 6604 passed. The 8 failures are in
`test_emulator_integration.py`, which only runs when a local Firestore
emulator is up; it hit a stale emulator from another session. That
covers job creation, which this PR doesn't touch.
- Checked a real CAA cover (200 via archive.org redirect, ~2s cold) and
a missing cover (404).
- Code review: CodeRabbit CLI unavailable (invalid token / org
mismatch), so a local Superpowers review ran instead. Its fixes are in
the last commit.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_019MixBuvzxB1c5AYc3mbDyW

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 12:41:59 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 5f72e618cc feat(cloudflare): serve app-link files on kjbox.cc/.well-known (#1209)
## Summary
Mobile Phase 5 "Tonight": the printed venue QR
(`HTTP://KJBOX.CC/<code>`) should open the Nomad Karaoke app when
installed (Android App Links now; iOS universal links once the
team-signed build exists in Phase 6).
- New Pulumi-managed Worker `kjbox-cc-well-known` on
`kjbox.cc/.well-known/*` and `www.kjbox.cc/.well-known/*` serving
`assetlinks.json` (fingerprints from config
`kjboxcc_android_sha256_fingerprints`, `[]` until set) and
`apple-app-site-association` (team 55B76HN73D,
`com.nomadkaraoke.mobile`).
- The kjbox.cc `/<code>` redirect rule now skips `/.well-known/`;
everything else unchanged.

**Already applied** (`pulumi up`, +3 created, ~1 ruleset). Verified:
both hosts' `assetlinks.json` 200 JSON with no redirect, AASA 200
`application/json`, `http://kjbox.cc/2121` still 302 →
`sing.nomadkaraoke.com/?t=2121`, `/` still 302.

## Testing
`infrastructure/cloudflare/tests`: 11 passed (Worker script exercised
under node). CodeRabbit CLI: no findings.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_013Bkasn6trBadg1Zb39vSVR

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 04:12:39 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 032ac11a45 fix(youtube): comment bot survives long runs (Broken pipe on first prod run) (v0.280.3) (#1208)
## Summary

The first prod run of the YouTube comment bot (#1205/#1207) posted 2
correct replies and then died with `BrokenPipeError`.

**Cause:** `commentThreads.list` pages were fetched lazily between slow
per-thread work. Background tasks on Cloud Run are CPU-throttled, so
Gemini classification of a 100-thread page took about 10 minutes. The
shared httplib2 connection idled out and the next page fetch broke.

**Fix:**
- **Reads:** all comment pages are fetched up front, and reads retry
(`num_retries=3`, which reconnects). If a page still fails, the threads
already fetched are processed, the run is marked incomplete (high-water
not advanced), and #alerts gets a throttled message.
- **Replies:** each post uses a freshly built client, built **before**
the comment is claimed, so a client failure leaves it unclaimed. Posts
are never retried, because a lost response could mean a double reply.

Re-running is safe: the 2 posted replies have action docs, and the
failed run saved no state, so the one-time backfill start still applies.

## Testing

- `backend/tests/test_youtube_comment_bot.py`: reads use `num_retries`,
posts don't.
- `backend/tests/test_youtube_comment_bot_worker.py`: a later page
failure still processes the fetched threads and alerts; a posting-client
failure leaves the comment unclaimed.
- 55 bot tests pass. Local CodeRabbit: 2 cycles, all findings fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_016E3ibN2EcKxnGQvEf6T4V8

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 03:41:20 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 055a5d3f73 feat(youtube): enable comment auto-reply in prod + one-time backfill (v0.280.2) (#1207)
## Summary

Switches on the YouTube comment auto-reply bot from #1205 in prod
(Andrew authorised go-live after the 6-month replay sign-off):

- `YOUTUBE_COMMENT_AUTOREPLY_ENABLED=true`
- `YOUTUBE_COMMENT_AUTOREPLY_START=2026-04-22T00:00:00Z`. This is used
on the first run only, as a one-time backfill (Andrew: "yes please, will
be a good test of the system working once in prod"). The replay predicts
3 replies: Fightmaster "Nobody's Baby", Bear Ghost "Tiny Hitman", Playa
Fly "N God We Trust". Threads already answered by hand are skipped.

Already verified in prod before this PR:
- The endpoint returns `disabled` while the flag is off.
- A `?dry_run=true` run loads secret `youtube-comment-bot-credentials`,
builds the channel index and reads comments with no errors.
- The Cloud Scheduler job `youtube-comment-autoreply-hourly` and the
secret were applied through Pulumi in #1205.

## Testing

Config-only change; the code and tests are in #1205.


🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_016E3ibN2EcKxnGQvEf6T4V8

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 02:41:15 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 53bed95e76 feat(local-stack): kjbox partner secret + fake quick version (v0.280.1) (#1206)
## Summary
Local-stack only (no prod impact). For mobile Phase 5 "Tonight": the
kjbox local stack points at gen's local stack for make-it.
- `scripts/local-stack.sh`: `KJBOX_PARTNER_SECRET` and
`DECIDE_GEN_PARTNER_SECRET` default to `local-stack-secret`, external
values win.
- Fake workers emit a quick version (`state_data.quick_version` ready +
`file_urls.quick.video_mp4`) for kjbox-sourced jobs, using prod's
`should_build_quick_version` gate and quick-version helpers.
`FAKE_WORKERS` is still refused in production.
- docs/TESTING.md § Local stack updated.

## Testing
`test_kjbox_partner_api.py`, `test_fake_workers.py`,
`test_kjbox_resolve_partner.py`, `test_fake_match_judge.py`: all pass.
Reviewed by a task reviewer (1 fix round).

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_013Bkasn6trBadg1Zb39vSVR

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 02:06:01 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 6c53f0471a feat(youtube): auto-reply to song requests in YouTube comments with requests-board links (v0.280.0) (#1205)
## Summary

Backlog Frog #6 (YouTube channel → gen funnel). Andrew's spec: *"we
could probably also solve that by building some automation to detect
youtube comments asking for songs and auto-reply with a link to the
requests board"*.

An hourly bot reads new top-level comments on the Nomad Karaoke channel
and, when someone asks for a song:
- **Song already on the channel:** replies with that video's link.
- **Otherwise:** adds the song to the requests board (or finds the
existing request) and replies with a link straight to it, e.g. `Added
"Tiny Hitman" by Bear Ghost to our requests board 🥳 upvote it here and
we'll email you when it's made: <link>`.
- **Vague asks** ("more Johnnie please") get a generic board link.
- **Non-English comments** get a translated reply, which falls back to
English if a URL doesn't survive translation.

The reply promotes only the board, not gen (Andrew's call). The goal is
turning anonymous YouTubers into board users with an email address.

**Andrew signed this off on a 6-month local replay**
(`scripts/youtube-comments/replay.py`, read-only):
- 265 threads, 33 song requests.
- It flagged all 29 requests Andrew had answered by hand, and all 7
vague asks.
- It posted no replies to non-requests.
- His calls on borderline cases became prompt rules: yes to Playa Fly,
no to "Joe Cuervo Karaoke 🎤" on an unrelated video.

## Changes

- **Decision core:** `backend/services/youtube_comment_bot.py`, a pure
module with injectable ports (Gemini classifier/translator, board,
channel index), shared by the replay and the worker.
- **Worker:** `backend/workers/youtube_comment_bot_worker.py`, called
through `POST /api/internal/youtube-comments/run` (`?dry_run=true`).
- Creates a create-only action doc per comment **before** posting, so
nothing gets a double reply.
- Keeps a high-water mark with a 6h overlap. A Gemini error causes a
retry, not a permanent skip.
  - Per-run cap and quota coordination.
  - Posts to #noise per reply; failures go to #alerts, throttled.
- **Kill switch:** `YOUTUBE_COMMENT_AUTOREPLY_ENABLED` defaults to
**false**, and this PR leaves it off. A follow-up PR switches it on
after a prod dry run. `YOUTUBE_COMMENT_AUTOREPLY_START` enables a
one-time backfill of unanswered requests.
- **Requests board:**
- New `source="youtube_comment"`; bot-added requests are owned by the
placeholder `youtube-comment@nomadkaraoke.com` with no votes.
- The **first real up-voter claims the request**, inside the vote
transaction. The daily picker skips unclaimed requests, so a free credit
never goes to the placeholder.
- Frontend `?request=<id>` deep link: highlight, scroll, banner (33
locales).
- Signing in from the link uses purpose `requests_board:<id>`, so the
magic link returns to the request with `&upvote=1` and the upvote is
cast once.
- **OAuth:** the bot uses its **own** Desktop OAuth client and secret
`youtube-comment-bot-credentials` (`youtube` + `youtube.force-ssl`),
because Google refuses `force-ssl` in the device flow. The upload token
is untouched.
- **Infra (Pulumi, already applied locally):** secret container + Cloud
Scheduler `youtube-comment-autoreply-hourly`.
- **Description template:** a make-your-own line moves above YouTube's
"...more" fold (`utm_campaign=make_cta_top`). The daily backfill
re-drains the channel over about 11 days.
- **Docs:** `docs/REQUESTS-BOARD.md` § YouTube comment auto-reply,
API.md, README.md, plus the plan in
`docs/archive/2026-10-22-youtube-comment-autoreply-plan.md`.

## Testing

- `backend/tests/test_youtube_comment_bot.py` (33): skips, routing,
vague asks, translation fallback, malformed classifier output, API
helpers.
- `backend/tests/test_youtube_comment_bot_worker.py` (12): kill switch,
dry run, idempotency (second run and concurrent claim), go-live start,
cap, quota, missing scope, post failure, classifier-error retry.
- `backend/tests/emulator/test_song_request_service.py`: bot add/dedupe,
claim on first upvote, downvote doesn't claim, picker skips unclaimed,
human resubmit claims, `find_published_by_key`.
- `backend/tests/test_admin_login_link.py`: `requests_board:<id>`
redirect and purpose helper.
- `frontend/e2e/regression/requests-board-deep-link.spec.ts` (3):
highlight + sign-in purpose, upvote-once on return + param dropped, no
toggle-off when already upvoted.
- Full `make test`: backend 6584 passed; the only failures are the
local-environment `test_emulator_integration` cases, which fail
identically on `origin/main` (port 8080 is held by another session's
emulator). Frontend Jest: 1760 passed.
- Behavioural acceptance: the 6-month replay above.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_016E3ibN2EcKxnGQvEf6T4V8

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 01:51:15 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 714ac74324 fix(resolve): keep Gemini SDK deadline >= 10 s (v0.279.7) — hotfix for #1203 (#1204)
## Problem
#1203 (v0.279.6) set the Gemini SDK timeout for free-text resolve to 7
s. The Gemini API rejects that:

```
resolve_free_text attempt 1/2 failed: 400 INVALID_ARGUMENT. 'Manually set deadline 7s is too short. Minimum allowed deadline is 10s.'
```

So since the 21:16 UTC deploy, every uncached "Describe the song" /
kjbox resolve answers kind `none` (seen on prod for 6 new descriptions
at 21:20).

## Fix
- The SDK timeout goes back to `match_judge_timeout_ms` (12 s), with a
floor of 10 s (`GEMINI_MIN_DEADLINE_MS`).
- The per-attempt cap (now 8 s) is client-side only
(`asyncio.wait_for`). Worst case is still about 20 s, and decide waits
25 s.
- 4xx errors other than 429 are no longer retried.
- Tests: the SDK deadline respects the API minimum, and a client error
is not retried.

Urgent hotfix: no local CodeRabbit run, so this PR is left open for the
bot review.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01G81fc5sBB8GwG5Xhz2UMgc

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

- **Bug Fixes**
- Improved AI-powered match judgments by handling timeouts and temporary
service failures more reliably.
- Prevented unnecessary retries for invalid requests, allowing
unavailable results to be returned faster.
- Improved timeout handling to reduce premature failures during AI
evaluations.

- **Chores**
  - Updated the application version to 0.279.7.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 17:36:33 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 2172306a21 fix(resolve): bound free-text resolve so a first-time description answers in time (v0.279.6) (#1203)
## Problem
Prod 2026-10-20: mobile "Describe the song" returned "we couldn't work
out…" on the first ask of every new description; repeating worked.

Evidence (Cloud Logging, `karaoke-backend`):
- `/api/kjbox/catalog/resolve` from decide: 12.1 s, 12.7 s, 17.2 s, 10.6
s.
- 20:28:41 "The song from Titanic": Gemini call at :41.6, then `Failed
to search tracks from karaoke-decide` at :53.7 — the catalog tidy waited
10 s on decide's `/api/catalog/tracks`, which took 37.6 s (cold start;
it routinely takes 35-44 s cold).
- 20:32:44: `resolve_free_text failed: 504 DEADLINE_EXCEEDED` after 10.5
s (one stalled Gemini call; a fresh call answered in ~2 s).
- decide's 12 s upstream timeout fired first → `{}`; gen finished and
cached → second ask worked.

## Fix (`match_judge/free_text.py`)
- Gemini: 7 s per attempt (SDK + `asyncio.wait_for`), one retry on
transient failures; quota/billing errors are not retried. Same
provider/model/config as before.
- Catalog tidy: 4 s cap. On timeout, answer with the AI verdict
(`reason: unavailable`, so not cached) and let the shielded catalog
search finish in the background, warming decide and the catalog proxy's
cache.
- Worst case ~18-20 s; decide now waits 25 s (karaoke-decide#163).

Tests: retry on a stalled call, retry-then-give-up, no retry on quota,
slow catalog tidy answers with the AI verdict while the search finishes
in the background.

CodeRabbit (local): 2 findings. Fixed the minor one (hold the task from
creation). Skipped the suggestion to coalesce concurrent catalog
searches: the route already rate-limits per partner and the catalog
proxy caches results, so it isn't worth the extra complexity here.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01G81fc5sBB8GwG5Xhz2UMgc

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 17:01:20 -04:00
Andrew Beveridge ☄️andClaude Sonnet 5.5 83e097fa3b feat(local-stack): fake review fixtures and knobs for mobile native review (v0.279.5) (#1202)
## Summary
Local-stack (FAKE_WORKERS=1) support for nomad-mobile Phase 2, which is
the native lyrics and instrumental review. Maestro can now drive every
review state against the real review endpoints. Nothing changes in prod:
all hooks are gated on `fake_workers_enabled()`.

## Changes
- **New per-user `FakeConfig` knobs:**
- `review_fixture` (`default` / `corrections` / `duet` / `rtl` /
`large`). The fixtures are built from the real `CorrectionResult` types:
AI spans with `undo_info`, an untimed word, an evidenced vocal gap with
synced reference lines, duet singers, Hebrew/Arabic lines, and 300
lines.
  - `backing_vocals` (`silent` / `audible`)
  - `stems_delay_seconds` (stems land after review opens)
- `review_mode` and `auto_approval` (writes
`processing_metadata.auto_approval`, so the confident and skipped-step
states can be reached)
  - `review_complete_fail_once`
- **Fake media gets an amplitude envelope**, so waveforms aren't flat.
There is also an audible backing stem variant.
- **`/api/review/{id}/audio` dev-proxy branch** now uses
`_ranged_response` (Range → 206), so seeking works on iOS. The prod
signed-URL path is untouched.
- **`GET /api/test/jobs/{id}/review-result`** returns the submitted
review (selection, `is_duet`, corrected lines) for Maestro assertions.
- `docs/TESTING.md` § Local stack documents the knobs.

## Testing
- `make test`: backend 2184 + 6532 + 163 passed, frontend 1760 passed.
- 29 new tests in `backend/tests/fake_workers/test_review_fixtures.py`.
- Smoke-tested on the local stack: every fixture loads; `with_backing`
analysis; Range 206; 503 then 200 on complete; `review-result`;
`auto_approval` states; late stems.

Reviewed locally (CodeRabbit CLI, 1 finding fixed).

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01G81fc5sBB8GwG5Xhz2UMgc

---------

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-20 00:20:38 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 e050cb8b3d fix: rescue repetitive-song reference lyrics the relevance filter undercounts (v0.279.4) (#1201)
@coderabbitai ignore

## Why
Andrew: job 5c1d24eb loaded no reference lyrics. LRCLIB and Genius both
returned the correct song, but the relevance filter scored them 26% and
25%, below its 30% threshold. Anchor coverage credits a repeated phrase
only once, so chorus-heavy songs undercount. The transcription was
decent, with a few consistent mishearings.

## What
`filter_irrelevant_sources(..., transcribed_text=)` now does a second,
**additive** check. A source the anchor filter rejects is kept when its
word-trigram overlap with the transcription clears four guards:
- **≥ 25%** of reference trigrams matched, repeats counted;
- **≥ 20** reference trigrams, so tiny references can't score by chance;
- **≥ 10** matches;
- **≥ 15 distinct** matched trigrams, so a shared refrain alone can't
pass.

Trigrams must contain three distinct words, so "na na na" and "oh oh oh"
don't count. Tokens are NFC-normalized and keep combining marks.

Both call sites pass the transcription: the corrector, which also serves
the review UI's lyrics search and add-source flow, and the controller's
uncorrected path. Without `transcribed_text`, behaviour is unchanged.
The rescue **never rejects** a source the anchor filter keeps. The
serialized `rejected_sources` metadata is unchanged.

## Calibration (459 recent production jobs; per-job reference txt vs
`(Lyrics Uncorrected).txt`)
- **Wrong-song baseline:** 491k cross-job pairs, excluding pairs with
the same title.
  - Worst genuine pair: 9.4%; p99.99: 5.5%.
- The only cross-job pairs that pass are one job whose audio is another
job's song under a wrong title. Their transcriptions overlap 45%, where
unrelated songs score under 10%.
- A naive n-gram rule would have admitted a wrong song at 41% through
shared "na na" filler. Requiring three distinct words removes it.
- **Rescued:** 22 sources that are currently rejected, **all the correct
song**, scoring 26–88%. Examples: the 5c1d24eb song, Deacon Blue, UB40,
Atomic Kitten, Sezen Aksu, Grateful Dead, Etherwood, Sub Focus, Terry
Allen.
- **Left rejected by the distinct-trigram guard:** one refrain-only drum
& bass track (8 distinct matches; it's the right song but has the risk
profile). This was a deliberate, cautious choice.
- No lyrics text is committed; the tests use invented lyrics.

## Testing
- `tests/correction/test_relevance.py`, 17 passed (8 new):
  - a repetitive correct song is rescued;
  - no transcription means unchanged behaviour;
  - a wrong song is still rejected;
  - filler doesn't count;
  - a tiny reference isn't rescued;
  - a source the anchor filter keeps is never rejected;
  - a shared refrain alone can't rescue;
  - combining marks are kept.
- `tests/unit`: 2184 passed. Running `tests/unit`, `tests/correction`
and the backend review tests together gives the same 12 order-dependent
failures on untouched main; they pass separately, as CI runs them.
- Local CodeRabbit: 2 cycles. Fixed the combining-marks and
distinct-trigram findings.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Jw21syo4HizJJjYw2mg2Vp

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 00:08:38 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 b67b273056 feat(kjbox): decide partner secret for catalog resolve + fake judge + error monitor for decide search (v0.278.10) (#1190)
## ⚠️ Merge prerequisite (do BEFORE merging)

The Secret Manager secret `decide-gen-partner-secret` must have a
version before this merges, otherwise the CI deploy's `--set-secrets
...DECIDE_GEN_PARTNER_SECRET=decide-gen-partner-secret:latest` fails.

```
gcloud auth login admin@nomadkaraoke.com
cd ~/Projects/nomadkaraoke/karaoke-gen-decide-partner-resolve/infrastructure
GOOGLE_OAUTH_ACCESS_TOKEN=$(gcloud auth print-access-token) pulumi up     # review: should only add the decide-gen-partner-secret Secret
openssl rand -hex 32 | tr -d '\n' | gcloud secrets versions add decide-gen-partner-secret --project=nomadkaraoke --data-file=-
```

No extra IAM grant needed: gen's API runs as `karaoke-backend@`, which
already has project-level `secretmanager.secretAccessor`.

## Summary
- `POST /api/kjbox/catalog/resolve` now also accepts karaoke-decide via
header `X-Partner-Secret` (new setting `decide_partner_secret` / env
`DECIDE_GEN_PARTNER_SECRET`). Other kjbox endpoints stay
`X-Kjbox-Secret` only. decide gets its own AI-call budget (60/min) so
kjbox nights are never starved; the verdict cache is shared.
- Fake match judge / resolve for the local E2E stack (`FAKE_WORKERS`):
`judge_kind` / `judge_canonical_*` knobs; no Gemini or catalog calls.
- Error monitor now watches decide's new `karaoke-decide-search` service
and `song-index-build` job.
- Pulumi: declares the `decide-gen-partner-secret` Secret (value added
manually); CI deploy mounts it.
- Version 0.278.10.

## Testing
- [x] Changed-module + kjbox/catalog/error-monitor/fake-worker backend
tests pass (133 passed). Full `backend/tests` suite exceeds 10 min
locally, so left to CI.

## Review
- CodeRabbit CLI failed twice (WebSocket closed / connection error), so
no local CodeRabbit review completed; I did a manual diff review instead
(no issues found). Deliberately NO `@coderabbitai ignore`, so the GitHub
bot reviews this PR.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01VsLw9cTTMBab1tiRxCsYrm

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 21:42:27 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 4286d98dbe fix: LRCLIB as a registered server-side app; honour Retry-After, retry 5xx (v0.279.2) (#1200)
@coderabbitai ignore

## Why
LRCLIB now applies stricter rate limits to unregistered clients
(https://lrclib.net/docs#implementation-requirements-rate-limiting), and
Cloud Run shares egress IPs. On 10-19, prod jobs got intermittent 503s,
and job 80e7ffdf lost its LRCLIB lyrics. LRCLIB is our only free source;
it supplied lyrics for 165 of 220 jobs in 30 days. Andrew registered a
server-side app.

## What
- **`karaoke_gen/lyrics_transcriber/lyrics/lrclib_client.py`** (new):
- Sends `Lrclib-App-Id` + `Lrclib-Api-Key` from `LRCLIB_APP_ID` /
`LRCLIB_SECRET_KEY`.
- Applies LRCLIB's rules: every 429 sets a **process-wide cooldown** of
`Retry-After` plus up to 30% jitter. The wait is never shortened, and
overlapping 429s keep the latest expiry. Nothing is sent until the
cooldown has passed. It waits and retries once only when the wait is
≤30s.
  - Retries 5xx after 1s/2s, honouring a 5xx `Retry-After`.
- A **401 on our credentials** logs a WARNING and drops to anonymous
access for the process, so a revoked key degrades instead of losing
LRCLIB.
- `LRCLIBProvider` (both `get` and `search`) uses it. The
`quick_version` renderer uses it with `retry=False`, so the fast path
gets the auth and the cooldown but never blocks.
- **Infra:**
  - Pulumi secrets `lrclib-app-id` / `lrclib-api-key`.
  - Env vars on the `lyrics-transcription-job`.
- The backend service deploy's `--set-secrets` (`ci.yml`). The
karaoke-backend SA already has project-wide `secretAccessor`.

## Deploy order (do NOT merge before steps 1–2)
1. `pulumi up` to create the two secrets and the job env. Needs
`/gcp-admin`.
2. Add the secret versions from Andrew's `.envrc` values.
3. Merge. The CI backend deploy references the secrets.

## Testing
- New `tests/unit/test_lrclib_client.py` (14 tests): headers; anonymous
use; Retry-After with jitter; a long Retry-After is not shortened and
blocks further calls; no-retry mode; a second 429; 5xx backoff; 5xx
Retry-After; 401 fallback; overlap merge (mutation-checked); provider
routes through the client.
- 164 passed across the client, `quick_version` and `search_lyrics`
tests.
- **Live** with the real app key: Dar Williams - Iowa and Wolfmother -
Where Eagles Have Been were found. One 503 was recovered by the retry.
- Local CodeRabbit, 3 cycles. I fixed the Retry-After cap, the cooldown
after a final 429, the `quick_version` 401 path, the overlap merge and
5xx Retry-After. I skipped one false positive about the tests' fake
clock (the retry path sleeps, then sends without re-checking the clock).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Jw21syo4HizJJjYw2mg2Vp

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 17:33:19 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 653801cd04 fix: Musixmatch provider uses the RapidAPI v2 /lyrics?artist=&title= endpoint (v0.279.1) (#1199)
@coderabbitai ignore

## Why
BACKLOG "Musixmatch lyrics provider dead for 30+ days (0/287 requests
succeed)". We're still paying for the RapidAPI listing
(`cavsn/musixmatch-song-lyrics-api`, Pro $14.99/mo, active), but the old
`GET /lyrics/{artist}/{title}/` path now 404s for every song. I checked
the listing's playground (via Andrew's RapidAPI login). The API moved to
`GET /lyrics?artist=&title=` (optional `trackId`, `translate`) and
returns a flat `data` object.

## What
- `MusixmatchProvider._fetch_data_from_source` calls the new endpoint
with query params. A miss comes back as **HTTP 200 with `data.error`**,
or with empty `lyrics`. Both are now treated as not-found and logged at
INFO, so they don't alert.
- `_convert_result_format` reads `trackName` / `artistName` / `trackId`
/ `lyrics`. Raw cache entries in the old `macro_calls` shape still
convert.

## Testing
- New `tests/unit/test_musixmatch_provider.py` uses real response shapes
captured today: endpoint and params, the 200 miss logged with no ERROR,
empty lyrics, no key, v2 conversion, legacy conversion.
- 47 passed across the provider, base provider and `search_lyrics`
tests.
- **Live check** with the production key: Queen - Bohemian Rhapsody 47
lines, Adele - Hello 48 lines. The upstream misses some songs (piri -
dog; Taylor Swift - Shake It Off), so coverage is partial but no longer
zero.
- Local CodeRabbit: 0 findings.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Jw21syo4HizJJjYw2mg2Vp

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 16:34:13 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 867617626a feat: requests board picks the visitor's language + header language switcher (#1198)
## Summary
Follow-up to #1193. `requests.nomadkaraoke.com` always served the
English board (the old redirect was also hard-coded to `/en/requests`).

- **Worker (`requests-host`, already deployed):** `/` serves
`/{locale}/requests/`. The locale comes from the `nk_requests_locale`
cookie, else `Accept-Language` (primary-subtag match like gen's
`LocaleRedirect`, so `pt-BR` → `pt` and `nn` → `nb`), else `en`.
`/{locale}/` (e.g. `/es/`) forces a language. Responses send `Vary:
Accept-Language, Cookie`.
- **Board header:** gen's `LanguageSwitcher` sits top-right. On the
vanity host it sets the cookie and reloads `/`, so the copyable URL
stays bare. On gen it behaves as before (`router.replace` to
`/{locale}/requests`).
- **`LanguageSwitcher`:** new optional `onSelectLocale` prop that
replaces the default navigation.

## Testing
- Worker: `node --test infrastructure/cloudflare/requests-host/`, 10
pass. This includes a check that the Worker's locale list matches
`frontend/i18n/routing.ts`.
- New `components/__tests__/LanguageSwitcher.test.tsx` (2 tests); the
703 component tests pass.
- Live checks against the deployed Worker: `es`, `ja`, `pt-BR`, `en-US`
each get the translated board, and the cookie beats `Accept-Language`.
- Mobile (390px) screenshot of a local build: the switcher shows as a
compact flag top-right and the dropdown fits the screen.
- Prod E2E spec extended with a browser-language check and a switcher
round-trip. It will run after deploy.

Local CodeRabbit review: no findings.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_016FYpoNVXzJzqKFaowqpwxr

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 16:15:08 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 37b4f5126a fix: serialise translation-preview renders and keep them in GCS (OOM SIGKILLs) (v0.278.17) (#1197)
@coderabbitai ignore

## Why
BACKLOG "translation-preview renders get the API instance SIGKILLed
(OOM)" (#alerts msgs 1557981404242116608, 1560563690053439628). On 10-09
and 10-16, two concurrent `GET /api/themes/translation-preview` renders
for **different** languages, each a 4K ffmpeg frame taking about 20s,
were followed by `Container terminated on signal 9`. Every in-flight
request got a 503, including a real user's `create-from-search` on
10-09. The existing per-key lock only prevented duplicate renders of the
*same* language.

## What
- `_render_slot = BoundedSemaphore(1)`: one render at a time per
instance.
- **Persistent cache:** rendered frames go to GCS at
`themes/_translation_previews/<theme>/<lang>-<sha256(style_params +
sample)[:16]>.jpg`. Each language renders once per theme change rather
than once per cold instance, so very few requests ever wait on the slot.
A theme edit changes the digest, so stale frames are never served. Cache
reads and writes are best-effort: failures log a WARNING and fall back
to rendering. They use the bucket directly, so a cache miss isn't logged
as a "not found".
- **Fixed an existing test blind spot:** this module's autouse
`no_sleep` fixture stubs the global `time.sleep`. As a result
`test_concurrent_cold_previews_render_once` passed without any threads
overlapping, and so did my first version of the new concurrency test.
Both now wait on `threading.Event().wait()`.

## Testing
- New: a GCS hit serves without rendering; a theme change gets a new
key; four languages never render concurrently. Mutation-checked: with
`BoundedSemaphore(8)` it fails with `assert 4 == 1`.
- `test_lyrics_translation.py`: 33 passed. Local CodeRabbit: 0 findings.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Jw21syo4HizJJjYw2mg2Vp

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 15:38:28 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 acb02e138a fix: un-pin audio-separator to 0.47.1 with RoFormer overlap=2 (v0.278.16) (#1196)
@coderabbitai ignore

## Why
BACKLOG "audio-separator 0.45–0.47 regression: mel-band RoFormer karaoke
models 3–5× slower on L4". The root cause is **python-audio-separator
#299 (0.47.0)**, not #298. #299 made RoFormer `overlap` a *window count*
that defaults to each model YAML's `num_overlap`. Before that, `overlap`
was effectively an 8-second step, which meant almost no overlap. The
karaoke models' YAMLs set `num_overlap` 4/8, so a 3-minute track went
from 23 forward passes to 87 (aufr33) and 124 (gabox_v2, becruily). That
matches the measured 25→75s, 26→149s and 27→157s. 0.45.0 and 0.46.0
still had the old step formula.

python-audio-separator #313 (released as **0.47.1**) keeps #299. It adds
a per-model forward-pass budget test on every PR and a GPU per-pass
timing test (T4: 1.21 s/pass), plus a README note. Andrew chose
overlap=2 for gen: smoother chunk seams than 0.44's no-overlap, at about
1.5× the 0.44 compute instead of 4–6×.

## What
- `karaoke_gen/audio_processor.py`: `SEPARATOR_MDXC_PARAMS` (overlap=2)
goes to all four `Separator(...)` constructions (stage 1, stage 2
karaoke ensemble, single-model paths) **only when audio-separator ≥
0.47**. On older versions `overlap=2` meant a 2-second step, which is 4×
slower, so they keep their defaults. Ensembles load each model through
`load_model`, so the param reaches every model in the preset.
- `backend/Dockerfile.gpu-base`: `AUDIO_SEPARATOR_VERSION=0.47.1`, with
the comment rewritten to explain the root cause. 0.47.1 needs
`torch>=2.3` on Linux, so the pre-installed `torch==2.6.0+cu126`
(required by the L4 driver) is kept.
- The poetry lock stays on 0.44.1. The version-aware params keep CI and
CPU images on the old semantics, and there's no torch churn.

## Expected cost per ~3-min track (forward passes)
| model | 0.44.2 | 0.47 YAML default | 0.47 + overlap=2 |
|---|---|---|---|
| karaoke_aufr33 | 23 | 87 | 44 |
| karaoke_gabox_v2 | 23 | 124 | 32 |
| karaoke_becruily | 23 | 124 | 32 |
| instrumental fv7z / resurrection | 23 | 32 / 21 | 32 / 21 |

## Verification plan (before /backlog-finish)
After the deploy, time the separation of the daily E2E or a real job on
the audio-separation-job logs. I expect roughly 1.5× the 0.44.2
karaoke-stage times (~40–50s per karaoke model rather than 75–157s). If
it's worse, revert the pin to 0.44.2. That's one line.

## Testing
- `tests/unit/test_audio_processor_local_gpu.py`: version gating
(0.47.x/0.48/1.0 → overlap 2; 0.44/0.46 → defaults; unknown → defaults;
fresh copy), and both stages' constructors receive `mdxc_params.overlap
== 2`.
- `make test`: the backend unit suite passes. The only failures are 8
`test_emulator_integration.py` tests, caused by my local environment:
another session's emulator setup is on :8080/:4443, and they fail
identically on untouched main. CI is green on main.
- Local CodeRabbit: 0 findings.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Jw21syo4HizJJjYw2mg2Vp

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 15:09:41 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 92fd647203 fix: preview with untimed in-browser edits fills timings or 422s, not a KeyError 500 (v0.278.15) (#1195)
@coderabbitai ignore

## Why
BACKLOG "Lyrics preview 500 `KeyError 'end_time'` on unsaved edits"
(#alerts msg 1559340473439617117). On 10-12 a paying user's review
preview failed 3× (job 16792df7). Both the GCE and the local fallback
failed with `'end_time'`. `JSON.stringify` drops `undefined`, so a word
or line edited in the browser can arrive without its timing keys, and
`CorrectionOperations.update_correction_result_with_data` indexes
`w["end_time"]` / `s["end_time"]`.

## What
- `timing_sanitizer.sanitize_corrections`: a segment with missing or
non-finite bounds now gets them from its words' timings, or else from
the neighbouring lines' gap. It used to be skipped. The existing word
clamp then fills untimed words inside that window.
- New `find_missing_timings()` returns the text of a line that still has
no usable timing.
- `POST /api/review/{id}/preview-video` now sanitizes the payload up
front, as review submit already did. If a line can't be recovered, it
returns **422** `The line "…" has a word with no timing. Sync that line,
then try the preview again.` (en/es/de) instead of a 500.
- The frontend payload is unchanged. The backend now treats a missing
key and a null value the same way, so a frontend change would add
nothing.

## Testing
- `test_timing_sanitizer.py`: bounds from words, missing `end_time`
**key** (the 16792df7 shape), untimed line filled from its neighbours'
gap, unrecoverable line reported. The old "non-finite segment bounds
left untouched" test is replaced, because that is the behaviour this PR
changes.
- `test_routes_review.py::TestPreviewVideoMissingTimings`: a 422 naming
the line, with no render attempted; fillable gaps reach the renderer
filled.
- 131 passed across review routes, sanitizer, and submit-corrections
tests. Local CodeRabbit: 0 findings.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Jw21syo4HizJJjYw2mg2Vp

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 14:57:13 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 3bbf397dc4 fix: keep auth, tenant lookup and job-log writes off the API event loop (v0.278.14) (#1194)
@coderabbitai ignore

## Why
The residual event-loop freezes BACKLOG item. Seven days of
`EVENT_LOOP_STALL in progress` stacks (2026-10-12→19, 355 samples),
grouped by their outermost app frame, ranked the app-level loop
blockers:

| Source | Samples | Max |
|---|---|---|
| in-process screens worker → job-log handler → sync Firestore write per
log line | 30 | 2.2s |
| `require_auth` → `validate_token_full` (session read on **every**
authed request) | 19 | 1.9s |
| preview encodes
(`_run_preview_encode`/`_encode_preview_in_background`) → job-log write
| 15 | 2.4s |
| `change_visibility` → sync Dropbox/GCS deletes | 6 | 1.2s |
| tenant middleware → `get_tenant_config` (GCS on cache miss) | 5 | 1.2s
|
| `require_review_auth` → `get_job` | 5 | 2.2s |
| `auto_correct_worker._run_suggest` → `get_job` (the only stall since
#1192) | 1 | 1.2s |

(The render/video in-process stalls stopped with #1192.)

## What
- **`backend/services/log_write_offload.py`**: when a job-log write is
made on a running event loop in the API process, it goes to one
background writer thread. The queue is bounded at 5000 and a drop is
logged at WARNING, rate-limited; `atexit` drains the queue. Writes stay
inline in threads and Cloud Run Jobs, so the worker code paths behave
exactly as before. Logs are read back ordered by their own `timestamp`,
so the hand-off doesn't reorder them.
- **`asyncio.to_thread`** for
`require_auth`/`require_auth_legacy`/`require_review_auth`, the tenant
middleware, `rerender_with_current_theme`'s tenant lookup,
`auto_correct._run_suggest`, and `change_visibility`'s `get_job`.
- **Visibility change:** `_delete_*_outputs` are now plain `def`, and
the sync steps run in one worker thread. Moving them off the loop
removed the implicit serialisation, so both directions now **claim
`visibility_change_in_progress` atomically** before touching anything.
They use the new `FirestoreService.try_set_job_state_flag`, a
transactional test-and-set. A lost claim returns 409. The
regenerate→private continuation passes `holds_claim=True`, because its
regenerate already set the flag. The claim is released if the reset
fails, and a raising screens trigger rolls back like a `False` one.
- **Guard (`test_no_blocking_async_routes.py`):** `BLOCKING_CALLS` gains
`validate_token(_full)`, `validate_session`, `get_tenant_config`,
`append_log_to_subcollection` and `append_worker_log`. Every `async def`
in `api/dependencies.py` and `middleware/` is now checked, not just
decorated routes. Mutation-checked: reverting `require_auth` fails the
guard.
- `docs/LESSONS-LEARNED.md` entry. About 70 direct
`get_job`/`update_job` calls in async routes remain. They're single
round-trips that rarely show in the watchdog, and the doc says to move
them when one does.

## Testing
- New: `test_log_write_offload.py` (inline vs offloaded, Cloud Run Job
inline, ordering, full-queue drop, Firestore append off the loop
thread), claim tests in `test_change_visibility.py` (lost claim, release
on failure, chained continuation, trigger exception, route 409), and
emulator `test_try_set_job_state_flag.py` (once-only, already set,
missing job, one winner among 4 concurrent callers).
- Backend: 6478 passed. The 8 failures are
`test_emulator_integration.py`, caused by my local environment: another
session's emulator setup has an `ssh` forward on :4443 instead of
fake-GCS, and they fail identically on untouched main. CI's emulator job
covers them and the new emulator test.
- Local CodeRabbit: 3 cycles. I fixed the claim race (two findings),
trigger-exception rollback, and the guarded claim release. I skipped two
minor nitpicks on the pre-existing trigger-failure rollback.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Jw21syo4HizJJjYw2mg2Vp

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 14:41:34 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 b02cce3cc7 feat: serve the requests board on requests.nomadkaraoke.com itself (#1193)
## Summary
`requests.nomadkaraoke.com` used to 302 to
`gen.nomadkaraoke.com/en/requests`, so the short link couldn't be copied
out of a mobile address bar (Andrew pastes it into YouTube comment
replies). Now the board is served on the vanity host itself.

- **New Worker `requests-host`**
(`infrastructure/cloudflare/requests-host/`): `/` serves gen's
`/en/requests/` (query kept), paths with an extension are proxied from
gen, and every other path 302s to the same path on gen.
- **Pulumi (already applied):** Worker route
`requests.nomadkaraoke.com/*`; old redirect rule removed; DNS comment
updated.
- **Tenant detection:** `requests` added to the non-tenant subdomains.
The browser and Pages Function now share one list
(`frontend/lib/edge/nonTenantSubdomains.ts`); the browser copy had
drifted and was missing `requests`. Backend list updated to match.
- **Board "Make it yourself"** goes to `gen.nomadkaraoke.com/app` when
on the vanity host.
- **Docs:** runbook rewritten, including why a Pages custom domain
didn't work. Attaching the domain bypassed zone redirect rules and
served the gen homepage for ~3 min before rollback.

## Testing
- `node --test infrastructure/cloudflare/requests-host/`: 4 pass
- Jest tenant + new non-tenant list tests: pass. Backend
`test_tenant_middleware.py`: 36 pass
- `make test`: 6465 pass. 8 fail in `test_emulator_integration.py`, and
the same 8 fail on a clean `origin/main` checkout (pre-existing,
env-related)
- New prod E2E `e2e/production/requests-vanity-host.spec.ts`: board at
`/` with no redirect and the API working, other paths 302 to gen. The
pass/fail status of the "not a tenant" check (still sends `X-Tenant-ID:
requests` until this frontend deploys) is listed below.
- Manual browser check on prod: URL stays
`requests.nomadkaraoke.com/?utm_source=…`, board renders, no console
errors

| Prod E2E check | Before this deploy |
|---|---|
| serves board at `/` | ✅ |
| other paths → gen | ✅ |
| not treated as tenant | ❌ expected, fixed by this PR |

Local CodeRabbit review done (1 minor doc finding, fixed).

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_016FYpoNVXzJzqKFaowqpwxr

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 14:18:57 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 1344a086da fix: worker Cloud Run Jobs never dispatch video/render into the API process (v0.278.12) (#1192)
## Problem: a regression from #1191, caught by #alerts triage
At 07:03 UTC on 2026-10-19, two new freeze alerts fired: **52s with 29
requests affected** (culprit
`firestore_service.append_log_to_subcollection`, during CDG packaging)
and **81s with 33 requests** (culprit `dropbox_service.upload_file`).
The stacks show `video_worker.generate_video` running **inside
karaoke-backend**.

Cause: in today's daily E2E run (job 5a5e5181), #1191's park/resume
worked as designed (parked 06:55:04, resumed 06:57:50). But the resume
runs in `audio-separation-job`, which doesn't set
`USE_CLOUD_RUN_JOBS_FOR_VIDEO`. So `trigger_video_worker` fell back to
Cloud Tasks, which runs it as a `BackgroundTask` on the API, putting the
whole video worker on the API's event loop. The same latent bug already
applied to the audio worker's auto-approval → render trigger.

## Fix
- `backend/config.py`: inside a Cloud Run Job (`CLOUD_RUN_JOB` is set by
the platform), `use_cloud_run_jobs_for_video` and
`use_cloud_run_jobs_for_render` default to **true**. An explicit env
value still wins. This covers every worker job (audio, lyrics, download,
...) with no infra change, through the normal image deploy.
- Error monitor normalizer: `requests_affected=N` →
`requests_affected=<N>`, so a stall culprit is one alert pattern instead
of one per request count.

## Testing
- New `backend/tests/test_config_cloud_run_job_dispatch.py`: inside a
job → true; outside → unchanged; explicit env wins.
- New normalizer test.
- `backend/tests` + `tests/unit/services/error_monitor`: 6871 passed.
- Local CodeRabbit review: 0 findings.

## Verify after deploy
The next parked-then-resumed job logs `Started Cloud Run Job
video-encoding-job` from audio-separation-job, and there are no
`EVENT_LOOP_STALL` lines on karaoke-backend with `video_worker` in the
stack.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_019pmG2XAgtUguEJWLFpkKHN

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 13:17:49 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 0d0605d7b5 fix: final encode waits for stem separation instead of stranding the job; pin audio-separator 0.44.2 (v0.278.11) (#1191)
## Problem
The daily E2E Stage 2 failed on 2026-10-17 (run #216) and 2026-10-18
(#217). When a customer finishes review **before** stem separation does,
`video-encoding-job` fails its prerequisites (`Missing instrumental:
instrumental_clean`) 3× within seconds. The job then sits at
`instrumental_selected` ("Starting final encode") forever: it's never
marked failed, nothing alerts, and separation finishing doesn't
re-trigger the encode.

The race became common when the GPU image's unpinned `pip install
"audio-separator[gpu]"` picked up **0.47.0** on the 10-17 rebuild. On
L4, the mel-band RoFormer **karaoke** models run 3–5× slower than on
0.44.2, so total separation went from ~5 to ~10–16 min:

| model (~3-min track) | 0.44.2 | 0.47.0 |
|---|---|---|
| karaoke_aufr33_viperx | ~25s | 75s |
| karaoke_gabox_v2 | ~26s | 149s |
| karaoke_becruily | ~27s | 157s |
| instrumental fv7z / resurrection | ~50–60s | ~62s |

The model YAMLs still say `num_overlap: 8, batch_size: 1`, so this isn't
the #299 YAML-defaults change; the cause is somewhere in 0.45–0.47's
RoFormer inference changes. That root cause is tracked separately.

## Fix
- **Pin** `audio-separator[gpu]==0.44.2` in
`backend/Dockerfile.gpu-base` (`ARG AUDIO_SEPARATOR_VERSION`). Changing
this file triggers the GPU base image rebuild in CI.
- **Video worker parks instead of failing.** When the selected stem is
missing *and* separation is still running (`audio_complete` false, valid
status), it sets `state_data.video_waiting_for_stems`, puts a
user-visible message ("Waiting for audio separation to finish; the final
video starts automatically") and exits 0, so Cloud Run doesn't retry. It
re-checks after parking, and an atomic Firestore test-and-clear
(`claim_job_state_flag`) means exactly one of the video worker and the
audio worker proceeds if the stems land mid-park.
- **Audio worker resumes it.** After `mark_audio_complete`,
`resume_video_if_waiting_for_stems` re-triggers a parked encode. The
fake-workers pipeline does the same, so the local stack matches prod.
- **Backstop in `recover_stuck_jobs`.** `instrumental_selected` for more
than 10 min with the stem present → re-trigger the video worker (covers
a lost resume or lost dispatch). More than 60 min with separation never
finishing → `mark_job_failed` (`stage=video_waiting_for_stems`), so the
job fails loudly. Jobs older than 24h are never revived. (No jobs are in
`instrumental_selected` in prod right now.)

## Testing
- New `backend/tests/test_video_waits_for_stems.py` (16 tests): pending
detection (including that a wrong status never parks), park / continue /
claim-race outcomes, and resume (flag claimed / no flag / job moved on).
- `test_recover_stuck_jobs_repark.py`: 5 new sweep tests (re-trigger,
fresh, still separating, never-ready → fail, >24h ignored).
- Full `backend/tests` (excluding emulator): 6455 passed.
- Local CodeRabbit review: 0 findings.

## Verify after deploy
- The GPU job logs `Separator version 0.44.2`, and karaoke-model
separation is back to ~20–30s.
- The next daily E2E Stage 2 passes.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_019pmG2XAgtUguEJWLFpkKHN

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 02:21:52 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 70b2f4279a feat(alerts): collapse error bursts, suppress link previews, de-noise #alerts (v0.278.9) (#1188)
Andrew's goal: "I'd like to make sure that our alerts channel only has
alerts in it that are actionable and important, or are just signal and
not noise." This follows #1186 (#noise routing).

## Changes
1. **Burst collapse** (`error_monitor/monitor.py`, `discord.py`,
`config.py`). When a run has more than 3 new patterns for one service,
including leftover un-alerted ones, the monitor sends ONE #alerts
message. It has the count, total occurrences, the top 5 by count,
first/last seen, and a `scripts/query-error-patterns.py --service X
--status new` pointer. All of those patterns are marked alerted. The
grouping is deterministic, since LLM grouping is off in prod. The
10-messages-per-run cap still applies, and unsent patterns are picked up
again next run. Evidence: on 2026-10-07/08, one mb-refresh run produced
73 separate alerts.
2. **No link previews.** Every error-monitor post (`_post`) and every
`ops_alerts` post now sets `flags: 4` (SUPPRESS_EMBEDS), through a new
`suppress_embeds` kwarg on `DiscordNotificationService.post_message`.
The YouTube "New upload" post keeps its embed.
3. **Loop watchdog.** `EVENT_LOOP_STALL in progress` is now always
WARNING. The `ended` line keeps the existing ERROR rule. In the last two
weeks there were 24 "in progress" pattern alerts on top of 15 "ended"
ones. I checked `infrastructure/modules/monitoring.py`
`EVENT_LOOP_STALL_LOG_FILTER`: it keys on `"EVENT_LOOP_STALL ended"` +
`severity>=ERROR`, so this needs no Pulumi change.
4. **Frontend chunk-load errors** (`frontend_ingestion.py`,
`firestore_adapter.py`). Reports matching `isChunkLoadError` collapse
into one pattern per build with `alert_min_hourly_count=10`. The monitor
holds that pattern un-alerted until more than 10 arrive within an hour,
which means a broken deploy. Separately, Safari/Firefox stacks now get
the message prepended before hashing (a pattern was named
`@https://gen.nomadkaraoke.com`). Chrome hashes are unchanged. Existing
Safari/Firefox patterns will re-key once, and burst collapse groups
them.
5. **known_issues.** Added a service-scoped ignore for
`github-runner-manager` "no available instance" aborts during scale-up
(seen 09-20, 09-23, 09-27, 10-08). The same message from other services
still alerts.
6. **divebar-mirror schedulers.** Added `attempt_deadline="600s"` on
both `divebar-mirror-daily` and `divebar-mirror-refresh`; the function
timeout is 540s. With the 180s default, Cloud Scheduler retried syncs
that were still running, and the overlapping runs hit OOM (4x in 30
days).

## ⚠️ Pulumi
`infrastructure/modules/divebar_mirror.py` changed, so **this needs
`pulumi up` by Andrew before merge**. Auto-merge is NOT enabled.

## Testing
- New and updated unit tests in
`tests/unit/services/error_monitor/{test_monitor,test_discord,test_frontend_ingestion,test_firestore_adapter,test_known_issues}.py`,
`backend/tests/{test_loop_watchdog,test_discord_service,test_ops_alerts}.py`,
and `tests/unit/test_divebar_mirror_scheduler.py` (an AST guard:
deadline > function timeout).
- `pytest backend/tests --ignore=backend/tests/emulator`: 6421 passed,
36 skipped.
- `tests/unit/services/error_monitor`: all pass.
- Local CodeRabbit review (`--committed`) completed with 0 findings.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_019pmG2XAgtUguEJWLFpkKHN

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 05:13:13 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 9af5c8224e docs(medley): recommend backing_vocals_db -8, not -14 (#1189)
The customer reported that -14 dB made the backing singers inaudible
(about 25 dB under the band). -8 dB leaves them about 18 dB under: quiet
but present. This updates the MEDLEYS.md spec-table example and adds the
guidance. Docs only, so no version bump.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01QV2TQwyC7pwVkGPCk98Bqg

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 23:58:23 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 0e507852af fix(alerts): demote expected conditions off ERROR + Payment Link webhook fix (v0.278.9) (#1187)
## Summary

Alert hygiene, part of keeping Discord #alerts to actionable signal
only. ERROR is page-worthy: the error monitor turns every new ERROR
pattern into an #alerts message. This PR moves expected conditions down
to INFO/WARNING and fixes the one real bug among them, which was a
Payment Link checkout crashing the Stripe webhook. It covers the BACKLOG
items "Payment Link checkouts crash the Stripe webhook" and part of
"Alert-hygiene bundle".

## Changes

1. **Stripe webhook, non-gen checkouts** (`users.py`,
`stripe_service.py`, `stripe_admin_service.py`)
- A session that gen didn't create (a Stripe Payment Link:
`payment_link` is set, there's no gen metadata, and
`customer_email=None`) used to log ERROR "No user email in checkout
session". `store_payment` then crashed on `None.lower()`, so the payment
never reached `stripe_payments` or revenue.
- New `StripeService.is_gen_checkout()`. A non-gen checkout now logs
INFO "non-gen checkout (payment link ...)" and grants no credits. It is
marked processed for idempotency and is still stored, with
`customer_details.email`, `order_type="payment_link"`, `payment_link_id`
and the line-item description.
- `store_payment` now handles None for metadata, customer_details,
total_details and payment-method fields.
- **Not backfilled:** session `cs_live_a1ENsua3w6mDH04PPc…` (2026-10-13,
a $36 tip, no customer action needed). It could be backfilled through an
admin path that calls `store_payment` on the expanded session. GCP
writes from agents are restricted, so this PR doesn't do it.
2. **Lyrics providers** (`base_lyrics_provider._log_request_exception`):
a 404 means "no lyrics for this track" and now logs INFO. 429 moves to
WARNING alongside 5xx, connection errors and timeouts. Auth errors and
other 4xx stay ERROR. After 5 404s from one provider in a process, it
logs a single WARNING saying the provider may be dead (TODO points at
the Musixmatch backlog item).
3. **research_audio** (`audio_search.py`): no more
`AWAITING_AUDIO_SELECTION -> SEARCHING_AUDIO ->
AWAITING_AUDIO_SELECTION` transitions, which were invalid and logged 2
ERRORs per re-search. The job stays in AWAITING_AUDIO_SELECTION with no
status writes, so a crashed request can't strand it. After the search
the job is re-read. If a selection landed in the meantime, the fresh
results are discarded with a 409 instead of replacing the results that
selection indexed into.
4. **audio-download-job on a deleted job** (`audio_download_worker.py`):
if `get_job()` returns None at the start or after the download, or a
failure happens and a fresh read confirms the doc is gone, the worker
logs INFO and returns success, so it exits 0. That means no `fail_job`,
no further writes, no worker triggers and no Cloud Run retry. A GCS
NotFound on a job that still exists stays an ERROR. Seen with job
6ffe88ad on 2026-10-14.
5. **Loop watchdog**: "Failed to record loop stall (non-fatal)" now logs
WARNING with exc_info. Only that line changed, to keep the conflict with
feat/alert-noise-monitor trivial.
6. **Job-list queries** (`firestore_service.list_jobs`,
`list_jobs_summary`, `list_jobs_by_batch_id`): `query.stream()` now gets
`retry=` and `timeout=` of 30s. The client default is 300s for both,
which caused the "Error listing jobs summary: Timeout of 300.0s
exceeded" errors.

Version 0.278.9 (0.278.8 is taken by the still-open #1186).

## Testing

- New/updated unit tests:
- `test_stripe_webhook_non_gen_checkout.py`: webhook stores the payment,
logs INFO, logs nothing at ERROR, and marks the session processed.
- `test_stripe_admin_service.py`: Payment Link session and null fields.
- `tests/unit/test_base_lyrics_provider.py`: log levels for 404, 429 and
401, plus the dead-provider WARNING.
- `test_audio_search_research_attach.py`: no status writes, and a
selection made mid-search gets a 409.
- `test_audio_download_worker.py`: job deleted at start, mid-download,
or with NotFound on write.
  - `test_loop_watchdog.py`: recorder failure logs WARNING.
- `test_firestore_list_query_timeout.py`: bounded retry/timeout on all
three list queries.
- `pytest backend/tests --ignore=backend/tests/emulator`: 6434 passed.
`pytest tests/unit`: 2108 passed.
- Local CodeRabbit review: cycle 1 raised 2 findings (non-gen session
idempotency, re-search race), both fixed. Cycle 2 was clean.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_019pmG2XAgtUguEJWLFpkKHN

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 02:58:58 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 a6d4163d2e feat(alerts): route non-signal Discord posts to #noise (v0.278.8) (#1186)
## Summary
Andrew asked for #alerts to carry only real problems. Routine and
informational posts now go to the new **#noise** channel. The GitHub org
PR webhook was already moved to #noise separately.

| Message | Before | After |
|---|---|---|
| Error monitor: New Error Pattern / Spike / Incident | #alerts |
#alerts |
| Error monitor: ✅ Patterns Auto-Resolved | #alerts | **#noise** |
| Error monitor: 📊 Daily Error Digest | #alerts | **#noise** |
| Nightly Backup Report (clean run) | #alerts | **#noise** |
| Nightly backup failed/partial (title is now "🚨 Nightly Backup FAILED")
| #alerts | #alerts |
| ✅ E2E Daily Test PASSED | #alerts | **#noise** |
| 🩹 E2E self-healed Stripe (auto-merge PR opened) | #alerts | **#noise**
|
| 🩹 self-heal PR needs CI approval click / could not open | #alerts |
#alerts |
| E2E / Tenant E2E failures, DR freshness stale, Job failed, Publish
completeness SHADOW, Gemini unavailable, OAuth credential alerts,
quarterly DR drill reminder | #alerts | #alerts (unchanged) |

## How the noise webhook is resolved
- **Backend / error monitor and the backup function** read it at
runtime: env `DISCORD_NOISE_WEBHOOK_URL` first, then the Secret Manager
secret `discord-noise-webhook` (the name can be overridden with
`DISCORD_NOISE_WEBHOOK_SECRET_NAME`). No Pulumi change is needed,
because both service accounts already have project-wide
`secretAccessor`.
- **GitHub Actions** uses `secrets.DISCORD_NOISE_WEBHOOK_URL ||
secrets.DISCORD_ALERT_WEBHOOK_URL`.
- **Nothing is silently dropped.** If the noise webhook can't be
resolved, or a post to it fails, the message goes to #alerts instead. In
the workflow this uses `curl -fsS` with a retry on the alert webhook.
- The error monitor now rate-limits each channel separately (10 per run
each), so noise can't use up the alert budget during an incident.

## Deploy notes
- **Backup Cloud Function:** CI doesn't deploy its source. Run
`infrastructure/functions/backup_to_aws/deploy.sh` after merge.
- **Error monitor job:** runs `karaoke-backend:latest`, so it picks up
the change once CI pushes the image.
- **No `pulumi up` needed.**

## Testing
- `tests/unit/services/error_monitor`: 376 passed. New tests cover noise
vs alert routing, fallback when the noise webhook is missing or fails,
separate rate limits, and secret resolution.
- `backend/tests` (excluding emulator): 6418 passed, 36 skipped.
- `infrastructure/functions/backup_to_aws`: 60 passed, including the new
`test_discord_alert.py` and routing tests in `test_main.py`. These were
run in a scratch venv because `functions_framework` isn't in the conda
env, which was already the case before this change.
- `actionlint` on `e2e-daily.yml`: no new findings. The only findings
are info-level shellcheck notes that were already there.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_019pmG2XAgtUguEJWLFpkKHN

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 02:46:50 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 290f927175 fix(backend): move remaining event-loop blockers off the loop (v0.278.7) (#1185)
## Summary
Found while triaging Discord #alerts and 14 days of freeze telemetry
(2026-10-18). The event-loop watchdog still caught real freezes that hit
users:

- **151s freeze, 32 requests affected (2026-10-17 20:10 UTC).** `POST
/api/review/{id}/custom-lyrics/generate` called
`CustomLyricsService.generate` (a sync Gemini call) on the loop. Every
user on the instance saw "server unavailable". It now runs via
`asyncio.to_thread`.
- **14.9s freeze, 7 requests (2026-10-16).** The first `from
google.cloud import run_v2` in
`worker_service._trigger_worker_cloud_run_job` ran on the loop.
`google.cloud.run_v2` and `httpcore` (imported lazily by httpx) are now
in the startup `_LAZY_ROUTE_IMPORTS`, so they are imported off-loop
before readiness.
- **GCS on the loop in async routes.** `_prepare_theme_for_job` ran on
the loop in `upload_and_create_job`, `create_job_with_upload_urls`,
`create_job_from_url` and audio-search `search_audio`;
`submit_audio_edit` also uploaded `edit_log.json` on the loop. These now
go through `to_thread`.
- **Guard.** `upload_json`, `_prepare_theme_for_job` and `generate` are
added to the `BLOCKING_CALLS` denylist in
`test_no_blocking_async_routes.py`. The denylist caught the two extra
routes above.

## Testing
- `pytest backend/tests --ignore=backend/tests/emulator`: 6418 passed,
36 skipped.
- `test_audio_search` source check updated to accept the
`to_thread(_prepare_theme_for_job, …)` form.
- Local CodeRabbit review: 0 findings.

## Verify after deploy
No new `EVENT_LOOP_STALL ended … requests_affected>0` lines with
culprits `_call_gemini`, `run_v2` or `upload_json`.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_019pmG2XAgtUguEJWLFpkKHN

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 01:56:13 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 9d6aa3df7d feat(medley): per-section backing_vocals_db for the karaoke track (v0.278.6) (#1184)
## Summary
The Fiverr medley customer said the backing singers in two songs were
too loud to sing over in tune, and asked for them as a faint background.

- **New option:** a per-section `backing_vocals_db` in the medley spec.
For the karaoke track only, the section is rebuilt from the clean
instrumental plus the backing-vocal stem at that level, e.g. `-14` for a
faint background. The rest of the karaoke track, the clean instrumental
and the guide are unchanged.
- **Cached mix:** the mix is cached per job/level, written to a
`.partial.flac` and renamed, so an interrupted run never leaves a
truncated mix to be reused. If the job has no backing stem, it stops
with a clear error.
- **Docs:** `docs/MEDLEYS.md` spec table.

## Testing
- `test_make_medley.py`: parsing, the ffmpeg graph, only the karaoke
variant changing, the atomic write and the cache. 20 + chord-chart tests
pass.
- Real data: the customer's medley with `-14` on Remember When and On
the Other Hand.
- Only those two sections change; the others differ by ≤ -43 dB, which
is crossfade overlap.
  - Overall section loudness is unchanged.
- A local CodeRabbit review found 1 issue (stale or partial cache). It
is fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 15:04:43 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 03ee693f28 feat(medley): customer chord charts + static-gain master (v0.278.5) (#1183)
## Summary
Two follow-ups on the medley tooling, both from the Fiverr medley
customer's feedback.

- **Fixes the robotic artifact at the start of the karaoke audio.** The
final single-pass `loudnorm` ran in dynamic mode. On a song that opens a
cappella (HSLHT sings from 0.4s, band at 3s), the instrumental's first
3s is near-silent separation residue at -70 dB, and `loudnorm` boosted
it by about 50 dB into a loud warble. The master is now one static gain
to -14 LUFS plus a 4×-oversampled limiter at -1 dB (`master_filter`).
After the fix, the first 3s measure -61 dB (were -15 dB). Sync is
sample-identical and true peak is -0.9 dB.
- **Imports the customer's own chord chart.**
`scripts/medley/chord_chart.py` plus a new `make_medley.py chart` step
read a chords-over-lyrics chart (.txt or .docx in Courier, capo shapes,
`Capo N` / `Key change Capo 1` lines). They align it word-by-word to the
reviewed lyrics, so chords come out timed on the words the chart puts
them on. Detected chords fill intros, breaks and spoken verses,
transposed into the capo shapes.
- **Shows the capo in the chord box.** New `ChordChange.capo` field,
shown as "capo N" under the chord.
- **Docs:** `docs/MEDLEYS.md` gets a new section on importing a customer
chart, and the mastering guidance now says never to use single-pass
`loudnorm`. `docs/README.md` has an entry.

## Testing
- Unit tests:
  - New: `tests/unit/test_chord_chart.py` (9).
- Added: capo tests in `test_guitar_chords.py`, plus master and capo
tests in `test_make_medley.py`.
- Full `tests/unit` suite: 2100 passed, 65 skipped (run before the last
CodeRabbit fix; only the 9 chord-chart tests were re-run after it).
- Real data: the customer's 4-song chart against the medley. The
lyrics-with-chords output matched the chart line by line, and 4K frames
rendered with capo labels.
- A local CodeRabbit review found 1 issue (two chord groups after the
same word could come out in the wrong order). It is fixed and has a
test.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01QV2TQwyC7pwVkGPCk98Bqg

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 14:24:45 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 9a44a30368 fix(review): idempotent correction-data review transition (v0.278.4) (#1182)
## Summary
The same race as #1181, on the other door into review. `GET
/api/review/{job_id}/correction-data` moves `AWAITING_REVIEW` to
`IN_REVIEW`. When two loads arrive concurrently, both see
`AWAITING_REVIEW`, so the second transition raises `in_review ->
in_review`. That returns a 500 and the review page shows "Failed to load
lyrics". The iOS in-app browser hits this every time; mobile E2E caught
it.

**Fix:** a shared `ensure_in_review()` helper in
`backend/services/review_state.py`, now used by both `correction-data`
and `start-review`.
- An already-`IN_REVIEW` job does not transition again.
- A lost race re-reads the job and carries on.
- Any other invalid transition still errors.
- `record_review_started` runs only for the winning transition.

Version 0.278.4.

## Testing
- 4 new `TestCorrectionDataReviewTransitionIdempotent` cases.
- 493 review and correction backend tests pass.
- Local CodeRabbit found nothing.
- Verified against the local stack via the mobile Maestro review-bridge
flow.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_011yLH4kWXMqjeKfZUG3ZiWZ

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 12:07:03 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 072f85cad9 fix(review): idempotent start-review (no more in_review -> in_review error) (v0.278.3) (#1181)
## Summary
Opening a lyrics review could fail with `Invalid state transition
in_review -> in_review`.

`POST /api/jobs/{job_id}/start-review` always called
`transition_to_state(IN_REVIEW)` and never checked for `in_review`. So
any second open failed: a duplicate request, a double-mount, a second
tab, or the mobile app's in-app browser opening the review again. Mobile
E2E found this, and a screenshot captured gen's error page.

The endpoint is now idempotent:
- If the job is already `in_review`, it returns `{"status": "success",
"job_status": "in_review"}`.
- If the transition raises because a concurrent call won the race, it
re-reads the job. If the job is now `in_review`, it returns the same
success.
- Any other invalid transition still raises.
- `record_review_started` runs only for the winning `awaiting_review →
in_review` transition.

Version 0.278.3.

## Testing
- 4 new `TestStartReview` cases in `backend/tests/test_jobs_api.py`.
- The review-related backend tests pass: 452.
- Local CodeRabbit found nothing.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_011yLH4kWXMqjeKfZUG3ZiWZ

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 10:20:58 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 2838d2dd1d fix(credits): keep credits_charged through job creation; one helper for confirm/refund/select (v0.278.2) (#1180)
## Summary
Fixes `credits_charged` being wiped from `state_data`, which made
duration confirm, refunds and the web cost dialog disagree. Mobile E2E
against the local stack found it: a 15-minute track showed "1 credit in
total (1 more)", and confirming the true total was rejected with 409.

**Root cause:** `create_job` debits the credit and records
`state_data.credits_charged`. But create-from-search, audio-search
create, made-for-you orders (which also lost `payment_bypassed`) and the
admin source override then replaced the whole `state_data` map, deleting
the field. Readers filled the gap with different defaults:
- reconcile: 1
- confirm-duration: 0
- 48h expiry: 0

**Fix**
- **Writers:** per-key `state_data.<key>` updates instead of whole-map
writes. Retry-from-scratch keeps the billing keys. Reconcile records
`credits_charged` when it pauses a job.
- **Readers:** one helper, `backend/services/job_credits.py`
(`get_credits_charged`, `is_payment_bypassed`). Every reader uses it:
reconcile, confirm-duration, `/select`, the bulk worker,
`_refund_credit_for_job` and the 48h expiry.
- For legacy jobs that lack the field, it returns what `create_job`
actually debited: 0 for bypassed, admin, tenant or no-email jobs,
otherwise 1.
- **`/select` and the bulk worker** never bill payment-bypassed jobs
(made-for-you, tenant, admin-created). This includes legacy ones whose
flag was wiped.
- **API responses:** `GET /api/jobs/{id}` and the job lists return the
resolved `credits_charged` when the stored field is missing. This is
response-only and never persisted, so the web and mobile clients
acknowledge exactly `credits_charged + pending`.
- **Confirm-duration** records the full total, so cancelling after a
confirm refunds it all. The 48h expiry refunds once:
`cancel_job(refund=False)` stops a second refund.
- **Admin source override** clears the old source's duration, selection
and confirm markers and keeps `credits_charged`. The next `/select` then
charges only the difference.
- **Web cost hook:** uses the server's resolved figure.
- Version 0.278.2.

## Testing
- New `test_credits_charged_consistency.py` covers:
- reconcile, then confirm with total 2, then cancel: 2 refunded, for new
and legacy jobs
  - 409 on the old total
  - refund amounts unchanged for jobs that have the field
  - `/select` and the bulk worker never deducting on bypassed jobs
  - the override followed by `/select` charging only the difference
- Stale-expiry legacy tests, and an emulator test for the dotted-path
merge (runs in CI).
- Backend unit tests: 6411 passed. Emulator integration tests run in CI;
locally they collided with a running local stack. Frontend jest: 1754
passed.
- An independent review found one Critical issue (`/select` billing
bypassed jobs) and one Important issue (web/server default mismatch).
Both are fixed and re-reviewed clean. Local CodeRabbit raised one minor,
which is fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_011yLH4kWXMqjeKfZUG3ZiWZ

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 08:38:35 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 1616bc997b fix(api): sign final MP4s in GET /api/jobs/{id} download_urls (v0.278.1) (#1179)
## Summary
`GET /api/jobs/{id}` has always returned an empty `download_urls` for
real jobs. `JobManager.get_output_urls` signs only `job.output_files`,
and current workers never populate that field. Final videos live in
`file_urls.finals`.

This change also signs each `file_urls.finals` entry whose key ends in
`_mp4` (for example `lossy_720p_mp4`):
- URLs expire after 120 minutes.
- When a key exists in both, the `output_files` entry wins.
- Entries that aren't strings are skipped.
- After a real signing failure, it stops signing further entries, so a
stalled signer can't add a timeout per MP4.

**Why:** the mobile app (nomad-mobile Phase 1) streams the 720p MP4 in
`expo-video` and shows download progress. Both need a URL that supports
HTTP Range requests. The authenticated `/download/` proxy doesn't
support them.

Signing still goes through the bounded executor in `storage_service.py`
with its 12s timeout. The route is a sync `def`, so the event loop is
never blocked.

Version 0.278.1.

## Testing
- 6 new `TestGetOutputUrls` cases in
`backend/tests/test_job_manager.py`. That file passes: 58 tests.
- Local CodeRabbit raised one finding (the type guard), which is fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_011yLH4kWXMqjeKfZUG3ZiWZ

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 05:25:45 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 695b8ddf84 feat: mobile app backend — handoff token, app checkout return, Google/Apple sign-in (v0.278.0) (#1178)
## Summary
Backend support for the Nomad Karaoke mobile app (nomad-mobile Phase 1,
"Make it").

- **`POST /api/users/auth/handoff-token`**: the app opens a job's web
review already signed in. The token is a single-use, 120s
`job_review:<id>` magic link, returned as
`{FRONTEND_URL}/{locale}/auth/verify?token=…&handoff=1`.
- The verify page skips the "Complete Sign-In" tap only when the URL has
`handoff=1` **and** the server confirms the token was minted as a
handoff (`link-status` → `handoff: true`). Emailed links keep the
scanner guard.
  - The web locale is validated against the 33 web locales.
- **Checkout `return_to: "app"`**: Stripe success and cancel go to `GET
/api/users/credits/app-return`, which 302s to
`nomadkaraoke://checkout?status=success|cancelled`.
  - The client never supplies a URL, so this adds no open redirect.
  - `session_id` is never forwarded.
- **`POST /api/users/auth/google` and `/auth/apple`**: ID-token sign-in
(PyJWT + JWKS: RS256, issuer, audience, expiry, `email_verified`). It
returns the same `VerifyMagicLinkResponse` as email-code sign-in,
through a shared `finish_verified_sign_in`.
- Email-code, Google and Apple all share the same abuse checks
(`_sign_in_checks`).
  - A JWKS outage returns 503 `identity_provider_unavailable`.
- Google audiences default to the new mobile web and iOS OAuth clients,
which are recorded in ARCHITECTURE.md.
- **Job summary** now includes `credit_refunded`.
- **Local stack:**
- fake-config `search_confidence` (perfect / recommended / unsure), with
a drift guard tied to the frontend `getSearchConfidence`
  - `youtube_url_enabled`
  - `fake:<email>` social tokens, accepted only in fake mode
- PyJWT is now a direct dependency (`pyjwt[crypto]`). Before this it was
only present transitively.
- Version 0.278.0.

## Testing
- `make test` is green:
  - CLI: 2088 passed
  - backend unit: 6360 passed
  - emulator: 156 passed, plus 4 handoff lifecycle tests
  - frontend jest: 1751 passed
- New suites:
  - `test_handoff_token.py`
  - `emulator/test_handoff_token_lifecycle.py`
  - `test_checkout_app_return.py`
  - `test_social_signin.py`
  - `test_magic_link_status.py`
  - `fake-search-tiers.test.ts`
  - verify-page tests
- Reviews:
- An independent code review was approved, and its hardening suggestions
were applied.
  - A local CodeRabbit review found nothing.

Plan: `docs/archive/2026-10-17-mobile-app-auth-handoff-plan.md`. Spec:
nomad-mobile
`docs/superpowers/specs/2026-10-17-mobile-phase1-make-it-design.md`.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_011yLH4kWXMqjeKfZUG3ZiWZ

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 04:03:05 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 f8e9a8b3b6 feat(bulk): guided Artist → Album → Tracks steps in bulk album mode (v0.277.1) (#1177)
## Summary
Bulk album mode was hard to figure out — the one-line text hint was easy
to skip and it wasn't obvious the list was albums. This adds visual
cues:

- **Step strip** `1. Artist › 2. Album › 3. Tracks` with icons; current
step highlighted, completed steps show the pick and are clickable to
jump back (replaces back buttons)
- **Per-step headings** with icons ("Choose an album — Albums by Britney
Spears")
- **Artist/album results as clickable cards** (icon + chevron, hover
border)
- **Tracklist**: Select all / none + "N of M tracks selected"
- Icons on **By album / By text** tabs
- Album mode **hides batch settings + submit until a tracklist is
loaded** (no more "Create 0 karaoke videos" on step 1)
- Leaving the tracklist clears its rows; stale in-flight lookups are
dropped (CodeRabbit finding) so a previous album can't be submitted

Cover art deliberately not used (would hit MetaBrainz's Cover Art
Archive live).

## Testing
- New `frontend/__tests__/bulk-album-mode.test.tsx` (7 tests: step
navigation, select all/none, rows cleared on back, stale-response drop,
settings/submit gating)
- Full frontend Jest suite: 142 suites / 1741 tests passed
- Visually verified locally against prod API (desktop + 390px)
- 33 locales translated

Local CodeRabbit review completed; 1 finding fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01BrgLoS3bpnku9n21g6YGgw

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 01:11:44 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 7cf5beede8 feat: guitar chords in karaoke videos + ChordMini ensemble chord detection (v0.277.0) (#1175)
## Summary
Adds optional guitar chords to karaoke videos, first used for a Fiverr
medley customer who wants chords on their video.

- **Rendering (gen library):**
`OutputGenerator.generate_outputs(chords=[ChordChange(start, end,
label)])`.
- Each chord change during a sung line is drawn above the word it lands
on, in a row reserved above every line (the same slot mechanism as
translated lyrics).
- Chords light up when played, using the same `\k` highlight as lyrics:
gold while upcoming (theme `chord_color`), sung-lyric colour once
played. Quick changes on one word never overlap.
- A top-right "now / next" chord box runs for the whole song, covering
intros and instrumental breaks.
- With no chords supplied, output is unchanged. A test asserts the ASS
is byte-identical.
- **Detection (`scripts/medley/detect_chords.py`):**
- Ensembles both ChordMini checkpoints (BTC + 2E1D, MIT; about 80%
majmin/MIREX, the best open weights we found) over the full mix, gen's
clean instrumental, and an htdemucs_6s harmonic mix (guitar + piano +
bass + other).
- Weighted per-frame vote, then merges short blips, snaps changes to
beats and spells chords for the key (Ab/Db/Eb7, F#m, borrowed
bIII/bVI/bVII as flats).
- **Medley script:** new `chords` step plus a `"chords": true` spec
option. Chords are clipped per section and hand over mid-crossfade, then
shifted for the countdown.
- **Docs:** `docs/MEDLEYS.md` § Guitar chords (setup, method, accuracy
caveats, theme keys) and a docs/README.md entry.

There's no job or UI option yet. The medley pipeline uses this for now;
a product version would need a chords toggle plus detection in a worker.

## Testing
- New: `test_guitar_chords.py` (12: chord timeline helpers, word
placement measured against the font, layout geometry, highlight timing,
now/next box, theme colour, no-chords path byte-identical),
`test_detect_chords.py` (7), and 2 medley chord-shifting tests.
- `tests/unit`: 2087 passed, 65 skipped.
- End to end on the 5 real medley songs: models/inputs agree on 91–99%
of major/minor labels, and the output matches the expected I–IV–V
progressions. Rendered 1080p and 4K medley videos with 146 chord
changes; spot-checked frames (verse, instrumental screen, chorus).
- Local CodeRabbit: 1 finding (doc section reference), fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01QV2TQwyC7pwVkGPCk98Bqg

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 00:24:11 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 8ec789a9aa fix(bulk): album lookup reads our MusicBrainz mirror, never the live API (v0.277.0) (#1176)
## Summary
Bulk → By album → "Britney Spears" showed **"Lookup failed. Please try
again."** `/api/bulk/album/artists` returned 502 because every attempt
to `musicbrainz.org/ws/2` from Cloud Run failed with
`httpx.ConnectError('')`. The same request from a home IP returns 200.
It last worked on 9/29. Most likely MusicBrainz is refusing our shared
GCP egress.

Per Andrew, we should only query our own infra, with no live-API
fallback. `musicbrainz_service` now reads **only our BigQuery
MusicBrainz mirror** (`karaoke_decide.mb_*`, which karaoke-decide's
`mb-refresh` rebuilds weekly from the official dumps).

## Changes
- **`backend/services/musicbrainz_service.py`**
- The HTTP client, throttle and retries for musicbrainz.org are removed.
Queries go through the BigQuery REST `jobs.query` endpoint (httpx + ADC;
no new dependency).
- Queries are parameterized, capped with `maximumBytesBilled` = 2 GiB,
and cached in memory for 10 min.
  - Lookups by table:
- artists: `mb_artists_normalized`. Ranking is exact match (including
"The "+q), then prefix, then substring, then popularity.
    - albums: `mb_artist_credit_artists` → `mb_release_groups`.
    - editions: `mb_releases`.
- tracks: `mb_release_tracks`, the new table in karaoke-decide#158,
clustered by release group + release.
- Measured against prod data: about 0.6–1 s per query, MB to 0.4 GB
billed.
- Canonical-pressing selection now ranks an edition by its best
release-event country.
- **Route:** passes `release_group_mbid` through on edition switches. An
unknown release returns **404** instead of 502.
- **Frontend:** `switchEdition` sends the album's `release_group_mbid`
so the lookup stays cluster-pruned. Older clients still work through a
one-off lookup that scans about 0.2 GB.
- **Infra:** the `karaoke-backend` SA gets project-level
`bigquery.dataViewer` + `bigquery.jobUser`. decide's Pulumi owns the
dataset ACL, so a dataset-scoped grant would fight it.
- **Docs:** `docs/API.md` and
`docs/archive/2026-10-16-bulk-album-musicbrainz-mirror.md`.

## Rollout (order matters)
1. Merge karaoke-decide#158, then build `mb_release_tracks`: by hand
(SQL in the archive doc, about $0.08) or at Sunday's refresh.
2. Run `pulumi up` for the IAM change. The agent's credentials are
read-only, so this needs Andrew.
3. Merge this PR.

## Testing
- `make test`: all passed (backend + 1735 frontend tests).
- New and rewritten unit tests cover normalization, BigQuery row
decoding, the REST request shape (no musicbrainz.org, billing cap,
params, cache), the HTTP/timeout errors, every lookup, the release-group
lookup path, and the 404.
- Ran the service live against the prod mirror: Britney Spears → 39
albums → …Baby One More Time (11 tracks, 6 variants); also checked
"beatles", "Beyoncé", "ac/dc", "Sigur Rós".
- CodeRabbit CLI: 1 minor finding (404 vs 502), fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01C1sMXk2MaxBLbep5jgiFab

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 00:06:16 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 85390fd12b docs: medley-making guide + spec-driven make_medley script (#1174)
## Summary
Documents the multi-song medley process used for the first
classic-country medley, and turns it into a reusable script so the next
one only needs a spec file.

- `docs/MEDLEYS.md`: the full process. One gen job per original song,
then identical cuts on every audio variant, and the reviewed lyrics
shifted onto the medley timeline and rendered with gen's own
`OutputGenerator`. Also covers how to choose cut points (crossfades,
pot-out/pot-in, keys/tempos, checking the brief against the recording)
and gotchas.
- `scripts/medley/make_medley.py`: `fetch` (Firestore + gcloud CLI) →
`audio` (per-section LUFS gain, crossfade/gap/pot fades, `loudnorm` to
-14 LUFS / -1 dBTP, three variants) → `video` (shifted reviewed lyrics +
countdown + themed title/end screens + karaoke and guide MP4s).
- `scripts/medley/example-spec.json`: the country medley as a worked
example.
- Linked from `CLAUDE.md` Quick Reference and `docs/README.md`.

Ops tooling only. No backend or frontend change, so no version bump.

## Testing
- `tests/unit/test_make_medley.py`: 12 unit tests covering spec
validation, the timeline offset maths, lyric-segment shifting (including
the same song used twice) and the ffmpeg filtergraph.
- End-to-end run against the real prod jobs: `fetch` → `audio`
reproduces the hand-built v1 medley **bit-for-bit** (max sample diff 0.0
on both the instrumental and guide WAVs). `video --resolution 360p`
renders both MP4s and the LRC.
- Local CodeRabbit review: 1 finding (unchecked ffmpeg exit code in
`measure_lufs`), fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01QV2TQwyC7pwVkGPCk98Bqg

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 19:44:27 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 bb55efd3c2 fix(cloudflare): pin email catch-all name to stop refresh drift (#1173)
Every `pulumi preview --refresh` on the nomad-cloudflare stack planned a
no-op update on the 3 `EmailRoutingCatchAll` resources. The live rules
matched the code, but the provider reads back `name: ""`. Setting
`name=""` explicitly fixes it (same pattern as
`support_subaddress=False`).

Already applied locally. `pulumi preview --refresh` now reports **132
unchanged** (zero drift).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01QuVxnKWqXrdn3ux2VWLxS8

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Updated the configuration for catch-all email routing to specify an
empty name. This is an infrastructure-level change; no changes to
end-user email features or behavior are described.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 14:33:32 -04:00