2052 Commits
Author SHA1 Message Date
Andrew Beveridge ☄️andClaude Opus 5.5 6b7e057e50 feat(cloudflare): Google login for Cloudflare Access (#1172)
Adds a Google identity provider to the Nomad Karaoke Zero Trust org, so
KJ Controller / FiverrBot login pages offer **Sign in with Google**
alongside One-time PIN and Cloudflare.

- OAuth web client `Cloudflare Access
(nomadkaraoke.cloudflareaccess.com)` created in GCP project
`nomadkaraoke` via the console (no API exists for web OAuth clients).
Redirect URI:
`https://nomadkaraoke.cloudflareaccess.com/cdn-cgi/access/callback`.
- Client ID in Pulumi config `googleOauthClientId`, secret encrypted as
`googleOauthClientSecret`.
- **Already applied locally** (`pulumi up`: 1 created, 131 unchanged),
so CI's run will be a no-op. Verified a full Google sign-in to
kjbox.nomadkaraoke.com.
- Access policies are unchanged; they still allow only the listed
emails/domains.

Infra-only change (no app code, no version bump). No local CodeRabbit
run, so the bot can review this one.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01QuVxnKWqXrdn3ux2VWLxS8

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Google sign-in is now available as an identity provider for Cloudflare
Zero Trust access, alongside the existing one-time PIN option.
* Production access now supports the Google OAuth credentials needed to
enable this sign-in option. Users can choose Google authentication where
it is configured, while existing one-time PIN access remains available.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 14:26:41 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 8a16e33a34 fix(review): show duet singer chips in Waveforms view (v0.276.3) (#1171)
## Summary
In duet mode, the Simple/Advanced lyrics views show a per-line singer
chip (● 1 / ● 2 / Both) to see and toggle which singer each line belongs
to. The Waveforms view was missing it, because `WaveformSegmentRow`
never got the duet props.

## Changes
- `WaveformSegmentRow`: new optional `singer` / `hasSingerOverrides` /
`onSingerChange` props render the same `SingerChip` after the
index/trash/play controls, plus `onFocus`/`onBlur`/`className` props.
- `TranscriptionView`: in duet mode, passes the singer, the singer row
tint and focus tracking to each waveform row, so the singer keyboard
shortcuts work there too. Moved the tint logic into a shared
`singerTintClass` helper.

## Testing
- New `TranscriptionView.waveformsDuet.test.tsx`: chips render per line
in duet mode, clicking cycles the singer, and no chips appear when duet
mode is off.
- `WaveformSegmentRow.test.tsx`: chip renders and cycles; focus and blur
get reported.
- Full frontend jest: 141 suites / 1735 tests pass.
- Local CodeRabbit review: no findings.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_015cvzKpvvadUsJGZFMv9DXy

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 13:38:18 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 3d4c9c4e6e fix(gdrive): retry Drive 5xx/429, check publish completeness after fallback pass (v0.276.3) (#1170)
## Summary
The Discord "Publish completeness SHADOW" alert for **NOMAD-1783** (job
`dd885ea9`) was a false alarm. Drive returned a one-off `HttpError 500
internalError` on resumable-upload init, and the worker's fallback Drive
pass re-uploaded all three files 43s later. Investigating it turned up
four gaps:

1. **No retry on Drive 5xx.** `gdrive_service` only retried connection
errors, so one 500 dropped the whole orchestrator upload.
2. **Check ran too early.** The shadow check ran inside the
orchestrator, before `_handle_native_distribution`'s fallback pass.
3. **Fallback dropped the original-vocals guide.** kjbox's "Original
Vocals" slider had no guide for NOMAD-1783.
4. **Recovered state was overwritten.** The main path persisted the
orchestrator's empty `gdrive_files` and the stale "Google Drive upload
failed" warning over the recovered state (`state_data.gdrive_files` was
`None`).

## Changes
- `is_transient_error`: Drive 429/500/502/503/504, plus 403 with
`rateLimitExceeded`/`userRateLimitExceeded`, now retry like connection
errors. Uploads get 4 attempts (2/4/8s backoff). Retries stay idempotent
because each attempt re-runs the same-name delete. With
`replace_existing=False`, only connection errors retry.
- `VideoWorkerOrchestrator.defer_publish_completeness_check`:
`generate_video_orchestrated` sets it and runs
`check_publish_completeness()` after the fallback pass.
`redistribute_video` is unchanged.
- `_adopt_fallback_gdrive_upload`: carries the fallback's file IDs onto
the result. The Drive-failure warning is dropped only when the recovered
set is complete.
- `OrchestratorResult.original_vocals_guide`: the fallback pass now
ships the guide too.

## Prod remediation (already done)
- Built the NOMAD-1783 vocals guide with the orchestrator's own
`_emit_original_vocals_guide` (5s intro, silent intro verified) and
copied it to `gs://nomadkaraoke-divebar-files/files/Nomad
Karaoke/vocals-padded/`.
- Restored `state_data.gdrive_files` from the actual Drive IDs (one file
per folder, no duplicate) and cleared the stale warning.
- Logs show no other `Google Drive upload failed` in the last 30 days.

## Testing
- New `backend/tests/test_gdrive_fallback_recovery.py` (26 tests): the
retry predicate, upload retry after a 500, no retry without cleanup,
deferred and default check, guide kept on a failed upload,
adopt/partial/no-recovery cases, and the fallback shipping the guide.
- Full backend suite: 6268 passed, 36 skipped.
- Local CodeRabbit: 2 cycles. Fixed partial-recovery warning handling,
403 rate-limit reasons, and the non-replace retry guard. Skipped
folder-create idempotency, since the MP4/, MP4-720p/ and CDG/ folders
always exist.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_0199crntgXrDkR93dQeqvnGY

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.276.3
2026-10-16 13:32:17 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 b2ef88a861 fix(review): unify backing-vocals recommendation on the 3-stem scorer (v0.276.2) (#1169)
## Summary
- `backing_vocals_analysis.recommended_selection` (legacy
`AudioAnalyzer` rule: keep only if >20% audible or a segment > −20 dB)
disagreed with the auto-approval `score_backing` verdict on **59/389
jobs (15%, Jul–Oct)** — every case 0.5–20% audible with no loud segment.
The finish modal followed the scorer, the instrumental screens followed
the heuristic, so the same job showed different "✓ recommended" stems
(job 58555663).
- Where they disagreed, admin/Andrew picks matched the scorer 14/18;
customer picks lean clean regardless (38% clean even when both signals
say keep). The heuristic also recommended keeping on 5 jobs the scorer
flagged as "backing stem IS the lead" (humans picked clean 4/5).
- 58555663 itself: the intro backing (16–29s) is a voiced B3–C♯4 hum
motif with the lead stem silent, pitch-matched to the "Mm-mm-mm" hum
lines — `with_backing` was right.

## Changes
- `unified_backing_recommendation` / `with_unified_recommendation` in
`auto_approval/instrumental.py`: scorer's confident verdict → `clean` if
backing-is-lead → else the heuristic (preserved as
`heuristic_recommended_selection`). Idempotent.
- Applied at write time (`screens_worker._analyze_backing_vocals`) and
read time (correction-data, `/instrumental-urls`,
`/instrumental-analysis`) so already-analysed jobs are consistent too.
- `backing_is_lead()` predicate extracted from `score_backing` and
shared.
- No change to auto-completion / enforcement (`backing_decision`
untouched); frontend unchanged.
- Docs: API.md note, LESSONS-LEARNED entry. Version 0.276.2.

## Testing
- [x] `make test` — backend 2054 + 6243 + emulator 152, frontend 1730,
all passing
- [x] New unit tests (helpers matrix), worker write-path test, route
test with the 58555663 signal profile

## Review
- [x] CodeRabbit CLI review completed locally — no findings

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_018aUxu88FZajta46BRdmpcc

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.276.2
2026-10-16 13:08:49 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 82a332dbe1 fix(review): proceed to instrumental review while stems separate; tolerate old backend (v0.276.1) (#1168)
Follow-up to #1167, from Andrew trying it on job 58555663.

## Changes
- **Don't block the CTA while stems are separating.** #1167 disabled
"Proceed to Instrumental Review" until the stems landed, but the full
instrumental screen waits for them itself, and reviewers in a hurry
(happy with clean) used to go straight there. The button is enabled
again, and the waiting box adds "In a hurry? You can continue to the
full instrumental review now instead."
- **Tolerate a backend without `audio_complete`.** The frontend deploys
well before Cloud Run, so the new page polled the 0.275.0 endpoint,
never saw `audio_complete`, and spun until the 10-min cap even though
separation had finished. The hook now treats "both stems present" as
ready when the field is absent.
- i18n: dropped `waitingForStems`, added `stemsWaitingProceed` (33
locales).

## Testing
- Jest: hook + modal 31/31 (new: legacy-backend fallback; waiting-state
CTA enabled + submits)
- Playwright `cloud-mode-review.spec.ts`: 13/13
- Local CodeRabbit: no findings

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01QKJ7TxzFoUeNTug1PR8Cwh

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.276.1
2026-10-16 00:42:50 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 abf7b693b4 feat(review): instrumental chooser appears when stems land after review opens (v0.276.0) (#1167)
## Summary

Recent jobs didn't show the instrumental chooser / backing-vocals
waveform in the Preview Video modal. Root cause: review opens as soon as
**lyrics** finish (~2 min after download), while audio separation lands
2-3 min later. The review page fetches `instrumental_options` once at
load, so stems that arrive mid-review never reach the modal, which
silently fell back to "Proceed to Instrumental Review".

Evidence (last 5 admin jobs): in 4/5 the review page was opened 11-23 s
after ready-for-review, and separation finished 1m46s-3m00s **after**
the page opened; the one opened after separation (6cb0f55f) was fine.

## Changes

- **Backend:** `GET /api/review/{id}/instrumental-urls` also returns
`audio_complete`, `job_status`, `backing_vocals_analysis` and a freshly
derived `auto_approval` summary (backwards-compatible).
- **Frontend:** new `useLateInstrumentalStems` hook. When the page
loaded without stems it polls every 15 s (max 10 min) until
`audio_complete`, which is set only after stems, OGGs and backing
analysis are all written, so the chooser never shows half its data.
Fresh data is overlaid as a separate modal prop, not merged into `data`,
so undo history, saved sessions and the in-flight preview render are
untouched. The getter is held in a ref because the page re-creates its
API client each render.
- **Modal:** while waiting, a spinner box ("Still separating the music
from the vocals…") and a disabled CTA ("Waiting for the instrumental…").
The chooser, ✓ badge, backing waveform and preview toggle appear in
place when ready. On failure or timeout it falls back to the old path
with a short note.
- i18n: 4 new strings, 33 locales.
- Fixed an existing Playwright test that expected "Proceed to
Instrumental Review" with both stems present (stale since v0.214; it now
picks Advanced mode).

## Testing

- `make test`: all pass (backend, plus frontend 140 suites / 1729 tests)
- New: 2 backend endpoint tests, 6 hook tests, 3 modal tests, 1
Playwright regression test (loads without stems → waiting state →
chooser appears)
- Local CodeRabbit: 1 finding (false positive: `advancedModeClean`
exists in all 33 locales)

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01QKJ7TxzFoUeNTug1PR8Cwh

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.276.0
2026-10-16 00:21:35 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 090bbf9ada feat(tenant-theme): CD+G instrumental/closing previews + transitions; per-tab previews, wider editor (v0.275.0) (#1166)
Follow-up to #1164, from Andrew's review of the theme editor.

## CD+G tab
- **All four CD+G screens previewed** from one real CDG: up card, lyrics
page, **instrumental break** (the sample lyrics include a gap ≥ a
preview threshold; the fully drawn frame is found by colour) and
**closing screen**. ~1.3 s.
- **"How it's drawn in"** for the up card, instrumental break and
closing screen. These are the "mask" images: cdgmaker's built-in
greyscale draw-order masks (`cdgmaker/transitions/*.png`, e.g.
`centertexttoplogobottomtext`) that decide the order tiles are revealed.
They change the animation, not the finished frame. Validated server-side
against the built-ins; the editor GET returns `cdg_transitions`.
- **Instrumental gap threshold** slider (3–120 s; stored in
centiseconds, validated 300–12000).

## Theme & style modal
1. ~96vw wide (max 1800px); the preview column is sticky.
2. **Each tab previews only its own screen** (Title card → title card,
Lyrics video → lyrics frame, End screen → end screen, CD+G → the 4 CD+G
screens, Advanced → all video screens). The server renders only what's
asked (new preview `kinds`), so previews are faster too.
3. CD+G previews in a compact 2×2 grid.

## Tests
- Backend: instrumental + closing frames render (distinct colours),
one-line sample still shows the break, `kinds` limits rendering,
transitions validated (incl. non-string), threshold bounds, editor lists
transitions, route `kinds` + 422 on unknown kinds.
- Frontend: per-tab previews/requests, Advanced shows all, modal width,
4 CD+G previews in a 2-col grid, transition picks saved, threshold
seconds↔centiseconds, unset placeholder. Full jest 1720 pass; backend
suite 6126 pass; package suite 2054 pass.
- 32 locales translated. CodeRabbit: 4 minor findings fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Gp5fBmsmWJxgkEHb4JfhXR

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.275.0
2026-10-15 16:01:56 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 c277bc8ee3 feat: local E2E stack (fake workers + test API) + email-code sign-in (v0.275.0) (#1165)
## Summary
- **Local stack:** `scripts/local-stack.sh up --frontend` runs the
**real** backend on the Firestore + GCS emulators with `FAKE_WORKERS=1`.
In-process fake workers move jobs through their real states in seconds,
using tiny real media, so web Playwright and mobile Maestro full-journey
tests can run offline against the same endpoints.
- **Email-code sign-in** (nomad-mobile spec §4.1): `POST
/api/users/auth/send-code` + `verify-code`. The code logic is shared
with kjbox via `LoginCodeStore`, and sends are throttled per email and
per IP.
- **Bug fix found by the new suite:** `confirm-duration` moved jobs to
`separating_stage1`, from which they can never reach review. It now
resumes at `downloading`. No prod job had taken this path yet.

## Changes
- `backend/services/fake_workers/`: dispatch seam in
`WorkerService.trigger_*`, plus fake search, download, checkout and
outbox, scenarios, and per-user config. The real stem upload, transcode,
backing-vocals analysis and preview render are reused. Every check is
strict (`is True`), so a MagicMock settings object never enables the
fakes, and startup **refuses FAKE_WORKERS in production**.
- `/api/test/*` router (mounted only in that mode): `status`,
`scenario`, `fake-config`, `emails`, `reset`, plus the fake Stripe
checkout page. Paying it runs the real webhook handler.
- `frontend/e2e/local-stack/` (9 specs, ~50s):
  - sign-up → make → review → instrumental → download
  - buy and cancel credits
  - duration confirm, failed, complete and review states
  - code sign-in and lockout
- New required CI job `local-stack-e2e`.
- Docs: TESTING.md § Local stack, DEVELOPMENT.md, API.md, README,
LESSONS-LEARNED.
- `MAX_SIGNUPS_PER_IP` can now be set by env. Only the local stack
raises it; production keeps the default of 2.

## Testing
- [x] Backend unit: 6175 passed
- [x] Emulator: 152 passed
- [x] Frontend Jest: 1706 passed
- [x] Local-stack Playwright: 9/9 passed (CI mode)

## Review
- [x] CodeRabbit CLI review completed locally. 1 minor finding declined:
it suggested swallowing infrastructure errors in send-code as fake
"sent" responses, which would hide real outages from users.

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01AGuxtgoscF6RJGtpjHPg3W

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 16:00:24 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 eeb08578c9 feat(tenant-theme): CD+G tab in the theme editor with real CD+G previews (v0.274.0) (#1164)
## Why
Commercial producers (Vocal Star first) care a lot about their CD+G
output, but the tenant theme editor only covered the MP4 title card /
lyrics / end screen. CD+G styling needed an admin to hand-edit the theme
JSON.

## What
**New "CD+G" tab** (only on portals with `features.enable_cdg` and a
`cdg` theme section):
- **Up card**: image, title/artist colours, title/artist sizes,
separator text between them (e.g. "MADE FAMOUS BY") + its colour, text
position from top (keeps text below a logo baked into the image),
title–artist gap.
- **Lyrics**: sung / unsung / screen / border colours, size, outline
width + colour, lyrics font ("Same as main font" or a distinct one, e.g.
condensed Oswald).
- **Instrumental breaks**: image, text, colour. **Closing screen**:
image, line colours.

**Real previews**: on the CD+G tab the preview request sets
`include_cdg`; the backend composes a short CDG with the real
`CDGGenerator` from the draft's `cdg` section and renders the up card +
a part-sung lyrics page with the new `karaoke_gen/.../cdg_decoder.py`
(single pass, multi-frame; moved out of the CDG test). 300×216 upscaled
3× with hard pixels, ~1 s. Other tabs never pay for it.

**Images**: uploads from the CD+G tab are fitted (letterboxed) to
300×216 PNG (`purpose=cdg`); the CD+G dropdowns only offer 300×216
images (`cdg_images` from the editor GET, header-only reads);
save/preview reject other sizes since the CDG renderer never resizes.
All 4 existing themes' CD+G images verified 300×216.

**Validation**: `cdg` hex colours, numeric bounds, single-line text
limits; distinct CDG font resolved for the preview the same way renders
do (`cdg_font` asset), title/outro use the main font.

## Tests
- Backend: CDG validation/fit/size-check/upload/editor list
(tenant_theme_service), real CD+G preview renders incl. colour change +
distinct font + missing asset + timeout (theme_preview_service), route
include_cdg + upload purpose. CDG title-card tests now use the shared
decoder.
- Frontend: new `TenantThemeEditorCdg.test.tsx` (10): tab gating,
includeCdg only on that tab, edits map to `cdg`, outline applies to both
strokes, font follow/distinct, fitted upload, CD+G-sized image list,
pixelated 300/216 previews. Full jest 1716 pass.
- Backend suite: all pass except `test_emulator_integration.py`, which
fails identically on main locally (a stray local Firestore emulator
enables it); CI runs emulator tests separately.
- 32 locales translated. CodeRabbit: 2 minor findings fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Gp5fBmsmWJxgkEHb4JfhXR

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.274.0
2026-10-15 15:02:24 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 35c0af444f fix(rerender): recover expired input audio before rendering (v0.273.6) (#1163)
## Problem
Re-rendering Vocal Star job `5fca18dd` (created 2026-08-25) failed in
the encoder:
`404 ... uploads/5fca18dd/audio/S1106-1 Ruby Murray - Softly Softly
Guide.mp3: No such object`.
Jobs created before v0.268.2 kept their uploaded mix and bring-your-own
instrumental only under `uploads/`, which had a 7-day lifecycle delete.
Every pre-October tenant job would fail the same way on re-render (incl.
the theme editor's "Re-render all").

## Fix
New `backend/services/input_recovery.recover_missing_inputs`, called by
the render worker (both GCE + local paths, off the event loop) and the
video worker, right after `ensure_job_inputs_persisted`:
- **Mix** (`input_media_gcs_path`) missing → repoint to the persisted
`jobs/{id}/input/` copy; else reuse a previous recovery; else rebuild
from the job's with-vocals render (`finals/with_vocals_mp4.mp4`,
`videos/with_vocals.*`): its audio track is the original mix (AAC).
Countdown padding (from `state_data.lyrics_metadata` /
`processing_metadata.rendering`) is trimmed; decoded to
`jobs/{id}/input/recovered_mix.flac` (input/ is keep-forever). Tries the
next video if one can't be decoded.
- **Instrumental** (`existing_instrumental_gcs_path`) missing →
persisted copy, else the staged `existing_instrumental.*` /
`custom_instrumental.*` at the job root or in `stems/` (kept by storage
retention).
- Nothing left → the job fails with an actionable message instead of a
raw 404. Recovery is recorded in `state_data.input_recovery`.

Verified on the real job's files: `with_vocals_mp4.mp4` audio = 148.69s,
same as `custom_instrumental.mp3`, same 0.61s leading silence (no
padding on this job).

## Tests
- `backend/tests/test_input_recovery.py` (17): no-op, persisted copy,
real ffmpeg rebuild, countdown trim, padding read from all known
locations, reuse of previous recovery, fall-through on an unreadable
video, actionable errors, instrumental repoint (4 locations), gs://
paths.
- Render-worker tests whose storage mocks report every blob missing stub
the recovery (autouse fixture).
- Full backend suite: 6083 passed.
- CodeRabbit: 3 findings fixed (padding location, to_thread,
fall-through);

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Gp5fBmsmWJxgkEHb4JfhXR

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.273.6
2026-10-15 14:35:52 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 23a0355363 fix(tenant): keep the tenant site title after Next.js rewrites <title> (v0.273.5) (#1162)
Follow-up to #1161. Prod check after #1161 showed the edge now serves
`<title>Vocal Star Karaoke Generator</title>` and the logo flash is
gone, but Next.js metadata rewrites the title on hydration and route
changes ("Nomad Karaoke: Generator", "Karaoke Generator"), overriding
both the edge-injected title and `applyTenantBranding`.

- `TenantProvider` pins `branding.site_title` on tenant portals
(MutationObserver on `<head>`); the default site is untouched.
- Tests: title restored after a simulated metadata rewrite; default
site's title left alone. Also unmount before resetting the store in
`afterEach` (an old mounted provider was refetching and polluting the
next test).
- CodeRabbit: one finding (my version bump had touched dev-dependency
pins) fixed; fresh re-review clean.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Gp5fBmsmWJxgkEHb4JfhXR

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.273.5
2026-10-15 13:28:55 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 c883173ca3 fix(tenant): no Nomad logo flash on tenant portals (v0.273.4) (#1161)
## Problem
Loading https://vocalstar.nomadkaraoke.com briefly showed the Nomad logo
(and on `/en/`, the whole consumer landing page) before the Vocal Star
branding appeared. The static export is prerendered with default
branding; the tenant swap only happens after hydration (measured: Nomad
logo in DOM at ~2ms, tenant logo at ~111ms, longer on cold loads). The
tab title also stayed "Nomad Karaoke: Generator".

## Fix
- Edge Pages Function (already knows the tenant) now also:
- sets `<html data-tenant-pending>` + `html[data-tenant-pending]
body{visibility:hidden}`, with a 4s safety reveal if JS never runs
  - preloads the tenant logo
  - replaces `<title>` with the tenant's `site_title`
- `TenantProvider` removes the marker once tenant config is initialised
(parent effect → runs after the tenant UI has committed; also on
config-fetch failure).
- Injection logic moved from `functions/[[path]].ts` into
`lib/edge/tenantHtmlInjection.ts` so it is unit-tested. Non-tenant hosts
are untouched.

## Testing
- New: `lib/__tests__/tenant-html-injection.test.ts` (hide/reveal script
executed in jsdom, timeout reveal, preload, title escaping, XSS
escaping, gs:// logo mapping, non-tenant passthrough) and
`components/__tests__/tenant-provider-reveal.test.tsx` (page is revealed
only after tenant branding rendered; revealed on fetch failure).
- Full frontend jest: 138 suites / 1704 tests pass. CodeRabbit local
review: no findings.
- Post-deploy: verify on vocalstar.nomadkaraoke.com with a Playwright
img/visibility timeline.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Gp5fBmsmWJxgkEHb4JfhXR

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.273.4
2026-10-15 13:03:57 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 0c1aa16469 Turn on conversion emails + fix doubled signature on credit-denied email (v0.273.3) (#1160)
## Summary
- **Conversion emails go live.** Andrew approved the copy, the 25%
offer, the timings and the 612 Joan St postal address. Adds
`CONVERSION_EMAILS_ENABLED=true,REVIEW_LAST_CHANCE_ENABLED=true,CONVERSION_OFFER_PERCENT=25`
to the backend `--set-env-vars` in ci.yml (feature shipped dark in
#1156).
- **Doubled signature fix.** The "About your Nomad Karaoke free credits"
(credit-denied) email appended `_get_email_signature()` itself and also
got it from the shared footer, so it showed twice. I audited all 22
`send_*` emails by rendering each one; this was the only duplicate.
- New regression test: renders every `EmailService.send_*` and asserts
the signature appears exactly once. It fails on main with
`send_credit_denied_email: 2`.

## Testing
- `make test` green (backend 2054 + 6071 + 150, frontend 1692).
- Local CodeRabbit review: no findings.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_019KuzHAU4rxkv8JfD8NmHTN

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.273.3
2026-10-15 12:44:43 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 182b4eace4 fix(tenant): second Dropbox pass keeps the tenant track-code prefix (v0.273.2) (#1159)
## Summary
Found by verifying a real Vocal Star portal job in prod after #1158 (job
`0976cc34`, code `E2E4635623`). The delivery folder had **both**
`E2E4635623 - Artist - Title (…)` and plain `Artist - Title (…)` copies
of every file.

The cause: `video_worker._handle_native_distribution` re-uploads the
folder after the orchestrator (adding `stems/` and `lyrics/`, with
`skip_unchanged`) without the tenant filename prefix. It now passes
`filename_prefix=tenant_track_code_prefix(job)`, so both passes write
the same names. The unchanged-skip then also matches, so files aren't
uploaded twice.

## Testing
- New test: the post-orchestrator pass passes the tenant prefix.
- Backend: 6069 passed (emulator tests excluded locally).
- Local CodeRabbit with `--fresh`: no findings.
- Will re-verify with a real prod job after deploy by listing the
Dropbox folder.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Gp5fBmsmWJxgkEHb4JfhXR

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.273.2
2026-10-15 02:53:21 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 621077474e feat(tenant): track code prefixes every delivered file + CDG title shrinks unbreakable words (v0.273.1) (#1158)
## Summary
Follow-up to #1153 and #1157, found by running a real Vocal Star job
through the portal in prod (job `8dc6945f`, E2E with track code
`E2E1531717`, which passed).

1. **Track code on every file.** #1153 put the tenant's code on the
Dropbox **folder** (`VS6588 - Artist - Title/`), but the files inside
were still `Artist - Title (…)`. Matt's April 2025 ask was the VS code
at the start of **each file**, and Andrew had confirmed "all files
prefixed with the track code".
- Tenant jobs with their own code now get `VS6588 - Artist - Title (…)`
on every Dropbox file.
- The entries inside the CDG/TXT zips are renamed too, so each CDG+MP3
pair still matches.
- Nomad's own `keep_brand_code` flows (finalise-only, re-render) are
unchanged, because this requires `tenant_id`.
2. **CDG title card:** a single word wider than the screen can't wrap,
so the card now shrinks the font instead of clipping it. Seen on the
first real prod render, where the E2E title has a 24-character timestamp
token.

## Changes
- `dropbox_service.upload_folder(filename_prefix=…)`.
- `prefix_zip_entries()` streams entries and keeps their
timestamps/compression, so a repack is deterministic.
  - `skip_unchanged` also works for repacked zips.
- `tenant_track_code_prefix(job)` is set into
`OrchestratorConfig.output_filename_prefix`. Tenant-only, and the code
is sanitized again before use.
- cdgmaker composer: width check in the title-card shrink loop.
- Version 0.273.0 → 0.273.1.

## Testing
- Prefixed upload, including zip entries and subfolders.
- No-prefix path unchanged.
- Deterministic, streamed repack.
- Tenant-only prefix, with sanitization.
- Real-render test for the unbreakable word.
- Package unit tests: 2054 passed. Backend: 6066 passed (emulator tests
excluded locally).
- CodeRabbit: 3 cycles, all findings fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Gp5fBmsmWJxgkEHb4JfhXR

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.273.1
2026-10-15 01:52:35 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 bfc8f7fb0f feat(cdg): configurable title card + per-theme CDG lyrics font (Vocal Star up card) (v0.273.0) (#1157)
## Summary
Vocal Star (and other commercial producers) need CDG files in their own
house style. Matt's 2025-02-20 email showed our CDG up card overflowing
its bordered design. He asked for his simpler card instead: logo on top,
then title, a small "MADE FAMOUS BY" line, then the artist.

Two pipeline limits blocked that:
- the CDG title card was hardcoded;
- a theme could only have **one font** for everything. The shared `font`
asset (`intro.font`) overwrites `karaoke.font_path` and `cdg.font_path`
at render time.

## Changes
- **Title card (cdgmaker composer):** new optional keys; all defaults
reproduce today's output.
- `title_separator_text` / `_color` / `_font_size`: a line between title
and artist.
  - `title_font_size`, `artist_font_size`, `title_max_height`.
- `title_block_top`: centre the block below a logo baked into the
background.
- It shrinks the title first, then the artist, so it can't loop forever.
- **Outro:** respects `title_block_top` and shrinks long text. Before,
an `assert` crashed when the text was taller than 200px.
- **Fonts:**
- The CDG title card and outro use the theme's main font
(`cdg.title_font_path`, fed by the shared `font` asset).
- A theme may keep a **distinct CDG lyrics font** via a new `cdg_font`
asset. The theme service emits it only when it differs from
`intro.font`, and `update_asset_paths` applies the shared font first so
section overrides win.
  - No visible change for existing themes, which all use one font.
- **Tenant theme editor:** keeps a distinct, available CDG font on save,
and copies it into the theme's assets if it's bundled.
- Docs: API.md.

## Testing
- New `tests/unit/lyrics_transcriber/output/test_cdg_title_card.py`
renders **real CDG files**, decodes the packets back into frames and
checks:
  - the separator sits between title and artist;
  - text stays below `title_block_top` and inside the safe area;
  - the title font is applied, and a missing one falls back;
  - long outro text shrinks instead of crashing.
- New tests for asset-mapping order, `cdg_font` emission, theme-editor
font handling, and the updated mapping assertion.
- Package unit tests: 2053 passed. Backend: 6056 passed (emulator tests
excluded locally).
- Local CodeRabbit with `--fresh`: no findings.
- Rendered Vocal Star samples (short, wrapped-long and very long
title/artist), decoded and visually checked against Matt's design.

After deploy, the Vocal Star theme gets: a new up-card background (logo
cropped from Matt's own card), separator "MADE FAMOUS BY",
`title_block_top`, black border, Avenir as the main font and Oswald kept
for CDG lyrics.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Gp5fBmsmWJxgkEHb4JfhXR

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.273.0
2026-10-15 00:58:42 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 5047f39da2 feat(tenant): per-track catalogue codes prefix tenant outputs (Vocal Star VS codes) (v0.272.0) (#1153)
## Summary
Vocal Star (Matt, April 2025) asked for their own VS catalogue code at
the start of every output file, with all files in one folder (e.g.
`VS6588 - Gerry and the Pacemakers - Ferry Cross The Mersey`). Tenant
operators can now set a per-track **Track code**. It becomes the job's
preserved brand code (`keep_brand_code`, which both render paths already
honour), so the Dropbox folder and every delivered file use it instead
of an allocated `VSTAR-NNNN`.

## Changes
- **Backend:** `POST /api/jobs/create-with-upload-urls` accepts
`track_code`.
  - Tenant portals only; `400` elsewhere.
  - Strict filesystem-safe validation, upper-cased.
  - Maps to `JobCreate.keep_brand_code`.
- **Bulk analyze:** a leading code in filenames (`VS6588 Artist - Title
Guide.mp3`) is split off and returned as `track_code`.
  - Files pair only when code **and** song match.
- LLM pairs with conflicting codes are rejected, and the code is
stripped from LLM-extracted artists.
- **Frontend:** an optional Track code field on the tenant single-track
form and on each bulk row.
  - Client validation mirrors the backend, with a visible error.
  - Resumed bulk rows keep their code, locked.
  - 33 locales translated.
- Docs: API.md. Version 0.271.0 → 0.272.0.

## Testing
- Backend: new tests for endpoint mapping, rejection outside tenants,
unsafe codes, normalisation, filename parsing, the band-name false
positive (UB40), same-code/different-song and LLM conflicts.
- Full backend suite: 5985 passed. The only 30 failures are emulator
tests, because the local Firestore emulator wasn't running.
- Frontend: full Jest 1680/1680 passing, plus new TenantJobFlow /
TenantBulkFlow tests. `tsc` shows no new errors.
- Local CodeRabbit: 3 cycles, all real findings fixed. The one remaining
note is on a pre-existing string.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Gp5fBmsmWJxgkEHb4JfhXR

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.272.0
2026-10-15 00:05:57 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 4ad946c3aa feat: free-to-paid conversion emails + last-chance review nudge, OFF until copy approved (v0.271.0) (#1156)
## Summary
This is BACKLOG Frog #2 ("plug the gen revenue leaks"), conversion-email
track. 44% of signups finish a free video, but only 6% ever pay, and
buyers decide fast (median about 5h). 45 users opened lyrics review and
let the job expire.

**Ships dark.** Nothing sends until Andrew approves the copy and
`CONVERSION_EMAILS_ENABLED` / `REVIEW_LAST_CHANCE_ENABLED` are added to
the ci.yml deploy env.

| Email | When | Stream |
|---|---|---|
| E1 "Your karaoke video is done. What's next?" + 25% off any pack for
48h | 2h after a never-paid user's **first** video completes | broadcast
|
| E2 "Your 25% off ends in 8 hours" | last 8h of the offer, if unpaid |
broadcast |
| R2 "6 hours left to finish your karaoke video" | 42h into review (the
existing reminder is at 24h; expiry is 48h) | transactional |

- `services/conversion_emails.py`:
- A `JobManager` hook marks the user on their first unpaid completion.
Tenant, Made For You and test accounts are excluded.
- The hourly `process-stale-reviews` cron sends the emails. It's
idempotent per user, and due times more than 24h stale are skipped, so
turning it on never emails a backlog. No infra change is needed.
- **Promo:** E1 grants `promo_discount_percent` /
`promo_discount_expires_at`. Checkout applies coupon
`conversion-offer-{pct}pct` when it beats a referral discount.
- The buy dialog shows a banner, and its prices use the effective
discount (CodeRabbit finding).
  - The email button links to `/app?buy=1`, which opens the dialog.
- **Compliance:** E1 and E2 go on Postmark's `broadcast` stream, where
Postmark manages unsubscribes (verified: it appends the link). They
include a postal-address footer (`COMMERCIAL_EMAIL_POSTAL_ADDRESS`).
`EmailProvider` gains `message_stream`, and email_log records the
stream.
- Copy is in `backend/translations` (33 locales). Docs:
`docs/CONVERSION-EMAILS.md`, covering how it works, measurement and how
to turn it on.

## Testing
- `make test`: backend 6017, emulator 150, frontend Jest 1688, package
2044. All green. Re-ran the targeted suites after rebasing onto the tip
PR.
- New tests:
- `test_conversion_emails.py` (26): eligibility, E1/E2 windows,
staleness, promo vs referral, broadcast stream, rendering.
- Stale-review R2 tests (3), `BuyCreditsDialog.test.tsx` (3), and an
AuthStatus `?buy=1` test.
- Rendered E1 and R2 HTML and checked them visually.
- Local CodeRabbit: 1 finding fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_019KuzHAU4rxkv8JfD8NmHTN

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.271.0
2026-10-14 23:50:59 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 542ad66b65 feat: opt-in 'tip the creator' add-on at gen checkout (v0.270.0) (#1155)
## Summary
This is BACKLOG Frog #2 ("plug the gen revenue leaks"), tip track. Tips
were **27% of old order-form revenue** ($950 on 178 of 231 orders), and
gen had nothing like them.

- Both gen checkouts (credit packs and Made For You) now offer an
**opt-in** Stripe Checkout `optional_items` tip: "Add to your order →
Tip the creator (optional) $5 [+ Add]".
- It's never pre-added. The old order form put a $5 tip line in the cart
at quantity 1 (opt-out), which is banned for UK/EU consumers.
- **Price:** $1/unit, found or created by
`lookup_key=gen_checkout_tip_usd_1`. It defaults to 5 units and is
adjustable from 1 to 100.
- Live product `prod_VRY9FhfNhiS0NN` already exists, with a description
explaining "each unit is $1".
- Any Stripe error means no tip is offered, so checkout is never
blocked. Kill switch: `CHECKOUT_TIP_ENABLED=false`.
- **Accounting:** credits are untouched. `stripe_payments.tip_cents`
records the tip separately.
- Referral earnings exclude tips. If the tip can't be read from Stripe,
the base is capped at the package list price (CodeRabbit finding).
- Docs: STRIPE-SETUP.md has a new "Checkout Tip" section.

## Testing
- Verified against **live Stripe**: created a real session (since
expired, nothing charged), and the hosted page shows the tip as an
opt-in "Add" item with the total unchanged.
- `make test`: backend 5977, emulator 150, frontend Jest 1676, package
2044. All green.
- New `backend/tests/test_checkout_tip.py`: price lookup and creation,
kill switch, optional item on both checkouts, opt-in only, tip_cents,
referral base.
- Local CodeRabbit: 1 finding fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_019KuzHAU4rxkv8JfD8NmHTN

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.270.0
2026-10-14 23:39:06 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 ff9b716435 feat: first-touch UTM/referrer signup attribution + UTM on YouTube description (v0.269.0) (#1154)
## Summary
This is BACKLOG Frog #2 ("plug the gen revenue leaks"), attribution
track. Until now we stored no acquisition source, so we couldn't tell
which channel brings the people who pay.

- **Capture (frontend):** `lib/attribution.ts` stores the first-touch
`utm_*` params, the external referrer host and the landing path for 30
days. It runs in the `[locale]` layout and in `LocaleRedirect`, before
`location.replace` wipes `document.referrer`. The data is sent with
`POST /auth/magic-link`.
- **Store (backend):** `services/signup_attribution.py` whitelists,
length-caps and sanitises the data. On account creation it sets
`signup_attribution` and `signup_source`: the `utm_source`, or a
referrer channel label such as `youtube`, `google`, `website`, `decide`
or `kjbox`. Malformed input never blocks sign-in, and the request field
is bounded (16 keys, values up to 500 chars).
- **Report:** `GET /api/users/admin/attribution-summary?days=90` returns
signups, verified, made a job, paid and revenue, per source. The admin
user detail shows the signup source and attribution.
- **YouTube:** description template links now carry
`utm_source=youtube&utm_medium=description&utm_campaign=requests|make_cta`.
They're static, so they're still derivable from the title, and the daily
backfill rolls them out. The `requests.nomadkaraoke.com` Cloudflare
redirect now has `preserve_query_string` (Pulumi `nomad-cloudflare`,
**already applied** locally).
- **Docs:** STRIPE-SETUP.md gets a "Legacy Payment Links" section. The
old order-form link `dR67sx…` was **deactivated 2026-10-14** and points
people to gen; it had 0 completed checkouts in about 30 sessions since
Dec. Also updated API.md.

Companion: nomadkaraoke/public-website#34 makes
`nomadkaraoke.com/r/CODE` pass the UTM params through.

## Testing
- `make test`: backend 5986 passed, emulator 150, frontend Jest 1683,
package 2044. All green.
- New tests: `backend/tests/test_signup_attribution.py` (cleaning,
channel labels, magic-link wiring, bounded payload, summary endpoint)
and `frontend/lib/__tests__/attribution.test.ts`.
- Local CodeRabbit: 3 findings fixed (malformed referrer, NaN expiry,
unbounded payload); re-review found 0.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_019KuzHAU4rxkv8JfD8NmHTN

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.269.0
2026-10-14 23:31:16 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 559d159145 infra(cloudflare): route fiverrbot /approve to its approval server (#1152)
## Summary
- Adds a tunnel ingress rule on `nomad-hf2`:
`fiverrbot.nomadkaraoke.com` path `^/approve` → `http://fiverrbot:8088`
(fiverrbot's new delivery approval page), ahead of the existing noVNC
rule.
- The existing FiverrBot Access app (Andrew-only) covers the whole
hostname, so the page is behind Access.

## Applied
`pulumi up --target` on `tunnel-config-nomad-hf2` only (2026-10-14).
`pulumi preview` also shows an **unrelated pending diff** on the
nomadkaraoke.com dynamic redirect ruleset ("Requests Voting Board
Redirect", `preserveQueryString`). That diff was left untouched.

Verified: unauthenticated `GET /approve/x` and `/vnc.html` both return
302 to `nomadkaraoke.cloudflareaccess.com`.

Companion: nomadkaraoke/fiverrbot delivery phase (v0.4.0).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_017rMyzhBpH5tgh9WDgGxg4n

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added routing for approval-related requests to the Fiverr bot service.
  * Existing traffic continues to use the default bot service route.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 23:29:30 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 3ece8baa51 docs: Cloudflare account move + Pulumi adoption lessons (#1151)
Docs-only follow-up to #1147, #1148 and #1150:
- `docs/ARCHITECTURE.md`: the Edge Security section now points to the
`nomad-cloudflare` project in `infrastructure/cloudflare/`, which
manages the rest of the account.
- `docs/LESSONS-LEARNED.md`: new section on the zero-downtime Registrar
account move and on re-pointing and adopting Pulumi stacks, plus
pulumi-cloudflare 6.22.1 gotchas.

Local CodeRabbit review: 0 findings.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_014XpBykm3FJmoFTjuNQ18Bi

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 21:02:28 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 acbdb326eb feat(infra): Nomad Karaoke Cloudflare account as a Pulumi project (#1150)
## Summary
Adopts the Nomad Karaoke Cloudflare account (`4585249d…`) into IaC as a
second Pulumi project, `infrastructure/cloudflare/` (project
`nomad-cloudflare`, stack `nomadkaraoke/prod`). It sits beside the GCP
platform project, and `infrastructure/` is now documented as the
**shared platform IaC home**. App repos keep only app-specific stacks.

**130 resources, all adopted with `pulumi import`, with no recreation:**
- 5 zones (protected) + `ssl` mode
- 84 DNS records
- 8 zone rulesets
- Email Routing: address, enablement (protected), catch-alls, rules
- `nomad-kjbox` / `nomad-hf2` tunnels (protected) + ingress
- Zero Trust: org (protected), OTP IdP, service token, **3 reusable
policies**, 2 apps
- Pages projects + `gen` domain, Worker custom domains + interceptor
route

**Single ownership, explicitly excluded (see
`infrastructure/cloudflare/README.md`):**
- gen `edge_security` phases + `api`/`api-edge-test` DNS
- runtime tenant domains
- decide's route/Worker
- read-only records
- karaokehunt's `http_config_settings` (pulumi-cloudflare 6.22.1 drops
`set_config bic:false`, so refresh fails)

**Applied already (repo rule: apply locally before merge):**
- Final `pulumi preview` shows **131 unchanged, 0 changes**.
- The only live mutation was moving the two Access apps from app-scoped
to reusable code-managed policies, with the same rules and 730h
sessions. Verified afterwards: anonymous → 302 to
`nomadkaraoke.cloudflareaccess.com`, and the kjbox service token → 200.
- The post-import "updates" with no property diff were one-time state
settlements. Each was confirmed as a no-op against a live before/after
snapshot.

## Testing
- `infrastructure/cloudflare/tests/test_build_import_list.py`: 4 passed
(ownership exclusions, pagination, unique names).
- `pulumi preview`: 131 unchanged.
- Live spot-check of site, gen, api, decide, sing, singa, kjbox.cc,
kjtip.me, karaokehunt and nomadkaroke.
- Local CodeRabbit: cycle 1 found 1 issue (pagination), fixed; cycle 2
found 0.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_014XpBykm3FJmoFTjuNQ18Bi

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 00:43:57 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 a404526ff4 docs: Cloudflare references point at the Nomad Karaoke account (#1149)
Docs-only follow-up to #1147 / #1148 (code already moved; no version
bump).

- `infrastructure/cloudflare/karaokehunt-interceptor/README.md` +
`worker.js` comment: karaokehunt.com zone is now
`a9136845466472d41ea82a156059ad9b` in the Nomad Karaoke account
(kill-switch curl updated).
- `docs/DEVELOPMENT.md`: manual Pages deploy needs the Nomad account
(`4585249d…`) creds; notes `karaoke-gen-9k4.pages.dev` /
`karaoke-gen-tenant-x0h.pages.dev` and that CI secrets point at the new
account.
- `docs/ARCHITECTURE.md`: edge-security zone/account and tenant CNAME
target (`karaoke-gen-tenant-x0h.pages.dev`, legacy target still
recognised).

Local CodeRabbit review: completed, 0 findings.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_014XpBykm3FJmoFTjuNQ18Bi

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 23:29:00 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 ece56b31a8 chore(cloudflare): tenant domains + karaokehunt interceptor on the Nomad Karaoke Cloudflare account (v0.268.11) (#1147)
## Summary
Nomad Karaoke moved out of Andrew's personal Cloudflare account into a
dedicated **Nomad Karaoke** account (`4585249dcf5d2f9fb4e075c5382cd071`)
on 2026-10-14. `nomadkaraoke.com` is now active there (new zone
`2324a342622e93343cf4e08227d82c02`).

- `tenant_domain_service`: new account/zone defaults. The tenant CNAME
target is now configurable (`TENANT_PAGES_TARGET`, default
`karaoke-gen-tenant-x0h.pages.dev`), because the recreated Pages
project's `pages.dev` subdomain is globally unique and isn't derivable
from the project name. CNAMEs at the old `karaoke-gen-tenant.pages.dev`
target still count as ours: `provision` repoints them (proxied) and
`deprovision` removes them.
- `karaokehunt-interceptor/deploy.sh`: new karaokehunt.com zone ID.
- Version 0.268.11.

Out of band, already done: GitHub secrets
`CLOUDFLARE_API_TOKEN`/`CLOUDFLARE_ACCOUNT_ID` point at the new account
(Pages-only token). Secret Manager `cloudflare-tenant-domains-token`
gets a new-account token (Pages + nomadkaraoke.com DNS) before merge.

## Testing
- `backend/tests/test_tenant_domain_service.py` +
`test_tenant_admin_service.py`: 76 passed, including new tests for the
legacy-target repoint/deprovision.
- Local CodeRabbit review: 1 finding (keep repointed CNAME proxied),
fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_014XpBykm3FJmoFTjuNQ18Bi

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.268.11
2026-10-14 23:20:22 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 60cb4d5e94 chore(infra): edge Pulumi stack on the Nomad Karaoke Cloudflare account (#1148)
## Summary
Follow-up to #1147. `nomadkaraoke.com` now lives in the Nomad Karaoke
Cloudflare account (zone `2324a342622e93343cf4e08227d82c02`).

- `Pulumi.prod.yaml`: new zone-scoped `cloudflare:apiToken` (DNS, WAF,
settings and transform rules on nomadkaraoke.com only), plus
`edge:cloudflareZoneId` and `edge:cloudflareAccountId`.
- `config.py`: `DEFAULT_ZONE_ID` set to the new zone.

## State (already applied locally, per repo rule)
- The 6 edge resources (`api-dns`, `api-edge-test-dns`,
`edge-waf-custom`, `edge-ratelimit`, `edge-origin-header`,
`zone-security-level`) and the stale cloudflare default provider were
removed from state.
- The resources were then imported from the new zone with `pulumi
import`.
- A targeted preview shows all 6 as `(same)`.
- The only remaining stack diffs are pre-existing divebar function
drift, which is unrelated.

## Testing
- `pulumi preview` (targeted): 6 unchanged.
- Local CodeRabbit review: 0 findings.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_014XpBykm3FJmoFTjuNQ18Bi

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 19:19:07 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 39dcf2598e fix: PyPI prune snippet + force marker only in PR title (v0.268.10) (#1146)
## Summary
Two follow-ups to #1145, both found while doing today's prune.

1. **Prune console snippet silently did nothing.** PyPI's delete-version
form now requires five `acknowledge_*` checkboxes. A submit without them
gets a **200 re-render**, which the old snippet counted as "deleted".
The snippet now reads each release's real form (CSRF plus checkboxes),
submits it, and counts a version deleted only when its manage page 404s.
This is the exact approach used for today's prune (88 releases, 10.73 →
~4.1 GB).
2. **The force marker matched the commit body.** A squash commit's body
is the PR description. #1145's description just *mentioned* the marker,
which forced a publish on merge. That publish ran before the prune, so
it 400'd and uploaded nothing. The marker now only counts in the commit
subject (the PR title).

Docs: the runbook covers the new form, the per-version JSON cache lag,
and a prune log.

## Testing
- Unit tests: 30 pass across both scripts, with new tests for
checkbox/404 handling and for a marker in the body not forcing a publish
- `node --check` on the generated snippet
- Replayed #1145's real merge message through the gate: no publish
- Local CodeRabbit review: 0 findings
- Full `make test` not re-run (no app code touched); PR CI covers the
package suites

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01MSW9yjwGnzz1S8MgCGm6t5

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.268.10
2026-10-14 13:34:57 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 5005d3cb80 ci: publish to PyPI only when CLI code changed (v0.268.9) (#1145)
## Summary
PyPI publishing failed on every deploy since v0.268.4. karaoke-gen hit
PyPI's 10 GB project cap for the third time (June, Sept 29, Oct 12).
Each wheel is ~62 MiB and we published on every merge (~100/month), so
each prune only bought a couple of weeks.

PyPI is only the public `pip install` channel for the CLIs. Cloud Run
builds from source, and the encoding worker pulls the wheel from GCS. So
we now publish only when something CLI users get has changed.

## Changes
- `scripts/pypi_publish_gate.py` (stdlib): diffs
`scripts/pypi_publish_paths.txt` (`karaoke_gen/` plus the `backend/`
modules it imports) and `pyproject.toml` (ignoring the package `version
=` line only) against **the latest version on PyPI**. A skipped or
failed publish therefore never loses a CLI change. Fails open.
- `ci.yml` `package-publish`: new "Decide PyPI publish" step gates
`poetry publish` (now `--skip-existing`). **Tags and GitHub releases
still happen on every version**, because the encoding-worker gate diffs
against tags. Override with `[publish-pypi]` in the PR title/commit, or
the new workflow_dispatch input "Publish to PyPI".
- Not gated on purpose: `frontend/**` (bundled review UI) and
`backend/config.py` (cloud settings). A test keeps the backend import
list in sync, with `config.py` as the one documented exemption.
- Docs: PYPI-STORAGE-PRUNE.md, TROUBLESHOOTING.md, prune script and
reminder workflow comments.

## Impact
Replayed over v0.200 to v0.268: **24 of 136** releases would have
published (~0.7 GB/month instead of ~4 GB/month). Run against current
main, it decides "no CLI-affecting changes since v0.268.3", which is
correct for v0.268.4 to v0.268.8.

## Testing
- 15 new unit tests in `tests/unit/test_pypi_publish_gate.py`
- `make test` all green
- actionlint clean on the new step
- Local CodeRabbit review: 1 finding (doc recovery path), fixed

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01MSW9yjwGnzz1S8MgCGm6t5

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 12:56:24 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 0329d47e38 fix: YouTube description backfill reverted translated-lyrics notices (v0.268.9) (#1144)
## Summary
The daily YouTube description backfill worker was undoing the
translated-lyrics notice on videos with translated lyrics.

**Root cause:** `classify()` parsed `Artist - Title (With Translation
into Spanish) (Karaoke)` with the translation label left inside the song
title, and `render_for()` never passed a `translation_language_name`.
Every translated upload therefore looked "pending". The worker re-checks
the whole channel every day, even after a cycle completes, so it rewrote
each new translated video. The rewrite dropped the `This karaoke video
includes translated lyrics (…)` notice, put the label into `{title}`,
and added it to the tags as well.

## Changes
- `backend/services/youtube_backfill.py`: `classify()` strips the `(With
Translation into X)` label from the parsed song title and records
`translation_language_name`. `render_for()` passes it to
`render_youtube_description`, which is the same thing the upload
pipeline does. Tags now use the clean song title.
- On its next run the worker restores the notice on the translated
videos it already reverted, because their titles still carry the label.
- README: added a safety note that any per-video description variant has
to be recoverable from the live title.
- Version 0.268.9

## Testing
- 5 new tests in `backend/tests/test_youtube_backfill.py`: parsing,
untranslated case, matching upload left alone, reverted description
restored, title preserved + clean tags.
- `make test-backend-unit`: 5961 passed.
- Local CodeRabbit review: no findings.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_013RzGSdPUXLn1nK5eMEZZTW

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 12:32:23 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 7775bb603e fix: send_push blocked the API loop; watchdog samples after a freeze (v0.268.8) (#1143)
## Summary
Two follow-ups from the stall-alert investigation (#1140).

**1. `send_push` blocked the event loop.** Once #1140 was live, the
watchdog's new `requests_affected` field showed a real stall:
`push_notification_service.py:166 in send_push`, 1.2s, with a request
waiting. `send_push` was declared `async def`, but its whole body was
synchronous: a Firestore read, one `webpush` HTTP POST per device, and a
Firestore update. The route AST guard can't see service methods.
- The body is now `send_push_sync`. `send_push` runs it with `await
asyncio.to_thread(...)`, so callers don't change.
- Invalid-subscription cleanup now uses `ArrayRemove(stale)` instead of
rewriting the whole `push_subscriptions` array. With the body in a
thread, it could otherwise overwrite a device that `add_subscription`
registers at the same moment. This came from a CodeRabbit finding.

**2. Watchdog blind spot.** A process freeze could carry a stall past
every 1s/5s/15s sample point without taking a sample. Every ERROR stall
on 10-10 through 10-13 had `culprit=` empty.
- If the loop is still blocked at the first on-time check after the
thaw, the watchdog now takes one sample.
- That sample's log level uses blocked time (frozen time discounted),
matching how the final duration is computed. This also came from a
CodeRabbit finding.

## Testing
- New tests:
- `send_push` runs off the loop thread with its arguments passed
through;
  - the 410 cleanup uses `ArrayRemove` for only the gone entry;
  - a block after a freeze gets a culprit;
  - a pure freeze stays silent;
  - the post-thaw sample is WARNING when the real block is under 10s.
- `make test`: all passed (before the rebase onto #1141/#1142, which
touched only the frontend and the version).
- Local CodeRabbit: 2 findings, both fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01DTfrTaaZQfXUUduAhZpLeq

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 19:05:00 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 e39894d705 fix: edge tenant config fetch threw 'Illegal invocation' (v0.268.7) (#1142)
## Summary
This fixes a regression from #1141, which I found while verifying it in
prod with `wrangler pages deployment tail`. Every tenant portal page
logged:

```
Failed to fetch tenant config: TypeError: Illegal invocation: function called with incorrect `this` reference.
```

`deps.fetchFn(...)` invoked Workers' `fetch` as a method, so `this` was
`deps`, which Workers rejects. As a result, the edge never injected
`__TENANT_CONFIG__`. `TenantProvider`'s client-side fallback still
applied the branding, but with a brief unbranded flash.

## Changes
- `tenantConfigCache.ts`: calls `fetchFn` unbound.
- `[[path]].ts`: passes `(input, init) => fetch(input, init)`.
- Regression test: a `fetchFn` that throws like Workers when `this`
isn't undefined or global. It fails on the old code and passes on the
new code.

## Testing
- Full frontend Jest suite passes.
- `wrangler pages functions build` compiles.
- After deploy, I will verify by tailing the function logs and checking
that `__TENANT_CONFIG__` is present in the singa HTML.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01DTfrTaaZQfXUUduAhZpLeq

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 18:49:51 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 ccf6da020f fix: tenant portal edge config stampede spawned cold API instances (v0.268.6) (#1141)
## Summary
This follows up #1140. While investigating the stall-alert flapping, I
found that 18 of the 28 requests that were the first on a new autoscaled
instance (each ≥10s) over 4 days came from the tenant portal's Pages
Function. Each one was `/api/tenant/config?tenant=…` with no user agent.

At 03:55 UTC, something loaded 17 `singa.nomadkaraoke.com` pages at
once. All 17 missed the 5-minute edge cache in the same second and
called the API. Cloud Run spawned 5 cold instances, and those HTML
responses waited 15–17s.

## Changes
New `frontend/lib/edge/tenantConfigCache.ts`, used by
`functions/[[path]].ts`:
- **Stale-while-revalidate:**
  - Under 5 minutes old, the cached config is served as is.
- Older than that, the stale copy is still served and refreshed in the
background via `waitUntil`. Copies are stored for 7 days.
- Entries cached before this deploy have no timestamp, so they are
treated as stale.
- **Bounded cold miss:** the backend fetch has a 3s
`AbortSignal.timeout`. On timeout or error the HTML is served without
injection, and `TenantProvider` falls back to fetching the config
client-side, as it already did.
- **Single flight:** concurrent misses in one isolate share one fetch.
- **Payload shape check:** a malformed 200 response isn't cached for a
week. This came from a CodeRabbit finding.

Other changes:
- `jest.setup.js` guards the `window.matchMedia` mock so
`@jest-environment node` tests can run. It's a no-op for jsdom tests.
- `docs/ARCHITECTURE.md` has a new section on the edge config cache.

**Trade-off:** a branding change now reaches each colo on the first page
view after 5 minutes, and that view still shows the old branding.
Before, it showed up on the first view after expiry.

## Testing
- 10 new Jest tests (node environment):
  - cold miss;
  - fresh hit;
  - stale hit with background refresh;
  - a 17-request burst makes one fetch;
  - 503 and timeout return null;
  - a failed refresh keeps the stale copy;
  - an entry without a timestamp is treated as stale;
  - a malformed payload isn't cached.
- `wrangler pages functions build` compiles the bundle.
- `make test`: all passed.
- Local CodeRabbit review: 1 finding, fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01DTfrTaaZQfXUUduAhZpLeq

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 18:29:45 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 3566fd136e fix: loop-stall alert flapped on idle autoscaled instances (v0.268.5) (#1140)
## Summary
The `>=2 event-loop stalls of 10s+ in 30 minutes` alert flapped on
2026-10-12 and 2026-10-13. Investigation:
- All 11 ERROR-level `EVENT_LOOP_STALL ended` lines in the last 4 days
(10-40s each) were on instances with **zero requests** in or near the
stall window. The check joined `labels.instanceId` between the stall
lines and the request logs.
- None of them had a culprit. A process freeze advances the 1s/5s/15s
sample schedule without taking a sample.
- **Pattern:** Cloud Run autoscaling starts an extra instance, it serves
one request (e.g. `/api/tenant/config`), then it sits idle and
CPU-throttled. The startup warmup thread competes for CPU slices
meanwhile (SpaCy preload took 690s on one instance). The watchdog reads
that as a loop stall.

## Changes
- `RequestTracker` + pure-ASGI `RequestTrackingMiddleware` (outermost)
count in-flight and started requests.
- `LoopWatchdog(request_tracker=...)`: a stall logs at ERROR, the level
the alert counts, only when a request was in flight at its start or
arrived during it. The verdict waits 0.5s after the loop resumes so
requests queued in the socket are counted. The same rule applies to `in
progress` samples, so they don't show up in the error monitor either.
- Idle stalls still log at WARNING with `requests_affected=0` and are
still recorded to `client_events` (`detail.requests_affected`).
- Without a tracker, behaviour is unchanged.
- Docs: TROUBLESHOOTING, LESSONS-LEARNED, API.

No infra change: the alert metric already filters on `severity>=ERROR`.

## Testing
- 10 new unit tests:
  - idle → WARNING;
  - in-flight request → ERROR;
  - request queued during the stall and accepted after resume → ERROR;
  - request finished mid-stall → still ERROR;
  - in-progress sample levels;
  - no-tracker behaviour unchanged;
  - middleware counting, including exceptions and non-http scopes;
  - middleware is installed outermost.
- `make test`: all passed.
- Local CodeRabbit review: 2 findings, both fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01DTfrTaaZQfXUUduAhZpLeq

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 12:38:01 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 fef212823d fix: non-Latin songs all matched each other in community-version check (v0.268.4) (#1139)
## Summary
Searching **Луна - Золотые Лепестки** showed "A karaoke version of this
song already exists!" with 166 unrelated community versions (e.g. *从灵乐队
- 永活盼望*).

**Root cause:** `normalize_for_match` filtered with `[^0-9a-z\s]`,
erasing every non-Latin character. Every Cyrillic/CJK/Arabic/Hebrew/etc.
artist+title normalized to an empty key, so they all collided in the
community index (one `'\x1f'` bucket held 166 tracks). The same
normalizer feeds request-board dedup, KaraokeHunt dedup and the match
judge.

## Changes
- `classifier.normalize_for_match`: keep letters, digits, whitespace and
combining marks of every script; diacritics are folded only on
Latin/Greek/Cyrillic bases (`Beyoncé`→`beyonce` still works, but kana
dakuten and Indic vowel signs survive).
- `karaokenerds_service`: rows whose artist/title normalize to empty
aren't indexed; empty-key lookups never match.
- Lesson added to `docs/LESSONS-LEARNED.md`; version 0.268.4.

## Testing
- Regression tests for Cyrillic/CJK non-collision, self-match,
Indic/kana marks, punctuation-only input (failed before the fix).
- Validated against the real community export: index 59,190 → 59,376
distinct songs, empty bucket gone, Луна lookup → no match.
- Firestore `song_requests` / `karaokehunt_requests` have no docs with
the collided `"|"` dedupe key → no data migration needed.
- `make test` green (backend 5942 + emulator 150 + frontend 1665). Local
CodeRabbit review: 2 findings fixed (Indic vowel signs, doc wording).

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01WVzkWBjPawKbTjpsrR4JAH

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 18:00:09 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 77bec3327c fix: encoding-worker boot loop (poisoned gcloud cache) + stale file_urls after change-to-public (v0.268.3) (#1138)
## Summary
Two bugs, both hit by job `82f48738` (a private → public visibility
change) on 2026-10-12.

### 1. Encoding worker VMs boot into an unrecoverable restart loop
If gcloud's first call (the API key fetch in `startup.sh`) happens
before DHCP, gcloud caches "not on GCE" in `/root/.config/gcloud/gce`.
Every later gcloud call in that boot then has no credentials, so the API
key comes back empty, the v0.266.4 self-heal refuses to serve, and
systemd re-runs `startup.sh` against the same poisoned cache until the
VM idle-stops. worker-b was up 20:30–20:42 and never answered /health,
so the render failed with `Cannot connect to host 34.10.189.118:8080`.
The same thing happened on 10-09 (`f4310aea`): it was fixed by hand and
the durable fix was noted but never shipped.

- `startup.sh`: `rm -f /root/.config/gcloud/gce`, wait up to 60s for the
metadata server, retry the secret fetch 6× (clearing the cache each
time). CI uploads this script on every deploy and bootstrap downloads it
on every service start, so existing VM disks pick it up.
- Packer `startup-fallback.sh`: clears the cache too (new images).

### 2. Retry after a failed change-to-public 404s forever
`change_to_public` deletes screens, `videos/with_vocals.mkv` and the
finals but left their `file_urls`. `/retry` then saw
`videos.with_vocals` + `instrumental_selection`, skipped the render and
404'd on the deleted MKV. It now drops those refs, as
`ThemeRerenderService` already does.

## Testing
- New `backend/tests/test_encoding_worker_startup_script.py` and
`test_change_visibility.py::test_drops_file_urls_for_deleted_artifacts`
- `pytest backend/tests/test_encoding_worker_startup_script.py
backend/tests/test_encoding_worker_self_heal.py
backend/tests/test_encoding_worker_code_paths.py
backend/tests/test_change_visibility.py`: all pass

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01JUFVxMZME6ydzkmLvyuXRo

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* When a private job is made public, references to deleted screen and
video files are cleared before processing resumes, preventing stale
files from appearing.
* Improved encoding worker startup reliability by retrying API-key
retrieval when the metadata service or secret store is temporarily
unavailable.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

@coderabbitai ignore

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.268.3
2026-10-12 17:29:43 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 983f6ecdcd fix(storage): keep user-uploaded inputs forever; uploads/ expires after 180d not 7d (v0.268.2) (#1137)
## Summary
`uploads/` had a bucket lifecycle **delete after 7 days**, and user
inputs were only partly copied out of it. The only copy step was the
audio worker (stem separation) copying the mix to `jobs/{id}/input/`,
best-effort. Everything else stayed in `uploads/`:
- bring-your-own-instrumental / tenant jobs, which skip separation;
- the uploaded instrumental, which was only staged at render time;
- lyrics and style uploads, which were never copied.

Result:
- randy-vild re-renders 404'd on their own input (2026-10-08). That data
was restored from Andrew's Dropbox.
- Firestore shows 64 completed Jan–Feb 2026 consumer jobs, plus
BYO-instrumental jobs, whose inputs are gone permanently.

## Changes
- **`backend/services/input_persistence.py`** (new):
- Server-side copies `uploads/{id}/**` to `jobs/{id}/input/**` and
repoints every job field that referenced the upload:
`input_media_gcs_path`, `existing_instrumental_gcs_path`,
`lyrics_file_gcs_path`, `style_params_gcs_path`, plus any matching value
under `file_urls` / `state_data` / `style_assets`.
  - Idempotent: unchanged copies are skipped.
- The audio, lyrics, screens, render-video and video workers all call it
on entry, via `asyncio.to_thread`. On failure it logs an ERROR, which
alerts, but doesn't fail the job.
- **`POST /api/admin/persist-uploads`** (dry run by default): backfill
for jobs no worker will touch again. Lookup failures are isolated per
job.
- **GCE encoder:** prefers the explicitly downloaded top-level
`existing_instrumental.*`. The recursive glob could otherwise return the
raw, unconformed copy now kept under `input/audio/`.
- **Pulumi:** `uploads/` lifecycle goes from 7 to **180 days**; `temp/`
stays at 7. **Already applied** and verified directly on the bucket.
- **Docs:** LESSONS-LEARNED, DISASTER-RECOVERY (new user-inputs row; the
stale "30d noncurrent" figure corrected to 7d), API, README.

## Verification
- `make test` all green.
- 16 new tests:
- copy mapping, repointing, idempotency, re-copy when a kept copy
differs, already-expired paths left alone;
  - the worker wrapper never raises;
- an AST guard that every pipeline worker calls the copy step via
`to_thread`;
  - backfill dry run / apply / error isolation;
  - the encoder's instrumental preference.
- Local CodeRabbit: 2 findings, both fixed (lyrics path, per-job
lookup); re-review clean.

## After merge
Run `POST /api/admin/persist-uploads` (dry run, then `?dry_run=false`)
to copy the inputs of the 51 jobs that still have objects in `uploads/`.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 17:33:06 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 60be7606a3 fix: stop double Dropbox upload, ignore partial-throttle stalls, pre-import lazy route modules (v0.268.2) (#1136)
## Summary
Follow-ups from #1135.

1. **Every job was uploaded to Dropbox twice.**
`generate_video_orchestrated` runs the orchestrator's Dropbox upload.
`_handle_native_distribution` then builds `stems/` and `lyrics/` and
re-uploaded the **whole** folder (job 8a10a22d: ~53s + ~70s, ~400 MB
twice). Neither pass can simply go:
   - the orchestrator's `dropbox_link` is the one written to the job;
- building the subfolders deletes instrumentals the orchestrator still
needs.
2. **Short (1-2.5s) EVENT_LOOP_STALLs in asyncio internals.** I compared
24h of stalls with each instance's request logs:
- 126 of 128 had no request waiting on them. These are partial CPU
throttling: the watchdog thread gets an occasional time slice, so its
wake gap stayed under the 2s freeze threshold from #1135.
- The apparent overlaps are requests that started one watchdog tick
(~0.26s) before the logged end and finished in ~4ms. They're the
requests that brought the CPU back.
- The only user-felt ones were 3 cold-start `importlib` stalls (3s, 4s,
15s): routes lazily importing modules that pull in `httpx`.

## Changes
- `DropboxService.upload_folder(..., skip_unchanged=True)` lists the
destination once (recursively) and skips files whose Dropbox
`content_hash` (SHA-256 of 4 MiB block hashes) matches. The native pass
uses it, so only new or changed files are sent. Listing errors fall back
to a full upload. It's off by default, so the orchestrator's upload and
other callers are unchanged.
- Watchdog `frozen_gap_s` is now 0.75s (3 missed ticks), down from 2s.
Measured watchdog wake gaps stay ≤0.27s with the loop blocked on sync
I/O, Python CPU work or C-extension CPU work, so real stalls are still
reported.
- Lifespan startup imports `_LAZY_ROUTE_IMPORTS` (httpx + 6 services)
via `asyncio.to_thread` before readiness, bounded at 30s with an ERROR
log on timeout.
- Docs: `LESSONS-LEARNED.md`: the double-upload entry; how to tell a
harmless stall class from a user-felt one.

## Testing
- Backend unit suite: **5908 passed**, 36 skipped.
- New tests:
- `content_hash` matches Dropbox's algorithm (multi-block and empty
files);
- skip-unchanged handles identical, changed and new files, plus
pagination, a missing folder, a listing failure and the off-by-default
case;
  - the native pass uses `skip_unchanged`;
- a short partial-throttle freeze is ignored, while I/O and CPU stalls
are still reported;
- lazy imports run in a worker thread before readiness, failures are
non-fatal, and the timeout doesn't block startup.
- Local CodeRabbit: 3 cycles. Fixed the startup-import race, the
stall-impact criterion in the doc, and the unbounded startup wait.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.268.2
2026-10-08 17:26:53 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 f2612ac110 fix: close edit→delete refund loop, retry lost YouTube uploads, unblock event loop (v0.268.1) (#1135)
## Summary
Investigation of two 2026-10-08 Discord alerts (plus a third that
arrived mid-session) turned up four issues:

1. **Credit-refund loop.** A brand-new welcome-credit account received a
finished video (job 8a10a22d), clicked **Edit** (status back to
`awaiting_review`), then **deleted** the job and got its credit back.
`delete_job` refunds any non-complete status, and Edit makes a delivered
job look "never completed".
2. **YouTube 410 "Gone" dropped the upload.** The resumable session was
lost 1.3s into the upload. `upload_video` called `execute()` once with
no retries, and the orchestrator only queued quota errors, so any other
failure lost the upload.
3. **EVENT_LOOP_STALL 21s in `upload_json`.** `complete_review` (an
async route) wrote the corrections JSON to GCS directly on the event
loop. Ranking a week of watchdog stacks by route showed the same pattern
in `recover_stuck_jobs` (~260 stalls/week), the YouTube queue processor
(its `_process_single_upload` ran a whole upload on the loop),
`create_job_from_search` (theme prep), `_validate_audio_durations`
(15s), and the sync `run_job` / generation bump inside worker triggers.
4. **False 30s "stall" in asyncio `get_debug`.** Cloud Run CPU
throttling froze the whole instance mid-callback with no request in
flight. The watchdog only discounted freezes where the loop was parked
in `select()`.

## Changes
- **Refund:** `_refund_credit_for_job` skips jobs with `edit_count > 0`
(already delivered). This covers delete, cancel, fail and the 48h review
auto-expiry.
- **YouTube:** on 404/410 the upload restarts with a fresh session (3
attempts, with backoff). `execute(num_retries=3)` lets googleapiclient
resume chunks on 5xx/429 within a session. Any other failure is queued
in `youtube_upload_queue` (`reason="upload_error"`) for the existing
processor, which retries up to 5 times. Admins still get a Discord
warning: "queued for automatic retry".
- **Event loop:**
- New `backend/utils/loop_bridge.py`: `run_in_body_thread` runs sync
route bodies on a dedicated 16-thread pool, so they can't exhaust the
default executor the dispatched coroutines need. `run_on_loop`
dispatches coroutines back to the API loop (`run_coroutine_threadsafe`,
120s timeout).
- `complete_review`: its sync save runs via `to_thread`, and the render
trigger stays on the loop.
- `create_job_from_search`, `recover_stuck_jobs` and
`retry_pending_render_jobs`: bodies run in the route-body pool.
`_retrigger_parked_download` now does its writes in that thread.
- `_validate_audio_durations` and the YouTube queue's
`_process_single_upload` keep their async signatures but run in threads.
The queue's quota and Firestore calls go through `to_thread`.
- `worker_service`: the generation bump and `client.run_job` go through
`to_thread`.
- **Watchdog:** if the watchdog thread itself oversleeps by 2s or more
while the process uses less than 25% CPU, the whole process was frozen.
That time is subtracted from the stall and isn't sampled. Blocking I/O
(the watchdog still wakes on time) and GIL-hogging CPU work (CPU keeps
advancing) are still reported.
- Docs: `API.md` (edit/refund note) and `LESSONS-LEARNED.md`
(mostly-sync routes that await one trigger; throttling freezes).

## Testing
- Backend unit suite: **5863 passed**, 36 skipped (`pytest backend/tests
--ignore=backend/tests/emulator`).
- New tests:
- edited-track refund cases (delete in awaiting_review /
lyrics_complete, cancel);
  - YouTube restart on 410, give up after 3 attempts, no restart on 403;
  - orchestrator queues failed uploads and warns when queueing fails;
  - loop-responsiveness ticker tests for each offloaded path;
  - `run_on_loop` result, exception and timeout-with-cancel;
  - `run_in_body_thread` copies context and uses its own pool;
- watchdog: freeze ignored, I/O stall and CPU hog still reported, freeze
subtracted from a real stall.
- Local CodeRabbit: 3 cycles. Fixed the nested-executor deadlock risk,
the missing `run_on_loop` timeout, the parked-download writes on the
loop, and a coroutine leak. The last cycle had 0 findings.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.268.1
2026-10-08 16:15:47 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 60687d6cd6 feat(theme): editable + previewed end screen; optional title/end screens (v0.268.0) (#1134)
## Summary
Randy Vild (tenant `randy-vild`) emailed: *"I have completed 3 songs but
all three songs produce the large text at the end even though I have in
my settings smaller text like in my beginning."*

Root cause: the theme editor's size/position sliders only edit
`intro.*`. The end screen has its own `end.*` layout (title 350 /
closing message 400 / yellow capitals artist in Randy's theme). The
editor neither exposed it nor previewed it ("The end screen isn't
previewed").

## Changes
- **End screen tab**: own title/artist colours, size + position sliders
(closing message, song title, artist), show/hide song title / artist
name, capitals toggles, and **Copy title card layout**. That copies the
title card's regions/colours/capitals and moves the closing message just
above the title, or below the artist if there's no room, so they never
overlap.
- **End screen preview**: `theme_preview_service.render_end_screen` uses
the same `VideoGenerator.create_end_video` as `screens_worker`. The
preview column shows the screen for the active tab plus the lyrics
frame; Advanced shows all three.
- **Leave out the title card / end screen**: optional `intro.enabled` /
`end.enabled` (absent = included; not added to the DEFAULT_*
required-field lists). The setting is honoured end to end:
- The screens worker skips the omitted screen and records
`state_data.screens_included`.
- The video worker's prerequisite check and the `/retry` ladder
(`_has_title_screen`, which also requires `screens_progress: complete`)
treat an omitted screen as done.
- The orchestrator sets the omitted screen's path to `None` and gives
the original-vocals guide an intro offset of 0.
- Encoding passes explicit `include_title_screen` /
`include_end_screen=false` to the GCE worker. Without them, its loose
`*title*.png` / `*end*.png` globs would rebuild the screen from
`style/*_background.png`.
- `LocalEncodingService.encode_lossless_mp4` builds the concat from
whichever segments exist (1–3).
  - The portrait renderer drops omitted cards.
- 11 new strings × 33 locales (`endScreenHint` removed). Docs: API.md,
README.md.

## Verification
- `make test` all green: frontend 1665 tests; backend unit, integration
and emulator suites.
- New `backend/tests/test_theme_screen_omission.py`, 28 tests:
  - screens worker for all 4 include combinations;
  - prerequisites and the retry helper;
  - orchestrator config and encoder options;
  - GCE `run_encoding` with decoy theme PNGs present;
  - concat for n = 1/2/3.
- Real-ffmpeg renders:
- end-screen preview (its own background; text size changes only the end
screen);
  - portrait durations with screens omitted.
- Frontend: editor tests for the end tab, copy layout, region toggles
and enable switches, plus unit tests for the `copyTitleCardLayoutToEnd`
util.
- Rendered Randy's real theme locally:
  - as saved, the end screen reproduces the large text;
  - after Copy title card layout, it matches his title card.
- Local CodeRabbit review: 0 findings.

## Deploy notes
- The worker paths changed (`gce_encoding/`,
`local_encoding_service.py`, `karaoke_gen/`), so CI blue-green redeploys
the encoding worker.
- Existing themes are unaffected: they have no `enabled` key.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.268.0
2026-10-08 15:09:25 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 437e72a29b fix(backend): move blocking credential checks + GDrive validation off the event loop (v0.267.1) (#1133)
## Summary
Two recurring `EVENT_LOOP_STALL`s alerted on 2026-10-07:
- **16s** in `mark_uploads_complete` → `check_dropbox_credentials()` →
expired-token refresh (HTTP + Secret Manager `add_secret_version` +
version cleanup), all on the event loop. Also seen 10-06 14:03, 10-07
05:49/06:00 via `create_job_with_upload_urls`.
- **5-10s** after every job completion in
`trigger_gdrive_validation_endpoint` (sync OIDC token fetch +
`requests.post(timeout=60)`). Seen 7× since 10-04.

These routes `await request.json()` etc., so the v0.264.0 "async route
without await" guard didn't catch them.

## Changes
- `file_upload.py` (`upload_and_create_job`,
`create_job_with_upload_urls`, `mark_uploads_complete`,
`create_job_from_url`) and `audio_search.py` (`search_audio`): 18
`check_*_credentials()` calls → `await asyncio.to_thread(...)`.
Plain-`def` routes were already in the threadpool and are unchanged.
- `internal.py`: `trigger_gdrive_validation` → `asyncio.to_thread`.
- `test_no_blocking_async_routes.py`: new `BLOCKING_CALLS` denylist
guard (credential checks, `trigger_gdrive_validation`, `requests.*`,
`time.sleep`) for direct calls inside `async def` routes, plus a
detector unit test. Run against `origin/main` it flags exactly the 19
sites fixed here.
- LESSONS-LEARNED entry; version 0.267.1.

## Testing
- `make test`: backend 5838 passed, 2021 passed (second backend set),
emulator 150 passed, frontend 1654 passed.
- Local CodeRabbit review: no findings.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.267.1
2026-10-07 21:25:56 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 9d04274de7 feat(tenant): theme edits reach in-progress tracks + outdated-track re-render prompts (v0.267.0) (#1132)
## Why
Randy Vild edited his portal theme and his next video (RVILD-0003) still
had the old title card. Each job copies the theme when it's created.
This one was created 30 minutes before his edit and was waiting for
encoding capacity when he saved. Edits only reached jobs created
afterwards, finished tracks gave no sign they were outdated, and admins
saw two identical "Re-render" buttons on portal tracks.

## What
- **Theme save → in-progress tracks**
(`services/theme_change_service.py`): every tenant track that hasn't
started rendering gets a fresh copy of the theme. Tracks whose screens
may already exist (`generating_screens`…`in_review`,
`render_pending_capacity`) are flagged `state_data.theme_screens_stale`.
- **Render-worker divert:** a flagged job in `review_complete` goes once
through the screens worker (`LYRICS_COMPLETE` +
`regen_restore_status=review_complete`, the same bounce the theme
re-render uses). The screens worker regenerates the screens and
re-triggers the render. Doing this at render time covers every path into
render (review submit, auto-approve, retry, capacity retry) and never
interrupts someone mid-review. Stale `finals/*.mov` files are deleted
too, so the encoder can't splice the old intro back in.
- **Edit of a finished tenant track** picks up the current theme when
the track's copy is outdated.
- **New endpoints:** `GET /api/tenant/theme/outdated-jobs`, plus `POST
/api/tenant/theme/rerender-outdated`, which is quiet (no per-track
emails), capped at 20 per call, and keeps its blocking I/O off the event
loop. The save response now includes `refreshed_jobs` and
`outdated_job_ids`. Outdated means the theme's `style_params.json` was
written after the job's `theme_applied_at` (or `created_at` if that's
unset).
- **UI:** after a save, the theme editor says how many in-progress
tracks were updated and offers **Re-render all N**. Outdated job cards
get an "Older theme" badge on Re-render. On portal cards, admins see
only the portal Re-render, not the duplicate admin one.
- i18n: 12 new strings, translated into all 33 locales; 3 superseded
keys removed.

## Testing
- `make test` passes (backend unit, emulator and e2e; frontend Jest and
Playwright).
- New `backend/tests/test_theme_change.py` (36 tests, 98% coverage of
the new service), plus a render-worker divert test, editor dialog tests
and OutputLinks tests.
- Local CodeRabbit review: 1 major finding (bulk re-render blocking the
event loop / unbounded), fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.267.0
2026-10-06 18:32:27 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 0ea2a23766 fix(encoding): worker self-heals after a bad boot; default metadata mTLS off (v0.266.4) (#1131)
## Summary

On 2026-10-06 the primary encoding VM (`encoding-worker-b`) booted
before its network was up. `bootstrap.sh` couldn't download `startup.sh`
from GCS (`Network is unreachable`) and fell back to the image's
`startup-fallback.sh`. That script writes an env file without
`ENCODING_API_KEY` or `GCE_METADATA_MTLS_MODE=none`, so google-auth used
mTLS to the metadata server and every render and preview failed with
`CERTIFICATE_VERIFY_FAILED`. The backend deliberately keeps retrying the
primary on infra errors, so a Randy Vild render (job `ebfe4344`) looped
in `render_pending_capacity` for an hour until the service was restarted
by hand.

## Changes

- **Worker defaults `GCE_METADATA_MTLS_MODE=none`** in
`backend/services/gce_encoding/main.py` before any google import, so
every boot path gets it. Ships with the wheel; no image rebuild needed.
- **Worker self-heal** (`backend/services/gce_encoding/self_heal.py`),
active only on worker VMs (systemd `INVOCATION_ID` **and**
`/opt/encoding-worker/bootstrap.sh`):
- Boot: refuses to start if the API key is missing or it can't get GCP
credentials. uvicorn exits, and systemd re-runs the bootstrap.
- Runtime: after a job fails with an infra/auth error, it waits 30s (so
the backend sees the failure), re-probes credentials, and exits once
idle if they're still failing.
- **Packer bootstrap** (`provision.sh`) retries the `startup.sh`
download for about 2 minutes. The fallback writes
`GCE_METADATA_MTLS_MODE=none` and exits if it can't fetch the API key,
instead of starting unauthenticated. Takes effect with the next image
build.
- **Alerting:** an infra failure while the primary is serving now logs
at ERROR (was WARNING).
- **UI:** `state_data.render_pending_capacity` is cleared once a retried
render reports progress, so the job card stops saying "(waiting for
encoding server availability)" during a healthy render.
- Docs: TROUBLESHOOTING runbook, LESSONS-LEARNED, README.

## Testing

- New: `test_encoding_worker_self_heal.py` (probe/boot/restart logic,
`main.py` wiring, mTLS default) and
`test_render_capacity_marker_clear.py`.
- `make test`: 5799 backend unit, 150 emulator, 1646 frontend, all
passing.
- Local CodeRabbit review: no findings.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.266.4
2026-10-06 11:55:51 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 950c96fa48 feat(e2e): checkout-only (no-charge) payment mode for daily E2E Stage 1 (v0.266.3) (#1130)
There's no working test card right now (the issuer declines it), so the
daily E2E Stage 1 defaults to a **no-charge** mode until the repo
variable `E2E_REAL_PAYMENT=true` is set.

## Checkout-only mode (`E2E_PAYMENT_MODE=checkout-only`)
- **Unchanged:** real signup via the `e2etest95` referral link, the Buy
Credits dialog, and real Stripe Checkout session creation.
- **Card:** fills the form with Stripe's public test card `4242…`
(`NO_CHARGE_CARD`; no secret needed). The self-healing steps still run,
except `submitPayment`.
- **Checks:** the $0.50 referral price and the Pay button are visible,
then it **stops before Pay**. Live mode always declines that test card,
so even an accidental click can't charge anything.
- **Stage 2:** gets the new account's session token and spends its
signup credit. The test asserts the account has at least 1 credit.
- **Not exercised:** the Stripe payment, the webhook credit grant, the
purchase email, and self-heal PRs (no verified purchase to learn from).
- **Email:** the daily email marks Stage 1 `[checkout-only: no real
payment]`.

## Testing
- Offline suite: 31 passed. The new test checks that no-charge mode
fills the form with the test card, never calls confirm and never leaves
Checkout.
- `--list` loads the production spec in both modes. Lint and typecheck
are clean.
- A live E2E Daily run on this branch (37262437261) is in progress
against prod.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.266.3
2026-10-05 00:28:56 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 57b319a4b4 fix(e2e): make credit-purchase test.use top-level so the spec loads (v0.266.2) (#1129)
Follow-up to #1128. In `credit-purchase-real.spec.ts`, `test.use({
trace, video, screenshot })` was inside a `describe` block. Those
options force a new worker, so Playwright refuses to load the spec. The
first post-merge E2E Daily run (37256162735) failed Stage 1 in about 1s,
before any test ran. This moves the call to the top level of the file.

Verified: `CI=1 npx playwright test
--config=playwright.production.config.ts --list` loads all 78 production
tests, including the credit-purchase test.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.266.2
2026-10-04 22:45:46 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 feef49a588 fix(e2e): stop publishing card data/admin token in artifacts; fail fast on card decline (v0.266.1) (#1128)
## Summary
E2E Daily **#200** failed in Stage 1. The cause was **the issuer
declining the test card**: Stripe `confirm` returned 402 `card_error /
card_declined / generic_decline` on all 3 attempts. It was not a Stripe
Checkout change. The self-healing runner still treated it as one: a 90s
verify wait, then 6 Gemini calls "re-entering" a form that was already
correct.

The investigation also found that **every run's artifacts published
secrets** in this public repo:
- the full test card (number, expiry, CVC) in screenshots, videos,
`error-context.md` and the DOM dump;
- `X-Admin-Token` in the trace network logs.

All 140 unexpired `e2e-*` artifacts have already been deleted via the
API. The admin token and the test card still need rotating (Andrew).

## Changes
- **Fail fast on decline:** `stripe-checkout.ts` watches
`/v1/payment_pages/<id>/confirm`. A `card_error` aborts `submitPayment`
immediately with `STRIPE_CARD_DECLINED: …`: no LLM calls, no second Pay
click.
- New `CheckoutStep.abortReason`, honoured by the runner, `pollVerify`
and the LLM loop.
  - Non-card errors (e.g. a missing field) still go to the LLM fallback.
- **Alerts say why:** Stage 1 exposes a `failure_reason` output ("Test
card declined, not a checkout bug (…)"). It appears in Discord and in
the daily email. The Discord payload is now built with `jq` from env, so
Stripe's text never reaches the shell.
- **No secrets in artifacts:**
- The Stripe helper's screenshots mask every input, and the DOM dump
redacts input values.
- The credit-purchase spec turns off trace, video and Playwright's
automatic screenshots.
- `playwright.production.config.ts` turns traces off on CI
(`E2E_TRACE=on` overrides).
  - New `frontend/e2e/scripts/scrub-artifacts.mjs`:
- deletes `*.zip` traces and `e2e-session-token.txt` (and videos with
`--drop-videos`);
    - redacts secret env values and card-number-shaped digit runs;
- with `--blank-inputs`, blanks `<input value>` and ARIA textbox values;
    - deletes any file that still contains a secret.
- Every production-E2E upload (`e2e-daily` ×2, `e2e-tenant-daily`, the
`ci.yml` happy path) now runs the scrubber first and **uploads only if
the scrub succeeded**.
- Docs: `TESTING.md` (decline behaviour, a "E2E Artifacts Are Public"
section), `LESSONS-LEARNED.md`.

## Testing
- `npx playwright test --config=playwright.selfheal.config.ts`: **30
passed**. New tests:
- a declined card fails in ~7s with 0 planner calls and exactly 1
confirm request;
  - the DOM dump contains no typed card values;
  - card-error parsing and the confirm-URL matcher;
- scrubber units (literal/grouped/PAN redaction, short secrets only
blanked inside inputs, deletion rules).
- Ran the scrubber against a copy of the real #200 artifacts: no card
data remains, and traces and videos were removed.
- actionlint: no new findings (only shellcheck info notes; ci.yml:996 is
pre-existing).
- Local CodeRabbit review: 2 findings (shell interpolation of the
reason; extension allowlist in the scrubber), both fixed.
- The full `make test` suite was not run: the change is limited to E2E
helpers, workflows and docs, with no backend or app code.

## Follow-ups (Andrew)
- Rotate the `admin-tokens` secret and the `E2E_ADMIN_TOKEN` repo
secret.
- Replace or unlock the test card; ideally a low-limit virtual card.
Then update the `E2E_STRIPE_CARD_*` secrets and re-run E2E Daily.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.266.1
2026-10-04 22:25:52 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 3ecea6f2e7 feat(storage): enable real purge with deletion audit log and orphan cleanup mode (v0.266.0) (#1127)
## Summary
Andrew approved this follow-up to #1125 and #1126. It turns on real
deletion, adds an audit log of every deleted object, and adds an
orphan-folder cleanup mode.

### Real deletion on
`STORAGE_RETENTION_DRY_RUN=false` and
`STORAGE_RETENTION_MAX_JOBS_PER_RUN=50` are added to the backend
deploy's `--set-env-vars`. The API service is the only thing that reads
them: it passes `--dry-run` to `storage-retention-job`. That keeps the
setting in one place, so a later deploy can't silently revert it.

### Deletion audit log
- `storage-retention/deletion-logs/<ts>-<mode>.jsonl` has one line per
deleted or failed object: timestamp, run_id, mode, job_id, path,
size_bytes, generation, category and result.
- The log is rewritten after every job or folder, so a crash still
leaves a complete record. Each run also writes
`<ts>-<mode>.summary.json`.
- Deletes are pinned to the logged generation (`if_generation_match`).
- `assert_deletable` refuses any path outside `jobs/{id}/`, so
`storage-retention/` is never touched. No lifecycle rule matches live
objects there.
- The Firestore manifest on each job doc now also records generations
and the log path.

### Orphan cleanup
- `POST /api/internal/storage-retention/orphans` runs on the same Cloud
Run job with `--mode orphans`.
- It targets `jobs/{id}/` folders with no doc in `jobs`, `jobs-dev` or
`youtube_upload_queue`, re-checked live per folder against a fresh
listing.
- It skips any folder with an object modified in the last 30 days, keeps
`input/`, and deletes the rest.
- `encoding_worker_jobs` doesn't count as a record: it's the encoder's
expiring result cache, which deleted jobs leave behind.
- It is a dry-run unless called with
`dry_run=false&confirm=delete-orphan-job-folders`, and the scheduler
never runs it.

### Docs
- `docs/STORAGE-RETENTION.md`: run modes, log locations, jq review
commands and restore one-liners (noncurrent `cp #gen` and soft-delete
`restore`).
- `scripts/storage_retention_report.py`: list runs, summarise a run, or
find which run deleted a path or job.

## Testing
New tests cover the log contents, generation-pinned deletes, per-job
flushing across a crash, the path guard, orphan planning (input kept,
30-day skip, live record re-check, batch limit), and the endpoint and
worker modes. Full backend suite: 5774 passed.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added admin-triggered orphan-folder cleanup, with dry-run enabled by
default, optional folder limits, and explicit confirmation required for
deletion.
* Added deletion reports and audit logs covering removed objects,
reclaimed space, and failures, plus a read-only tool to review runs and
search records.
* **Bug Fixes**
* Added safeguards to prevent deleting changed objects or files outside
their intended job folders.
* **Documentation**
* Expanded storage-retention guidance with recovery details, review
commands, and cleanup instructions.
* **Chores**
* Set the production cleanup limit to 50 jobs per run and updated the
package version.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
v0.266.0
2026-10-04 11:54:05 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 60e48dbb72 fix(storage): retention pass as a Cloud Run Job and stems-restore screens handoff (v0.265.1) (#1126)
## Summary
This fixes two problems found while verifying #1125 in prod.

1. **The scheduled pass never finished.** It ran as a FastAPI
BackgroundTask, which is starved on the API service:
`cpu-throttling=true` gives it CPU only while a request is open. Running
it synchronously doesn't work either, because Cloudflare cuts requests
at 100s.
- The pass now runs as a new Cloud Run Job, `storage-retention-job`
(us-east4, CPU image `:latest`). Pulumi is already applied (targeted).
   - New CLI: `backend/workers/storage_retention_worker.py`.
- The endpoint starts the job and returns the report path. It returns
503 if the job can't be started.
- The dry-run default is unchanged: the API passes `--dry-run` based on
`STORAGE_RETENTION_DRY_RUN`, which defaults to true.
2. **The regenerate E2E stalled after the stems restore.** The first
screens dispatch had marked `screens_progress=running`, so the restore's
re-trigger was skipped as "already running".
- The stems gate now drops that mark when it hands off, and again on
completion.
- `recover-stuck-jobs` now re-triggers re-render/regenerate jobs that
stay parked at `lyrics_complete` for more than 10 minutes.

## Testing
- New tests cover the worker CLI, the job trigger, the endpoint, the
screens-mark handling and the watchdog. Full backend suite: 5754 passed.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Scheduled storage-retention runs now launch as managed jobs, with
options for dry runs, report paths, job limits, and orphan handling.
  * Retention jobs can also run locally outside production.

* **Bug Fixes**
* The retention endpoint reports an error when a job cannot be started.
* Stalled regeneration flows can resume screens processing after a lost
dispatch or stems restore.

* **Documentation**
* Updated the architecture overview to describe the managed
storage-retention job workflow.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
v0.265.1
2026-10-04 01:10:39 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 6c3c70654d feat(storage): retention purge (dry-run) and on-demand regenerate (v0.265.0) (#1125)
## Summary
Implements the approved storage-retention plan (2026-10-03): purge
regenerable files from jobs completed more than 30 days ago, and
regenerate them on demand.

### Purge (`backend/services/storage_retention.py`)
- Daily Cloud Scheduler `storage-retention-daily` calls `POST
/api/internal/storage-retention/run` (Pulumi, already applied).
- **DRY-RUN by default** (`STORAGE_RETENTION_DRY_RUN`, default true). It
writes a report to
`gs://karaoke-gen-storage-nomadkaraoke/storage-retention/reports/<ts>-dry-run.json`.
Real deletion needs the env flipped after a human review.
- Walks completed Firestore jobs. Completion time is the newest
`complete` timeline entry.
- **Keeps:** input (incl. `edited.flac`), lyrics, style, review
sessions, audio_edit, uploads, packages, analysis, screen images, the
720p final (both copies), `stems/custom_instrumental.*`,
`stems/vocals_derived.*`, and the root custom/existing instrumentals.
Unknown files are kept too.
- **Purges:** other finals, `videos/`, `previews/`, `encoded/`,
`quick/*.mp4`, `screens/*.mov`, `review-audio/`, and separated stems
(only when the input is kept in `jobs/{id}/input/`).
- **Skips:**
  - finalise-only jobs
  - jobs whose input audio is gone (66 old jobs)
- jobs with an in-flight edit, re-render, visibility change or
regenerate
  - pending YouTube uploads
  - excluded tenants (env)
- Real runs:
  - batched, with a resumable cursor
  - re-check eligibility atomically via a claim
- record `renders_purged_at`, `stems_purged_at` and a `storage_purge`
manifest
  - drop purged `file_urls` entries
- `?job_ids=…&dry_run=false&min_age_days=0` runs a scoped real pass, for
testing. A global real purge only happens through the setting.

### Regenerate (`backend/services/regenerate_service.py`)
- `POST /api/jobs/{id}/regenerate` (owner or admin, rate-limited per job
and per user) plus a job-page **Regenerate video** button and progress
notice. There is also `POST /api/admin/jobs/{id}/regenerate`.
- GCS-only: claim, then screens, render, encode. It uses the existing
lyrics, instrumental selection and style. No brand-code allocation, no
YouTube/Dropbox/GDrive/Discord. The owner is notified when it's done.
- **Stems gate** in the screens worker (`stems_restore.py`): if
`stems_purged_at` is set, it runs the audio-separation job in restore
mode first, and the audio worker re-triggers screens afterwards. This
covers regenerate, admin/theme re-render, private→public, Edit (which
now restores stems before review) and retry.
- Public→private on a job with archived finals regenerates first, then
redistributes (no more "No finals" failure).
- Archived downloads return **410** `output_archived` with a
`regenerate_url`.

### Fixes
- Encoder: the with-backing and clean instrumental lookups no longer
fall back to another instrumental (that was a silent wrong-instrumental
bug); a missing instrumental now fails loudly.
- `existing_instrumental_gcs_path` is persisted at upload to
`jobs/{id}/custom_instrumental.<ext>`, because `uploads/` expires after
7 days. Older jobs get a resolve-and-repoint fallback in the video
worker and orchestrator.
- Bucket noncurrent-version retention reduced from 30 to 7 days (Pulumi,
applied).

## Testing
- New: `test_storage_retention.py` (108), `test_regenerate_service.py`
(66), `test_instrumental_resolution.py` (16), plus edit-flow, upload and
visibility updates. Full backend unit suite: 5731 passed.
- Frontend: OutputLinks regenerate tests and job-status helper tests
(Jest). All 33 locales translated.
- Local read-only dry-run against prod: 538 jobs eligible, about 401 GiB
purgeable.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added an option to regenerate archived or missing video outputs from a
completed job. Existing publishing links remain unchanged, and job
progress is shown while processing.
* Added automatic restoration of archived audio stems when needed for
regeneration or job edits.
* **Improvements**
* Added scheduled cleanup of eligible files from older completed jobs
while retaining key source and review assets.
* Improved recovery when requested downloads have been archived by
offering a regeneration option.
* Improved handling of saved instrumentals so they remain available for
future processing.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
v0.265.0
2026-10-03 23:59:23 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 33d6ac5d2a fix(backend): build encoding service off the event loop (v0.264.1) (#1124)
## Summary
This is a follow-up to #1120. The new event-loop watchdog's first prod
catch was a 9.9s stall on a fresh instance, culprit
`backend/services/encoding_service.py:1392 in get_encoding_service`. On
first use it imports `google.cloud.compute_v1` (~10s), on the loop, via
`/api/health/detailed`.

- The singleton is now built in the startup warmup thread
(`main._run_background_warmup`).
- `get_encoding_service()` is now thread-safe (double-checked lock), and
it publishes the service only once fully wired.
- The async callers (`check_encoding_worker_status`,
`_run_preview_encode`, `generate_preview_video`) use `await
asyncio.to_thread(get_encoding_service)`.
- The version goes to 0.264.1, which also resolves the 0.264.0 collision
between #1120 and #1122.

## Testing
- Backend unit suite: 5538 passed.
- New tests:
- The health encoding check keeps the loop responsive during a slow
build.
  - The singleton is built exactly once under 5 concurrent threads.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Reliability**
* Health checks and encoding previews remain responsive while the
encoding service initializes.
  * Concurrent requests safely share a single encoding service instance.
* Service initialization now also runs during startup; if it fails, the
service can still initialize when needed.
* **Documentation**
  * Updated the recent changes log and release version to 0.264.1.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 14:59:59 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 98883dea8a fix(audio-search): best pick prefers the requested artist among title matches (v0.264.1) (#1123)
## Summary
Follow-up to #1122, found by checking it against a real prod search for
**Braxton Keith - The Chair**: George Strait's 1985 "The Chair" (Spotify
popularity 68) beat Braxton Keith's (65) on the popularity tiebreak. The
result was a confident "Recommended" pick of the wrong artist.

- `getBestResult(results, title, artist)` /
`getSearchConfidence(results, title, artist)`: among confirmed title
matches, those by the requested artist win.
- `isArtistMatch`: tolerates a leading "The", featured artists ("Braxton
Keith, X") and small typos (edit distance ≤ len/5, only for names of 5+
chars, so "U2" ≠ "UB40").
- A Spotify best pick by another artist is no longer a confident tier-2
match.
- `AudioSourceStep` passes the artist.

## Test plan
- [x] New tests: the real George Strait vs Braxton Keith case,
another-artist-only Spotify → tier 3, `isArtistMatch` cases (89 in the
audio-search suites)
- [x] Checked against the real prod search results: best = Braxton Keith
"The Chair" (Single 2025), tier 2
- [x] Local CodeRabbit: 1 finding (short-name fuzzy match) fixed

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v0.264.1
2026-10-03 14:47:06 -04:00