Commit Graph
205 Commits
Author SHA1 Message Date
Andrew Beveridge ☄️andClaude Sonnet 5.5 286e604b75 feat(worker): voice-pool separate-only GPU job (v0.283.1) (#1217)
## Summary
- Adds `voice-pool-separate-job`, a Cloud Run GPU (L4) job for
karaoke-decide's voice session. decide writes a JSONL manifest
(`voice-pool/manifests/<run>.jsonl`) and starts the job with a per-run
shard; each task separates its share of songs with the
`instrumental_clean` Separator and writes
`voice-pool/stems/<id>/vocals.flac` (skipped if already present).
Results are written back to the manifest area every 10 songs.
- No Firestore or gen job state. Model-load failure or an
all-songs-failed task exits non-zero with a single ERROR; partial
failures exit 0.
- Task timeout is 3600s (Cloud Run's GPU maximum, found by the local
CodeRabbit review), so decide must size the shard count to keep each
task under 1h. Re-runs skip finished songs.

## Changes
- `backend/workers/voice_pool_separate_worker.py` +
`backend/tests/test_voice_pool_separate_worker.py` (9 tests, 91%
coverage of the worker)
- `infrastructure/modules/cloud_run.py`, `infrastructure/__main__.py`:
Pulumi job definition
- `.github/workflows/ci.yml`: CI guard step (tolerates a missing job)
- `docs/ARCHITECTURE.md`: paragraph on the job
- Version 0.283.0 -> 0.283.1

## Deploy notes
**Andrew: run `pulumi up` in gen infrastructure/ BEFORE merging (creates
voice-pool-separate-job; the CI guard tolerates a missing job).**
decide's Pulumi (later PR) will create the voice-pool SA and grant it
`roles/run.jobsExecutorWithOverrides` on this job.

## Testing
- [x] `pytest backend/tests/ --ignore=backend/tests/emulator`: 6691
passed, 40 skipped
- [x] Local CodeRabbit review completed (2 cycles): the critical timeout
finding fixed; the remaining minor (results flush inside `finally`) is a
known deferred item

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_0188txBF47GpTo7hym1FoifX

---------

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-22 18:42:35 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 055a5d3f73 feat(youtube): enable comment auto-reply in prod + one-time backfill (v0.280.2) (#1207)
## Summary

Switches on the YouTube comment auto-reply bot from #1205 in prod
(Andrew authorised go-live after the 6-month replay sign-off):

- `YOUTUBE_COMMENT_AUTOREPLY_ENABLED=true`
- `YOUTUBE_COMMENT_AUTOREPLY_START=2026-04-22T00:00:00Z`. This is used
on the first run only, as a one-time backfill (Andrew: "yes please, will
be a good test of the system working once in prod"). The replay predicts
3 replies: Fightmaster "Nobody's Baby", Bear Ghost "Tiny Hitman", Playa
Fly "N God We Trust". Threads already answered by hand are skipped.

Already verified in prod before this PR:
- The endpoint returns `disabled` while the flag is off.
- A `?dry_run=true` run loads secret `youtube-comment-bot-credentials`,
builds the channel index and reads comments with no errors.
- The Cloud Scheduler job `youtube-comment-autoreply-hourly` and the
secret were applied through Pulumi in #1205.

## Testing

Config-only change; the code and tests are in #1205.


🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_016E3ibN2EcKxnGQvEf6T4V8

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 02:41:15 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 b67b273056 feat(kjbox): decide partner secret for catalog resolve + fake judge + error monitor for decide search (v0.278.10) (#1190)
## ⚠️ Merge prerequisite (do BEFORE merging)

The Secret Manager secret `decide-gen-partner-secret` must have a
version before this merges, otherwise the CI deploy's `--set-secrets
...DECIDE_GEN_PARTNER_SECRET=decide-gen-partner-secret:latest` fails.

```
gcloud auth login admin@nomadkaraoke.com
cd ~/Projects/nomadkaraoke/karaoke-gen-decide-partner-resolve/infrastructure
GOOGLE_OAUTH_ACCESS_TOKEN=$(gcloud auth print-access-token) pulumi up     # review: should only add the decide-gen-partner-secret Secret
openssl rand -hex 32 | tr -d '\n' | gcloud secrets versions add decide-gen-partner-secret --project=nomadkaraoke --data-file=-
```

No extra IAM grant needed: gen's API runs as `karaoke-backend@`, which
already has project-level `secretmanager.secretAccessor`.

## Summary
- `POST /api/kjbox/catalog/resolve` now also accepts karaoke-decide via
header `X-Partner-Secret` (new setting `decide_partner_secret` / env
`DECIDE_GEN_PARTNER_SECRET`). Other kjbox endpoints stay
`X-Kjbox-Secret` only. decide gets its own AI-call budget (60/min) so
kjbox nights are never starved; the verdict cache is shared.
- Fake match judge / resolve for the local E2E stack (`FAKE_WORKERS`):
`judge_kind` / `judge_canonical_*` knobs; no Gemini or catalog calls.
- Error monitor now watches decide's new `karaoke-decide-search` service
and `song-index-build` job.
- Pulumi: declares the `decide-gen-partner-secret` Secret (value added
manually); CI deploy mounts it.
- Version 0.278.10.

## Testing
- [x] Changed-module + kjbox/catalog/error-monitor/fake-worker backend
tests pass (133 passed). Full `backend/tests` suite exceeds 10 min
locally, so left to CI.

## Review
- CodeRabbit CLI failed twice (WebSocket closed / connection error), so
no local CodeRabbit review completed; I did a manual diff review instead
(no issues found). Deliberately NO `@coderabbitai ignore`, so the GitHub
bot reviews this PR.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01VsLw9cTTMBab1tiRxCsYrm

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 21:42:27 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 4286d98dbe fix: LRCLIB as a registered server-side app; honour Retry-After, retry 5xx (v0.279.2) (#1200)
@coderabbitai ignore

## Why
LRCLIB now applies stricter rate limits to unregistered clients
(https://lrclib.net/docs#implementation-requirements-rate-limiting), and
Cloud Run shares egress IPs. On 10-19, prod jobs got intermittent 503s,
and job 80e7ffdf lost its LRCLIB lyrics. LRCLIB is our only free source;
it supplied lyrics for 165 of 220 jobs in 30 days. Andrew registered a
server-side app.

## What
- **`karaoke_gen/lyrics_transcriber/lyrics/lrclib_client.py`** (new):
- Sends `Lrclib-App-Id` + `Lrclib-Api-Key` from `LRCLIB_APP_ID` /
`LRCLIB_SECRET_KEY`.
- Applies LRCLIB's rules: every 429 sets a **process-wide cooldown** of
`Retry-After` plus up to 30% jitter. The wait is never shortened, and
overlapping 429s keep the latest expiry. Nothing is sent until the
cooldown has passed. It waits and retries once only when the wait is
≤30s.
  - Retries 5xx after 1s/2s, honouring a 5xx `Retry-After`.
- A **401 on our credentials** logs a WARNING and drops to anonymous
access for the process, so a revoked key degrades instead of losing
LRCLIB.
- `LRCLIBProvider` (both `get` and `search`) uses it. The
`quick_version` renderer uses it with `retry=False`, so the fast path
gets the auth and the cooldown but never blocks.
- **Infra:**
  - Pulumi secrets `lrclib-app-id` / `lrclib-api-key`.
  - Env vars on the `lyrics-transcription-job`.
- The backend service deploy's `--set-secrets` (`ci.yml`). The
karaoke-backend SA already has project-wide `secretAccessor`.

## Deploy order (do NOT merge before steps 1–2)
1. `pulumi up` to create the two secrets and the job env. Needs
`/gcp-admin`.
2. Add the secret versions from Andrew's `.envrc` values.
3. Merge. The CI backend deploy references the secrets.

## Testing
- New `tests/unit/test_lrclib_client.py` (14 tests): headers; anonymous
use; Retry-After with jitter; a long Retry-After is not shortened and
blocks further calls; no-retry mode; a second 429; 5xx backoff; 5xx
Retry-After; 401 fallback; overlap merge (mutation-checked); provider
routes through the client.
- 164 passed across the client, `quick_version` and `search_lyrics`
tests.
- **Live** with the real app key: Dar Williams - Iowa and Wolfmother -
Where Eagles Have Been were found. One 503 was recovered by the retry.
- Local CodeRabbit, 3 cycles. I fixed the Retry-After cap, the cooldown
after a final 429, the `quick_version` 401 path, the overlap merge and
5xx Retry-After. I skipped one false positive about the tests' fake
clock (the retry path sleeps, then sends without re-checking the clock).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Jw21syo4HizJJjYw2mg2Vp

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 17:33:19 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 a6d4163d2e feat(alerts): route non-signal Discord posts to #noise (v0.278.8) (#1186)
## Summary
Andrew asked for #alerts to carry only real problems. Routine and
informational posts now go to the new **#noise** channel. The GitHub org
PR webhook was already moved to #noise separately.

| Message | Before | After |
|---|---|---|
| Error monitor: New Error Pattern / Spike / Incident | #alerts |
#alerts |
| Error monitor: ✅ Patterns Auto-Resolved | #alerts | **#noise** |
| Error monitor: 📊 Daily Error Digest | #alerts | **#noise** |
| Nightly Backup Report (clean run) | #alerts | **#noise** |
| Nightly backup failed/partial (title is now "🚨 Nightly Backup FAILED")
| #alerts | #alerts |
| ✅ E2E Daily Test PASSED | #alerts | **#noise** |
| 🩹 E2E self-healed Stripe (auto-merge PR opened) | #alerts | **#noise**
|
| 🩹 self-heal PR needs CI approval click / could not open | #alerts |
#alerts |
| E2E / Tenant E2E failures, DR freshness stale, Job failed, Publish
completeness SHADOW, Gemini unavailable, OAuth credential alerts,
quarterly DR drill reminder | #alerts | #alerts (unchanged) |

## How the noise webhook is resolved
- **Backend / error monitor and the backup function** read it at
runtime: env `DISCORD_NOISE_WEBHOOK_URL` first, then the Secret Manager
secret `discord-noise-webhook` (the name can be overridden with
`DISCORD_NOISE_WEBHOOK_SECRET_NAME`). No Pulumi change is needed,
because both service accounts already have project-wide
`secretAccessor`.
- **GitHub Actions** uses `secrets.DISCORD_NOISE_WEBHOOK_URL ||
secrets.DISCORD_ALERT_WEBHOOK_URL`.
- **Nothing is silently dropped.** If the noise webhook can't be
resolved, or a post to it fails, the message goes to #alerts instead. In
the workflow this uses `curl -fsS` with a retry on the alert webhook.
- The error monitor now rate-limits each channel separately (10 per run
each), so noise can't use up the alert budget during an incident.

## Deploy notes
- **Backup Cloud Function:** CI doesn't deploy its source. Run
`infrastructure/functions/backup_to_aws/deploy.sh` after merge.
- **Error monitor job:** runs `karaoke-backend:latest`, so it picks up
the change once CI pushes the image.
- **No `pulumi up` needed.**

## Testing
- `tests/unit/services/error_monitor`: 376 passed. New tests cover noise
vs alert routing, fallback when the noise webhook is missing or fails,
separate rate limits, and secret resolution.
- `backend/tests` (excluding emulator): 6418 passed, 36 skipped.
- `infrastructure/functions/backup_to_aws`: 60 passed, including the new
`test_discord_alert.py` and routing tests in `test_main.py`. These were
run in a scratch venv because `functions_framework` isn't in the conda
env, which was already the case before this change.
- `actionlint` on `e2e-daily.yml`: no new findings. The only findings
are info-level shellcheck notes that were already there.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_019pmG2XAgtUguEJWLFpkKHN

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 02:46:50 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 c277bc8ee3 feat: local E2E stack (fake workers + test API) + email-code sign-in (v0.275.0) (#1165)
## Summary
- **Local stack:** `scripts/local-stack.sh up --frontend` runs the
**real** backend on the Firestore + GCS emulators with `FAKE_WORKERS=1`.
In-process fake workers move jobs through their real states in seconds,
using tiny real media, so web Playwright and mobile Maestro full-journey
tests can run offline against the same endpoints.
- **Email-code sign-in** (nomad-mobile spec §4.1): `POST
/api/users/auth/send-code` + `verify-code`. The code logic is shared
with kjbox via `LoginCodeStore`, and sends are throttled per email and
per IP.
- **Bug fix found by the new suite:** `confirm-duration` moved jobs to
`separating_stage1`, from which they can never reach review. It now
resumes at `downloading`. No prod job had taken this path yet.

## Changes
- `backend/services/fake_workers/`: dispatch seam in
`WorkerService.trigger_*`, plus fake search, download, checkout and
outbox, scenarios, and per-user config. The real stem upload, transcode,
backing-vocals analysis and preview render are reused. Every check is
strict (`is True`), so a MagicMock settings object never enables the
fakes, and startup **refuses FAKE_WORKERS in production**.
- `/api/test/*` router (mounted only in that mode): `status`,
`scenario`, `fake-config`, `emails`, `reset`, plus the fake Stripe
checkout page. Paying it runs the real webhook handler.
- `frontend/e2e/local-stack/` (9 specs, ~50s):
  - sign-up → make → review → instrumental → download
  - buy and cancel credits
  - duration confirm, failed, complete and review states
  - code sign-in and lockout
- New required CI job `local-stack-e2e`.
- Docs: TESTING.md § Local stack, DEVELOPMENT.md, API.md, README,
LESSONS-LEARNED.
- `MAX_SIGNUPS_PER_IP` can now be set by env. Only the local stack
raises it; production keeps the default of 2.

## Testing
- [x] Backend unit: 6175 passed
- [x] Emulator: 152 passed
- [x] Frontend Jest: 1706 passed
- [x] Local-stack Playwright: 9/9 passed (CI mode)

## Review
- [x] CodeRabbit CLI review completed locally. 1 minor finding declined:
it suggested swallowing infrastructure errors in send-code as fake
"sent" responses, which would hide real outages from users.

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01AGuxtgoscF6RJGtpjHPg3W

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 16:00:24 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 0c1aa16469 Turn on conversion emails + fix doubled signature on credit-denied email (v0.273.3) (#1160)
## Summary
- **Conversion emails go live.** Andrew approved the copy, the 25%
offer, the timings and the 612 Joan St postal address. Adds
`CONVERSION_EMAILS_ENABLED=true,REVIEW_LAST_CHANCE_ENABLED=true,CONVERSION_OFFER_PERCENT=25`
to the backend `--set-env-vars` in ci.yml (feature shipped dark in
#1156).
- **Doubled signature fix.** The "About your Nomad Karaoke free credits"
(credit-denied) email appended `_get_email_signature()` itself and also
got it from the shared footer, so it showed twice. I audited all 22
`send_*` emails by rendering each one; this was the only duplicate.
- New regression test: renders every `EmailService.send_*` and asserts
the signature appears exactly once. It fails on main with
`send_credit_denied_email: 2`.

## Testing
- `make test` green (backend 2054 + 6071 + 150, frontend 1692).
- Local CodeRabbit review: no findings.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_019KuzHAU4rxkv8JfD8NmHTN

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 12:44:43 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 39dcf2598e fix: PyPI prune snippet + force marker only in PR title (v0.268.10) (#1146)
## Summary
Two follow-ups to #1145, both found while doing today's prune.

1. **Prune console snippet silently did nothing.** PyPI's delete-version
form now requires five `acknowledge_*` checkboxes. A submit without them
gets a **200 re-render**, which the old snippet counted as "deleted".
The snippet now reads each release's real form (CSRF plus checkboxes),
submits it, and counts a version deleted only when its manage page 404s.
This is the exact approach used for today's prune (88 releases, 10.73 →
~4.1 GB).
2. **The force marker matched the commit body.** A squash commit's body
is the PR description. #1145's description just *mentioned* the marker,
which forced a publish on merge. That publish ran before the prune, so
it 400'd and uploaded nothing. The marker now only counts in the commit
subject (the PR title).

Docs: the runbook covers the new form, the per-version JSON cache lag,
and a prune log.

## Testing
- Unit tests: 30 pass across both scripts, with new tests for
checkbox/404 handling and for a marker in the body not forcing a publish
- `node --check` on the generated snippet
- Replayed #1145's real merge message through the gate: no publish
- Local CodeRabbit review: 0 findings
- Full `make test` not re-run (no app code touched); PR CI covers the
package suites

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01MSW9yjwGnzz1S8MgCGm6t5

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 13:34:57 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 5005d3cb80 ci: publish to PyPI only when CLI code changed (v0.268.9) (#1145)
## Summary
PyPI publishing failed on every deploy since v0.268.4. karaoke-gen hit
PyPI's 10 GB project cap for the third time (June, Sept 29, Oct 12).
Each wheel is ~62 MiB and we published on every merge (~100/month), so
each prune only bought a couple of weeks.

PyPI is only the public `pip install` channel for the CLIs. Cloud Run
builds from source, and the encoding worker pulls the wheel from GCS. So
we now publish only when something CLI users get has changed.

## Changes
- `scripts/pypi_publish_gate.py` (stdlib): diffs
`scripts/pypi_publish_paths.txt` (`karaoke_gen/` plus the `backend/`
modules it imports) and `pyproject.toml` (ignoring the package `version
=` line only) against **the latest version on PyPI**. A skipped or
failed publish therefore never loses a CLI change. Fails open.
- `ci.yml` `package-publish`: new "Decide PyPI publish" step gates
`poetry publish` (now `--skip-existing`). **Tags and GitHub releases
still happen on every version**, because the encoding-worker gate diffs
against tags. Override with `[publish-pypi]` in the PR title/commit, or
the new workflow_dispatch input "Publish to PyPI".
- Not gated on purpose: `frontend/**` (bundled review UI) and
`backend/config.py` (cloud settings). A test keeps the backend import
list in sync, with `config.py` as the one documented exemption.
- Docs: PYPI-STORAGE-PRUNE.md, TROUBLESHOOTING.md, prune script and
reminder workflow comments.

## Impact
Replayed over v0.200 to v0.268: **24 of 136** releases would have
published (~0.7 GB/month instead of ~4 GB/month). Run against current
main, it decides "no CLI-affecting changes since v0.268.3", which is
correct for v0.268.4 to v0.268.8.

## Testing
- 15 new unit tests in `tests/unit/test_pypi_publish_gate.py`
- `make test` all green
- actionlint clean on the new step
- Local CodeRabbit review: 1 finding (doc recovery path), fixed

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01MSW9yjwGnzz1S8MgCGm6t5

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 12:56:24 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 950c96fa48 feat(e2e): checkout-only (no-charge) payment mode for daily E2E Stage 1 (v0.266.3) (#1130)
There's no working test card right now (the issuer declines it), so the
daily E2E Stage 1 defaults to a **no-charge** mode until the repo
variable `E2E_REAL_PAYMENT=true` is set.

## Checkout-only mode (`E2E_PAYMENT_MODE=checkout-only`)
- **Unchanged:** real signup via the `e2etest95` referral link, the Buy
Credits dialog, and real Stripe Checkout session creation.
- **Card:** fills the form with Stripe's public test card `4242…`
(`NO_CHARGE_CARD`; no secret needed). The self-healing steps still run,
except `submitPayment`.
- **Checks:** the $0.50 referral price and the Pay button are visible,
then it **stops before Pay**. Live mode always declines that test card,
so even an accidental click can't charge anything.
- **Stage 2:** gets the new account's session token and spends its
signup credit. The test asserts the account has at least 1 credit.
- **Not exercised:** the Stripe payment, the webhook credit grant, the
purchase email, and self-heal PRs (no verified purchase to learn from).
- **Email:** the daily email marks Stage 1 `[checkout-only: no real
payment]`.

## Testing
- Offline suite: 31 passed. The new test checks that no-charge mode
fills the form with the test card, never calls confirm and never leaves
Checkout.
- `--list` loads the production spec in both modes. Lint and typecheck
are clean.
- A live E2E Daily run on this branch (37262437261) is in progress
against prod.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 00:28:56 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 feef49a588 fix(e2e): stop publishing card data/admin token in artifacts; fail fast on card decline (v0.266.1) (#1128)
## Summary
E2E Daily **#200** failed in Stage 1. The cause was **the issuer
declining the test card**: Stripe `confirm` returned 402 `card_error /
card_declined / generic_decline` on all 3 attempts. It was not a Stripe
Checkout change. The self-healing runner still treated it as one: a 90s
verify wait, then 6 Gemini calls "re-entering" a form that was already
correct.

The investigation also found that **every run's artifacts published
secrets** in this public repo:
- the full test card (number, expiry, CVC) in screenshots, videos,
`error-context.md` and the DOM dump;
- `X-Admin-Token` in the trace network logs.

All 140 unexpired `e2e-*` artifacts have already been deleted via the
API. The admin token and the test card still need rotating (Andrew).

## Changes
- **Fail fast on decline:** `stripe-checkout.ts` watches
`/v1/payment_pages/<id>/confirm`. A `card_error` aborts `submitPayment`
immediately with `STRIPE_CARD_DECLINED: …`: no LLM calls, no second Pay
click.
- New `CheckoutStep.abortReason`, honoured by the runner, `pollVerify`
and the LLM loop.
  - Non-card errors (e.g. a missing field) still go to the LLM fallback.
- **Alerts say why:** Stage 1 exposes a `failure_reason` output ("Test
card declined, not a checkout bug (…)"). It appears in Discord and in
the daily email. The Discord payload is now built with `jq` from env, so
Stripe's text never reaches the shell.
- **No secrets in artifacts:**
- The Stripe helper's screenshots mask every input, and the DOM dump
redacts input values.
- The credit-purchase spec turns off trace, video and Playwright's
automatic screenshots.
- `playwright.production.config.ts` turns traces off on CI
(`E2E_TRACE=on` overrides).
  - New `frontend/e2e/scripts/scrub-artifacts.mjs`:
- deletes `*.zip` traces and `e2e-session-token.txt` (and videos with
`--drop-videos`);
    - redacts secret env values and card-number-shaped digit runs;
- with `--blank-inputs`, blanks `<input value>` and ARIA textbox values;
    - deletes any file that still contains a secret.
- Every production-E2E upload (`e2e-daily` ×2, `e2e-tenant-daily`, the
`ci.yml` happy path) now runs the scrubber first and **uploads only if
the scrub succeeded**.
- Docs: `TESTING.md` (decline behaviour, a "E2E Artifacts Are Public"
section), `LESSONS-LEARNED.md`.

## Testing
- `npx playwright test --config=playwright.selfheal.config.ts`: **30
passed**. New tests:
- a declined card fails in ~7s with 0 planner calls and exactly 1
confirm request;
  - the DOM dump contains no typed card values;
  - card-error parsing and the confirm-URL matcher;
- scrubber units (literal/grouped/PAN redaction, short secrets only
blanked inside inputs, deletion rules).
- Ran the scrubber against a copy of the real #200 artifacts: no card
data remains, and traces and videos were removed.
- actionlint: no new findings (only shellcheck info notes; ci.yml:996 is
pre-existing).
- Local CodeRabbit review: 2 findings (shell interpolation of the
reason; extension allowlist in the scrubber), both fixed.
- The full `make test` suite was not run: the change is limited to E2E
helpers, workflows and docs, with no backend or app code.

## Follow-ups (Andrew)
- Rotate the `admin-tokens` secret and the `E2E_ADMIN_TOKEN` repo
secret.
- Replace or unlock the test card; ideally a low-limit virtual card.
Then update the `E2E_STRIPE_CARD_*` secrets and re-run E2E Daily.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 22:25:52 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 3ecea6f2e7 feat(storage): enable real purge with deletion audit log and orphan cleanup mode (v0.266.0) (#1127)
## Summary
Andrew approved this follow-up to #1125 and #1126. It turns on real
deletion, adds an audit log of every deleted object, and adds an
orphan-folder cleanup mode.

### Real deletion on
`STORAGE_RETENTION_DRY_RUN=false` and
`STORAGE_RETENTION_MAX_JOBS_PER_RUN=50` are added to the backend
deploy's `--set-env-vars`. The API service is the only thing that reads
them: it passes `--dry-run` to `storage-retention-job`. That keeps the
setting in one place, so a later deploy can't silently revert it.

### Deletion audit log
- `storage-retention/deletion-logs/<ts>-<mode>.jsonl` has one line per
deleted or failed object: timestamp, run_id, mode, job_id, path,
size_bytes, generation, category and result.
- The log is rewritten after every job or folder, so a crash still
leaves a complete record. Each run also writes
`<ts>-<mode>.summary.json`.
- Deletes are pinned to the logged generation (`if_generation_match`).
- `assert_deletable` refuses any path outside `jobs/{id}/`, so
`storage-retention/` is never touched. No lifecycle rule matches live
objects there.
- The Firestore manifest on each job doc now also records generations
and the log path.

### Orphan cleanup
- `POST /api/internal/storage-retention/orphans` runs on the same Cloud
Run job with `--mode orphans`.
- It targets `jobs/{id}/` folders with no doc in `jobs`, `jobs-dev` or
`youtube_upload_queue`, re-checked live per folder against a fresh
listing.
- It skips any folder with an object modified in the last 30 days, keeps
`input/`, and deletes the rest.
- `encoding_worker_jobs` doesn't count as a record: it's the encoder's
expiring result cache, which deleted jobs leave behind.
- It is a dry-run unless called with
`dry_run=false&confirm=delete-orphan-job-folders`, and the scheduler
never runs it.

### Docs
- `docs/STORAGE-RETENTION.md`: run modes, log locations, jq review
commands and restore one-liners (noncurrent `cp #gen` and soft-delete
`restore`).
- `scripts/storage_retention_report.py`: list runs, summarise a run, or
find which run deleted a path or job.

## Testing
New tests cover the log contents, generation-pinned deletes, per-job
flushing across a crash, the path guard, orphan planning (input kept,
30-day skip, live record re-check, batch limit), and the endpoint and
worker modes. Full backend suite: 5774 passed.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added admin-triggered orphan-folder cleanup, with dry-run enabled by
default, optional folder limits, and explicit confirmation required for
deletion.
* Added deletion reports and audit logs covering removed objects,
reclaimed space, and failures, plus a read-only tool to review runs and
search records.
* **Bug Fixes**
* Added safeguards to prevent deleting changed objects or files outside
their intended job folders.
* **Documentation**
* Expanded storage-retention guidance with recovery details, review
commands, and cleanup instructions.
* **Chores**
* Set the production cleanup limit to 50 jobs per run and updated the
package version.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 11:54:05 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 c8461e56d0 fix(e2e): self-heal CI approval — correct pending detection, re-approve open PRs (#1121)
## Summary
This fixes the approval step from #1118, found by the live test on run
37140934446. That run reported `approved=true`, but bot PR #1119's CI
run was still `action_required`: the old check treated *any* existing
`pull_request` run as already running.

- `approve_ci` now succeeds only once a `pull_request` CI run exists
that is past `action_required`. It approves pending runs as it finds
them.
- When a self-heal PR is already open, the job now runs `approve_ci` for
that PR instead of exiting. So a stuck bot PR gets unstuck on the next
run.
- Polling is tolerant of transient `gh`/`jq` failures (from the local
CodeRabbit review). An empty branch name fails fast.

## Testing
- `actionlint` is clean for the changed lines; local CodeRabbit review:
2 findings, both fixed.
- After merge: a forced-LLM run on `main` will hit the "PR already open"
path for #1119. That approves its CI with the workflow's own token and
should let it auto-merge with no manual input.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 17:50:38 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 1c9104a79d fix(e2e): self-heal job approves its PR's CI run (no manual step) (#1118)
## Summary
This follows #1116. The live test of the self-heal loop (bot PR #1117)
merged, but only after one manual step: approving its CI run.

- **Approve the bot PR's CI run.** A bot-authored PR's `pull_request` CI
run is created as **action_required** (waiting for approval), and until
it runs it outranks the passing CI Gate from the dispatched run. The
`self-heal-pr` job now approves that run via `POST
/actions/runs/{id}/approve`. If approval fails, it dispatches `ci.yml`
and the Discord alert says the PR needs a one-click approval, with a
link.
- **Ruleset dependency, now documented.**
`require_extra_approval_for_unattributed_changes` is now set to `false`
on the `protect main` ruleset (11423976). It's a newer GitHub setting
that defaults to true and blocks bot-authored PRs until a human
approves. This is recorded in `docs/TESTING.md` and
`docs/LESSONS-LEARNED.md`.

## Testing
- `actionlint` is clean for the changed lines.
- After merge: a forced-LLM run on `main` on a step with no learned
recipe yet (`fillPostalCode`) will open a fresh bot PR and exercise
approval → CI → auto-merge without manual input.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 17:30:35 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 4768164a94 feat(e2e): self-healing Stripe Checkout (LLM fallback + auto-merging fix PRs) (#1116)
## Summary
Stripe's hosted checkout page has broken the daily E2E **5 times since
April** (#916, #919, #945, #1115…). Each fix added another selector, and
each new Stripe change still needed a human. This PR makes the Stripe
stage repair itself, with no manual steps.

**How it works.** `stripe-checkout.ts` now runs the checkout as verified
steps (`selectCard`, `fillCardNumber`, `fillCardExpiry`, `fillCardCvc`,
`fillCardholderName`, `fillPostalCode`, `uncheckSaveInfo`,
`submitPayment`). Each step tries, in order:
1. **learned variants** (`learned-variants.json`, newest first),
2. the existing hand-written **built-in** selectors,
3. a **Gemini fallback** that drives the page one action at a time until
the step's check passes. Model list: `gemini-3.5-flash-lite` →
`gemini-3.1-flash-lite` → `gemini-3.8-flash`; IDs that aren't available
are skipped.

A step only counts as done when its **check** passes, never just because
a click didn't throw (the root cause of run #192's misleading failure).

**Fully self-healing loop.** When the LLM repairs a step *and* the
purchase is confirmed server-side (credit granted), the new
`self-heal-pr` job (on `main` only):
- opens a **data-only** PR that changes only `learned-variants.json`
(label `e2e-self-heal`, `@coderabbitai ignore`),
- enables auto-merge,
- dispatches `ci.yml` on the branch, because PRs opened with the
workflow token don't trigger CI.

The next run replays the recipe without the LLM. Discord and the daily
email report 🩹 when this happens, and only one self-heal PR is open at a
time.

**Safety**
- Card data never reaches the LLM. It may only use per-step
`{{CARD_NUMBER}}`-style placeholders, which are substituted locally.
Field values and card-number-like digit runs are redacted from what it
sees, and inputs are masked in screenshots. Tests assert no secret
appears in the prompt.
- Guardrails refuse clicks on Link/Klarna/Cash App/Bank/"Back to…", and
on the final Pay button outside `submitPayment`. They check the
**resolved element** (text, aria-label, test ID, submit type), not just
the model's label.
- The Pay step's check is "redirected to the success page OR credit
granted server-side" (throttled), so the fallback never submits twice.
The cancel URL doesn't count as success.

**Gemini key:** the AI Studio `gemini-api-key`, billed outside the
nomadkaraoke project. A `GEMINI_API_KEY` repo secret is used if set;
otherwise it's read from Secret Manager via keyless login, and the
credential file is deleted before the tests run. With no key, the test
runs deterministic-only, as before.

Also passes the existing `E2E_STRIPE_ZIP` secret, which was never wired
up.

## Testing
- **Offline:** `playwright.selfheal.config.ts`, 21 tests with fixture
Stripe pages served on fake checkout.stripe.com/js.stripe.com and a stub
LLM. Covers: known layout uses no LLM; unknown layout → LLM repairs it →
recipe learned → replayed without the LLM; no secrets reach the LLM;
guardrails, including CSS targets; deterministic-only failure mode; unit
tests for redaction, merging and frame keys. Added to CI's smoke job.
- **Live (real Stripe, real Gemini) on this branch,** with
`self_heal_force_llm=selectCard`:
- [run
37137960148](https://github.com/nomadkaraoke/karaoke-gen/actions/runs/37137960148):
`gemini-3.5-flash-lite` completed "choose Card" in one call, credit
granted 1→2. This exposed the session ID in Stripe's URL path, now
fixed.
- [run
37138459027](https://github.com/nomadkaraoke/karaoke-gen/actions/runs/37138459027):
repeated after the review fixes. Recipe
`[data-testid="card-accordion-item"]` @ `checkout.stripe.com/c/pay`,
credit granted.
  - The PR-opening job is skipped on branches by design.
- After merge, I'll dispatch once on `main` with
`self_heal_force_llm=selectCard` to exercise the PR → CI → auto-merge
path.
- Local CodeRabbit review: 2 findings, both fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 17:02:12 +00:00
Andrew Beveridge ☄️andClaude Opus 5.5 8490ab75eb chore(gemini): move every Gemini call from Vertex AI to the Gemini Developer API (#1108)
## Summary
Moves **every** Gemini call in gen off **Vertex AI** (billed to the
`nomadkaraoke` GCP project, ~$85/mo) onto the **Gemini Developer API**
using the `gemini-api-key` Secret Manager secret (an AI Studio key
billed outside this project). Once deployed and verified,
`aiplatform.googleapis.com` / `generativelanguage.googleapis.com` get
disabled in nomadkaraoke.

### Key injection
- `backend/services/gemini_client.get_genai_client(timeout_ms)` — key
from `GEMINI_API_KEY`, else Secret Manager `gemini-api-key` via
`settings.get_secret` (runtime SA already has project-level
`secretAccessor`). No `project`/`location`.
- Service: CI `--set-secrets ... GEMINI_API_KEY=gemini-api-key:latest`.
- Jobs: Pulumi secret env on `lyrics-transcription-job`,
`video-encoding-job` (renders + translations), `nomad-error-monitor` —
**already applied** (targeted `pulumi up`, only that env added). Other
jobs fall back to Secret Manager.
- Local scripts (`frontend/scripts/translate.py`,
`scripts/measure_autocorrect_cost.py`,
`scripts/test_agentic_correction.py`): env, else `gcloud secrets
versions access` (in-process only).
- **Rotation:** add a new secret version; new instances / job executions
pick it up. No code change.

### Degradation (quota / billing / invalid key; per-minute 429s stay
retryable)
| Caller | Behaviour |
|---|---|
| auto-correct | non-retryable 503 → compare mode drops the Gemini leg,
keeps Opus |
| match-judge / free-text / parse-titles / tenant bulk | existing
fallbacks (no suggestion / blanks / regex) |
| credit eval | `pending_review` (fail-closed, unchanged) |
| lyrics translation | no retry; render proceeds without translation,
status recorded |
| custom lyrics | clean 503 "temporarily unavailable…" (+ 5-min timeout
so it can't hang) |
| error-monitor LLM | no retry; alerts sent without AI summary |
| translate.py | exit 2 "Gemini quota/credit exhausted — top up AI
Studio or rotate gemini-api-key" |

`note_gemini_failure()` also sends one Discord ops alert per 6h
(Firestore-throttled across instances via new
`ops_alerts.send_throttled_ops_alert`).

### Infra
- Pulumi: removes `roles/aiplatform.user` from `karaoke-backend` —
**will be applied after this deploys and Vertex traffic is confirmed at
zero**.
- Agentic correction default model `gemini/gemini-3.8-flash`
(`vertexai/` prefix still accepted, routed to the Developer API).
- Version 0.261.1 → 0.262.0. Docs: CLAUDE.md, LESSONS-LEARNED, README,
ARCHITECTURE, API, etc.

## Testing
- New: `backend/tests/services/test_backend_gemini_client.py`
(classifier, key resolution, no-Vertex client kwargs, alert throttle,
per-caller degradation),
`tests/unit/lyrics_transcriber/test_agentic_gemini_provider.py`,
`frontend/scripts/test_gemini_client.py`, lyrics-translation quota test.
- `pytest backend/tests --ignore=backend/tests/emulator` — all pass;
`pytest tests/unit` — 2021 passed.
- Verified `gemini-3.8-flash` is listed on the Developer API and
answered a 5-token test call with the key.
- `translate.py --dry-run` on `frontend/messages` and
`backend/translations` OK.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Updates**
* AI-powered lyric translation, custom lyrics, auto-correction, title
parsing, match judging, and related features now use the Gemini
Developer API.
* AI features use the configured Gemini API key. Quota, billing, or key
issues may show an unavailable message or use an existing fallback
instead of repeated attempts.
* **Documentation**
* Updated product, API, setup, and recovery guidance to reflect Gemini
API key configuration and usage.
* Updated guidance for handling AI service availability and quota
issues.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 21:58:53 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 49464e8fcc chore(cost): GCP cost cuts round 2 (v0.261.4) (#1107)
Round 2 of the GCP cost cuts, approved by Andrew. Target: the whole
project under $300/mo; the current run-rate is about $400.

## Changes
- **karaoke-backend min-instances 1 → 0** (`ci.yml`), about $24/mo. The
backend Cloud Run service isn't defined in Pulumi, so the CI flag is the
only place this is set. Andrew accepts a ~15s cold start. The 60s
`/api/health` uptime check and the `*/10` schedulers keep an instance
warm most of the time. With `--cpu-throttling` (request-based billing),
a warm instance above the minimum costs nothing while idle.
- **audio-separator Cloud Run GPU service shut down** (us-east4), about
$22/mo. It is now behind the Pulumi flag `audioSeparatorServiceEnabled`,
off by default. The service, SA, IAM bindings, output bucket and AR repo
were already removed with a targeted `pulumi up`; the bucket needed the
break-glass SA. `audio-separation-job` is untouched; it uses
`karaoke-backend-gpu/karaoke-backend`. The separator health check no
longer depends on `AUDIO_SEPARATOR_API_URL`, which also fixes a status
card that always showed "offline". The CLI now logs a clear hint when
the remote API fails. Companion PR:
nomadkaraoke/python-audio-separator#309 makes its Cloud Run deploy
manual-only.
- **backup-to-aws no longer copies job files to S3**, about $15/mo
egress. Job files now rely on GCS versioning plus soft-delete.
Firestore, secrets, BigQuery, kn-data and git-bundle backups are
unchanged. The freshness monitor no longer checks `gcs/job-files/`, and
DISASTER-RECOVERY.md is updated.
- **Artifact Registry cleanup policies**, about $8–10/mo. Per package:
keep the 10 most recent versions plus anything tagged `latest`,
`content-` or `cache`; delete all other versions older than 7d. Until
now, tagged per-commit images were never deleted. Dropped the CI
fallback that copied the GPU base once from us-east4, and deleted the
legacy 13 GB us-east4 GPU base package.
- Docs: GCP-COST-OPTIMIZATION.md (new section, including how to redeploy
the separator on demand), ARCHITECTURE.md, DISASTER-RECOVERY.md.

Changes made outside Pulumi (gcf-artifacts cleanup policy, destroying
superseded secret versions) are recorded in GCP-COST-OPTIMIZATION.md.

## Tests
- `backend/tests/test_system_status_endpoint.py` (new separator status
tests)
- `infrastructure/functions/backup_to_aws` (49 tests, including a new
one that checks job files aren't synced)
- `tests/unit/test_audio_remote.py`, `test_audio_processor_local_gpu.py`

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Service Availability**
* The hosted audio-separation service is no longer available. Audio
separation can run locally; users relying on the hosted service will
need another deployment.
* **Backup and Recovery**
* New job files are no longer copied to off-site S3 backups. S3 recovery
is limited to files backed up through September 30, 2026; newer outputs
may need to be regenerated.
* **System Status**
* Audio-separator health status now reports the locally installed
package version without checking a remote service.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 21:22:49 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 e4dc50d9e3 fix(ci): sanitize PR title in deploy env vars (v0.261.3) (#1109)
The v0.261.2 main deploy (run 36945703908) failed at **Deploy to Cloud
Run (candidate revision)** with:

`argument --set-env-vars: Bad syntax for dict arg`

The #1106 PR title contained a comma, and `PR_TITLE` is spliced into the
comma-delimited `--set-env-vars`. The bug was already there and that
title was the first to trigger it. Its Cloud Build step succeeded.

Fix: in both build-meta steps, commas become `;`, and newlines and
shell-active characters (`\" \` $ \\`) are removed. Version bumped to
0.261.3.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 20:50:43 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 dd795c66a0 ci(cost): build backend images in Cloud Build, run deploy-backend on ubuntu-latest (#1106)
## Summary
Retires the self-hosted `gha-build-*` runner VMs (e2-standard-8,
`docker-build` label, ~$21/mo). `deploy-backend` now runs on free
`ubuntu-latest`; the Docker images are built by **Cloud Build in
us-central1**.

## Why not build on the hosted runner? (egress math)
The app images are built `FROM` bases in us-central1 Artifact Registry,
and the app layer has a `RUN pip install`, so the full base must be
pulled each deploy:
- CPU base `karaoke-backend-base`: **4.22 GB** compressed
- GPU base `karaoke-backend-gpu-base`: **13.1 GB** compressed
- ~120 main deploys/month (119 in Sept, 11 so far today)

On a GitHub-hosted runner that is GCP internet egress: 17.3 GB × 120 ×
$0.12/GB ≈ **$250/mo** (CPU-only would still be ≈ $61/mo). Both are
worse than the $21 VM.

Cloud Build in us-central1 pulls same-region (free). Build-minutes on
the default-pool e2-standard-2 machine are in the **2,500 min/month free
tier** ($0.006/min after that). A test build of the real images (CPU and
GPU in parallel) took **6m55s**, about the same as today's ~7.3 min of
sequential builds on the VM. That's ~840 min/mo, so the expected cost is
~$0 (worst case ~$5/mo).

## Changes
- `infrastructure/cloudbuild/backend-images.yaml` +
`build-backend-images.sh`: buildx (docker-container) builds with the
**same tags and registry caches** as before. CPU and GPU build as
parallel steps, and each build gets a fresh metadata-server token.
- `ci.yml` `deploy-backend`: `runs-on: ubuntu-latest`. It keeps the hash
checks (manifest + ~10 KB config blob only) and submits `gcloud builds
submit --async`, then streams the logs and checks the status.
- Base rebuilds (≈ monthly) use e2-highcpu-8 with 300 GB disk and a 2 h
timeout.
- A Cloud Build still running when the job is cancelled or fails is
cancelled. CI concurrency cancels superseded deploys, and an orphaned
build would otherwise still push `:latest`.
- Removed the disk-space step, the local base pull, and the one-time
us-east4→us-central1 GPU-base copy. The copy finished 2026-09-26, and on
a hosted runner it would stream 13 GB through egress.
- `build-runner-images.yml`: `variants=all` no longer bakes the unused
`build` family. It can still be built with an explicit `variants=build`.
- Docs: EPHEMERAL-GHA-RUNNERS, SELF-HOSTED-RUNNERS,
GCP-COST-OPTIMIZATION.

No Pulumi or IAM changes were needed. `github-actions-deployer` already
has `cloudbuild.builds.editor`, `storage.admin` and
`iam.serviceAccountUser`, and the default Cloud Build SA (compute
default) already has `artifactregistry.writer` on both repos.

## Rollback (one commit)
`git revert <squash-merge sha of this PR>` and push to main. That
restores `runs-on: [self-hosted, linux, gcp, docker-build]` and the old
in-job buildx steps. Nothing outside the repo was changed: the
runner-manager `build` family is untouched, and the `gha-runner-build`
GCE images (20260812, 20260901) are kept so the dispatcher can still
launch `gha-build-*` VMs. If the build image has been pruned by then,
re-bake it with Actions → Build GHA Runner Images → `variants=build`
(only `variants=all` no longer includes it).

## Testing
- I ran the real CPU and GPU builds from this branch's config in Cloud
Build us-central1, with `cbtest-*` tags and no cache writes. Both
succeeded in 6m55s, and the us-east4 push uploaded only the app layers.
- `actionlint` and `shellcheck` report no new findings.
- Full verification is the first main deploy after merge.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Improvements**
* Backend image builds now run through a managed build service in the
central US region, with CPU and GPU images published as part of the
deployment workflow.
* The default “all variants” selection now excludes the build variant;
explicitly requested variants remain available.
* **Documentation**
* Updated runner and cost documentation to reflect the current build and
deployment setup.
* **Maintenance**
  * Updated the package version to 0.261.2.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 20:22:59 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 354e43bab2 Render Hebrew/Arabic karaoke right-to-left; fix title-card tofu (v0.255.0) (#1094)
## Summary

A Hebrew KaraokeHunt job (5710831e, Omer Adam – "שני משוגעים") shipped
with its lyrics **laid out and highlighted left-to-right** and a **title
card of "?" boxes**. Diagnosed by measuring rendered pixels (not by
eye), reproduced locally and with prod's static ffmpeg 7.0.2 in Docker.

**Root causes**
- ASS style `Encoding=0` makes libass force LTR paragraphs; the `{\kf}`
before each word splits the line into runs, so each word is shaped
correctly but the words go left→right.
- `\kf` always wipes left→right within a word
([libass#406](https://github.com/libass/libass/issues/406)).
- PIL (title/end cards) has no per-glyph font fallback; the theme font
(Avenir Next) has no Hebrew glyphs.

**Fixes**
- `build_karaoke_styles` forces `Encoding=-1` (libass auto paragraph
direction; LTR unchanged).
- RTL lines get `{\frz(180+angle)\frx180\fry180}` — a visual no-op that
makes libass ≥0.15 sweep fills right→left without splitting words
(Arabic still joins). Theme angle preserved; re-emitted after duet
`{\r}` resets.
- Lead-in indicator mirrored for RTL lines.
- Text measurement mirrors libass: per-glyph fontconfig fallback, size
scaled from each font's metrics (old fixed 0.70 was 20%+ off for
fallback fonts).
- New `karaoke_gen/utils/font_fallback.py` (fontTools cmap +
fc-match/fc-list) used by title/end cards, portrait headers and ASS
measurement.
- Found by the new tests: inter-word gaps ≤0.1s were dropped, so the
highlight ran up to ~0.2s early by line end (all languages). Karaoke
tags now derive from one absolute centisecond timeline.
- Fixed pre-existing `test_render_video_worker_gce` failure on main
(MagicMock generation → `_g<mock>` id).

## Testing
- **Pixel-level render tests** (`test_ass_render_highlight.py` +
`ass_render_harness.py`): real `SubtitlesGenerator` output rendered with
ffmpeg/libass in a pure-colour palette, asserting on frames for English,
Hebrew, Arabic, Chinese, Japanese, Korean + an RTL duet line: highlight
starts at the reading-start edge, sung pixels always precede unsung in
reading order (word order + in-word fill direction), monotonic and in
time with the words, lead-in on the correct side, no missing glyphs, RTL
tags don't move the line (incl. rotated themes).
- 32 fail on pre-fix main; mutation checks ("Encoding only", "no re-emit
after `\r`", "ignore theme angle") each fail exactly the expected cases.
- 123/123 pass on Linux against **both** Debian ffmpeg (CI-equivalent)
and **prod's johnvansickle static ffmpeg** via
`scripts/run-render-tests-linux.sh`.
- CI now installs `fonts-noto-core fonts-noto-cjk`; under `CI=true`
these tests fail rather than skip on missing preconditions. (macOS
libass uses CoreText → font-sensitive checks skip locally.)
- Title-card tests (`test_font_fallback_render.py`): no `.notdef` tofu
for Hebrew/Arabic/CJK titles; Hebrew title reads right-to-left.
- Unit tests: `test_rtl_ass_output.py`, `test_font_fallback.py`.
- `make test`: unit 1972 passed; emulator 150 passed; frontend 1491
passed.
- Full local re-render of job 5710831e reviewed by Andrew: RTL highlight
smooth and correct, title card correct.

## Follow-ups (not in this PR)
- CDG output still renders RTL left-to-right / tofu (separate renderer).
- Separate investigation: transcription dropped 3 sung lines in this
job, which the section detector then labelled INSTRUMENTAL.

No local CodeRabbit review (CLI has no seat for this org), so leaving
this open for the CodeRabbit bot. A local code review ran; its findings
are addressed in the second commit.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Karaoke lyrics now render more reliably for Hebrew, Arabic, Chinese,
Japanese, and Korean, with improved reading-direction highlighting,
timing, and lead-in placement.
* Titles and portrait text can use fallback fonts when the selected font
lacks the necessary characters.

* **Bug Fixes**
* Improved lyric centering, word highlighting, and glyph rendering
across supported scripts.

* **Documentation**
  * Added guidance on lyric rendering and multilingual test coverage.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 21:02:53 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 badd845a1d perf(jobs): move video-encoding-job (video + render) to us-east4 (v0.249.1) (#1083)
## Summary
Follow-up to #1077. `video-encoding-job` runs both the post-review
render (`render_video_worker`) and the final video (`video_worker`). In
us-central1 each run waited **2–5 min** for Cloud Run Jobs to start it,
so every customer job paid that twice. This moves it to us-east4
alongside the download, lyrics and bulk-search jobs.

## Cost (measured, not estimated)
- Over the last 30 days the job read **156 GB** from Google services,
mostly the us-central1 bucket, across 494 runs (Cloud Monitoring
`container/network/received_bytes_count`, kind=google). That's about
0.32 GB per run.
- Pricing from the Billing Catalog: GCS inter-region within North
America is **free for the first 100 GB/month, then $0.02/GB**. Cloud Run
writes to Google APIs are **$0**.
- Internet egress (224 GB/month, Dropbox etc.) and Cloud Run CPU cost
the same in either region.
- Existing cross-region reads are about 24 GB/month (GPU job and
lyrics), which the billing export shows at about $0.
- **Net: about $1.60/month**, or about $3/month worst case. Time saved:
roughly 4–8 min per customer job.

## Changes
- **backend**: `trigger_video_worker` (via `_trigger_cloud_run_job`) and
`trigger_render_video_worker` now use `cpu_jobs_region`. `GCP_REGION`
stays us-central1; encoding VM and zone logic are unchanged.
- **infra (Pulumi, already applied)**: new us-east4
`video-encoding-job`, with the legacy us-central1 copy kept for rollback
and its deletion protection turned off. A preview after applying showed
290 unchanged.
- **ci**: the video job image update now targets us-east4, using the
us-east4 CPU image copy.
- **Also updated**: rollback.sh, tests (video and render routing),
TROUBLESHOOTING/ARCHITECTURE/LESSONS-LEARNED.

## Testing
- `make test` passes: backend tests plus 1443 frontend tests.
- Prod smoke test of the new us-east4 job: it imports `video_worker` and
`render_video_worker` and calls the encoding VM's `/health` on its
public IP (port 8080 is open to 0.0.0.0/0 with API-key auth). **Started
in 25 s, exit 0.**

🤖 Generated with [Claude Code](https://claude.com/claude-code)


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Updates**
- Video encoding and render jobs now run in the `us-east4` region,
alongside other CPU jobs.
- Video encoding continues to access storage and encoding resources in
`us-central1`; cross-region transfer may incur costs.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 16:33:23 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 cb5ce7f162 chore(pypi): prune keep-window default 60→30 days (#1082)
Ran the PyPI prune runbook today (2026-09-29): publishing was blocked
(`Project size too large`). The 60-day policy would only have freed 1.08
GB (→9.64 GB), refilling within ~a week, so pruned with `--keep-days 30`
instead (69 releases deleted, 0.188.5–0.207.0; newest release of each
older month retained) → ~6.8 GB. Re-ran the failed publish; 0.249.0 is
live on PyPI.

This PR makes 30 days the default in the planner script, the monthly
reminder workflow, and the runbook. Unit tests pass (13).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 16:17:43 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 3e855ff65d perf(jobs): run download/lyrics/bulk-search Cloud Run Jobs in us-east4 — cut 2-5 min job start latency (v0.247.1) (#1077)
## Summary
Every gen job waited **2–5 min** before `audio-download-job` (and then
`lyrics-transcription-job`) even started. That blocked kjbox's goal of
singing within about 4 minutes of making a song.

**Root cause: Cloud Run Jobs in us-central1 are slow to start on
Google's side.** Our image, CPU and code aren't the cause.
- The `Started` condition fires about 10 s before the first Python log,
so our process startup is fine.
- decide's 203 MB image on 4 CPU is just as slow in us-central1.
- A throwaway job running Google's tiny sample image, 4 runs per region:
**us-central1 4m–5m07s**, **us-east4 8–19 s**, **us-east1 8–25 s**. All
four us-central1 runs started in the same second, as if released from a
queue. Quotas are far from their limits.

## Changes
- **backend**: new `CPU_JOBS_REGION` setting (default `us-east4`) for
triggering `audio-download-job`, `lyrics-transcription-job` and
`bulk-search-job`. `GCP_REGION` is unchanged. `video-encoding-job` stays
in us-central1, because it pulls multi-GB outputs from the us-central1
bucket.
- **infra (Pulumi, already applied)**: the three jobs are created in
us-east4 as new resources. The legacy us-central1 copies are kept as the
rollback path, with deletion protection turned off so a follow-up can
remove them. The preview after applying showed 289 unchanged, so there's
no drift.
- **ci**: the CPU app image is also pushed to
`us-east4-docker.pkg.dev/nomadkaraoke/karaoke-backend-gpu/karaoke-backend-cpu`,
so jobs pull from their own region. The lyrics job image update now
targets us-east4. After each deploy, CI starts a no-op
`audio-download-job` execution so the one-time ~2.5 min image import
doesn't land on the first customer.
- `scripts/rollback.sh`: lyrics job moves to the us-east4 image, and
jobs whose target tag is missing are skipped instead of aborting the
loop.
- Docs: TROUBLESHOOTING runbook (measure + rollback), ARCHITECTURE,
LESSONS-LEARNED.

## Testing
- `make test`: all backend tests pass, plus 1422 frontend tests.
- New tests in `backend/tests/test_worker_service_cpu_jobs_region.py`:
routing of each moved job, video job stays in `GCP_REGION`, env override
for rollback, and the Pulumi/backend region constants must match.
- Prod smoke test of the new us-east4 `audio-download-job` with the real
image (`import backend.workers.audio_download_worker`): **Started in 7.0
s, 8.6 s, 10.0 s** (first run of a new digest adds a one-time ~2.5 min
import, which the CI step covers).
- Local review via a reviewer agent: no must-fix findings; the
rollback-path suggestions are addressed.

## Rollout
One-time image seed (Cloud Build gcrane copy) and `pulumi up` are done.
After merge, CI deploys the backend, which starts triggering the
us-east4 jobs.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Performance**
* Audio downloads, bulk searches, and lyric transcriptions now run in a
region with faster job startup times, reducing delays before processing
begins. Video encoding continues to run in its existing region.
* **Reliability**
* Rollbacks now skip jobs when the target image is unavailable and
continue with the remaining jobs. Other image lookup errors still stop
the rollback.
* **Documentation**
* Updated architecture and troubleshooting guides with regional job
details, startup-delay guidance, and rollback instructions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 01:02:19 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 8a5f754efd ci: remove post-deploy canary + never email test accounts on stale review (v0.245.2) (#1068)
## Summary
- Removes the `deploy-verify` job ("Deploy - Post-Deploy Canary (Happy
Path)") from `ci.yml`, plus the `Detect backend version change` step /
`version_changed` output that only gated it.
- `stale_review_processor`: jobs owned by automated-test accounts
(`is_test_email` — `e2e-test-runner@nomadkaraoke.com`,
`@inbox.testmail.app`) no longer get the 24h reminder, review-expired,
or duration-confirm-expired emails. They are still auto-expired at 48h
so stranded test jobs don't linger.
- Updates comments + `docs/TROUBLESHOOTING.md` / `docs/TESTING.md` that
referenced the canary. Version → 0.245.2.

## Why
The canary created a real production job (piri – dog, as
`e2e-test-runner@nomadkaraoke.com`) on every backend version bump.
`ci.yml` uses `cancel-in-progress: true`, so back-to-back merges
cancelled it mid-run — the process is killed before Playwright's
`afterEach` cleanup runs, stranding the job in `awaiting_review`. The
stale-review processor then emailed the test account (which forwards to
Andrew's inbox) at 24h/48h (e.g. job `adb8f393` from cancelled run
36288893657; earlier `a5f84cc3`, `10b532e2`). The canary was overkill;
the daily E2E and the traffic-gated candidate-revision smoke remain, and
the email guard backstops the daily E2E if it ever strands a job.

## Test plan
- [x] New unit tests: test-account reminder is silent (+ still flagged),
review expiry + duration-confirm expiry cancel without email
- [x] `pytest backend/tests --ignore=backend/tests/emulator` — 4883
passed
- [x] `actionlint` — no new findings

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* CI no longer runs a post-deployment production canary or sends its
test results and failure notifications. Candidate-revision smoke testing
remains in place.
* Production real-job verification is handled manually or by the daily
E2E run.
* **Documentation**
* Testing and rollback guidance now reflects the updated verification
and recovery steps. Daily E2E failures are identified as a reason to
roll back.
* **Bug Fixes**
* Automated test accounts no longer receive stale-review reminder or
expiry emails; existing expiry and cancellation handling remains in
place.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-27 23:37:45 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 87a36bd2a5 feat(encoding): GCP cost cuts — Spot 16-vCPU workers, 3-VM fleet, 5-min idle, gated worker deploys (v0.245.0) (#1061)
## Summary
Approved GCP cost cuts for the encoding pipeline (target: whole project
< $300/mo; encoding VMs were ~$300/mo for ~5-20 jobs/day). Details +
before/after table: `docs/archive/2026-09-26-encoding-cost-cuts.md`.

| | Before | After |
|---|---|---|
| Primary pair a/b | c4d-highcpu-32 on-demand | **c4d-highcpu-16 Spot**
(STOP on preemption) |
| Fallbacks | 8 VMs / 6 families | **1 on-demand**
`encoding-worker-fallback-c2df` (c2d-highcpu-16, us-central1-f) |
| Idle shutdown | 15 min / check every 5 | **5 min / check every 2** |
| CI worker blue-green | every push to main | only when
`infrastructure/encoding-worker/worker_code_paths.txt` paths changed
since the serving version (+ `workflow_dispatch`
`force_encoding_worker_deploy`) |
| Nomad Dropbox upload | whole folder | minus lossless 4K **MP4**
(tenant folders unchanged) |

**Infra already applied locally (targeted Pulumi):** 7 fallback VMs +
disks + 7 static IPs deleted; a/b + c2df resized in place; idle function
env + scheduler updated. The Spot switch was done in place with `gcloud
compute instances set-scheduling` on the stopped VMs + `pulumi refresh`
(pulumi-gcp treats a `scheduling` diff as *replace*, and recreating a
c4d VM risks a stockout). `encoding-worker-fallback-vms` secret → v6
(c2df only). CI's pulumi run should be a no-op for these resources.

### Scope change (per Andrew)
**Task 5 dropped:** review previews stay on the encoding VM
(`USE_GCE_PREVIEW_ENCODING=true` unchanged) and the review-page warmup
is kept, since it deliberately warms the VM for the preview and the
final render. To make sure the new 5-min idle timeout doesn't undermine
that, the review page now also heartbeats **every 2 min** while the tab
is visible and was used in the last 15 min, and re-warms on tab refocus
(`useEncodingWorkerKeepAlive`). Before this, the heartbeat only fired on
lyric edits. The idle checker keeps the routed VM alive while
`last_activity_at` < 5 min, and both warmup and heartbeat write it.
Tests cover both sides.

### Code changes
- **Spot preemption mid-encode**: when a poll fails,
`wait_for_completion` checks the pinned worker VM. If it is
STOPPING/TERMINATED it raises `EncodingJobLostError`, so
`run_with_lost_job_resubmit` resubmits the encode, and that resubmit
restarts the Spot VM or falls back to c2df. Before, the failure showed
up after ~7 min as a "lost contact" error that was not resubmitted.
- **Stale fallback lists**: a candidate VM that no longer exists
(`NotFound`) is skipped instead of aborting the whole failover. This
protects the window before running instances pick up the new secret.
- `SPEED_RANK` is now keyed by machine family, so the ranking is
independent of VM size.
- **Idle-stop vs. in-flight encode** (review follow-up): the idle
function's `/health active_jobs` only counts RUNNING jobs, and
`last_activity_at` was only written when a VM was started. With a 5-min
window, a long encode's VM could be stopped in the seconds between the
job finishing and the next poll. The poll loop now refreshes
`last_activity_at` at most once a minute. A VM seen in
STAGING/PROVISIONING after it accepted a job also counts as lost. CI
`Read Firestore config` now writes empty values instead of "None".
- **Worker deploy gate**:
`backend/tests/test_encoding_worker_code_paths.py` derives the worker's
transitive `backend.*` import set and fails if `worker_code_paths.txt`
doesn't cover it. Every deploy still uploads the wheel and `startup.sh`,
so a cold-booted VM always runs the latest code.
- **Dropbox**: nothing automated reads lossless files from Dropbox. The
lossless **MKV** is still uploaded because the completion email and the
live Fiverr auto-reply bot both tell customers it is in the Dropbox
folder, and the VocalStar tenant expects lossless files. To drop the MKV
as well, update those two texts first, then add the MKV suffix to
`DROPBOX_SKIP_OUTPUT_SUFFIXES`.

## Testing
- New/updated unit tests: preemption → lost job → resubmit, NotFound
candidate skip, family ranking, idle 5-min/heartbeat window, fleet-shape
invariants, worker-path coverage, Dropbox exclusion, keep-alive hook
(Jest, fake timers).
- `pytest backend/tests --ignore=backend/tests/emulator`: 4802 passed.
The only failures are 6 tests that need the spaCy `en_core_web_sm`
model, which isn't installed in my local venv; unrelated to this PR.
- Jest lyrics-review + lib/lyrics-review: 423 passed. Infra tests
(`test_encoding_worker_config.py`, idle function): all pass.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-26 23:20:54 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 8cbcc03099 infra: GCP cost cuts (VPC connector, min-instances, Autoclass, weekly incremental git backup, hourly error monitor, PITR off) (#1063)
## Summary
These are the cost cuts Andrew approved to get the whole GCP project
under $300/mo. Performance compromises were accepted.

- **VPC connector removed.** `cloud-run-connector` only existed to reach
the old flacfetch GCE VM's private IP. flacfetch now runs on Netcup
behind a public HTTPS URL. I checked every URL secret the backend uses
and none are private IPs. The backend is detached with
`--clear-vpc-connector`: just dropping the `--vpc-connector` flag from
`gcloud run deploy` would keep the old setting. `audio-download-job` and
`bulk-search-job` no longer set `vpc_access`. Saves ~$12/mo plus the
Network Intelligence resource-hours.
- **`karaoke-backend` min-instances 2 → 1** (ci.yml).
- **Autoclass** on `karaoke-gen-storage-nomadkaraoke` (terminal
NEARLINE, because customers re-download old jobs) and on
`nomadkaraoke-divebar-files` (terminal ARCHIVE). Autoclass charges no
retrieval or early-deletion fees.
- **backup-to-aws**
- **Git bundles:** now weekly (Sundays) and incremental. A repo is
re-bundled only when its GitHub `pushed_at` is newer than its bundle's
S3 `LastModified` minus a 1h lag, so S3 itself is the state store. Every
repo is re-bundled on the first Sunday of each month.
`?mode=git_repos[&full=1]` runs the git step alone.
- **Bug fix:** since #997 every S3 **overwrite** has failed. The
`backup-writer` IAM user has no `s3:GetObject`, so HEAD on an existing
key returns 403. Git bundles and the manifest were stuck at 09-15, and
the DR freshness monitor has been red every day. The existence check now
uses `ListObjectsV2`.
- **`gcs_sync`:** detects changes by `time_created` instead of
`updated`. Autoclass transitions bump `updated`, which would have made
the sync re-copy every object and pull it back to Standard.
- **DR freshness monitor:** `git-repos/` is now judged on
`manifest.json` (192h) plus a check of the manifest's contents.
- **Error monitor:** runs hourly instead of every 15 min. Lookback is 60
min, equal to the interval, so there are no gaps or double counts.
`MAX_LOG_ENTRIES` goes from 500 to 2000.
- **Firestore PITR disabled.** The nightly export to GCS staging
remains, plus the weekly copy to S3.
- **`recover-stuck-downloads` / `retry-pending-render-jobs`:** every 5 →
10 min. Both are recovery sweeps with ~10-min stall thresholds.
- Docs updated: DISASTER-RECOVERY (PITR restore replaced, git RPO,
incident write-up), ARCHITECTURE, LESSONS-LEARNED,
GCP-COST-OPTIMIZATION, TROUBLESHOOTING.

**Kept:** the `audio-separator` GPU service. Its request logs show ~40
external `/separate` calls/month from remote-CLI users. It's already
min-instances 0.

## Already applied to prod (before merge)
- `gcloud run services update karaoke-backend --clear-vpc-connector
--min-instances 1`, same image. Health is OK, and
`/api/health/flacfetch` and `/api/health/encoding-worker` are both OK
without the connector.
- Targeted `pulumi up` on the Firestore DB, both buckets, both jobs and
both 5-min schedulers.
- backup-to-aws redeployed with `deploy.sh`. A manual `?mode=git_repos`
run bundled 13 of 121 repos (106 unchanged, 2 skipped) and uploaded 120
files with 0 errors.
- Deleted `gs://…/backups/flacfetch-torrent-backup-20260122-1931.tar.gz`
(1 GB).

## After merge
Targeted `pulumi up` for `error-monitor-scheduler`, which must wait for
the image with LOOKBACK=60, then delete the connector and drop the
`vpcaccess` API service from state (`disable_on_destroy=False`, so the
API itself stays enabled).

## Tests
`make test` passes locally. The backup_to_aws function tests (43,
including new incremental, list-based-existence and gcs_sync tests) were
run separately because CI doesn't run them.

Related: nomadkaraoke/karaoke-decide#140 (ARCHIVE lifecycle on
`nomadkaraoke-data`, already applied).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Updates**
* Pending-render retries and stuck-download recovery now run every 10
minutes instead of every 5 minutes.
  * Error monitoring now runs hourly and checks a 60-minute window.
* Cloud Run candidate deployments use one minimum instance instead of
two and no longer use a VPC connector.

* **Backups and Recovery**
* Git repository backups now run weekly, with monthly full refreshes and
incremental updates between them.
* Firestore point-in-time recovery is disabled; recovery uses nightly
exports.
  * Storage buckets now use automatic storage-class management.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-26 22:59:06 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 7179e2863f ci: move test/package jobs to GitHub-hosted runners; GPU base image same-region (cost) (#1059)
## Summary
GCP cost cuts in CI (approved):

1. **GitHub-hosted runners for test/package jobs** (~\$28/mo of
`gha-general-*` VMs, ~390 VMs/30d). Moved to `ubuntu-latest` (free,
public repo): `package-unit-tests`, `package-integration-tests`,
`package-build-test`, `backend-unit-tests`, `backend-emulator-tests`,
`package-build` (main), `frontend-e2e-full` (label-gated). None used GCP
auth. Self-hosted disk checks replaced by a free-disk-space step (drop
dotnet/android/ghc/CodeQL); venv cache keys renamed so self-hosted venvs
aren't restored onto a different interpreter path.
2. **Removed `propagate-base-image`** — pre-pulled the base image onto
"the 3 runners"; runners are now single-use ephemeral VMs so this just
spawned 3 VMs per base rebuild for nothing.
3. **AR inter-region egress** (~\$12-16/mo): the GPU app image was built
on a us-central1 runner `FROM` the ~13 GB GPU base in **us-east4**,
pulling it cross-region on every deploy. The GPU base + all BuildKit
caches now live in us-central1 `karaoke-repo`; only the final app image
is pushed to us-east4 (`karaoke-backend-gpu`) where
`audio-separation-job` pulls it same-region. Push uploads only the small
changed app layers (base layer digests are identical). First deploy
copies the current base from us-east4 via `buildx imagetools create`
(byte-identical, no from-scratch rebuild); falls back to a rebuild only
if that fails.
4. `build-runner-images.yml`: `KEEP_IMAGES_PER_FAMILY` 2 → 1
(GPU/windows images are 15-21 GiB).
5. `ci.yml` changes now trigger the package/backend test jobs (so this
PR exercises the moved jobs).

`deploy-backend` stays on the `docker-build` GCE runner (Docker builds,
WIF, same-region AR). GPU runners untouched.

## Testing
- This PR's CI runs the moved jobs on hosted runners.
- `image_hash` helper + `imagetools create` cross-registry copy verified
locally against two `registry:2` instances (copy wraps in a single-entry
index; helper follows it).
- actionlint: no new findings (fewer than before).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Chores**
* CI test, package, and frontend end-to-end jobs now run on
GitHub-hosted runners. GPU image builds use a centralized base-image
cache, while application images continue to be published in their
existing region.
* Runner image cleanup retains the newest image per family; images
backing existing disks remain protected.
* **Documentation**
* Updated the runner guide to describe which jobs use hosted or
self-hosted runners and how image retention works.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-26 22:34:09 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 f2b633175e chore(infra): cut encoding-worker disk cost (~$75/mo) + fix runner image pruning (#1055)
## Summary

This cuts the GCP cost of the encoding-worker VM disks and custom images
by about **$75/mo**. Functionality and instance-type diversity are
unchanged.

| Lever | Change | Est. saving |
|---|---|---|
| Boot disk size | All 10 workers go from 100 GB to **50 GB**. The
Packer image `disk_size` goes from 100 to 30, because a disk can't be
smaller than its source image. | ~$44/mo |
| Hyperdisk IOPS/throughput | `-c4a` and `-n4db` were at 3600 IOPS / 290
MB/s, the paid defaults. They are now at the free baseline of 3000 /
140. New hyperdisks are created at 3000 / 140. | ~$18/mo |
| Custom images | Deleted 23 old runner images, keeping the newest 2 per
family. Fixed the prune step in `build-runner-images.yml` so they don't
pile up again. | ~$13/mo |

Measurements behind the sizing:
- A worker 8 months into service used 15 GB of 100 GB. Fresh disks use 8
GB.
- Job-folder scratch (1,904 jobs): p99 1.3 GB, max 4 GB. The deepest
queue seen on a worker was 8.

Full analysis, benchmarks and the rollout procedure are in
`docs/archive/2026-09-26-encoding-worker-disk-cost.md`.

## Runner image prune bug
The monthly step filtered with `-deprecated.state:*`, which is invalid.
gcloud wrote the error to stderr, stdout was empty, and the step printed
"Nothing to deprecate" every month. Deprecating wouldn't have saved
anything anyway, since deprecated images are still billed. Also,
gcloud's `family=gha-runner-gpu` filter is a word-boundary regex, so it
matches `gha-runner-gpu-windows` too. The step now matches the family
exactly on the client side, deletes images beyond the newest 2, and
never deletes an image that an existing disk was created from.

## Rollout (stockout-safe)
Pulumi can't shrink a boot disk. The only way it would get there is
replacing the VM, which needs c4d/n4d/c2d capacity. So each **stopped**
VM keeps running as the same instance, and only its boot disk is swapped
with gcloud: detach, delete, create a 50 GB disk from the new image,
attach it as boot. Then `pulumi refresh --target` on that instance.
After that, `pulumi preview` shows no diff for the swapped VMs.

## Testing
- `pytest infrastructure/test_encoding_worker_config.py`: 8 passed. A
new invariant checks that the Packer image size is ≤ the worker disk
size and that the worker disk is ≥ 40 GB.
- Each swapped VM was started and checked with `/health` and a real full
4K `/encode`, and `df` was checked afterwards. Before/after timings are
in the archive doc. c2df was the same before and after (142/139 s →
142/137 s), and n2f matched the August benchmark (247 s).

⚠️ Do NOT merge until every VM is swapped and refreshed. Merging with
`size=50` while a 100 GB VM is still in Pulumi state would make a later
`pulumi up` try to **replace** `encoding-worker-a/-b`. Status is tracked
in the comments.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Cost Optimization**
* Reduced encoding-worker boot disks from 100 GB to 50 GB, lowering
estimated stopped-VM disk costs to about $4–5 per month.
  * Reduced the worker image disk size to 30 GB.

* **Infrastructure Maintenance**
* Image cleanup keeps the two newest active images per family and
deletes older ones, while preserving images used by existing disks.

* **Documentation**
* Updated infrastructure guidance with disk sizing, image cleanup, and
cost details. Added a report with rollout, verification, and rollback
information.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-26 16:39:16 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 e9465c73d4 chore: gemini-3.8-flash everywhere + auto-correct on Opus 5.5 (cost) (#1054)
## Why
GCP credits expired 2026-09-19; Vertex Gemini 3.1 Pro spend had grown $9
→ $37 → $105/30d. `gemini-3.8-flash` is $0.75/$3.75 per 1M (vs $2/$12
for 3.1 Pro; rises to $1.50/$7.50 on 2027-01-01). Opus 5.5 is $4/$20 (vs
$5/$25 for 4.8).

## Changes
- All Gemini defaults → `gemini-3.8-flash`: credit eval, custom lyrics,
auto-correct (default + compare leg), match judge, parse titles, tenant
bulk analyze, agentic AI router, error-monitor LLM analysis (also
`us-central1` → `global`, required for Gemini 3.x), frontend
`translate.py`.
- Auto-correct Claude leg `claude-opus-4-8` → `claude-opus-5-5` (ci.yml
env), with `effort: "high"` pinned (5.5 defaults to medium). Pricing
table gains both new models.
- Docs updated; version 0.241.1.

## Evidence (12 recent prod jobs, pre-review lyrics scored against the
human-reviewed final)
| Model | Precision | Word-level recall | $/job |
|---|---|---|---|
| gemini-3.1-pro-preview | 91% | 74% | $0.067 |
| gemini-3.8-flash | 98% | 38–41% | $0.020 |
| gemini-3.8-flash @ high thinking | 97% | 60% (4/12 timed out, 504) |
$0.041 |
| claude-opus-4-8 | 94% | 33% | $0.080 |
| claude-opus-5-5 (effort high) | 93% | 58% | $0.048 |

Flash for the Gemini leg is a deliberate cost/recall trade-off (Andrew
chose it); Opus 5.5 more than compensates. Also verified:
`gemini-3.8-flash` works on Vertex `global` (rejects
`thinking_level=minimal`; nothing here uses it), match-judge latency
median 3.3s / max 6.2s (< 12s timeout), Japanese translation smoke test
preserves ICU placeholders. Translation cache is keyed on English text +
locale, so no re-translation churn.

Tests: auto_correct, credit eval, custom lyrics, parse_titles,
error_monitor, agentic router suites pass locally.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* AI-powered correction, translation, matching, title parsing, and
evaluation now use updated models, including Gemini 3.8 Flash and Claude
Opus 5.5.
* **Updates**
* AI service pricing reflects scheduled rate changes, including a Gemini
3.8 Flash rate change beginning January 1, 2027.
  * Application version updated to 0.241.1.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-26 14:53:04 -04:00
Andrew Beveridge ☄️andClaude Opus 5.5 d570b1f139 feat(kjbox): partner API for singer email-code sign-in + show credits (v0.239.0) (#1051)
## Summary
Partner API so the kjbox singer UI (live karaoke nights) can turn
patrons into real karaoke-gen customers: the singer verifies their email
with a 6-digit code typed into the kjbox page, and kjbox then searches
audio / creates the job **as that user** (normal gen delivery emails).

All under `/api/kjbox`, gated by `X-Kjbox-Secret` (Secret Manager
`kjbox-partner-secret`, wired in ci.yml; 503 when unset):
- `POST /auth/send-code` — emails a 6-digit code (hashed at rest, 10-min
expiry, new code supersedes old). Exempt from the 2-signups-per-IP cap
(a venue shares one IP) — replaced by a partner-wide 100 new accounts /
24h cap + 5 codes / email / hour. New users tagged `signup_source=kjbox`
(+ venue).
- `POST /auth/verify-code` — 5 attempts then burned, single-use; shares
the post-login path with magic-link verify (refactored into
`complete_verified_login`: welcome credit w/ AI eval, locale, referral,
session, welcome email).
- `POST /credits/show-credit` (+ user Bearer) — +1 credit, no email
("free at Andrew's shows"), idempotent per key, 5 / user / 24h.
- Review-needed + 24h reminder emails for `X-Client-Id: kjbox*` jobs
carry a one-click sign-in link to the review page (singers never signed
in on gen's site).

Paired kjbox PR: nomadkaraoke/kjbox (singer make-it flow, v0.116.0).
Plan: kjbox `docs/archive/2026-09-24-singer-make-it-auto-flow-plan.md`.

## Testing
- New: `backend/tests/test_kjbox_partner_api.py`,
`backend/tests/emulator/test_kjbox_partner_service.py` (+ additions to
existing tests for the verify refactor, review-email links, X-Client-Id
on create-from-search)
- Backend unit suite: 4644 passed; emulator suite passed.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added partner sign-in with one-time email codes and show-credit
claiming, with limits on code requests and credit grants to prevent
repeated or duplicate claims.
* Added localized sign-in emails and single-use review links for
eligible reminders.
  * Sign-in supports welcome-credit evaluation and referral attribution.
* **Improvements**
* Review reminders can use custom links, and new accounts can record
signup source and venue.
  * Credential-like request headers are redacted from stored metadata.
* **Documentation**
  * Added partner API usage and behavior details.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-26 00:03:10 -04:00
Andrew Beveridge ☄️andClaude Fable 5 7b4af11362 perf: cut backend cold start ~16s → ~4s via lazy imports + background warmup (v0.235.0) (#1036)
## Summary

A freshly-spawned `karaoke-backend` Cloud Run instance took **~16s
before serving its first request** — Cloud Run holds routed requests
during boot, so every scale-out was a 16s latency cliff (measured
2026-09-22 during the 16-tab load test: warm instance 1.2s vs fresh
instances 15.8–18.8s for the same request). Handoff spec:
`docs/archive/2026-09-22-gen-cold-start-reduction-handoff.md`
(workspace).

Measured breakdown: **~8–9s Python imports** + **~7.5s lifespan
preloads** (spaCy ~1.0s, NLTK cmudict+punkt ~2.3s, Langfuse ~3s,
credential validation ~1.6s — all gating readiness, from prod logs).

## Changes

**Lazy imports, fixed at the source module** so every importer benefits
and the CLI keeps working (imports now happen at first use inside worker
code paths):

| Module | Deferred | Cost removed |
|---|---|---|
| `karaoke_gen/.../utils/syllable_counter.py` | spacy, nltk, pyphen,
syllables → `__init__`/methods | ~2.6s |
| `karaoke_gen/.../correction/phrase_analyzer.py` | spacy → `__init__`
(was also pulling **torch** via thinc) | ~1.7s |
| `karaoke_gen/audio_processor.py` | audio_separator
(torch/onnxruntime/librosa), pydub | ~2.9s |
| `backend/config.py` | google.cloud.secretmanager → `get_secret()` |
~1.7s |
| `backend/services/{custom_lyrics,auto_correct}/service.py` |
google.genai → `_call_gemini()` | ~1.2s |
| audioshake/whisper transcribers, cdgmaker composer | pydub → using
methods | small |

`audio_processor` keeps `Separator` / `AudioSeparatorAPIClient` /
`REMOTE_API_AVAILABLE` / `SEPARATOR_AVAILABLE` as (lazy) module
attributes via PEP 562 `__getattr__`, so existing test patch targets are
unchanged; intra-module code uses helpers because LOAD_GLOBAL bypasses
module `__getattr__`. Same pattern keeps `custom_lyrics.service.genai`
patchable.

**Lifespan** (`backend/main.py`): the spaCy/NLTK/Langfuse preloads +
credential validation moved into a daemon thread
(`_run_background_warmup`). They warm caches for the *first
lyrics-processing job* — they never needed to gate HTTP readiness.
Workers racing the warmup fall back to the preloaders' existing lazy
paths. Heavy imports happen in this thread, not in async handlers
(event-loop-blocking gotcha from v0.230.0/v0.231.1).

## Results

- Local `import backend.main`: **10.63s → 2.75s** (remaining is
dominated by `google.cloud.firestore` ~0.8s, a genuine serving
dependency)
- Lifespan startup: ~7.5s → instant ("Application startup complete"
same-second)
- Expected prod cold start: **~16s → ~3–4s** (to be verified post-deploy
with `scripts/load-test-review.py --tabs 16` + fresh-instance request
latency)

## Tests

- **New:** `backend/tests/test_cold_start_imports.py` —
subprocess-imports `backend.main`, asserts none of the heavy libraries
(torch, spacy, nltk, audio_separator, matplotlib, pydub, google.genai,
secretmanager, librosa, onnxruntime) leak into `sys.modules`;
`backend/tests/test_main_startup_warmup.py` — warmup calls all preloads,
survives individual failures, and doesn't block readiness.
- Full backend + package suites: **4541 passed** (incl. the
audio_processor/custom_lyrics/auto_correct patch-target suites). Ruff:
no new findings on changed files.

## Follow-up (separate PR after prod verification)

If fresh-instance first-request latency lands <5s, drop `--min-instances
2 → 1` in `ci.yml` — the second warm instance was the cold-start
mitigation (~$15/mo).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **Performance**
- Reduced backend cold-start time by deferring heavy dependencies until
they are needed.
- Startup readiness is no longer blocked by model, language-resource,
analytics, or credential warmups; these now run in the background.
- Secret Manager and audio, transcription, lyric, and correction tools
load on demand.

- **Documentation**
- Added guidance and measured results for diagnosing and reducing
backend cold starts.

- **Quality**
- Added automated checks to ensure startup remains lightweight and
background warmups continue safely after individual failures.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-22 21:32:43 -04:00
Andrew Beveridge ☄️andClaude Fable 5 3d1ee6a621 feat: KaraokeHunt retired-app request interceptor — auto-convert to Gen users (v0.235.0) (#1037)
## Summary

The retired KaraokeHunt mobile app still has installs submitting ~5-10
song requests/month. Each request POSTs to the dead
`create.karaokehunt.com`, fires a client-side Pushbullet push to Andrew,
and shows the user a hardcoded *"you should receive an email in 5-10
minutes with a link to your new karaoke video"* — a promise broken for
years. This PR makes it true and converts those users into Gen users
(Andrew's spec, BACKLOG Frog #2 extension).

## How

- **Cloudflare Worker**
(`infrastructure/cloudflare/karaokehunt-interceptor/`, deployed via
`deploy.sh` — the karaokehunt.com zone is not IaC-managed): answers the
app instantly, `waitUntil`-forwards the payload to the new backend
intake with a shared secret.
- **`POST /api/karaokehunt/request`**: unauthenticated intake, gated by
`X-KH-Forwarder-Secret` (Secret Manager `karaokehunt-forwarder-secret`,
already created; wired in ci.yml `--set-secrets`), 10/min/IP rate limit,
lenient payload parsing, Firestore `karaokehunt_requests` audit trail.
- **Conversion** (`backend/workers/karaokehunt_conversion.py`, mirrors
`community_daily_pick`):
- new email → account + 1 credit (`karaokehunt_app_conversion`) → job
created **as them** (consumes it); first sign-in still grants the
welcome credit (deliberate — next song covered)
- existing user with credits → job on their credit (email offers to
restore it on reply)
- no credits, or over `KARAOKEHUNT_DAILY_JOB_CAP` (default 3/day, bounds
unattended spend) → free community requests board (dedup → upvote)
- conservative `pick_auto_selection` (not `select_best`): unconfident
matches park in `AWAITING_AUDIO_SELECTION` and the email asks them to
pick — garbage app inputs must never auto-publish to YouTube
- 14-day per-email+song dedup; failures retryable via admin `POST
/api/karaokehunt/internal/reprocess/{doc_id}`
- **Email** `send_karaokehunt_conversion`: 3 variants (job started /
pick the recording / on the board), 7-day one-click sign-in link,
KaraokeNerds community-version bonus link when one exists. EN + all 32
locales via translate.py.

## Safety

Deploys **dark**: endpoint 503s until the secret env lands, and receives
no traffic until the Worker + DNS flip (done post-merge via
`deploy.sh`). Kill switch = delete the Worker route; backend-side =
remove the secret mapping or `KARAOKEHUNT_DAILY_JOB_CAP=0`.

## Testing

- 36 new tests (`test_karaokehunt_intake.py`,
`test_karaokehunt_conversion.py`): secret/rate-limit/JSON gates, all
conversion routings, credit-grant guards, park-on-unconfident-search,
email variants + HTML escaping, reprocess auth.
- Full `make test` green (backend + frontend).
- Post-deploy: live end-to-end POST through `create.karaokehunt.com`
mimicking the app payload, verifying account/job/email/Firestore.

Plan doc: `docs/archive/2026-09-22-karaokehunt-app-interceptor-plan.md`

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **New Features**
- Requests submitted through the retired KaraokeHunt app are now
received and converted into Nomad Karaoke jobs, accounts, or
community-board requests.
- Users receive localized email updates with job status, credit usage,
audio-selection prompts, and tracking links.
  - Requests can be retried by administrators when processing fails.
- Duplicate protection, request limits, and daily processing caps help
prevent repeated or excessive submissions.
- **Documentation**
- Added API and deployment documentation for the request-forwarding
workflow.
- **Chores**
  - Released version 0.235.0.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-22 21:26:18 -04:00
Andrew Beveridge ☄️andClaude Fable 5 6514673fde perf: Cloud Run burst headroom — concurrency 32 + min-instances 2 (#1035)
## Summary

Final tuning from the 16-tab test diagnosis (#1033 fixed the edge rate
limit eating the requests; this fixes the remaining latency). With
nothing failing anymore, the warm 16-tab run still showed every tab at
~17–19s — server logs pinpointed why: the one warm instance answered
`correction-data` in **1.2s** while the three instances the burst
spawned answered in **15.8–18.8s** — Cloud Run holds requests during the
~16s app boot.

Concurrency 16 (set in #1028 while the event loop still had blocking
hazards) now overshoots: the loop is clean, one instance juggles dozens
of light requests fine, and forcing scale-out per burst just converts
load into cold-start waits.

## Change

`--concurrency 16 → 32` and `--min-instances 1 → 2`: 64 warm request
slots, so a 16-tab reload (~48 concurrent API calls) is absorbed with
zero cold starts. Still far below the original 80 that allowed the
2026-09-18 single-instance wedge.

## 💰 Cost note (Andrew)

The second min-instance idles at Cloud Run idle rates — roughly
**$15/month**. Single-flag revert if you'd rather not pay it; without it
the first burst after idle pays one ~16s cold-start wave for the
overflow beyond 32 concurrent slots.

## Follow-up worth considering (not in this PR)

App cold start is ~16s — heavy import graph (matplotlib, pydub, GCP
clients). Lazy-importing the heavy tails would benefit every scale-out.

## Testing

Deploy-flag-only change. Post-deploy: re-run
`scripts/load-test-review.py --tabs 16` — expect single-digit-second
tabs with no cold-start waves.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-22 18:26:41 -04:00
Andrew Beveridge ☄️andClaude Fable 5 30cf41e7d6 feat: concurrent-reliability phases 2-5 — Cloud Run shape, degradation telemetry, banner tuning, load-test guard (#1028)
## Summary

Phases 2–5 of the concurrent-load reliability initiative (Phase 1 =
#1024, verified live: the 10-tab meltdown went from 9/30 failed requests
to 30/30 OK). Verbatim spec:
`docs/archive/2026-09-17-concurrent-review-reliability-verbatim-record.md`;
plan:
`docs/archive/2026-09-18-concurrent-review-reliability-investigation-and-plan.md`.

## P2 — Cloud Run shape
- **`--concurrency 80 → 16`** for karaoke-backend: the service is one
uvicorn process per 2-vCPU instance, and review loads are heavy; at 80 a
burst never triggers scale-out and all lands on the single min-instance.
16 makes bursts fan out while staying well above normal per-instance
load. (Deploy flags live in ci.yml — the service is not Pulumi-managed.)
- **Local preview ffmpeg fallback bounded**
(`LOCAL_PREVIEW_MAX_CONCURRENCY=1`, 900s queue timeout, fast
RuntimeError after that): the GCE-outage fallback runs ffmpeg ON the API
instance; unbounded, a few concurrent Preview clicks during an encoder
outage pegged both vCPUs, stalled the health probe, and showed everyone
the orange banner (Andrew's banner-during-preview screenshot).

## P3 — degradation telemetry (the explicit ask)
- New **`POST /api/client-events`** (unauthenticated so review-token
users report too; 30/min/IP rate limit; bot-filtered; detail capped;
URLs query-stripped). Each event → Firestore `client_events` collection
**and** a structured `client_event type=…` log line, so "how often are
users seeing this?" is reviewable via Firestore aggregation or Cloud
Logging correlation.
- Frontend reporter `lib/degradation-events.ts` (60s per-type throttle,
localhost no-op, never throws), wired into:
- `BackendStatusBanner` → `banner_reconnecting` / `banner_unavailable`
with stall age, in-flight count, probe state
- review lyrics-load failure → `lyrics_load_failed` with message/status
- `VocalsAudioDataLoader` → `waveform_slow` (>20s to first strips) and
`waveform_failed` (gave up) with poll/retry counts

## P4 — banner tuning
- Escalating to the full "We're having trouble reaching our servers"
card now requires **two consecutive failed health probes**
(`UNAVAILABLE_PROBE_FAILURES=2`). One failed 4s probe against a
briefly-pegged instance shows only the calm "Reconnecting…" pill.
Single-failure behavior for the pill is unchanged, as is instant
clearing on recovery.

## P5 — regression guard
- **`scripts/load-test-review.py`** checked in: replays the 10-tab
incident (Firestore-ADC review tokens, curl request trios, pass/fail
thresholds, exit code). Documented in `docs/TESTING.md`. Validated
against prod post-Phase-1: 9/9 requests OK, tabs < 3s.

## Testing
- Backend: 6 new tests for client-events
(persist/validate/rate-limit/bot/fail-soft/detail-cap) + local-preview
bound test — full suite **4517 passed**.
- Frontend: new degradation-events suite, two-probe escalation +
debug-snapshot tests; one existing api-fetch test updated to the new
escalation contract — full suite **1298 passed**.
- Live validation of the load-test tool against prod (small 3-tab run):
PASS.

## Notes for review
- Cloud Run concurrency change means bursts now spin up extra instances
(billable during requests only; max-instances unchanged at 20).
- Firestore `client_events` has no TTL policy yet (needs a
gcloud/console action our read-only ADC can't do); volume is tiny
(throttled per client) — can add later if it ever matters.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Added automatic reporting for degraded experiences, including
reconnecting or unavailable backend status, failed lyrics loads, and
slow or failed waveform loads.
- Added concurrent review-page load testing to help verify performance
under multiple open tabs.

- **Improvements**
- Backend status now remains in “reconnecting” until multiple health
checks fail, reducing premature unavailable messages.
- Local preview processing is limited during heavy demand to improve
stability.

- **Documentation**
- Added API documentation for client telemetry and guidance for running
concurrent-load regression tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-19 12:48:21 -04:00
Andrew Beveridge ☄️andClaude Fable 5 3cdb62a058 fix: run render-video worker as Cloud Run Job to survive backend deploys (#994)
## Summary

Tenant E2E run #107 (`singa`) failed with job `41e06b90` frozen at
`rendering_video` — even though the GCE encoder **finished the render
and uploaded every output**. Root cause: a backend deploy landed
mid-render and killed the render poll, which ran as a FastAPI
BackgroundTask on the Cloud Run service.

Full investigation:
`docs/archive/2026-09-13-render-deploy-survival-investigation.md`

### The failure chain (4 layers, each structurally broken)

1. **BackgroundTask execution** — the render endpoint returns 200
immediately and polls post-response; Cloud Run SIGKILLs old-revision
instances ~10s after SIGTERM on every deploy.
2. **Unreachable shutdown parking** — uvicorn waits for background tasks
*before* running lifespan shutdown, so
`park_active_render_jobs_for_shutdown()` never runs in exactly the
scenario it was written for. Its comment claimed a 600s grace period;
Cloud Run gives 10s. Verified in logs: the old revision's last lines
were uvicorn's `Waiting for background tasks to complete.` — then
SIGKILL.
3. **Cloud Tasks retry defeated** — the queue got its 200 at dispatch;
`dispatch_deadline=1800s` protects nothing.
4. **45-min stuck sweep** — reachable but slower than the E2E's 30-min
budget and a ~50-60 min silent stall + wasted re-render for a real
customer, on every deploy that lands mid-render (deploys go out on every
merge to main).

This is the **identical bug class fixed for the video worker in
2026-03** (`docs/archive/2026-03-08-fix-encoding-deploy-survival.md`,
`USE_CLOUD_RUN_JOBS_FOR_VIDEO`) — the render worker was never migrated.

### Changes

- **Route `trigger_render_video_worker` through a Cloud Run Job** —
reuses `video-encoding-job` with args override `python -m
backend.workers.render_video_worker --job-id <id>`, behind
`USE_CLOUD_RUN_JOBS_FOR_RENDER` (enabled in deploy env). Job executions
run to completion; deploys never touch them.
- **New CLI entrypoint** in `render_video_worker.py`. Exit-code contract
differs from video_worker deliberately: clean returns (True AND False)
exit 0 because `process_render_video` parks/fails/supersedes internally;
only an escaped crash exits 1 (job template has `max_retries=2`).
- **Worker-side idempotency gate** (review finding): the CRJ path
bypasses the HTTP endpoint's `_check_worker_idempotency`, and CRJ
retries could re-enter a job in `RENDERING_VIDEO` (hard-failing it via
the same-state transition guard) or resurrect a terminal job. The worker
now skips terminal + already-rendering jobs cleanly regardless of
dispatch path.
- **Surface failed video-worker dispatch** (review finding): the ignored
`trigger_video_worker` return after render completion now logs at ERROR
instead of silently stalling the job at `instrumental_selected`.
- **Pulumi**: `video-encoding-job` env gains both
`USE_CLOUD_RUN_JOBS_FOR_*` flags so the render execution's in-process
video-worker trigger also dispatches a CRJ. **Already applied to prod**
(`pulumi up` run locally, verified on the job).
- **`main.py`**: corrected the false 600s comment; shrank the
unreachable 480s lifespan wait to 5s. Parking hook kept as
legacy/flag-off net.

### Rollback

Remove `USE_CLOUD_RUN_JOBS_FOR_RENDER=true` from the service env → falls
back to the legacy Cloud Tasks path.

## Testing

- Backend: 4422 unit + 151 emulator tests pass locally.
- New tests: CRJ routing (flag on/off), CLI exit-code contract,
worker-side idempotency gate (terminal skip, already-rendering skip).
- Post-deploy: will dispatch the Tenant Portals E2E workflow (the
failing scenario) against prod.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Video rendering now runs as durable background jobs, allowing renders
to continue through service deployments.
- Added safeguards to prevent duplicate processing and safely handle
completed or in-progress jobs.
- Improved failure reporting when a video-processing job cannot be
started.

- **Bug Fixes**
- Fixed render jobs becoming stuck when deployments interrupted
background processing.

- **Documentation**
- Added troubleshooting guidance and incident lessons for interrupted or
stuck renders.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-13 03:02:15 -04:00
Andrew Beveridge ☄️andClaude Opus 4.8 d93dbddd48 ci: reliable R1 canary version-gate + retry (stop false pages) (#988)
Follow-up to the incident-hardening initiative — fixes the false
"Post-deploy canary FAILED (v0.222.3)" page from the #987 deploy.

**Root cause:** the R1 version-change gate did a single un-retried curl
to the raw Cloud Run **origin** `/api/health/detailed`, which came back
empty (cold start on the scaled-to-zero prior revision and/or origin
edge-auth on the detailed endpoint). It logged `Currently-serving
version: <unknown>` and **fail-opened → ran the canary on a no-bump
deploy**, which then flake-failed on a review-page timing wait and
paged.

**Fixes:**
1. **Reliable gate** — read the currently-serving version via the
**edge** (`https://api.nomadkaraoke.com/api/health/detailed`, warm +
Cloudflare-fronted, verified to return the version) with retries. A
no-bump deploy now correctly **skips** the canary.
2. **Retry the canary** — `--retries=2` so a single flaky UI wait
doesn't page; only a failure surviving retries is real. (The
happy-path's review-page waits are inherently flaky; the identical spec
passed on the 0.223.2 deploy.)

No app code changed → no version bump. YAML validated.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-09-12 23:22:10 +00:00
Andrew Beveridge ☄️andClaude Opus 4.8 5b5264dc54 Incident hardening: deploy/CI/runner resilience (R1/R2/R3/R4/C1/C2) (#987)
Part 3 of 3 of the **Incident hardening (post-NOMAD-1632)** initiative
(plan: `docs/archive/2026-09-12-incident-hardening-plan.md`). Closes the
deploy/CI/runner gaps: no functional gate, no codified rollback, bad
revision serves 100% immediately, fire-and-forget runner stranding, and
a flaky smoke. **No backend app code changed → no version bump.**

## ⚠️ Reviewer note — read first
**R3/R4 rewrites the live `deploy-backend` step.** It's designed to be
*auto-revert-safe* (a failed candidate smoke means we simply never
migrate traffic, so the prior revision keeps serving 100% — a bad
revision serves **0%**, not an outage). But it changes how **every**
backend deploy works and **can't be fully exercised in CI**. This is the
one to watch on first merge. If you'd rather de-risk, R3/R4 is the only
item here that's safe to split out — the rest (R1/R2/C1/C2) are
independent and lower-risk.

## R2 — codified fast rollback (`scripts/rollback.sh` + runbook)
Dry-run-able, explicit-target-required rollback for the Cloud Run
**service** (traffic-shift to a prior revision — instant, no rebuild)
and the three **jobs** (image re-pin, correct per-job region/repo —
verified against ci.yml). `--list` shows available revisions/tags;
`--dry-run` prints every gcloud command. `docs/TROUBLESHOOTING.md` "Fast
rollback" runbook added as the first section.

## R3/R4 — service traffic-canary
`deploy-backend` now deploys `--no-traffic --tag candidate`, smokes the
tagged candidate URL (health + expected version, with retries), then
`update-traffic --to-latest` **only on pass**. Cloud Run Jobs can't
traffic-split (noted inline) → they rely on R1 + R2.

## R1 — post-deploy functional canary, version-change-gated
New `deploy-verify` job runs the real production happy-path
(`happy-path-real-user.spec.ts`, admin-credit variant — reuses
e2e-daily's impersonate token refresh, secret names verified identical)
after a backend deploy, **only when the backend version actually
changed** (new "Detect backend version change" step compares `poetry
version` to the currently-served `/api/health/detailed`).
Frontend/docs/infra-only merges short-circuit. On failure → Discord
alert + loud fail printing the exact `scripts/rollback.sh` invocation.
Per the plan's caveat, a fresh happy-path won't reproduce every incident
class (e.g. Failure B's stale-cache) — an admin reset+retry variant is a
noted follow-up.

## C1 — ephemeral-runner stranding fix
`infrastructure/functions/runner_manager/ephemeral.py` now confirms the
VM insert **operation** succeeded instead of fire-and-forget: a late
stockout/quota/transient error raises → existing zone-fallback / webhook
redelivery kicks in instead of silently stranding a queued job for ~15
min. An insert-confirm *timeout* is treated as "still provisioning" (not
a failure); guarded for older clients. **+4 tests (46 pass).**

## C2 — de-flake the E2E smoke
`networkidle` → `domcontentloaded` + explicit element waits (the
marketing page's background requests made `networkidle` time out and
redden real deploys).

## Validation (local only — no deploy)
- `ci.yml` + `e2e-daily.yml` parse; **actionlint clean** on new code.
- `rollback.sh`: `bash -n` + `shellcheck` clean; constants cross-checked
against ci.yml.
- `test_ephemeral.py`: **46 passed**.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-09-12 16:58:12 -04:00
Andrew Beveridge ☄️andClaude Opus 4.8 05f024bdd4 Requests voting board Phase 2: daily free-community-track automation (#977)
## Summary

Phase 2 of the requests voting board (`requests.nomadkaraoke.com`) — the
daily automation that turns the top-voted request into a free,
auto-generated, published karaoke video. **Deploys dark** behind a
kill-switch (`COMMUNITY_DAILY_PICK_ENABLED`, default off); the daily
Scheduler only shadow-logs until it's flipped on.

Picks up
`docs/archive/2026-09-03-requests-voting-board-phase2-handoff.md`.
Design + Andrew's decisions:
`docs/archive/2026-09-03-requests-board-phase2-plan.md`.

## What's included
- **Daily picker** (`community_daily_pick.py`, Scheduler noon US
Eastern, DST-aware): claims the UTC day with a create-only lock
(`daily_community_pick/{YYYY-MM-DD}` → **one free track/day, total**),
picks the top open request with net votes ≥ 0 (source-agnostic; skips
community-rejected), grants the requester a free credit, and submits the
job **as that user** (owned by them, credit consumed) via the same
search → auto-select → download path the web flow uses. Fully idempotent
(resumable per-day lock phase + per-request guard flags).
Shadow/`?dry_run=true` mode.
- **24h ownership handoff** (`community_handoff.py`, hourly): reassigns
a stale review to the next up-voter (compare-and-set guarded), up to **5
voters**, then parks the track (`stalled`). Community jobs excluded from
the normal stale-review auto-cancel.
- **Voter publish fan-out**: YouTube upload-queue hook marks the request
`published` and emails every up-voter (except the owner) a new "track
you voted for is live" email (33 locales). Retry-safe (per-voter success
tracking).
- **Two internal endpoints** + two Cloud Scheduler jobs (Pulumi).

## Andrew's decisions (via AskUserQuestion)
- Trigger: **noon US Eastern**. Trending-agent fallback source:
**deferred** to a follow-up. Net-negative requests: **skipped**. Handoff
cap: **5 voters then park**.

## Deferred / follow-ups (not in this PR)
- **Trending-agent fallback source** (auto-submit a candidate when the
board is empty) — deliberately deferred; must generate candidates the
careful way (karaoke-decide `candidates` + KaraokeNerds).
- **YouTube description board link** — left untouched to avoid
clobbering Andrew's parallel descriptions work; add
`requests.nomadkaraoke.com` to `default_youtube_description` once that
lands.

## Testing
- Worker unit tests (kill-switch shadow, empty board, day-lock
idempotency, credit-grant/job-create guards, handoff cap/stale/CAS,
voter fan-out retry-safety) + emulator tests for the new service methods
against real Firestore transactions.
- All touched existing suites (youtube queue, stale-review,
requests-board routes/logic) pass. 79 passed locally.

## Review
CodeRabbit reviewed locally (2 cycles completed; 2 major + 1 minor
findings all addressed — retry-safe fan-out, CAS owner assignment,
up-voter ordering). 3rd cycle hit the free-OSS rate limit; it would only
re-verify the applied fixes.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-09-03 01:34:31 -04:00
Andrew Beveridge ☄️andClaude Opus 4.8 3c58143f75 fix(review): re-sign instrumental preview URLs on expiry (refresh-on-error) (#973)
## Problem

Completing a combined review failed to play the instrumental in the
preview modal — toggling to **Instrumental** loaded nothing, with
`NS_ERROR_DOM_NETWORK_ERR` in DevTools (Firefox) on the
`review-audio/instrumental_with_backing.ogg` GCS request.

**Root cause:** the overlaid instrumental `<audio>` uses the signed GCS
URL baked into the review payload at page load, which has a **120-minute
expiry**. A long review session outlives it, so the stem fetch returns
`400 ExpiredToken` and playback silently fails. Confirmed by fetching
the exact URL from the report: `ExpiredToken — Request signature expired
at 2026-09-01T20:16:26Z` (page had been open ~2h11m).

Reloading didn't help because the review UI persists the full
`CorrectionData` — including `instrumental_options[].audio_url` — to
`localStorage` (`lyrics_analyzer_data`) and rehydrates it when the user
has unsaved edits, so the *same* expired URL came back.

The `[SW] Service worker registered` log is a red herring:
`frontend/public/sw.js` has no `fetch` handler.

## Fix (refresh-on-error)

- **Backend:** extract a shared `_build_instrumental_options` helper and
add `GET /api/review/{job_id}/instrumental-urls` returning
freshly-signed stem URLs.
- **Frontend:** on the stem `<audio>` `error` event, re-fetch fresh URLs
via the new endpoint and swap them in. The existing video↔audio sync
effect (keyed on the URL) resumes playback in place. Capped at 3
refreshes per modal open so a genuinely-broken stem (404) can't loop.

No new user-facing strings (the recovery is silent) → no i18n changes.

## Testing

- Backend: 3 new tests for the endpoint (both stems / single stem / 404)
— `TestInstrumentalUrlsRefresh`. Full `test_routes_review.py` +
`test_combined_review_flow.py` + `test_audio_transcoding_service.py` =
111 passed.
- Frontend: 3 new `PreviewVideoSection` tests (swap on error / cap at 3
/ no-op without refresh fn). Full unit suite = 1248 passed.
- CodeRabbit local review: no findings.

@coderabbitai ignore

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-09-01 17:21:35 -04:00
Andrew Beveridge ☄️andClaude Opus 4.8 37b9e6088b ci(pypi): recurring storage-cap prune reminder (fix failing PyPI publish) (#960)
## Problem

The `Deploy - Publish to PyPI` step on `main` has been failing:

```
HTTP Error 400: Project size too large. Limit for project 'karaoke-gen'
total size is 10 GB.
```

This is a recurrence of the known 10 GB PyPI project cap. The project is
at
**10.71 GB / 479 releases**; each release is a ~62 MiB universal wheel
(bundled
33-locale frontend) auto-published on every merge (~3 GB/month), so it
refills
the cap every few months. **Only the public `pip install` publish step
fails** —
Cloud Run and the GCE encoding worker (pulls the wheel from GCS) are
unaffected.

The prior fix (v0.187.5) was wheel-only publishing + a one-time manual
deletion,
which only bought ~2 months. This makes the cleanup **recurring and
durable**.

## Why not just automate deletion in CI?

PyPI has **no deletion API** (upload tokens / Trusted Publishing are
upload-only,
and 2FA is mandatory). Deletion is only possible from a logged-in
browser
session, which can't be safely/durably stored in CI. So we automate the
*decision + reminder* and keep the *deletion* a ~1-minute human step.

## What this adds

- **`scripts/prune_pypi_releases.py`** (stdlib-only) — reads the public
JSON API
and computes a retention plan: **keep last 60 days + newest release per
older
calendar month + latest overall**. Renders a table / JSON / Markdown
report and
a **paste-into-browser-console** deletion snippet. Never deletes
anything
  itself. Today's plan: keep 82, delete 397, **10.71 GB → 4.17 GB**.
- **`.github/workflows/pypi-prune-reminder.yml`** — monthly cron (+
manual
dispatch) that runs the planner and, once the project crosses a 7 GB
watermark,
opens/updates a single tracking issue with the plan + snippet. No
secrets.
- **`docs/PYPI-STORAGE-PRUNE.md`** runbook + a `TROUBLESHOOTING.md`
entry.
- **`tests/unit/test_prune_pypi_releases.py`** — 13 tests covering the
policy
(recency window, per-month keeper, latest-overall, same-day timestamp
ranking,
  empty/undated releases, sizes, rendering).

## Testing

- `pytest tests/unit/test_prune_pypi_releases.py` → 13 passed.
- Ran the planner against live PyPI data; generated JS validated with
`node --check`.
- `black` clean; workflow YAML validated; `should_nag` gate verified.

## Notes

- **No version bump** — tooling/CI/docs only, nothing in the wheel
changes.
Merging therefore **skips** the publish step (tag `v0.209.1` already
exists),
so this PR does not itself add another fat wheel while we're over the
cap.
- **To unblock publishing now:** after merge, run the prune (locally via
the
script or trigger the workflow with `workflow_dispatch`), paste the
snippet on
the logged-in PyPI releases page, then re-run the publish. The monthly
cron
  keeps it under the cap going forward.
- Related: filed
[pypi/support#12050](https://github.com/pypi/support/issues/12050)
requesting a 50 GB limit as a one-off buffer; the recurring prune is the
  durable fix.

@coderabbitai ignore

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-29 21:46:10 -04:00
Andrew Beveridge ☄️andClaude Fable 5 b45efbdbc5 feat: enable confident backing-keep enforcement in prod (#953)
## Summary

One-line deploy-env change: `AUTO_APPROVAL_BACKING_KEEP_ENABLED=true` on
the backend service. A confident WITH_BACKING verdict from the 3-stem
backing decider (#950) now auto-selects the with-backing instrumental,
letting jobs with confident lyrics AND clear backing vocals complete
fully automatically.

## Shadow validation evidence (why now)

- **20-job replay corpus**: 13/14 true keeps confidently retained (incl.
both jobs the human under-kept); catastrophic backing-is-lead case
gated; zero unsafe outcomes.
- **Live prod battery**: 7/7 keep verdicts on harmony tracks with
consistent signal profiles; the misclassified-lead gate fired correctly
on an adversarial shoegaze track; one job completed fully zero-touch end
to end.
- **Human review pass on 11 fresh jobs (2026-08-29)**: 10/11
instrumental picks matched the shadow verdict. The single
lyrics-auto-eligible job was shipped by the human with **zero manual
edits and the identical instrumental selection** full-auto would have
made. Every job needing real lyric work was correctly gated (up to 127
manual edits on gated jobs; zero false-autos).

Kill switch: set the var to false (or remove it — code default is
false). Per-job opt-out: `review_mode="always_review"`.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-29 01:17:31 -04:00
Andrew Beveridge ☄️andClaude Opus 4.8 8457a3bfca fix(ci): pin google-cloud-firestore in backend deploy (2.29.0 breaks deploys) (#941)
## Problem — backend deploys are broken

The **Deploy - Backend (Cloud Run)** job's *Read Firestore config* step
crashes:

```
google.api_core.exceptions.InvalidArgument: 400 Invalid database id %28default%29
```

`%28default%29` is URL-encoded `(default)`. **`google-cloud-firestore`
2.29.0** (published 2026-08-24T21:55Z) percent-encodes the default
database id in the resource path. The deploy step ran an **unpinned**
`pip install google-cloud-firestore`, so it started pulling 2.29.0 and
failing.

**Impact:** every backend deploy since 2.29.0 shipped has failed
(post-merge runs for `c36bcb08` and `7a3c1d66`). #939 (worker-trigger
retry) and #940 (job-list summary field) merged but **never reached
Cloud Run** — the active revision is still `00913` from #937.

## Fix

Pin the deploy step to `google-cloud-firestore==2.26.0` — the version
`pyproject.toml` and prod already run (the running backend reads
Firestore fine on 2.26.0). One line + a comment explaining why.

## Testing

- PR CI runs unit tests (deploy jobs are push-to-main only); the pin
doesn't affect them.
- On merge, the backend deploy will run with 2.26.0 and should succeed,
deploying main (0.199.3 = retry fix + summary field).
- Verified 2.29.0 is the regression via PyPI upload timestamps vs the
last good deploy (21:06, used 2.28.1).

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Updated the backend deployment to use a fixed Firestore library
version, improving deployment consistency and reliability.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-08-24 23:32:48 -04:00
Andrew Beveridge ☄️ 7729d30937 build: build frontend into the wheel in CI instead of committing out/ to git (supersedes #892) (#926)
Build the Next.js frontend into the wheel in CI instead of committing the generated out/ to git (fixes #876; supersedes #892). Validated: package-build-test green (npm ci → make build-frontend → poetry build → verify_wheel_frontend guard → install). Thanks @netux for surfacing it.
2026-08-23 01:19:11 -04:00
Andrew Beveridge ☄️andClaude Opus 4.8 6fa4400d3d feat(dr): back up all GitHub repos to S3 (survive a GitHub ban) (#929)
## Summary

Extends the nightly DR backup Cloud Function (`backup-to-aws`) to bundle
**every git repo under `github.com/nomadkaraoke` and
`github.com/beveradb`** into a per-repo `git bundle` on S3, so code +
full history survive a complete loss of GitHub access (e.g. an automated
copyright-strike account ban). Each bundle is directly cloneable —
restore is `git clone --mirror <bundle>` → `git push --mirror
<new-remote>`.

## Changes

- **`git_repos_export.py`** (new): enumerate repos via GitHub
`/user/repos` (includes private), mirror-clone + `git bundle create
--all`, upload to `git-repos/{owner}/{repo}.bundle` in staging, plus a
`manifest.json` inventory. Forks excluded, oversized/empty repos
skipped, per-repo errors non-fatal, org backed up **before** the
personal account for deadline resilience. PAT is read from Secret
Manager at **runtime** (not a deploy-time secret env var) so the
function deploys before the secret has a value and skips gracefully if
absent.
- **`main.py`**: new nightly Step 4b (between secrets export and S3
upload).
- **`infrastructure/modules/backup.py`**: bump function to 4Gi/2CPU
(mirror-clone uses in-memory `/tmp`); add `GIT_BACKUP_OWNERS` env.
- **`infrastructure/modules/secrets.py`**: new `github-backup-token`
secret (value added manually).
- **`.github/workflows/dr-backup-freshness.yml`**: monitor `git-repos/`
at 36h, counting only `.bundle` objects.
- **`docs/DISASTER-RECOVERY.md`**: new "Scenario 3: Loss of GitHub
access" + restore steps + token setup.

## Security

- The GitHub PAT is injected via a `GIT_CONFIG_*` `Authorization` header
(like `actions/checkout`), never in the clone URL/argv — so it can't
leak via `ps`, `TimeoutExpired`, or captured stderr. Token is also
redacted from any error text persisted into `manifest.json`.

## Testing

- 14 new unit tests for the module
(enumerate/filter/bundle/skip/error/security paths); full
`backup_to_aws` suite: **27 passing**.
- CodeRabbit local review (cycle 1) completed; all 4 findings (1
critical PAT leak, 2 major, 1 minor) addressed. Cycle-2 verification was
rate-limited (free-tier), so **left for the GitHub bot to review the
final diff**.

## Deploy notes

- **Pulumi already applied locally** (`github-backup-token` secret
created + function on 4Gi/2CPU) — CI re-run is a no-op.
- After merge, run `infrastructure/functions/backup_to_aws/deploy.sh` to
ship the new function source, then set the PAT:
`printf %s '<PAT>' | gcloud secrets versions add github-backup-token
--data-file=- --project=nomadkaraoke`
- Requires a classic GitHub PAT (owner: beveradb, member of nomadkaraoke
org) with `repo` + `read:org`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---
@coderabbitai ignore

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-23 00:44:52 -04:00
Andrew Beveridge ☄️andClaude Opus 4.8 c7959ea2d5 feat(encoding): mature multi-instance-type pool with availability-aware selection (#911)
## Summary
- Replaces "c4d primary + single n2 fallback family" with a **ranked
pool of 6 x86_64 / 32-vCPU / ≥32 GB highcpu types** across 4 silicon
lineages (c4d, c4, n4d, c2d, n2d, n2), selected **fastest-first** but
**availability-aware** — a type that stocks out is demoted for a 15-min
cooldown then re-probed. c4d stays the top pick whenever it can start; a
render no longer stalls even if 2+ types are fully stocked out in
us-central1.
- One **pure preference module**
(`encoding_worker_preference.ordered_candidates`) is now the single
source of truth for candidate ranking, driving **both** runtime
selection and deploy green selection so they can't drift.
- Keeps the per-VM start/stop model (MIG deferred with documented
triggers; no reservation). Full plan + rollout in
`docs/archive/2026-08-15-encoding-instance-type-diversity-plan.md`.

## Changes
- **New** `backend/services/encoding_worker_preference.py` — speed-rank
+ Firestore `capacity_state` cooldown; stdlib-only so `deploy_promote`
imports it (with a shadow-proof by-path fallback).
- **Runtime** (`encoding_service`, `encoding_worker_manager`): builder
ranks primary + fallbacks; manager records/clears per-(type,zone)
stockout; override routing now keys off an explicit `is_primary` flag
instead of list position.
- **Deploy** (`deploy_promote`, `ci.yml`): green selection uses the
shared ranker + threads `capacity_state`.
- **IaC** (`config.py`, tests): `FALLBACKS` broadened to 6 types
(additive-only) with family-correct disk types (Titanium c4/c4d/n4/n4d =
hyperdisk-balanced; c2d/n2/n2d = pd-balanced); secret schema gains
optional `machine_type`.
- **Packer** (`provision.sh`, `startup.sh`): bake full dep tree with
**CPU-only torch** so fresh type VMs boot self-sufficient; build-time
import verify.
- **New** `benchmark_types.py` to measure per-type encode wall-time and
refine the seeded `SPEED_RANK`.

## Testing
- [x] Full backend unit suite: **3871 passed, 36 skipped** (emulator
tests need external services)
- [x] New/updated tests: preference module (16), capacity_state +
`is_primary` manager tests, builder-ordering + non-list-JSON service
tests, deploy ranking tests, infra disk-capability + pool-breadth
invariants
- [ ] Operator rollout (needs write creds): `pulumi up` (verify +8
create, 0 replace) → rebuild image (dep-bake) → grow fallback secret
with `machine_type` → run `benchmark_types.py`

## Review
- [x] Local CodeRabbit review completed (4 major findings fixed; cycle 2
clean)
- [x] Review feedback addressed

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-15 20:52:56 -04:00
Andrew Beveridge ☄️ 9a752ab3eb fix(deploy): capacity-aware encoding-worker deploy (reach serving fallback in c4d stockout) (#910)
## Problem

Follow-up to #908/#909. The CI encoding-worker deploy is a blue-green
over the **c4d primary/secondary** VMs. us-central1 is in a persistent
**c4d Spot stockout**, so both are down and all traffic runs on an n2
fallback recorded as `active_override_vm`. The deploy can't start the
c4d secondary → rolls back → **never refreshes the serving fallback**,
so worker-side `main.py` changes (the #908/#909 fixes) never reach prod
until a manual restart. Evidence: the config doc's `primary_version` was
stuck at `0.192.9`, and I had to hand-start the n2f fallback to get
0.194.0 live.

## Fix — validate on a fresh green, then promote it (zero-downtime,
stockout-proof)

Decision logic is a pure, unit-tested module
(`infrastructure/encoding-worker/deploy_promote.py`) since the branching
can't be exercised in CI:
- **`select_green_candidates`** — try the c4d `secondary` first, else a
fresh n2 fallback (n2 before c4d), **excluding the current override** so
it keeps serving while we validate a *different* fresh worker.
- **`promote_plan`** — for a c4d green, the classic primary/secondary
swap **and clear any stale override**; for a fallback green, set it as
`active_override`. Returns the exact Firestore update + which retired
VM(s) to drain/stop (**never the just-promoted green**).

Workflow (`.github/workflows/ci.yml`):
- "Select & start green target" (capacity-aware, `continue-on-error`,
emits `green_ok`), health + real-encode-test on the green IP, then
"Promote green to serving" (apply plan + drained stop of retired VMs,
non-RUNNING skipped, one shared 20-min drain budget).
- Rollback uses `always()` so `deploy_in_progress` is always cleared;
only stops a green we started fresh, and **recovers the serving
override** (starts it back up) if a last-resort in-place restart left it
down.

## Testing
- `backend/tests/test_deploy_promote.py` — 10 cases: candidate ordering
(secondary-first, n2-before-c4d, override excluded), both promote
branches, and never-drain-the-green guard.
- `actionlint` clean on the new steps; YAML validated.
- Two full local CodeRabbit cycles; all 7 findings addressed (self-drain
guard, stuck `deploy_in_progress`, None-IP guard, drain time-bound,
override recovery).

## Deploy note
This PR's own merge is the first live run of the new logic (deploy runs
on every push to main). Expected: it selects an n2 fallback as green
(c4d won't start), validates it, sets `active_override` to the fresh
one, and drains the old override. Will be watched closely.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Improved encoding-worker deployments with capacity-aware green worker
selection and fallback support.
* Added safer blue-green promotion, health validation, job draining, and
rollback handling.
* Prevented active jobs from being interrupted during worker
replacement.

* **Bug Fixes**
* Added recovery for capacity shortages, including validated in-place
restarts as a last resort.
* Automatically clears stale deployment overrides and deployment locks.

* **Documentation**
* Updated troubleshooting guidance for worker restarts and
capacity-related recovery.

* **Tests**
* Added coverage for worker selection, promotion, fallback handling, and
draining behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

@coderabbitai ignore
2026-08-15 18:09:51 -04:00
Andrew Beveridge ☄️andClaude Opus 4.8 d9ecc0d0d0 Edge cutover complete: api.nomadkaraoke.com behind Cloudflare (WAF + origin lock enforced) (#888)
Completes the edge-security rollout started in #886. **The prod cutover
is done and validated** — `api.nomadkaraoke.com` now serves all traffic
through Cloudflare with WAF + rate-limiting, and the origin lock is
enforced.

## Cutover (applied via targeted `pulumi up`, validated live)
1. Imported the existing prod `api` DNS record (adopt, not duplicate)
2. Provisioned prod WAF/rate-limit/header rules while `api` stayed
DNS-only
3. Flipped `api` to proxied (the one reversible change)
4. `EDGE_AUTH_MODE` off → warn → **enforce** (validated safe via prod
E2E)

## Validation results (2026-07-20)
- `api.nomadkaraoke.com` → **200 + cf-ray**, exploit paths → **403**
(WAF), `/api/health` → 200, **CORS ok**, frontend loads, **0 backend
errors**
- **Daily prod E2E PASSED** (real Stripe payment + karaoke generation
through the edge)
- **0 edge-auth warnings** during the full E2E → enforce enabled
- Origin lock verified: direct-to-`*.run.app` non-exempt → **403**;
health/root exempt → 200

## Key finding: SSL "full" mode resolves the cert dragon
The Cloud Run managed cert can't provision behind Cloudflare (ACME
challenge not visible), **but the zone SSL mode is "full" (not strict)**
so Cloudflare doesn't validate the origin cert — the edge works and is
**renewal-safe** with no run.app Origin Rule / LB / cert wait. Rule:
keep zone SSL on "full", never "full (strict)".

## Free-plan constraints handled (in #886)
WAF `contains` not regex; rate-limit `period=10`+`cf.colo.id`; managed
OWASP off.

## Changes
- `config.py`: SSL-"full" note (replaces the reverted run.app-origin
config)
- `Pulumi.prod.yaml`: cutover config (`rolloutStage=prod`,
`proxyProdApi=true`, `proxyStaging=true`)
- `ci.yml`: `EDGE_AUTH_MODE=enforce` (persist the origin lock across
deploys)
- Plan §4 + runbook Phase B: the resolution + actual results

## Rollback
`pulumi config set edge:proxyProdApi false && pulumi up --target
…api-dns` (DNS-only within a TTL); `EDGE_AUTH_MODE=off` disables the
lock.

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-21 00:25:25 +00:00
Andrew Beveridge ☄️andClaude Opus 4.8 fb2bd6b108 Cloudflare edge security (WAF + rate limit + origin lock) for karaoke-backend (#886)
## Summary
Puts `api.nomadkaraoke.com` behind the Cloudflare edge that already
fronts the rest of the domain, after a vuln-scanner hammered the
directly-exposed Cloud Run origin on 2026-07-20 (`/.env`,
`/.ssh/id_rsa`, `wp-config.php`… — all 404'd, but the burst of 404s
tripped the "High Error Rate" alert). The API was the one
internet-facing surface with **no WAF, no rate limiting** — DNS-only
CNAME straight to `ghs.googlehosted.com`, Cloud Run ingress `all`.

**Zero-downtime by construction:** everything is gated behind
`edge:enabled` (default off) and staged via two independent switches
(`edge:rolloutStage` for rule scope, `edge:proxyProdApi` for the DNS
flip). The whole Cloud-Run×Cloudflare interaction is validated on a
throwaway staging host first; the only prod-affecting action is a
single, instantly-reversible DNS proxy flip. Full sequence in
`docs/archive/2026-07-20-edge-security-cutover-runbook.md`.

## Changes
- **`infrastructure/modules/edge_security.py`** (new) —
`pulumi_cloudflare` WAF (exploit-path block), per-IP rate limiting
(excludes `/api/internal/*`), origin-lock header transform, zone
security; staging + prod DNS records. Managed OWASP ruleset gated off
(zone is Free plan).
- **`backend/middleware/edge_auth.py`** (new) — off/warn/enforce origin
lock via a secret header Cloudflare injects; **fails closed (503) in
enforce** when the secret is missing; health/root exempt. +13 unit
tests.
- **`monitoring.py`** — retune the high-error-rate alert to **5xx-only**
(+ fix the rate-vs-ratio naming) so client/bot 404 bursts no longer
page.
- Wiring: `secrets.py` (edge-origin-secret container), `config.py`,
`__main__.py`, pinned `pulumi-cloudflare==6.18.0`, `ci.yml`
(`EDGE_AUTH_MODE=off` + secret ref), `Pulumi.prod.yaml` (encrypted
`edge:originSecret`).
- Docs: hardening plan + cutover runbook in `docs/archive/`.

## Already applied to prod (targeted `pulumi up`)
- ✅ `edge-origin-secret` Secret Manager container + v1 value
- ✅ 5xx error-rate alert retune (fixes the original paging issue
immediately)
- Everything else stays dormant: `edge:enabled=false`, backend
`EDGE_AUTH_MODE=off`.

## Testing
- [x] `make test` — full backend + frontend suite green (1085 frontend
tests; backend incl. 13 new edge-auth tests)
- [x] Cloudflare token verified against live API (DNS create+delete,
rulesets, zone settings)
- [x] `pulumi preview` clean (targeted apply of only the 2 additive
resources)
- [ ] Staging validation + prod cutover — follow the runbook (separate,
gated)

## Review
- [x] CodeRabbit CLI review completed locally; findings addressed
(fail-closed enforce, decoupled cutover, fixed origin-lock test,
threshold wording). One pre-existing PR_TITLE shell-interp finding in
the deploy step left out of scope.

## Notes for reviewer
- Nothing here touches prod behavior until someone sets `edge:enabled
true` and walks the runbook. The prod `api` DNS record must be `pulumi
import`ed (already exists) — documented in runbook step 4a.
- Pre-existing cross-worktree Cloud Function source drift is visible in
a full `pulumi preview`; not introduced here (applied via `--target` to
avoid it).

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-20 19:12:28 -04:00
Andrew Beveridge ☄️andClaude Fable 5 ac4d7f7b3d feat(runner-manager): Windows GPU ephemeral runner family (gpu-windows) (#881)
## Summary

Adds a Windows Server + T4 family to the ephemeral GHA runner fleet,
following the existing pattern exactly (JIT-config single-use VMs from
pre-baked image families). Built for python-audio-separator's upcoming
`windows-directml` integration tests — part 2 of the RoFormer-on-Windows
plan (nomadkaraoke/python-audio-separator#292; see plan in
nomadkaraoke/python-audio-separator#294).

### Dispatcher (`ephemeral.py`)
- `FamilySpec.os_label` (default `linux`) replaces the hardcoded `linux`
advertised label
- New `gpu-windows` family: n1-standard-4 + T4, `gha-runner-gpu-windows`
image family, 100GB disk
- `resolve_family` precedence: windows → gpu → build → general
- Windows VMs get the JIT startup script via the
`windows-startup-script-ps1` metadata key (`run.cmd --jitconfig` +
guaranteed shutdown in `finally`); Linux families unchanged
- +8 unit tests (41 pass): family routing, advertised labels, per-family
startup metadata key, TERMINATE scheduling

### Image bake
- `runner-image-provision.ps1`: NVIDIA **GRID** driver (WDDM mode — the
datacenter driver runs T4 in TCC mode with no DirectX, which breaks
DirectML), with TCC→WDDM switch + bounded reboot retries; Python 3.12
(torch-directml has no 3.13 wheels); Git, FFmpeg, system-wide Poetry,
actions-runner win-x64; lean DirectML model set at
`C:\audio-separator-models`. Idempotent via phase markers (startup
scripts re-run every boot); emits the same READY/FAILED serial-console
markers as the Linux provisioner so the existing wait step works
unchanged.
- `build-runner-images.yml`: `gpu-windows` variant
(windows-cloud/`windows-2022` base, per-variant image + startup-metadata
flags), included in monthly "all" builds
- `docs/EPHEMERAL-GHA-RUNNERS.md`: fourth image family + label-routing
rules

## Deploy sequence (operator)

1. **Before merge: `pulumi up` locally** (runner-manager function
redeploys with the new family — my ADC is read-only so I could not run
it)
2. Merge, then bake the first image: `gh workflow run
build-runner-images.yml -f variants=gpu-windows`
3. First consumer job (`windows-directml`, `runs-on: [self-hosted,
windows, gpu]`) lands in python-audio-separator next — it doubles as the
verification job

The companion label fix (`runs-on: [self-hosted, linux, gpu]` on
existing jobs, so Windows runners can't steal Linux jobs) is already in
nomadkaraoke/python-audio-separator#294.

## Testing

- `test_ephemeral.py`: 41/41 pass
- Workflow YAML validated; CodeRabbit local review run, findings applied
(windows-2022 family name, bounded WDDM retry, reboot on installer exit
1, atomic downloads)
- Real-world validation happens at first bake + first `windows-directml`
job (next PR)

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 14:15:38 -04:00
Andrew Beveridge ☄️andClaude Opus 4.8 6f86b3d901 test(e2e): rewrite tenant daily E2E as a faithful frontend journey (#873)
## Why

The daily tenant E2E (`scripts/e2e/tenant_e2e.py`) drove the whole flow
via **direct backend API calls**, bypassing the frontend. Two problems:
1. **Not representative** — it never exercised the tenant portal UI a
real operator uses.
2. **Intermittent stuck-render orphans** (Run #31 vocalstar, #32 singa)
— because it never loaded the lyrics-review page, it never fired
`encoding-worker/warmup`, so the encoder was cold when the render
started and the job orphaned in `rendering_video` (no recovery).
Root-caused this session; the fix is to make the test faithful, which
fires the warmup a real user fires.

## What

Replaces `tenant_e2e.py` with
**`frontend/e2e/production/tenant-happy-path.spec.ts`** — a Playwright
journey mirroring `happy-path-real-user.spec.ts`, run per-tenant via the
existing `e2e-tenant-daily.yml` matrix (vocalstar, singa; `max-parallel:
1`):

login (real magic-link UI) → submit track (tenant form + real audio
upload) → wait for transcription → **open review UI (fires warmup)** →
approve → wait for render → assert tenant distribution (Dropbox present,
no YouTube, downloads, `is_private`) → cleanup.

- **Tenant auth**: added `inbox.testmail.app` to both tenants'
`allowed_email_domains` (setup scripts + live GCS config via
`scripts/e2e/update_tenant_allowlist.py`) so the test signs in through
the real magic-link UI.
- **Test audio**: short 60s clips fetched from private GCS in CI
(`scripts/e2e/fetch_test_audio.sh`) — fast, reliable uploads and quick
renders.
- Workflow reworked to Node/Playwright; `tenant_e2e.py` deleted; version
`0.190.3 → 0.190.4`.

## Findings surfaced (validating the exercise)

- **Browser upload was CORS-blocked in Playwright** by the shared
config's global `x-playwright-test` header (it made the GCS signed-URL
PUT preflight request a header the bucket's CORS doesn't allow).
Verified the same PUT succeeds in a real browser — a **test-harness
artifact, not a product bug**. Worked around by stripping the header for
`storage.googleapis.com` requests. (Note: any prod E2E doing a browser
signed-URL upload is affected by this global header.)

## Verification

Both tenants run **green end-to-end against prod** locally (login →
submit → review → render → distribution; render completed without
orphaning). Reworked workflow dispatched on this branch to confirm CI
wiring (Node setup + private-audio fetch + secrets).

Test-only + CI + tenant-config change. No product code.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

@coderabbitai ignore

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 02:33:20 -04:00