mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
main
214
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
0449d0c107 |
fix(infra): build Cloud Function source archives without __pycache__ (v0.297.4) (#1272)
## Summary
A full `pulumi preview` from main showed runner-manager drift ("replace
source, update function"), last applied from the
`karaoke-gen-runner-scan-retry` worktree. I checked it:
- #1262 (that worktree's branch) is fully on main. The squash merge is
byte-identical to the branch tip. The live scheduler job (`retryCount
2`, `maxRetryDuration 60s`, `attemptDeadline 300s`) also matches the
code exactly.
- The drift comes from the **deployed zip containing
`__pycache__/*.pyc`** (including a pytest pyc) from the worktree's test
run. `pulumi.FileArchive` hashes every file, so every clean checkout saw
a phantom diff. Separately, the 7-day lifecycle rule had already deleted
the bucket object.
## Changes
- New `infrastructure/modules/source_archive.clean_source_archive()`
builds an `AssetArchive` of source files only. It skips `__pycache__`,
`*.pyc`, `.pytest_cache`, other tool caches and `.DS_Store`.
- Used for runner-manager, divebar-lookup and divebar-mirror (the three
`FileArchive` users).
- Unit tests, added to the "Infra - Cloud Function tests" CI job.
- LESSONS-LEARNED entry.
## Applied (before merge, per repo convention)
- I ran a targeted preview with `__pycache__` present in the worktree.
Runner-manager showed source replace + function update (the one-time
correction). Divebar-lookup and divebar-mirror showed **no change**.
- Targeted `pulumi up` ran as claude-admin (Andrew's grant): 2 changes,
289 unchanged.
- The new revision is `github-runner-manager-00031-sam`, and the
deployed zip now holds only the 4 source files.
- The re-preview shows 291 unchanged.
## Testing
- `pytest infrastructure/test_source_archive.py`: 4 passed
- `pytest infrastructure/functions/runner_manager`: 75 passed
- Local CodeRabbit review: no findings
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_012RYLVSfwoV9WyncjqMPHrq
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
ac844b3933 |
ci: pin Playwright jobs to ubuntu-24.04 (chromium unsupported on 26.04) (#1265)
## Summary GitHub's `ubuntu-latest` now resolves to Ubuntu 26.04, where our Playwright (1.57) fails at `npx playwright install --with-deps chromium`: > Error: ERROR: Playwright does not support chromium on ubuntu26.04-x64 That breaks every PR's E2E Smoke / Full / Local-Stack jobs (seen on #1264) and both daily E2E workflows on main (Payment + Happy Path, Tenant Portals failed today, 2026-11-11). ## Change Pins only the 7 jobs that install Playwright browsers to `ubuntu-24.04` (ci.yml ×4, e2e-daily.yml ×2, e2e-tenant-daily.yml ×1). Everything else stays on `ubuntu-latest`. Workflow-only, so no version bump. A Playwright upgrade that supports 26.04 can follow later and drop the pin. ## Testing This PR's own CI runs the pinned E2E jobs. Workflow-only change, no local CodeRabbit review run, so the bot review is left on. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVVUdF2J1bNeVa2uLXA <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Chores** * Updated the operating system used by automated smoke, end-to-end, and daily test runs to Ubuntu 24.04. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
a1cbe1c0d4 |
feat(encoding): retire the GCE c4d pair; run on AWS + c2df only (0.293.0) (#1255)
## Why Since 2026-10-31 the AWS EC2 worker (`aws-encoding-worker-1`) has handled every encode and distribution job. 27 jobs went to it, with no Cloud Run fallbacks, no `encoding_stuck`, and nothing encoding-related in #alerts. The c4d Spot pair (`encoding-worker-a`/`-b`) now only costs ~$8/mo of Hyperdisk. `encoding-worker-fallback-c2df` stays as the GCP fallback. ## What Deleting the pair needs code changes first, because several paths assumed it exists: - **`encoding_worker_manager`** - The primary is optional: an empty `primary_vm` means none. - `ranked_candidates` skips a missing primary and always sets `is_primary`, so with no primary a successful start records the AWS/c2df override. Previously the legacy index-0 rule would have cleared it. - `ensure_primary_running` and `ensure_preferred_running` start a ranked candidate. If there are no candidates they raise `NO_CANDIDATES`. - `demote_active_worker` looks up the override's `machine_type`, so the AWS box can now be cooled down. It then clears the override so the retry goes elsewhere. - **`encoding_service`** - When `active_url` is empty, it targets the top-ranked candidate. - It also invalidates its cached URL after a single-candidate fallback. - **CI deploy (`ci.yml` + `deploy_promote.py`)** - While AWS serves, a c2df green is validate-only: it writes `validated_version`, keeps the AWS override, and stops the green. Before this, it would have replaced the AWS override with c2df. - The worker-change gate now diffs against `validated_version`, because `primary_version` stops moving. - An in-place restart of a serving override first waits for it to drain (15-min cap). If it is still busy, the restart is skipped. - **Idle function:** a missing VM is logged as a WARNING returning `NOT_FOUND`, not an ERROR. - **Error monitor:** watches c2df instead of a/b. - **Pulumi:** drops the a/b VMs and static IPs. The prod preview shows exactly 4 deletes. A separate, unrelated `youtube-comment-autoreply-scheduler` drift shows up too; it will not be applied. - **Docs:** CLAUDE.md, ARCHITECTURE, DEVELOPMENT and TROUBLESHOOTING. ## Rollout (after merge + deploy) 1. Back up `config/encoding-worker`, then set `primary_vm`, `primary_ip`, `secondary_vm` and `secondary_ip` to `""`. 2. Apply `pulumi destroy --target` on the 4 a/b URNs only. 3. Redeploy the idle function with `deploy.sh`. 4. Verify: one prod encode on AWS, warmup, and the logs. **Rollback:** in Firestore, delete `speed_rank_overrides` and set `aws_video_worker=false`. Restoring the pair would need a revert plus `pulumi up`, which gives new static IPs. ## Testing - New `backend/tests/test_encoding_worker_no_primary.py` (13 tests), plus new deploy_promote, idle-function and error-monitor tests. - Full backend suite: 7188 passed. The 8 `test_emulator_integration` failures also fail on main, because of a local emulator from another session. - CodeRabbit CLI: no findings. An independent review found 2 issues, both fixed: the demote retry path, and the in-place restart drain. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVTkatGnrYFhCzq4PTe @coderabbitai ignore Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
ca05245469 |
feat(youtube): TornadoAPI first for YouTube downloads, flacfetch as fallback (v0.291.0) (#1247)
YouTube downloads depended on one logged-in cookie session on one VM. This tries TornadoAPI first and keeps flacfetch as the fallback. Plan: workspace `docs/archive/2026-11-03-tornado-youtube-download-plan.md`. Alert: https://discord.com/channels/1198691101943541770/1265517528608669707/1567241547739570298 Alert: https://discord.com/channels/1198691101943541770/1265517528608669707/1567252495535841386 Alert: https://discord.com/channels/1198691101943541770/1265517528608669707/1567252497620537415 ## Inactive until the secret exists Nothing changes in prod when this merges. The deploy adds `TORNADO_API_KEY` only once secret `tornado-api-key` has a version, and creating it needs `/gcp-admin`. `YOUTUBE_DOWNLOAD_PROVIDER=flacfetch` switches it off. ## What changed - **`tornado_client.py`:** create job, poll with a 180s deadline (cancels on timeout), stream the delivered file, delete it from Tornado. Retries with one idempotency key per download; honours `Retry-After`. - **`YouTubeDownloadService`:** Tornado first for YouTube, then flacfetch. WARNING per fallback, ERROR on 401/402/403, so an outage or lapsed billing is visible. Non-YouTube URLs stay on flacfetch. - **Availability check:** Tornado metadata first, because flacfetch's check rejects valid videos while its cookies are bot-blocked. If the lookup fails, flacfetch decides as before. ## Spike results (real API, 11 jobs) - `audio_only` + `format: opus` gives **packet-identical audio** to flacfetch's real job input (`Tk3ty5ansqs`, stream MD5 match). `audio_codec: copy` or no `format` gives 128k AAC instead, so the request omits it. - `mBJivIsN0n0`, which bot-blocked job 526bf46c, completed in about 5s of processing. - An age-restricted video completed without cookies; a missing video ends in `Warning`. - Billed about 25.6 MB for a 4-minute song. ## Testing - 79 mocked tests pass (`test_tornado_client.py`, `test_youtube_download_service.py`). - Live run through `YouTubeDownloadService` with the GCS upload stubbed, using a URL with a playlist parameter: only the single video was sent. - Local CodeRabbit: all findings fixed; the last small fix wasn't re-reviewed. ## Not verified - **A full pipeline job with this input.** It can only run once the secret exists. Files are Ogg Opus named `.ogg` (flacfetch's were WebM Opus), so the codec is the same but the container is new to this path. Run the daily E2E or one admin job first. - **Deviation from the plan:** the Tornado job id is logged but not saved on the gen job, because a restart loses the in-request download on the flacfetch path too. A restart can orphan one job, which costs a fraction of a cent. - **CI's secret check:** it depends on the deploy service account being able to read the secret; the log line `tornado-api-key wired:` shows the result. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVTvqXyL4QyA2EjzQMG Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
ea69386e50 |
fix(ci): Java 25 for the Firestore emulator (#1245)
`google-cloud-cli-firestore-emulator` 592.0.0 is compiled for Java 25 (class file 69), and the package is installed unpinned from apt, so CI's Java 21 fails with `UnsupportedClassVersionError`. This blocks every PR: Backend Emulator Integration and Local Stack E2E both failed on #1243 and #1244 (runs 41705112128, 41706068761). Main last passed 11-02. CI-only change; this PR's own run is the test. @coderabbitai ignore not added: no local review was run on this one-line CI change, so the bot may review it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVTvqXyL4QyA2EjzQMG Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
78264fa23a |
feat(encoding): run the video worker (distribution) on the AWS box (v0.289.0) (#1240)
## Summary Phase 2 of moving encoding to AWS; Phase 1 was #1236 and #1239. Design and rationale are in `docs/archive/2026-10-29-aws-encoding-worker-plan.md` § Phase 2. When Firestore `config/encoding-worker.aws_video_worker` is true **and** the AWS worker ranks first, `trigger_video_worker` runs the video worker **on the AWS box** instead of starting the Cloud Run job `video-encoding-job`. - **Uploads move to AWS:** the Dropbox/YouTube/Drive uploads (~187 GB/mo, ~$18/mo of Cloud Run internet egress) leave from AWS, on credits. - **No Cloud Run CPU** for the job (~$11/mo). - **Off by default.** Any failure falls back to the Cloud Run job, so this path can't drop a job. ## Changes - **`POST /run-worker` (worker):** - Spawns `python -m backend.workers.video_worker --job-id …` with the box's video-worker env. - Status is reported via `/status/{id}`. - A run counts in `/health` `active_jobs` from the moment it's accepted, so the idle watchdog won't stop the box mid-run. - Returns 503 unless the env file exists. - `job_id` is pattern-validated. - Any startup failure marks the run failed. - **`backend/services/local_encoded_outputs.py`:** a same-box handoff. - When the co-located video worker asks (`stash_local_outputs`, set only where `LOCAL_ENCODED_OUTPUTS_DIR` exists), the encoder keeps local copies of the finals, keyed by blob path. - The video worker takes those instead of downloading them back out of GCS. Without this, the move would *add* ~400 MB/job of GCS egress. - Copies are made off the event loop and purged after 6 h. - No change on GCE or Cloud Run. - **`video-worker-env.py` (box, every boot):** - Mirrors the **live** `video-encoding-job` env, with secrets resolved, so nothing is hand-copied and nothing drifts. - Runs as the job's identity, `karaoke-backend@`, via a second WIF binding (Pulumi `aws-encoding-worker-wif-backend-user`). - On failure it writes no env file, so `/run-worker` answers 503 and the Cloud Run job is used. - **`WorkerService._trigger_video_worker_on_aws`:** checks the flag and that the AWS worker ranks first, ensures it's running, waits for health, then POSTs `/run-worker`. Returns False on any doubt. - **Also:** `startup-aws.sh` creates the backend credential config and runs the env helper; CI uploads `video-worker-env.py`; `worker_code_paths.txt` now gates `local_encoded_outputs.py`. ## Rollout (needs `/gcp-admin`) 1. **Before merge:** targeted `pulumi up` of the backend WIF binding. It's additive. 2. **After stage B of Phase 1** (AWS worker preferred, timings verified): stop/start the box once so the new `startup-aws.sh` writes the env, then set `aws_video_worker: true`. Set it to `false` to roll back. ## Testing - **27 new tests:** - handoff stash/take/purge, path-escape guard, and the per-request stash flag; - `/run-worker`: disabled, bad module, dedupe, active immediately, spawn args/env, failure log tail, wheel-not-ready, safe start, unsafe ids; - env builder; - the trigger's AWS branch: off, not first, happy path, didn't start, refusals, errors. - **Full backend + unit suites:** no new failures (the 13 local failures are identical on main). - **Local CodeRabbit:** 5 findings, all fixed. The code-paths gate test also caught a missing worker path, now fixed. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
de5dfcd50d |
feat(encoding): AWS EC2 encoding worker as a pool candidate (v0.288.0) (#1236)
## Summary
Phase 1 of moving encoding to AWS. EC2 is paid for by the AWS Activate
credits (acct 790139457751, expiring 2028-10-31).
- **Plan:** `docs/archive/2026-10-29-aws-encoding-worker-plan.md`.
- **Benchmark** (3 real prod jobs replayed): an EC2 c7a.4xlarge matched
or beat GCE c4d-highcpu-16.
The EC2 box runs the **unchanged** `gce_encoding` worker service and
joins the existing candidate pool as `provider: aws`. The existing
failover, active override, lost-job detection and stockout cooldowns all
apply to it as they are.
## Changes
- **`backend/services/aws_encoding_compute.py` (new):**
`AwsComputeAdapter` looks to the manager like the
`compute_v1.InstancesClient` methods it uses.
- EC2 states map to GCE statuses.
- EC2 capacity errors are added to `CAPACITY_ERROR_CODES`.
- A missing instance raises `NotFound`.
- Controller credentials come from Secret Manager
`aws-encoding-controller-credentials`.
- **`EncodingWorkerManager`:**
- `_compute_for(vm)` sends AWS VMs to the adapter; the GCE paths are
untouched.
- A shared `parse_fallback_vms` passes `provider`/`instance_id`/`region`
through.
- `ranked_candidates()` is used by both the encode path and the warmups.
- `ensure_preferred_running()` behaves **exactly like
`ensure_primary_running` whenever the GCE primary ranks first**, which
is today's case.
- An AWS control-plane fault fails over to the GCE candidates.
- **Ranking:** `SPEED_RANK["c7a"] = 90`, so AWS is the last resort by
default. Firestore `config/encoding-worker.speed_rank_overrides =
{"c7a": 5}` makes it preferred, with no deploy.
- **Isolation from GCE-only tooling:**
- The override now records `active_override_provider`.
- The CI blue-green deploy, `deploy_promote` and the idle Cloud Function
all ignore the AWS worker; it hot-updates its wheel per job.
- A GCE promotion clears any stale provider tag.
- **Worker:**
- Uses `gcloud storage` on AWS, because gsutil can't use WIF credential
files. GCE keeps gsutil.
- `infrastructure/encoding-worker/aws/` adds:
- `provision.sh`, as EC2 user-data;
- `startup-aws.sh`, uploaded by CI;
- `idle-watchdog.py`, an idle self-stop that runs as its own systemd
timer.
- **Infra (GCP), already applied with `/gcp-admin`:**
- The WIF pool `aws-encoding`, plus an AWS provider that only admits the
`nomad-encoding-worker` role, federated to `encoding-worker@`.
- The secret `aws-encoding-controller-credentials`.
- Preview and apply: 4 creates, 288 unchanged.
- **Infra (AWS):** a new Pulumi project `infrastructure/aws` (stack
`nomadkaraoke/prod`):
- the c7a.8xlarge **persistent Spot** worker, which stops on
interruption or shutdown;
- an Elastic IP;
- a security group on 8080 (API key, same as the GCE firewall);
- the worker role with SSM;
- a tag-scoped `nomad-encoding-controller` user.
- Preview: 10 creates. **I apply it after merge**, because the box
provisions from the deployed wheel and boot scripts.
- `boto3` is now an explicit dependency.
- CI also runs the `encoding_worker_idle` function tests, in the job
renamed "Infra - Cloud Function tests".
## Rollout (after merge)
1. **Stage A:** `pulumi up` the AWS stack. Store the controller key in
the secret, then add the worker's entry to
`encoding-worker-fallback-vms`. It's inert unless both c4d and c2d stock
out.
2. **Stage B:** a supervised `speed_rank_overrides {"c7a": 5}` window.
Run real jobs, then compare timings and outputs, including a visual
check of the portrait video.
3. **Stage C:** keep it preferred. After a clean week, delete the c4d
pair.
Phase 2, running distribution (Dropbox/YouTube/Drive uploads) from AWS,
follows in a separate PR.
## Testing
- **New tests:**
- adapter: botocore `Stubber`, covering the state map, capacity and
NotFound handling;
- manager routing and ranking with overrides;
- `ensure_preferred_running` branches;
- failover on an AWS control-plane fault;
- provider tagging;
- deploy_promote exclusions;
- the idle function skipping AWS;
- watchdog decisions;
- the gcs CLI choice.
- **Updated** the candidate and warmup-route tests to the new seams.
- **Full backend + unit suites** have no new failures. The local
emulator and cross-suite failures are identical on main.
- **Local CodeRabbit:** 2 findings, both fixed (failover on AWS faults;
stale provider tag).
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
1abdd8c860 |
fix(cost): runner-manager only dispatches VMs for jobs it can serve; enforce AR cleanup (v0.286.1) (#1228)
## Summary On 2026-10-27, **238 GCP `gha-general` VMs (~$67)** ran in one day. This pushed October's GCP bill past the $300 budget. **Cause:** - nomad-mobile's E2E jobs run on the utility MacBook (`[self-hosted, macOS, nomad-mbp]`). - The runner manager acted on *any* job labelled `self-hosted`, so each macOS job got a Linux VM that could never take it. - While the job waited for an MBP lane, `redispatch_stalled_jobs` launched another VM every 5 min for up to 60 min. - Each of those VMs then idled until the 2 h reaper deleted it. ## Changes - `ephemeral.can_serve(labels)`: GitHub assigns a job only to a runner that advertises every label the job requests, so a job is ours only if its labels ⊆ the resolved family's `runner_labels_for()`. - The new check gates the webhook (`main.py`), `list_queued_self_hosted_jobs` and `redispatch_stalled_jobs`. - I checked every `runs-on: [self-hosted, …]` label set in the org's workflows. All of them except the macOS lanes are subsets of our families' labels, so no existing job loses its VM. - CI: these function tests never ran in CI before. A new `runner-manager-tests` job runs them on infra or ci.yml changes and is enforced by CI Gate. - Docs: `docs/EPHEMERAL-GHA-RUNNERS.md` § "Only jobs our runners can serve". ## Testing - 6 new tests: macOS jobs, unknown labels, our families, the redispatch skip, queued-scan exclusion, and the webhook no-op. All 71 runner_manager tests pass locally. - Local CodeRabbit review: 2 cycles, both CI-wiring findings fixed. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
d76a136ee6 |
feat: mobile Phase 6 — RevenueCat IAP webhook, Expo push, self-serve account deletion (flagged off), App Review demo login (v0.284.0) (#1220)
Phase 6 (mobile release readiness) — gen side. Spec: nomad-mobile
`docs/superpowers/specs/2026-10-25-mobile-phase6-release-readiness-design.md`
(§4 payments, §5 push, §6 deletion, §9 demo account).
## What
- **`/me` app fields:** opaque `app_user_id` (UUID, used for
RevenueCat/Sentry — no email to third parties),
`capabilities.account_deletion`, `push_prefs`.
- **In-app purchases:** `POST /api/users/webhooks/revenuecat` (shared
Authorization header). Exactly-once grants keyed by
`{store}:{transaction_id}` inside the same Firestore transaction as the
balance change; refunds (REFUND / CANCELLATION with CUSTOMER_SUPPORT or
negative price) debit the granted account, refund tombstones block late
grants; other cancellations ignored. `store_payments` for accounting +
revenue sums; referral earnings on **net** proceeds; sandbox grants but
excluded from revenue/referrals. Credit packages gain `store_product_id`
+ server-driven `web_link` (iOS US only for now).
- **Push:** `POST/DELETE /api/push/expo-token` (token moves between
users atomically), `PATCH /api/users/me/push-prefs` (ready / review /
problems, gates Expo **and** Web Push), Expo sender (batched,
DeviceNotRegistered cleanup), new FAILED / AWAITING_AUDIO_SELECTION
triggers.
- **Account deletion** (`ACCOUNT_DELETION_ENABLED`, **off in prod**
until Andrew's manual pass): `GET /api/users/me/deletion-preview`,
`DELETE /api/users/me`. Refuses admin / unpaid referral balance / Stripe
Connect / made-for-you in progress (409). Cancels active jobs, deletes
decide data (partner call, twice), RevenueCat subscriber, revokes Sign
in with Apple (when a code is given), queues kjbox
(`/api/kjbox/account-deletions` pull + ack), deletes **inputs only**
(uploads, input/, audio_edit/, style/, staged instrumentals) — **never
published outputs** — strips personal fields, anonymises
jobs/referrals/song requests/show-credit records, keeps payment records.
Every step idempotent; failures → 503 retry; sessions revoked last.
- **Web:** `/account/delete` page (Google Play web deletion path),
hidden while the flag is off.
- **App Review demo login:** fixed code for one configured email — inert
until `APP_REVIEW_DEMO_EMAIL` is set (not set in prod).
- **CI:** mounts `REVENUECAT_WEBHOOK_AUTH`, `REVENUECAT_SECRET_API_KEY`,
`APP_REVIEW_DEMO_CODE` (secrets exist; runtime SA already has accessor).
## Tests
6838 backend unit tests pass (emulator suite needs emulators;
iap/deletion emulator tests pass locally), frontend jest 1785/1785.
Per-task reviews + local CodeRabbit review (5 minor findings fixed).
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01CZf1cM9vefnNw5djARRhyA
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
286e604b75 |
feat(worker): voice-pool separate-only GPU job (v0.283.1) (#1217)
## Summary - Adds `voice-pool-separate-job`, a Cloud Run GPU (L4) job for karaoke-decide's voice session. decide writes a JSONL manifest (`voice-pool/manifests/<run>.jsonl`) and starts the job with a per-run shard; each task separates its share of songs with the `instrumental_clean` Separator and writes `voice-pool/stems/<id>/vocals.flac` (skipped if already present). Results are written back to the manifest area every 10 songs. - No Firestore or gen job state. Model-load failure or an all-songs-failed task exits non-zero with a single ERROR; partial failures exit 0. - Task timeout is 3600s (Cloud Run's GPU maximum, found by the local CodeRabbit review), so decide must size the shard count to keep each task under 1h. Re-runs skip finished songs. ## Changes - `backend/workers/voice_pool_separate_worker.py` + `backend/tests/test_voice_pool_separate_worker.py` (9 tests, 91% coverage of the worker) - `infrastructure/modules/cloud_run.py`, `infrastructure/__main__.py`: Pulumi job definition - `.github/workflows/ci.yml`: CI guard step (tolerates a missing job) - `docs/ARCHITECTURE.md`: paragraph on the job - Version 0.283.0 -> 0.283.1 ## Deploy notes **Andrew: run `pulumi up` in gen infrastructure/ BEFORE merging (creates voice-pool-separate-job; the CI guard tolerates a missing job).** decide's Pulumi (later PR) will create the voice-pool SA and grant it `roles/run.jobsExecutorWithOverrides` on this job. ## Testing - [x] `pytest backend/tests/ --ignore=backend/tests/emulator`: 6691 passed, 40 skipped - [x] Local CodeRabbit review completed (2 cycles): the critical timeout finding fixed; the remaining minor (results flush inside `finally`) is a known deferred item @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_0188txBF47GpTo7hym1FoifX --------- Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
|
055a5d3f73 |
feat(youtube): enable comment auto-reply in prod + one-time backfill (v0.280.2) (#1207)
## Summary Switches on the YouTube comment auto-reply bot from #1205 in prod (Andrew authorised go-live after the 6-month replay sign-off): - `YOUTUBE_COMMENT_AUTOREPLY_ENABLED=true` - `YOUTUBE_COMMENT_AUTOREPLY_START=2026-04-22T00:00:00Z`. This is used on the first run only, as a one-time backfill (Andrew: "yes please, will be a good test of the system working once in prod"). The replay predicts 3 replies: Fightmaster "Nobody's Baby", Bear Ghost "Tiny Hitman", Playa Fly "N God We Trust". Threads already answered by hand are skipped. Already verified in prod before this PR: - The endpoint returns `disabled` while the flag is off. - A `?dry_run=true` run loads secret `youtube-comment-bot-credentials`, builds the channel index and reads comments with no errors. - The Cloud Scheduler job `youtube-comment-autoreply-hourly` and the secret were applied through Pulumi in #1205. ## Testing Config-only change; the code and tests are in #1205. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_016E3ibN2EcKxnGQvEf6T4V8 Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
b67b273056 |
feat(kjbox): decide partner secret for catalog resolve + fake judge + error monitor for decide search (v0.278.10) (#1190)
## ⚠️ Merge prerequisite (do BEFORE merging) The Secret Manager secret `decide-gen-partner-secret` must have a version before this merges, otherwise the CI deploy's `--set-secrets ...DECIDE_GEN_PARTNER_SECRET=decide-gen-partner-secret:latest` fails. ``` gcloud auth login admin@nomadkaraoke.com cd ~/Projects/nomadkaraoke/karaoke-gen-decide-partner-resolve/infrastructure GOOGLE_OAUTH_ACCESS_TOKEN=$(gcloud auth print-access-token) pulumi up # review: should only add the decide-gen-partner-secret Secret openssl rand -hex 32 | tr -d '\n' | gcloud secrets versions add decide-gen-partner-secret --project=nomadkaraoke --data-file=- ``` No extra IAM grant needed: gen's API runs as `karaoke-backend@`, which already has project-level `secretmanager.secretAccessor`. ## Summary - `POST /api/kjbox/catalog/resolve` now also accepts karaoke-decide via header `X-Partner-Secret` (new setting `decide_partner_secret` / env `DECIDE_GEN_PARTNER_SECRET`). Other kjbox endpoints stay `X-Kjbox-Secret` only. decide gets its own AI-call budget (60/min) so kjbox nights are never starved; the verdict cache is shared. - Fake match judge / resolve for the local E2E stack (`FAKE_WORKERS`): `judge_kind` / `judge_canonical_*` knobs; no Gemini or catalog calls. - Error monitor now watches decide's new `karaoke-decide-search` service and `song-index-build` job. - Pulumi: declares the `decide-gen-partner-secret` Secret (value added manually); CI deploy mounts it. - Version 0.278.10. ## Testing - [x] Changed-module + kjbox/catalog/error-monitor/fake-worker backend tests pass (133 passed). Full `backend/tests` suite exceeds 10 min locally, so left to CI. ## Review - CodeRabbit CLI failed twice (WebSocket closed / connection error), so no local CodeRabbit review completed; I did a manual diff review instead (no issues found). Deliberately NO `@coderabbitai ignore`, so the GitHub bot reviews this PR. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01VsLw9cTTMBab1tiRxCsYrm --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
4286d98dbe |
fix: LRCLIB as a registered server-side app; honour Retry-After, retry 5xx (v0.279.2) (#1200)
@coderabbitai ignore ## Why LRCLIB now applies stricter rate limits to unregistered clients (https://lrclib.net/docs#implementation-requirements-rate-limiting), and Cloud Run shares egress IPs. On 10-19, prod jobs got intermittent 503s, and job 80e7ffdf lost its LRCLIB lyrics. LRCLIB is our only free source; it supplied lyrics for 165 of 220 jobs in 30 days. Andrew registered a server-side app. ## What - **`karaoke_gen/lyrics_transcriber/lyrics/lrclib_client.py`** (new): - Sends `Lrclib-App-Id` + `Lrclib-Api-Key` from `LRCLIB_APP_ID` / `LRCLIB_SECRET_KEY`. - Applies LRCLIB's rules: every 429 sets a **process-wide cooldown** of `Retry-After` plus up to 30% jitter. The wait is never shortened, and overlapping 429s keep the latest expiry. Nothing is sent until the cooldown has passed. It waits and retries once only when the wait is ≤30s. - Retries 5xx after 1s/2s, honouring a 5xx `Retry-After`. - A **401 on our credentials** logs a WARNING and drops to anonymous access for the process, so a revoked key degrades instead of losing LRCLIB. - `LRCLIBProvider` (both `get` and `search`) uses it. The `quick_version` renderer uses it with `retry=False`, so the fast path gets the auth and the cooldown but never blocks. - **Infra:** - Pulumi secrets `lrclib-app-id` / `lrclib-api-key`. - Env vars on the `lyrics-transcription-job`. - The backend service deploy's `--set-secrets` (`ci.yml`). The karaoke-backend SA already has project-wide `secretAccessor`. ## Deploy order (do NOT merge before steps 1–2) 1. `pulumi up` to create the two secrets and the job env. Needs `/gcp-admin`. 2. Add the secret versions from Andrew's `.envrc` values. 3. Merge. The CI backend deploy references the secrets. ## Testing - New `tests/unit/test_lrclib_client.py` (14 tests): headers; anonymous use; Retry-After with jitter; a long Retry-After is not shortened and blocks further calls; no-retry mode; a second 429; 5xx backoff; 5xx Retry-After; 401 fallback; overlap merge (mutation-checked); provider routes through the client. - 164 passed across the client, `quick_version` and `search_lyrics` tests. - **Live** with the real app key: Dar Williams - Iowa and Wolfmother - Where Eagles Have Been were found. One 503 was recovered by the retry. - Local CodeRabbit, 3 cycles. I fixed the Retry-After cap, the cooldown after a final 429, the `quick_version` 401 path, the overlap merge and 5xx Retry-After. I skipped one false positive about the tests' fake clock (the retry path sleeps, then sends without re-checking the clock). 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Jw21syo4HizJJjYw2mg2Vp --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
a6d4163d2e |
feat(alerts): route non-signal Discord posts to #noise (v0.278.8) (#1186)
## Summary Andrew asked for #alerts to carry only real problems. Routine and informational posts now go to the new **#noise** channel. The GitHub org PR webhook was already moved to #noise separately. | Message | Before | After | |---|---|---| | Error monitor: New Error Pattern / Spike / Incident | #alerts | #alerts | | Error monitor: ✅ Patterns Auto-Resolved | #alerts | **#noise** | | Error monitor: 📊 Daily Error Digest | #alerts | **#noise** | | Nightly Backup Report (clean run) | #alerts | **#noise** | | Nightly backup failed/partial (title is now "🚨 Nightly Backup FAILED") | #alerts | #alerts | | ✅ E2E Daily Test PASSED | #alerts | **#noise** | | 🩹 E2E self-healed Stripe (auto-merge PR opened) | #alerts | **#noise** | | 🩹 self-heal PR needs CI approval click / could not open | #alerts | #alerts | | E2E / Tenant E2E failures, DR freshness stale, Job failed, Publish completeness SHADOW, Gemini unavailable, OAuth credential alerts, quarterly DR drill reminder | #alerts | #alerts (unchanged) | ## How the noise webhook is resolved - **Backend / error monitor and the backup function** read it at runtime: env `DISCORD_NOISE_WEBHOOK_URL` first, then the Secret Manager secret `discord-noise-webhook` (the name can be overridden with `DISCORD_NOISE_WEBHOOK_SECRET_NAME`). No Pulumi change is needed, because both service accounts already have project-wide `secretAccessor`. - **GitHub Actions** uses `secrets.DISCORD_NOISE_WEBHOOK_URL || secrets.DISCORD_ALERT_WEBHOOK_URL`. - **Nothing is silently dropped.** If the noise webhook can't be resolved, or a post to it fails, the message goes to #alerts instead. In the workflow this uses `curl -fsS` with a retry on the alert webhook. - The error monitor now rate-limits each channel separately (10 per run each), so noise can't use up the alert budget during an incident. ## Deploy notes - **Backup Cloud Function:** CI doesn't deploy its source. Run `infrastructure/functions/backup_to_aws/deploy.sh` after merge. - **Error monitor job:** runs `karaoke-backend:latest`, so it picks up the change once CI pushes the image. - **No `pulumi up` needed.** ## Testing - `tests/unit/services/error_monitor`: 376 passed. New tests cover noise vs alert routing, fallback when the noise webhook is missing or fails, separate rate limits, and secret resolution. - `backend/tests` (excluding emulator): 6418 passed, 36 skipped. - `infrastructure/functions/backup_to_aws`: 60 passed, including the new `test_discord_alert.py` and routing tests in `test_main.py`. These were run in a scratch venv because `functions_framework` isn't in the conda env, which was already the case before this change. - `actionlint` on `e2e-daily.yml`: no new findings. The only findings are info-level shellcheck notes that were already there. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_019pmG2XAgtUguEJWLFpkKHN Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
c277bc8ee3 |
feat: local E2E stack (fake workers + test API) + email-code sign-in (v0.275.0) (#1165)
## Summary - **Local stack:** `scripts/local-stack.sh up --frontend` runs the **real** backend on the Firestore + GCS emulators with `FAKE_WORKERS=1`. In-process fake workers move jobs through their real states in seconds, using tiny real media, so web Playwright and mobile Maestro full-journey tests can run offline against the same endpoints. - **Email-code sign-in** (nomad-mobile spec §4.1): `POST /api/users/auth/send-code` + `verify-code`. The code logic is shared with kjbox via `LoginCodeStore`, and sends are throttled per email and per IP. - **Bug fix found by the new suite:** `confirm-duration` moved jobs to `separating_stage1`, from which they can never reach review. It now resumes at `downloading`. No prod job had taken this path yet. ## Changes - `backend/services/fake_workers/`: dispatch seam in `WorkerService.trigger_*`, plus fake search, download, checkout and outbox, scenarios, and per-user config. The real stem upload, transcode, backing-vocals analysis and preview render are reused. Every check is strict (`is True`), so a MagicMock settings object never enables the fakes, and startup **refuses FAKE_WORKERS in production**. - `/api/test/*` router (mounted only in that mode): `status`, `scenario`, `fake-config`, `emails`, `reset`, plus the fake Stripe checkout page. Paying it runs the real webhook handler. - `frontend/e2e/local-stack/` (9 specs, ~50s): - sign-up → make → review → instrumental → download - buy and cancel credits - duration confirm, failed, complete and review states - code sign-in and lockout - New required CI job `local-stack-e2e`. - Docs: TESTING.md § Local stack, DEVELOPMENT.md, API.md, README, LESSONS-LEARNED. - `MAX_SIGNUPS_PER_IP` can now be set by env. Only the local stack raises it; production keeps the default of 2. ## Testing - [x] Backend unit: 6175 passed - [x] Emulator: 152 passed - [x] Frontend Jest: 1706 passed - [x] Local-stack Playwright: 9/9 passed (CI mode) ## Review - [x] CodeRabbit CLI review completed locally. 1 minor finding declined: it suggested swallowing infrastructure errors in send-code as fake "sent" responses, which would hide real outages from users. @coderabbitai ignore --- 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01AGuxtgoscF6RJGtpjHPg3W --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0c1aa16469 |
Turn on conversion emails + fix doubled signature on credit-denied email (v0.273.3) (#1160)
## Summary - **Conversion emails go live.** Andrew approved the copy, the 25% offer, the timings and the 612 Joan St postal address. Adds `CONVERSION_EMAILS_ENABLED=true,REVIEW_LAST_CHANCE_ENABLED=true,CONVERSION_OFFER_PERCENT=25` to the backend `--set-env-vars` in ci.yml (feature shipped dark in #1156). - **Doubled signature fix.** The "About your Nomad Karaoke free credits" (credit-denied) email appended `_get_email_signature()` itself and also got it from the shared footer, so it showed twice. I audited all 22 `send_*` emails by rendering each one; this was the only duplicate. - New regression test: renders every `EmailService.send_*` and asserts the signature appears exactly once. It fails on main with `send_credit_denied_email: 2`. ## Testing - `make test` green (backend 2054 + 6071 + 150, frontend 1692). - Local CodeRabbit review: no findings. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_019KuzHAU4rxkv8JfD8NmHTN --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
39dcf2598e |
fix: PyPI prune snippet + force marker only in PR title (v0.268.10) (#1146)
## Summary Two follow-ups to #1145, both found while doing today's prune. 1. **Prune console snippet silently did nothing.** PyPI's delete-version form now requires five `acknowledge_*` checkboxes. A submit without them gets a **200 re-render**, which the old snippet counted as "deleted". The snippet now reads each release's real form (CSRF plus checkboxes), submits it, and counts a version deleted only when its manage page 404s. This is the exact approach used for today's prune (88 releases, 10.73 → ~4.1 GB). 2. **The force marker matched the commit body.** A squash commit's body is the PR description. #1145's description just *mentioned* the marker, which forced a publish on merge. That publish ran before the prune, so it 400'd and uploaded nothing. The marker now only counts in the commit subject (the PR title). Docs: the runbook covers the new form, the per-version JSON cache lag, and a prune log. ## Testing - Unit tests: 30 pass across both scripts, with new tests for checkbox/404 handling and for a marker in the body not forcing a publish - `node --check` on the generated snippet - Replayed #1145's real merge message through the gate: no publish - Local CodeRabbit review: 0 findings - Full `make test` not re-run (no app code touched); PR CI covers the package suites @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01MSW9yjwGnzz1S8MgCGm6t5 --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
5005d3cb80 |
ci: publish to PyPI only when CLI code changed (v0.268.9) (#1145)
## Summary PyPI publishing failed on every deploy since v0.268.4. karaoke-gen hit PyPI's 10 GB project cap for the third time (June, Sept 29, Oct 12). Each wheel is ~62 MiB and we published on every merge (~100/month), so each prune only bought a couple of weeks. PyPI is only the public `pip install` channel for the CLIs. Cloud Run builds from source, and the encoding worker pulls the wheel from GCS. So we now publish only when something CLI users get has changed. ## Changes - `scripts/pypi_publish_gate.py` (stdlib): diffs `scripts/pypi_publish_paths.txt` (`karaoke_gen/` plus the `backend/` modules it imports) and `pyproject.toml` (ignoring the package `version =` line only) against **the latest version on PyPI**. A skipped or failed publish therefore never loses a CLI change. Fails open. - `ci.yml` `package-publish`: new "Decide PyPI publish" step gates `poetry publish` (now `--skip-existing`). **Tags and GitHub releases still happen on every version**, because the encoding-worker gate diffs against tags. Override with `[publish-pypi]` in the PR title/commit, or the new workflow_dispatch input "Publish to PyPI". - Not gated on purpose: `frontend/**` (bundled review UI) and `backend/config.py` (cloud settings). A test keeps the backend import list in sync, with `config.py` as the one documented exemption. - Docs: PYPI-STORAGE-PRUNE.md, TROUBLESHOOTING.md, prune script and reminder workflow comments. ## Impact Replayed over v0.200 to v0.268: **24 of 136** releases would have published (~0.7 GB/month instead of ~4 GB/month). Run against current main, it decides "no CLI-affecting changes since v0.268.3", which is correct for v0.268.4 to v0.268.8. ## Testing - 15 new unit tests in `tests/unit/test_pypi_publish_gate.py` - `make test` all green - actionlint clean on the new step - Local CodeRabbit review: 1 finding (doc recovery path), fixed @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01MSW9yjwGnzz1S8MgCGm6t5 --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
950c96fa48 |
feat(e2e): checkout-only (no-charge) payment mode for daily E2E Stage 1 (v0.266.3) (#1130)
There's no working test card right now (the issuer declines it), so the daily E2E Stage 1 defaults to a **no-charge** mode until the repo variable `E2E_REAL_PAYMENT=true` is set. ## Checkout-only mode (`E2E_PAYMENT_MODE=checkout-only`) - **Unchanged:** real signup via the `e2etest95` referral link, the Buy Credits dialog, and real Stripe Checkout session creation. - **Card:** fills the form with Stripe's public test card `4242…` (`NO_CHARGE_CARD`; no secret needed). The self-healing steps still run, except `submitPayment`. - **Checks:** the $0.50 referral price and the Pay button are visible, then it **stops before Pay**. Live mode always declines that test card, so even an accidental click can't charge anything. - **Stage 2:** gets the new account's session token and spends its signup credit. The test asserts the account has at least 1 credit. - **Not exercised:** the Stripe payment, the webhook credit grant, the purchase email, and self-heal PRs (no verified purchase to learn from). - **Email:** the daily email marks Stage 1 `[checkout-only: no real payment]`. ## Testing - Offline suite: 31 passed. The new test checks that no-charge mode fills the form with the test card, never calls confirm and never leaves Checkout. - `--list` loads the production spec in both modes. Lint and typecheck are clean. - A live E2E Daily run on this branch (37262437261) is in progress against prod. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
feef49a588 |
fix(e2e): stop publishing card data/admin token in artifacts; fail fast on card decline (v0.266.1) (#1128)
## Summary
E2E Daily **#200** failed in Stage 1. The cause was **the issuer
declining the test card**: Stripe `confirm` returned 402 `card_error /
card_declined / generic_decline` on all 3 attempts. It was not a Stripe
Checkout change. The self-healing runner still treated it as one: a 90s
verify wait, then 6 Gemini calls "re-entering" a form that was already
correct.
The investigation also found that **every run's artifacts published
secrets** in this public repo:
- the full test card (number, expiry, CVC) in screenshots, videos,
`error-context.md` and the DOM dump;
- `X-Admin-Token` in the trace network logs.
All 140 unexpired `e2e-*` artifacts have already been deleted via the
API. The admin token and the test card still need rotating (Andrew).
## Changes
- **Fail fast on decline:** `stripe-checkout.ts` watches
`/v1/payment_pages/<id>/confirm`. A `card_error` aborts `submitPayment`
immediately with `STRIPE_CARD_DECLINED: …`: no LLM calls, no second Pay
click.
- New `CheckoutStep.abortReason`, honoured by the runner, `pollVerify`
and the LLM loop.
- Non-card errors (e.g. a missing field) still go to the LLM fallback.
- **Alerts say why:** Stage 1 exposes a `failure_reason` output ("Test
card declined, not a checkout bug (…)"). It appears in Discord and in
the daily email. The Discord payload is now built with `jq` from env, so
Stripe's text never reaches the shell.
- **No secrets in artifacts:**
- The Stripe helper's screenshots mask every input, and the DOM dump
redacts input values.
- The credit-purchase spec turns off trace, video and Playwright's
automatic screenshots.
- `playwright.production.config.ts` turns traces off on CI
(`E2E_TRACE=on` overrides).
- New `frontend/e2e/scripts/scrub-artifacts.mjs`:
- deletes `*.zip` traces and `e2e-session-token.txt` (and videos with
`--drop-videos`);
- redacts secret env values and card-number-shaped digit runs;
- with `--blank-inputs`, blanks `<input value>` and ARIA textbox values;
- deletes any file that still contains a secret.
- Every production-E2E upload (`e2e-daily` ×2, `e2e-tenant-daily`, the
`ci.yml` happy path) now runs the scrubber first and **uploads only if
the scrub succeeded**.
- Docs: `TESTING.md` (decline behaviour, a "E2E Artifacts Are Public"
section), `LESSONS-LEARNED.md`.
## Testing
- `npx playwright test --config=playwright.selfheal.config.ts`: **30
passed**. New tests:
- a declined card fails in ~7s with 0 planner calls and exactly 1
confirm request;
- the DOM dump contains no typed card values;
- card-error parsing and the confirm-URL matcher;
- scrubber units (literal/grouped/PAN redaction, short secrets only
blanked inside inputs, deletion rules).
- Ran the scrubber against a copy of the real #200 artifacts: no card
data remains, and traces and videos were removed.
- actionlint: no new findings (only shellcheck info notes; ci.yml:996 is
pre-existing).
- Local CodeRabbit review: 2 findings (shell interpolation of the
reason; extension allowlist in the scrubber), both fixed.
- The full `make test` suite was not run: the change is limited to E2E
helpers, workflows and docs, with no backend or app code.
## Follow-ups (Andrew)
- Rotate the `admin-tokens` secret and the `E2E_ADMIN_TOKEN` repo
secret.
- Replace or unlock the test card; ideally a low-limit virtual card.
Then update the `E2E_STRIPE_CARD_*` secrets and re-run E2E Daily.
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
3ecea6f2e7 |
feat(storage): enable real purge with deletion audit log and orphan cleanup mode (v0.266.0) (#1127)
## Summary Andrew approved this follow-up to #1125 and #1126. It turns on real deletion, adds an audit log of every deleted object, and adds an orphan-folder cleanup mode. ### Real deletion on `STORAGE_RETENTION_DRY_RUN=false` and `STORAGE_RETENTION_MAX_JOBS_PER_RUN=50` are added to the backend deploy's `--set-env-vars`. The API service is the only thing that reads them: it passes `--dry-run` to `storage-retention-job`. That keeps the setting in one place, so a later deploy can't silently revert it. ### Deletion audit log - `storage-retention/deletion-logs/<ts>-<mode>.jsonl` has one line per deleted or failed object: timestamp, run_id, mode, job_id, path, size_bytes, generation, category and result. - The log is rewritten after every job or folder, so a crash still leaves a complete record. Each run also writes `<ts>-<mode>.summary.json`. - Deletes are pinned to the logged generation (`if_generation_match`). - `assert_deletable` refuses any path outside `jobs/{id}/`, so `storage-retention/` is never touched. No lifecycle rule matches live objects there. - The Firestore manifest on each job doc now also records generations and the log path. ### Orphan cleanup - `POST /api/internal/storage-retention/orphans` runs on the same Cloud Run job with `--mode orphans`. - It targets `jobs/{id}/` folders with no doc in `jobs`, `jobs-dev` or `youtube_upload_queue`, re-checked live per folder against a fresh listing. - It skips any folder with an object modified in the last 30 days, keeps `input/`, and deletes the rest. - `encoding_worker_jobs` doesn't count as a record: it's the encoder's expiring result cache, which deleted jobs leave behind. - It is a dry-run unless called with `dry_run=false&confirm=delete-orphan-job-folders`, and the scheduler never runs it. ### Docs - `docs/STORAGE-RETENTION.md`: run modes, log locations, jq review commands and restore one-liners (noncurrent `cp #gen` and soft-delete `restore`). - `scripts/storage_retention_report.py`: list runs, summarise a run, or find which run deleted a path or job. ## Testing New tests cover the log contents, generation-pinned deletes, per-job flushing across a crash, the path guard, orphan planning (input kept, 30-day skip, live record re-check, batch limit), and the endpoint and worker modes. Full backend suite: 5774 passed. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added admin-triggered orphan-folder cleanup, with dry-run enabled by default, optional folder limits, and explicit confirmation required for deletion. * Added deletion reports and audit logs covering removed objects, reclaimed space, and failures, plus a read-only tool to review runs and search records. * **Bug Fixes** * Added safeguards to prevent deleting changed objects or files outside their intended job folders. * **Documentation** * Expanded storage-retention guidance with recovery details, review commands, and cleanup instructions. * **Chores** * Set the production cleanup limit to 50 jobs per run and updated the package version. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
c8461e56d0 |
fix(e2e): self-heal CI approval — correct pending detection, re-approve open PRs (#1121)
## Summary This fixes the approval step from #1118, found by the live test on run 37140934446. That run reported `approved=true`, but bot PR #1119's CI run was still `action_required`: the old check treated *any* existing `pull_request` run as already running. - `approve_ci` now succeeds only once a `pull_request` CI run exists that is past `action_required`. It approves pending runs as it finds them. - When a self-heal PR is already open, the job now runs `approve_ci` for that PR instead of exiting. So a stuck bot PR gets unstuck on the next run. - Polling is tolerant of transient `gh`/`jq` failures (from the local CodeRabbit review). An empty branch name fails fast. ## Testing - `actionlint` is clean for the changed lines; local CodeRabbit review: 2 findings, both fixed. - After merge: a forced-LLM run on `main` will hit the "PR already open" path for #1119. That approves its CI with the workflow's own token and should let it auto-merge with no manual input. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
1c9104a79d |
fix(e2e): self-heal job approves its PR's CI run (no manual step) (#1118)
## Summary This follows #1116. The live test of the self-heal loop (bot PR #1117) merged, but only after one manual step: approving its CI run. - **Approve the bot PR's CI run.** A bot-authored PR's `pull_request` CI run is created as **action_required** (waiting for approval), and until it runs it outranks the passing CI Gate from the dispatched run. The `self-heal-pr` job now approves that run via `POST /actions/runs/{id}/approve`. If approval fails, it dispatches `ci.yml` and the Discord alert says the PR needs a one-click approval, with a link. - **Ruleset dependency, now documented.** `require_extra_approval_for_unattributed_changes` is now set to `false` on the `protect main` ruleset (11423976). It's a newer GitHub setting that defaults to true and blocks bot-authored PRs until a human approves. This is recorded in `docs/TESTING.md` and `docs/LESSONS-LEARNED.md`. ## Testing - `actionlint` is clean for the changed lines. - After merge: a forced-LLM run on `main` on a step with no learned recipe yet (`fillPostalCode`) will open a fresh bot PR and exercise approval → CI → auto-merge without manual input. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
4768164a94 |
feat(e2e): self-healing Stripe Checkout (LLM fallback + auto-merging fix PRs) (#1116)
## Summary Stripe's hosted checkout page has broken the daily E2E **5 times since April** (#916, #919, #945, #1115…). Each fix added another selector, and each new Stripe change still needed a human. This PR makes the Stripe stage repair itself, with no manual steps. **How it works.** `stripe-checkout.ts` now runs the checkout as verified steps (`selectCard`, `fillCardNumber`, `fillCardExpiry`, `fillCardCvc`, `fillCardholderName`, `fillPostalCode`, `uncheckSaveInfo`, `submitPayment`). Each step tries, in order: 1. **learned variants** (`learned-variants.json`, newest first), 2. the existing hand-written **built-in** selectors, 3. a **Gemini fallback** that drives the page one action at a time until the step's check passes. Model list: `gemini-3.5-flash-lite` → `gemini-3.1-flash-lite` → `gemini-3.8-flash`; IDs that aren't available are skipped. A step only counts as done when its **check** passes, never just because a click didn't throw (the root cause of run #192's misleading failure). **Fully self-healing loop.** When the LLM repairs a step *and* the purchase is confirmed server-side (credit granted), the new `self-heal-pr` job (on `main` only): - opens a **data-only** PR that changes only `learned-variants.json` (label `e2e-self-heal`, `@coderabbitai ignore`), - enables auto-merge, - dispatches `ci.yml` on the branch, because PRs opened with the workflow token don't trigger CI. The next run replays the recipe without the LLM. Discord and the daily email report 🩹 when this happens, and only one self-heal PR is open at a time. **Safety** - Card data never reaches the LLM. It may only use per-step `{{CARD_NUMBER}}`-style placeholders, which are substituted locally. Field values and card-number-like digit runs are redacted from what it sees, and inputs are masked in screenshots. Tests assert no secret appears in the prompt. - Guardrails refuse clicks on Link/Klarna/Cash App/Bank/"Back to…", and on the final Pay button outside `submitPayment`. They check the **resolved element** (text, aria-label, test ID, submit type), not just the model's label. - The Pay step's check is "redirected to the success page OR credit granted server-side" (throttled), so the fallback never submits twice. The cancel URL doesn't count as success. **Gemini key:** the AI Studio `gemini-api-key`, billed outside the nomadkaraoke project. A `GEMINI_API_KEY` repo secret is used if set; otherwise it's read from Secret Manager via keyless login, and the credential file is deleted before the tests run. With no key, the test runs deterministic-only, as before. Also passes the existing `E2E_STRIPE_ZIP` secret, which was never wired up. ## Testing - **Offline:** `playwright.selfheal.config.ts`, 21 tests with fixture Stripe pages served on fake checkout.stripe.com/js.stripe.com and a stub LLM. Covers: known layout uses no LLM; unknown layout → LLM repairs it → recipe learned → replayed without the LLM; no secrets reach the LLM; guardrails, including CSS targets; deterministic-only failure mode; unit tests for redaction, merging and frame keys. Added to CI's smoke job. - **Live (real Stripe, real Gemini) on this branch,** with `self_heal_force_llm=selectCard`: - [run 37137960148](https://github.com/nomadkaraoke/karaoke-gen/actions/runs/37137960148): `gemini-3.5-flash-lite` completed "choose Card" in one call, credit granted 1→2. This exposed the session ID in Stripe's URL path, now fixed. - [run 37138459027](https://github.com/nomadkaraoke/karaoke-gen/actions/runs/37138459027): repeated after the review fixes. Recipe `[data-testid="card-accordion-item"]` @ `checkout.stripe.com/c/pay`, credit granted. - The PR-opening job is skipped on branches by design. - After merge, I'll dispatch once on `main` with `self_heal_force_llm=selectCard` to exercise the PR → CI → auto-merge path. - Local CodeRabbit review: 2 findings, both fixed. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
8490ab75eb |
chore(gemini): move every Gemini call from Vertex AI to the Gemini Developer API (#1108)
## Summary Moves **every** Gemini call in gen off **Vertex AI** (billed to the `nomadkaraoke` GCP project, ~$85/mo) onto the **Gemini Developer API** using the `gemini-api-key` Secret Manager secret (an AI Studio key billed outside this project). Once deployed and verified, `aiplatform.googleapis.com` / `generativelanguage.googleapis.com` get disabled in nomadkaraoke. ### Key injection - `backend/services/gemini_client.get_genai_client(timeout_ms)` — key from `GEMINI_API_KEY`, else Secret Manager `gemini-api-key` via `settings.get_secret` (runtime SA already has project-level `secretAccessor`). No `project`/`location`. - Service: CI `--set-secrets ... GEMINI_API_KEY=gemini-api-key:latest`. - Jobs: Pulumi secret env on `lyrics-transcription-job`, `video-encoding-job` (renders + translations), `nomad-error-monitor` — **already applied** (targeted `pulumi up`, only that env added). Other jobs fall back to Secret Manager. - Local scripts (`frontend/scripts/translate.py`, `scripts/measure_autocorrect_cost.py`, `scripts/test_agentic_correction.py`): env, else `gcloud secrets versions access` (in-process only). - **Rotation:** add a new secret version; new instances / job executions pick it up. No code change. ### Degradation (quota / billing / invalid key; per-minute 429s stay retryable) | Caller | Behaviour | |---|---| | auto-correct | non-retryable 503 → compare mode drops the Gemini leg, keeps Opus | | match-judge / free-text / parse-titles / tenant bulk | existing fallbacks (no suggestion / blanks / regex) | | credit eval | `pending_review` (fail-closed, unchanged) | | lyrics translation | no retry; render proceeds without translation, status recorded | | custom lyrics | clean 503 "temporarily unavailable…" (+ 5-min timeout so it can't hang) | | error-monitor LLM | no retry; alerts sent without AI summary | | translate.py | exit 2 "Gemini quota/credit exhausted — top up AI Studio or rotate gemini-api-key" | `note_gemini_failure()` also sends one Discord ops alert per 6h (Firestore-throttled across instances via new `ops_alerts.send_throttled_ops_alert`). ### Infra - Pulumi: removes `roles/aiplatform.user` from `karaoke-backend` — **will be applied after this deploys and Vertex traffic is confirmed at zero**. - Agentic correction default model `gemini/gemini-3.8-flash` (`vertexai/` prefix still accepted, routed to the Developer API). - Version 0.261.1 → 0.262.0. Docs: CLAUDE.md, LESSONS-LEARNED, README, ARCHITECTURE, API, etc. ## Testing - New: `backend/tests/services/test_backend_gemini_client.py` (classifier, key resolution, no-Vertex client kwargs, alert throttle, per-caller degradation), `tests/unit/lyrics_transcriber/test_agentic_gemini_provider.py`, `frontend/scripts/test_gemini_client.py`, lyrics-translation quota test. - `pytest backend/tests --ignore=backend/tests/emulator` — all pass; `pytest tests/unit` — 2021 passed. - Verified `gemini-3.8-flash` is listed on the Developer API and answered a 5-token test call with the key. - `translate.py --dry-run` on `frontend/messages` and `backend/translations` OK. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Updates** * AI-powered lyric translation, custom lyrics, auto-correction, title parsing, match judging, and related features now use the Gemini Developer API. * AI features use the configured Gemini API key. Quota, billing, or key issues may show an unavailable message or use an existing fallback instead of repeated attempts. * **Documentation** * Updated product, API, setup, and recovery guidance to reflect Gemini API key configuration and usage. * Updated guidance for handling AI service availability and quota issues. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
49464e8fcc |
chore(cost): GCP cost cuts round 2 (v0.261.4) (#1107)
Round 2 of the GCP cost cuts, approved by Andrew. Target: the whole project under $300/mo; the current run-rate is about $400. ## Changes - **karaoke-backend min-instances 1 → 0** (`ci.yml`), about $24/mo. The backend Cloud Run service isn't defined in Pulumi, so the CI flag is the only place this is set. Andrew accepts a ~15s cold start. The 60s `/api/health` uptime check and the `*/10` schedulers keep an instance warm most of the time. With `--cpu-throttling` (request-based billing), a warm instance above the minimum costs nothing while idle. - **audio-separator Cloud Run GPU service shut down** (us-east4), about $22/mo. It is now behind the Pulumi flag `audioSeparatorServiceEnabled`, off by default. The service, SA, IAM bindings, output bucket and AR repo were already removed with a targeted `pulumi up`; the bucket needed the break-glass SA. `audio-separation-job` is untouched; it uses `karaoke-backend-gpu/karaoke-backend`. The separator health check no longer depends on `AUDIO_SEPARATOR_API_URL`, which also fixes a status card that always showed "offline". The CLI now logs a clear hint when the remote API fails. Companion PR: nomadkaraoke/python-audio-separator#309 makes its Cloud Run deploy manual-only. - **backup-to-aws no longer copies job files to S3**, about $15/mo egress. Job files now rely on GCS versioning plus soft-delete. Firestore, secrets, BigQuery, kn-data and git-bundle backups are unchanged. The freshness monitor no longer checks `gcs/job-files/`, and DISASTER-RECOVERY.md is updated. - **Artifact Registry cleanup policies**, about $8–10/mo. Per package: keep the 10 most recent versions plus anything tagged `latest`, `content-` or `cache`; delete all other versions older than 7d. Until now, tagged per-commit images were never deleted. Dropped the CI fallback that copied the GPU base once from us-east4, and deleted the legacy 13 GB us-east4 GPU base package. - Docs: GCP-COST-OPTIMIZATION.md (new section, including how to redeploy the separator on demand), ARCHITECTURE.md, DISASTER-RECOVERY.md. Changes made outside Pulumi (gcf-artifacts cleanup policy, destroying superseded secret versions) are recorded in GCP-COST-OPTIMIZATION.md. ## Tests - `backend/tests/test_system_status_endpoint.py` (new separator status tests) - `infrastructure/functions/backup_to_aws` (49 tests, including a new one that checks job files aren't synced) - `tests/unit/test_audio_remote.py`, `test_audio_processor_local_gpu.py` 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Service Availability** * The hosted audio-separation service is no longer available. Audio separation can run locally; users relying on the hosted service will need another deployment. * **Backup and Recovery** * New job files are no longer copied to off-site S3 backups. S3 recovery is limited to files backed up through September 30, 2026; newer outputs may need to be regenerated. * **System Status** * Audio-separator health status now reports the locally installed package version without checking a remote service. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
e4dc50d9e3 |
fix(ci): sanitize PR title in deploy env vars (v0.261.3) (#1109)
The v0.261.2 main deploy (run 36945703908) failed at **Deploy to Cloud Run (candidate revision)** with: `argument --set-env-vars: Bad syntax for dict arg` The #1106 PR title contained a comma, and `PR_TITLE` is spliced into the comma-delimited `--set-env-vars`. The bug was already there and that title was the first to trigger it. Its Cloud Build step succeeded. Fix: in both build-meta steps, commas become `;`, and newlines and shell-active characters (`\" \` $ \\`) are removed. Version bumped to 0.261.3. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
dd795c66a0 |
ci(cost): build backend images in Cloud Build, run deploy-backend on ubuntu-latest (#1106)
## Summary Retires the self-hosted `gha-build-*` runner VMs (e2-standard-8, `docker-build` label, ~$21/mo). `deploy-backend` now runs on free `ubuntu-latest`; the Docker images are built by **Cloud Build in us-central1**. ## Why not build on the hosted runner? (egress math) The app images are built `FROM` bases in us-central1 Artifact Registry, and the app layer has a `RUN pip install`, so the full base must be pulled each deploy: - CPU base `karaoke-backend-base`: **4.22 GB** compressed - GPU base `karaoke-backend-gpu-base`: **13.1 GB** compressed - ~120 main deploys/month (119 in Sept, 11 so far today) On a GitHub-hosted runner that is GCP internet egress: 17.3 GB × 120 × $0.12/GB ≈ **$250/mo** (CPU-only would still be ≈ $61/mo). Both are worse than the $21 VM. Cloud Build in us-central1 pulls same-region (free). Build-minutes on the default-pool e2-standard-2 machine are in the **2,500 min/month free tier** ($0.006/min after that). A test build of the real images (CPU and GPU in parallel) took **6m55s**, about the same as today's ~7.3 min of sequential builds on the VM. That's ~840 min/mo, so the expected cost is ~$0 (worst case ~$5/mo). ## Changes - `infrastructure/cloudbuild/backend-images.yaml` + `build-backend-images.sh`: buildx (docker-container) builds with the **same tags and registry caches** as before. CPU and GPU build as parallel steps, and each build gets a fresh metadata-server token. - `ci.yml` `deploy-backend`: `runs-on: ubuntu-latest`. It keeps the hash checks (manifest + ~10 KB config blob only) and submits `gcloud builds submit --async`, then streams the logs and checks the status. - Base rebuilds (≈ monthly) use e2-highcpu-8 with 300 GB disk and a 2 h timeout. - A Cloud Build still running when the job is cancelled or fails is cancelled. CI concurrency cancels superseded deploys, and an orphaned build would otherwise still push `:latest`. - Removed the disk-space step, the local base pull, and the one-time us-east4→us-central1 GPU-base copy. The copy finished 2026-09-26, and on a hosted runner it would stream 13 GB through egress. - `build-runner-images.yml`: `variants=all` no longer bakes the unused `build` family. It can still be built with an explicit `variants=build`. - Docs: EPHEMERAL-GHA-RUNNERS, SELF-HOSTED-RUNNERS, GCP-COST-OPTIMIZATION. No Pulumi or IAM changes were needed. `github-actions-deployer` already has `cloudbuild.builds.editor`, `storage.admin` and `iam.serviceAccountUser`, and the default Cloud Build SA (compute default) already has `artifactregistry.writer` on both repos. ## Rollback (one commit) `git revert <squash-merge sha of this PR>` and push to main. That restores `runs-on: [self-hosted, linux, gcp, docker-build]` and the old in-job buildx steps. Nothing outside the repo was changed: the runner-manager `build` family is untouched, and the `gha-runner-build` GCE images (20260812, 20260901) are kept so the dispatcher can still launch `gha-build-*` VMs. If the build image has been pruned by then, re-bake it with Actions → Build GHA Runner Images → `variants=build` (only `variants=all` no longer includes it). ## Testing - I ran the real CPU and GPU builds from this branch's config in Cloud Build us-central1, with `cbtest-*` tags and no cache writes. Both succeeded in 6m55s, and the us-east4 push uploaded only the app layers. - `actionlint` and `shellcheck` report no new findings. - Full verification is the first main deploy after merge. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Improvements** * Backend image builds now run through a managed build service in the central US region, with CPU and GPU images published as part of the deployment workflow. * The default “all variants” selection now excludes the build variant; explicitly requested variants remain available. * **Documentation** * Updated runner and cost documentation to reflect the current build and deployment setup. * **Maintenance** * Updated the package version to 0.261.2. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
354e43bab2 |
Render Hebrew/Arabic karaoke right-to-left; fix title-card tofu (v0.255.0) (#1094)
## Summary
A Hebrew KaraokeHunt job (5710831e, Omer Adam – "שני משוגעים") shipped
with its lyrics **laid out and highlighted left-to-right** and a **title
card of "?" boxes**. Diagnosed by measuring rendered pixels (not by
eye), reproduced locally and with prod's static ffmpeg 7.0.2 in Docker.
**Root causes**
- ASS style `Encoding=0` makes libass force LTR paragraphs; the `{\kf}`
before each word splits the line into runs, so each word is shaped
correctly but the words go left→right.
- `\kf` always wipes left→right within a word
([libass#406](https://github.com/libass/libass/issues/406)).
- PIL (title/end cards) has no per-glyph font fallback; the theme font
(Avenir Next) has no Hebrew glyphs.
**Fixes**
- `build_karaoke_styles` forces `Encoding=-1` (libass auto paragraph
direction; LTR unchanged).
- RTL lines get `{\frz(180+angle)\frx180\fry180}` — a visual no-op that
makes libass ≥0.15 sweep fills right→left without splitting words
(Arabic still joins). Theme angle preserved; re-emitted after duet
`{\r}` resets.
- Lead-in indicator mirrored for RTL lines.
- Text measurement mirrors libass: per-glyph fontconfig fallback, size
scaled from each font's metrics (old fixed 0.70 was 20%+ off for
fallback fonts).
- New `karaoke_gen/utils/font_fallback.py` (fontTools cmap +
fc-match/fc-list) used by title/end cards, portrait headers and ASS
measurement.
- Found by the new tests: inter-word gaps ≤0.1s were dropped, so the
highlight ran up to ~0.2s early by line end (all languages). Karaoke
tags now derive from one absolute centisecond timeline.
- Fixed pre-existing `test_render_video_worker_gce` failure on main
(MagicMock generation → `_g<mock>` id).
## Testing
- **Pixel-level render tests** (`test_ass_render_highlight.py` +
`ass_render_harness.py`): real `SubtitlesGenerator` output rendered with
ffmpeg/libass in a pure-colour palette, asserting on frames for English,
Hebrew, Arabic, Chinese, Japanese, Korean + an RTL duet line: highlight
starts at the reading-start edge, sung pixels always precede unsung in
reading order (word order + in-word fill direction), monotonic and in
time with the words, lead-in on the correct side, no missing glyphs, RTL
tags don't move the line (incl. rotated themes).
- 32 fail on pre-fix main; mutation checks ("Encoding only", "no re-emit
after `\r`", "ignore theme angle") each fail exactly the expected cases.
- 123/123 pass on Linux against **both** Debian ffmpeg (CI-equivalent)
and **prod's johnvansickle static ffmpeg** via
`scripts/run-render-tests-linux.sh`.
- CI now installs `fonts-noto-core fonts-noto-cjk`; under `CI=true`
these tests fail rather than skip on missing preconditions. (macOS
libass uses CoreText → font-sensitive checks skip locally.)
- Title-card tests (`test_font_fallback_render.py`): no `.notdef` tofu
for Hebrew/Arabic/CJK titles; Hebrew title reads right-to-left.
- Unit tests: `test_rtl_ass_output.py`, `test_font_fallback.py`.
- `make test`: unit 1972 passed; emulator 150 passed; frontend 1491
passed.
- Full local re-render of job 5710831e reviewed by Andrew: RTL highlight
smooth and correct, title card correct.
## Follow-ups (not in this PR)
- CDG output still renders RTL left-to-right / tofu (separate renderer).
- Separate investigation: transcription dropped 3 sung lines in this
job, which the section detector then labelled INSTRUMENTAL.
No local CodeRabbit review (CLI has no seat for this org), so leaving
this open for the CodeRabbit bot. A local code review ran; its findings
are addressed in the second commit.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Karaoke lyrics now render more reliably for Hebrew, Arabic, Chinese,
Japanese, and Korean, with improved reading-direction highlighting,
timing, and lead-in placement.
* Titles and portrait text can use fallback fonts when the selected font
lacks the necessary characters.
* **Bug Fixes**
* Improved lyric centering, word highlighting, and glyph rendering
across supported scripts.
* **Documentation**
* Added guidance on lyric rendering and multilingual test coverage.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
||
|
|
badd845a1d |
perf(jobs): move video-encoding-job (video + render) to us-east4 (v0.249.1) (#1083)
## Summary Follow-up to #1077. `video-encoding-job` runs both the post-review render (`render_video_worker`) and the final video (`video_worker`). In us-central1 each run waited **2–5 min** for Cloud Run Jobs to start it, so every customer job paid that twice. This moves it to us-east4 alongside the download, lyrics and bulk-search jobs. ## Cost (measured, not estimated) - Over the last 30 days the job read **156 GB** from Google services, mostly the us-central1 bucket, across 494 runs (Cloud Monitoring `container/network/received_bytes_count`, kind=google). That's about 0.32 GB per run. - Pricing from the Billing Catalog: GCS inter-region within North America is **free for the first 100 GB/month, then $0.02/GB**. Cloud Run writes to Google APIs are **$0**. - Internet egress (224 GB/month, Dropbox etc.) and Cloud Run CPU cost the same in either region. - Existing cross-region reads are about 24 GB/month (GPU job and lyrics), which the billing export shows at about $0. - **Net: about $1.60/month**, or about $3/month worst case. Time saved: roughly 4–8 min per customer job. ## Changes - **backend**: `trigger_video_worker` (via `_trigger_cloud_run_job`) and `trigger_render_video_worker` now use `cpu_jobs_region`. `GCP_REGION` stays us-central1; encoding VM and zone logic are unchanged. - **infra (Pulumi, already applied)**: new us-east4 `video-encoding-job`, with the legacy us-central1 copy kept for rollback and its deletion protection turned off. A preview after applying showed 290 unchanged. - **ci**: the video job image update now targets us-east4, using the us-east4 CPU image copy. - **Also updated**: rollback.sh, tests (video and render routing), TROUBLESHOOTING/ARCHITECTURE/LESSONS-LEARNED. ## Testing - `make test` passes: backend tests plus 1443 frontend tests. - Prod smoke test of the new us-east4 job: it imports `video_worker` and `render_video_worker` and calls the encoding VM's `/health` on its public IP (port 8080 is open to 0.0.0.0/0 with API-key auth). **Started in 25 s, exit 0.** 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Updates** - Video encoding and render jobs now run in the `us-east4` region, alongside other CPU jobs. - Video encoding continues to access storage and encoding resources in `us-central1`; cross-region transfer may incur costs. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
cb5ce7f162 |
chore(pypi): prune keep-window default 60→30 days (#1082)
Ran the PyPI prune runbook today (2026-09-29): publishing was blocked (`Project size too large`). The 60-day policy would only have freed 1.08 GB (→9.64 GB), refilling within ~a week, so pruned with `--keep-days 30` instead (69 releases deleted, 0.188.5–0.207.0; newest release of each older month retained) → ~6.8 GB. Re-ran the failed publish; 0.249.0 is live on PyPI. This PR makes 30 days the default in the planner script, the monthly reminder workflow, and the runbook. Unit tests pass (13). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
3e855ff65d |
perf(jobs): run download/lyrics/bulk-search Cloud Run Jobs in us-east4 — cut 2-5 min job start latency (v0.247.1) (#1077)
## Summary Every gen job waited **2–5 min** before `audio-download-job` (and then `lyrics-transcription-job`) even started. That blocked kjbox's goal of singing within about 4 minutes of making a song. **Root cause: Cloud Run Jobs in us-central1 are slow to start on Google's side.** Our image, CPU and code aren't the cause. - The `Started` condition fires about 10 s before the first Python log, so our process startup is fine. - decide's 203 MB image on 4 CPU is just as slow in us-central1. - A throwaway job running Google's tiny sample image, 4 runs per region: **us-central1 4m–5m07s**, **us-east4 8–19 s**, **us-east1 8–25 s**. All four us-central1 runs started in the same second, as if released from a queue. Quotas are far from their limits. ## Changes - **backend**: new `CPU_JOBS_REGION` setting (default `us-east4`) for triggering `audio-download-job`, `lyrics-transcription-job` and `bulk-search-job`. `GCP_REGION` is unchanged. `video-encoding-job` stays in us-central1, because it pulls multi-GB outputs from the us-central1 bucket. - **infra (Pulumi, already applied)**: the three jobs are created in us-east4 as new resources. The legacy us-central1 copies are kept as the rollback path, with deletion protection turned off so a follow-up can remove them. The preview after applying showed 289 unchanged, so there's no drift. - **ci**: the CPU app image is also pushed to `us-east4-docker.pkg.dev/nomadkaraoke/karaoke-backend-gpu/karaoke-backend-cpu`, so jobs pull from their own region. The lyrics job image update now targets us-east4. After each deploy, CI starts a no-op `audio-download-job` execution so the one-time ~2.5 min image import doesn't land on the first customer. - `scripts/rollback.sh`: lyrics job moves to the us-east4 image, and jobs whose target tag is missing are skipped instead of aborting the loop. - Docs: TROUBLESHOOTING runbook (measure + rollback), ARCHITECTURE, LESSONS-LEARNED. ## Testing - `make test`: all backend tests pass, plus 1422 frontend tests. - New tests in `backend/tests/test_worker_service_cpu_jobs_region.py`: routing of each moved job, video job stays in `GCP_REGION`, env override for rollback, and the Pulumi/backend region constants must match. - Prod smoke test of the new us-east4 `audio-download-job` with the real image (`import backend.workers.audio_download_worker`): **Started in 7.0 s, 8.6 s, 10.0 s** (first run of a new digest adds a one-time ~2.5 min import, which the CI step covers). - Local review via a reviewer agent: no must-fix findings; the rollback-path suggestions are addressed. ## Rollout One-time image seed (Cloud Build gcrane copy) and `pulumi up` are done. After merge, CI deploys the backend, which starts triggering the us-east4 jobs. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Performance** * Audio downloads, bulk searches, and lyric transcriptions now run in a region with faster job startup times, reducing delays before processing begins. Video encoding continues to run in its existing region. * **Reliability** * Rollbacks now skip jobs when the target image is unavailable and continue with the remaining jobs. Other image lookup errors still stop the rollback. * **Documentation** * Updated architecture and troubleshooting guides with regional job details, startup-delay guidance, and rollback instructions. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
8a5f754efd |
ci: remove post-deploy canary + never email test accounts on stale review (v0.245.2) (#1068)
## Summary
- Removes the `deploy-verify` job ("Deploy - Post-Deploy Canary (Happy
Path)") from `ci.yml`, plus the `Detect backend version change` step /
`version_changed` output that only gated it.
- `stale_review_processor`: jobs owned by automated-test accounts
(`is_test_email` — `e2e-test-runner@nomadkaraoke.com`,
`@inbox.testmail.app`) no longer get the 24h reminder, review-expired,
or duration-confirm-expired emails. They are still auto-expired at 48h
so stranded test jobs don't linger.
- Updates comments + `docs/TROUBLESHOOTING.md` / `docs/TESTING.md` that
referenced the canary. Version → 0.245.2.
## Why
The canary created a real production job (piri – dog, as
`e2e-test-runner@nomadkaraoke.com`) on every backend version bump.
`ci.yml` uses `cancel-in-progress: true`, so back-to-back merges
cancelled it mid-run — the process is killed before Playwright's
`afterEach` cleanup runs, stranding the job in `awaiting_review`. The
stale-review processor then emailed the test account (which forwards to
Andrew's inbox) at 24h/48h (e.g. job `adb8f393` from cancelled run
36288893657; earlier `a5f84cc3`, `10b532e2`). The canary was overkill;
the daily E2E and the traffic-gated candidate-revision smoke remain, and
the email guard backstops the daily E2E if it ever strands a job.
## Test plan
- [x] New unit tests: test-account reminder is silent (+ still flagged),
review expiry + duration-confirm expiry cancel without email
- [x] `pytest backend/tests --ignore=backend/tests/emulator` — 4883
passed
- [x] `actionlint` — no new findings
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* CI no longer runs a post-deployment production canary or sends its
test results and failure notifications. Candidate-revision smoke testing
remains in place.
* Production real-job verification is handled manually or by the daily
E2E run.
* **Documentation**
* Testing and rollback guidance now reflects the updated verification
and recovery steps. Daily E2E failures are identified as a reason to
roll back.
* **Bug Fixes**
* Automated test accounts no longer receive stale-review reminder or
expiry emails; existing expiry and cancellation handling remains in
place.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
||
|
|
87a36bd2a5 |
feat(encoding): GCP cost cuts — Spot 16-vCPU workers, 3-VM fleet, 5-min idle, gated worker deploys (v0.245.0) (#1061)
## Summary Approved GCP cost cuts for the encoding pipeline (target: whole project < $300/mo; encoding VMs were ~$300/mo for ~5-20 jobs/day). Details + before/after table: `docs/archive/2026-09-26-encoding-cost-cuts.md`. | | Before | After | |---|---|---| | Primary pair a/b | c4d-highcpu-32 on-demand | **c4d-highcpu-16 Spot** (STOP on preemption) | | Fallbacks | 8 VMs / 6 families | **1 on-demand** `encoding-worker-fallback-c2df` (c2d-highcpu-16, us-central1-f) | | Idle shutdown | 15 min / check every 5 | **5 min / check every 2** | | CI worker blue-green | every push to main | only when `infrastructure/encoding-worker/worker_code_paths.txt` paths changed since the serving version (+ `workflow_dispatch` `force_encoding_worker_deploy`) | | Nomad Dropbox upload | whole folder | minus lossless 4K **MP4** (tenant folders unchanged) | **Infra already applied locally (targeted Pulumi):** 7 fallback VMs + disks + 7 static IPs deleted; a/b + c2df resized in place; idle function env + scheduler updated. The Spot switch was done in place with `gcloud compute instances set-scheduling` on the stopped VMs + `pulumi refresh` (pulumi-gcp treats a `scheduling` diff as *replace*, and recreating a c4d VM risks a stockout). `encoding-worker-fallback-vms` secret → v6 (c2df only). CI's pulumi run should be a no-op for these resources. ### Scope change (per Andrew) **Task 5 dropped:** review previews stay on the encoding VM (`USE_GCE_PREVIEW_ENCODING=true` unchanged) and the review-page warmup is kept, since it deliberately warms the VM for the preview and the final render. To make sure the new 5-min idle timeout doesn't undermine that, the review page now also heartbeats **every 2 min** while the tab is visible and was used in the last 15 min, and re-warms on tab refocus (`useEncodingWorkerKeepAlive`). Before this, the heartbeat only fired on lyric edits. The idle checker keeps the routed VM alive while `last_activity_at` < 5 min, and both warmup and heartbeat write it. Tests cover both sides. ### Code changes - **Spot preemption mid-encode**: when a poll fails, `wait_for_completion` checks the pinned worker VM. If it is STOPPING/TERMINATED it raises `EncodingJobLostError`, so `run_with_lost_job_resubmit` resubmits the encode, and that resubmit restarts the Spot VM or falls back to c2df. Before, the failure showed up after ~7 min as a "lost contact" error that was not resubmitted. - **Stale fallback lists**: a candidate VM that no longer exists (`NotFound`) is skipped instead of aborting the whole failover. This protects the window before running instances pick up the new secret. - `SPEED_RANK` is now keyed by machine family, so the ranking is independent of VM size. - **Idle-stop vs. in-flight encode** (review follow-up): the idle function's `/health active_jobs` only counts RUNNING jobs, and `last_activity_at` was only written when a VM was started. With a 5-min window, a long encode's VM could be stopped in the seconds between the job finishing and the next poll. The poll loop now refreshes `last_activity_at` at most once a minute. A VM seen in STAGING/PROVISIONING after it accepted a job also counts as lost. CI `Read Firestore config` now writes empty values instead of "None". - **Worker deploy gate**: `backend/tests/test_encoding_worker_code_paths.py` derives the worker's transitive `backend.*` import set and fails if `worker_code_paths.txt` doesn't cover it. Every deploy still uploads the wheel and `startup.sh`, so a cold-booted VM always runs the latest code. - **Dropbox**: nothing automated reads lossless files from Dropbox. The lossless **MKV** is still uploaded because the completion email and the live Fiverr auto-reply bot both tell customers it is in the Dropbox folder, and the VocalStar tenant expects lossless files. To drop the MKV as well, update those two texts first, then add the MKV suffix to `DROPBOX_SKIP_OUTPUT_SUFFIXES`. ## Testing - New/updated unit tests: preemption → lost job → resubmit, NotFound candidate skip, family ranking, idle 5-min/heartbeat window, fleet-shape invariants, worker-path coverage, Dropbox exclusion, keep-alive hook (Jest, fake timers). - `pytest backend/tests --ignore=backend/tests/emulator`: 4802 passed. The only failures are 6 tests that need the spaCy `en_core_web_sm` model, which isn't installed in my local venv; unrelated to this PR. - Jest lyrics-review + lib/lyrics-review: 423 passed. Infra tests (`test_encoding_worker_config.py`, idle function): all pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
8cbcc03099 |
infra: GCP cost cuts (VPC connector, min-instances, Autoclass, weekly incremental git backup, hourly error monitor, PITR off) (#1063)
## Summary These are the cost cuts Andrew approved to get the whole GCP project under $300/mo. Performance compromises were accepted. - **VPC connector removed.** `cloud-run-connector` only existed to reach the old flacfetch GCE VM's private IP. flacfetch now runs on Netcup behind a public HTTPS URL. I checked every URL secret the backend uses and none are private IPs. The backend is detached with `--clear-vpc-connector`: just dropping the `--vpc-connector` flag from `gcloud run deploy` would keep the old setting. `audio-download-job` and `bulk-search-job` no longer set `vpc_access`. Saves ~$12/mo plus the Network Intelligence resource-hours. - **`karaoke-backend` min-instances 2 → 1** (ci.yml). - **Autoclass** on `karaoke-gen-storage-nomadkaraoke` (terminal NEARLINE, because customers re-download old jobs) and on `nomadkaraoke-divebar-files` (terminal ARCHIVE). Autoclass charges no retrieval or early-deletion fees. - **backup-to-aws** - **Git bundles:** now weekly (Sundays) and incremental. A repo is re-bundled only when its GitHub `pushed_at` is newer than its bundle's S3 `LastModified` minus a 1h lag, so S3 itself is the state store. Every repo is re-bundled on the first Sunday of each month. `?mode=git_repos[&full=1]` runs the git step alone. - **Bug fix:** since #997 every S3 **overwrite** has failed. The `backup-writer` IAM user has no `s3:GetObject`, so HEAD on an existing key returns 403. Git bundles and the manifest were stuck at 09-15, and the DR freshness monitor has been red every day. The existence check now uses `ListObjectsV2`. - **`gcs_sync`:** detects changes by `time_created` instead of `updated`. Autoclass transitions bump `updated`, which would have made the sync re-copy every object and pull it back to Standard. - **DR freshness monitor:** `git-repos/` is now judged on `manifest.json` (192h) plus a check of the manifest's contents. - **Error monitor:** runs hourly instead of every 15 min. Lookback is 60 min, equal to the interval, so there are no gaps or double counts. `MAX_LOG_ENTRIES` goes from 500 to 2000. - **Firestore PITR disabled.** The nightly export to GCS staging remains, plus the weekly copy to S3. - **`recover-stuck-downloads` / `retry-pending-render-jobs`:** every 5 → 10 min. Both are recovery sweeps with ~10-min stall thresholds. - Docs updated: DISASTER-RECOVERY (PITR restore replaced, git RPO, incident write-up), ARCHITECTURE, LESSONS-LEARNED, GCP-COST-OPTIMIZATION, TROUBLESHOOTING. **Kept:** the `audio-separator` GPU service. Its request logs show ~40 external `/separate` calls/month from remote-CLI users. It's already min-instances 0. ## Already applied to prod (before merge) - `gcloud run services update karaoke-backend --clear-vpc-connector --min-instances 1`, same image. Health is OK, and `/api/health/flacfetch` and `/api/health/encoding-worker` are both OK without the connector. - Targeted `pulumi up` on the Firestore DB, both buckets, both jobs and both 5-min schedulers. - backup-to-aws redeployed with `deploy.sh`. A manual `?mode=git_repos` run bundled 13 of 121 repos (106 unchanged, 2 skipped) and uploaded 120 files with 0 errors. - Deleted `gs://…/backups/flacfetch-torrent-backup-20260122-1931.tar.gz` (1 GB). ## After merge Targeted `pulumi up` for `error-monitor-scheduler`, which must wait for the image with LOOKBACK=60, then delete the connector and drop the `vpcaccess` API service from state (`disable_on_destroy=False`, so the API itself stays enabled). ## Tests `make test` passes locally. The backup_to_aws function tests (43, including new incremental, list-based-existence and gcs_sync tests) were run separately because CI doesn't run them. Related: nomadkaraoke/karaoke-decide#140 (ARCHIVE lifecycle on `nomadkaraoke-data`, already applied). 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Updates** * Pending-render retries and stuck-download recovery now run every 10 minutes instead of every 5 minutes. * Error monitoring now runs hourly and checks a 60-minute window. * Cloud Run candidate deployments use one minimum instance instead of two and no longer use a VPC connector. * **Backups and Recovery** * Git repository backups now run weekly, with monthly full refreshes and incremental updates between them. * Firestore point-in-time recovery is disabled; recovery uses nightly exports. * Storage buckets now use automatic storage-class management. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
7179e2863f |
ci: move test/package jobs to GitHub-hosted runners; GPU base image same-region (cost) (#1059)
## Summary GCP cost cuts in CI (approved): 1. **GitHub-hosted runners for test/package jobs** (~\$28/mo of `gha-general-*` VMs, ~390 VMs/30d). Moved to `ubuntu-latest` (free, public repo): `package-unit-tests`, `package-integration-tests`, `package-build-test`, `backend-unit-tests`, `backend-emulator-tests`, `package-build` (main), `frontend-e2e-full` (label-gated). None used GCP auth. Self-hosted disk checks replaced by a free-disk-space step (drop dotnet/android/ghc/CodeQL); venv cache keys renamed so self-hosted venvs aren't restored onto a different interpreter path. 2. **Removed `propagate-base-image`** — pre-pulled the base image onto "the 3 runners"; runners are now single-use ephemeral VMs so this just spawned 3 VMs per base rebuild for nothing. 3. **AR inter-region egress** (~\$12-16/mo): the GPU app image was built on a us-central1 runner `FROM` the ~13 GB GPU base in **us-east4**, pulling it cross-region on every deploy. The GPU base + all BuildKit caches now live in us-central1 `karaoke-repo`; only the final app image is pushed to us-east4 (`karaoke-backend-gpu`) where `audio-separation-job` pulls it same-region. Push uploads only the small changed app layers (base layer digests are identical). First deploy copies the current base from us-east4 via `buildx imagetools create` (byte-identical, no from-scratch rebuild); falls back to a rebuild only if that fails. 4. `build-runner-images.yml`: `KEEP_IMAGES_PER_FAMILY` 2 → 1 (GPU/windows images are 15-21 GiB). 5. `ci.yml` changes now trigger the package/backend test jobs (so this PR exercises the moved jobs). `deploy-backend` stays on the `docker-build` GCE runner (Docker builds, WIF, same-region AR). GPU runners untouched. ## Testing - This PR's CI runs the moved jobs on hosted runners. - `image_hash` helper + `imagetools create` cross-registry copy verified locally against two `registry:2` instances (copy wraps in a single-entry index; helper follows it). - actionlint: no new findings (fewer than before). 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * CI test, package, and frontend end-to-end jobs now run on GitHub-hosted runners. GPU image builds use a centralized base-image cache, while application images continue to be published in their existing region. * Runner image cleanup retains the newest image per family; images backing existing disks remain protected. * **Documentation** * Updated the runner guide to describe which jobs use hosted or self-hosted runners and how image retention works. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
f2b633175e |
chore(infra): cut encoding-worker disk cost (~$75/mo) + fix runner image pruning (#1055)
## Summary This cuts the GCP cost of the encoding-worker VM disks and custom images by about **$75/mo**. Functionality and instance-type diversity are unchanged. | Lever | Change | Est. saving | |---|---|---| | Boot disk size | All 10 workers go from 100 GB to **50 GB**. The Packer image `disk_size` goes from 100 to 30, because a disk can't be smaller than its source image. | ~$44/mo | | Hyperdisk IOPS/throughput | `-c4a` and `-n4db` were at 3600 IOPS / 290 MB/s, the paid defaults. They are now at the free baseline of 3000 / 140. New hyperdisks are created at 3000 / 140. | ~$18/mo | | Custom images | Deleted 23 old runner images, keeping the newest 2 per family. Fixed the prune step in `build-runner-images.yml` so they don't pile up again. | ~$13/mo | Measurements behind the sizing: - A worker 8 months into service used 15 GB of 100 GB. Fresh disks use 8 GB. - Job-folder scratch (1,904 jobs): p99 1.3 GB, max 4 GB. The deepest queue seen on a worker was 8. Full analysis, benchmarks and the rollout procedure are in `docs/archive/2026-09-26-encoding-worker-disk-cost.md`. ## Runner image prune bug The monthly step filtered with `-deprecated.state:*`, which is invalid. gcloud wrote the error to stderr, stdout was empty, and the step printed "Nothing to deprecate" every month. Deprecating wouldn't have saved anything anyway, since deprecated images are still billed. Also, gcloud's `family=gha-runner-gpu` filter is a word-boundary regex, so it matches `gha-runner-gpu-windows` too. The step now matches the family exactly on the client side, deletes images beyond the newest 2, and never deletes an image that an existing disk was created from. ## Rollout (stockout-safe) Pulumi can't shrink a boot disk. The only way it would get there is replacing the VM, which needs c4d/n4d/c2d capacity. So each **stopped** VM keeps running as the same instance, and only its boot disk is swapped with gcloud: detach, delete, create a 50 GB disk from the new image, attach it as boot. Then `pulumi refresh --target` on that instance. After that, `pulumi preview` shows no diff for the swapped VMs. ## Testing - `pytest infrastructure/test_encoding_worker_config.py`: 8 passed. A new invariant checks that the Packer image size is ≤ the worker disk size and that the worker disk is ≥ 40 GB. - Each swapped VM was started and checked with `/health` and a real full 4K `/encode`, and `df` was checked afterwards. Before/after timings are in the archive doc. c2df was the same before and after (142/139 s → 142/137 s), and n2f matched the August benchmark (247 s). ⚠️ Do NOT merge until every VM is swapped and refreshed. Merging with `size=50` while a 100 GB VM is still in Pulumi state would make a later `pulumi up` try to **replace** `encoding-worker-a/-b`. Status is tracked in the comments. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Cost Optimization** * Reduced encoding-worker boot disks from 100 GB to 50 GB, lowering estimated stopped-VM disk costs to about $4–5 per month. * Reduced the worker image disk size to 30 GB. * **Infrastructure Maintenance** * Image cleanup keeps the two newest active images per family and deletes older ones, while preserving images used by existing disks. * **Documentation** * Updated infrastructure guidance with disk sizing, image cleanup, and cost details. Added a report with rollout, verification, and rollback information. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
e9465c73d4 |
chore: gemini-3.8-flash everywhere + auto-correct on Opus 5.5 (cost) (#1054)
## Why GCP credits expired 2026-09-19; Vertex Gemini 3.1 Pro spend had grown $9 → $37 → $105/30d. `gemini-3.8-flash` is $0.75/$3.75 per 1M (vs $2/$12 for 3.1 Pro; rises to $1.50/$7.50 on 2027-01-01). Opus 5.5 is $4/$20 (vs $5/$25 for 4.8). ## Changes - All Gemini defaults → `gemini-3.8-flash`: credit eval, custom lyrics, auto-correct (default + compare leg), match judge, parse titles, tenant bulk analyze, agentic AI router, error-monitor LLM analysis (also `us-central1` → `global`, required for Gemini 3.x), frontend `translate.py`. - Auto-correct Claude leg `claude-opus-4-8` → `claude-opus-5-5` (ci.yml env), with `effort: "high"` pinned (5.5 defaults to medium). Pricing table gains both new models. - Docs updated; version 0.241.1. ## Evidence (12 recent prod jobs, pre-review lyrics scored against the human-reviewed final) | Model | Precision | Word-level recall | $/job | |---|---|---|---| | gemini-3.1-pro-preview | 91% | 74% | $0.067 | | gemini-3.8-flash | 98% | 38–41% | $0.020 | | gemini-3.8-flash @ high thinking | 97% | 60% (4/12 timed out, 504) | $0.041 | | claude-opus-4-8 | 94% | 33% | $0.080 | | claude-opus-5-5 (effort high) | 93% | 58% | $0.048 | Flash for the Gemini leg is a deliberate cost/recall trade-off (Andrew chose it); Opus 5.5 more than compensates. Also verified: `gemini-3.8-flash` works on Vertex `global` (rejects `thinking_level=minimal`; nothing here uses it), match-judge latency median 3.3s / max 6.2s (< 12s timeout), Japanese translation smoke test preserves ICU placeholders. Translation cache is keyed on English text + locale, so no re-translation churn. Tests: auto_correct, credit eval, custom lyrics, parse_titles, error_monitor, agentic router suites pass locally. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * AI-powered correction, translation, matching, title parsing, and evaluation now use updated models, including Gemini 3.8 Flash and Claude Opus 5.5. * **Updates** * AI service pricing reflects scheduled rate changes, including a Gemini 3.8 Flash rate change beginning January 1, 2027. * Application version updated to 0.241.1. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
d570b1f139 |
feat(kjbox): partner API for singer email-code sign-in + show credits (v0.239.0) (#1051)
## Summary
Partner API so the kjbox singer UI (live karaoke nights) can turn
patrons into real karaoke-gen customers: the singer verifies their email
with a 6-digit code typed into the kjbox page, and kjbox then searches
audio / creates the job **as that user** (normal gen delivery emails).
All under `/api/kjbox`, gated by `X-Kjbox-Secret` (Secret Manager
`kjbox-partner-secret`, wired in ci.yml; 503 when unset):
- `POST /auth/send-code` — emails a 6-digit code (hashed at rest, 10-min
expiry, new code supersedes old). Exempt from the 2-signups-per-IP cap
(a venue shares one IP) — replaced by a partner-wide 100 new accounts /
24h cap + 5 codes / email / hour. New users tagged `signup_source=kjbox`
(+ venue).
- `POST /auth/verify-code` — 5 attempts then burned, single-use; shares
the post-login path with magic-link verify (refactored into
`complete_verified_login`: welcome credit w/ AI eval, locale, referral,
session, welcome email).
- `POST /credits/show-credit` (+ user Bearer) — +1 credit, no email
("free at Andrew's shows"), idempotent per key, 5 / user / 24h.
- Review-needed + 24h reminder emails for `X-Client-Id: kjbox*` jobs
carry a one-click sign-in link to the review page (singers never signed
in on gen's site).
Paired kjbox PR: nomadkaraoke/kjbox (singer make-it flow, v0.116.0).
Plan: kjbox `docs/archive/2026-09-24-singer-make-it-auto-flow-plan.md`.
## Testing
- New: `backend/tests/test_kjbox_partner_api.py`,
`backend/tests/emulator/test_kjbox_partner_service.py` (+ additions to
existing tests for the verify refactor, review-email links, X-Client-Id
on create-from-search)
- Backend unit suite: 4644 passed; emulator suite passed.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added partner sign-in with one-time email codes and show-credit
claiming, with limits on code requests and credit grants to prevent
repeated or duplicate claims.
* Added localized sign-in emails and single-use review links for
eligible reminders.
* Sign-in supports welcome-credit evaluation and referral attribution.
* **Improvements**
* Review reminders can use custom links, and new accounts can record
signup source and venue.
* Credential-like request headers are redacted from stored metadata.
* **Documentation**
* Added partner API usage and behavior details.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
||
|
|
7b4af11362 |
perf: cut backend cold start ~16s → ~4s via lazy imports + background warmup (v0.235.0) (#1036)
## Summary
A freshly-spawned `karaoke-backend` Cloud Run instance took **~16s
before serving its first request** — Cloud Run holds routed requests
during boot, so every scale-out was a 16s latency cliff (measured
2026-09-22 during the 16-tab load test: warm instance 1.2s vs fresh
instances 15.8–18.8s for the same request). Handoff spec:
`docs/archive/2026-09-22-gen-cold-start-reduction-handoff.md`
(workspace).
Measured breakdown: **~8–9s Python imports** + **~7.5s lifespan
preloads** (spaCy ~1.0s, NLTK cmudict+punkt ~2.3s, Langfuse ~3s,
credential validation ~1.6s — all gating readiness, from prod logs).
## Changes
**Lazy imports, fixed at the source module** so every importer benefits
and the CLI keeps working (imports now happen at first use inside worker
code paths):
| Module | Deferred | Cost removed |
|---|---|---|
| `karaoke_gen/.../utils/syllable_counter.py` | spacy, nltk, pyphen,
syllables → `__init__`/methods | ~2.6s |
| `karaoke_gen/.../correction/phrase_analyzer.py` | spacy → `__init__`
(was also pulling **torch** via thinc) | ~1.7s |
| `karaoke_gen/audio_processor.py` | audio_separator
(torch/onnxruntime/librosa), pydub | ~2.9s |
| `backend/config.py` | google.cloud.secretmanager → `get_secret()` |
~1.7s |
| `backend/services/{custom_lyrics,auto_correct}/service.py` |
google.genai → `_call_gemini()` | ~1.2s |
| audioshake/whisper transcribers, cdgmaker composer | pydub → using
methods | small |
`audio_processor` keeps `Separator` / `AudioSeparatorAPIClient` /
`REMOTE_API_AVAILABLE` / `SEPARATOR_AVAILABLE` as (lazy) module
attributes via PEP 562 `__getattr__`, so existing test patch targets are
unchanged; intra-module code uses helpers because LOAD_GLOBAL bypasses
module `__getattr__`. Same pattern keeps `custom_lyrics.service.genai`
patchable.
**Lifespan** (`backend/main.py`): the spaCy/NLTK/Langfuse preloads +
credential validation moved into a daemon thread
(`_run_background_warmup`). They warm caches for the *first
lyrics-processing job* — they never needed to gate HTTP readiness.
Workers racing the warmup fall back to the preloaders' existing lazy
paths. Heavy imports happen in this thread, not in async handlers
(event-loop-blocking gotcha from v0.230.0/v0.231.1).
## Results
- Local `import backend.main`: **10.63s → 2.75s** (remaining is
dominated by `google.cloud.firestore` ~0.8s, a genuine serving
dependency)
- Lifespan startup: ~7.5s → instant ("Application startup complete"
same-second)
- Expected prod cold start: **~16s → ~3–4s** (to be verified post-deploy
with `scripts/load-test-review.py --tabs 16` + fresh-instance request
latency)
## Tests
- **New:** `backend/tests/test_cold_start_imports.py` —
subprocess-imports `backend.main`, asserts none of the heavy libraries
(torch, spacy, nltk, audio_separator, matplotlib, pydub, google.genai,
secretmanager, librosa, onnxruntime) leak into `sys.modules`;
`backend/tests/test_main_startup_warmup.py` — warmup calls all preloads,
survives individual failures, and doesn't block readiness.
- Full backend + package suites: **4541 passed** (incl. the
audio_processor/custom_lyrics/auto_correct patch-target suites). Ruff:
no new findings on changed files.
## Follow-up (separate PR after prod verification)
If fresh-instance first-request latency lands <5s, drop `--min-instances
2 → 1` in `ci.yml` — the second warm instance was the cold-start
mitigation (~$15/mo).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Performance**
- Reduced backend cold-start time by deferring heavy dependencies until
they are needed.
- Startup readiness is no longer blocked by model, language-resource,
analytics, or credential warmups; these now run in the background.
- Secret Manager and audio, transcription, lyric, and correction tools
load on demand.
- **Documentation**
- Added guidance and measured results for diagnosing and reducing
backend cold starts.
- **Quality**
- Added automated checks to ensure startup remains lightweight and
background warmups continue safely after individual failures.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
||
|
|
3d1ee6a621 |
feat: KaraokeHunt retired-app request interceptor — auto-convert to Gen users (v0.235.0) (#1037)
## Summary The retired KaraokeHunt mobile app still has installs submitting ~5-10 song requests/month. Each request POSTs to the dead `create.karaokehunt.com`, fires a client-side Pushbullet push to Andrew, and shows the user a hardcoded *"you should receive an email in 5-10 minutes with a link to your new karaoke video"* — a promise broken for years. This PR makes it true and converts those users into Gen users (Andrew's spec, BACKLOG Frog #2 extension). ## How - **Cloudflare Worker** (`infrastructure/cloudflare/karaokehunt-interceptor/`, deployed via `deploy.sh` — the karaokehunt.com zone is not IaC-managed): answers the app instantly, `waitUntil`-forwards the payload to the new backend intake with a shared secret. - **`POST /api/karaokehunt/request`**: unauthenticated intake, gated by `X-KH-Forwarder-Secret` (Secret Manager `karaokehunt-forwarder-secret`, already created; wired in ci.yml `--set-secrets`), 10/min/IP rate limit, lenient payload parsing, Firestore `karaokehunt_requests` audit trail. - **Conversion** (`backend/workers/karaokehunt_conversion.py`, mirrors `community_daily_pick`): - new email → account + 1 credit (`karaokehunt_app_conversion`) → job created **as them** (consumes it); first sign-in still grants the welcome credit (deliberate — next song covered) - existing user with credits → job on their credit (email offers to restore it on reply) - no credits, or over `KARAOKEHUNT_DAILY_JOB_CAP` (default 3/day, bounds unattended spend) → free community requests board (dedup → upvote) - conservative `pick_auto_selection` (not `select_best`): unconfident matches park in `AWAITING_AUDIO_SELECTION` and the email asks them to pick — garbage app inputs must never auto-publish to YouTube - 14-day per-email+song dedup; failures retryable via admin `POST /api/karaokehunt/internal/reprocess/{doc_id}` - **Email** `send_karaokehunt_conversion`: 3 variants (job started / pick the recording / on the board), 7-day one-click sign-in link, KaraokeNerds community-version bonus link when one exists. EN + all 32 locales via translate.py. ## Safety Deploys **dark**: endpoint 503s until the secret env lands, and receives no traffic until the Worker + DNS flip (done post-merge via `deploy.sh`). Kill switch = delete the Worker route; backend-side = remove the secret mapping or `KARAOKEHUNT_DAILY_JOB_CAP=0`. ## Testing - 36 new tests (`test_karaokehunt_intake.py`, `test_karaokehunt_conversion.py`): secret/rate-limit/JSON gates, all conversion routings, credit-grant guards, park-on-unconfident-search, email variants + HTML escaping, reprocess auth. - Full `make test` green (backend + frontend). - Post-deploy: live end-to-end POST through `create.karaokehunt.com` mimicking the app payload, verifying account/job/email/Firestore. Plan doc: `docs/archive/2026-09-22-karaokehunt-app-interceptor-plan.md` 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Requests submitted through the retired KaraokeHunt app are now received and converted into Nomad Karaoke jobs, accounts, or community-board requests. - Users receive localized email updates with job status, credit usage, audio-selection prompts, and tracking links. - Requests can be retried by administrators when processing fails. - Duplicate protection, request limits, and daily processing caps help prevent repeated or excessive submissions. - **Documentation** - Added API and deployment documentation for the request-forwarding workflow. - **Chores** - Released version 0.235.0. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
6514673fde |
perf: Cloud Run burst headroom — concurrency 32 + min-instances 2 (#1035)
## Summary Final tuning from the 16-tab test diagnosis (#1033 fixed the edge rate limit eating the requests; this fixes the remaining latency). With nothing failing anymore, the warm 16-tab run still showed every tab at ~17–19s — server logs pinpointed why: the one warm instance answered `correction-data` in **1.2s** while the three instances the burst spawned answered in **15.8–18.8s** — Cloud Run holds requests during the ~16s app boot. Concurrency 16 (set in #1028 while the event loop still had blocking hazards) now overshoots: the loop is clean, one instance juggles dozens of light requests fine, and forcing scale-out per burst just converts load into cold-start waits. ## Change `--concurrency 16 → 32` and `--min-instances 1 → 2`: 64 warm request slots, so a 16-tab reload (~48 concurrent API calls) is absorbed with zero cold starts. Still far below the original 80 that allowed the 2026-09-18 single-instance wedge. ## 💰 Cost note (Andrew) The second min-instance idles at Cloud Run idle rates — roughly **$15/month**. Single-flag revert if you'd rather not pay it; without it the first burst after idle pays one ~16s cold-start wave for the overflow beyond 32 concurrent slots. ## Follow-up worth considering (not in this PR) App cold start is ~16s — heavy import graph (matplotlib, pydub, GCP clients). Lazy-importing the heavy tails would benefit every scale-out. ## Testing Deploy-flag-only change. Post-deploy: re-run `scripts/load-test-review.py --tabs 16` — expect single-digit-second tabs with no cold-start waves. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
30cf41e7d6 |
feat: concurrent-reliability phases 2-5 — Cloud Run shape, degradation telemetry, banner tuning, load-test guard (#1028)
## Summary Phases 2–5 of the concurrent-load reliability initiative (Phase 1 = #1024, verified live: the 10-tab meltdown went from 9/30 failed requests to 30/30 OK). Verbatim spec: `docs/archive/2026-09-17-concurrent-review-reliability-verbatim-record.md`; plan: `docs/archive/2026-09-18-concurrent-review-reliability-investigation-and-plan.md`. ## P2 — Cloud Run shape - **`--concurrency 80 → 16`** for karaoke-backend: the service is one uvicorn process per 2-vCPU instance, and review loads are heavy; at 80 a burst never triggers scale-out and all lands on the single min-instance. 16 makes bursts fan out while staying well above normal per-instance load. (Deploy flags live in ci.yml — the service is not Pulumi-managed.) - **Local preview ffmpeg fallback bounded** (`LOCAL_PREVIEW_MAX_CONCURRENCY=1`, 900s queue timeout, fast RuntimeError after that): the GCE-outage fallback runs ffmpeg ON the API instance; unbounded, a few concurrent Preview clicks during an encoder outage pegged both vCPUs, stalled the health probe, and showed everyone the orange banner (Andrew's banner-during-preview screenshot). ## P3 — degradation telemetry (the explicit ask) - New **`POST /api/client-events`** (unauthenticated so review-token users report too; 30/min/IP rate limit; bot-filtered; detail capped; URLs query-stripped). Each event → Firestore `client_events` collection **and** a structured `client_event type=…` log line, so "how often are users seeing this?" is reviewable via Firestore aggregation or Cloud Logging correlation. - Frontend reporter `lib/degradation-events.ts` (60s per-type throttle, localhost no-op, never throws), wired into: - `BackendStatusBanner` → `banner_reconnecting` / `banner_unavailable` with stall age, in-flight count, probe state - review lyrics-load failure → `lyrics_load_failed` with message/status - `VocalsAudioDataLoader` → `waveform_slow` (>20s to first strips) and `waveform_failed` (gave up) with poll/retry counts ## P4 — banner tuning - Escalating to the full "We're having trouble reaching our servers" card now requires **two consecutive failed health probes** (`UNAVAILABLE_PROBE_FAILURES=2`). One failed 4s probe against a briefly-pegged instance shows only the calm "Reconnecting…" pill. Single-failure behavior for the pill is unchanged, as is instant clearing on recovery. ## P5 — regression guard - **`scripts/load-test-review.py`** checked in: replays the 10-tab incident (Firestore-ADC review tokens, curl request trios, pass/fail thresholds, exit code). Documented in `docs/TESTING.md`. Validated against prod post-Phase-1: 9/9 requests OK, tabs < 3s. ## Testing - Backend: 6 new tests for client-events (persist/validate/rate-limit/bot/fail-soft/detail-cap) + local-preview bound test — full suite **4517 passed**. - Frontend: new degradation-events suite, two-probe escalation + debug-snapshot tests; one existing api-fetch test updated to the new escalation contract — full suite **1298 passed**. - Live validation of the load-test tool against prod (small 3-tab run): PASS. ## Notes for review - Cloud Run concurrency change means bursts now spin up extra instances (billable during requests only; max-instances unchanged at 20). - Firestore `client_events` has no TTL policy yet (needs a gcloud/console action our read-only ADC can't do); volume is tiny (throttled per client) — can add later if it ever matters. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added automatic reporting for degraded experiences, including reconnecting or unavailable backend status, failed lyrics loads, and slow or failed waveform loads. - Added concurrent review-page load testing to help verify performance under multiple open tabs. - **Improvements** - Backend status now remains in “reconnecting” until multiple health checks fail, reducing premature unavailable messages. - Local preview processing is limited during heavy demand to improve stability. - **Documentation** - Added API documentation for client telemetry and guidance for running concurrent-load regression tests. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
3cdb62a058 |
fix: run render-video worker as Cloud Run Job to survive backend deploys (#994)
## Summary Tenant E2E run #107 (`singa`) failed with job `41e06b90` frozen at `rendering_video` — even though the GCE encoder **finished the render and uploaded every output**. Root cause: a backend deploy landed mid-render and killed the render poll, which ran as a FastAPI BackgroundTask on the Cloud Run service. Full investigation: `docs/archive/2026-09-13-render-deploy-survival-investigation.md` ### The failure chain (4 layers, each structurally broken) 1. **BackgroundTask execution** — the render endpoint returns 200 immediately and polls post-response; Cloud Run SIGKILLs old-revision instances ~10s after SIGTERM on every deploy. 2. **Unreachable shutdown parking** — uvicorn waits for background tasks *before* running lifespan shutdown, so `park_active_render_jobs_for_shutdown()` never runs in exactly the scenario it was written for. Its comment claimed a 600s grace period; Cloud Run gives 10s. Verified in logs: the old revision's last lines were uvicorn's `Waiting for background tasks to complete.` — then SIGKILL. 3. **Cloud Tasks retry defeated** — the queue got its 200 at dispatch; `dispatch_deadline=1800s` protects nothing. 4. **45-min stuck sweep** — reachable but slower than the E2E's 30-min budget and a ~50-60 min silent stall + wasted re-render for a real customer, on every deploy that lands mid-render (deploys go out on every merge to main). This is the **identical bug class fixed for the video worker in 2026-03** (`docs/archive/2026-03-08-fix-encoding-deploy-survival.md`, `USE_CLOUD_RUN_JOBS_FOR_VIDEO`) — the render worker was never migrated. ### Changes - **Route `trigger_render_video_worker` through a Cloud Run Job** — reuses `video-encoding-job` with args override `python -m backend.workers.render_video_worker --job-id <id>`, behind `USE_CLOUD_RUN_JOBS_FOR_RENDER` (enabled in deploy env). Job executions run to completion; deploys never touch them. - **New CLI entrypoint** in `render_video_worker.py`. Exit-code contract differs from video_worker deliberately: clean returns (True AND False) exit 0 because `process_render_video` parks/fails/supersedes internally; only an escaped crash exits 1 (job template has `max_retries=2`). - **Worker-side idempotency gate** (review finding): the CRJ path bypasses the HTTP endpoint's `_check_worker_idempotency`, and CRJ retries could re-enter a job in `RENDERING_VIDEO` (hard-failing it via the same-state transition guard) or resurrect a terminal job. The worker now skips terminal + already-rendering jobs cleanly regardless of dispatch path. - **Surface failed video-worker dispatch** (review finding): the ignored `trigger_video_worker` return after render completion now logs at ERROR instead of silently stalling the job at `instrumental_selected`. - **Pulumi**: `video-encoding-job` env gains both `USE_CLOUD_RUN_JOBS_FOR_*` flags so the render execution's in-process video-worker trigger also dispatches a CRJ. **Already applied to prod** (`pulumi up` run locally, verified on the job). - **`main.py`**: corrected the false 600s comment; shrank the unreachable 480s lifespan wait to 5s. Parking hook kept as legacy/flag-off net. ### Rollback Remove `USE_CLOUD_RUN_JOBS_FOR_RENDER=true` from the service env → falls back to the legacy Cloud Tasks path. ## Testing - Backend: 4422 unit + 151 emulator tests pass locally. - New tests: CRJ routing (flag on/off), CLI exit-code contract, worker-side idempotency gate (terminal skip, already-rendering skip). - Post-deploy: will dispatch the Tenant Portals E2E workflow (the failing scenario) against prod. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Video rendering now runs as durable background jobs, allowing renders to continue through service deployments. - Added safeguards to prevent duplicate processing and safely handle completed or in-progress jobs. - Improved failure reporting when a video-processing job cannot be started. - **Bug Fixes** - Fixed render jobs becoming stuck when deployments interrupted background processing. - **Documentation** - Added troubleshooting guidance and incident lessons for interrupted or stuck renders. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
d93dbddd48 |
ci: reliable R1 canary version-gate + retry (stop false pages) (#988)
Follow-up to the incident-hardening initiative — fixes the false "Post-deploy canary FAILED (v0.222.3)" page from the #987 deploy. **Root cause:** the R1 version-change gate did a single un-retried curl to the raw Cloud Run **origin** `/api/health/detailed`, which came back empty (cold start on the scaled-to-zero prior revision and/or origin edge-auth on the detailed endpoint). It logged `Currently-serving version: <unknown>` and **fail-opened → ran the canary on a no-bump deploy**, which then flake-failed on a review-page timing wait and paged. **Fixes:** 1. **Reliable gate** — read the currently-serving version via the **edge** (`https://api.nomadkaraoke.com/api/health/detailed`, warm + Cloudflare-fronted, verified to return the version) with retries. A no-bump deploy now correctly **skips** the canary. 2. **Retry the canary** — `--retries=2` so a single flaky UI wait doesn't page; only a failure surviving retries is real. (The happy-path's review-page waits are inherently flaky; the identical spec passed on the 0.223.2 deploy.) No app code changed → no version bump. YAML validated. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
|
5b5264dc54 |
Incident hardening: deploy/CI/runner resilience (R1/R2/R3/R4/C1/C2) (#987)
Part 3 of 3 of the **Incident hardening (post-NOMAD-1632)** initiative (plan: `docs/archive/2026-09-12-incident-hardening-plan.md`). Closes the deploy/CI/runner gaps: no functional gate, no codified rollback, bad revision serves 100% immediately, fire-and-forget runner stranding, and a flaky smoke. **No backend app code changed → no version bump.** ## ⚠️ Reviewer note — read first **R3/R4 rewrites the live `deploy-backend` step.** It's designed to be *auto-revert-safe* (a failed candidate smoke means we simply never migrate traffic, so the prior revision keeps serving 100% — a bad revision serves **0%**, not an outage). But it changes how **every** backend deploy works and **can't be fully exercised in CI**. This is the one to watch on first merge. If you'd rather de-risk, R3/R4 is the only item here that's safe to split out — the rest (R1/R2/C1/C2) are independent and lower-risk. ## R2 — codified fast rollback (`scripts/rollback.sh` + runbook) Dry-run-able, explicit-target-required rollback for the Cloud Run **service** (traffic-shift to a prior revision — instant, no rebuild) and the three **jobs** (image re-pin, correct per-job region/repo — verified against ci.yml). `--list` shows available revisions/tags; `--dry-run` prints every gcloud command. `docs/TROUBLESHOOTING.md` "Fast rollback" runbook added as the first section. ## R3/R4 — service traffic-canary `deploy-backend` now deploys `--no-traffic --tag candidate`, smokes the tagged candidate URL (health + expected version, with retries), then `update-traffic --to-latest` **only on pass**. Cloud Run Jobs can't traffic-split (noted inline) → they rely on R1 + R2. ## R1 — post-deploy functional canary, version-change-gated New `deploy-verify` job runs the real production happy-path (`happy-path-real-user.spec.ts`, admin-credit variant — reuses e2e-daily's impersonate token refresh, secret names verified identical) after a backend deploy, **only when the backend version actually changed** (new "Detect backend version change" step compares `poetry version` to the currently-served `/api/health/detailed`). Frontend/docs/infra-only merges short-circuit. On failure → Discord alert + loud fail printing the exact `scripts/rollback.sh` invocation. Per the plan's caveat, a fresh happy-path won't reproduce every incident class (e.g. Failure B's stale-cache) — an admin reset+retry variant is a noted follow-up. ## C1 — ephemeral-runner stranding fix `infrastructure/functions/runner_manager/ephemeral.py` now confirms the VM insert **operation** succeeded instead of fire-and-forget: a late stockout/quota/transient error raises → existing zone-fallback / webhook redelivery kicks in instead of silently stranding a queued job for ~15 min. An insert-confirm *timeout* is treated as "still provisioning" (not a failure); guarded for older clients. **+4 tests (46 pass).** ## C2 — de-flake the E2E smoke `networkidle` → `domcontentloaded` + explicit element waits (the marketing page's background requests made `networkidle` time out and redden real deploys). ## Validation (local only — no deploy) - `ci.yml` + `e2e-daily.yml` parse; **actionlint clean** on new code. - `rollback.sh`: `bash -n` + `shellcheck` clean; constants cross-checked against ci.yml. - `test_ephemeral.py`: **46 passed**. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
|
05f024bdd4 |
Requests voting board Phase 2: daily free-community-track automation (#977)
## Summary
Phase 2 of the requests voting board (`requests.nomadkaraoke.com`) — the
daily automation that turns the top-voted request into a free,
auto-generated, published karaoke video. **Deploys dark** behind a
kill-switch (`COMMUNITY_DAILY_PICK_ENABLED`, default off); the daily
Scheduler only shadow-logs until it's flipped on.
Picks up
`docs/archive/2026-09-03-requests-voting-board-phase2-handoff.md`.
Design + Andrew's decisions:
`docs/archive/2026-09-03-requests-board-phase2-plan.md`.
## What's included
- **Daily picker** (`community_daily_pick.py`, Scheduler noon US
Eastern, DST-aware): claims the UTC day with a create-only lock
(`daily_community_pick/{YYYY-MM-DD}` → **one free track/day, total**),
picks the top open request with net votes ≥ 0 (source-agnostic; skips
community-rejected), grants the requester a free credit, and submits the
job **as that user** (owned by them, credit consumed) via the same
search → auto-select → download path the web flow uses. Fully idempotent
(resumable per-day lock phase + per-request guard flags).
Shadow/`?dry_run=true` mode.
- **24h ownership handoff** (`community_handoff.py`, hourly): reassigns
a stale review to the next up-voter (compare-and-set guarded), up to **5
voters**, then parks the track (`stalled`). Community jobs excluded from
the normal stale-review auto-cancel.
- **Voter publish fan-out**: YouTube upload-queue hook marks the request
`published` and emails every up-voter (except the owner) a new "track
you voted for is live" email (33 locales). Retry-safe (per-voter success
tracking).
- **Two internal endpoints** + two Cloud Scheduler jobs (Pulumi).
## Andrew's decisions (via AskUserQuestion)
- Trigger: **noon US Eastern**. Trending-agent fallback source:
**deferred** to a follow-up. Net-negative requests: **skipped**. Handoff
cap: **5 voters then park**.
## Deferred / follow-ups (not in this PR)
- **Trending-agent fallback source** (auto-submit a candidate when the
board is empty) — deliberately deferred; must generate candidates the
careful way (karaoke-decide `candidates` + KaraokeNerds).
- **YouTube description board link** — left untouched to avoid
clobbering Andrew's parallel descriptions work; add
`requests.nomadkaraoke.com` to `default_youtube_description` once that
lands.
## Testing
- Worker unit tests (kill-switch shadow, empty board, day-lock
idempotency, credit-grant/job-create guards, handoff cap/stale/CAS,
voter fan-out retry-safety) + emulator tests for the new service methods
against real Firestore transactions.
- All touched existing suites (youtube queue, stale-review,
requests-board routes/logic) pass. 79 passed locally.
## Review
CodeRabbit reviewed locally (2 cycles completed; 2 major + 1 minor
findings all addressed — retry-safe fan-out, CAS owner assignment,
up-voter ordering). 3rd cycle hit the free-OSS rate limit; it would only
re-verify the applied fixes.
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
|
3c58143f75 |
fix(review): re-sign instrumental preview URLs on expiry (refresh-on-error) (#973)
## Problem
Completing a combined review failed to play the instrumental in the
preview modal — toggling to **Instrumental** loaded nothing, with
`NS_ERROR_DOM_NETWORK_ERR` in DevTools (Firefox) on the
`review-audio/instrumental_with_backing.ogg` GCS request.
**Root cause:** the overlaid instrumental `<audio>` uses the signed GCS
URL baked into the review payload at page load, which has a **120-minute
expiry**. A long review session outlives it, so the stem fetch returns
`400 ExpiredToken` and playback silently fails. Confirmed by fetching
the exact URL from the report: `ExpiredToken — Request signature expired
at 2026-09-01T20:16:26Z` (page had been open ~2h11m).
Reloading didn't help because the review UI persists the full
`CorrectionData` — including `instrumental_options[].audio_url` — to
`localStorage` (`lyrics_analyzer_data`) and rehydrates it when the user
has unsaved edits, so the *same* expired URL came back.
The `[SW] Service worker registered` log is a red herring:
`frontend/public/sw.js` has no `fetch` handler.
## Fix (refresh-on-error)
- **Backend:** extract a shared `_build_instrumental_options` helper and
add `GET /api/review/{job_id}/instrumental-urls` returning
freshly-signed stem URLs.
- **Frontend:** on the stem `<audio>` `error` event, re-fetch fresh URLs
via the new endpoint and swap them in. The existing video↔audio sync
effect (keyed on the URL) resumes playback in place. Capped at 3
refreshes per modal open so a genuinely-broken stem (404) can't loop.
No new user-facing strings (the recovery is silent) → no i18n changes.
## Testing
- Backend: 3 new tests for the endpoint (both stems / single stem / 404)
— `TestInstrumentalUrlsRefresh`. Full `test_routes_review.py` +
`test_combined_review_flow.py` + `test_audio_transcoding_service.py` =
111 passed.
- Frontend: 3 new `PreviewVideoSection` tests (swap on error / cap at 3
/ no-op without refresh fn). Full unit suite = 1248 passed.
- CodeRabbit local review: no findings.
@coderabbitai ignore
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
|
37b9e6088b |
ci(pypi): recurring storage-cap prune reminder (fix failing PyPI publish) (#960)
## Problem The `Deploy - Publish to PyPI` step on `main` has been failing: ``` HTTP Error 400: Project size too large. Limit for project 'karaoke-gen' total size is 10 GB. ``` This is a recurrence of the known 10 GB PyPI project cap. The project is at **10.71 GB / 479 releases**; each release is a ~62 MiB universal wheel (bundled 33-locale frontend) auto-published on every merge (~3 GB/month), so it refills the cap every few months. **Only the public `pip install` publish step fails** — Cloud Run and the GCE encoding worker (pulls the wheel from GCS) are unaffected. The prior fix (v0.187.5) was wheel-only publishing + a one-time manual deletion, which only bought ~2 months. This makes the cleanup **recurring and durable**. ## Why not just automate deletion in CI? PyPI has **no deletion API** (upload tokens / Trusted Publishing are upload-only, and 2FA is mandatory). Deletion is only possible from a logged-in browser session, which can't be safely/durably stored in CI. So we automate the *decision + reminder* and keep the *deletion* a ~1-minute human step. ## What this adds - **`scripts/prune_pypi_releases.py`** (stdlib-only) — reads the public JSON API and computes a retention plan: **keep last 60 days + newest release per older calendar month + latest overall**. Renders a table / JSON / Markdown report and a **paste-into-browser-console** deletion snippet. Never deletes anything itself. Today's plan: keep 82, delete 397, **10.71 GB → 4.17 GB**. - **`.github/workflows/pypi-prune-reminder.yml`** — monthly cron (+ manual dispatch) that runs the planner and, once the project crosses a 7 GB watermark, opens/updates a single tracking issue with the plan + snippet. No secrets. - **`docs/PYPI-STORAGE-PRUNE.md`** runbook + a `TROUBLESHOOTING.md` entry. - **`tests/unit/test_prune_pypi_releases.py`** — 13 tests covering the policy (recency window, per-month keeper, latest-overall, same-day timestamp ranking, empty/undated releases, sizes, rendering). ## Testing - `pytest tests/unit/test_prune_pypi_releases.py` → 13 passed. - Ran the planner against live PyPI data; generated JS validated with `node --check`. - `black` clean; workflow YAML validated; `should_nag` gate verified. ## Notes - **No version bump** — tooling/CI/docs only, nothing in the wheel changes. Merging therefore **skips** the publish step (tag `v0.209.1` already exists), so this PR does not itself add another fat wheel while we're over the cap. - **To unblock publishing now:** after merge, run the prune (locally via the script or trigger the workflow with `workflow_dispatch`), paste the snippet on the logged-in PyPI releases page, then re-run the publish. The monthly cron keeps it under the cap going forward. - Related: filed [pypi/support#12050](https://github.com/pypi/support/issues/12050) requesting a 50 GB limit as a one-off buffer; the recurring prune is the durable fix. @coderabbitai ignore --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
|
b45efbdbc5 |
feat: enable confident backing-keep enforcement in prod (#953)
## Summary One-line deploy-env change: `AUTO_APPROVAL_BACKING_KEEP_ENABLED=true` on the backend service. A confident WITH_BACKING verdict from the 3-stem backing decider (#950) now auto-selects the with-backing instrumental, letting jobs with confident lyrics AND clear backing vocals complete fully automatically. ## Shadow validation evidence (why now) - **20-job replay corpus**: 13/14 true keeps confidently retained (incl. both jobs the human under-kept); catastrophic backing-is-lead case gated; zero unsafe outcomes. - **Live prod battery**: 7/7 keep verdicts on harmony tracks with consistent signal profiles; the misclassified-lead gate fired correctly on an adversarial shoegaze track; one job completed fully zero-touch end to end. - **Human review pass on 11 fresh jobs (2026-08-29)**: 10/11 instrumental picks matched the shadow verdict. The single lyrics-auto-eligible job was shipped by the human with **zero manual edits and the identical instrumental selection** full-auto would have made. Every job needing real lyric work was correctly gated (up to 127 manual edits on gated jobs; zero false-autos). Kill switch: set the var to false (or remove it — code default is false). Per-job opt-out: `review_mode="always_review"`. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |