mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
main
2052
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
3a6e175fbf |
feat(tenant): fully automatic tenant lifecycle — subdomain provisioning, delete, per-email access (#1072)
## Summary
The admin tenant console now creates and deletes **fully working**
tenants.
- **Create:**
- Provisions `{id}.nomadkaraoke.com` through the Cloudflare API: a Pages
custom domain on `karaoke-gen-tenant` plus a proxied CNAME.
- If a non-tenant DNS record already owns the hostname, create returns
`409` before anything is written.
- If Cloudflare fails, the GCS tenant is rolled back and create returns
`502`.
- The subdomain must be `{id}.nomadkaraoke.com`, because the edge
function and middleware derive the tenant from the first hostname label.
- **Delete:** new `DELETE /api/admin/tenants/{id}` removes the subdomain
(only records that point at the tenant Pages project), the theme (unless
it's the default or shared), the registry entry and the config. Jobs are
kept.
- **Retry/backfill:** new `POST /api/admin/tenants/{id}/domain` is
idempotent.
- **Per-email access:** new `auth.allowed_emails`, alongside domains. If
either list has entries, only listed emails/domains can sign in; both
empty = open. `@nomadkaraoke.com` admins can always sign in; the
docstring already claimed this but the code didn't do it.
- **Admin UI:**
- domain status badge with set-up/retry
- access editor with an "anyone can sign in" warning
- delete that requires typing the tenant id
- after create, polls until the certificate is live
- **Infra:** secret `cloudflare-tenant-domains-token`. The Pulumi
container is already applied; the value is an account-owned Cloudflare
token scoped to Pages R/W on the account plus DNS R/W on the
nomadkaraoke.com zone only.
## Testing
- New `test_tenant_domain_service.py` runs against a stateful fake
Cloudflare (httpx MockTransport). It covers provisioning, idempotency,
foreign-record conflicts, deprovisioning only our records, API errors
and a missing token.
- Service tests cover:
- create provisions the domain
- a DNS conflict or unusable Cloudflare means no writes
- a provisioning failure means a full rollback
- delete removes everything, aborts if Cloudflare fails, and keeps a
shared theme
- access-list normalisation and enforcement
- Route tests cover delete (204/404/502), the domain endpoint (200/409),
502 on create, allowed_emails passthrough, and admin-only access.
- Model tests cover individual emails, combined lists, the legacy flag,
and the admin bypass.
- Frontend (Jest) covers access editing, the open-portal warning, domain
set-up, and type-to-confirm delete.
- Results: `pytest backend/tests --ignore=backend/tests/emulator` gives
4927 passed; `jest` gives 1421 passed.
No local CodeRabbit review was run, so this PR is left open for the
GitHub bot. A separate agent review was done before merge.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Tenant creation provisions a portal subdomain, with its setup status
shown in tenant management.
* Admins can retry domain setup and delete tenants after confirming the
tenant ID.
* Tenant access can be restricted to specific email addresses, allowed
email domains, or both. Addresses are normalized, and admin access is
retained.
* **Bug Fixes**
* Email restrictions are enforced for magic-link sign-in even when no
domain allowlist is configured.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
v0.246.0
|
||
|
|
13ecce4f79 |
feat: monitor karaoke-decide lb-refresh job errors (v0.245.4) (#1073)
## Summary Adds `lb-refresh` (karaoke-decide's new ListenBrainz statistics → BigQuery Cloud Run Job) to `MONITORED_CLOUD_RUN_JOBS`, so a failed run or its stale-data ERROR reaches the error monitor and Discord. This matches what #1067 did for `mb-refresh`. ## Changes - `backend/services/error_monitor/config.py`: add `lb-refresh` - `tests/unit/services/error_monitor/test_config.py`: expect 6 jobs and include `lb-refresh` - Version 0.245.4 ## Testing - `pytest tests/unit/services/error_monitor/`: 361 passed 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
849167603a |
fix(ci-runners): re-dispatch self-hosted jobs stranded in queued (v0.245.4) (#1071)
## Summary Three merge-to-main runs left `Deploy - Backend` stuck `queued`, and twice prod was left half-deployed. The dispatcher launches **one** VM per `workflow_job.queued` webhook and never retries. GitHub never redelivers a failed webhook. Root causes found in the dispatcher logs: - **2026-09-27:** the deploy job's webhook got a **Cloud Run 429 "no available instance"**. `max_instance_count=5`, and each webhook holds an instance for up to 90s while it confirms the VM insert, so a CI burst drops webhooks. 429s went 1/day (9/15) → 7 (9/23) → 15 (9/27). - **2026-08-08 / 09-25:** the dispatched GPU-family VM was preempted before it registered, and nothing replaced it. ## Changes - **Stalled-job re-dispatch** in the scheduled pass, which now runs every 5 min instead of 15: - Scans org repos for self-hosted jobs that have been `queued` for 5–60 min. - Per family, compares queued jobs against spare runners (booting VMs, including JIT-registered-but-offline ones, plus idle online runners). - Launches the shortfall, at most 5 per pass, in parallel. - Gives up after 60 min, so fleet-wide runner breakage can't turn into a VM-launch loop. - `max_instance_count` 5 → 20. - Scheduler `attempt_deadline` = function timeout (300s), so a slow pass isn't retried while it is still running. - Env knobs, including a kill switch: `REDISPATCH_ENABLED`, `REDISPATCH_AFTER_MINUTES`, `REDISPATCH_GIVE_UP_MINUTES`, `MAX_REDISPATCH_PER_TICK`. - Docs: `docs/EPHEMERAL-GHA-RUNNERS.md` § Stalled-job re-dispatch. ## Testing - `infrastructure/functions/runner_manager/test_ephemeral.py`: 65 pass (16 new). The new tests cover the dropped-webhook case, the preempted-VM case, spare capacity per family, booting VMs registered as offline, the give-up limit, the per-pass cap, the kill switch and error isolation. CI doesn't run these; they were run locally. - Ran the real repo scan against GitHub read-only: 28 repos in about 20s. - `/code-review` (the CodeRabbit CLI has no seat): all 3 findings fixed in the second commit. The CodeRabbit bot reviews this PR as well. - `pulumi preview`: the targeted diff is the runner-manager source object replace, a function update (code + maxInstanceCount) and a scheduler update (schedule + attempt deadline). ## Deploy (infra deploys are local, not CI) Run a **targeted** `pulumi up` before merging. The untargeted preview also includes unrelated pending drift from other work (the `karaoke-storage` CORS update and the `cloudflare-tenant-domains-token` secret delete): ``` cd infrastructure && pulumi up --stack prod \ --target 'urn:pulumi:prod::karaoke-gen-infrastructure::gcp:storage/bucketObject:BucketObject::runner-manager-source' \ --target 'urn:pulumi:prod::karaoke-gen-infrastructure::gcp:cloudfunctionsv2/function:Function::runner-manager-function' \ --target 'urn:pulumi:prod::karaoke-gen-infrastructure::gcp:cloudscheduler/job:Job::runner-manager-idle-check' ``` 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>v0.245.4 |
||
|
|
cada71a0f2 |
fix(infra): make karaoke-gen storage bucket private; CORS any origin (#1070)
## Summary - `public_access_prevention = "enforced"` on `karaoke-gen-storage-nomadkaraoke`. The out-of-band `allUsers` objectViewer binding was also removed with `gcloud storage buckets remove-iam-policy-binding`. - CORS origins changed to `*`, so tenant subdomains created at runtime work without a per-tenant list. This is safe now that the bucket is private: a signed URL or origin-bound resumable session is the auth. - Docs: ARCHITECTURE tenant runbook, plus a LESSONS-LEARNED entry. ## Audit (why it was public) - Added by hand on **2025-12-22 06:37 UTC** by admin@ via `gsutil iam ch`. That's the same minute as commit `92affebd` (Job Inspector HTML page, `public/job-inspector.html`, the only object under `public/`). It was never in Pulumi and isn't mentioned in any doc. - No consumer relied on it. The backend and all Cloud Run jobs, the encoding VMs, flacfetch, audio-separator, backup-to-aws, decide and CI all use service accounts with their own grants. Browsers only get V4 signed URLs. kjbox goes through the gen API. No code builds unsigned bucket URLs, and the last 14 days of logs show no unsigned bucket URLs. - Known casualties: `public/job-inspector.html` is no longer public, and the `remote_cli.download_file_via_gsutil` anonymous fallback now returns False; the primary path is the API download. ## Verification (prod, after local `pulumi up`) - Anonymous list: **401**; anonymous object GET: **403** - Signed preview URL from `/api/themes`: **200**; with `Origin: https://randy-vild.nomadkaraoke.com` it returns `access-control-allow-origin: *` No local CodeRabbit review was run, so this PR is left open for the GitHub bot. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Security** * Storage buckets now prevent public access, helping keep customer files private. Signed URLs continue to provide authorized access. * **Bug Fixes** * Mutable JSON uploads avoid stale caching, and reads use the latest stored version. * **Documentation** * Updated setup guidance to reflect current bucket access, CORS, and caching behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
3607ccf745 |
fix(tenant): theme/tenant edits take effect immediately; no orphan jobs on bad theme (#1069)
## Problem Creating the `randy-vild` tenant via `/admin/tenants` and bulk-submitting 2 minutes later failed every row with **"Theme not found: randy-vild"**, and left 10 orphaned "Waiting for upload" jobs. Root cause: `karaoke-gen-storage-nomadkaraoke` grants `allUsers` objectViewer, so GCS serves objects with no explicit Cache-Control as `public, max-age=3600`. The backend's own reads of `themes/_metadata.json` returned the **pre-tenant** registry (logged `Loaded theme metadata: 4 themes` at 03:32:38, although the 5-theme generation was written at 03:30:40). Theme edits (`style_params.json`, assets) had the same problem: new jobs could pick up the old style for up to an hour. ## Fix - `StorageService.upload_json` writes `Cache-Control: no-store` - `StorageService.download_json` resolves the live generation (`get_blob`) and downloads that exact generation, which skips any cached copy (including objects written before this change) - tenant theme asset and logo uploads pass `cache_control=no-store` - `ThemeService.theme_exists` force-refreshes the per-instance 5-min registry cache on a miss - all 4 job-creation endpoints validate the theme **before** `create_job` (no more orphan jobs) - GCS CORS: add `https://randy-vild.nomadkaraoke.com` (**needs local `pulumi up` before merge**) - docs: tenant subdomain runbook + caching note in ARCHITECTURE.md ## Testing - New/updated unit tests: fresh `download_json` + NotFound, `no-store` on JSON/asset writes, theme_exists refresh-on-miss, unknown theme rejected before `create_job` - `pytest backend/tests --ignore=backend/tests/emulator`: 4888 passed, 36 skipped (emulator tests need a local Firestore emulator, unrelated) No local CodeRabbit review was run, so this PR is left open for the GitHub bot. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>v0.245.3 |
||
|
|
8a5f754efd |
ci: remove post-deploy canary + never email test accounts on stale review (v0.245.2) (#1068)
## Summary
- Removes the `deploy-verify` job ("Deploy - Post-Deploy Canary (Happy
Path)") from `ci.yml`, plus the `Detect backend version change` step /
`version_changed` output that only gated it.
- `stale_review_processor`: jobs owned by automated-test accounts
(`is_test_email` — `e2e-test-runner@nomadkaraoke.com`,
`@inbox.testmail.app`) no longer get the 24h reminder, review-expired,
or duration-confirm-expired emails. They are still auto-expired at 48h
so stranded test jobs don't linger.
- Updates comments + `docs/TROUBLESHOOTING.md` / `docs/TESTING.md` that
referenced the canary. Version → 0.245.2.
## Why
The canary created a real production job (piri – dog, as
`e2e-test-runner@nomadkaraoke.com`) on every backend version bump.
`ci.yml` uses `cancel-in-progress: true`, so back-to-back merges
cancelled it mid-run — the process is killed before Playwright's
`afterEach` cleanup runs, stranding the job in `awaiting_review`. The
stale-review processor then emailed the test account (which forwards to
Andrew's inbox) at 24h/48h (e.g. job `adb8f393` from cancelled run
36288893657; earlier `a5f84cc3`, `10b532e2`). The canary was overkill;
the daily E2E and the traffic-gated candidate-revision smoke remain, and
the email guard backstops the daily E2E if it ever strands a job.
## Test plan
- [x] New unit tests: test-account reminder is silent (+ still flagged),
review expiry + duration-confirm expiry cancel without email
- [x] `pytest backend/tests --ignore=backend/tests/emulator` — 4883
passed
- [x] `actionlint` — no new findings
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* CI no longer runs a post-deployment production canary or sends its
test results and failure notifications. Candidate-revision smoke testing
remains in place.
* Production real-job verification is handled manually or by the daily
E2E run.
* **Documentation**
* Testing and rollback guidance now reflects the updated verification
and recovery steps. Daily E2E failures are identified as a reason to
roll back.
* **Bug Fixes**
* Automated test accounts no longer receive stale-review reminder or
expiry emails; existing expiry and cancellation handling remains in
place.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
v0.245.2
|
||
|
|
5b2a0f8c3e |
feat(error-monitor): watch decide's mb-refresh job (#1067)
## Summary - Adds `mb-refresh` (karaoke-decide's new weekly MusicBrainz → BigQuery Cloud Run Job, nomadkaraoke/karaoke-decide PR to follow) to `MONITORED_CLOUD_RUN_JOBS`, so failed refreshes and staleness warnings raise error-monitor patterns / Discord alerts. - Version bump 0.245.0 → 0.245.1. ## Testing - `pytest tests/unit/services/error_monitor` — 361 passed (count + membership assertions updated). 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Reliability** * The weekly MusicBrainz data refresh is now included in service monitoring. Issues with the refresh can be detected alongside problems with other monitored services, improving visibility into the availability of refreshed music data. * **Release** * This update is version 0.245.1. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>v0.245.1 |
||
|
|
87a36bd2a5 |
feat(encoding): GCP cost cuts — Spot 16-vCPU workers, 3-VM fleet, 5-min idle, gated worker deploys (v0.245.0) (#1061)
## Summary Approved GCP cost cuts for the encoding pipeline (target: whole project < $300/mo; encoding VMs were ~$300/mo for ~5-20 jobs/day). Details + before/after table: `docs/archive/2026-09-26-encoding-cost-cuts.md`. | | Before | After | |---|---|---| | Primary pair a/b | c4d-highcpu-32 on-demand | **c4d-highcpu-16 Spot** (STOP on preemption) | | Fallbacks | 8 VMs / 6 families | **1 on-demand** `encoding-worker-fallback-c2df` (c2d-highcpu-16, us-central1-f) | | Idle shutdown | 15 min / check every 5 | **5 min / check every 2** | | CI worker blue-green | every push to main | only when `infrastructure/encoding-worker/worker_code_paths.txt` paths changed since the serving version (+ `workflow_dispatch` `force_encoding_worker_deploy`) | | Nomad Dropbox upload | whole folder | minus lossless 4K **MP4** (tenant folders unchanged) | **Infra already applied locally (targeted Pulumi):** 7 fallback VMs + disks + 7 static IPs deleted; a/b + c2df resized in place; idle function env + scheduler updated. The Spot switch was done in place with `gcloud compute instances set-scheduling` on the stopped VMs + `pulumi refresh` (pulumi-gcp treats a `scheduling` diff as *replace*, and recreating a c4d VM risks a stockout). `encoding-worker-fallback-vms` secret → v6 (c2df only). CI's pulumi run should be a no-op for these resources. ### Scope change (per Andrew) **Task 5 dropped:** review previews stay on the encoding VM (`USE_GCE_PREVIEW_ENCODING=true` unchanged) and the review-page warmup is kept, since it deliberately warms the VM for the preview and the final render. To make sure the new 5-min idle timeout doesn't undermine that, the review page now also heartbeats **every 2 min** while the tab is visible and was used in the last 15 min, and re-warms on tab refocus (`useEncodingWorkerKeepAlive`). Before this, the heartbeat only fired on lyric edits. The idle checker keeps the routed VM alive while `last_activity_at` < 5 min, and both warmup and heartbeat write it. Tests cover both sides. ### Code changes - **Spot preemption mid-encode**: when a poll fails, `wait_for_completion` checks the pinned worker VM. If it is STOPPING/TERMINATED it raises `EncodingJobLostError`, so `run_with_lost_job_resubmit` resubmits the encode, and that resubmit restarts the Spot VM or falls back to c2df. Before, the failure showed up after ~7 min as a "lost contact" error that was not resubmitted. - **Stale fallback lists**: a candidate VM that no longer exists (`NotFound`) is skipped instead of aborting the whole failover. This protects the window before running instances pick up the new secret. - `SPEED_RANK` is now keyed by machine family, so the ranking is independent of VM size. - **Idle-stop vs. in-flight encode** (review follow-up): the idle function's `/health active_jobs` only counts RUNNING jobs, and `last_activity_at` was only written when a VM was started. With a 5-min window, a long encode's VM could be stopped in the seconds between the job finishing and the next poll. The poll loop now refreshes `last_activity_at` at most once a minute. A VM seen in STAGING/PROVISIONING after it accepted a job also counts as lost. CI `Read Firestore config` now writes empty values instead of "None". - **Worker deploy gate**: `backend/tests/test_encoding_worker_code_paths.py` derives the worker's transitive `backend.*` import set and fails if `worker_code_paths.txt` doesn't cover it. Every deploy still uploads the wheel and `startup.sh`, so a cold-booted VM always runs the latest code. - **Dropbox**: nothing automated reads lossless files from Dropbox. The lossless **MKV** is still uploaded because the completion email and the live Fiverr auto-reply bot both tell customers it is in the Dropbox folder, and the VocalStar tenant expects lossless files. To drop the MKV as well, update those two texts first, then add the MKV suffix to `DROPBOX_SKIP_OUTPUT_SUFFIXES`. ## Testing - New/updated unit tests: preemption → lost job → resubmit, NotFound candidate skip, family ranking, idle 5-min/heartbeat window, fleet-shape invariants, worker-path coverage, Dropbox exclusion, keep-alive hook (Jest, fake timers). - `pytest backend/tests --ignore=backend/tests/emulator`: 4802 passed. The only failures are 6 tests that need the spaCy `en_core_web_sm` model, which isn't installed in my local venv; unrelated to this PR. - Jest lyrics-review + lib/lyrics-review: 423 passed. Infra tests (`test_encoding_worker_config.py`, idle function): all pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>v0.245.0 |
||
|
|
c138c9d1ac |
feat(review): vocal waveforms for own-instrumental jobs (v0.244.1) (#1062)
## Summary
Jobs with a user-supplied instrumental (`existing_instrumental`) skip
GPU separation. That left the lyrics review **Waveforms** mode with no
vocals stem, so `/vocals-peaks` returned 404 and no strips were drawn.
The waveform only needs a peak envelope, so a CPU approximation is
enough.
- **`backend/services/derived_vocals.py`**: decodes the mix and the
instrumental to mono at 22.05 kHz (ffmpeg). It aligns them by FFT
cross-correlation (numpy, ±2s search), fits a least-squares gain, and
subtracts, leaving roughly the vocals. It writes a FLAC and logs offset,
gain and residual ratio. A few seconds of CPU.
- **Lyrics worker**: for existing-instrumental jobs, runs this in a
thread alongside transcription. It uploads
`jobs/{id}/stems/vocals_derived.flac` and registers
`stems.vocals_derived`, and it's awaited before advancing to screens so
the screens step pre-computes peaks as usual. Best-effort: failures log
and the job continues.
- **`vocals_stem_path`**: adds `vocals_derived` as the last-resort key.
It's waveform-only: deliverable stems, the original-vocals guide and the
auto-approval timing gate don't read it.
## Testing
`backend/tests/test_derived_vocals.py` covers:
- offset recovery for positive, negative and large lags
- gain and vocals recovery on synthetic signals
- an unrelated instrumental leaving the mix intact
- a real ffmpeg round trip (24-bit stereo WAVs → mono 22.05 kHz FLAC)
- stem precedence (never overrides real separated vocals)
- worker upload and registration, plus the non-fatal failure path
The full backend unit suite passes in an isolated venv (4812 passed).
The only exceptions are 6 tests needing the spaCy `en_core_web_sm`
model, which isn't installed in that venv; CI has it.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
||
|
|
8cbcc03099 |
infra: GCP cost cuts (VPC connector, min-instances, Autoclass, weekly incremental git backup, hourly error monitor, PITR off) (#1063)
## Summary These are the cost cuts Andrew approved to get the whole GCP project under $300/mo. Performance compromises were accepted. - **VPC connector removed.** `cloud-run-connector` only existed to reach the old flacfetch GCE VM's private IP. flacfetch now runs on Netcup behind a public HTTPS URL. I checked every URL secret the backend uses and none are private IPs. The backend is detached with `--clear-vpc-connector`: just dropping the `--vpc-connector` flag from `gcloud run deploy` would keep the old setting. `audio-download-job` and `bulk-search-job` no longer set `vpc_access`. Saves ~$12/mo plus the Network Intelligence resource-hours. - **`karaoke-backend` min-instances 2 → 1** (ci.yml). - **Autoclass** on `karaoke-gen-storage-nomadkaraoke` (terminal NEARLINE, because customers re-download old jobs) and on `nomadkaraoke-divebar-files` (terminal ARCHIVE). Autoclass charges no retrieval or early-deletion fees. - **backup-to-aws** - **Git bundles:** now weekly (Sundays) and incremental. A repo is re-bundled only when its GitHub `pushed_at` is newer than its bundle's S3 `LastModified` minus a 1h lag, so S3 itself is the state store. Every repo is re-bundled on the first Sunday of each month. `?mode=git_repos[&full=1]` runs the git step alone. - **Bug fix:** since #997 every S3 **overwrite** has failed. The `backup-writer` IAM user has no `s3:GetObject`, so HEAD on an existing key returns 403. Git bundles and the manifest were stuck at 09-15, and the DR freshness monitor has been red every day. The existence check now uses `ListObjectsV2`. - **`gcs_sync`:** detects changes by `time_created` instead of `updated`. Autoclass transitions bump `updated`, which would have made the sync re-copy every object and pull it back to Standard. - **DR freshness monitor:** `git-repos/` is now judged on `manifest.json` (192h) plus a check of the manifest's contents. - **Error monitor:** runs hourly instead of every 15 min. Lookback is 60 min, equal to the interval, so there are no gaps or double counts. `MAX_LOG_ENTRIES` goes from 500 to 2000. - **Firestore PITR disabled.** The nightly export to GCS staging remains, plus the weekly copy to S3. - **`recover-stuck-downloads` / `retry-pending-render-jobs`:** every 5 → 10 min. Both are recovery sweeps with ~10-min stall thresholds. - Docs updated: DISASTER-RECOVERY (PITR restore replaced, git RPO, incident write-up), ARCHITECTURE, LESSONS-LEARNED, GCP-COST-OPTIMIZATION, TROUBLESHOOTING. **Kept:** the `audio-separator` GPU service. Its request logs show ~40 external `/separate` calls/month from remote-CLI users. It's already min-instances 0. ## Already applied to prod (before merge) - `gcloud run services update karaoke-backend --clear-vpc-connector --min-instances 1`, same image. Health is OK, and `/api/health/flacfetch` and `/api/health/encoding-worker` are both OK without the connector. - Targeted `pulumi up` on the Firestore DB, both buckets, both jobs and both 5-min schedulers. - backup-to-aws redeployed with `deploy.sh`. A manual `?mode=git_repos` run bundled 13 of 121 repos (106 unchanged, 2 skipped) and uploaded 120 files with 0 errors. - Deleted `gs://…/backups/flacfetch-torrent-backup-20260122-1931.tar.gz` (1 GB). ## After merge Targeted `pulumi up` for `error-monitor-scheduler`, which must wait for the image with LOOKBACK=60, then delete the connector and drop the `vpcaccess` API service from state (`disable_on_destroy=False`, so the API itself stays enabled). ## Tests `make test` passes locally. The backup_to_aws function tests (43, including new incremental, list-based-existence and gcs_sync tests) were run separately because CI doesn't run them. Related: nomadkaraoke/karaoke-decide#140 (ARCHIVE lifecycle on `nomadkaraoke-data`, already applied). 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Updates** * Pending-render retries and stuck-download recovery now run every 10 minutes instead of every 5 minutes. * Error monitoring now runs hourly and checks a 60-minute window. * Cloud Run candidate deployments use one minimum instance instead of two and no longer use a VPC connector. * **Backups and Recovery** * Git repository backups now run weekly, with monthly full refreshes and incremental updates between them. * Firestore point-in-time recovery is disabled; recovery uses nightly exports. * Storage buckets now use automatic storage-class management. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>v0.244.1 |
||
|
|
7179e2863f |
ci: move test/package jobs to GitHub-hosted runners; GPU base image same-region (cost) (#1059)
## Summary GCP cost cuts in CI (approved): 1. **GitHub-hosted runners for test/package jobs** (~\$28/mo of `gha-general-*` VMs, ~390 VMs/30d). Moved to `ubuntu-latest` (free, public repo): `package-unit-tests`, `package-integration-tests`, `package-build-test`, `backend-unit-tests`, `backend-emulator-tests`, `package-build` (main), `frontend-e2e-full` (label-gated). None used GCP auth. Self-hosted disk checks replaced by a free-disk-space step (drop dotnet/android/ghc/CodeQL); venv cache keys renamed so self-hosted venvs aren't restored onto a different interpreter path. 2. **Removed `propagate-base-image`** — pre-pulled the base image onto "the 3 runners"; runners are now single-use ephemeral VMs so this just spawned 3 VMs per base rebuild for nothing. 3. **AR inter-region egress** (~\$12-16/mo): the GPU app image was built on a us-central1 runner `FROM` the ~13 GB GPU base in **us-east4**, pulling it cross-region on every deploy. The GPU base + all BuildKit caches now live in us-central1 `karaoke-repo`; only the final app image is pushed to us-east4 (`karaoke-backend-gpu`) where `audio-separation-job` pulls it same-region. Push uploads only the small changed app layers (base layer digests are identical). First deploy copies the current base from us-east4 via `buildx imagetools create` (byte-identical, no from-scratch rebuild); falls back to a rebuild only if that fails. 4. `build-runner-images.yml`: `KEEP_IMAGES_PER_FAMILY` 2 → 1 (GPU/windows images are 15-21 GiB). 5. `ci.yml` changes now trigger the package/backend test jobs (so this PR exercises the moved jobs). `deploy-backend` stays on the `docker-build` GCE runner (Docker builds, WIF, same-region AR). GPU runners untouched. ## Testing - This PR's CI runs the moved jobs on hosted runners. - `image_hash` helper + `imagetools create` cross-registry copy verified locally against two `registry:2` instances (copy wraps in a single-entry index; helper follows it). - actionlint: no new findings (fewer than before). 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * CI test, package, and frontend end-to-end jobs now run on GitHub-hosted runners. GPU image builds use a centralized base-image cache, while application images continue to be published in their existing region. * Runner image cleanup retains the newest image per family; images backing existing disks remain protected. * **Documentation** * Updated the runner guide to describe which jobs use hosted or self-hosted runners and how image retention works. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
7ebe929d71 |
feat: quick scrolling-lyrics version for kjbox make-it jobs (v0.244.0) (#1064)
## Summary
kjbox singer "make it" jobs take ~30 min. For kjbox jobs (`X-Client-Id:
kjbox*`) the audio worker now also produces a rough **scrolling-lyrics
draft video** within a few minutes of the audio landing, so the singer
can choose to sing right away (kjbox side: nomadkaraoke/kjbox PR to
follow).
- **Quick separation** — one fast single-model pass
(`mel_band_roformer_instrumental_fv7z_gabox.ckpt`, already baked in the
GPU image, MDXC overlap 2; `QUICK_VERSION_MODEL` override) on the warm
L4 **before** the ensemble.
- **Background render** (`backend/services/quick_version/`, ported from
kjbox `fastgen/`) while the ensemble runs: LRCLIB lyrics (synced →
time-anchored scroll; plain → constant crawl; none → title card), PIL
crawl PNG + one ffmpeg overlay pass, 854×480, vocals mixed back at 30%
as a guide.
- Uploads `jobs/{id}/quick/quick.mp4` → `file_urls.quick.video_mp4`;
`state_data.quick_version = {status, lyrics_tier, ready_at, timings…}`.
- The worker joins the render (≤300 s) before temp-dir cleanup, even if
the ensemble fails. Quick-version failures never fail the job. Skipped
on retries once `ready`, in remote-API mode, or with
`KJBOX_QUICK_VERSION_ENABLED=false`.
- No new endpoint: kjbox polls `GET /api/jobs/{id}` and downloads
`/api/jobs/{id}/download/quick/video_mp4`.
Plan: workspace
`docs/archive/2026-09-26-make-it-quick-fastgen-version-plan.md`.
## Testing
- `backend/tests/test_quick_version.py` (37): lyrics parsing/LRCLIB
fallbacks, layout + scroll expression, ffmpeg cmd, real PIL+ffmpeg
render (skips without ffmpeg), gating, separator params/output
detection, render thread success/failure, audio-worker wiring (join on
ensemble failure, timeout → failed, remote mode skips).
- Full backend suite: 4831 passed.
- Local real runs: Fv7z single-model separation produced both stems with
the custom names; ABBA "Waterloo" rendered in ~9 s (LRCLIB synced).
- Not yet measured in prod: GPU job cold start + L4 quick-separation
time — first thing to check after deploy.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
v0.244.0
|
||
|
|
a2e8e98eaf |
fix: user-instrumental renders for any audio format + crash-report diagnostics (v0.243.1) (#1060)
## Summary
**1. User-instrumental renders failed for WAV/M4A/OGG/AAC** (job
3ea34552: "Encoding failed on retry: No instrumental audio found")
- The orchestrator render path sent only
`instrumental_selection="custom"` to the GCE encoder.
- The encoder's custom lookup accepted just
`*custom_instrumental*.{flac,mp3}`, and the staged copy kept its
original extension. Tenant MP3s worked; a `.wav` didn't.
- Fix: the orchestrator now passes `existing_instrumental`, so the
encoder downloads the file and matches any extension. The custom lookup
(extracted as `resolve_instrumental`, now tested directly rather than
via copied globs) also accepts any audio extension.
**2. Crash reports now carry enough to diagnose.** A Firefox "Uncaught
out of memory" froze a tab and the alert was useless:
- There was no Error object, so the reporter's own stack became the
pattern.
- The stack pushed URL/UA/build out of Discord's view.
- `extra` was never kept.
Changes:
- **Frontend:** `lib/diagnostics.ts` keeps a 30-entry breadcrumb trail
(nav, clicks, upload phases, errors) and snapshots page age,
DOM/canvas/audio counts, live blob URLs and the JS heap (Chromium).
Errors without an Error object are flagged `synthetic` (reporter stack
dropped, location kept). Each page load can send at most 20 reports.
Benign media `AbortError` rejections are still filtered.
- **Backend:**
- Sample order is now message → location → URL/UA/build → `Diag:` →
`Trail:` → trimmed stack.
- Stackless errors group by message.
- The full `extra` is logged as `client_extra` on
`frontend_crash_reported`.
- `TROUBLESHOOTING.md` has a runbook entry.
## Testing
- New backend tests: `resolve_instrumental` across extensions,
orchestrator config, GCE backend config forwarding, and ingestion
sample/grouping.
- New Jest tests: diagnostics, reporter, global handlers, upload
breadcrumbs.
- Targeted backend suites pass. The frontend Jest suite passes in full.
- The full local backend run hit 93 failures from the shared dev env's
`idna` being upgraded mid-run by another process (`ImportError:
uts46data`). These are unrelated to this diff; CI is the gate.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Custom instrumentals can now be used in renders from WAV, M4A, OGG,
AAC, MP3, and FLAC files. When multiple supported versions are
available, FLAC is preferred.
* Crash reports now include additional browser diagnostics and recent
activity, helping provide more context when an error occurs.
* **Bug Fixes**
* Added a clearer message when an uploaded instrumental exceeds the size
limit, including the maximum size and supported format suggestions.
* Improved grouping of frontend errors when a stack trace is
unavailable.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
v0.243.1
|
||
|
|
455fb11a14 |
feat(upload): unified browser upload flow, bring-your-own instrumental, stranded-upload cleanup (v0.243.0) (#1058)
## Summary Two invisible upload failures on the same song led to this: - **Job 8729f19a**: large audio upload (signed-URL flow) abandoned mid-upload. The only UI signal was the "Creating..." button text and the card already said "Setting up", so the job sat `pending` forever with the credit spent. - **Job 2d0c1ae0**: 44 MB instrumental upload on the review screen was a multipart POST through Cloud Run (32 MiB cap) and failed silently. Only the CORS preflight was logged. ## Changes **Shared upload layer (frontend)**: all user uploads now go through one path: - `lib/upload.ts`: signed-URL PUT with progress, sequential multi-file upload with aggregate progress, client-side duration check. - `hooks/useUploadTask.ts`: progress state plus a `beforeunload` guard. - `components/upload/UploadProgressModal.tsx`: a modal the user can't close, showing progress, speed, ETA, "file N of M" and a keep-tab-open warning. - Used by guided job creation, tenant single job and instrumental review. Tenant bulk keeps its per-row table and uses the shared guard. **Guided flow** - Always uses the signed-URL path, which also fixes `requires_audio_edit` being silently dropped for uploads of 25MB or more. - Private upload jobs can attach their own instrumental (`existing_instrumental`). This skips separation and instrumental review. The duration is checked before upload, and the backing-vocals option is hidden. **Instrumental review** - New `instrumental-upload-url` and `instrumental-upload-complete` endpoints: direct-to-GCS upload, then a finalize step. - One shared processing helper with the multipart route: duration check, FLAC conversion, `stems.custom_instrumental`. - Returns a signed `audio_url`. The old post-upload preview URL had no backend route. - Inline error plus toast for mismatches and failures. **Backend robustness** - `state_data.awaiting_upload`: the dashboard shows "Waiting for upload" instead of "Setting up". - The hourly stale-review run also cancels and refunds non-tenant jobs still awaiting upload after 2h with nothing uploaded. - `uploads-complete` now checks ownership (previously any signed-in user could finalize any pending job). On an instrumental duration mismatch it cancels and refunds the job instead of stranding it. ## Testing - `make test`: backend 1839 + 4788 + 150 emulator passed; frontend Jest 1391 passed across 117 suites. - New tests: - backend: stale-upload processor; signed instrumental endpoints (15 cases); `requires_audio_edit` persistence; uploads-complete ownership and cancel-on-mismatch - frontend: upload lib, `useUploadTask`, modal (speed/ETA, can't be dismissed), `OwnInstrumentalField`, `TenantJobFlow`, API flows - Manual check against the real frontend (local dev, prod read-only; create, upload and finalize endpoints mocked, uploads sent to a throttled local PUT server, so no prod data was changed): - guided flow with and without an instrumental - instrumental review upload on 2d0c1ae0, including a mismatch caught before upload - Escape and outside clicks don't close the modal; navigating away triggers the `beforeunload` prompt Plan: `docs/archive/2026-09-26-unified-upload-flow-plan.md` 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>v0.243.0 |
||
|
|
763ce6592e |
fix: honor "edit audio first" for uploaded files (v0.242.1) (#1057)
## Summary Found while verifying #1056 in prod: an uploaded file with **requires_audio_edit** went straight to lyrics review, never stopping at the audio editor. - `state_data.requires_audio_edit` was only checked by the audio-download worker (URL/search downloads). - File uploads (`/api/jobs/upload`, signed-URL `uploads-complete`) and the URL/search fallback paths start via `JobManager.start_job_processing`, which ignored it. So users who uploaded a file and chose to edit the audio first never got the editor. - Fix: `start_job_processing` now parks flagged jobs at `AWAITING_AUDIO_EDIT` via the shared `enter_audio_edit` helper (renamed from the worker's private `_enter_audio_edit`), run in a thread to keep the transcode/waveform pre-generation off the API event loop. Duration reconcile and the workers then run at audio-edit submit, same as the download flow. ## Testing - New `TestStartJobProcessing::test_requires_audio_edit_parks_at_editor_instead_of_processing` (parks, no workers, no reconcile). - Related suites pass (job_manager, audio_download_worker, upload preflight, job creation regression, made-for-you, state machine flow: 216 passed). - Prod: re-run of the tempo/fade verification job after deploy. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Jobs requiring audio editing now enter the audio editor after upload or URL-based processing begins, rather than proceeding to audio or lyrics processing. * Duration reconciliation is skipped while a job is awaiting audio editing. * If audio-edit preparation fails, the job is marked as failed with an explanatory error instead of continuing through processing. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>v0.242.1 |
||
|
|
9db834c182 |
feat(audio-editor): tempo change with labeled outputs + mid-track fades (v0.242.0) (#1056)
## Summary ### Tempo change (new) - New whole-track `tempo` audio-edit operation (50–150%), pitch-preserving via ffmpeg **Rubber Band** (prod static ffmpeg 7.0.2 has librubberband; logged `atempo` fallback otherwise). Edits compound; undo/redo/session restore work unchanged. - Editor: **Tempo** button → dialog with slider + presets, resulting length, live pitch-preserving browser preview, persistent "Tempo: 90% of original" badge, and a labeling warning on submit. Audio edit happens before separation/transcription, so lyrics sync to the new tempo automatically. ### Labeling tempo-changed tracks in published outputs - At `/audio-edit/submit` (before workers start) `job.title` becomes `"Title (90% Tempo)"`. `job.title` is the display title every output reads: title/end screens + thumbnail, CDG title, filenames, Dropbox, GDrive/kjbox mirror, YouTube, downloads, emails. The original is pinned to `lyrics_title` for lyrics search; new `job.tempo_factor`. - YouTube descriptions lead with a tempo notice. Title edits on completed tracks keep the label; admin resets that restore normal-speed audio remove it. Lyrics-review search boxes default to the unlabeled title. - Also prevents a real hazard: server YouTube uploads replace same-titled videos, so an unlabeled variant would have deleted the normal-tempo original. ### Mid-track fades (from user feedback, 2026-09-12) > i would like the ability to fade-in and fade-out mid track rather than just at the beginning and end. - Fades now apply across the selection only (Audacity-style; atrim/afade/concat, sample-accurate, duration preserved), so they work anywhere. Edge selections behave as before (1s snap). Trim Start/End remain edge-gated. Design record: `docs/archive/2026-09-26-audio-editor-tempo-change.md`. ## Testing - `make test`: all backend + 1351 frontend Jest tests pass. - New tests: tempo op + real-ffmpeg fade/tempo tests, route validation, submit labeling (compounding, idempotent relabel, rounding parity with frontend), admin reset, title-edit enforcement, correction-data lyrics search title, YouTube description, AudioEditor component tests, `lib/tempo` tests, mocked-API Playwright E2E `e2e/regression/audio-editor.spec.ts`. - Local review: an AI code-review agent (CodeRabbit CLI unavailable for this org), 2 cycles. Fixed half-up rounding parity, the lyrics search default title, and slider-focus key handling. - Prod verification with a private test job planned after deploy. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Adjust track tempo from 50% to 150% while preserving pitch, with preview and cumulative change details. Tempo-adjusted tracks are labeled in titles and descriptions. * **Improvements** * Apply fade-in and fade-out effects to any selected audio region, leaving audio outside the selection unchanged. * Lyrics search continues to use the original title and artist when track titles include tempo labels. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>v0.242.0 |
||
|
|
f2b633175e |
chore(infra): cut encoding-worker disk cost (~$75/mo) + fix runner image pruning (#1055)
## Summary This cuts the GCP cost of the encoding-worker VM disks and custom images by about **$75/mo**. Functionality and instance-type diversity are unchanged. | Lever | Change | Est. saving | |---|---|---| | Boot disk size | All 10 workers go from 100 GB to **50 GB**. The Packer image `disk_size` goes from 100 to 30, because a disk can't be smaller than its source image. | ~$44/mo | | Hyperdisk IOPS/throughput | `-c4a` and `-n4db` were at 3600 IOPS / 290 MB/s, the paid defaults. They are now at the free baseline of 3000 / 140. New hyperdisks are created at 3000 / 140. | ~$18/mo | | Custom images | Deleted 23 old runner images, keeping the newest 2 per family. Fixed the prune step in `build-runner-images.yml` so they don't pile up again. | ~$13/mo | Measurements behind the sizing: - A worker 8 months into service used 15 GB of 100 GB. Fresh disks use 8 GB. - Job-folder scratch (1,904 jobs): p99 1.3 GB, max 4 GB. The deepest queue seen on a worker was 8. Full analysis, benchmarks and the rollout procedure are in `docs/archive/2026-09-26-encoding-worker-disk-cost.md`. ## Runner image prune bug The monthly step filtered with `-deprecated.state:*`, which is invalid. gcloud wrote the error to stderr, stdout was empty, and the step printed "Nothing to deprecate" every month. Deprecating wouldn't have saved anything anyway, since deprecated images are still billed. Also, gcloud's `family=gha-runner-gpu` filter is a word-boundary regex, so it matches `gha-runner-gpu-windows` too. The step now matches the family exactly on the client side, deletes images beyond the newest 2, and never deletes an image that an existing disk was created from. ## Rollout (stockout-safe) Pulumi can't shrink a boot disk. The only way it would get there is replacing the VM, which needs c4d/n4d/c2d capacity. So each **stopped** VM keeps running as the same instance, and only its boot disk is swapped with gcloud: detach, delete, create a 50 GB disk from the new image, attach it as boot. Then `pulumi refresh --target` on that instance. After that, `pulumi preview` shows no diff for the swapped VMs. ## Testing - `pytest infrastructure/test_encoding_worker_config.py`: 8 passed. A new invariant checks that the Packer image size is ≤ the worker disk size and that the worker disk is ≥ 40 GB. - Each swapped VM was started and checked with `/health` and a real full 4K `/encode`, and `df` was checked afterwards. Before/after timings are in the archive doc. c2df was the same before and after (142/139 s → 142/137 s), and n2f matched the August benchmark (247 s). ⚠️ Do NOT merge until every VM is swapped and refreshed. Merging with `size=50` while a 100 GB VM is still in Pulumi state would make a later `pulumi up` try to **replace** `encoding-worker-a/-b`. Status is tracked in the comments. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Cost Optimization** * Reduced encoding-worker boot disks from 100 GB to 50 GB, lowering estimated stopped-VM disk costs to about $4–5 per month. * Reduced the worker image disk size to 30 GB. * **Infrastructure Maintenance** * Image cleanup keeps the two newest active images per family and deletes older ones, while preserving images used by existing disks. * **Documentation** * Updated infrastructure guidance with disk sizing, image cleanup, and cost details. Added a report with rollout, verification, and rollback information. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
e9465c73d4 |
chore: gemini-3.8-flash everywhere + auto-correct on Opus 5.5 (cost) (#1054)
## Why GCP credits expired 2026-09-19; Vertex Gemini 3.1 Pro spend had grown $9 → $37 → $105/30d. `gemini-3.8-flash` is $0.75/$3.75 per 1M (vs $2/$12 for 3.1 Pro; rises to $1.50/$7.50 on 2027-01-01). Opus 5.5 is $4/$20 (vs $5/$25 for 4.8). ## Changes - All Gemini defaults → `gemini-3.8-flash`: credit eval, custom lyrics, auto-correct (default + compare leg), match judge, parse titles, tenant bulk analyze, agentic AI router, error-monitor LLM analysis (also `us-central1` → `global`, required for Gemini 3.x), frontend `translate.py`. - Auto-correct Claude leg `claude-opus-4-8` → `claude-opus-5-5` (ci.yml env), with `effort: "high"` pinned (5.5 defaults to medium). Pricing table gains both new models. - Docs updated; version 0.241.1. ## Evidence (12 recent prod jobs, pre-review lyrics scored against the human-reviewed final) | Model | Precision | Word-level recall | $/job | |---|---|---|---| | gemini-3.1-pro-preview | 91% | 74% | $0.067 | | gemini-3.8-flash | 98% | 38–41% | $0.020 | | gemini-3.8-flash @ high thinking | 97% | 60% (4/12 timed out, 504) | $0.041 | | claude-opus-4-8 | 94% | 33% | $0.080 | | claude-opus-5-5 (effort high) | 93% | 58% | $0.048 | Flash for the Gemini leg is a deliberate cost/recall trade-off (Andrew chose it); Opus 5.5 more than compensates. Also verified: `gemini-3.8-flash` works on Vertex `global` (rejects `thinking_level=minimal`; nothing here uses it), match-judge latency median 3.3s / max 6.2s (< 12s timeout), Japanese translation smoke test preserves ICU placeholders. Translation cache is keyed on English text + locale, so no re-translation churn. Tests: auto_correct, credit eval, custom lyrics, parse_titles, error_monitor, agentic router suites pass locally. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * AI-powered correction, translation, matching, title parsing, and evaluation now use updated models, including Gemini 3.8 Flash and Claude Opus 5.5. * **Updates** * AI service pricing reflects scheduled rate changes, including a Gemini 3.8 Flash rate change beginning January 1, 2027. * Application version updated to 0.241.1. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>v0.241.1 |
||
|
|
b177c36641 |
feat(kjbox): free-text song resolve for singer search auto-correct (v0.241.0) (#1053)
## Summary
Andrew wants the kjbox singer song search to auto-correct like gen's job
creation does, e.g. "the strokes max picu" → The Strokes — Machu Picchu,
with Undo. gen's match-judge only takes a separate artist and title, and
its catalog stage only fixes formatting, not typos. So this adds:
- **`POST /api/kjbox/catalog/resolve`** (`X-Kjbox-Secret`) `{query}` →
match-judge verdict shape (`kind` cosmetic/content/ambiguous/none,
`confident`, `canonical_*`, `alternatives`) plus
`typed_artist`/`typed_title`.
- **`backend/services/match_judge/free_text.py`**: one small Gemini call
(the match judge's model and settings) that splits and corrects
together. Bad or incomplete model output, or any model failure, degrades
to `kind: "none"`, never a 5xx.
- **Cost and abuse controls:**
- Results are cached per case-folded query (in-process, 2000 entries);
transient failures are not cached.
- A partner-wide cap of 120 AI calls/min per instance (429
`rate_limited`); cache hits are free.
- kjbox only calls this after its own catalogue search came back empty.
- Docs: `docs/API.md` § kjbox Partner API.
## Testing
- `test_match_judge_free_text.py` (8)
- `TestCatalogResolve` (4)
- Full backend suite passes (emulator tests excluded)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added free-text song search resolution that identifies artist and
title, corrects typos, and can suggest alternatives when a match is
ambiguous.
* Repeated searches can use cached results. If resolution is
unavailable, the request returns without an error; uncached requests may
be rate-limited.
* **Documentation**
* Documented the song-resolution API, including its response behavior
and request limits.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
v0.241.0
|
||
|
|
36e95c7fe9 |
feat(kjbox): on-demand review sign-in link + record who started the review (v0.240.0) (#1052)
## Summary
Backs Andrew's feedback on the kjbox singer make-it flow (kjbox
companion PR to follow):
- **`POST /api/kjbox/jobs/{job_id}/review-link`** (partner secret +
singer Bearer session): mints the same one-click `job_review:<id>`
sign-in link the kjbox review email carries, on demand. This powers
"**Tap here** to review it yourself" on the singer's My songs. It is
owner-only (404 otherwise, including unknown ids) and only while
`awaiting_review`/`in_review` (409 `not_in_review`). It also returns
`review_started_by`.
- **`state_data.review_started_by` / `review_started_at`** are recorded
on the first `AWAITING_REVIEW → IN_REVIEW` transition. The value is
`"admin"` when an admin other than the owner (the KJ) opened it, and
`"owner"` otherwise (review-token links count as owner). It is wired
into `GET /api/review/{id}/correction-data` (`require_review_auth` now
stashes the reviewer on `request.state`) and into the legacy
`start-review` / `corrections` routes. kjbox reads it through its admin
job poll to show "Andrew has already started reviewing".
- Docs: `docs/API.md` § kjbox Partner API.
## Testing
- New `TestReviewLink` (6 tests) in `test_kjbox_partner_api.py`
- New `test_review_started_by.py` (7 tests)
- New route-level `TestCorrectionDataRecordsReviewStarter` (2 tests)
- Full backend suite: 4671 passed (excluding emulator tests)
CodeRabbit CLI has no seat on this org, so this PR is opened without the
ignore line and the GitHub bot will review it.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added a partner API endpoint for job owners to request a localized
sign-in link for jobs awaiting or undergoing review.
* Review details now indicate whether the owner or an administrator
started the review.
* **Documentation**
* Updated the API documentation with the required partner secret header
for review-link requests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
v0.240.0
|
||
|
|
d570b1f139 |
feat(kjbox): partner API for singer email-code sign-in + show credits (v0.239.0) (#1051)
## Summary
Partner API so the kjbox singer UI (live karaoke nights) can turn
patrons into real karaoke-gen customers: the singer verifies their email
with a 6-digit code typed into the kjbox page, and kjbox then searches
audio / creates the job **as that user** (normal gen delivery emails).
All under `/api/kjbox`, gated by `X-Kjbox-Secret` (Secret Manager
`kjbox-partner-secret`, wired in ci.yml; 503 when unset):
- `POST /auth/send-code` — emails a 6-digit code (hashed at rest, 10-min
expiry, new code supersedes old). Exempt from the 2-signups-per-IP cap
(a venue shares one IP) — replaced by a partner-wide 100 new accounts /
24h cap + 5 codes / email / hour. New users tagged `signup_source=kjbox`
(+ venue).
- `POST /auth/verify-code` — 5 attempts then burned, single-use; shares
the post-login path with magic-link verify (refactored into
`complete_verified_login`: welcome credit w/ AI eval, locale, referral,
session, welcome email).
- `POST /credits/show-credit` (+ user Bearer) — +1 credit, no email
("free at Andrew's shows"), idempotent per key, 5 / user / 24h.
- Review-needed + 24h reminder emails for `X-Client-Id: kjbox*` jobs
carry a one-click sign-in link to the review page (singers never signed
in on gen's site).
Paired kjbox PR: nomadkaraoke/kjbox (singer make-it flow, v0.116.0).
Plan: kjbox `docs/archive/2026-09-24-singer-make-it-auto-flow-plan.md`.
## Testing
- New: `backend/tests/test_kjbox_partner_api.py`,
`backend/tests/emulator/test_kjbox_partner_service.py` (+ additions to
existing tests for the verify refactor, review-email links, X-Client-Id
on create-from-search)
- Backend unit suite: 4644 passed; emulator suite passed.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added partner sign-in with one-time email codes and show-credit
claiming, with limits on code requests and credit grants to prevent
repeated or duplicate claims.
* Added localized sign-in emails and single-use review links for
eligible reminders.
* Sign-in supports welcome-credit evaluation and referral attribution.
* **Improvements**
* Review reminders can use custom links, and new accounts can record
signup source and venue.
* Credential-like request headers are redacted from stored metadata.
* **Documentation**
* Added partner API usage and behavior details.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
v0.239.0
|
||
|
|
5d56ee9069 |
fix(frontend): stop reporting opaque cross-origin "Script error." (v0.238.6) (#1050)
## Summary The error monitor keeps alerting on a frontend pattern `@https://gen.nomadkaraoke.com/` with sample `Error: Script error.` (iOS Safari, 4 occurrences since June). That's the browser's opaque cross-origin error: a script from another origin (a third-party tag, in-app browser or extension injection) threw. The browser hides everything, so there's no error object, filename or line. Our `window.onerror` handler wrapped it in `new Error(event.message)`, so the only "stack" frame (`63bfc60a…js:6:874`) is our own handler. There's nothing actionable in these reports. ## Changes - `frontend/lib/client-error-setup.ts`: new exported `isOpaqueCrossOriginError(event)`. The window `error` listener returns early when `event.error == null` and the message is `Script error.`. - `backend/services/error_monitor/frontend_ingestion.py` + `backend/api/routes/client_errors.py`: `is_opaque_script_error()` drops bare `Script error.` / `Error: Script error.` reports at ingestion. They're accepted (202) but not persisted, the same way the bot filter works. This also stops older cached bundles from alerting. - `docs/ARCHITECTURE.md`: a note on the crash-report noise filters. - Version 0.238.5 → 0.238.6. ## Testing - New Jest `lib/__tests__/client-error-setup.test.ts`: tests the predicate; an opaque event is not reported; a real error is still reported. Full Jest: 1342 passed. - Backend: parametrized API tests (dropped vs "Script error in …" still persisted) and unit tests for the predicate. `tests/unit` passed (1834), and `backend/tests` without the emulator passed (4591). Emulator tests are left to CI. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Opaque cross-origin “Script error” events without an error object and location details are no longer recorded. Other client errors, including script errors with location details or different messages, continue through the reporting flow. * Matching is case- and whitespace-tolerant and allows an optional trailing period. * **Documentation** * Updated Error Monitor guidance to describe which browser and server-side errors are ignored. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>v0.238.6 |
||
|
|
bcdfd6b6bb |
fix(jobs): unstick audio-edit jobs — waveform OOM, retry skip, screens handoff (#1049)
## Summary Job `2c1b922e` (Dave Matthews Band – Why I Am, 24-bit/96kHz FLAC, audio edit requested) stalled twice in prod. Three bugs: 1. **Waveform OOM** — `WaveformGenerator.generate_data_only` decoded the full source at native resolution through pydub. For a 6-min hi-res FLAC that peaked at ~1 GB RSS and OOM-killed the 1 GiB `audio-download-job` while pre-generating editor assets. It now decodes straight to 16 kHz mono PCM via ffmpeg and computes the envelope with numpy: **156 MB vs 1.07 GB peak** on the real file, amplitudes within 0.003 of the old output. The same function backs the editor's on-demand waveform in the API. 2. **Retry skipped unfinished audio-edit prep** — the Cloud Run auto-retry saw `downloading`, treated the download as done and exited 0, leaving the job at `downloading` forever. A retry that finds an audio-edit job still at `downloading` now resumes the editor prep. That path triggers nothing downstream, so it's safe. Non-edit jobs keep the existing skip (downstream may already be triggered). 3. **Screens handoff ignored `audio_edit_complete`** — `advance_to_screens_if_ready` (added in #1002) and the `recover-stuck-jobs` lost-screens sweep only accepted `downloading`. Audio-edited jobs prep at `audio_edit_complete`, so **every audio-edited job since Sept 15 stalled** after lyrics and separation finished (log: `screens not (re)triggered — status=audio_edit_complete`). Both now use a shared `PREP_PHASE_STATUSES`. 2c1b922e was unstuck manually (admin reset → `awaiting_audio_edit`, then a direct `/api/internal/workers/screens` dispatch). Added a TROUBLESHOOTING entry covering both signatures and the manual fixes. ## Testing - New tests: hi-res 96k/24-bit decode path (low-rate mono args, envelope matches pydub reference, silent → 0, undecodable → error); download-worker resume vs skip; handoff at `audio_edit_complete`; watchdog recovers `audio_edit_complete` and queries both statuses. - `pytest tests/unit/` → 1824 passed, 8 skipped - `pytest backend/tests/ --ignore=backend/tests/emulator` → 4586 passed, 36 skipped - Version 0.238.4 → 0.238.5 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Audio-edit jobs can resume preparation after an interruption without redownloading existing audio. * Screen generation can proceed after audio editing is complete, and stalled screen-generation triggers can be recovered. * Waveform previews are more reliable for high-resolution and silent audio. Invalid audio and decoding timeouts now produce clear errors. * **Documentation** * Added troubleshooting guidance for jobs stuck during audio editing or before screen generation. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v0.238.5 |
||
|
|
daa46347f7 |
docs: KaraokeHunt outreach handoffs + session record (#1048)
Docs-only. Brings the KaraokeHunt outreach handoff docs (2026-09-13, 09-19, 09-24 final phase) and the 09-19 session record onto main before the outreach worktree is cleaned up. Covers Segments A/B/C being sent (87 + 238 + 35) and the Postmark outbound-stream sending method. This is a public repo, so customer emails and handles are redacted to `<user>`. The unredacted working copies stay in the local PII directory. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
6c28a8cc3c |
fix: /karaokehuntnews wording - original song, not a cover band (v0.238.4) (#1047)
Wording change requested by Andrew for the Segment C / Frog #1 launch letter, to match the email as sent: "(the vocals are genuinely gone, not just quieter)" -> "(original song, not a cover band)". en + 32 locales retranslated. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Updated the karaoke generator description across supported languages to clarify that instrumental separation uses the original song, not a cover band. The description also highlights word-by-word synced lyrics, an estimated half-hour turnaround, and support for 33 languages. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v0.238.4 |
||
|
|
20f9334995 |
feat: /karaokehuntnews letter for KaraokeHunt mailing-list subscribers (v0.238.3) (#1046)
## Summary Localized letter page for the launch email to KaraokeHunt **mailing-list subscribers who never used the app** (outreach Segment C, folded into Frog #1). Linked from that email's PS, same pattern as `/karaokehunt` and `/karaokehuntlist`. ## Changes - `KaraokeHuntLetter`: optional `link` for the credits paragraph (defaults to gen) + `hasCreditsAfter` toggle - `/[locale]/karaokehuntnews` + bare `/karaokehuntnews` locale redirect (no-JS fallback) - `karaokehuntNewsLetter` strings, 33 locales; CTA link = `nomadkaraoke.com/r/thankyou50` (50%-off vanity referral link, created via admin API) - Version 0.238.3 ## Testing - Jest: new case for the news letter (custom link, no trailing text); full frontend suite 1337/1337 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v0.238.3 |
||
|
|
c8bd29654b |
feat: /karaokehuntlist letter for KaraokeHunt app users (v0.238.2) (#1045)
## Summary Localized letter page for **Segment B** of the KaraokeHunt outreach (app users who signed up but never requested a song). The existing `/karaokehunt` page opens with "you requested a song", which is wrong for these users, so the Segment B email's PS links here instead. ## Changes - `components/karaokehunt/KaraokeHuntLetter.tsx`: shared letter layout extracted from `/karaokehunt` (namespace + ordered paragraph keys) - `/[locale]/karaokehunt`: now uses the shared component (rendering unchanged) - `/[locale]/karaokehuntlist` + bare `/karaokehuntlist` locale redirect (with no-JS fallback, same as `/karaokehunt`) - `karaokehuntListLetter` strings in all 33 locales - Version 0.238.2 ## Testing - New Jest tests for the shared letter (both variants + credits link); full frontend suite 1336/1336 passing - Visually verified locally: `/karaokehuntlist` (en, he RTL) and `/karaokehunt` unchanged 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a localized note for former KaraokeHunt users who signed up but never requested a song, available in 33 languages. * The note introduces Nomad Karaoke, explains how to sign in using the email associated with KaraokeHunt, and offers three free credits. * **Updates** * Refreshed the existing KaraokeHunt letter page with a shared, language-aware layout. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v0.238.2 |
||
|
|
77132b38fb |
feat: localized KaraokeHunt letter page at /karaokehunt (v0.238.1) (#1044)
## Summary - New page **/karaokehunt** - "A note for KaraokeHunt users", linked from the outreach emails (Frog #2 / Phase C). - Bare URL is a `LocaleRedirect` shim, so one link lands every visitor on their own language (`/[locale]/karaokehunt`); page includes the LanguageSwitcher + ThemeToggle. - Copy mirrors the approved batch-1 email wording (hyphens only, no em-dashes), generalized: apology, made-good requests, free credits + sign-in, app retired/uninstall note. - Strings in `messages/en.json` under `karaokehuntLetter`; translated to all 33 locales via `translate.py --target all` (100% key coverage). ## Testing - `npx jest --ci`: 1333 tests, 108 suites, all green. - `npm run build`: production build green, `/karaokehunt` + SSG locale routes emitted. - ESLint clean on new files; no tsc errors in new code. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a letter for former KaraokeHunt users explaining the app’s retirement, where to find the new service, and how to access complimentary credits. * The letter is available in 33 languages, with language and theme controls. * **Improvements** * Theme-control labels are now translated to match the selected language. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>v0.238.1 |
||
|
|
c00d64553c |
docs: session record — Recent Jobs sort control (#1043)
## Summary - Adds the session record for the Recent Jobs sort control feature (PR #1042, v0.238.0), documenting what changed, decisions/rationale, and gotchas for future sessions - Docs-only, no code changes 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
|
0ec6909eb0 |
Add sort control to Recent Jobs dashboard list (#1042)
## Summary - Adds a "Sort by" dropdown (submission date, recently active, completion date, artist, title) plus an ascending/descending toggle to the Recent Jobs list on the dashboard, replacing the fixed newest-first order - `updated_at` was missing from the Firestore summary-field projection (`SUMMARY_FIELD_PATHS`) — added it, needed for "recently active" sorting - No dedicated completion timestamp exists in the data model, so "Completion date" reuses `updated_at` for jobs in a terminal success status (`complete`/`prep_complete`) and pushes everything else to the end of the list - Sort preference persists to localStorage like the existing status/limit filters - Artist/title collation uses the active dashboard locale (`next-intl`) rather than the browser default (found + fixed via local CodeRabbit review) ## Test plan - [x] `make test` — full backend + frontend suite (106 suites, 1322 frontend tests) passes - [x] New unit tests for `sortJobs()` in `frontend/__tests__/job-status.test.ts` (date fields, artist/title collation incl. locale, missing-value handling, immutability) - [x] New backend regression test asserting `updated_at` stays in `SUMMARY_FIELD_PATHS` - [x] New Playwright e2e test (`job-management.spec.ts`) exercising the sort dropdown + direction toggle against mocked job data - [x] `tsc --noEmit` and `eslint` clean on all changed files - [x] Local CodeRabbit review: 1 finding (locale-aware collation), fixed; verification pass clean @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
|
342d0a5e67 |
feat(review): instant backing-vocals audition + resizable/collapsible reference panel + Ctrl-click delete in Waveforms (v0.238.0) (#1041)
## Summary
### Preview modal: backing-vocals waveform click played nothing for 7–15
s
**Root cause (prod logs + GCS timestamps):** review OGGs are "eagerly"
transcoded in `screens_worker`, but screens triggers on *lyrics*
completion and audio separation usually finishes later — so the stem
OGGs didn't exist and the API transcoded on demand (`Cache miss,
transcoding …`) on the modal's first fetch. First `/instrumental-audio`
request per job: 7.7–15.5 s vs 0.1–0.7 s once cached. A sweep found
**29/38** in-review jobs affected.
- `audio_worker` now transcodes the review stems (clean, with-backing,
backing vocals — in parallel, forced) right after uploading them.
- Review page pre-warms the browser cache with the stem URLs once the
main review audio is ready (also warms the server transcode for older
jobs).
- While encoding, standalone stem playback shows **"Loading: <stem>…"**
+ spinner until `playing` fires (and again on `waiting`), instead of
claiming "Playing" over silence.
- Clicking the pink waveform now also **selects "Instrumental + backing
vocals"**.
- Ops (already done): backfilled OGGs for the 29 affected in-review jobs
via `POST /api/admin/jobs/{id}/prepare-review-audio`.
### Waveforms mode
- **Draggable divider** between Synced and Reference Lyrics (min 160 px,
max 70 %); narrower reference lines wrap. Double-click resets to the
auto width; ←/→ when focused. Persisted in localStorage.
- **Collapse button** in the Reference header hides the column; a slim
vertical "Reference Lyrics" strip re-expands it. Persisted.
- **Ctrl/Cmd-click deletes a word** (was: opened Edit Segment modal),
matching Simple/Advanced — handled on mousedown so no drag starts.
## Testing
- `make test`: backend 4580 + 1820 + 140 emulator passed; frontend Jest
1324 passed; Playwright passed.
- New tests: `TestTranscodeReviewStems`,
`test_audio_worker_review_stems.py`, `useReferencePanelLayout.test.tsx`
(clamp/persist/drag), PreviewVideoSection loading indicator,
ReviewChangesModal waveform-click selection, WaveformSegmentRow
Ctrl-click delete.
- Manually verified in the local fixture harness (Playwright): drag →
reference 329 px / synced 1031 px, lines wrap, no horizontal overflow;
collapse → synced full width; Ctrl-click deleted a word without opening
the modal.
- i18n: 4 new keys × 33 locales.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* In Waveforms review, resize or collapse the Reference panel;
double-click its divider to reset the width.
* Delete words directly from waveform rows using the existing
delete-word interaction.
* Clicking the backing-vocals waveform selects that instrumental for the
final video.
* **Improvements**
* Review audio stems are prepared earlier for smoother playback. A
loading indicator appears while stems load or playback stalls.
* Instrumental stems preload after review audio is ready, except in
read-only mode.
* **Bug Fixes**
* Waveforms and Reference columns retain their intended sizing and
layout when adjusting the panel.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
v0.238.0
|
||
|
|
6dd944d0f3 |
fix(review): Complete no longer waits on the preview video; abandon preview encodes once the job leaves review (v0.237.0) (#1040)
## Summary - Clicking **Complete** in the lyrics review before the preview video loaded appeared to wait for the preview. The frontend never gated on it — the API instance's **event loop was frozen by synchronous Compute API VM-start calls**, stalling every request on that instance (prod logs, job e80caa57, 2026-09-23: `POST /corrections` handler finished in 0.5 s but responded after 13.9 s; `POST /complete` took 18.6 s; a trivial `GET /jobs` took 5.4 s in the same window). - 14-day sample: 62/142 review completes took > 3 s (max 35 s); "Encoding worker unreachable — started VM" fires from the API service 2–6×/day, freezing an instance each time. - Investigation + numbers: `docs/archive/2026-09-23-complete-skip-preview-wait-plan.md`. ## Changes - `worker_service`: the encoder warmup (documented as fire-and-forget, but called inline and blocking) now runs via `asyncio.to_thread` in a tracked background task; render/video dispatch returns immediately. Correctness unaffected — the render worker already self-heals with its own warmup fallback. - `encoding_service` / `encoding_worker_manager`: `ensure_any_running` / `ensure_primary_running` and the per-poll `get_vm_status` in `wait_for_worker_ready` wrapped in `asyncio.to_thread`. - `review.py`: the background preview encode gets a watcher that re-reads job status from Firestore every 3 s (cross-instance safe) and cancels the encode — no VM cold start, submission, polling, local fallback or error marker — once the job leaves `awaiting_review` / `in_review`. - Not done: killing an already-running ffmpeg on the GCE worker (≈10 s in the light lane; would need a Popen registry + worker restart to deploy). The expensive waste — cold-starting a VM and API-side polling for a discarded preview — is what the watcher prevents. - `docs/LESSONS-LEARNED.md` entry; version 0.237.0. ## Testing - [x] `backend/tests/test_services.py`: trigger returns without waiting on a slow warmup and the loop stays responsive; warmup still runs; sync callers unchanged - [x] `backend/tests/test_encoding_service.py`: warmup fallback no longer blocks the event loop - [x] `backend/tests/test_routes_review.py`: preview abandoned when the job leaves review; continues while in review; transient status-read error doesn't abandon - [x] Full backend suite: 4570 passed, 36 skipped - [ ] Prod verification after deploy: complete a review before the preview loads on a cold encoder; expect `/complete` ≈ 1–2 s and a "Abandoning preview … job left review" log line ## Follow-up (out of scope) - Preview submission's 30 s aiohttp timeout against a stopped VM (TCP black-hole) delays cold previews ~25 s; a short `connect` timeout would fix it. --- 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Review completion and other requests remain responsive while video-encoding workers start, rather than being held up by that process. * Background preview encoding stops when a job leaves the review stage, avoiding unnecessary work and preventing stale previews or error indicators. * Temporary status-check errors no longer prematurely stop preview encoding. While a preview is rendering, it is discarded if the job leaves review before it can be published. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>v0.237.0 |
||
|
|
a9c6de0617 |
feat: KaraokeHunt interceptor — one freebie per email + community-version short-circuit (v0.236.0) (#1039)
## Summary Follow-ups from Andrew's live test of the v0.235.0 interceptor (#1037): 1. **Community short-circuit** — his "abba - Waterloo" test generated a real job despite a community karaoke version existing. Now `check_community_versions` gates the whole routing: existing version → no job, no board; the email's main button is the YouTube link (outcome `community_existing`), and new users keep their conversion credit for a custom song. 2. **One freebie EVER per email** (Andrew's spec): the goal is converting installs into direct gen.nomadkaraoke.com users, not keeping the dead app alive. First-time emails (all variants) now state the app is retired and they must uninstall it and use Gen directly. A second request from a converted address gets no job/board/credit — just the "please uninstall" email (`repeat_request`), throttled to one per address per 7 days. Accidental same-song double-taps stay silent (`duplicate` dedup runs first). ## Implementation - `karaokehunt_conversion.py`: `CONVERTED_OUTCOMES` + `_freebie_already_used` (fails open) + `_uninstall_email_recently_sent` (fails closed — anti-spam) + community short-circuit before routing; queried fields re-verified in Python (defense in depth). - `send_karaokehunt_conversion`: new `community` and `uninstall` variants; rewritten intro (retired app, one-time freebie); bold uninstall note on every first-time variant; new keys translated to all 32 locales. - Docs: API.md + interceptor README routing updated. ## Testing 42 karaokehunt tests (6 new: community short-circuit ×2, repeat/uninstall ×2, email variants ×2 more); full `make test` green. Post-merge: live POSTs through create.karaokehunt.com for (a) a song with a community version and (b) a repeat request from a converted address. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * KaraokeHunt requests now receive a one-time conversion per email address. Repeat requests receive an uninstall reminder instead of a new conversion, with reminders limited to once every seven days. * When a community version is available, the email links directly to it; new users keep their free credit. * Conversion emails explain credit status and include a Nomad Karaoke sign-in link. * **Documentation** * Updated guidance on request outcomes and the one-time conversion process. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>v0.236.0 |
||
|
|
a382af349d |
docs: record karaokehunt.com zone changes for the interceptor (#1038)
Docs-only addendum to infrastructure/cloudflare/karaokehunt-interceptor/README.md: records the two zone-level Cloudflare changes (catch-all redirect scoped to exclude create.karaokehunt.com; Bot Fight Mode disabled) that were required to make the Worker reachable by the retired app's HTTP client. Follow-up to #1037. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Documentation** - Documented Cloudflare zone configuration updates affecting access to the interceptor. - Recorded that redirect handling now excludes the application host, allowing requests to reach the Worker. - Documented the disabling of Bot Fight Mode to prevent managed browser challenges from blocking non-browser clients. - Clarified that apex and `www` redirects remain unchanged and included guidance for reverting the configuration. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
7b4af11362 |
perf: cut backend cold start ~16s → ~4s via lazy imports + background warmup (v0.235.0) (#1036)
## Summary
A freshly-spawned `karaoke-backend` Cloud Run instance took **~16s
before serving its first request** — Cloud Run holds routed requests
during boot, so every scale-out was a 16s latency cliff (measured
2026-09-22 during the 16-tab load test: warm instance 1.2s vs fresh
instances 15.8–18.8s for the same request). Handoff spec:
`docs/archive/2026-09-22-gen-cold-start-reduction-handoff.md`
(workspace).
Measured breakdown: **~8–9s Python imports** + **~7.5s lifespan
preloads** (spaCy ~1.0s, NLTK cmudict+punkt ~2.3s, Langfuse ~3s,
credential validation ~1.6s — all gating readiness, from prod logs).
## Changes
**Lazy imports, fixed at the source module** so every importer benefits
and the CLI keeps working (imports now happen at first use inside worker
code paths):
| Module | Deferred | Cost removed |
|---|---|---|
| `karaoke_gen/.../utils/syllable_counter.py` | spacy, nltk, pyphen,
syllables → `__init__`/methods | ~2.6s |
| `karaoke_gen/.../correction/phrase_analyzer.py` | spacy → `__init__`
(was also pulling **torch** via thinc) | ~1.7s |
| `karaoke_gen/audio_processor.py` | audio_separator
(torch/onnxruntime/librosa), pydub | ~2.9s |
| `backend/config.py` | google.cloud.secretmanager → `get_secret()` |
~1.7s |
| `backend/services/{custom_lyrics,auto_correct}/service.py` |
google.genai → `_call_gemini()` | ~1.2s |
| audioshake/whisper transcribers, cdgmaker composer | pydub → using
methods | small |
`audio_processor` keeps `Separator` / `AudioSeparatorAPIClient` /
`REMOTE_API_AVAILABLE` / `SEPARATOR_AVAILABLE` as (lazy) module
attributes via PEP 562 `__getattr__`, so existing test patch targets are
unchanged; intra-module code uses helpers because LOAD_GLOBAL bypasses
module `__getattr__`. Same pattern keeps `custom_lyrics.service.genai`
patchable.
**Lifespan** (`backend/main.py`): the spaCy/NLTK/Langfuse preloads +
credential validation moved into a daemon thread
(`_run_background_warmup`). They warm caches for the *first
lyrics-processing job* — they never needed to gate HTTP readiness.
Workers racing the warmup fall back to the preloaders' existing lazy
paths. Heavy imports happen in this thread, not in async handlers
(event-loop-blocking gotcha from v0.230.0/v0.231.1).
## Results
- Local `import backend.main`: **10.63s → 2.75s** (remaining is
dominated by `google.cloud.firestore` ~0.8s, a genuine serving
dependency)
- Lifespan startup: ~7.5s → instant ("Application startup complete"
same-second)
- Expected prod cold start: **~16s → ~3–4s** (to be verified post-deploy
with `scripts/load-test-review.py --tabs 16` + fresh-instance request
latency)
## Tests
- **New:** `backend/tests/test_cold_start_imports.py` —
subprocess-imports `backend.main`, asserts none of the heavy libraries
(torch, spacy, nltk, audio_separator, matplotlib, pydub, google.genai,
secretmanager, librosa, onnxruntime) leak into `sys.modules`;
`backend/tests/test_main_startup_warmup.py` — warmup calls all preloads,
survives individual failures, and doesn't block readiness.
- Full backend + package suites: **4541 passed** (incl. the
audio_processor/custom_lyrics/auto_correct patch-target suites). Ruff:
no new findings on changed files.
## Follow-up (separate PR after prod verification)
If fresh-instance first-request latency lands <5s, drop `--min-instances
2 → 1` in `ci.yml` — the second warm instance was the cold-start
mitigation (~$15/mo).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Performance**
- Reduced backend cold-start time by deferring heavy dependencies until
they are needed.
- Startup readiness is no longer blocked by model, language-resource,
analytics, or credential warmups; these now run in the background.
- Secret Manager and audio, transcription, lyric, and correction tools
load on demand.
- **Documentation**
- Added guidance and measured results for diagnosing and reducing
backend cold starts.
- **Quality**
- Added automated checks to ensure startup remains lightweight and
background warmups continue safely after individual failures.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
||
|
|
3d1ee6a621 |
feat: KaraokeHunt retired-app request interceptor — auto-convert to Gen users (v0.235.0) (#1037)
## Summary The retired KaraokeHunt mobile app still has installs submitting ~5-10 song requests/month. Each request POSTs to the dead `create.karaokehunt.com`, fires a client-side Pushbullet push to Andrew, and shows the user a hardcoded *"you should receive an email in 5-10 minutes with a link to your new karaoke video"* — a promise broken for years. This PR makes it true and converts those users into Gen users (Andrew's spec, BACKLOG Frog #2 extension). ## How - **Cloudflare Worker** (`infrastructure/cloudflare/karaokehunt-interceptor/`, deployed via `deploy.sh` — the karaokehunt.com zone is not IaC-managed): answers the app instantly, `waitUntil`-forwards the payload to the new backend intake with a shared secret. - **`POST /api/karaokehunt/request`**: unauthenticated intake, gated by `X-KH-Forwarder-Secret` (Secret Manager `karaokehunt-forwarder-secret`, already created; wired in ci.yml `--set-secrets`), 10/min/IP rate limit, lenient payload parsing, Firestore `karaokehunt_requests` audit trail. - **Conversion** (`backend/workers/karaokehunt_conversion.py`, mirrors `community_daily_pick`): - new email → account + 1 credit (`karaokehunt_app_conversion`) → job created **as them** (consumes it); first sign-in still grants the welcome credit (deliberate — next song covered) - existing user with credits → job on their credit (email offers to restore it on reply) - no credits, or over `KARAOKEHUNT_DAILY_JOB_CAP` (default 3/day, bounds unattended spend) → free community requests board (dedup → upvote) - conservative `pick_auto_selection` (not `select_best`): unconfident matches park in `AWAITING_AUDIO_SELECTION` and the email asks them to pick — garbage app inputs must never auto-publish to YouTube - 14-day per-email+song dedup; failures retryable via admin `POST /api/karaokehunt/internal/reprocess/{doc_id}` - **Email** `send_karaokehunt_conversion`: 3 variants (job started / pick the recording / on the board), 7-day one-click sign-in link, KaraokeNerds community-version bonus link when one exists. EN + all 32 locales via translate.py. ## Safety Deploys **dark**: endpoint 503s until the secret env lands, and receives no traffic until the Worker + DNS flip (done post-merge via `deploy.sh`). Kill switch = delete the Worker route; backend-side = remove the secret mapping or `KARAOKEHUNT_DAILY_JOB_CAP=0`. ## Testing - 36 new tests (`test_karaokehunt_intake.py`, `test_karaokehunt_conversion.py`): secret/rate-limit/JSON gates, all conversion routings, credit-grant guards, park-on-unconfident-search, email variants + HTML escaping, reprocess auth. - Full `make test` green (backend + frontend). - Post-deploy: live end-to-end POST through `create.karaokehunt.com` mimicking the app payload, verifying account/job/email/Firestore. Plan doc: `docs/archive/2026-09-22-karaokehunt-app-interceptor-plan.md` 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Requests submitted through the retired KaraokeHunt app are now received and converted into Nomad Karaoke jobs, accounts, or community-board requests. - Users receive localized email updates with job status, credit usage, audio-selection prompts, and tracking links. - Requests can be retried by administrators when processing fails. - Duplicate protection, request limits, and daily processing caps help prevent repeated or excessive submissions. - **Documentation** - Added API and deployment documentation for the request-forwarding workflow. - **Chores** - Released version 0.235.0. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>v0.235.0 |
||
|
|
d4b5768d26 |
feat: divebar catalog GCS export + kjbox read access to KN exports (v0.234.1) (#1032)
## Summary Gen-side groundwork for the kjbox **local catalog mirror** (search-unification handoff rec 4 — sub-0.3s offline-capable searches at live shows; kjbox PR to follow): - **`divebar_mirror` CF — catalog snapshot export.** After the staging MERGE (must be after: `gcs_path` lives only in the main table, set by the file-sync VM), the index build now writes `gs://nomadkaraoke-divebar-files/exports/divebar-catalog-latest.json.gz` — gzipped NDJSON with exactly the CF `search` column set (`file_id, brand, brand_code, artist, title, filename, format, file_size, drive_path, in_gcs`), `row_count`/`exported_at` object metadata, and deterministic gzip (mtime=0) so the box's sync can hash-skip unchanged content. Best-effort: an export failure is logged into the result but never fails the index build. **No new exposure**: the bucket is already public (allUsers objectViewer — it's how CF download URLs work) and every row is already publicly queryable via the divebar-lookup `search` action. - **`kn_data_sync` Pulumi module — box read access.** `nomad-master-sync@nomadkaraoke.iam.gserviceaccount.com` (the SA already on the box for the NOMAD-720p master rsync, key at `/opt/nomad/secrets/`) gets `roles/storage.objectViewer` on the **private** kn-data bucket, so the box can fetch `full/full-data-latest.json.gz` + `community/community-data-latest.json.gz`. KN data stays private — no public access added. ## Testing - 13 CF tests green (new `test_export_catalog.py`): NDJSON shape pinned to the CF column set (kjbox depends on it), accent round-trip, in_gcs booleans, metadata, public-bucket target, gzip determinism, and best-effort wiring (export raising → index build still returns ok with the error recorded). - divebar_lookup CF tests unaffected (43 green). ## Deploy Infra-only. Will `pulumi up` locally before merge per repo policy, then force-run the nightly `divebar-mirror-daily` job once to produce the first export. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Catalog data is now exported as a compressed snapshot to Cloud Storage for the local KJ box mirror. - Snapshots include row-count and export-time metadata, with deterministic output for consistent re-exports. - **Bug Fixes** - Catalog export runs best-effort: failures are captured in the sync response without breaking the main synchronization flow. - When no catalog rows are produced, the export is skipped and noted in the response. - **Infrastructure / Security** - Added a bucket-level IAM grant to support KJ box export mirroring. - **Tests** - Added coverage for snapshot determinism and sync/export wiring. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>v0.234.1 |
||
|
|
6514673fde |
perf: Cloud Run burst headroom — concurrency 32 + min-instances 2 (#1035)
## Summary Final tuning from the 16-tab test diagnosis (#1033 fixed the edge rate limit eating the requests; this fixes the remaining latency). With nothing failing anymore, the warm 16-tab run still showed every tab at ~17–19s — server logs pinpointed why: the one warm instance answered `correction-data` in **1.2s** while the three instances the burst spawned answered in **15.8–18.8s** — Cloud Run holds requests during the ~16s app boot. Concurrency 16 (set in #1028 while the event loop still had blocking hazards) now overshoots: the loop is clean, one instance juggles dozens of light requests fine, and forcing scale-out per burst just converts load into cold-start waits. ## Change `--concurrency 16 → 32` and `--min-instances 1 → 2`: 64 warm request slots, so a 16-tab reload (~48 concurrent API calls) is absorbed with zero cold starts. Still far below the original 80 that allowed the 2026-09-18 single-instance wedge. ## 💰 Cost note (Andrew) The second min-instance idles at Cloud Run idle rates — roughly **$15/month**. Single-flag revert if you'd rather not pay it; without it the first burst after idle pays one ~16s cold-start wave for the overflow beyond 32 concurrent slots. ## Follow-up worth considering (not in this PR) App cold start is ~16s — heavy import graph (matplotlib, pydub, GCP clients). Lazy-importing the heavy tails would benefit every scale-out. ## Testing Deploy-flag-only change. Post-deploy: re-run `scripts/load-test-review.py --tabs 16` — expect single-digit-second tabs with no cold-start waves. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
438efb751d |
feat: audition instrumental stems while the preview video is still encoding (#1034)
## Summary
In the lyrics-review **Preview Video** modal, the pink backing-vocals
waveform said "click to hear them" — but while the preview was still
encoding (regularly 30s–3min on cold encoder starts), clicking silently
did nothing: the hidden stem `<audio>` only mounted alongside the ready
`<video>`, and all stem playback was slaved to video events.
The stem URLs and waveform amplitudes are already available the moment
the modal opens, so this makes the instrumental(+backing) audio playable
immediately:
- **Mount the hidden stem player in every preview state** (not just
`ready`), so it can play standalone
- **Clicking the pink waveform while encoding seeks and plays the stem
directly**; the stem's `timeupdate` drives the waveform playhead
- **"Stop audio" control** appears under the encoding spinner while a
stem is audibly playing
- **Seamless handoff**: when encoding completes mid-audition, the video
starts from the stem's current position (still in instrumental mode)
instead of jumping back to 0
- Radio choices ("Instrumental + backing vocals" / "Clean instrumental")
still audition-switch the stem while encoding, but don't start playback
unprompted
- Standalone stem is silenced if the encode errors; audio mode resets
only when a fresh generation starts
## i18n
Two new strings (`playingWhileEncoding`, `stopAudio`) translated to all
33 locales.
## Testing
- 6 new Jest cases (standalone audition, playhead forwarding, stop
control, stem swap mid-encode, radio no-autoplay, encode-complete
handoff) with an HTMLMediaElement play/pause emulation scoped to the new
describe block
- Full frontend suite: 106 suites / 1304 tests passing
- ESLint clean on changed files; `react-hooks/exhaustive-deps` warning
fixed by memoizing `options`
- No backend changes
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
v0.234.0
|
||
|
|
925ec79c56 |
fix: 16-tab burst tripped the Cloudflare rate limit — raise + scope it, survive edge blocks gracefully (#1033)
## Summary Root-cause fix for Andrew's failed 16-tab reload test (2026-09-22 21:37Z), which the new degradation telemetry captured end-to-end (`client_events`: banner_reconnecting ×6, banner_unavailable ×2 with failed probes, waveform_slow with peaks-fallback). **Diagnosis:** the origin was completely healthy during the test — zero backend errors, 3 instances serving — but only ~207 API requests reached it (16 tabs should have sent 2–3× that; only 9 `correction-data` arrived). The missing traffic was blocked by **our own Cloudflare rate limit (50 req/10s/IP)**: 16 tabs × ~10 API calls + CORS preflights ≈ hundreds of edge requests in seconds. CF block pages have no CORS headers, so cross-origin fetches fail opaquely — producing "Access denied / Failed to load job details" (blocked job fetch), "temporarily unavailable" (blocked correction-data, 3 fast retries all inside the 10s block), and outage banners (the `/api/health` probes were themselves rate-limited, falsely "confirming" an outage). ## Changes **Edge (Pulumi `infrastructure/`)** - `RATE_LIMIT_REQUESTS` 50 → **200 per 10s/IP** — still flood control (20 req/s sustained blocks), but a legitimate many-tab power user passes. - Rate-limit expression now **excludes OPTIONS preflights** (double-counting; a blocked preflight kills the real request) and **`/api/health`** (the frontend's connectivity probe must reflect origin health — never edge-block state — or every tab shows the outage banner for a healthy origin). **Frontend** - `429` treated as transient; retry honors `Retry-After` or waits out the full 10s mitigation window (fast retries fail *and* prolong the block). - Network-failure retry backoff 0.6s/1.5s → **2s/9s**, so the final attempt can land after a 10s block expires (blocked cross-origin requests surface as network errors). - Review-page access check: `BackendUnavailableError` now shows **"Trouble reaching our servers" + Try again** instead of the misleading **"Access denied"** — unreachability is not an authorization verdict. ## Testing - Frontend: 2 new 429 tests (waits out window; honors Retry-After), network-backoff tests updated to the new schedule with probe-interference handled — **1309 passed**. Backend suite (untouched code): **4518 passed**. - Post-merge: Pulumi applies via CI (local `pulumi up` unavailable to the agent — read-only ADC can't write Pulumi state; flagged below). Will verify the ruleset values via the Cloudflare API afterwards and re-run the review load test. ## ⚠️ Deploy note `infrastructure/` changes normally get a local `pulumi up` before merge; the agent's read-only ADC can't write Pulumi state, so this relies on the CI apply on merge (idempotent single-ruleset update — low partial-state risk). Andrew: if you prefer, run `pulumi up` locally before merging. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>v0.233.0 |
||
|
|
758d74f44e |
feat: divebar-lookup 'search' action shares kn_search matching — token-AND, accent-fold, typo-tolerant (v0.232.1) (#1030)
## Summary Third quick win from the kjbox search-unification research (workspace `docs/archive/2026-09-19-kjbox-search-unification-handoff.md`, finding F1): the divebar-lookup CF's `search` action (GCS-mirror rows + kjbox loose-CDG sibling pairing) matched by **whole-string folded LIKE**, while `kn_search`/`kn_community_search` use token-AND with accent folding and typo tolerance. Live-proven divergence: in one rotation search, `"books maximo park"` (word-order swap) returned the KN rows but **0** mirror rows; typos likewise dropped only the mirror rows. `_search_divebar` now shares `_kn_match_parts` with the KN searches: token-AND over the diacritic-folded artist+title haystack, per-word `EDIT_DISTANCE` typo tolerance (length-scaled), STRPOS literal substring (no LIKE escaping). Token-AND is strictly more permissive than the old whole-string substring, so no previously-matching query regresses — this matters for kjbox's `find_sibling_audio`, which re-runs this search on approval. Also adds the same `maximum_bytes_billed` cap the KN searches have (public endpoint, CWE-400). ## Validation Ran the real generated SQL against live BigQuery (read-only ADC, CF imported with stubbed `functions_framework`): | Query | Before | After | |---|---|---| | `books maximo park` (order swap) | 0 rows | 3 rows (Maxïmo Park — Books from Boxes) | | `maximo park boks` (typo) | 0 rows | 3 rows | | `jose feliciano feliz navidad` (accent case, gen#1026) | 2 rows | 2 rows (SDK cdg+mp3 — sibling pair intact) | CF unit tests: 41 passed (TestDivebarSearch rewritten for the shared engine: token-AND per-token conditions, EDIT_DISTANCE, folded params, blank-query short-circuit, bytes-billed cap). ## Deploy Infra-only (`infrastructure/functions/divebar_lookup/`). Applied locally via `pulumi up` (stack prod, 2 changes / 298 unchanged) before merge; deployed action smoke-tested live (order-swap 3 rows, typo 3 rows, accent case 2 rows). 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved Divebar search with accent-insensitive, typo-tolerant, and order-independent matching. * Searches now require all usable terms to match for more relevant results. * Empty or whitespace-only searches return no results without querying. * Added a 1 GB limit on billed search data. * Limited excessively long search terms while preserving matching behavior. * **Chores** * Updated the application version to 0.232.1. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>v0.232.1 |
||
|
|
d85b6e271c |
feat: precomputed vocals peaks — Waveforms strips paint sub-second (#1031)
## Summary Final piece of the review-page speed work (#1024 → #1028 → #1029): the Waveforms-mode strips no longer require downloading + WebAudio-decoding the entire vocals stem (~7.5MB, ~4–6s) — the backend now serves a **precomputed 400-peaks/s envelope as ~150KB of uint8/base64 JSON**, cached in GCS and pre-warmed at prep time, so strips paint sub-second. ## How - **`WaveformGenerator.generate_peaks()`** — numpy max-abs envelope, bucket-for-bucket identical to the frontend's `computePeaks` (same drift-free fractional-stride bucketing; exact match vs a naive reference in tests). - **`AudioAnalysisService.get_vocals_peaks()`** — GCS cache at `review-audio/vocals_peaks_400.json`, validated against the stem path; miss decodes the small transcoded OGG; upload best-effort. - **`GET /api/review/{job_id}/vocals-peaks`** — off the event loop under the shared waveform semaphore; `202 Retry-After: 15` while separation runs (mirrors `/audio/vocals`); `Cache-Control: private, max-age=600`; `peaks_per_second` clamped 50–1000. - **`screens_worker`** pre-computes the envelope next to the existing OGG + waveform pre-warm, so it's a pure cache read by the time a job reaches review. - **Frontend** — `VocalsAudioDataLoader` prefers the peaks endpoint (one quick retry on transient errors) and **falls back to the original full download+decode** on any terminal peaks failure, so a peaks-side problem can never cost a user their waveforms. Local mode (no peaks URL) uses the old path unchanged. Vocal stem selection factored into `backend/utils/stems.py` (shared by routes + worker). ## Testing - Package: 4 tests incl. exact naive-reference parity (**106 passed**) - Backend: 4 service + 4 endpoint tests (**4518 passed**) - Frontend: `fetchVocalsPeaks` units + 5 loader-behavior tests (prefers peaks / falls back / one-retry-then-fallback / 202 polling / audio-only mode) (**1307 passed**) - `review-fast-load.spec.ts` prod E2E extended with a cached vocals-peaks latency/shape check - Post-deploy: will verify the endpoint on a real in-review job and confirm payload size + latency ## Notes - Existing in-review jobs have no cache yet — their first Waveforms open computes + caches lazily (bounded); every open after that is sub-second. New jobs are pre-warmed. - 8-bit amplitude quantization is visually lossless at strip heights (14px). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>v0.232.0 |
||
|
|
7cb5de729a |
fix: encoding-worker warmup/heartbeat off the event loop (11s instance freeze on review open) (#1029)
## Summary Follow-up to the concurrent-load reliability work (#1024, #1028), triggered by Andrew observing a ~5s "Loading audio…" stall opening job `7a16be60`'s review page. **Diagnosis from prod request logs (2026-09-20 00:25Z):** correction-data returned in 0.57s, then the page's parallel batch — `audio/vocals` (7.5MB), playback audio 302, `sessions`, and the encoding-worker `warmup` + `heartbeat` — ALL took a uniform ~12s on the same warm instance, with an **11-second gap containing zero log lines** (00:25:38.05 → 00:25:49.13). The last logger before the gap and the first after it: `encoding_worker_manager`. **Root cause:** `POST /api/internal/encoding-worker/warmup/{job_id}` (fired by every lyrics-review page open, per its JIT pre-warm design) called `manager.ensure_primary_running()` — Firestore reads + GCE compute `instances.get`/`instances.start` (10s+ when the VM is cold/slow) — **synchronously inside an async handler**. On the single-process/single-event-loop server this froze the whole instance, stalling the very page load that fired it. Same bug class as the v0.230.0 review-path fixes, one router over. ## Changes - `warmup_encoding_worker` and `heartbeat_encoding_worker` run their manager calls via `asyncio.to_thread` (behavior otherwise unchanged, including the soft-fail on `EncodingWorkerStartError`). - New `backend/tests/test_encoding_worker_routes.py` — success/soft-fail/heartbeat passthrough plus a ticker-based test proving the event loop keeps running during a slow warmup (would fail on the old code). ## Testing Full backend suite: **4510 passed**. Post-deploy I'll re-check a review-page open against a cold encoding VM in the request logs (the audio requests should no longer correlate with warmup latency). 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Bug Fixes** - Improved encoding-worker startup and heartbeat responsiveness by preventing slow operations from blocking other asynchronous activity. - Encoding-worker startup failures now return a clear soft-failure response while allowing the service to continue operating. - **Maintenance** - Updated the package version to 0.231.1. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>v0.231.1 |
||
|
|
30cf41e7d6 |
feat: concurrent-reliability phases 2-5 — Cloud Run shape, degradation telemetry, banner tuning, load-test guard (#1028)
## Summary Phases 2–5 of the concurrent-load reliability initiative (Phase 1 = #1024, verified live: the 10-tab meltdown went from 9/30 failed requests to 30/30 OK). Verbatim spec: `docs/archive/2026-09-17-concurrent-review-reliability-verbatim-record.md`; plan: `docs/archive/2026-09-18-concurrent-review-reliability-investigation-and-plan.md`. ## P2 — Cloud Run shape - **`--concurrency 80 → 16`** for karaoke-backend: the service is one uvicorn process per 2-vCPU instance, and review loads are heavy; at 80 a burst never triggers scale-out and all lands on the single min-instance. 16 makes bursts fan out while staying well above normal per-instance load. (Deploy flags live in ci.yml — the service is not Pulumi-managed.) - **Local preview ffmpeg fallback bounded** (`LOCAL_PREVIEW_MAX_CONCURRENCY=1`, 900s queue timeout, fast RuntimeError after that): the GCE-outage fallback runs ffmpeg ON the API instance; unbounded, a few concurrent Preview clicks during an encoder outage pegged both vCPUs, stalled the health probe, and showed everyone the orange banner (Andrew's banner-during-preview screenshot). ## P3 — degradation telemetry (the explicit ask) - New **`POST /api/client-events`** (unauthenticated so review-token users report too; 30/min/IP rate limit; bot-filtered; detail capped; URLs query-stripped). Each event → Firestore `client_events` collection **and** a structured `client_event type=…` log line, so "how often are users seeing this?" is reviewable via Firestore aggregation or Cloud Logging correlation. - Frontend reporter `lib/degradation-events.ts` (60s per-type throttle, localhost no-op, never throws), wired into: - `BackendStatusBanner` → `banner_reconnecting` / `banner_unavailable` with stall age, in-flight count, probe state - review lyrics-load failure → `lyrics_load_failed` with message/status - `VocalsAudioDataLoader` → `waveform_slow` (>20s to first strips) and `waveform_failed` (gave up) with poll/retry counts ## P4 — banner tuning - Escalating to the full "We're having trouble reaching our servers" card now requires **two consecutive failed health probes** (`UNAVAILABLE_PROBE_FAILURES=2`). One failed 4s probe against a briefly-pegged instance shows only the calm "Reconnecting…" pill. Single-failure behavior for the pill is unchanged, as is instant clearing on recovery. ## P5 — regression guard - **`scripts/load-test-review.py`** checked in: replays the 10-tab incident (Firestore-ADC review tokens, curl request trios, pass/fail thresholds, exit code). Documented in `docs/TESTING.md`. Validated against prod post-Phase-1: 9/9 requests OK, tabs < 3s. ## Testing - Backend: 6 new tests for client-events (persist/validate/rate-limit/bot/fail-soft/detail-cap) + local-preview bound test — full suite **4517 passed**. - Frontend: new degradation-events suite, two-probe escalation + debug-snapshot tests; one existing api-fetch test updated to the new escalation contract — full suite **1298 passed**. - Live validation of the load-test tool against prod (small 3-tab run): PASS. ## Notes for review - Cloud Run concurrency change means bursts now spin up extra instances (billable during requests only; max-instances unchanged at 20). - Firestore `client_events` has no TTL policy yet (needs a gcloud/console action our read-only ADC can't do); volume is tiny (throttled per client) — can add later if it ever matters. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added automatic reporting for degraded experiences, including reconnecting or unavailable backend status, failed lyrics loads, and slow or failed waveform loads. - Added concurrent review-page load testing to help verify performance under multiple open tabs. - **Improvements** - Backend status now remains in “reconnecting” until multiple health checks fail, reducing premature unavailable messages. - Local preview processing is limited during heavy demand to improve stability. - **Documentation** - Added API documentation for client telemetry and guidance for running concurrent-load regression tests. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>v0.231.0 |
||
|
|
e0f3d1bb45 |
feat(divebar-lookup): combined kn_search action — community + full KN catalogs in one query (#1025)
## Summary The kjbox "Search Karaoke Nerds" panel went community-only in the scrape removal (kjbox #214 / `kn_community_search`), so songs that only exist on commercial disc brands — e.g. **Jason Aldean – Big Green Tractor** (13 commercial brands, zero community versions) — now show "No results found" even though karaokenerds.com lists them. ## Changes - New CF action **`kn_search`**: searches BOTH our KN catalog copies in a **single BigQuery job** (`UNION ALL` + per-source `QUALIFY ROW_NUMBER()` limit so the ~300k-row `karaokenerds_raw` table can't starve community rows). - `community` rows: `{artist, title, brand, watch}` (unchanged shape, playable) - `full` rows: `{artist, title, brands}` — comma-separated brand codes, no URLs (disc-only releases) - Matching logic (token-AND, accent-fold, typo `EDIT_DISTANCE`) extracted to `_kn_match_parts`, shared with the **unchanged** `kn_community_search` action. - Same 1 GB `maximum_bytes_billed` cost cap as the existing action (public endpoint). ## Testing - `test_main.py`: new `TestKnSearch` class (dispatch validation, per-source split, shared matching semantics, no-BQ-on-blank-query). **35 passed.** - Verified the real generated SQL against live BigQuery: `Big Green Tractor Jason Aldean` (and the `tracter` typo) → 1 full row with all 13 brands; Tribute / Maxïmo Park return both community + full rows. ## Deploy Pulumi packages `functions/divebar_lookup/` via content-hashed `FileArchive` — `pulumi up` from `infrastructure/` redeploys the CF. Per repo rules: **run `pulumi up` locally before merging.** The paired kjbox PR nomadkaraoke/kjbox#217 (v0.102.0) depends on this action being live. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added a unified KaraokeNerds search that searches community and full catalogs together. - Results are separated into community and full-catalog sections, with relevant details such as artist, title, brand, and available video links. - Search supports accent-insensitive, typo-tolerant, multi-word matching. - Added validation for search queries and result limits, with limits capped at 200 results. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>v0.230.3 |
||
|
|
01440a1ec0 |
fix: retry flacfetch search on transient network errors (v0.230.2) (#1027)
## Summary Fixes the 2026-09-18 15:21 UTC error-monitor alert: `Remote flacfetch search failed (not falling back to local): Search request failed: ReadTimeout for POST https://flacfetch.nomadkaraoke.com/search` — a single ReadTimeout surfaced as a user-facing 500 on `/api/audio-search/search-standalone`. The flacfetch server was busy with an uncached ~40s provider sweep (its event loop blocks during sweeps — fixed separately in nomadkaraoke/flacfetch#50), the 60s client timeout fired, and `search()` — unlike `download()`/`download_by_id()` — had **no retry at all**. The user's manual retry succeeded in 11s via the now-warm server cache; an automatic retry would have hidden the incident entirely. ## Changes - `with_retry()` accepts keyword-only `retry_max_attempts`/`retry_min_wait`/`retry_max_wait` overrides (defaults unchanged). - `search()` now retries via `with_retry` with a **light profile**: 2 attempts, 1s wait (configurable via `FLACFETCH_SEARCH_RETRY_MAX_ATTEMPTS`/`FLACFETCH_SEARCH_RETRY_WAIT`) — deliberately lighter than the download profile (9 attempts, 10–60s backoff) because search sits on a user-facing path behind Cloudflare's ~100s edge limit. Worst case: 60s timeout + 1s + fast cache-hit retry. - Still **no fallback to local search** on failure — preserves the fallback-audit 2026-06-09 decision; failure stays loud. ## Testing - New `TestFlacfetchSearchRetryLogic`: retries on ReadTimeout then succeeds; fails after 2 attempts using the search profile (not the download profile's 9); no retry on 4xx; 404 still returns empty results without retry. - `backend/tests/test_flacfetch_client.py` + `test_audio_search.py`: 123 passed. Full `make test` running. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved search reliability by retrying temporary failures with a shorter, search-specific retry policy. * Search requests now enforce an overall timeout, preventing prolonged waits and returning a clear service error when the limit is reached. * Search requests fail faster when repeated attempts are unsuccessful. * 404 responses continue to return empty results without unnecessary retries. * Other client errors are no longer retried. * **Chores** * Updated the project version to 0.230.2. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>v0.230.2 |
||
|
|
54edd0a8f9 |
fix: accent-insensitive divebar catalog search (José Feliciano etc.) (#1026)
## Bug Tonight (live show) approving a kjbox sing request for the Sandell loose-CDG **"Feliz Navidad — Jose Feliciano"** failed with *"This CDG has no audio track available in the mirror — pick another version"* even though the sibling `(SDK) José Feliciano - Feliz Navidad.mp3` exists (both files mirrored in GCS). Root cause here: the divebar-lookup CF `search` action matches by plain `LIKE` over `LOWER(artist || ' ' || title)`, so the KN-normalized, accent-free query `"Jose Feliciano Feliz Navidad"` can never match the accented catalog row `"José Feliciano"`. kjbox's loose-CDG sibling pairing re-searches with exactly that text, got zero rows, and correctly failed closed. Verified against the live CF: `"Jose Feliciano Feliz Navidad"` → 0 results; `"Feliz Navidad"` → both SDK rows. ## Fix Diacritic-fold both sides of the match (NFD + strip combining marks + lower) — the query in Python, the haystack in SQL — mirroring exactly what `_search_kn_community` in the same CF already does since #1000. Symmetric, so accented queries also match unaccented rows. Companion PR (defense-in-depth, kjbox side): nomadkaraoke/kjbox#216 makes the sibling pairing fall back to title-only / artist-only queries. ## Testing - New `TestDivebarSearch` unit tests: haystack SQL contains the fold expression; accented query is folded to `%jose feliciano feliz navidad%` before matching. - Full `test_main.py` suite: 30 passed. ## Deploy note Infrastructure change (Cloud Function source) — per repo convention, apply Pulumi locally before merge. My ADC is read-only so I have NOT run `pulumi up`; Andrew, run it locally (or accept the CI apply on merge). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>v0.230.1 |
||
|
|
b336f36f4e |
perf: survive concurrent review-page loads (10-tab meltdown fix, Phase 1) (#1024)
## Summary
Andrew can reliably take the backend down by opening ~10 lyrics-review
tabs at once: 1–2 load, the rest show "We're having trouble reaching our
servers" / "The server is temporarily unavailable". This PR is Phase 1
of the systematic fix (investigation + full phased plan in
`docs/archive/2026-09-18-concurrent-review-reliability-investigation-and-plan.md`;
Andrew's verbatim spec in
`docs/archive/2026-09-17-concurrent-review-reliability-verbatim-record.md`).
**Reproduced before the fix (prod, 2026-09-18 03:35Z):** 10 simulated
tabs → 8/30 requests HTTP 500, survivors took 44–72s (vs 0.8–1.3s solo),
Cloud Run logged 9× "request aborted — no available instance". Root
cause: `/waveform-data` decodes a whole stem (pydub→ffmpeg + Python RMS
loop) **synchronously on the single uvicorn event loop, uncached**,
while every other review endpoint also does blocking Firestore/GCS on
that loop — one wedged instance takes all traffic (concurrency=80 → no
scale-out at 10 tabs).
## Changes
**Backend**
- `/waveform-data`: persistent GCS cache
(`jobs/{id}/review-audio/waveform_review_{n}.json`, validated against
source path + num_points), computed via `asyncio.to_thread` under a
bounded semaphore (`REVIEW_WAVEFORM_MAX_CONCURRENCY`, default 4). Cache
miss decodes the ~3MB transcoded OGG instead of the ~35MB raw FLAC.
Pre-computed in `screens_worker` (next to the existing OGG pre-warm) so
review opens are pure cache reads.
- Blocking Firestore/GCS calls (`get_job`, `file_exists`,
`download_json`, `transition_to_state`) moved off the event loop via
`asyncio.to_thread` in the review hot-path endpoints (correction-data,
waveform-data, audio proxies, instrumental-urls/-analysis).
- `/audio/vocals`: now served from the bounded LRU byte cache
(previously a fresh multi-MB GCS download per request), honors HTTP
Range (206), and ranged review-audio responses carry `Cache-Control:
private, max-age=600`.
**Frontend**
- `VocalsAudioDataLoader`: transient failures (5xx/429/network) retry
with backoff `[5s, 15s, 30s, 60s]` instead of silently giving up — the
"Waveforms mode shows no waveforms until manual reload" symptom. 404
stays terminal via new typed `AudioFetchError`; 202 polling unchanged.
## Testing
- 12 new backend tests (waveform cache
hit/miss/stale-source/num_points/OGG-preference/upload-failure; vocals
LRU + Range 206 + Cache-Control; waveform-data endpoint routing) — full
backend suite **4500 passed**.
- 3 new frontend tests (transient retry → load, retry exhaustion, 404
terminal) — full Jest suite **1289 passed** across 105 suites.
- Post-deploy: will re-run the 10-tab burst repro against prod and
record before/after numbers in the plan doc.
## Follow-ups (later phases, tracked in the plan doc + BACKLOG)
Cloud Run shape (lower concurrency so bursts scale out),
`/api/client-events` degradation telemetry (Andrew's explicit ask),
banner threshold tuning, `instrumental-analysis` still signs 4 URLs
(13.7s baseline), load-test regression guard.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Review waveforms load faster with cached waveform data and improved
concurrent-load handling.
* Audio playback supports partial-range requests and short-term caching
for smoother streaming.
* Transient audio failures are automatically retried up to four times.
* **Bug Fixes**
* Reduced review-page failures and delays during simultaneous loads.
* Waveform generation falls back to available audio when backing vocals
are unavailable.
* **Documentation**
* Added API documentation for waveform data and updated audio streaming
behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
v0.230.0
|
||
|
|
527d55ef63 |
fix: preserve source audio extension for temp downloads (ffmpeg extension-fallback failure) (#1023)
## Summary
Root-cause fix for the 2026-09-17 19:47 ET error-pattern cluster (job
`e026e727`): a paying user's uploaded MP3 had a ~1.2 MB ID3 tag
(embedded cover art from a ytmp3 rip). The backend downloads job audio
to temp files hardcoded as `*.flac` — ffmpeg's content probe is
inconclusive inside the huge tag, so it falls back to the **file
extension**, commits to the flac demuxer, and fails with `Could not find
codec parameters for stream 0 (Audio: flac, 0 channels)`. That single
file broke every consumer of the input audio:
- `GET /api/review/{id}/waveform-data` → 500 ("Decoding failed. ffmpeg
returned error code: 8")
- Review OGG transcode → "Cannot determine format of input 0:0 after
EOF" (fell back to raw FLAC serving)
- Preview video encode → failed on **both** GCE worker and local
fallback ("Invalid data found when processing input")
- Duration probe → `Failed to get audio duration: 'duration'` (KeyError)
**Repro:** the file is a perfectly valid 227 s MP3 — `ffprobe` succeeds
when the copy is named `.mp3` or has no extension, and fails when named
`.flac` (verified on ffmpeg 8.0 locally and 7.0.2-static in prod). The
`majorbrand: dash` metadata is a copied ID3 tag, not a mislabeled MP4.
## Changes
- New `backend/utils/audio_filenames.py` —
`local_audio_filename(gcs_path, stem)` names local copies with the
source object's real (lowercased) extension; unknown/missing extension →
bare name, forcing pure content probing.
- Applied at every site that downloads job audio to a temp file:
- `backend/services/audio_analysis_service.py` (waveform data/image,
analysis, mute-region waveform, stem comparison)
- `backend/services/audio_transcoding_service.py` (review OGG transcode)
- `backend/api/routes/review.py` `_prepare_preview_inputs` (preview
local encode)
- `backend/services/gce_encoding/main.py` (GCE preview + render audio
download)
- `backend/workers/render_video_worker.py`,
`backend/workers/audio_worker.py`, `backend/workers/lyrics_worker.py`
- `docs/TROUBLESHOOTING.md`: new runbook entry for this failure class.
- Version bump 0.229.3 → 0.229.4.
## Testing
- New `tests/unit/test_audio_filenames.py` — 9 tests: helper behavior
(extension preservation, lowercasing, unknown/missing-extension
fallback) + service-level assertions that the transcoding and analysis
services download to source-matching temp names.
- Full `make test` suite passes locally (backend + frontend, 105 suites
/ 1286 frontend tests).
## Impact / rollout
- No data repair needed for job `e026e727` — its stems/lyrics are
intact; the review page starts working once this deploys.
- `gce_encoding/main.py` ships in the wheel; the encoding worker's
per-job wheel version check picks it up (restart `encoding-worker`
service to force immediate pickup).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Improved audio processing for files with non-FLAC extensions,
including MP3 and OGG files.
- Preview rendering, waveform generation, transcription, and audio
conversion now preserve source file extensions, reducing decoding
failures.
- Audio conversion fallbacks now use the correct source content type.
- **Documentation**
- Added troubleshooting guidance for waveform and preview failures
related to audio metadata or file extensions.
- **Tests**
- Added coverage for extension handling across audio download and
processing workflows.
- **Release**
- Updated the application version to 0.229.4.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
v0.229.4
|
||
|
|
f997cd7de2 |
docs: session record — review fast-full-load (#1018) + E2E canary regression fix (#1022)
Session record for 2026-09-17: the review fast-full-load re-architecture (#1018, v0.229.0) and the diagnosis/verification of the post-deploy canary regression (#1019 → fixed by #1020). Docs-only. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |