@coderabbitai ignore
Follow-up to #828. That PR staged the instrumental in `_encode_via_gce`
(legacy `video_worker.py`), but production jobs run the **orchestrator**
pipeline (`video_worker_orchestrator.py`), whose `_run_encoding` builds
its own `EncodingInput` and never made the user-supplied instrumental
available under `jobs/{job_id}/`. Tenant jobs still failed at the GCE
encoder with "No instrumental audio found" (confirmed — the "Staged
existing instrumental" log never appeared).
**Fix:** in the orchestrator's `_run_encoding`, when using the GCE
backend, upload the local instrumental (existing/custom — identified by
the `(Instrumental User)` / `(Instrumental Custom)` filename) into
`jobs/{job_id}/` so the encoder's recursive `find_file` locates it.
Consumer `clean`/`backing` jobs are untouched (their stems already live
under `jobs/{job_id}/`).
Also: the daily E2E uses a tenant-specific title so the two matrix
tenants don't share a `base_name` (the GCE encoder caches by base_name).
Validated by re-running the full prod E2E to completion after deploy.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@coderabbitai ignore
## Final render blocker (bug #8)
With #824/#825/#826/#827, tenant jobs now render the karaoke video — but
the GCE encoder failed with **"No instrumental audio found"**.
**Root cause:** the user-supplied instrumental lives under
`uploads/{job}/audio/`, but the encoder only pulls input from
`jobs/{job}/` (`input_gcs_path`) and locates the instrumental via
`find_file("*existing_instrumental*", "*Instrumental User*")`. On top of
that, the running encoder VM (`encoding-worker-b`) boots a **2026-01-23
image** that predates the encoder's own `existing_instrumental`
config-download path (added 2026-03-04), so it could never fetch it from
`uploads/` either.
**Fix (image-agnostic — no risky shared-VM recreation, no consumer
impact):** `_encode_via_gce` server-side copies the instrumental into
`jobs/{job_id}/existing_instrumental.<ext>` before submitting. The
encoder downloads `jobs/{job}/` and its **recursive** `find_file` then
locates it on any image version. Added `StorageService.copy_blob` + unit
test.
> Note: the stale encoder VM image is a separate infra item — the
`encoding-worker` VMs should be recreated from the image family's newest
image (which has 3 months of encoder fixes). Tracked for a Pulumi pass;
not required for this fix.
## Daily tenant E2E (second ask)
- `scripts/e2e/tenant_e2e.py` — full per-tenant prod health check:
submit → approve review → render → assert (completes, Dropbox link
present, no YouTube, downloads available) → clean up job + uploads.
- `.github/workflows/e2e-tenant-daily.yml` — daily matrix (vocalstar,
singa), WIF auth, Discord + email on failure.
- `docs/archive/2026-06-15-tenant-portals-e2e.md` — the full story (8
bugs, all fixed).
## Testing
- `copy_blob` unit test; storage suite green (20).
- Full tenant E2E re-run to completion after deploy.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@coderabbitai ignore
## Two final render blockers (found by the Vocal Star E2E)
With attribution (#824), the audio slot (#825), and transcription
padding (#826) fixed, the first real tenant job advanced through
transcription → screens → review → render, surfacing the last two
never-exercised issues:
### 1. Render prerequisites — "No instrumental selected"
existing-instrumental jobs (every tenant submission) have no
instrumental-selection UI step, so `instrumental_selection` was never
written to state_data. The video orchestrator then defaults to `'clean'`
(a separated stem that was never produced for these jobs), and
`_validate_prerequisites` rejects the job, leaving it stuck retrying.
**Fix:** `complete-review` now defaults `instrumental_selection` to
`'custom'` (use the provided instrumental) when
`existing_instrumental_gcs_path` is set. An explicit request value still
wins. The render path already supports `'custom'`.
### 2. Theme asset paths (reproducibility)
The tenant setup scripts wrote full `gs://bucket/themes/.../assets/X`
URLs, but `theme_service._build_style_assets_mapping` only resolves bare
basenames (the consumer convention). With `gs://` URLs, `style_assets`
came back empty → assets never downloaded → encoder failed with "Video
background image not found". Scripts now emit basenames, and add the
newly-required `extra_text_text_transform` field the strict theme
validator expects.
(The two live GCS themes — vocalstar, singa — were corrected out-of-band
to match.)
## Testing
- New regression tests: complete-review defaults to `custom` for
existing-instrumental jobs; explicit selection still wins.
- Review suites green (102). Full E2E re-run to completion after deploy.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@coderabbitai ignore
## Problem
The white-label tenant portals (`vocalstar.nomadkaraoke.com`,
`singa.nomadkaraoke.com`) looked "done" but **had never worked
end-to-end**: 0 of the last 484 production jobs carried a `tenant_id`.
No tenant job had ever run.
**Root cause:** the frontend API client never sent the `X-Tenant-ID`
header. Tenant portals call the shared `api.nomadkaraoke.com`, whose
Host isn't the tenant subdomain, and the `?tenant=` query param is
disabled in prod — so the backend could never detect the tenant. Every
tenant sign-in and job submission silently fell back to consumer
behavior (default theme, no tenant Dropbox, etc.). The
`getTenantHeaders()` helper existed and was unit-tested, but was never
wired into requests.
## Changes
**Frontend** (`lib/api.ts`, `lib/tenant.ts`)
- Attach `X-Tenant-ID` on all authenticated requests **and** the
magic-link request, sourced from `window.__TENANT_CONFIG__`
(edge-injected by the Cloudflare function; mirrored from the tenant
store for dev/preview). Consumer build resolves no tenant → no header →
**behavior unchanged**.
**Backend** (`_apply_tenant_overrides` in `file_upload.py`)
- Make tenant config **authoritative** over global distribution
defaults. Prod sets `DEFAULT_DROPBOX_PATH` / `DEFAULT_GDRIVE_FOLDER_ID`
/ `DEFAULT_BRAND_PREFIX` globally; these seed `dist`, and the old "fill
only when unset" overlay let them leak — so every tenant job would have
landed in the **shared consumer Dropbox folder** and uploaded to the
**global Google Drive**. Now Dropbox/GDrive routing is gated by the
tenant feature flags and cleared when disabled.
**Tenant configs** (GCS + setup scripts, Vocal Star + Singa)
- Disable Google Drive (`gdrive_upload=false`, `gdrive_folder_id=null`),
keep YouTube off, enable Dropbox to each tenant's own folder. B2B =
locked theme, no YouTube/GDrive, Dropbox + portal download only.
## Why it's consumer-safe
- Frontend: header only attached when a tenant is resolved (never on
`gen.nomadkaraoke.com`).
- Backend: `_apply_tenant_overrides` early-returns when `tenant_config
is None` (all consumer jobs).
## Testing
- New: frontend header-attachment tests (tenant build vs consumer);
backend authoritative-override + feature-flag-gating regression guards
(incl. the global-default-leak case).
- Full suites green: **3466 backend**, **1013 frontend**.
- Verified live tenant configs via API after update.
Manual prod E2E (real track through each portal) + a daily per-tenant
E2E workflow follow in a second PR, once this deploys and is verified in
prod.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
Claude **Fable 5** (`claude-fable-5`) is no longer available on the
Anthropic API. Verified live against `api.anthropic.com`:
```
404 not_found_error: "Claude Fable 5 is not available. Please use Opus 4.8.
Learn more: https://www.anthropic.com/news/fable-mythos-access"
```
`claude-opus-4-8` responds normally and is the API's own prescribed
replacement. Opus 4.8 was also ≈ Gemini 3.1 Pro in our 30-job offline
eval, so switching to it preserves the two-model compare quality.
## Changes
- **`.github/workflows/ci.yml`** (load-bearing):
`AUTO_CORRECT_MODEL=claude-opus-4-8` and
`AUTO_CORRECT_COMPARE_MODELS=claude-opus-4-8;gemini-3.1-pro-preview`.
Compare mode now runs **Opus 4.8 + Gemini 3.1 Pro** instead of Fable +
Gemini.
- **`scripts/measure_autocorrect_cost.py`**: default models + docstring
example → Opus, so the cost tool measures what prod runs.
- **`docs/API.md`**: example auto-correct response reflects Opus.
- **`pyproject.toml`**: `0.182.1` → `0.182.2`.
## Re-enable path
Fable is **disabled, not deleted** — its pricing entry in `pricing.py`
and all Fable unit tests are intentionally left intact. If access
returns, flip it back by reverting the two env-var values in `ci.yml`.
## Testing
- Auto-correct suite: 73 passed (`backend/tests/services/auto_correct/`
+ `test_proactive_auto_correct.py`).
- Live API probe confirmed Fable 404s and Opus 4.8 works.
- CodeRabbit (uncommitted): 0 findings.
@coderabbitai ignore
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Summary
Makes AI lyrics auto-correction run **automatically on every job**
(multi-model), so suggestions are cached and ready the moment the
reviewer opens the lyrics UI — plus token/cost instrumentation and a UI
cleanup that removes all "AI"/"model" wording.
Measured cost first (`scripts/measure_autocorrect_cost.py`):
**~$0.13/track** multi-model on normal songs — cheap enough to run
proactively.
## Changes
**Cost instrumentation**
- Capture per-model `usage` from Anthropic + Gemini; log greppable
`auto-correct usage` / `auto-correct cost … total_est_cost_usd=…` lines
from `suggest()`.
- `backend/services/auto_correct/pricing.py` (Anthropic rates
authoritative; Gemini estimated) + `scripts/measure_autocorrect_cost.py`
offline harness.
**Proactive generation (every job)**
- `lyrics_worker._run_proactive_auto_correct()` runs the multi-model
path after `mark_lyrics_complete` (downstream already triggered →
non-blocking), gated by `AUTO_CORRECT_PROACTIVE_ENABLED`, bounded by a
180s timeout, and **swallows every error** — it never blocks or fails
the karaoke job (flaky AI / out of credits / timeout all degrade
gracefully).
- Cache-key aligned with the UI request, so the review page's on-load
call is an instant cache hit (no double spend).
- `ci.yml`: wires `AUTO_CORRECT_*` env + the `anthropic-api-key` secret
onto `lyrics-transcription-job` (same SA as the API service — no IAM
change).
**Review UI**
- Always multi-model (removed the model-selection toggle); button
renamed **"Auto-correct"**; stripped model/consensus/cached attribution
badges; reworded strings to drop "AI"/"model" (translated to all 32
locales).
- Auto-runs once on review load so suggestions are ready without a
click; the button stays for the paste-references-later flow.
## Testing
- Backend: **3460 passed** (`backend/tests/`, emulator excluded) — incl.
new `test_pricing.py` + `test_proactive_auto_correct.py` (gating,
graceful failure, cache alignment).
- Frontend: **1006 jest tests pass**; type-clean; 33-locale key parity
verified.
## Notes / follow-ups
- Proactive activates on the **next deploy** (the lyrics-job env/secret
apply via this PR's ci.yml on merge).
- Left "AI" wording in two *separate* features untouched (different
subsystems): `lyricsReview.agenticMetrics` and
`lyricsReview.modals.correctionAnnotation` — flagged for a separate
decision.
- The production E2E spec's click-to-open flow may want a tweak now that
suggestions auto-run on load.
@coderabbitai ignore
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
Three independent fixes bundled together:
1. **`revoke_session` idempotency** — stops the recurring `Error
revoking session` prod error on `karaoke-backend`. Firestore
`document.update()` raises `NotFound` (404) when a session doc is
already gone (logged out / expired / cleaned up); logout against an
already-absent session now succeeds instead of erroring.
2. **`query-error-patterns.py` crash** — `last_seen`/`first_seen` are
stored as ISO-8601 strings in Firestore, but the sort key and `--hours`
filter did `float(value)`, raising `ValueError` and breaking the
`--json` path that `/prod-investigate` relies on. Added a `_to_epoch()`
helper handling datetime objects, ISO strings, and numeric epochs.
3. **Mobile header overlap** — `AppHeader` was `position: fixed` and
each consumer cleared it with a hardcoded top padding. On narrow
viewports the header wraps to 3 rows (149px on a 412px Pixel), exceeding
the 112px offset, hiding the "Create Karaoke Video" heading and the top
of the form behind it (worse across locales with longer button labels).
Switched `AppHeader` to `position: sticky` (matching the already-working
jobs-page header) so content always clears it at any height, and removed
the now-redundant magic-number offsets.
## Changes
- `backend/services/user_service.py` — catch
`google_exceptions.NotFound` in `revoke_session`, return `True`
- `backend/tests/test_revoke_session.py` — new unit tests (existing /
missing / unexpected-error paths)
- `scripts/query-error-patterns.py` — `_to_epoch()` helper for sort +
`--hours` filter
- `frontend/components/app-header.tsx` — `fixed` → `sticky`
- `frontend/app/[locale]/app/page.tsx`,
`frontend/app/[locale]/app/referrals/page.tsx` — drop hardcoded top
padding
- `pyproject.toml` — bump to 0.174.17
## Testing
- [x] Backend tests pass locally (47 passed incl. 3 new `revoke_session`
tests)
- [x] `query-error-patterns.py --json` and `--hours` verified against
live Firestore
- [x] Mobile header fix verified in Playwright at 412px (Pixel) and
1280px — overlap eliminated (title moves from 12px behind the header to
a clear gap below it)
- [x] CodeRabbit CLI review: 0 findings
## Review
- [x] Local CodeRabbit review completed
- [x] Review feedback addressed (none)
@coderabbitai ignore
---
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
- Fixes the failing e2e-daily notification job ([run
#25983930960](https://github.com/nomadkaraoke/karaoke-gen/actions/runs/25983930960/job/76378786060))
which still used \`smtp.sendgrid.net\` — SendGrid was cancelled ~3 weeks
ago
- Drops the \`sendgrid\` Python dep, removes \`SendGridEmailProvider\`,
simplifies provider selection in \`email_service.py\`
- Removes \`SENDGRID_API_KEY\` env bindings from Pulumi
(gdrive-validator Cloud Function + video Cloud Run Job) and from
ci.yml's Cloud Run \`--set-secrets\`
- \`sendgrid-api-key\` Secret resource intentionally kept in
\`infrastructure/modules/secrets.py\` with a "delete manually" marker so
an unrelated \`pulumi up\` doesn't destroy the live secret + versions
- Version bumped 0.174.10 → 0.174.11
## Pulumi applied
\`pulumi up --target\` already run locally with the two affected
resources (video-encoding-job + gdrive-validator-function). PR #776's
runner-manager state intentionally untouched.
@coderabbitai ignore
## Test plan
- [x] Full backend test suite (3178 tests) passes locally
- [x] CodeRabbit local review: no findings
- [x] Pulumi up applied locally (clean)
- [ ] CI green on this PR
- [ ] After merge: e2e-daily.yml notification job should pass on next
scheduled run
- [ ] Verify gdrive-validator function still emails on next post-job
trigger
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
- **Bug**: `record_earning()` in `referral_service.py` compared naive
`datetime.utcnow()` against TZ-aware `earning_expires` (derived from
Firestore's TZ-aware `referred_at`), raising `TypeError: can't compare
offset-naive and offset-aware datetimes`. The exception was swallowed by
a broad `except` in the Stripe webhook handler and only logged as a
warning, so **every referral earning was silently lost** since the
referral system shipped (~3 weeks).
- **Symptom (user report)**: referral code `angel` showed 0 purchases on
the admin referrals page despite `tnt.travistoo@gmail.com` (a real user
referred by angel) paying $28 on 2026-04-27. Production logs at the
exact webhook moment confirmed: `Referral earning recording failed:
can't compare offset-naive and offset-aware datetimes`.
- **Fix**: same pattern previously applied to `_build_user_public()` —
promote `now` to TZ-aware before comparing.
- **Regression test**: existing tests mocked `referred_at` with naive
`datetime.utcnow()`, so the bug was invisible. Added two TZ-aware
variants matching production reality.
- **Backfill script**: `scripts/backfill_referral_earnings.py`
retroactively creates the missed `ReferralEarning` records and bumps
per-link `stats.purchases` / `stats.total_earned_cents`. Idempotent,
dry-run by default. Skips purchases predating attribution and refunded
sessions.
## Backfill dry-run output (verified)
```
Earnings to create: 20
Skipped (not credit_purchase): 7
Skipped (before attribution): 7
Skipped (refunded / unsuccessful / outside window / zero): 0
Stats updates per link:
angel: +1 purchases, +$5.60 (Travis)
e2etest70: +15 purchases, +$9.00 (e2e tests)
loyalty50: +4 purchases, +$7.15 (Angelina + John)
```
## Test plan
- [x] Unit test `test_creates_earning_with_tz_aware_referred_at`
reproduces the production `TypeError` before the fix and passes after.
- [x] Unit test
`test_no_earning_outside_window_with_tz_aware_referred_at` confirms
expired-window path also handles TZ-aware datetimes cleanly.
- [x] Full referral test suite passes (71 tests).
- [x] Full backend test suite passes (2987 tests).
- [ ] **Post-merge**: run `python scripts/backfill_referral_earnings.py
--apply` with full Firestore write credentials (claude-readonly ADC
blocks writes). Expect 20 earnings created, 4 link stats updated.
- [ ] **Post-merge**: verify admin referrals page now shows `angel: 1
purchase / $5.60 earned`.
@coderabbitai ignore
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
- Replaces single encoding worker VM with a blue-green pair (a/b) for
zero-downtime deployments
- VMs auto-start on demand (JIT warmup when user opens lyrics review)
and auto-shutdown after 15 min idle
- CI deploys to secondary VM, runs a real encode test, then swaps the
Firestore pointer atomically
- Deep health check catches issues like PR #587's pip crash before they
reach users
- Cloud Function checks idle VMs every 5 minutes and stops them to
minimize cost (~$10-15/mo idle vs $800/mo always-on)
## Architecture
- **Firestore config doc** (`config/encoding-worker`) is single source
of truth for primary/secondary routing
- **Backend** reads primary IP from Firestore (30s TTL cache), starts VM
on demand via Compute API
- **CI workflow** handles blue-green swap: deploy → health check → real
encode test → atomic Firestore swap → drain & stop old
- **Cloud Function** auto-stops idle VMs based on active jobs, user
activity, and deploy status
- **Frontend** sends warmup signal on lyrics review page load +
heartbeat on interactions
## Changes
### New Files
- `backend/services/encoding_worker_manager.py` — VM lifecycle +
Firestore config management
- `backend/api/routes/encoding_worker.py` — Warmup and heartbeat API
endpoints
- `infrastructure/functions/encoding_worker_idle/` — Idle shutdown Cloud
Function
- `infrastructure/modules/encoding_worker_manager.py` — Pulumi module
for Cloud Function + Scheduler
- `scripts/seed-encoding-worker-config.py` — Firestore config seeding
script
- `infrastructure/encoding-worker/test-assets/test-preview.ass` — Test
file for deploy health check
### Modified Files
- `infrastructure/compute/encoding_worker_vm.py` — Single VM →
blue-green pair
- `infrastructure/__main__.py` — Wire up new VMs, Cloud Function, IAM
- `backend/services/encoding_service.py` — Dynamic URL from Firestore,
warmup-aware retry
- `.github/workflows/ci.yml` — Blue-green deploy flow
- `frontend/` — Warmup signal on lyrics review, heartbeat, warming-up UI
state
- `docs/` — Architecture, development, troubleshooting, lessons learned
## Test plan
- [x] 47 new backend tests pass (EncodingWorkerManager, routes, dynamic
URL resolution)
- [x] 9 Cloud Function unit tests pass
- [x] All 2808 existing backend tests pass (1 pre-existing failure
unrelated)
- [x] All 1116 existing unit tests pass (3 pre-existing failures
unrelated)
- [ ] Pulumi preview shows expected resource changes
- [ ] Pulumi up applied locally before merge
- [ ] Firestore config seeded with VM IPs
- [ ] End-to-end verification after deployment
## Migration Steps (post-merge)
1. `pulumi up` creates 2 VMs, removes old VM
2. Seed Firestore: `python scripts/seed-encoding-worker-config.py <ip-a>
<ip-b>`
3. Start VM A, verify health
4. First CI deploy exercises the blue-green flow
## Design Docs
- Spec:
`docs/superpowers/specs/2026-03-24-blue-green-encoding-worker-design.md`
- Plan:
`docs/superpowers/plans/2026-03-24-blue-green-encoding-worker.md`
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@coderabbitai ignore
## Summary
- **Revenue chart showed nothing**: `store_payment()` used
`datetime.utcnow()` for `created_at` instead of the Stripe session's
actual `created` timestamp. All backfilled payments appeared on a single
date. Fixed to use `session["created"]`.
- **Single data point invisible**: Recharts AreaChart doesn't render a
visible area with only one point. Added `dot={chart.length === 1}` so
single points show as dots.
- **Payment detail sidebar had no padding**: `SheetContent` had no
padding class. Added `p-6`.
- **Backfill date fix script**: Added `scripts/fix_backfill_dates.py`
(already run against production to correct 32 records).
## Test plan
- [x] All 32 payment tests pass locally
- [x] Revenue chart verified visually via dev server — shows proper date
distribution (Dec 2025 - Mar 2026)
- [x] Payment detail sidebar verified visually — proper padding on all
sides
- [x] Production API confirmed returning correct date-bucketed data
after fix script
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
- Add `--quiet` flag to `gcloud beta emulators firestore start` to
prevent interactive prompts that hang CI/agent sessions
- Fix `stop-emulators.sh` to kill Java child processes (`pkill -P`) with
port-based fallback for orphans on port 8080
- Generate unique Playwright dev server port per worktree (3100-3199
hash range) to prevent cross-worktree conflicts
## Test plan
- [ ] `./scripts/start-emulators.sh` starts without interactive prompts
- [ ] `./scripts/stop-emulators.sh` kills Java emulator child processes
(no orphans on port 8080)
- [ ] Playwright config generates different ports for different worktree
directories
- [ ] `E2E_PORT` env var still overrides the default port
<!-- @coderabbitai ignore -->
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
- Fix broken "Buy More Credits" flow for logged-in users — replaces
redirect (which bounced back to `/app`) with an in-app dialog
- Add `BuyCreditsDialog` component following the existing
`FeedbackDialog` pattern
- Make credits count in header clickable to open the buy dialog
- Update "Made For You" pricing from $15 to $50 (frontend labels +
Stripe backend)
- Update hero section: "Create It Yourself ($5)", green "2 free credits"
message, remove redundant line, tighten spacing
## Test plan
- [ ] Log in, click "Buy More Credits" in dropdown — dialog opens with
package grid
- [ ] Click credits count in header — same dialog opens
- [ ] Select package, click "Continue to Payment" — redirects to Stripe
checkout
- [ ] Landing page pricing still works for unauthenticated users
- [ ] Made For You order form shows $50 pricing
- [ ] Hero section shows updated copy and tighter spacing
<!-- @coderabbitai ignore -->
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
- Brand code `NOMAD-1271` was reused for a real song (Hot Mulligan - Pop
Shuvit) while old E2E test job files still existed in Google Drive,
creating a duplicate in the public share
- Root cause: `cleanup-distribution` endpoint had two bugs that allowed
brand codes to be recycled before GDrive was confirmed clean
## Root Causes
1. **Empty dict was falsy**: `if gdrive_files:` skips cleanup when
`gdrive_files = {}` (empty dict). This happened when GDrive uploads
silently failed (pre-PR #417), so test job files accumulated in GDrive
without ever being cleaned
2. **Brand code recycled too early**: Brand code was recycled
immediately after Dropbox cleanup succeeded, *without* confirming GDrive
was also clean — a new job could claim the recycled code while old
GDrive files still existed
## Changes
- **`gdrive_service.py`**: Added `find_files_by_brand_code()` — searches
CDG/, MP4/, MP4-720p/ subfolders by brand code prefix as a fallback when
tracked file IDs aren't available (old jobs, or jobs where upload failed
silently). Raises `RuntimeError` if any subfolder search fails so
callers can't misread an incomplete search as "GDrive is clean"
- **`jobs.py` (`cleanup-distribution`)**: Now uses tracked file IDs
(fast path) OR falls back to brand_code search; brand code is only
recycled after **both** Dropbox AND GDrive are confirmed clean. Fixed
`method` key being lost when results dict was replaced (use local
variable instead of interim mutation)
- **`scripts/cleanup_test_job_gdrive.py`**: Remediation script to delete
existing stranded test job files from GDrive
## Testing
- 41 tests passing, 94% coverage on `gdrive_service.py`
- New `TestDeleteFile` class (5 tests for `delete_file`/`delete_files` —
previously uncovered)
- New `test_cleanup_distribution.py` (9 tests covering fast path,
fallback, and brand code gating logic)
- Updated error handling test to verify `RuntimeError` is raised (not
swallowed) on subfolder search failure
- [x] CodeRabbit CLI review completed, 2 findings fixed
@coderabbitai ignore
---
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude <noreply@anthropic.com>
## Summary
- Added tenacity retry with connection reset to Google Drive API calls
(`get_or_create_folder`, `upload_file`, `delete_file`) to handle stale
HTTP connections on idle Cloud Run containers
- Added `distribution_warnings` field to `OrchestratorResult` and
`_handle_native_distribution()` to track non-fatal upload failures
(YouTube, Dropbox, GDrive)
- Added Pushbullet notification when distribution uploads fail, so
operator is alerted immediately
- Added backfill script (`scripts/backfill_gdrive_uploads.py`) for
affected jobs - already run successfully for 12 affected jobs
- Updated docs: LESSONS-LEARNED.md and TROUBLESHOOTING.md with GDrive
stale connection pattern
## Context
Every GDrive upload silently failed for ~1 month (64 errors, 12+
affected jobs). The root cause was the singleton `GoogleDriveService`
caching its HTTP connection, which went stale when Cloud Run containers
sat idle between jobs. The orchestrator caught all exceptions and
continued, so jobs completed with `gdrive_files: {}` and no alert was
sent.
## Test plan
- [x] 5 new retry behavior tests in `test_gdrive_service.py`
(BrokenPipeError, SSLError, give up after 3, service reset, no retry on
non-transient)
- [x] 3 new distribution warnings tests in
`test_video_worker_orchestrator.py`
- [x] All 76 affected tests pass
- [x] Full backend test suite passes
- [x] Backfill script already run successfully for all 12 affected jobs
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
- **Systemic fix**: Orchestrator now validates CDG/TXT outputs after
packaging, before encoding. If `enable_cdg=True` but no CDG ZIP was
produced, the job fails immediately with a clear error instead of
silently completing without the package.
- **Remediation**: Created `scripts/regenerate_cdg.py` to regenerate and
distribute CDG/TXT packages for affected jobs (5b6aba25, 5161b069).
Packages uploaded to GCS, GDrive, and Dropbox.
- **Root cause**: Before b671b83f, `instrumental_selection: custom` had
no case in the orchestrator config builder — fell through to "Backing"
suffix, CDG generator looked for wrong filename, `FileNotFoundError` was
silently caught.
## Changes
| File | Change |
|------|--------|
| `backend/workers/video_worker_orchestrator.py` | Post-packaging
validation + LRC-missing check update |
| `backend/tests/test_video_worker_orchestrator.py` | 8 new tests in
`TestPackagingValidation` |
| `scripts/regenerate_cdg.py` | One-time remediation script (GCS +
Firestore + GDrive + Dropbox) |
| `docs/LESSONS-LEARNED.md` | Pipeline stage validation pattern |
| `docs/TROUBLESHOOTING.md` | CDG missing recovery runbook |
| `docs/README.md` | Updated recent changes |
| `pyproject.toml` | Version 0.119.6 → 0.119.7 |
## Test plan
- [x] 8 new tests in `TestPackagingValidation` (CDG fails, TXT fails,
combined CDG+TXT fails, CDG succeeds, CDG disabled, LRC missing raises
with CDG, LRC missing raises with TXT, LRC missing skips without CDG)
- [x] All 2062+ backend tests pass
- [x] Remediation script run successfully for both affected jobs
- [x] Verified CDG/TXT ZIPs in GCS, Firestore, GDrive, and Dropbox
> **CodeRabbit note**: 1 finding (unreachable `else` branch in
`_run_packaging` LRC check) — kept as defensive code since the test
`test_lrc_missing_skips_when_cdg_disabled` exercises this path directly.
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
Merges the two-step human review process (lyrics review → instrumental
selection) into a single unified flow. Users now review lyrics AND
select instrumental track in one session.
**Benefits:**
- Better UX (one interaction vs two)
- Reduced drop-off between steps
- Single email notification instead of two
- Simpler workflow
## Changes
### Backend
- `complete_review` endpoint now accepts optional
`instrumental_selection` in request body
- Backing vocals analysis moved from `render_video_worker` to
`screens_worker` (runs before review)
- Removed standalone instrumental selection endpoints (now part of
combined review)
- Added `CompleteReviewRequest` model with validation
### Frontend
- New `InstrumentalSelectorEmbedded` component for combined review UI
- `LyricsAnalyzer` updated to proceed to instrumental review
- `ReviewChangesModal` shows "Proceed to Instrumental Review" button
- Mock data updated in E2E tests
### Local CLI
- `gen_cli.py` refactored to use `run_combined_review()` instead of
separate `run_instrumental_review()`
- `ReviewServer` extended with instrumental options and analysis
- Deleted standalone `InstrumentalReviewServer`
### Documentation
- Updated `docs/README.md` with recent changes entry
- Updated `docs/ARCHITECTURE.md` with new pipeline diagram
- Updated `docs/API.md` with endpoint changes
- Added lesson to `docs/LESSONS-LEARNED.md`
## Test plan
- [x] Backend unit tests: 1742 passed
- [x] Frontend unit tests: 234 passed
- [x] New integration tests: 5 tests for combined flow
- [x] New backend tests: 7 tests for CompleteReviewRequest and
state_data
- [x] New frontend E2E tests: 6 tests for combined flow UI
🤖 Generated with [Claude Code](https://claude.com/claude-code)
@coderabbitai ignore
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
## Summary
- Fix two bugs causing phantom gaps in lyrics correction
- Add post-filtering gap extension to absorb single-word gaps into
adjacent anchors
- Add comprehensive test fixtures from 14 production jobs
## Bugs Fixed
1. **Initial gap creation bug** (line 986): The condition `if
transcription_position > 0` was always False for position 0, causing the
first word to be completely uncovered. Fixed to `if
transcribed_word_ids`.
2. **Text normalization mismatch** (line 455): Transcription words used
simple `lower().strip()` while references used `clean_text()` which
removes apostrophes. This caused "Train's" != "trains" mismatches. Fixed
to use consistent `clean_text()` for both.
## Results
| Metric | Before | After |
|--------|--------|-------|
| Coverage (Nana Grizol) | 79.6% | 87.0% |
| Phantom gaps at boundaries | Common | Eliminated |
| Words with apostrophes | Mismatched | Correctly matched |
## Test plan
- [x] All 1736 existing tests pass
- [x] New test suite with 14 real production job fixtures
- [x] Visual review of all test fixtures confirms improvement
- [x] Coverage tests verify no regression
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
## Summary
- Remove the `cloudbuild-deploy-as-backend` IAM resource that was
causing Pulumi deployment failures
- Delete legacy Cloud Build files that were never working
## Problem
Every push to main has been failing in the "Deploy - Infrastructure
(Pulumi)" step with:
```
Permission 'iam.serviceAccounts.setIamPolicy' denied on resource
```
This resource was added to grant Cloud Build permission to deploy Cloud
Run services, but:
1. CI uses GitHub Actions for builds and deployments, not Cloud Build
2. The GitHub Actions SA doesn't have `iam.serviceAccounts.setIamPolicy`
permission
3. The Cloud Build yaml files were legacy/unused
## Changes
- Remove `cloudbuild-deploy-as-backend` IAM binding from Pulumi
- Delete `cloudbuild.yaml`, `cloudbuild-base.yaml`, `scripts/deploy.sh`
- Update README to note deployments happen via GitHub Actions CI
## Test plan
- [x] Verify Python syntax: `cd infrastructure && python -c "import
__main__"`
- [ ] CI should pass - Pulumi will no longer try to create the
problematic IAM binding
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.ai/code)
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
## Summary
Fixes the deployment failure from PR #297 by making lyrics_transcriber a
proper subpackage of karaoke_gen instead of a separate vendored library.
### Changes
- Move `libs/lyrics_transcriber/` to `karaoke_gen/lyrics_transcriber/`
- Update all imports from `from lyrics_transcriber.X` to `from
karaoke_gen.lyrics_transcriber.X`
- Simplify Dockerfile (no separate `libs/` COPY needed)
- Update pyproject.toml to remove separate package entry
- Update test mocks and logger names
### Why this is better
- Cleaner project structure - one package containing everything
- Simpler Dockerfile - just COPY karaoke_gen/
- No confusion about vendored vs internal code
## Test plan
- [x] `poetry install` succeeds
- [x] All imports work correctly
- [x] All 2614 tests pass
- [ ] Backend deployment succeeds
@coderabbitai ignore
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
## Summary
- Add support for `STRIPE_PAYMENT_METHOD_CONFIG` environment variable to
enable wallet payment methods (Google Pay, Apple Pay, Link) in Stripe
Checkout sessions
- Update `docs/STRIPE-SETUP.md` with configuration instructions
- Add diagnostic scripts for testing Stripe payment method setup
## Context
Investigation revealed that Google Pay and Link require the
`payment_method_configuration` parameter to be passed to Stripe Checkout
session creation. Without this, Stripe uses defaults which may not
include all enabled wallet methods.
Additional findings:
- Link visibility is subject to Stripe A/B testing (confirmed by Stripe
support)
- Google Pay requires Chrome/Chromium browser (Firefox not supported)
## Test plan
- [x] Tested checkout session creation with payment method config on
Android Chrome
- [x] Verified Google Pay and Link appear in checkout
- [x] Verified made-for-you checkout flow shows all payment methods
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
## Summary
Separates karaoke-gen user data from karaoke-decide by using a dedicated
`gen_users` collection instead of the shared `users` collection.
**Problem:** Both karaoke-gen and karaoke-decide were using the same
`users` collection in Firestore with incompatible schemas:
- karaoke-gen: email as doc ID, has `credits`, `role`, `is_active`
- karaoke-decide: hash as doc ID, has `user_id`, `is_guest`, `quiz_*`
fields
**Solution:** karaoke-gen now uses its own `gen_users` collection.
## Changes
- Updated `USERS_COLLECTION` constant from `"users"` to `"gen_users"`
- Updated all hardcoded `db.collection("users")` references to use the
constant
- Updated Pulumi Firestore indexes for `gen_users` collection
- Added migration script (already executed)
## Migration Status
✅ 42 karaoke-gen users migrated to `gen_users` collection
✅ 36 karaoke-decide users remain in `users` collection (untouched)
✅ Firestore indexes created for `gen_users`
## Test plan
- [ ] Admin dashboard loads user list from `gen_users`
- [ ] User login/signup works
- [ ] Credit operations work
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Replace hardcoded LLM prompts with LangFuse-managed prompts to enable
prompt iteration without redeploying. Key changes:
- Add LangFusePromptService for fetching prompts and examples from LangFuse
- Add fetch_prompt() and fetch_dataset() helpers with version fallback
- Update classifier to use LangFuse prompts with hardcoded fallback for dev
- Add migration script to upload prompts/examples to LangFuse
- Delete legacy LLM handler code (llm.py, llm_providers.py)
- Change default LangFuse host to US region (us.cloud.langfuse.com)
- Add unit tests for prompt service
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Use KARAOKE_GEN_AUTH_TOKEN env var instead of gcloud identity token
- Use production API URL (api.nomadkaraoke.com) by default
- Add HTTP status code checking with helpful error messages
- Handle null/empty timeline gracefully
- Add worker logs section (more accurate real-time status)
- Compute enhanced status from state_data (like frontend does)
- Show step-based progress (e.g., "[4/10] Transcribing")
- Warn when backend status is stale vs actual worker progress
- Show more useful job details (artist, title, dates, file URLs)
The backend job.status field can be stale during parallel processing,
but state_data contains accurate worker progress. This mirrors the
frontend's approach in lib/job-status.ts.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Changes:
- Reverted to gemini-3-flash-preview (latest model with best capabilities)
- Changed GCP_LOCATION default from us-central1 to global (required for Gemini 3)
- Fixed multimodal response parsing for Gemini 3 models (returns list of content parts)
- Added local test script for agentic correction workflow
The test script (scripts/test_agentic_correction.py) allows testing the agentic
correction locally without deploying to Cloud Run, making iteration faster.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Move os import to top of investigate_user.py
- Replace deprecated datetime.utcnow() with datetime.now(timezone.utc)
- Use UserRole enum comparison instead of string value in auth_service.py
- Prefix unused auth_result parameter with underscore in bulk_delete_jobs
- Add exception chaining (from e) for clearer tracebacks in jobs.py
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Security fixes:
- Remove insecure request.user_email fallback in create_job endpoint
Now: auth_result.user_email used for regular users, admins can
create jobs on behalf of others, token auth gets no owner
- Add job ownership checks to require_review_auth and
require_instrumental_auth when using full auth tokens
Nitpicks:
- user_service.py: Remove redundant exception in logging.exception
- auth_service.py: Prefix unused variable with underscore (_message)
- audit_logging.py: Use logging.exception instead of error for stack trace
- investigate_user.py: Use env var for project ID instead of hardcoding
- investigate_user.py: Remove unnecessary f-string prefix
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Security & Auth Enhancements:
- Add AuthResult dataclass with user_email and is_admin context
- Add admin-by-email-domain check for @nomadkaraoke.com
- Add job ownership checks to all job endpoints (users see only their jobs)
- Admins bypass ownership checks and see all jobs
Audit Logging:
- Add AuditLoggingMiddleware with request_id correlation
- Add auth event logging (success/failure) with request_id
- Enhance structured logging with audit fields
- Add list_sessions_for_user() method for user investigation
Investigation Script:
- Add scripts/investigate_user.py for querying user activity
- Shows: profile, jobs, sessions, credit transactions, magic links
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add comprehensive documentation for debugging jobs in production:
- Firestore job data access via Python script
- Cloud Logging queries for job-specific logs
- GCS file listing and inspection
- Cloud Run service health checks
- Cloud Tasks queue monitoring
- Common debugging scenarios
Also adds scripts/get_job.py helper for quick job lookups.
All commands tested against real job ID ffb0b8fa.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Adds a comprehensive theme system for karaoke video styles:
- Theme selection and customizable color overrides for karaoke jobs
- Public Themes API for listing, previewing, and fetching theme details
- Theme selector and color-overrides UI in job submission flows
- CLI option to specify a theme when preparing/submitting jobs
- Thread-safe singleton initialization for ThemeService
🤖 Generated with Claude Code
- Fix broken countdown processor tests (removed API methods that don't exist)
- Fix asyncio tests in test_instrumental_api.py (use pytest.mark.asyncio)
- Fix flaky test_search_returns_results (clear _remote_client to force mock)
- Fix youtube_description_template not passed through job_manager.create_job
- Fix Config fixture in E2E tests (add poll_interval and output_dir)
- Add Makefile with `make test` for running all tests
- Update CI to run all backend tests (not just specific files)
- Update cursor rule to specify full test suite requirements
- Version bump to 0.75.53
The download_file_via_url method was using requests.get() directly
without the Authorization header, causing all downloads to fail with
401/403 errors even though the job completed successfully.
Now uses self.session.get() which includes the auth token from headers.
Bump version to 0.75.51
- Added original audio path detection in gen_cli.py
- Pass original_audio_path to InstrumentalReviewServer
- Updated test-instrumental-review.py script with --original argument
- Added tests for original audio functionality
- Bumped version to 0.75.12
The Instrumental Review UI now shows an 'Original Audio' tab when the
original audio file is available, allowing users to compare the full
song against backing vocals and instrumentals when deciding which
regions to mute.
Closes #[issue_number] (if applicable)
Backend:
- Add original_audio_path parameter to InstrumentalReviewServer
- Add /api/audio/original endpoint for streaming original audio
- Add "original" to valid selection options
Frontend:
- Add "Original" toggle button (shown first when available)
- Add "Original Audio" selection option with description
- Add original to success message labels
Helper script:
- Find original audio files matching "(Original)" pattern
- Log and pass original_audio_path to server
Version bump: 0.71.58 → 0.71.59
- Fix invalid escape sequences in updateMuteRegionsPanel() - use string
concatenation instead of nested template literals
- Update helper script to find instrumental files in main job directory
(not just stems/) and handle "+BV" naming convention
Version bump: 0.71.56 → 0.71.57