## Summary
Backend support for the Nomad Karaoke mobile app (nomad-mobile Phase 1,
"Make it").
- **`POST /api/users/auth/handoff-token`**: the app opens a job's web
review already signed in. The token is a single-use, 120s
`job_review:<id>` magic link, returned as
`{FRONTEND_URL}/{locale}/auth/verify?token=…&handoff=1`.
- The verify page skips the "Complete Sign-In" tap only when the URL has
`handoff=1` **and** the server confirms the token was minted as a
handoff (`link-status` → `handoff: true`). Emailed links keep the
scanner guard.
- The web locale is validated against the 33 web locales.
- **Checkout `return_to: "app"`**: Stripe success and cancel go to `GET
/api/users/credits/app-return`, which 302s to
`nomadkaraoke://checkout?status=success|cancelled`.
- The client never supplies a URL, so this adds no open redirect.
- `session_id` is never forwarded.
- **`POST /api/users/auth/google` and `/auth/apple`**: ID-token sign-in
(PyJWT + JWKS: RS256, issuer, audience, expiry, `email_verified`). It
returns the same `VerifyMagicLinkResponse` as email-code sign-in,
through a shared `finish_verified_sign_in`.
- Email-code, Google and Apple all share the same abuse checks
(`_sign_in_checks`).
- A JWKS outage returns 503 `identity_provider_unavailable`.
- Google audiences default to the new mobile web and iOS OAuth clients,
which are recorded in ARCHITECTURE.md.
- **Job summary** now includes `credit_refunded`.
- **Local stack:**
- fake-config `search_confidence` (perfect / recommended / unsure), with
a drift guard tied to the frontend `getSearchConfidence`
- `youtube_url_enabled`
- `fake:<email>` social tokens, accepted only in fake mode
- PyJWT is now a direct dependency (`pyjwt[crypto]`). Before this it was
only present transitively.
- Version 0.278.0.
## Testing
- `make test` is green:
- CLI: 2088 passed
- backend unit: 6360 passed
- emulator: 156 passed, plus 4 handoff lifecycle tests
- frontend jest: 1751 passed
- New suites:
- `test_handoff_token.py`
- `emulator/test_handoff_token_lifecycle.py`
- `test_checkout_app_return.py`
- `test_social_signin.py`
- `test_magic_link_status.py`
- `fake-search-tiers.test.ts`
- verify-page tests
- Reviews:
- An independent code review was approved, and its hardening suggestions
were applied.
- A local CodeRabbit review found nothing.
Plan: `docs/archive/2026-10-17-mobile-app-auth-handoff-plan.md`. Spec:
nomad-mobile
`docs/superpowers/specs/2026-10-17-mobile-phase1-make-it-design.md`.
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_011yLH4kWXMqjeKfZUG3ZiWZ
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
44 KiB
Testing & Code Quality Guide
This document defines testing standards, code quality principles, and CI requirements for Nomad Karaoke Gen.
Critical: All Tests Must Pass
All tests must pass before merging. If tests fail:
- Fix them - Don't assume failures are "pre-existing issues"
- Investigate - Understand why the test is failing
- Don't dismiss - If a test fails, either the code or test needs fixing
CI blocks merging if tests fail. If you encounter failures locally, fix them as part of your work.
Running Tests
# Run ALL tests (backend + frontend) - installs deps automatically
make test
# Run subsets for faster iteration
make test-backend # Backend only (~2 min)
make test-frontend # Frontend only (~3 min)
Dependencies are installed automatically when needed.
Critical: Automate Manual Testing Steps
If your plan includes "Manual Testing" steps, convert them to automated Playwright E2E tests.
When you write a plan with steps like "manually verify the dialog opens" or "test that the button appears for complete jobs", these are perfect candidates for automated E2E tests. Don't leave verification as a manual step that may never happen.
Why Automate?
- Codifies expected behavior - The test documents exactly what should happen
- Repeatable verification - Can re-run after future changes to catch regressions
- Actually gets executed - Manual steps are often skipped; automated tests run reliably
- Proves the feature works - Passing test = working feature in production
Production E2E Tests Are Valuable
It's acceptable (and preferred) to write E2E tests that only run against production:
// frontend/e2e/production/admin-delete-outputs.spec.ts
test('Admin can delete job outputs', async ({ page }) => {
// This test runs against real production after deployment
// It verifies the feature actually works end-to-end
await page.goto(`${PROD_URL}/admin/jobs/${testJobId}`);
await page.click('[data-testid="delete-outputs-button"]');
await expect(page.getByText('Confirm deletion')).toBeVisible();
// ... complete the flow
});
Even if the test is only run once after deployment, it's still valuable because:
- It codifies the expected behavior completely
- It proves the deployed code works
- It can be re-run if issues are suspected
- It serves as documentation for the feature
Workflow for Production-Only E2E Tests
- Implement the feature with unit tests and regression E2E tests (mocked API)
- Write production E2E test that exercises the real flow
- Merge and deploy to production
- Run the production E2E test to verify:
npm run test:e2e:prod -- admin-delete-outputs.spec.ts - Commit passed - Feature is verified working in production
Example: Converting Manual Steps to E2E
Before (manual testing in plan):
Manual testing:
- Create/complete a test job with YouTube/Dropbox upload
- Navigate to admin job detail
- Click "Delete All Outputs" button
- Verify dialog shows, confirm deletion
- Verify "Outputs Deleted" badge appears
After (automated production E2E):
test('Delete outputs flow', async ({ page }) => {
// Use existing complete job or create one via API
const jobId = process.env.E2E_COMPLETE_JOB_ID;
await page.goto(`${PROD_URL}/admin/jobs/${jobId}`);
await page.click('[data-testid="delete-outputs-button"]');
await expect(page.getByRole('dialog')).toBeVisible();
await page.click('[data-testid="confirm-delete"]');
await expect(page.getByText('Outputs Deleted')).toBeVisible();
});
When to Use Each Test Type
| Scenario | Test Type | Location |
|---|---|---|
| UI component behavior | Jest unit test | components/__tests__/ |
| UI flow with mocked API | Playwright regression | e2e/regression/ |
| Multi-step journey / job states, real backend, offline | Playwright local stack | e2e/local-stack/ |
| Full flow against real backend | Playwright production | e2e/production/ |
| API endpoint logic | pytest unit test | backend/tests/ |
| Database operations | Emulator test | backend/tests/emulator/ |
Default to production E2E for any feature that involves user-facing workflows. The goal is automated verification of deployed code, not checkbox completion of manual steps.
Core Principles
SOLID Principles
All code should follow SOLID principles:
- Single Responsibility - Each class/function does one thing well
- Open/Closed - Open for extension, closed for modification
- Liskov Substitution - Subtypes must be substitutable for base types
- Interface Segregation - Many specific interfaces over one general interface
- Dependency Inversion - Depend on abstractions, not concretions
Code Quality Standards
- Maintainability - Code should be easy to understand and modify
- Testability - Code should be designed for easy testing (dependency injection, pure functions)
- No magic numbers - Use named constants
- Explicit over implicit - Clear is better than clever
- DRY but not prematurely - Don't repeat yourself, but don't abstract too early
Test Types
Unit Tests
Locations:
tests/unit/- CLI package (karaoke_gen/) unit testsbackend/tests/- Backend API unit tests (excludesemulator/)frontend/components/__tests__/- React component unit tests
Characteristics:
- Fast (< 1ms per test typical)
- No external dependencies (mocked)
- No network, database, or filesystem access
- Run on every commit
Coverage targets:
karaoke_gen/package: 69%+backend/package: No explicit threshold (but coverage collected)frontend/components: No explicit threshold (but coverage collected)
# Good unit test example (backend/tests/test_job_manager.py)
def test_create_job_applies_default_theme(mock_firestore, mock_gcs):
job_manager = JobManager()
job = job_manager.create_job(
artist="Test Artist",
title="Test Song",
user_email="test@example.com"
)
assert job.theme_id == "nomad" # Default theme applied
Critical: Mocks Must Match Real API Contracts
When mocking methods, ensure the mock's return_value matches what the real method returns. A common bug pattern:
# ❌ BAD: Assumes method returns boolean
mock_service.update_job.return_value = True
# Code under test:
success = job_manager.update_job(job_id, updates)
if not success: # BUG: if real method returns None, this is always True!
raise HTTPException(500)
The fix:
# ✅ GOOD: Check real method signature first
# def update_job(self, job_id: str, updates: Dict) -> None:
mock_service.update_job.return_value = None # Match real API
Before writing a mock, check the actual method's return type. Void methods should mock return_value = None. Methods that raise on error (rather than returning False) should have mocks that raise exceptions for error cases. See LESSONS-LEARNED.md for a production bug caused by this.
Critical: Test the Contract Between Caller and Callee
When function A calls function B, it's not enough to test B with hand-crafted ideal inputs. You must also verify A actually produces those inputs. This is a "contract test."
# ❌ BAD: Tests _store_metadata in isolation with hand-crafted input
# If transcribe_lyrics() never includes "lyrics_dir", this test still passes
# but production silently loses all metadata
def test_stores_audioshake_ids(tmp_path):
_store_lyrics_processing_metadata(
transcription_result={"lyrics_dir": str(tmp_path)}, # Too helpful!
...
)
assert call_args["audioshake_task_id"] == "task-abc" # ✅ passes, but useless
# ✅ GOOD: Also test that the caller produces the expected input
def test_transcribe_lyrics_returns_lyrics_dir(tmp_path):
result = processor.transcribe_lyrics(audio, artist, title, tmp_path)
assert "lyrics_dir" in result # Catches the actual bug
When to write contract tests:
- Function A builds a dict/object and passes it to function B
- Function B reads specific keys/fields from that dict/object
- The dict/object shape is implicit (no schema, no TypedDict, no dataclass)
Especially dangerous: Non-fatal code paths (try/except that swallows errors). If the function is designed to never crash, a test that only checks "didn't raise" provides zero signal. Assert on the positive outcome (data was stored, value was returned) not just the absence of exceptions.
See LESSONS-LEARNED.md for the production bug this pattern caused.
Render tests: verify lyrics as they actually render (CJK / RTL)
Correct-looking ASS tags can still render wrong (a Hebrew job shipped with words laid
out and highlighted left-to-right). tests/unit/lyrics_transcriber/output/test_ass_render_highlight.py
generates ASS with the real SubtitlesGenerator in a pure-colour palette (sung = green,
unsung = magenta, lead-in = cyan, on black), renders frames with ffmpeg/libass
(ass_render_harness.py), and asserts on pixels for English, Hebrew, Arabic, Chinese,
Japanese, Korean and an RTL duet line: highlight starts at the reading-start edge,
sung pixels always precede unsung ones in reading order (pins word order and in-word
fill direction), progress is monotonic and in time with the words, the lead-in lands
outside the start edge, and libass found every glyph. tests/unit/test_font_fallback_render.py
does the same for PIL-rendered title cards / portrait headers (no .notdef tofu, Hebrew
title reads right-to-left). test_translated_lyrics_render.py covers translated lyrics: the
translation row renders beneath its line (smaller, centred, never highlighted, every glyph
found) for Latin/Hebrew/Arabic/CJK translations and mixed directions (Hebrew/Arabic lyrics with
English or Arabic translations), and an RTL lyric still fills right-to-left.
- CI installs
fonts-noto-core fonts-noto-cjkand runs withCI=true, where these tests fail instead of skipping on a missing precondition. - macOS: Homebrew libass uses the CoreText font provider (different fallback fonts, no
CJK fallback), so font-sensitive checks skip. Run the full set in Linux against both
Debian's ffmpeg and prod's static ffmpeg with
scripts/run-render-tests-linux.sh. - When touching ASS generation, add a sample to
SAMPLESrather than asserting on tag strings.
Integration Tests (tests/integration/)
Purpose: Test CLI workflows and component interactions.
Characteristics:
- May mock high-level components
- Tests full command pipelines
- Uses
@pytest.mark.integrationmarker - Often async (
@pytest.mark.asyncio)
Example scenarios:
- CLI
--edit-lyricsflow end-to-end - Job creation → processing pipeline
- Audio separation → transcription coordination
Emulator Tests (backend/tests/emulator/)
Purpose: Test real Firestore/GCS behavior without production access.
Characteristics:
- Requires GCP emulators running
- Tests actual database operations
- Verifies subcollections, TTL policies, etc.
- Auto-skips if emulators unavailable
Setup:
# Start emulators
make emulators-start
# Run emulator tests
poetry run pytest backend/tests/emulator/ -v
# Stop emulators
make emulators-stop
E2E Tests (Frontend)
Locations:
frontend/e2e/regression/- Mocked API tests (CI-safe, fast)frontend/e2e/local-stack/- Real backend on emulators with fake workers (CI-safe, ~1 min) — see Local stackfrontend/e2e/production/- Real backend tests (comprehensive, slower)frontend/e2e/manual/- Tests requiring manual setup (not in CI)
Tool: Playwright
Regression tests run on every PR:
- Mock API responses via fixtures
- Test UI behavior in isolation
- Playwright auto-starts dev server via
webServerconfig - ~3 min total runtime
Production tests run on-demand or daily:
- Hit real
https://gen.nomadkaraoke.comandhttps://api.nomadkaraoke.com - Full user journeys (20-40 min for happy path)
- Require authentication tokens
Manual tests are NOT included in CI:
- Require separate dev servers (e.g., LyricsTranscriber frontend on port 5173)
- See
frontend/e2e/manual/README.mdfor setup instructions
Local stack (real backend, fake workers)
scripts/local-stack.sh runs the real FastAPI backend on the Firestore + GCS
emulators with FAKE_WORKERS=1. Every API route, the review UI's data endpoints,
credits, emails and downloads run for real. Only the workers are swapped out
(backend/services/fake_workers/): they walk each job through its real states in
seconds, writing small real media (ffmpeg tones, 320x180 video) and a
schema-valid corrections.json built from the real CorrectionResult types.
Web Playwright (frontend/e2e/local-stack/) and mobile Maestro flows run against
it, so both clients hit the same endpoints.
scripts/local-stack.sh up --frontend # emulators + API :8010 + Next dev :3010
make test-local-stack # or: cd frontend && npm run test:e2e:local-stack
scripts/local-stack.sh logs backend # tail the API log
scripts/local-stack.sh down
Requirements: Docker (fake-gcs-server), Java 21 + the gcloud Firestore emulator,
ffmpeg, and the poetry env. The Playwright config starts the stack itself if it
isn't running (scripts/local-stack.sh serve + next dev).
What is faked, and where. Each fake lives at one seam, guarded by
fake_workers_enabled(). That check is strict (is True), so a MagicMock
settings object in a unit test never turns it on.
| Real thing | Fake | Seam |
|---|---|---|
| Cloud Run Jobs / Cloud Tasks workers | in-process threads (fake_workers/pipeline.py) |
WorkerService.trigger_* |
| flacfetch / YouTube search | 3 deterministic results (lossless "perfect match", Spotify, YouTube) | get_audio_search_service() |
| audio download | canned tone song (then the REAL process_audio_download continues: duration reconcile etc.) |
audio_download_worker._download_audio |
| Stripe Checkout | /api/test/fake-checkout/{id} page; Pay runs the real webhook handler |
StripeService._create_checkout |
| Postmark | console provider + captured outbox | ConsoleEmailProvider |
| Secret Manager, DeBounce, encoding VM | skipped | get_secret, is_disposable_domain, warmup |
The audio worker reuses the real stem upload, OGG transcode and backing-vocals
analysis. The review preview video is the real local ffmpeg render. Startup
refuses FAKE_WORKERS in production (validate_production_config).
Test API (/api/test/*, mounted only with FAKE_WORKERS, no auth):
| Endpoint | Purpose |
|---|---|
GET /api/test/status |
readiness, effective config, running fake workers, scenario names |
POST /api/test/scenario {name, email?, credits?, artist?, title?} |
seed user, job_awaiting_review, job_complete, job_failed, job_awaiting_duration_confirm → {email, session_token, credits, job_id?}. Jobs are made through the real search + create-from-search routes. |
GET/PUT /api/test/fake-config[?email=] |
knobs: step_delay_seconds, auto_approve, fail_at, audio_duration_seconds, search_result_count, search_confidence (perfect / recommended / unsure → the UI's confidence tier 1 / 2 / 3), youtube_url_enabled (default true: the fake video step sets state_data.youtube_url = https://youtu.be/dQw4w9WgXcQ on non-private jobs). With ?email= the knobs apply only to that user's jobs, so tests that each use their own user can run in parallel. Search runs before a job exists, so search_result_count and search_confidence only take effect globally. |
GET /api/test/emails?to= |
captured outbound email, newest first, with extracted links and 6-digit codes |
POST /api/test/reset |
wipe the emulator Firestore + bucket (keeps seeded themes) |
Sign-in in tests: seed a user and put session_token in
localStorage.karaoke_access_token (signInAs in e2e/local-stack/stack.ts).
Alternatively, drive the real magic link or email code and read it from
/api/test/emails.
Google / Apple sign-in: POST /api/users/auth/google|apple {"id_token": "fake:<email>"}
signs in as <email> with no network call, only in fake mode (outside it, 401). The
mobile app's fake-social switch sends these.
Handoff to the web review: with a session for a job_awaiting_review scenario,
POST /api/users/auth/handoff-token {"target": "job_review", "job_id": …} returns a
2-minute, single-use …/auth/verify?token=…&handoff=1 URL (on FRONTEND_URL) that
signs in without the "Complete Sign-In" tap and lands on the review.
Checkout back to the app: POST /api/users/credits/checkout with
"return_to": "app" makes the fake checkout's Pay / Cancel redirect to
$LOCAL_STACK_API_URL/api/users/credits/app-return?status=…, which 302s to
nomadkaraoke://checkout?status=success|cancelled. Cancel never runs the webhook.
Fake search tiers: fake_workers/search.py builds the search_confidence
shapes, and frontend/lib/__tests__/fake-search-tiers.test.ts feeds them to the real
getSearchConfidence via frontend/lib/__tests__/fixtures/fake-search-tiers.json.
A backend test fails if that fixture is stale; regenerate it with
poetry run python -m backend.services.fake_workers.search --write-fixture.
Mobile (Maestro / dev builds): run the stack, then point the app at
http://10.0.2.2:8010 (Android emulator) or http://127.0.0.1:8010 (iOS
simulator). Set LOCAL_STACK_API_URL to that address before up, so fake-checkout
URLs open on the device. Seed state with curl -X POST .../api/test/scenario, and read
sign-in codes from /api/test/emails.
Ports: LOCAL_STACK_API_PORT (8010) and LOCAL_STACK_WEB_PORT (3010). The
emulators use the fixed 8080 / 4443 that start-emulators.sh uses, so stop other
emulator runs first.
Playwright Usage
Installation
cd frontend
npm install -D @playwright/test
npx playwright install
Running E2E Tests
# Regression tests (mocked API, default)
npm run test:e2e
# Production tests (real backend)
npm run test:e2e:prod
# With UI mode for debugging
npm run test:e2e:ui
# Specific test file
npx playwright test e2e/regression/smoke.spec.ts
# Production test with environment
E2E_TEST_TOKEN=xxx npm run test:e2e:prod
Investigating Frontend Issues
When debugging frontend issues, use Playwright to:
- Reproduce the issue - Write a test that fails
- Debug visually - Use
--uimode or--headed - Capture evidence - Screenshots, traces, console logs
- Verify the fix - Test passes after fix
# Debug mode with browser visible
npm run test:e2e:headed
# Generate trace for failed tests
npx playwright test --trace on
Writing E2E Tests
Regression test example (mocked API):
// frontend/e2e/regression/smoke.spec.ts
import { test, expect } from '@playwright/test';
import { setupApiFixtures, clearAuthToken } from '../fixtures/test-helper';
test.describe('Smoke Tests', () => {
test.beforeEach(async ({ page }) => {
await clearAuthToken(page);
});
test('landing page loads and shows hero', async ({ page }) => {
await setupApiFixtures(page, { mocks: [] });
await page.goto('/');
await expect(page.locator('h1')).toContainText('Karaoke');
});
});
Production test example (real backend):
// frontend/e2e/production/lyrics-review-only.spec.ts
import { test, expect } from '@playwright/test';
const PROD_URL = 'https://gen.nomadkaraoke.com';
test.describe('Lyrics Review', () => {
test('Complete lyrics review via UI', async ({ page }) => {
test.setTimeout(900_000); // 15 min - preview generation can be slow
const jobId = process.env.E2E_JOB_ID;
await page.goto(`${PROD_URL}/review/${jobId}`);
// ... test implementation
});
});
CI Pipeline
Required Checks (Block PR Merge)
All PRs must pass these checks (path-filtered for efficiency):
| Check | Description | Threshold |
|---|---|---|
backend-lint |
Ruff linter | No errors |
backend-unit-tests |
Backend tests (excludes emulator) | Must pass |
backend-emulator-tests |
GCP emulator integration | Must pass |
local-stack-e2e |
Playwright full journeys vs the local stack (backend or frontend changes) | Must pass |
package-unit-tests |
CLI package tests | 69% coverage |
package-integration-tests |
CLI integration tests | Must pass |
frontend-lint |
ESLint (Rules of Hooks) | No errors |
frontend-unit-tests |
Jest component tests | Must pass |
frontend-e2e-smoke |
Playwright smoke tests | Must pass |
frontend-build |
Next.js build | Must pass |
On-Demand Checks
| Check | Trigger | Description |
|---|---|---|
frontend-e2e-full |
e2e PR label |
Full regression E2E suite |
e2e-happy-path |
Daily 6 AM UTC | Complete production happy path |
Path Filtering
CI intelligently runs only relevant tests based on changed files:
| Changed Path | Tests Run |
|---|---|
frontend/ |
Frontend lint, unit, E2E, local-stack E2E |
backend/ |
Backend lint, unit, emulator, local-stack E2E |
karaoke_gen/ |
Package unit, integration |
tests/ |
Package tests |
infrastructure/ |
Infrastructure validation |
Known Testing Gaps
Infrastructure dependencies (Modal, AudioShake, Cloud Tasks, Firestore) are mocked in tests. This means IAM permission errors or API changes won't be caught until production. Mitigations:
- Emulator tests - Use
backend/tests/emulator/for Firestore/GCS integration - Manual verification after infrastructure changes (Pulumi updates)
- Health endpoints -
/api/healthand/api/health/detailedvalidate connectivity - Daily E2E -
e2e-happy-path.ymlruns full production flow
Production E2E Testing
For comprehensive production testing:
cd frontend
# Run full happy path (requires tokens)
E2E_TEST_TOKEN=<token> npx playwright test e2e/production/happy-path-real-user.spec.ts \
--config=playwright.production.config.ts
# Run focused tests (faster iteration)
E2E_JOB_ID=<id> E2E_REVIEW_TOKEN=<token> npx playwright test e2e/production/lyrics-review-only.spec.ts \
--config=playwright.production.config.ts
# Admin dashboard tests
KARAOKE_ADMIN_TOKEN=<admin-token> npx playwright test e2e/production/admin-dashboard.spec.ts \
--config=playwright.production.config.ts
Environment Variables
| Variable | Purpose | Required For |
|---|---|---|
E2E_TEST_TOKEN |
User authentication | Most production tests |
KARAOKE_ADMIN_TOKEN |
Admin authentication | Admin dashboard tests |
E2E_JOB_ID |
Existing job to test | Focused tests |
E2E_REVIEW_TOKEN |
Review page access | Lyrics review tests |
TESTMAIL_API_KEY |
TestMail.app API | Magic link auth tests |
TESTMAIL_NAMESPACE |
TestMail.app namespace | Magic link auth tests |
Getting Test Tokens
Option 1: From browser
- Login to https://gen.nomadkaraoke.com
- Open DevTools → Application → Local Storage
- Copy the
karaoke_access_tokenvalue
Option 2: From backend logs When the daily E2E runs, tokens are logged for debugging.
The production tests cover:
- Magic link authentication flow
- Audio search and selection
- YouTube URL processing
- File upload workflow
- Lyrics review and correction
- Preview video generation
- Instrumental selection
- Final encoding and download
- Admin dashboard functionality
Self-Healing Stripe Checkout (daily E2E Stage 1)
Payment mode (repo variable
E2E_REAL_PAYMENT). Unset/anything buttrue→E2E_PAYMENT_MODE=checkout-only(current default since 2026-10-05, no working test card): Stage 1 signs up via the referral link, creates a real Checkout session, fills it with Stripe's public test card4242…(NO_CHARGE_CARD), asserts the $0.50 price + Pay button, and stops before Pay — no charge, no credit grant, no self-heal PR. Stage 2 uses the new account's signup credit. SetE2E_REAL_PAYMENT=true(Settings → Variables) to resume buying a real credit with theE2E_STRIPE_CARD_*card.
Stripe changes its hosted checkout page without notice (5 breaking changes Apr–Oct 2026),
so frontend/e2e/helpers/stripe-checkout.ts runs the checkout as verified sub-goals
(selectCard, fillCardNumber, fillCardExpiry, fillCardCvc, fillCardholderName,
fillPostalCode, uncheckSaveInfo, submitPayment) via e2e/helpers/self-healing/:
- Learned variants (
self-healing/learned-variants.json, newest first) — recipes the LLM discovered on earlier live runs. - Builtin variants — the hand-written selectors in
stripe-checkout.ts. - LLM fallback — Gemini (
E2E_SELF_HEAL_MODELS, defaultgemini-3.5-flash-lite,gemini-3.1-flash-lite,gemini-3.8-flash; unavailable ids are skipped, failures escalate down the list) is shown a redacted ARIA snapshot + input-masked screenshot and drives the page one action at a time until the step'sverifypostcondition passes.
Each attempt is judged only by the step's verify (e.g. "card-number input visible",
"filled field holds the expected value and looks like a card-number field", "redirected to
our success page or credit granted server-side") — never by "the click didn't throw".
Card data never reaches the LLM: it may only use {{CARD_NUMBER}}-style placeholders
(per-step allowlist) that are substituted locally; textbox values and PAN-like digit runs are
redacted from observations; inputs are masked in screenshots. Guardrails refuse clicks on
Link/Klarna/Cash App/Bank/"Back to…" and on the final Pay button outside submitPayment.
Fully automatic loop. When the LLM heals a step and the purchase is verified server-side,
the spec writes test-results/self-heal/learned-variants.json; the workflow's
self-heal-pr job opens a data-only PR (label e2e-self-heal, @coderabbitai ignore),
enables auto-merge, and approves the PR's pull_request CI run (bot PRs' runs start as
"action_required" and block the CI Gate until run; falls back to dispatching ci.yml).
Next run replays the recipe without the LLM.
Repo setting this depends on (changed 2026-10-03, not in code): the protect main
ruleset (id 11423976) has require_extra_approval_for_unattributed_changes: false. GitHub
defaults it to true (server-side, ~Aug 2026), which silently blocks bot-authored PRs
(mergeStateStatus: BLOCKED with all checks green) until a human approves. If a ruleset
update ever omits the key, GitHub resets it to true — set it explicitly. Discord + the daily email
report 🩹 when this happens. Only one open self-heal PR at a time.
Declined card ≠ layout change. The helper watches Stripe's
/v1/payment_pages/<id>/confirm response; a card_error (e.g. card_declined/generic_decline)
fails submitPayment immediately with STRIPE_CARD_DECLINED: … — no LLM calls, no second Pay
click. The workflow surfaces it as the Stage 1 failure_reason in Discord and the daily email
("Test card declined, not a checkout bug"). Fix it on the card side (issuer fraud hold — e.g.
after many live $0.50 runs in a day — or an expired/replaced card), then re-run.
The Gemini key is the AI Studio gemini-api-key (billed outside the nomadkaraoke project):
a GEMINI_API_KEY repo secret if set, else read from Secret Manager via keyless auth (the
credential file is deleted before tests run). No key → deterministic-only, as before.
cd frontend
# Offline tests (fixture checkout pages + stub LLM) — also run in CI's smoke job
npx playwright test --config=playwright.selfheal.config.ts
# Exercise the live LLM path end-to-end: Actions → E2E Daily → Run workflow →
# self_heal_force_llm = selectCard (forces that step through Gemini; opens a PR)
E2E Artifacts Are Public — Scrub Before Upload
karaoke-gen is a public repo: any signed-in GitHub user can download Actions artifacts.
Every workflow that uploads production-E2E output runs frontend/e2e/scripts/scrub-artifacts.mjs
first and only uploads if the scrub succeeded:
- deletes Playwright traces (
*.zip— they contain every request header incl.X-Admin-Tokenand every typed value) ande2e-session-token.txt; - replaces literal secret values (admin token, testmail key, card number, session tokens…) and card-number-shaped digit runs in text files; deletes any text file still containing a secret;
- Stage 1 also passes
--drop-videos --blank-inputs(videos of the Stripe page;<input value>and ARIA-snapshot textbox values in DOM dumps /error-context.md).
In addition, playwright.production.config.ts turns traces off on CI (E2E_TRACE=on to
override), the credit-purchase spec disables trace/video/auto-screenshots, and the Stripe helper's
own screenshots mask every input. When adding a new secret to a production E2E job, add its env
var to SECRET_ENV_VARS in the scrubber and to the job's scrub step.
Concurrent-Load Regression Test (review pages)
scripts/load-test-review.py replays the 2026-09-18 "10 tabs melt the backend"
incident: N simultaneous review-tab loads, each firing the page's real GET trio
(correction-data, waveform-data, ranged audio/vocals), authenticated with per-job
review tokens read from Firestore via ADC (no admin secret needed).
python scripts/load-test-review.py # 10 tabs vs prod, warm thresholds
python scripts/load-test-review.py --tabs 20
python scripts/load-test-review.py --max-tab-seconds 45 # cold caches (first-ever waveforms)
Pass = every request 2xx and every tab under the threshold; exit 1 otherwise.
Run it after changes to the review hot path (backend/api/routes/review.py,
audio_transcoding_service, audio_analysis_service) or Cloud Run deploy flags.
Background + before/after numbers:
docs/archive/2026-09-18-concurrent-review-reliability-investigation-and-plan.md.
Ad-Hoc Production Debugging (for Agents)
When asked to "test it yourself in prod" or debug a specific production issue, use the debug script template:
Quick Start
# Copy the template (the .local suffix is gitignored)
cp frontend/e2e/helpers/debug-prod-template.mjs frontend/test-my-issue.local.mjs
# Edit the script to customize what you're testing
# You can hardcode tokens in .local files - they won't be committed
# Run it
cd frontend && node test-my-issue.local.mjs
With Environment Variables (Preferred)
# Get admin token from Secret Manager
export KARAOKE_ADMIN_TOKEN=$(gcloud secrets versions access latest --secret=admin-tokens --project=nomadkaraoke | cut -d',' -f1)
# Run the template directly
cd frontend && node e2e/helpers/debug-prod-template.mjs
# Or with a specific job
DEBUG_JOB_ID=abc123 node e2e/helpers/debug-prod-template.mjs
What the Template Provides
- Playwright browser automation against production
- Auth token injection (via localStorage)
- Console error capture
- HTTP error capture (4xx, 5xx responses)
- Screenshot capability
- Customizable debug steps
Gitignore Patterns
Files matching these patterns are gitignored in frontend/:
test-*.local.*- Ad-hoc test scriptsdebug-*.local.*- Debug scripts
Never commit tokens. Use env vars or .local files only.
When to Use What
| Scenario | Approach |
|---|---|
| Quick one-off debugging | Copy template → .local file |
| Reusable production test | Add to e2e/production/ with env var auth |
| Regression test (mocked) | Add to e2e/regression/ |
Example: Debug a Specific Job's Review Page
// frontend/test-review-debug.local.mjs
import { chromium } from 'playwright';
const TOKEN = 'your-token-here'; // OK in .local files
const JOB_ID = '26391a79';
async function debug() {
const browser = await chromium.launch({ headless: false }); // See the browser
const page = await browser.newContext().then(c => c.newPage());
await page.addInitScript(t => localStorage.setItem('karaoke_access_token', t), TOKEN);
await page.goto(`https://gen.nomadkaraoke.com/app/jobs#/${JOB_ID}/review`);
// Debug interactively - the browser stays open
await page.pause(); // Opens Playwright inspector
await browser.close();
}
debug();
Running Tests Locally
# All backend/package tests
make test
# Specific suites
make test-unit # Package unit tests (69% coverage)
make test-backend # Backend tests only
make test-e2e # E2E with emulators
# With coverage report
poetry run pytest tests/unit --cov=karaoke_gen --cov-report=html
open htmlcov/index.html
# Frontend tests
cd frontend
npm run test:unit # Jest watch mode
npm run test:ci # Jest with coverage
npm run test:e2e # Playwright regression
npm run test:all # Unit + E2E combined
Test File Organization
karaoke-gen/
├── tests/
│ ├── unit/ # CLI package unit tests
│ │ ├── conftest.py # Shared fixtures
│ │ ├── test_audio.py # Audio separation tests
│ │ ├── test_karaoke_finalise/ # Finalize module tests
│ │ └── ...
│ ├── integration/ # CLI integration tests
│ │ ├── test_cli_edit_lyrics.py # Edit lyrics workflow
│ │ └── test_cli_backend_integration.py
│ └── conftest.py # Root fixtures
├── backend/
│ └── tests/
│ ├── conftest.py # FastAPI TestClient, mocks
│ ├── test_job_manager.py # Job management tests
│ ├── test_dependencies.py # DI tests
│ ├── test_made_for_you_webhook.py # Webhook tests
│ └── emulator/ # GCP emulator tests
│ ├── conftest.py # Emulator setup
│ ├── test_emulator_integration.py
│ └── test_worker_logs_direct.py
└── frontend/
├── components/__tests__/ # Jest component tests
│ ├── JobCard.test.tsx
│ ├── ThemeSelector.test.tsx
│ └── ...
├── e2e/
│ ├── fixtures/ # Shared test helpers
│ │ ├── test-helper.ts # API mocking utilities
│ │ └── mock-responses.ts # Canned API responses
│ ├── regression/ # Mocked API E2E (CI)
│ │ ├── smoke.spec.ts
│ │ ├── authentication.spec.ts
│ │ ├── karaoke-generation.spec.ts
│ │ └── ...
│ ├── production/ # Real backend E2E
│ │ ├── happy-path-real-user.spec.ts
│ │ ├── lyrics-review-only.spec.ts
│ │ └── admin-dashboard.spec.ts
│ └── manual/ # Manual setup required (NOT in CI)
│ ├── README.md # Setup instructions
│ └── lyrics-review-mobile.spec.ts # LyricsTranscriber frontend
├── jest.config.js
├── jest.setup.js
├── playwright.config.ts # Default (regression)
└── playwright.production.config.ts
Writing Good Tests
Do
- Test behavior, not implementation
- Use descriptive test names:
test_create_job_applies_default_theme_when_none_specified - One assertion per test (when practical)
- Use fixtures for common setup
- Test edge cases and error conditions
- Mock external services (Modal, AudioShake, Genius, etc.)
Don't
- Test private methods directly
- Write tests that depend on execution order
- Use sleep/wait without timeouts
- Test framework code (FastAPI, React, Next.js)
- Skip tests without a reason
Coverage Enforcement
Coverage is enforced in CI for the CLI package:
# Package unit tests require 69% coverage
poetry run pytest tests/unit/ --cov=karaoke_gen --cov-fail-under=69
To check coverage locally:
# CLI package
poetry run pytest tests/unit --cov=karaoke_gen --cov-report=term-missing
# Backend (no threshold, but useful to see)
poetry run pytest backend/tests --cov=backend --cov-report=term-missing
# Frontend (via Jest)
cd frontend && npm run test:ci # Includes coverage
Mocking Guidelines
When to Mock
- External APIs (Modal, AudioShake, Genius, Spotify lyrics, YouTube)
- GCS/Firestore in unit tests
- FFmpeg in unit tests
- Time-dependent operations
- Network requests
When NOT to Mock
- Your own code (test the real thing)
- Simple data transformations
- In emulator tests (use real Firestore/GCS emulators)
- In production E2E tests (use real backend)
Mock Example
from unittest.mock import Mock, patch
def test_fetch_audio_handles_modal_timeout():
with patch('karaoke_gen.audio.modal_api.submit_job') as mock_submit:
mock_submit.side_effect = TimeoutError("Modal API timeout")
with pytest.raises(AudioSeparationError):
separate_audio("test.mp3")
Frontend Mock Example
// Using Playwright route interception
test('shows error on API failure', async ({ page }) => {
await page.route('**/api/jobs', route => {
route.fulfill({ status: 500, body: 'Server Error' });
});
await page.goto('/app');
await expect(page.getByText('Error loading jobs')).toBeVisible();
});
Debugging Failed Tests
Local Debugging
# Run single test with verbose output
poetry run pytest tests/unit/test_audio.py::test_separate_audio -vvs
# Drop into debugger on failure
poetry run pytest --pdb
# Show local variables on failure
poetry run pytest -l
# Frontend: Debug specific test
cd frontend && npx playwright test smoke.spec.ts --debug
CI Debugging
- Check the GitHub Actions logs
- Look for the specific assertion that failed
- Reproduce locally with the same command
- If flaky, check for timing/ordering issues
- Download test artifacts (screenshots, videos) from failed E2E runs
Adding Tests for New Features
When adding a new feature:
- Write tests first (TDD) or immediately after
- Unit tests for business logic in
karaoke_gen/orbackend/ - Backend tests for new API endpoints
- Frontend unit tests for new components
- E2E regression tests for UI flows with mocked API (add to
e2e/regression/) - Production E2E test for full workflow verification (add to
e2e/production/) - Update this doc if new patterns emerge
Remember: If your verification plan includes manual steps, write a production E2E test instead. See Critical: Automate Manual Testing Steps.
Quick Reference
| Task | Command |
|---|---|
| All tests (before commit) | make test |
| Backend only | make test-backend |
| Frontend only | make test-frontend |
| Package unit | make test-unit |
| Emulator tests | make test-e2e |
| Frontend E2E (production) | cd frontend && npm run test:e2e:prod |
| Coverage HTML | poetry run pytest tests/unit --cov=karaoke_gen --cov-report=html |
| Install deps | make install |
| Start emulators | make emulators-start |
| Stop emulators | make emulators-stop |
New Guard / New Enforcement Checklist
Added after the NOMAD-1632 incident pair (2026-09-11/12). The fix for a silent partial-publish bug became a louder outage: a new completeness guard inside
GCEEncodingBackend.encode()conflated an empty (recoverable, stale-cache) result with a partial (defective) one and shipped enforce-first with only mocked unit tests. Both the original bug and its fix lived in the encode→publish seam, which had no composed integration test. Seedocs/archive/2026-09-12-incident-hardening-plan.mdfor the full post-mortem.
Any new enforcement on a critical path (a check that can fail/block a job, reject a result, or change what gets published) MUST satisfy all of the following before it enforces:
- Distinguish "nothing yet / recoverable" from "wrong result." An empty, missing,
not-ready, or cached-but-stale input is usually a recoverable condition some
other layer already handles — it is NOT the same as a present-but-incorrect result.
Guarding the recoverable case short-circuits recovery and manufactures a new
failure mode (exactly Failure B: the guard fired on empty
output_files, which the orchestrator already recovers from by re-encoding). - Default to prior behavior on uncertainty — fail OPEN unless you are sure it's a defect. If you can't positively classify the input as wrong, let it through and log, rather than blocking. A guard that is wrong in the blocking direction is an outage; one that is wrong in the permissive direction degrades to the (known) prior behavior.
- Test against empty / partial / malformed / complete inputs — not just the happy shape. The data shape that broke us (empty) was simply never in the test corpus.
- Ship in SHADOW mode first (log + alert, do NOT fail/block), and enforce only after a clean shadow window (~1–2 weeks of real traffic with no false positives). Precedent: the timing gate's "G3 shadow" rollout.
- Have a composed integration test, not only isolated unit mocks. Wire the real
producer to the real consumer across the seam the guard lives in; patch only the
true external boundary (network/GCS). A whole-component mock proves nothing about
the seam. Reference implementation:
TestEncodePublishSeaminbackend/tests/test_video_worker_orchestrator.py(realGCEEncodingBackend.encode()through the real orchestrator) and the golden-fixture contract test inbackend/tests/test_encoding_contract.py.
Critical-Path Change Tier
Some code paths are critical path: a silent or loud defect there directly damages customer deliverables or revenue. Treat encode, publish, and distribution changes (and anything they call at that seam) as critical-path.
A change tagged critical-path requires, before it enforces new behavior:
- A composed integration test of the affected seam (producer + consumer, real, with only the external boundary mocked) — see the New Guard checklist item 5.
- Shadow-first rollout for any new enforcement (checklist item 4).
- A real-job verification in production (manual, or the daily E2E) before the change is trusted at 100% (a fresh happy-path job plus, where relevant, an admin-reset + retry to exercise the recovery path).
Mock drift is the recurring root cause here: every encoding mock was hand-authored and
the real "cached/empty" shape was never captured. Critical-path seams should be backed
by golden fixtures captured from real responses and a typed schema shared by
caller and callee (e.g. GceEncodeResponse in backend/services/encoding_interface.py
with fixtures under backend/tests/fixtures/encoding/), so caller and worker can't
drift apart silently.
Blameless Post-Mortem Template
Keep a running incident log in docs/ (archive the detailed write-up under
docs/archive/YYYY-MM-DD-<incident>.md; the NOMAD-1632 hardening plan is the first
entry). For each incident, capture — blamelessly, focused on the system and process,
not the person:
# Incident: <short title> (<date>)
## Summary
One or two sentences: what broke, who/what was affected, blast radius, duration.
## Impact
Jobs/customers affected, data integrity, revenue, how it was detected and when.
## Timeline
- <ts> trigger / deploy
- <ts> first symptom
- <ts> detected (by whom/what)
- <ts> mitigated
- <ts> resolved
## Root cause
The actual defect AND the process gap that let it reach prod (e.g. "no composed test
of the seam", "guard enforced without shadow"). Distinguish recoverable vs defective
if a guard/enforcement was involved.
## What went well / what went poorly
## Action items (with owners, tracked in BACKLOG.md)
- Detection, guardrail, test, deploy, and process fixes — prefer the cheapest net that
catches the whole *class*, not just this instance.