## Summary Phases 2–5 of the concurrent-load reliability initiative (Phase 1 = #1024, verified live: the 10-tab meltdown went from 9/30 failed requests to 30/30 OK). Verbatim spec: `docs/archive/2026-09-17-concurrent-review-reliability-verbatim-record.md`; plan: `docs/archive/2026-09-18-concurrent-review-reliability-investigation-and-plan.md`. ## P2 — Cloud Run shape - **`--concurrency 80 → 16`** for karaoke-backend: the service is one uvicorn process per 2-vCPU instance, and review loads are heavy; at 80 a burst never triggers scale-out and all lands on the single min-instance. 16 makes bursts fan out while staying well above normal per-instance load. (Deploy flags live in ci.yml — the service is not Pulumi-managed.) - **Local preview ffmpeg fallback bounded** (`LOCAL_PREVIEW_MAX_CONCURRENCY=1`, 900s queue timeout, fast RuntimeError after that): the GCE-outage fallback runs ffmpeg ON the API instance; unbounded, a few concurrent Preview clicks during an encoder outage pegged both vCPUs, stalled the health probe, and showed everyone the orange banner (Andrew's banner-during-preview screenshot). ## P3 — degradation telemetry (the explicit ask) - New **`POST /api/client-events`** (unauthenticated so review-token users report too; 30/min/IP rate limit; bot-filtered; detail capped; URLs query-stripped). Each event → Firestore `client_events` collection **and** a structured `client_event type=…` log line, so "how often are users seeing this?" is reviewable via Firestore aggregation or Cloud Logging correlation. - Frontend reporter `lib/degradation-events.ts` (60s per-type throttle, localhost no-op, never throws), wired into: - `BackendStatusBanner` → `banner_reconnecting` / `banner_unavailable` with stall age, in-flight count, probe state - review lyrics-load failure → `lyrics_load_failed` with message/status - `VocalsAudioDataLoader` → `waveform_slow` (>20s to first strips) and `waveform_failed` (gave up) with poll/retry counts ## P4 — banner tuning - Escalating to the full "We're having trouble reaching our servers" card now requires **two consecutive failed health probes** (`UNAVAILABLE_PROBE_FAILURES=2`). One failed 4s probe against a briefly-pegged instance shows only the calm "Reconnecting…" pill. Single-failure behavior for the pill is unchanged, as is instant clearing on recovery. ## P5 — regression guard - **`scripts/load-test-review.py`** checked in: replays the 10-tab incident (Firestore-ADC review tokens, curl request trios, pass/fail thresholds, exit code). Documented in `docs/TESTING.md`. Validated against prod post-Phase-1: 9/9 requests OK, tabs < 3s. ## Testing - Backend: 6 new tests for client-events (persist/validate/rate-limit/bot/fail-soft/detail-cap) + local-preview bound test — full suite **4517 passed**. - Frontend: new degradation-events suite, two-probe escalation + debug-snapshot tests; one existing api-fetch test updated to the new escalation contract — full suite **1298 passed**. - Live validation of the load-test tool against prod (small 3-tab run): PASS. ## Notes for review - Cloud Run concurrency change means bursts now spin up extra instances (billable during requests only; max-instances unchanged at 20). - Firestore `client_events` has no TTL policy yet (needs a gcloud/console action our read-only ADC can't do); volume is tiny (throttled per client) — can add later if it ever matters. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added automatic reporting for degraded experiences, including reconnecting or unavailable backend status, failed lyrics loads, and slow or failed waveform loads. - Added concurrent review-page load testing to help verify performance under multiple open tabs. - **Improvements** - Backend status now remains in “reconnecting” until multiple health checks fail, reducing premature unavailable messages. - Local preview processing is limited during heavy demand to improve stability. - **Documentation** - Added API documentation for client telemetry and guidance for running concurrent-load regression tests. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
30 KiB
Testing & Code Quality Guide
This document defines testing standards, code quality principles, and CI requirements for Nomad Karaoke Gen.
Critical: All Tests Must Pass
All tests must pass before merging. If tests fail:
- Fix them - Don't assume failures are "pre-existing issues"
- Investigate - Understand why the test is failing
- Don't dismiss - If a test fails, either the code or test needs fixing
CI blocks merging if tests fail. If you encounter failures locally, fix them as part of your work.
Running Tests
# Run ALL tests (backend + frontend) - installs deps automatically
make test
# Run subsets for faster iteration
make test-backend # Backend only (~2 min)
make test-frontend # Frontend only (~3 min)
Dependencies are installed automatically when needed.
Critical: Automate Manual Testing Steps
If your plan includes "Manual Testing" steps, convert them to automated Playwright E2E tests.
When you write a plan with steps like "manually verify the dialog opens" or "test that the button appears for complete jobs", these are perfect candidates for automated E2E tests. Don't leave verification as a manual step that may never happen.
Why Automate?
- Codifies expected behavior - The test documents exactly what should happen
- Repeatable verification - Can re-run after future changes to catch regressions
- Actually gets executed - Manual steps are often skipped; automated tests run reliably
- Proves the feature works - Passing test = working feature in production
Production E2E Tests Are Valuable
It's acceptable (and preferred) to write E2E tests that only run against production:
// frontend/e2e/production/admin-delete-outputs.spec.ts
test('Admin can delete job outputs', async ({ page }) => {
// This test runs against real production after deployment
// It verifies the feature actually works end-to-end
await page.goto(`${PROD_URL}/admin/jobs/${testJobId}`);
await page.click('[data-testid="delete-outputs-button"]');
await expect(page.getByText('Confirm deletion')).toBeVisible();
// ... complete the flow
});
Even if the test is only run once after deployment, it's still valuable because:
- It codifies the expected behavior completely
- It proves the deployed code works
- It can be re-run if issues are suspected
- It serves as documentation for the feature
Workflow for Production-Only E2E Tests
- Implement the feature with unit tests and regression E2E tests (mocked API)
- Write production E2E test that exercises the real flow
- Merge and deploy to production
- Run the production E2E test to verify:
npm run test:e2e:prod -- admin-delete-outputs.spec.ts - Commit passed - Feature is verified working in production
Example: Converting Manual Steps to E2E
Before (manual testing in plan):
Manual testing:
- Create/complete a test job with YouTube/Dropbox upload
- Navigate to admin job detail
- Click "Delete All Outputs" button
- Verify dialog shows, confirm deletion
- Verify "Outputs Deleted" badge appears
After (automated production E2E):
test('Delete outputs flow', async ({ page }) => {
// Use existing complete job or create one via API
const jobId = process.env.E2E_COMPLETE_JOB_ID;
await page.goto(`${PROD_URL}/admin/jobs/${jobId}`);
await page.click('[data-testid="delete-outputs-button"]');
await expect(page.getByRole('dialog')).toBeVisible();
await page.click('[data-testid="confirm-delete"]');
await expect(page.getByText('Outputs Deleted')).toBeVisible();
});
When to Use Each Test Type
| Scenario | Test Type | Location |
|---|---|---|
| UI component behavior | Jest unit test | components/__tests__/ |
| UI flow with mocked API | Playwright regression | e2e/regression/ |
| Full flow against real backend | Playwright production | e2e/production/ |
| API endpoint logic | pytest unit test | backend/tests/ |
| Database operations | Emulator test | backend/tests/emulator/ |
Default to production E2E for any feature that involves user-facing workflows. The goal is automated verification of deployed code, not checkbox completion of manual steps.
Core Principles
SOLID Principles
All code should follow SOLID principles:
- Single Responsibility - Each class/function does one thing well
- Open/Closed - Open for extension, closed for modification
- Liskov Substitution - Subtypes must be substitutable for base types
- Interface Segregation - Many specific interfaces over one general interface
- Dependency Inversion - Depend on abstractions, not concretions
Code Quality Standards
- Maintainability - Code should be easy to understand and modify
- Testability - Code should be designed for easy testing (dependency injection, pure functions)
- No magic numbers - Use named constants
- Explicit over implicit - Clear is better than clever
- DRY but not prematurely - Don't repeat yourself, but don't abstract too early
Test Types
Unit Tests
Locations:
tests/unit/- CLI package (karaoke_gen/) unit testsbackend/tests/- Backend API unit tests (excludesemulator/)frontend/components/__tests__/- React component unit tests
Characteristics:
- Fast (< 1ms per test typical)
- No external dependencies (mocked)
- No network, database, or filesystem access
- Run on every commit
Coverage targets:
karaoke_gen/package: 69%+backend/package: No explicit threshold (but coverage collected)frontend/components: No explicit threshold (but coverage collected)
# Good unit test example (backend/tests/test_job_manager.py)
def test_create_job_applies_default_theme(mock_firestore, mock_gcs):
job_manager = JobManager()
job = job_manager.create_job(
artist="Test Artist",
title="Test Song",
user_email="test@example.com"
)
assert job.theme_id == "nomad" # Default theme applied
Critical: Mocks Must Match Real API Contracts
When mocking methods, ensure the mock's return_value matches what the real method returns. A common bug pattern:
# ❌ BAD: Assumes method returns boolean
mock_service.update_job.return_value = True
# Code under test:
success = job_manager.update_job(job_id, updates)
if not success: # BUG: if real method returns None, this is always True!
raise HTTPException(500)
The fix:
# ✅ GOOD: Check real method signature first
# def update_job(self, job_id: str, updates: Dict) -> None:
mock_service.update_job.return_value = None # Match real API
Before writing a mock, check the actual method's return type. Void methods should mock return_value = None. Methods that raise on error (rather than returning False) should have mocks that raise exceptions for error cases. See LESSONS-LEARNED.md for a production bug caused by this.
Critical: Test the Contract Between Caller and Callee
When function A calls function B, it's not enough to test B with hand-crafted ideal inputs. You must also verify A actually produces those inputs. This is a "contract test."
# ❌ BAD: Tests _store_metadata in isolation with hand-crafted input
# If transcribe_lyrics() never includes "lyrics_dir", this test still passes
# but production silently loses all metadata
def test_stores_audioshake_ids(tmp_path):
_store_lyrics_processing_metadata(
transcription_result={"lyrics_dir": str(tmp_path)}, # Too helpful!
...
)
assert call_args["audioshake_task_id"] == "task-abc" # ✅ passes, but useless
# ✅ GOOD: Also test that the caller produces the expected input
def test_transcribe_lyrics_returns_lyrics_dir(tmp_path):
result = processor.transcribe_lyrics(audio, artist, title, tmp_path)
assert "lyrics_dir" in result # Catches the actual bug
When to write contract tests:
- Function A builds a dict/object and passes it to function B
- Function B reads specific keys/fields from that dict/object
- The dict/object shape is implicit (no schema, no TypedDict, no dataclass)
Especially dangerous: Non-fatal code paths (try/except that swallows errors). If the function is designed to never crash, a test that only checks "didn't raise" provides zero signal. Assert on the positive outcome (data was stored, value was returned) not just the absence of exceptions.
See LESSONS-LEARNED.md for the production bug this pattern caused.
Integration Tests (tests/integration/)
Purpose: Test CLI workflows and component interactions.
Characteristics:
- May mock high-level components
- Tests full command pipelines
- Uses
@pytest.mark.integrationmarker - Often async (
@pytest.mark.asyncio)
Example scenarios:
- CLI
--edit-lyricsflow end-to-end - Job creation → processing pipeline
- Audio separation → transcription coordination
Emulator Tests (backend/tests/emulator/)
Purpose: Test real Firestore/GCS behavior without production access.
Characteristics:
- Requires GCP emulators running
- Tests actual database operations
- Verifies subcollections, TTL policies, etc.
- Auto-skips if emulators unavailable
Setup:
# Start emulators
make emulators-start
# Run emulator tests
poetry run pytest backend/tests/emulator/ -v
# Stop emulators
make emulators-stop
E2E Tests (Frontend)
Locations:
frontend/e2e/regression/- Mocked API tests (CI-safe, fast)frontend/e2e/production/- Real backend tests (comprehensive, slower)frontend/e2e/manual/- Tests requiring manual setup (not in CI)
Tool: Playwright
Regression tests run on every PR:
- Mock API responses via fixtures
- Test UI behavior in isolation
- Playwright auto-starts dev server via
webServerconfig - ~3 min total runtime
Production tests run on-demand or daily:
- Hit real
https://gen.nomadkaraoke.comandhttps://api.nomadkaraoke.com - Full user journeys (20-40 min for happy path)
- Require authentication tokens
Manual tests are NOT included in CI:
- Require separate dev servers (e.g., LyricsTranscriber frontend on port 5173)
- See
frontend/e2e/manual/README.mdfor setup instructions
Playwright Usage
Installation
cd frontend
npm install -D @playwright/test
npx playwright install
Running E2E Tests
# Regression tests (mocked API, default)
npm run test:e2e
# Production tests (real backend)
npm run test:e2e:prod
# With UI mode for debugging
npm run test:e2e:ui
# Specific test file
npx playwright test e2e/regression/smoke.spec.ts
# Production test with environment
E2E_TEST_TOKEN=xxx npm run test:e2e:prod
Investigating Frontend Issues
When debugging frontend issues, use Playwright to:
- Reproduce the issue - Write a test that fails
- Debug visually - Use
--uimode or--headed - Capture evidence - Screenshots, traces, console logs
- Verify the fix - Test passes after fix
# Debug mode with browser visible
npm run test:e2e:headed
# Generate trace for failed tests
npx playwright test --trace on
Writing E2E Tests
Regression test example (mocked API):
// frontend/e2e/regression/smoke.spec.ts
import { test, expect } from '@playwright/test';
import { setupApiFixtures, clearAuthToken } from '../fixtures/test-helper';
test.describe('Smoke Tests', () => {
test.beforeEach(async ({ page }) => {
await clearAuthToken(page);
});
test('landing page loads and shows hero', async ({ page }) => {
await setupApiFixtures(page, { mocks: [] });
await page.goto('/');
await expect(page.locator('h1')).toContainText('Karaoke');
});
});
Production test example (real backend):
// frontend/e2e/production/lyrics-review-only.spec.ts
import { test, expect } from '@playwright/test';
const PROD_URL = 'https://gen.nomadkaraoke.com';
test.describe('Lyrics Review', () => {
test('Complete lyrics review via UI', async ({ page }) => {
test.setTimeout(900_000); // 15 min - preview generation can be slow
const jobId = process.env.E2E_JOB_ID;
await page.goto(`${PROD_URL}/review/${jobId}`);
// ... test implementation
});
});
CI Pipeline
Required Checks (Block PR Merge)
All PRs must pass these checks (path-filtered for efficiency):
| Check | Description | Threshold |
|---|---|---|
backend-lint |
Ruff linter | No errors |
backend-unit-tests |
Backend tests (excludes emulator) | Must pass |
backend-emulator-tests |
GCP emulator integration | Must pass |
package-unit-tests |
CLI package tests | 69% coverage |
package-integration-tests |
CLI integration tests | Must pass |
frontend-lint |
ESLint (Rules of Hooks) | No errors |
frontend-unit-tests |
Jest component tests | Must pass |
frontend-e2e-smoke |
Playwright smoke tests | Must pass |
frontend-build |
Next.js build | Must pass |
On-Demand Checks
| Check | Trigger | Description |
|---|---|---|
frontend-e2e-full |
e2e PR label |
Full regression E2E suite |
e2e-happy-path |
Daily 6 AM UTC | Complete production happy path |
Path Filtering
CI intelligently runs only relevant tests based on changed files:
| Changed Path | Tests Run |
|---|---|
frontend/ |
Frontend lint, unit, E2E |
backend/ |
Backend lint, unit, emulator |
karaoke_gen/ |
Package unit, integration |
tests/ |
Package tests |
infrastructure/ |
Infrastructure validation |
Known Testing Gaps
Infrastructure dependencies (Modal, AudioShake, Cloud Tasks, Firestore) are mocked in tests. This means IAM permission errors or API changes won't be caught until production. Mitigations:
- Emulator tests - Use
backend/tests/emulator/for Firestore/GCS integration - Manual verification after infrastructure changes (Pulumi updates)
- Health endpoints -
/api/healthand/api/health/detailedvalidate connectivity - Daily E2E -
e2e-happy-path.ymlruns full production flow
Production E2E Testing
For comprehensive production testing:
cd frontend
# Run full happy path (requires tokens)
E2E_TEST_TOKEN=<token> npx playwright test e2e/production/happy-path-real-user.spec.ts \
--config=playwright.production.config.ts
# Run focused tests (faster iteration)
E2E_JOB_ID=<id> E2E_REVIEW_TOKEN=<token> npx playwright test e2e/production/lyrics-review-only.spec.ts \
--config=playwright.production.config.ts
# Admin dashboard tests
KARAOKE_ADMIN_TOKEN=<admin-token> npx playwright test e2e/production/admin-dashboard.spec.ts \
--config=playwright.production.config.ts
Environment Variables
| Variable | Purpose | Required For |
|---|---|---|
E2E_TEST_TOKEN |
User authentication | Most production tests |
KARAOKE_ADMIN_TOKEN |
Admin authentication | Admin dashboard tests |
E2E_JOB_ID |
Existing job to test | Focused tests |
E2E_REVIEW_TOKEN |
Review page access | Lyrics review tests |
TESTMAIL_API_KEY |
TestMail.app API | Magic link auth tests |
TESTMAIL_NAMESPACE |
TestMail.app namespace | Magic link auth tests |
Getting Test Tokens
Option 1: From browser
- Login to https://gen.nomadkaraoke.com
- Open DevTools → Application → Local Storage
- Copy the
karaoke_access_tokenvalue
Option 2: From backend logs When the daily E2E runs, tokens are logged for debugging.
The production tests cover:
- Magic link authentication flow
- Audio search and selection
- YouTube URL processing
- File upload workflow
- Lyrics review and correction
- Preview video generation
- Instrumental selection
- Final encoding and download
- Admin dashboard functionality
Concurrent-Load Regression Test (review pages)
scripts/load-test-review.py replays the 2026-09-18 "10 tabs melt the backend"
incident: N simultaneous review-tab loads, each firing the page's real GET trio
(correction-data, waveform-data, ranged audio/vocals), authenticated with per-job
review tokens read from Firestore via ADC (no admin secret needed).
python scripts/load-test-review.py # 10 tabs vs prod, warm thresholds
python scripts/load-test-review.py --tabs 20
python scripts/load-test-review.py --max-tab-seconds 45 # cold caches (first-ever waveforms)
Pass = every request 2xx and every tab under the threshold; exit 1 otherwise.
Run it after changes to the review hot path (backend/api/routes/review.py,
audio_transcoding_service, audio_analysis_service) or Cloud Run deploy flags.
Background + before/after numbers:
docs/archive/2026-09-18-concurrent-review-reliability-investigation-and-plan.md.
Ad-Hoc Production Debugging (for Agents)
When asked to "test it yourself in prod" or debug a specific production issue, use the debug script template:
Quick Start
# Copy the template (the .local suffix is gitignored)
cp frontend/e2e/helpers/debug-prod-template.mjs frontend/test-my-issue.local.mjs
# Edit the script to customize what you're testing
# You can hardcode tokens in .local files - they won't be committed
# Run it
cd frontend && node test-my-issue.local.mjs
With Environment Variables (Preferred)
# Get admin token from Secret Manager
export KARAOKE_ADMIN_TOKEN=$(gcloud secrets versions access latest --secret=admin-tokens --project=nomadkaraoke | cut -d',' -f1)
# Run the template directly
cd frontend && node e2e/helpers/debug-prod-template.mjs
# Or with a specific job
DEBUG_JOB_ID=abc123 node e2e/helpers/debug-prod-template.mjs
What the Template Provides
- Playwright browser automation against production
- Auth token injection (via localStorage)
- Console error capture
- HTTP error capture (4xx, 5xx responses)
- Screenshot capability
- Customizable debug steps
Gitignore Patterns
Files matching these patterns are gitignored in frontend/:
test-*.local.*- Ad-hoc test scriptsdebug-*.local.*- Debug scripts
Never commit tokens. Use env vars or .local files only.
When to Use What
| Scenario | Approach |
|---|---|
| Quick one-off debugging | Copy template → .local file |
| Reusable production test | Add to e2e/production/ with env var auth |
| Regression test (mocked) | Add to e2e/regression/ |
Example: Debug a Specific Job's Review Page
// frontend/test-review-debug.local.mjs
import { chromium } from 'playwright';
const TOKEN = 'your-token-here'; // OK in .local files
const JOB_ID = '26391a79';
async function debug() {
const browser = await chromium.launch({ headless: false }); // See the browser
const page = await browser.newContext().then(c => c.newPage());
await page.addInitScript(t => localStorage.setItem('karaoke_access_token', t), TOKEN);
await page.goto(`https://gen.nomadkaraoke.com/app/jobs#/${JOB_ID}/review`);
// Debug interactively - the browser stays open
await page.pause(); // Opens Playwright inspector
await browser.close();
}
debug();
Running Tests Locally
# All backend/package tests
make test
# Specific suites
make test-unit # Package unit tests (69% coverage)
make test-backend # Backend tests only
make test-e2e # E2E with emulators
# With coverage report
poetry run pytest tests/unit --cov=karaoke_gen --cov-report=html
open htmlcov/index.html
# Frontend tests
cd frontend
npm run test:unit # Jest watch mode
npm run test:ci # Jest with coverage
npm run test:e2e # Playwright regression
npm run test:all # Unit + E2E combined
Test File Organization
karaoke-gen/
├── tests/
│ ├── unit/ # CLI package unit tests
│ │ ├── conftest.py # Shared fixtures
│ │ ├── test_audio.py # Audio separation tests
│ │ ├── test_karaoke_finalise/ # Finalize module tests
│ │ └── ...
│ ├── integration/ # CLI integration tests
│ │ ├── test_cli_edit_lyrics.py # Edit lyrics workflow
│ │ └── test_cli_backend_integration.py
│ └── conftest.py # Root fixtures
├── backend/
│ └── tests/
│ ├── conftest.py # FastAPI TestClient, mocks
│ ├── test_job_manager.py # Job management tests
│ ├── test_dependencies.py # DI tests
│ ├── test_made_for_you_webhook.py # Webhook tests
│ └── emulator/ # GCP emulator tests
│ ├── conftest.py # Emulator setup
│ ├── test_emulator_integration.py
│ └── test_worker_logs_direct.py
└── frontend/
├── components/__tests__/ # Jest component tests
│ ├── JobCard.test.tsx
│ ├── ThemeSelector.test.tsx
│ └── ...
├── e2e/
│ ├── fixtures/ # Shared test helpers
│ │ ├── test-helper.ts # API mocking utilities
│ │ └── mock-responses.ts # Canned API responses
│ ├── regression/ # Mocked API E2E (CI)
│ │ ├── smoke.spec.ts
│ │ ├── authentication.spec.ts
│ │ ├── karaoke-generation.spec.ts
│ │ └── ...
│ ├── production/ # Real backend E2E
│ │ ├── happy-path-real-user.spec.ts
│ │ ├── lyrics-review-only.spec.ts
│ │ └── admin-dashboard.spec.ts
│ └── manual/ # Manual setup required (NOT in CI)
│ ├── README.md # Setup instructions
│ └── lyrics-review-mobile.spec.ts # LyricsTranscriber frontend
├── jest.config.js
├── jest.setup.js
├── playwright.config.ts # Default (regression)
└── playwright.production.config.ts
Writing Good Tests
Do
- Test behavior, not implementation
- Use descriptive test names:
test_create_job_applies_default_theme_when_none_specified - One assertion per test (when practical)
- Use fixtures for common setup
- Test edge cases and error conditions
- Mock external services (Modal, AudioShake, Genius, etc.)
Don't
- Test private methods directly
- Write tests that depend on execution order
- Use sleep/wait without timeouts
- Test framework code (FastAPI, React, Next.js)
- Skip tests without a reason
Coverage Enforcement
Coverage is enforced in CI for the CLI package:
# Package unit tests require 69% coverage
poetry run pytest tests/unit/ --cov=karaoke_gen --cov-fail-under=69
To check coverage locally:
# CLI package
poetry run pytest tests/unit --cov=karaoke_gen --cov-report=term-missing
# Backend (no threshold, but useful to see)
poetry run pytest backend/tests --cov=backend --cov-report=term-missing
# Frontend (via Jest)
cd frontend && npm run test:ci # Includes coverage
Mocking Guidelines
When to Mock
- External APIs (Modal, AudioShake, Genius, Spotify lyrics, YouTube)
- GCS/Firestore in unit tests
- FFmpeg in unit tests
- Time-dependent operations
- Network requests
When NOT to Mock
- Your own code (test the real thing)
- Simple data transformations
- In emulator tests (use real Firestore/GCS emulators)
- In production E2E tests (use real backend)
Mock Example
from unittest.mock import Mock, patch
def test_fetch_audio_handles_modal_timeout():
with patch('karaoke_gen.audio.modal_api.submit_job') as mock_submit:
mock_submit.side_effect = TimeoutError("Modal API timeout")
with pytest.raises(AudioSeparationError):
separate_audio("test.mp3")
Frontend Mock Example
// Using Playwright route interception
test('shows error on API failure', async ({ page }) => {
await page.route('**/api/jobs', route => {
route.fulfill({ status: 500, body: 'Server Error' });
});
await page.goto('/app');
await expect(page.getByText('Error loading jobs')).toBeVisible();
});
Debugging Failed Tests
Local Debugging
# Run single test with verbose output
poetry run pytest tests/unit/test_audio.py::test_separate_audio -vvs
# Drop into debugger on failure
poetry run pytest --pdb
# Show local variables on failure
poetry run pytest -l
# Frontend: Debug specific test
cd frontend && npx playwright test smoke.spec.ts --debug
CI Debugging
- Check the GitHub Actions logs
- Look for the specific assertion that failed
- Reproduce locally with the same command
- If flaky, check for timing/ordering issues
- Download test artifacts (screenshots, videos) from failed E2E runs
Adding Tests for New Features
When adding a new feature:
- Write tests first (TDD) or immediately after
- Unit tests for business logic in
karaoke_gen/orbackend/ - Backend tests for new API endpoints
- Frontend unit tests for new components
- E2E regression tests for UI flows with mocked API (add to
e2e/regression/) - Production E2E test for full workflow verification (add to
e2e/production/) - Update this doc if new patterns emerge
Remember: If your verification plan includes manual steps, write a production E2E test instead. See Critical: Automate Manual Testing Steps.
Quick Reference
| Task | Command |
|---|---|
| All tests (before commit) | make test |
| Backend only | make test-backend |
| Frontend only | make test-frontend |
| Package unit | make test-unit |
| Emulator tests | make test-e2e |
| Frontend E2E (production) | cd frontend && npm run test:e2e:prod |
| Coverage HTML | poetry run pytest tests/unit --cov=karaoke_gen --cov-report=html |
| Install deps | make install |
| Start emulators | make emulators-start |
| Stop emulators | make emulators-stop |
New Guard / New Enforcement Checklist
Added after the NOMAD-1632 incident pair (2026-09-11/12). The fix for a silent partial-publish bug became a louder outage: a new completeness guard inside
GCEEncodingBackend.encode()conflated an empty (recoverable, stale-cache) result with a partial (defective) one and shipped enforce-first with only mocked unit tests. Both the original bug and its fix lived in the encode→publish seam, which had no composed integration test. Seedocs/archive/2026-09-12-incident-hardening-plan.mdfor the full post-mortem.
Any new enforcement on a critical path (a check that can fail/block a job, reject a result, or change what gets published) MUST satisfy all of the following before it enforces:
- Distinguish "nothing yet / recoverable" from "wrong result." An empty, missing,
not-ready, or cached-but-stale input is usually a recoverable condition some
other layer already handles — it is NOT the same as a present-but-incorrect result.
Guarding the recoverable case short-circuits recovery and manufactures a new
failure mode (exactly Failure B: the guard fired on empty
output_files, which the orchestrator already recovers from by re-encoding). - Default to prior behavior on uncertainty — fail OPEN unless you are sure it's a defect. If you can't positively classify the input as wrong, let it through and log, rather than blocking. A guard that is wrong in the blocking direction is an outage; one that is wrong in the permissive direction degrades to the (known) prior behavior.
- Test against empty / partial / malformed / complete inputs — not just the happy shape. The data shape that broke us (empty) was simply never in the test corpus.
- Ship in SHADOW mode first (log + alert, do NOT fail/block), and enforce only after a clean shadow window (~1–2 weeks of real traffic with no false positives). Precedent: the timing gate's "G3 shadow" rollout.
- Have a composed integration test, not only isolated unit mocks. Wire the real
producer to the real consumer across the seam the guard lives in; patch only the
true external boundary (network/GCS). A whole-component mock proves nothing about
the seam. Reference implementation:
TestEncodePublishSeaminbackend/tests/test_video_worker_orchestrator.py(realGCEEncodingBackend.encode()through the real orchestrator) and the golden-fixture contract test inbackend/tests/test_encoding_contract.py.
Critical-Path Change Tier
Some code paths are critical path: a silent or loud defect there directly damages customer deliverables or revenue. Treat encode, publish, and distribution changes (and anything they call at that seam) as critical-path.
A change tagged critical-path requires, before it enforces new behavior:
- A composed integration test of the affected seam (producer + consumer, real, with only the external boundary mocked) — see the New Guard checklist item 5.
- Shadow-first rollout for any new enforcement (checklist item 4).
- A post-deploy canary / real-job verification before the change is trusted at 100% (a fresh happy-path job plus, where relevant, an admin-reset + retry to exercise the recovery path).
Mock drift is the recurring root cause here: every encoding mock was hand-authored and
the real "cached/empty" shape was never captured. Critical-path seams should be backed
by golden fixtures captured from real responses and a typed schema shared by
caller and callee (e.g. GceEncodeResponse in backend/services/encoding_interface.py
with fixtures under backend/tests/fixtures/encoding/), so caller and worker can't
drift apart silently.
Blameless Post-Mortem Template
Keep a running incident log in docs/ (archive the detailed write-up under
docs/archive/YYYY-MM-DD-<incident>.md; the NOMAD-1632 hardening plan is the first
entry). For each incident, capture — blamelessly, focused on the system and process,
not the person:
# Incident: <short title> (<date>)
## Summary
One or two sentences: what broke, who/what was affected, blast radius, duration.
## Impact
Jobs/customers affected, data integrity, revenue, how it was detected and when.
## Timeline
- <ts> trigger / deploy
- <ts> first symptom
- <ts> detected (by whom/what)
- <ts> mitigated
- <ts> resolved
## Root cause
The actual defect AND the process gap that let it reach prod (e.g. "no composed test
of the seam", "guard enforced without shadow"). Distinguish recoverable vs defective
if a guard/enforcement was involved.
## What went well / what went poorly
## Action items (with owners, tracked in BACKLOG.md)
- Detection, guardrail, test, deploy, and process fixes — prefer the cheapest net that
catches the whole *class*, not just this instance.