## Summary - Add prominent guidance in TESTING.md about converting manual testing steps into automated Playwright E2E tests - Emphasize that production E2E tests are valuable even if only run once after deployment - Update CLAUDE.md to surface this guidance early for agents ## Changes - **docs/TESTING.md**: Added new "Critical: Automate Manual Testing Steps" section with: - Clear directive to convert manual steps to E2E tests - Explanation of why production E2E tests are valuable - Workflow for production-only E2E tests - Before/after example showing conversion - Decision table for test type selection - Updated "Adding Tests for New Features" section - **CLAUDE.md**: Added callout in Testing section pointing to new guidance ## Test plan - [x] Docs-only change, no code affected - [x] Ran `make test` to verify no regressions @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.ai/code) Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
20 KiB
Testing & Code Quality Guide
This document defines testing standards, code quality principles, and CI requirements for Nomad Karaoke Gen.
Critical: All Tests Must Pass
All tests must pass before merging. If tests fail:
- Fix them - Don't assume failures are "pre-existing issues"
- Investigate - Understand why the test is failing
- Don't dismiss - If a test fails, either the code or test needs fixing
CI blocks merging if tests fail. If you encounter failures locally, fix them as part of your work.
Running Tests
# Run ALL tests (backend + frontend) - installs deps automatically
make test
# Run subsets for faster iteration
make test-backend # Backend only (~2 min)
make test-frontend # Frontend only (~3 min)
Dependencies are installed automatically when needed.
Critical: Automate Manual Testing Steps
If your plan includes "Manual Testing" steps, convert them to automated Playwright E2E tests.
When you write a plan with steps like "manually verify the dialog opens" or "test that the button appears for complete jobs", these are perfect candidates for automated E2E tests. Don't leave verification as a manual step that may never happen.
Why Automate?
- Codifies expected behavior - The test documents exactly what should happen
- Repeatable verification - Can re-run after future changes to catch regressions
- Actually gets executed - Manual steps are often skipped; automated tests run reliably
- Proves the feature works - Passing test = working feature in production
Production E2E Tests Are Valuable
It's acceptable (and preferred) to write E2E tests that only run against production:
// frontend/e2e/production/admin-delete-outputs.spec.ts
test('Admin can delete job outputs', async ({ page }) => {
// This test runs against real production after deployment
// It verifies the feature actually works end-to-end
await page.goto(`${PROD_URL}/admin/jobs/${testJobId}`);
await page.click('[data-testid="delete-outputs-button"]');
await expect(page.getByText('Confirm deletion')).toBeVisible();
// ... complete the flow
});
Even if the test is only run once after deployment, it's still valuable because:
- It codifies the expected behavior completely
- It proves the deployed code works
- It can be re-run if issues are suspected
- It serves as documentation for the feature
Workflow for Production-Only E2E Tests
- Implement the feature with unit tests and regression E2E tests (mocked API)
- Write production E2E test that exercises the real flow
- Merge and deploy to production
- Run the production E2E test to verify:
npm run test:e2e:prod -- admin-delete-outputs.spec.ts - Commit passed - Feature is verified working in production
Example: Converting Manual Steps to E2E
Before (manual testing in plan):
Manual testing:
- Create/complete a test job with YouTube/Dropbox upload
- Navigate to admin job detail
- Click "Delete All Outputs" button
- Verify dialog shows, confirm deletion
- Verify "Outputs Deleted" badge appears
After (automated production E2E):
test('Delete outputs flow', async ({ page }) => {
// Use existing complete job or create one via API
const jobId = process.env.E2E_COMPLETE_JOB_ID;
await page.goto(`${PROD_URL}/admin/jobs/${jobId}`);
await page.click('[data-testid="delete-outputs-button"]');
await expect(page.getByRole('dialog')).toBeVisible();
await page.click('[data-testid="confirm-delete"]');
await expect(page.getByText('Outputs Deleted')).toBeVisible();
});
When to Use Each Test Type
| Scenario | Test Type | Location |
|---|---|---|
| UI component behavior | Jest unit test | components/__tests__/ |
| UI flow with mocked API | Playwright regression | e2e/regression/ |
| Full flow against real backend | Playwright production | e2e/production/ |
| API endpoint logic | pytest unit test | backend/tests/ |
| Database operations | Emulator test | backend/tests/emulator/ |
Default to production E2E for any feature that involves user-facing workflows. The goal is automated verification of deployed code, not checkbox completion of manual steps.
Core Principles
SOLID Principles
All code should follow SOLID principles:
- Single Responsibility - Each class/function does one thing well
- Open/Closed - Open for extension, closed for modification
- Liskov Substitution - Subtypes must be substitutable for base types
- Interface Segregation - Many specific interfaces over one general interface
- Dependency Inversion - Depend on abstractions, not concretions
Code Quality Standards
- Maintainability - Code should be easy to understand and modify
- Testability - Code should be designed for easy testing (dependency injection, pure functions)
- No magic numbers - Use named constants
- Explicit over implicit - Clear is better than clever
- DRY but not prematurely - Don't repeat yourself, but don't abstract too early
Test Types
Unit Tests
Locations:
tests/unit/- CLI package (karaoke_gen/) unit testsbackend/tests/- Backend API unit tests (excludesemulator/)frontend/components/__tests__/- React component unit tests
Characteristics:
- Fast (< 1ms per test typical)
- No external dependencies (mocked)
- No network, database, or filesystem access
- Run on every commit
Coverage targets:
karaoke_gen/package: 69%+backend/package: No explicit threshold (but coverage collected)frontend/components: No explicit threshold (but coverage collected)
# Good unit test example (backend/tests/test_job_manager.py)
def test_create_job_applies_default_theme(mock_firestore, mock_gcs):
job_manager = JobManager()
job = job_manager.create_job(
artist="Test Artist",
title="Test Song",
user_email="test@example.com"
)
assert job.theme_id == "nomad" # Default theme applied
Integration Tests (tests/integration/)
Purpose: Test CLI workflows and component interactions.
Characteristics:
- May mock high-level components
- Tests full command pipelines
- Uses
@pytest.mark.integrationmarker - Often async (
@pytest.mark.asyncio)
Example scenarios:
- CLI
--edit-lyricsflow end-to-end - Job creation → processing pipeline
- Audio separation → transcription coordination
Emulator Tests (backend/tests/emulator/)
Purpose: Test real Firestore/GCS behavior without production access.
Characteristics:
- Requires GCP emulators running
- Tests actual database operations
- Verifies subcollections, TTL policies, etc.
- Auto-skips if emulators unavailable
Setup:
# Start emulators
make emulators-start
# Run emulator tests
poetry run pytest backend/tests/emulator/ -v
# Stop emulators
make emulators-stop
E2E Tests (Frontend)
Locations:
frontend/e2e/regression/- Mocked API tests (CI-safe, fast)frontend/e2e/production/- Real backend tests (comprehensive, slower)frontend/e2e/manual/- Tests requiring manual setup (not in CI)
Tool: Playwright
Regression tests run on every PR:
- Mock API responses via fixtures
- Test UI behavior in isolation
- Playwright auto-starts dev server via
webServerconfig - ~3 min total runtime
Production tests run on-demand or daily:
- Hit real
https://gen.nomadkaraoke.comandhttps://api.nomadkaraoke.com - Full user journeys (20-40 min for happy path)
- Require authentication tokens
Manual tests are NOT included in CI:
- Require separate dev servers (e.g., LyricsTranscriber frontend on port 5173)
- See
frontend/e2e/manual/README.mdfor setup instructions
Playwright Usage
Installation
cd frontend
npm install -D @playwright/test
npx playwright install
Running E2E Tests
# Regression tests (mocked API, default)
npm run test:e2e
# Production tests (real backend)
npm run test:e2e:prod
# With UI mode for debugging
npm run test:e2e:ui
# Specific test file
npx playwright test e2e/regression/smoke.spec.ts
# Production test with environment
E2E_TEST_TOKEN=xxx npm run test:e2e:prod
Investigating Frontend Issues
When debugging frontend issues, use Playwright to:
- Reproduce the issue - Write a test that fails
- Debug visually - Use
--uimode or--headed - Capture evidence - Screenshots, traces, console logs
- Verify the fix - Test passes after fix
# Debug mode with browser visible
npm run test:e2e:headed
# Generate trace for failed tests
npx playwright test --trace on
Writing E2E Tests
Regression test example (mocked API):
// frontend/e2e/regression/smoke.spec.ts
import { test, expect } from '@playwright/test';
import { setupApiFixtures, clearAuthToken } from '../fixtures/test-helper';
test.describe('Smoke Tests', () => {
test.beforeEach(async ({ page }) => {
await clearAuthToken(page);
});
test('landing page loads and shows hero', async ({ page }) => {
await setupApiFixtures(page, { mocks: [] });
await page.goto('/');
await expect(page.locator('h1')).toContainText('Karaoke');
});
});
Production test example (real backend):
// frontend/e2e/production/lyrics-review-only.spec.ts
import { test, expect } from '@playwright/test';
const PROD_URL = 'https://gen.nomadkaraoke.com';
test.describe('Lyrics Review', () => {
test('Complete lyrics review via UI', async ({ page }) => {
test.setTimeout(900_000); // 15 min - preview generation can be slow
const jobId = process.env.E2E_JOB_ID;
await page.goto(`${PROD_URL}/review/${jobId}`);
// ... test implementation
});
});
CI Pipeline
Required Checks (Block PR Merge)
All PRs must pass these checks (path-filtered for efficiency):
| Check | Description | Threshold |
|---|---|---|
backend-lint |
Ruff linter | No errors |
backend-unit-tests |
Backend tests (excludes emulator) | Must pass |
backend-emulator-tests |
GCP emulator integration | Must pass |
package-unit-tests |
CLI package tests | 69% coverage |
package-integration-tests |
CLI integration tests | Must pass |
frontend-lint |
ESLint (Rules of Hooks) | No errors |
frontend-unit-tests |
Jest component tests | Must pass |
frontend-e2e-smoke |
Playwright smoke tests | Must pass |
frontend-build |
Next.js build | Must pass |
On-Demand Checks
| Check | Trigger | Description |
|---|---|---|
frontend-e2e-full |
e2e PR label |
Full regression E2E suite |
e2e-happy-path |
Daily 6 AM UTC | Complete production happy path |
Path Filtering
CI intelligently runs only relevant tests based on changed files:
| Changed Path | Tests Run |
|---|---|
frontend/ |
Frontend lint, unit, E2E |
backend/ |
Backend lint, unit, emulator |
karaoke_gen/ |
Package unit, integration |
tests/ |
Package tests |
infrastructure/ |
Infrastructure validation |
Known Testing Gaps
Infrastructure dependencies (Modal, AudioShake, Cloud Tasks, Firestore) are mocked in tests. This means IAM permission errors or API changes won't be caught until production. Mitigations:
- Emulator tests - Use
backend/tests/emulator/for Firestore/GCS integration - Manual verification after infrastructure changes (Pulumi updates)
- Health endpoints -
/api/healthand/api/health/detailedvalidate connectivity - Daily E2E -
e2e-happy-path.ymlruns full production flow
Production E2E Testing
For comprehensive production testing:
cd frontend
# Run full happy path (requires tokens)
E2E_TEST_TOKEN=<token> npx playwright test e2e/production/happy-path-real-user.spec.ts \
--config=playwright.production.config.ts
# Run focused tests (faster iteration)
E2E_JOB_ID=<id> E2E_REVIEW_TOKEN=<token> npx playwright test e2e/production/lyrics-review-only.spec.ts \
--config=playwright.production.config.ts
# Admin dashboard tests
KARAOKE_ADMIN_TOKEN=<admin-token> npx playwright test e2e/production/admin-dashboard.spec.ts \
--config=playwright.production.config.ts
Environment Variables
| Variable | Purpose | Required For |
|---|---|---|
E2E_TEST_TOKEN |
User authentication | Most production tests |
KARAOKE_ADMIN_TOKEN |
Admin authentication | Admin dashboard tests |
E2E_JOB_ID |
Existing job to test | Focused tests |
E2E_REVIEW_TOKEN |
Review page access | Lyrics review tests |
TESTMAIL_API_KEY |
TestMail.app API | Magic link auth tests |
TESTMAIL_NAMESPACE |
TestMail.app namespace | Magic link auth tests |
Getting Test Tokens
Option 1: From browser
- Login to https://gen.nomadkaraoke.com
- Open DevTools → Application → Local Storage
- Copy the
karaoke_access_tokenvalue
Option 2: From backend logs When the daily E2E runs, tokens are logged for debugging.
The production tests cover:
- Magic link authentication flow
- Audio search and selection
- YouTube URL processing
- File upload workflow
- Lyrics review and correction
- Preview video generation
- Instrumental selection
- Final encoding and download
- Admin dashboard functionality
Running Tests Locally
# All backend/package tests
make test
# Specific suites
make test-unit # Package unit tests (69% coverage)
make test-backend # Backend tests only
make test-e2e # E2E with emulators
# With coverage report
poetry run pytest tests/unit --cov=karaoke_gen --cov-report=html
open htmlcov/index.html
# Frontend tests
cd frontend
npm run test:unit # Jest watch mode
npm run test:ci # Jest with coverage
npm run test:e2e # Playwright regression
npm run test:all # Unit + E2E combined
Test File Organization
karaoke-gen/
├── tests/
│ ├── unit/ # CLI package unit tests
│ │ ├── conftest.py # Shared fixtures
│ │ ├── test_audio.py # Audio separation tests
│ │ ├── test_karaoke_finalise/ # Finalize module tests
│ │ └── ...
│ ├── integration/ # CLI integration tests
│ │ ├── test_cli_edit_lyrics.py # Edit lyrics workflow
│ │ └── test_cli_backend_integration.py
│ └── conftest.py # Root fixtures
├── backend/
│ └── tests/
│ ├── conftest.py # FastAPI TestClient, mocks
│ ├── test_job_manager.py # Job management tests
│ ├── test_dependencies.py # DI tests
│ ├── test_made_for_you_webhook.py # Webhook tests
│ └── emulator/ # GCP emulator tests
│ ├── conftest.py # Emulator setup
│ ├── test_emulator_integration.py
│ └── test_worker_logs_direct.py
└── frontend/
├── components/__tests__/ # Jest component tests
│ ├── JobCard.test.tsx
│ ├── ThemeSelector.test.tsx
│ └── ...
├── e2e/
│ ├── fixtures/ # Shared test helpers
│ │ ├── test-helper.ts # API mocking utilities
│ │ └── mock-responses.ts # Canned API responses
│ ├── regression/ # Mocked API E2E (CI)
│ │ ├── smoke.spec.ts
│ │ ├── authentication.spec.ts
│ │ ├── karaoke-generation.spec.ts
│ │ └── ...
│ ├── production/ # Real backend E2E
│ │ ├── happy-path-real-user.spec.ts
│ │ ├── lyrics-review-only.spec.ts
│ │ └── admin-dashboard.spec.ts
│ └── manual/ # Manual setup required (NOT in CI)
│ ├── README.md # Setup instructions
│ └── lyrics-review-mobile.spec.ts # LyricsTranscriber frontend
├── jest.config.js
├── jest.setup.js
├── playwright.config.ts # Default (regression)
└── playwright.production.config.ts
Writing Good Tests
Do
- Test behavior, not implementation
- Use descriptive test names:
test_create_job_applies_default_theme_when_none_specified - One assertion per test (when practical)
- Use fixtures for common setup
- Test edge cases and error conditions
- Mock external services (Modal, AudioShake, Genius, etc.)
Don't
- Test private methods directly
- Write tests that depend on execution order
- Use sleep/wait without timeouts
- Test framework code (FastAPI, React, Next.js)
- Skip tests without a reason
Coverage Enforcement
Coverage is enforced in CI for the CLI package:
# Package unit tests require 69% coverage
poetry run pytest tests/unit/ --cov=karaoke_gen --cov-fail-under=69
To check coverage locally:
# CLI package
poetry run pytest tests/unit --cov=karaoke_gen --cov-report=term-missing
# Backend (no threshold, but useful to see)
poetry run pytest backend/tests --cov=backend --cov-report=term-missing
# Frontend (via Jest)
cd frontend && npm run test:ci # Includes coverage
Mocking Guidelines
When to Mock
- External APIs (Modal, AudioShake, Genius, Spotify lyrics, YouTube)
- GCS/Firestore in unit tests
- FFmpeg in unit tests
- Time-dependent operations
- Network requests
When NOT to Mock
- Your own code (test the real thing)
- Simple data transformations
- In emulator tests (use real Firestore/GCS emulators)
- In production E2E tests (use real backend)
Mock Example
from unittest.mock import Mock, patch
def test_fetch_audio_handles_modal_timeout():
with patch('karaoke_gen.audio.modal_api.submit_job') as mock_submit:
mock_submit.side_effect = TimeoutError("Modal API timeout")
with pytest.raises(AudioSeparationError):
separate_audio("test.mp3")
Frontend Mock Example
// Using Playwright route interception
test('shows error on API failure', async ({ page }) => {
await page.route('**/api/jobs', route => {
route.fulfill({ status: 500, body: 'Server Error' });
});
await page.goto('/app');
await expect(page.getByText('Error loading jobs')).toBeVisible();
});
Debugging Failed Tests
Local Debugging
# Run single test with verbose output
poetry run pytest tests/unit/test_audio.py::test_separate_audio -vvs
# Drop into debugger on failure
poetry run pytest --pdb
# Show local variables on failure
poetry run pytest -l
# Frontend: Debug specific test
cd frontend && npx playwright test smoke.spec.ts --debug
CI Debugging
- Check the GitHub Actions logs
- Look for the specific assertion that failed
- Reproduce locally with the same command
- If flaky, check for timing/ordering issues
- Download test artifacts (screenshots, videos) from failed E2E runs
Adding Tests for New Features
When adding a new feature:
- Write tests first (TDD) or immediately after
- Unit tests for business logic in
karaoke_gen/orbackend/ - Backend tests for new API endpoints
- Frontend unit tests for new components
- E2E regression tests for UI flows with mocked API (add to
e2e/regression/) - Production E2E test for full workflow verification (add to
e2e/production/) - Update this doc if new patterns emerge
Remember: If your verification plan includes manual steps, write a production E2E test instead. See Critical: Automate Manual Testing Steps.
Quick Reference
| Task | Command |
|---|---|
| All tests (before commit) | make test |
| Backend only | make test-backend |
| Frontend only | make test-frontend |
| Package unit | make test-unit |
| Emulator tests | make test-e2e |
| Frontend E2E (production) | cd frontend && npm run test:e2e:prod |
| Coverage HTML | poetry run pytest tests/unit --cov=karaoke_gen --cov-report=html |
| Install deps | make install |
| Start emulators | make emulators-start |
| Stop emulators | make emulators-stop |