## Summary - **`make test` now runs ALL tests** (backend + frontend) and auto-installs dependencies if missing - Added clear "All tests must pass" guidance - agents should never dismiss failures as "pre-existing" - Fixed misplaced test that was always failing (tested wrong frontend) - Fixed logo locator bug in landing-page test ## Changes | File | Change | |------|--------| | `Makefile` | Auto-install deps, unified `make test`, added `test-backend`/`test-frontend` | | `CLAUDE.md` | Simplified to just `make test` | | `docs/DEVELOPMENT.md` | Consolidated testing section | | `docs/TESTING.md` | Updated guidance and quick reference | | `frontend/e2e/regression/landing-page.spec.ts` | Fixed `getByText` → `getByAltText` for logo | | `frontend/e2e/manual/` | Moved `lyrics-review-mobile.spec.ts` here with README | ## Test plan - [x] `make test` passes all 153 tests (76 backend + 77 frontend) - [x] Auto-install works in fresh worktree - [x] Documentation is clear and consistent @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
16 KiB
Testing & Code Quality Guide
This document defines testing standards, code quality principles, and CI requirements for Nomad Karaoke Gen.
Critical: All Tests Must Pass
All tests must pass before merging. If tests fail:
- Fix them - Don't assume failures are "pre-existing issues"
- Investigate - Understand why the test is failing
- Don't dismiss - If a test fails, either the code or test needs fixing
CI blocks merging if tests fail. If you encounter failures locally, fix them as part of your work.
Running Tests
# Run ALL tests (backend + frontend) - installs deps automatically
make test
# Run subsets for faster iteration
make test-backend # Backend only (~2 min)
make test-frontend # Frontend only (~3 min)
Dependencies are installed automatically when needed.
Core Principles
SOLID Principles
All code should follow SOLID principles:
- Single Responsibility - Each class/function does one thing well
- Open/Closed - Open for extension, closed for modification
- Liskov Substitution - Subtypes must be substitutable for base types
- Interface Segregation - Many specific interfaces over one general interface
- Dependency Inversion - Depend on abstractions, not concretions
Code Quality Standards
- Maintainability - Code should be easy to understand and modify
- Testability - Code should be designed for easy testing (dependency injection, pure functions)
- No magic numbers - Use named constants
- Explicit over implicit - Clear is better than clever
- DRY but not prematurely - Don't repeat yourself, but don't abstract too early
Test Types
Unit Tests
Locations:
tests/unit/- CLI package (karaoke_gen/) unit testsbackend/tests/- Backend API unit tests (excludesemulator/)frontend/components/__tests__/- React component unit tests
Characteristics:
- Fast (< 1ms per test typical)
- No external dependencies (mocked)
- No network, database, or filesystem access
- Run on every commit
Coverage targets:
karaoke_gen/package: 69%+backend/package: No explicit threshold (but coverage collected)frontend/components: No explicit threshold (but coverage collected)
# Good unit test example (backend/tests/test_job_manager.py)
def test_create_job_applies_default_theme(mock_firestore, mock_gcs):
job_manager = JobManager()
job = job_manager.create_job(
artist="Test Artist",
title="Test Song",
user_email="test@example.com"
)
assert job.theme_id == "nomad" # Default theme applied
Integration Tests (tests/integration/)
Purpose: Test CLI workflows and component interactions.
Characteristics:
- May mock high-level components
- Tests full command pipelines
- Uses
@pytest.mark.integrationmarker - Often async (
@pytest.mark.asyncio)
Example scenarios:
- CLI
--edit-lyricsflow end-to-end - Job creation → processing pipeline
- Audio separation → transcription coordination
Emulator Tests (backend/tests/emulator/)
Purpose: Test real Firestore/GCS behavior without production access.
Characteristics:
- Requires GCP emulators running
- Tests actual database operations
- Verifies subcollections, TTL policies, etc.
- Auto-skips if emulators unavailable
Setup:
# Start emulators
make emulators-start
# Run emulator tests
poetry run pytest backend/tests/emulator/ -v
# Stop emulators
make emulators-stop
E2E Tests (Frontend)
Locations:
frontend/e2e/regression/- Mocked API tests (CI-safe, fast)frontend/e2e/production/- Real backend tests (comprehensive, slower)frontend/e2e/manual/- Tests requiring manual setup (not in CI)
Tool: Playwright
Regression tests run on every PR:
- Mock API responses via fixtures
- Test UI behavior in isolation
- Playwright auto-starts dev server via
webServerconfig - ~3 min total runtime
Production tests run on-demand or daily:
- Hit real
https://gen.nomadkaraoke.comandhttps://api.nomadkaraoke.com - Full user journeys (20-40 min for happy path)
- Require authentication tokens
Manual tests are NOT included in CI:
- Require separate dev servers (e.g., LyricsTranscriber frontend on port 5173)
- See
frontend/e2e/manual/README.mdfor setup instructions
Playwright Usage
Installation
cd frontend
npm install -D @playwright/test
npx playwright install
Running E2E Tests
# Regression tests (mocked API, default)
npm run test:e2e
# Production tests (real backend)
npm run test:e2e:prod
# With UI mode for debugging
npm run test:e2e:ui
# Specific test file
npx playwright test e2e/regression/smoke.spec.ts
# Production test with environment
E2E_TEST_TOKEN=xxx npm run test:e2e:prod
Investigating Frontend Issues
When debugging frontend issues, use Playwright to:
- Reproduce the issue - Write a test that fails
- Debug visually - Use
--uimode or--headed - Capture evidence - Screenshots, traces, console logs
- Verify the fix - Test passes after fix
# Debug mode with browser visible
npm run test:e2e:headed
# Generate trace for failed tests
npx playwright test --trace on
Writing E2E Tests
Regression test example (mocked API):
// frontend/e2e/regression/smoke.spec.ts
import { test, expect } from '@playwright/test';
import { setupApiFixtures, clearAuthToken } from '../fixtures/test-helper';
test.describe('Smoke Tests', () => {
test.beforeEach(async ({ page }) => {
await clearAuthToken(page);
});
test('landing page loads and shows hero', async ({ page }) => {
await setupApiFixtures(page, { mocks: [] });
await page.goto('/');
await expect(page.locator('h1')).toContainText('Karaoke');
});
});
Production test example (real backend):
// frontend/e2e/production/lyrics-review-only.spec.ts
import { test, expect } from '@playwright/test';
const PROD_URL = 'https://gen.nomadkaraoke.com';
test.describe('Lyrics Review', () => {
test('Complete lyrics review via UI', async ({ page }) => {
test.setTimeout(900_000); // 15 min - preview generation can be slow
const jobId = process.env.E2E_JOB_ID;
await page.goto(`${PROD_URL}/review/${jobId}`);
// ... test implementation
});
});
CI Pipeline
Required Checks (Block PR Merge)
All PRs must pass these checks (path-filtered for efficiency):
| Check | Description | Threshold |
|---|---|---|
backend-lint |
Ruff linter | No errors |
backend-unit-tests |
Backend tests (excludes emulator) | Must pass |
backend-emulator-tests |
GCP emulator integration | Must pass |
package-unit-tests |
CLI package tests | 69% coverage |
package-integration-tests |
CLI integration tests | Must pass |
frontend-lint |
ESLint (Rules of Hooks) | No errors |
frontend-unit-tests |
Jest component tests | Must pass |
frontend-e2e-smoke |
Playwright smoke tests | Must pass |
frontend-build |
Next.js build | Must pass |
On-Demand Checks
| Check | Trigger | Description |
|---|---|---|
frontend-e2e-full |
e2e PR label |
Full regression E2E suite |
e2e-happy-path |
Daily 6 AM UTC | Complete production happy path |
Path Filtering
CI intelligently runs only relevant tests based on changed files:
| Changed Path | Tests Run |
|---|---|
frontend/ |
Frontend lint, unit, E2E |
backend/ |
Backend lint, unit, emulator |
karaoke_gen/ |
Package unit, integration |
tests/ |
Package tests |
infrastructure/ |
Infrastructure validation |
Known Testing Gaps
Infrastructure dependencies (Modal, AudioShake, Cloud Tasks, Firestore) are mocked in tests. This means IAM permission errors or API changes won't be caught until production. Mitigations:
- Emulator tests - Use
backend/tests/emulator/for Firestore/GCS integration - Manual verification after infrastructure changes (Pulumi updates)
- Health endpoints -
/api/healthand/api/health/detailedvalidate connectivity - Daily E2E -
e2e-happy-path.ymlruns full production flow
Production E2E Testing
For comprehensive production testing:
cd frontend
# Run full happy path (requires tokens)
E2E_TEST_TOKEN=<token> npx playwright test e2e/production/happy-path-real-user.spec.ts \
--config=playwright.production.config.ts
# Run focused tests (faster iteration)
E2E_JOB_ID=<id> E2E_REVIEW_TOKEN=<token> npx playwright test e2e/production/lyrics-review-only.spec.ts \
--config=playwright.production.config.ts
# Admin dashboard tests
KARAOKE_ADMIN_TOKEN=<admin-token> npx playwright test e2e/production/admin-dashboard.spec.ts \
--config=playwright.production.config.ts
Environment Variables
| Variable | Purpose | Required For |
|---|---|---|
E2E_TEST_TOKEN |
User authentication | Most production tests |
KARAOKE_ADMIN_TOKEN |
Admin authentication | Admin dashboard tests |
E2E_JOB_ID |
Existing job to test | Focused tests |
E2E_REVIEW_TOKEN |
Review page access | Lyrics review tests |
TESTMAIL_API_KEY |
TestMail.app API | Magic link auth tests |
TESTMAIL_NAMESPACE |
TestMail.app namespace | Magic link auth tests |
Getting Test Tokens
Option 1: From browser
- Login to https://gen.nomadkaraoke.com
- Open DevTools → Application → Local Storage
- Copy the
karaoke_access_tokenvalue
Option 2: From backend logs When the daily E2E runs, tokens are logged for debugging.
The production tests cover:
- Magic link authentication flow
- Audio search and selection
- YouTube URL processing
- File upload workflow
- Lyrics review and correction
- Preview video generation
- Instrumental selection
- Final encoding and download
- Admin dashboard functionality
Running Tests Locally
# All backend/package tests
make test
# Specific suites
make test-unit # Package unit tests (69% coverage)
make test-backend # Backend tests only
make test-e2e # E2E with emulators
# With coverage report
poetry run pytest tests/unit --cov=karaoke_gen --cov-report=html
open htmlcov/index.html
# Frontend tests
cd frontend
npm run test:unit # Jest watch mode
npm run test:ci # Jest with coverage
npm run test:e2e # Playwright regression
npm run test:all # Unit + E2E combined
Test File Organization
karaoke-gen/
├── tests/
│ ├── unit/ # CLI package unit tests
│ │ ├── conftest.py # Shared fixtures
│ │ ├── test_audio.py # Audio separation tests
│ │ ├── test_karaoke_finalise/ # Finalize module tests
│ │ └── ...
│ ├── integration/ # CLI integration tests
│ │ ├── test_cli_edit_lyrics.py # Edit lyrics workflow
│ │ └── test_cli_backend_integration.py
│ └── conftest.py # Root fixtures
├── backend/
│ └── tests/
│ ├── conftest.py # FastAPI TestClient, mocks
│ ├── test_job_manager.py # Job management tests
│ ├── test_dependencies.py # DI tests
│ ├── test_made_for_you_webhook.py # Webhook tests
│ └── emulator/ # GCP emulator tests
│ ├── conftest.py # Emulator setup
│ ├── test_emulator_integration.py
│ └── test_worker_logs_direct.py
└── frontend/
├── components/__tests__/ # Jest component tests
│ ├── JobCard.test.tsx
│ ├── ThemeSelector.test.tsx
│ └── ...
├── e2e/
│ ├── fixtures/ # Shared test helpers
│ │ ├── test-helper.ts # API mocking utilities
│ │ └── mock-responses.ts # Canned API responses
│ ├── regression/ # Mocked API E2E (CI)
│ │ ├── smoke.spec.ts
│ │ ├── authentication.spec.ts
│ │ ├── karaoke-generation.spec.ts
│ │ └── ...
│ ├── production/ # Real backend E2E
│ │ ├── happy-path-real-user.spec.ts
│ │ ├── lyrics-review-only.spec.ts
│ │ └── admin-dashboard.spec.ts
│ └── manual/ # Manual setup required (NOT in CI)
│ ├── README.md # Setup instructions
│ └── lyrics-review-mobile.spec.ts # LyricsTranscriber frontend
├── jest.config.js
├── jest.setup.js
├── playwright.config.ts # Default (regression)
└── playwright.production.config.ts
Writing Good Tests
Do
- Test behavior, not implementation
- Use descriptive test names:
test_create_job_applies_default_theme_when_none_specified - One assertion per test (when practical)
- Use fixtures for common setup
- Test edge cases and error conditions
- Mock external services (Modal, AudioShake, Genius, etc.)
Don't
- Test private methods directly
- Write tests that depend on execution order
- Use sleep/wait without timeouts
- Test framework code (FastAPI, React, Next.js)
- Skip tests without a reason
Coverage Enforcement
Coverage is enforced in CI for the CLI package:
# Package unit tests require 69% coverage
poetry run pytest tests/unit/ --cov=karaoke_gen --cov-fail-under=69
To check coverage locally:
# CLI package
poetry run pytest tests/unit --cov=karaoke_gen --cov-report=term-missing
# Backend (no threshold, but useful to see)
poetry run pytest backend/tests --cov=backend --cov-report=term-missing
# Frontend (via Jest)
cd frontend && npm run test:ci # Includes coverage
Mocking Guidelines
When to Mock
- External APIs (Modal, AudioShake, Genius, Spotify lyrics, YouTube)
- GCS/Firestore in unit tests
- FFmpeg in unit tests
- Time-dependent operations
- Network requests
When NOT to Mock
- Your own code (test the real thing)
- Simple data transformations
- In emulator tests (use real Firestore/GCS emulators)
- In production E2E tests (use real backend)
Mock Example
from unittest.mock import Mock, patch
def test_fetch_audio_handles_modal_timeout():
with patch('karaoke_gen.audio.modal_api.submit_job') as mock_submit:
mock_submit.side_effect = TimeoutError("Modal API timeout")
with pytest.raises(AudioSeparationError):
separate_audio("test.mp3")
Frontend Mock Example
// Using Playwright route interception
test('shows error on API failure', async ({ page }) => {
await page.route('**/api/jobs', route => {
route.fulfill({ status: 500, body: 'Server Error' });
});
await page.goto('/app');
await expect(page.getByText('Error loading jobs')).toBeVisible();
});
Debugging Failed Tests
Local Debugging
# Run single test with verbose output
poetry run pytest tests/unit/test_audio.py::test_separate_audio -vvs
# Drop into debugger on failure
poetry run pytest --pdb
# Show local variables on failure
poetry run pytest -l
# Frontend: Debug specific test
cd frontend && npx playwright test smoke.spec.ts --debug
CI Debugging
- Check the GitHub Actions logs
- Look for the specific assertion that failed
- Reproduce locally with the same command
- If flaky, check for timing/ordering issues
- Download test artifacts (screenshots, videos) from failed E2E runs
Adding Tests for New Features
When adding a new feature:
- Write tests first (TDD) or immediately after
- Unit tests for business logic in
karaoke_gen/orbackend/ - Backend tests for new API endpoints
- Frontend unit tests for new components
- E2E regression tests for critical UI flows (add to
e2e/regression/) - Update this doc if new patterns emerge
Quick Reference
| Task | Command |
|---|---|
| All tests (before commit) | make test |
| Backend only | make test-backend |
| Frontend only | make test-frontend |
| Package unit | make test-unit |
| Emulator tests | make test-e2e |
| Frontend E2E (production) | cd frontend && npm run test:e2e:prod |
| Coverage HTML | poetry run pytest tests/unit --cov=karaoke_gen --cov-report=html |
| Install deps | make install |
| Start emulators | make emulators-start |
| Stop emulators | make emulators-stop |