Files
karaoke-gen/docs/TESTING.md
T
Andrew Beveridge ☄️andClaude Opus 4.5 49158f99ac fix(testing): Unify test setup with auto-install and clearer guidance (#262)
## Summary

- **`make test` now runs ALL tests** (backend + frontend) and
auto-installs dependencies if missing
- Added clear "All tests must pass" guidance - agents should never
dismiss failures as "pre-existing"
- Fixed misplaced test that was always failing (tested wrong frontend)
- Fixed logo locator bug in landing-page test

## Changes

| File | Change |
|------|--------|
| `Makefile` | Auto-install deps, unified `make test`, added
`test-backend`/`test-frontend` |
| `CLAUDE.md` | Simplified to just `make test` |
| `docs/DEVELOPMENT.md` | Consolidated testing section |
| `docs/TESTING.md` | Updated guidance and quick reference |
| `frontend/e2e/regression/landing-page.spec.ts` | Fixed `getByText` →
`getByAltText` for logo |
| `frontend/e2e/manual/` | Moved `lyrics-review-mobile.spec.ts` here
with README |

## Test plan

- [x] `make test` passes all 153 tests (76 backend + 77 frontend)
- [x] Auto-install works in fresh worktree
- [x] Documentation is clear and consistent

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-09 21:05:03 -05:00

16 KiB

Testing & Code Quality Guide

This document defines testing standards, code quality principles, and CI requirements for Nomad Karaoke Gen.

Critical: All Tests Must Pass

All tests must pass before merging. If tests fail:

  1. Fix them - Don't assume failures are "pre-existing issues"
  2. Investigate - Understand why the test is failing
  3. Don't dismiss - If a test fails, either the code or test needs fixing

CI blocks merging if tests fail. If you encounter failures locally, fix them as part of your work.

Running Tests

# Run ALL tests (backend + frontend) - installs deps automatically
make test

# Run subsets for faster iteration
make test-backend   # Backend only (~2 min)
make test-frontend  # Frontend only (~3 min)

Dependencies are installed automatically when needed.

Core Principles

SOLID Principles

All code should follow SOLID principles:

  1. Single Responsibility - Each class/function does one thing well
  2. Open/Closed - Open for extension, closed for modification
  3. Liskov Substitution - Subtypes must be substitutable for base types
  4. Interface Segregation - Many specific interfaces over one general interface
  5. Dependency Inversion - Depend on abstractions, not concretions

Code Quality Standards

  • Maintainability - Code should be easy to understand and modify
  • Testability - Code should be designed for easy testing (dependency injection, pure functions)
  • No magic numbers - Use named constants
  • Explicit over implicit - Clear is better than clever
  • DRY but not prematurely - Don't repeat yourself, but don't abstract too early

Test Types

Unit Tests

Locations:

  • tests/unit/ - CLI package (karaoke_gen/) unit tests
  • backend/tests/ - Backend API unit tests (excludes emulator/)
  • frontend/components/__tests__/ - React component unit tests

Characteristics:

  • Fast (< 1ms per test typical)
  • No external dependencies (mocked)
  • No network, database, or filesystem access
  • Run on every commit

Coverage targets:

  • karaoke_gen/ package: 69%+
  • backend/ package: No explicit threshold (but coverage collected)
  • frontend/ components: No explicit threshold (but coverage collected)
# Good unit test example (backend/tests/test_job_manager.py)
def test_create_job_applies_default_theme(mock_firestore, mock_gcs):
    job_manager = JobManager()
    job = job_manager.create_job(
        artist="Test Artist",
        title="Test Song",
        user_email="test@example.com"
    )
    assert job.theme_id == "nomad"  # Default theme applied

Integration Tests (tests/integration/)

Purpose: Test CLI workflows and component interactions.

Characteristics:

  • May mock high-level components
  • Tests full command pipelines
  • Uses @pytest.mark.integration marker
  • Often async (@pytest.mark.asyncio)

Example scenarios:

  • CLI --edit-lyrics flow end-to-end
  • Job creation → processing pipeline
  • Audio separation → transcription coordination

Emulator Tests (backend/tests/emulator/)

Purpose: Test real Firestore/GCS behavior without production access.

Characteristics:

  • Requires GCP emulators running
  • Tests actual database operations
  • Verifies subcollections, TTL policies, etc.
  • Auto-skips if emulators unavailable

Setup:

# Start emulators
make emulators-start

# Run emulator tests
poetry run pytest backend/tests/emulator/ -v

# Stop emulators
make emulators-stop

E2E Tests (Frontend)

Locations:

  • frontend/e2e/regression/ - Mocked API tests (CI-safe, fast)
  • frontend/e2e/production/ - Real backend tests (comprehensive, slower)
  • frontend/e2e/manual/ - Tests requiring manual setup (not in CI)

Tool: Playwright

Regression tests run on every PR:

  • Mock API responses via fixtures
  • Test UI behavior in isolation
  • Playwright auto-starts dev server via webServer config
  • ~3 min total runtime

Production tests run on-demand or daily:

  • Hit real https://gen.nomadkaraoke.com and https://api.nomadkaraoke.com
  • Full user journeys (20-40 min for happy path)
  • Require authentication tokens

Manual tests are NOT included in CI:

  • Require separate dev servers (e.g., LyricsTranscriber frontend on port 5173)
  • See frontend/e2e/manual/README.md for setup instructions

Playwright Usage

Installation

cd frontend
npm install -D @playwright/test
npx playwright install

Running E2E Tests

# Regression tests (mocked API, default)
npm run test:e2e

# Production tests (real backend)
npm run test:e2e:prod

# With UI mode for debugging
npm run test:e2e:ui

# Specific test file
npx playwright test e2e/regression/smoke.spec.ts

# Production test with environment
E2E_TEST_TOKEN=xxx npm run test:e2e:prod

Investigating Frontend Issues

When debugging frontend issues, use Playwright to:

  1. Reproduce the issue - Write a test that fails
  2. Debug visually - Use --ui mode or --headed
  3. Capture evidence - Screenshots, traces, console logs
  4. Verify the fix - Test passes after fix
# Debug mode with browser visible
npm run test:e2e:headed

# Generate trace for failed tests
npx playwright test --trace on

Writing E2E Tests

Regression test example (mocked API):

// frontend/e2e/regression/smoke.spec.ts
import { test, expect } from '@playwright/test';
import { setupApiFixtures, clearAuthToken } from '../fixtures/test-helper';

test.describe('Smoke Tests', () => {
  test.beforeEach(async ({ page }) => {
    await clearAuthToken(page);
  });

  test('landing page loads and shows hero', async ({ page }) => {
    await setupApiFixtures(page, { mocks: [] });
    await page.goto('/');
    await expect(page.locator('h1')).toContainText('Karaoke');
  });
});

Production test example (real backend):

// frontend/e2e/production/lyrics-review-only.spec.ts
import { test, expect } from '@playwright/test';

const PROD_URL = 'https://gen.nomadkaraoke.com';

test.describe('Lyrics Review', () => {
  test('Complete lyrics review via UI', async ({ page }) => {
    test.setTimeout(900_000); // 15 min - preview generation can be slow

    const jobId = process.env.E2E_JOB_ID;
    await page.goto(`${PROD_URL}/review/${jobId}`);
    // ... test implementation
  });
});

CI Pipeline

Required Checks (Block PR Merge)

All PRs must pass these checks (path-filtered for efficiency):

Check Description Threshold
backend-lint Ruff linter No errors
backend-unit-tests Backend tests (excludes emulator) Must pass
backend-emulator-tests GCP emulator integration Must pass
package-unit-tests CLI package tests 69% coverage
package-integration-tests CLI integration tests Must pass
frontend-lint ESLint (Rules of Hooks) No errors
frontend-unit-tests Jest component tests Must pass
frontend-e2e-smoke Playwright smoke tests Must pass
frontend-build Next.js build Must pass

On-Demand Checks

Check Trigger Description
frontend-e2e-full e2e PR label Full regression E2E suite
e2e-happy-path Daily 6 AM UTC Complete production happy path

Path Filtering

CI intelligently runs only relevant tests based on changed files:

Changed Path Tests Run
frontend/ Frontend lint, unit, E2E
backend/ Backend lint, unit, emulator
karaoke_gen/ Package unit, integration
tests/ Package tests
infrastructure/ Infrastructure validation

Known Testing Gaps

Infrastructure dependencies (Modal, AudioShake, Cloud Tasks, Firestore) are mocked in tests. This means IAM permission errors or API changes won't be caught until production. Mitigations:

  1. Emulator tests - Use backend/tests/emulator/ for Firestore/GCS integration
  2. Manual verification after infrastructure changes (Pulumi updates)
  3. Health endpoints - /api/health and /api/health/detailed validate connectivity
  4. Daily E2E - e2e-happy-path.yml runs full production flow

Production E2E Testing

For comprehensive production testing:

cd frontend

# Run full happy path (requires tokens)
E2E_TEST_TOKEN=<token> npx playwright test e2e/production/happy-path-real-user.spec.ts \
  --config=playwright.production.config.ts

# Run focused tests (faster iteration)
E2E_JOB_ID=<id> E2E_REVIEW_TOKEN=<token> npx playwright test e2e/production/lyrics-review-only.spec.ts \
  --config=playwright.production.config.ts

# Admin dashboard tests
KARAOKE_ADMIN_TOKEN=<admin-token> npx playwright test e2e/production/admin-dashboard.spec.ts \
  --config=playwright.production.config.ts

Environment Variables

Variable Purpose Required For
E2E_TEST_TOKEN User authentication Most production tests
KARAOKE_ADMIN_TOKEN Admin authentication Admin dashboard tests
E2E_JOB_ID Existing job to test Focused tests
E2E_REVIEW_TOKEN Review page access Lyrics review tests
TESTMAIL_API_KEY TestMail.app API Magic link auth tests
TESTMAIL_NAMESPACE TestMail.app namespace Magic link auth tests

Getting Test Tokens

Option 1: From browser

  1. Login to https://gen.nomadkaraoke.com
  2. Open DevTools → Application → Local Storage
  3. Copy the karaoke_access_token value

Option 2: From backend logs When the daily E2E runs, tokens are logged for debugging.

The production tests cover:

  • Magic link authentication flow
  • Audio search and selection
  • YouTube URL processing
  • File upload workflow
  • Lyrics review and correction
  • Preview video generation
  • Instrumental selection
  • Final encoding and download
  • Admin dashboard functionality

Running Tests Locally

# All backend/package tests
make test

# Specific suites
make test-unit       # Package unit tests (69% coverage)
make test-backend    # Backend tests only
make test-e2e        # E2E with emulators

# With coverage report
poetry run pytest tests/unit --cov=karaoke_gen --cov-report=html
open htmlcov/index.html

# Frontend tests
cd frontend
npm run test:unit    # Jest watch mode
npm run test:ci      # Jest with coverage
npm run test:e2e     # Playwright regression
npm run test:all     # Unit + E2E combined

Test File Organization

karaoke-gen/
├── tests/
│   ├── unit/                        # CLI package unit tests
│   │   ├── conftest.py              # Shared fixtures
│   │   ├── test_audio.py            # Audio separation tests
│   │   ├── test_karaoke_finalise/   # Finalize module tests
│   │   └── ...
│   ├── integration/                 # CLI integration tests
│   │   ├── test_cli_edit_lyrics.py  # Edit lyrics workflow
│   │   └── test_cli_backend_integration.py
│   └── conftest.py                  # Root fixtures
├── backend/
│   └── tests/
│       ├── conftest.py              # FastAPI TestClient, mocks
│       ├── test_job_manager.py      # Job management tests
│       ├── test_dependencies.py     # DI tests
│       ├── test_made_for_you_webhook.py  # Webhook tests
│       └── emulator/                # GCP emulator tests
│           ├── conftest.py          # Emulator setup
│           ├── test_emulator_integration.py
│           └── test_worker_logs_direct.py
└── frontend/
    ├── components/__tests__/        # Jest component tests
    │   ├── JobCard.test.tsx
    │   ├── ThemeSelector.test.tsx
    │   └── ...
    ├── e2e/
    │   ├── fixtures/                # Shared test helpers
    │   │   ├── test-helper.ts       # API mocking utilities
    │   │   └── mock-responses.ts    # Canned API responses
    │   ├── regression/              # Mocked API E2E (CI)
    │   │   ├── smoke.spec.ts
    │   │   ├── authentication.spec.ts
    │   │   ├── karaoke-generation.spec.ts
    │   │   └── ...
    │   ├── production/              # Real backend E2E
    │   │   ├── happy-path-real-user.spec.ts
    │   │   ├── lyrics-review-only.spec.ts
    │   │   └── admin-dashboard.spec.ts
    │   └── manual/                  # Manual setup required (NOT in CI)
    │       ├── README.md            # Setup instructions
    │       └── lyrics-review-mobile.spec.ts  # LyricsTranscriber frontend
    ├── jest.config.js
    ├── jest.setup.js
    ├── playwright.config.ts         # Default (regression)
    └── playwright.production.config.ts

Writing Good Tests

Do

  • Test behavior, not implementation
  • Use descriptive test names: test_create_job_applies_default_theme_when_none_specified
  • One assertion per test (when practical)
  • Use fixtures for common setup
  • Test edge cases and error conditions
  • Mock external services (Modal, AudioShake, Genius, etc.)

Don't

  • Test private methods directly
  • Write tests that depend on execution order
  • Use sleep/wait without timeouts
  • Test framework code (FastAPI, React, Next.js)
  • Skip tests without a reason

Coverage Enforcement

Coverage is enforced in CI for the CLI package:

# Package unit tests require 69% coverage
poetry run pytest tests/unit/ --cov=karaoke_gen --cov-fail-under=69

To check coverage locally:

# CLI package
poetry run pytest tests/unit --cov=karaoke_gen --cov-report=term-missing

# Backend (no threshold, but useful to see)
poetry run pytest backend/tests --cov=backend --cov-report=term-missing

# Frontend (via Jest)
cd frontend && npm run test:ci  # Includes coverage

Mocking Guidelines

When to Mock

  • External APIs (Modal, AudioShake, Genius, Spotify lyrics, YouTube)
  • GCS/Firestore in unit tests
  • FFmpeg in unit tests
  • Time-dependent operations
  • Network requests

When NOT to Mock

  • Your own code (test the real thing)
  • Simple data transformations
  • In emulator tests (use real Firestore/GCS emulators)
  • In production E2E tests (use real backend)

Mock Example

from unittest.mock import Mock, patch

def test_fetch_audio_handles_modal_timeout():
    with patch('karaoke_gen.audio.modal_api.submit_job') as mock_submit:
        mock_submit.side_effect = TimeoutError("Modal API timeout")

        with pytest.raises(AudioSeparationError):
            separate_audio("test.mp3")

Frontend Mock Example

// Using Playwright route interception
test('shows error on API failure', async ({ page }) => {
  await page.route('**/api/jobs', route => {
    route.fulfill({ status: 500, body: 'Server Error' });
  });

  await page.goto('/app');
  await expect(page.getByText('Error loading jobs')).toBeVisible();
});

Debugging Failed Tests

Local Debugging

# Run single test with verbose output
poetry run pytest tests/unit/test_audio.py::test_separate_audio -vvs

# Drop into debugger on failure
poetry run pytest --pdb

# Show local variables on failure
poetry run pytest -l

# Frontend: Debug specific test
cd frontend && npx playwright test smoke.spec.ts --debug

CI Debugging

  1. Check the GitHub Actions logs
  2. Look for the specific assertion that failed
  3. Reproduce locally with the same command
  4. If flaky, check for timing/ordering issues
  5. Download test artifacts (screenshots, videos) from failed E2E runs

Adding Tests for New Features

When adding a new feature:

  1. Write tests first (TDD) or immediately after
  2. Unit tests for business logic in karaoke_gen/ or backend/
  3. Backend tests for new API endpoints
  4. Frontend unit tests for new components
  5. E2E regression tests for critical UI flows (add to e2e/regression/)
  6. Update this doc if new patterns emerge

Quick Reference

Task Command
All tests (before commit) make test
Backend only make test-backend
Frontend only make test-frontend
Package unit make test-unit
Emulator tests make test-e2e
Frontend E2E (production) cd frontend && npm run test:e2e:prod
Coverage HTML poetry run pytest tests/unit --cov=karaoke_gen --cov-report=html
Install deps make install
Start emulators make emulators-start
Stop emulators make emulators-stop