fix(testing): Unify test setup with auto-install and clearer guidance (#262)

## Summary

- **`make test` now runs ALL tests** (backend + frontend) and
auto-installs dependencies if missing
- Added clear "All tests must pass" guidance - agents should never
dismiss failures as "pre-existing"
- Fixed misplaced test that was always failing (tested wrong frontend)
- Fixed logo locator bug in landing-page test

## Changes

| File | Change |
|------|--------|
| `Makefile` | Auto-install deps, unified `make test`, added
`test-backend`/`test-frontend` |
| `CLAUDE.md` | Simplified to just `make test` |
| `docs/DEVELOPMENT.md` | Consolidated testing section |
| `docs/TESTING.md` | Updated guidance and quick reference |
| `frontend/e2e/regression/landing-page.spec.ts` | Fixed `getByText` →
`getByAltText` for logo |
| `frontend/e2e/manual/` | Moved `lyrics-review-mobile.spec.ts` here
with README |

## Test plan

- [x] `make test` passes all 153 tests (76 backend + 77 frontend)
- [x] Auto-install works in fresh worktree
- [x] Documentation is clear and consistent

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
Andrew Beveridge ☄️andClaude Opus 4.5 authored and GitHub committed 2026-01-09 21:05:03 -05:00
1 parent 98b4636006
commit 49158f99ac
7 files changed
+133 -65

No files matched your search

+8 -4
View File
@@ -46,13 +46,17 @@
**Before committing**, run all tests:
```bash
# Backend (10 min timeout, limit output)
make test 2>&1 | tail -n 500
# Frontend
cd frontend && npm run test:all 2>&1 | tail -n 200
```
This single command:
- Installs dependencies automatically (backend + frontend)
- Runs all backend tests (unit, integration, emulator)
- Runs all frontend tests (Jest unit + Playwright E2E)
- Takes ~5-10 minutes
**All tests must pass.** Don't dismiss failures as "pre-existing" - investigate and fix them.
### Version Bumping
- Bump `tool.poetry.version` in `pyproject.toml` for code changes
- Skip for docs-only changes
+38 -12
View File
@@ -1,34 +1,50 @@
# Makefile for karaoke-gen project
# Run `make help` to see available commands
.PHONY: help test test-unit test-backend test-integration test-e2e test-all lint clean emulators-start emulators-stop
.PHONY: help install install-backend install-frontend test test-unit test-backend test-e2e test-frontend test-all lint clean emulators-start emulators-stop
# Default target
help:
@echo "Available commands:"
@echo " make test - Run all tests (unit + backend + emulator-based)"
@echo " make test-unit - Run unit tests only (karaoke_gen)"
@echo " make test-backend - Run backend unit tests only"
@echo " make test-e2e - Run E2E tests with emulators (starts/stops automatically)"
@echo " make test-all - Alias for 'make test'"
@echo " make test - Run ALL tests (backend + frontend, installs deps automatically)"
@echo " make test-backend - Run backend tests only (unit + emulator)"
@echo " make test-frontend - Run frontend tests only (unit + E2E)"
@echo " make test-unit - Run unit tests only (karaoke_gen package)"
@echo " make test-e2e - Run emulator tests with auto-start/stop"
@echo " make install - Install all dependencies (backend + frontend)"
@echo " make lint - Run linter checks"
@echo " make emulators-start - Start GCP emulators for local development"
@echo " make emulators-stop - Stop GCP emulators"
@echo ""
@echo "Before committing, run: make test"
# Install dependencies (only if needed)
install-backend:
@if [ ! -d "$$(poetry env info --path 2>/dev/null)" ] || ! poetry run python -c "import fastapi" 2>/dev/null; then \
echo "=== Installing backend dependencies ==="; \
poetry install; \
fi
install-frontend:
@if [ ! -d "frontend/node_modules" ]; then \
echo "=== Installing frontend dependencies ==="; \
cd frontend && npm install; \
fi
install: install-backend install-frontend
# Run unit tests for karaoke_gen package
test-unit:
test-unit: install-backend
@echo "=== Running karaoke_gen unit tests ==="
poetry run pytest tests/unit/ -v --cov=karaoke_gen --cov-report=term-missing --cov-fail-under=69
# Run backend unit tests
test-backend:
# Run backend unit tests (excludes emulator tests)
test-backend-unit: install-backend
@echo "=== Running backend unit tests ==="
poetry run pytest backend/tests/ --ignore=backend/tests/emulator -v
# Run E2E integration tests with emulators
test-e2e:
test-e2e: install-backend
@echo "=== Running E2E integration tests with emulators ==="
@./scripts/start-emulators.sh || (echo "Failed to start emulators" && exit 1)
@export FIRESTORE_EMULATOR_HOST=127.0.0.1:8080 && \
@@ -42,8 +58,18 @@ test-e2e:
./scripts/stop-emulators.sh; \
exit $$TEST_RESULT
# Run all tests (use this before committing!)
test: test-unit test-backend test-e2e
# Run all backend tests (unit + emulator)
test-backend: test-unit test-backend-unit test-e2e
@echo ""
@echo "✅ Backend tests passed!"
# Run frontend tests (unit + E2E)
test-frontend: install-frontend
@echo "=== Running frontend tests ==="
cd frontend && npm run test:all
# Run ALL tests (backend + frontend) - use this before committing!
test: test-backend test-frontend
@echo ""
@echo "✅ All tests passed! Ready to commit."
+10 -38
View File
@@ -60,45 +60,21 @@ npm run dev
## Testing
### Backend Tests
**See [TESTING.md](TESTING.md) for complete testing guidance.**
Quick reference (run before committing):
```bash
# Run all tests (required before commit)
make test 2>&1 | tail -n 500
# This runs in order:
# 1. Unit tests with coverage
# 2. Backend unit tests
# 3. E2E integration tests with emulators
```
**Important**: Set 10 minute timeout. Tests can take a while with emulators.
This runs ALL tests (backend + frontend) and installs dependencies automatically.
### Frontend Tests
For faster iteration, run subsets:
- `make test-backend` - Backend only (~2 min)
- `make test-frontend` - Frontend only (~3 min)
```bash
cd frontend
npm run test:all 2>&1 | tail -n 200
# Individual commands:
npm run test:unit # Jest unit tests
npm run test:e2e # Playwright E2E tests
```
### Test Architecture
```
┌─────────────────────────────────┐
│ Unit Tests (Mocked) │ Fast, isolated
│ ~62 tests, <1s │
├─────────────────────────────────┤
│ Emulator Integration Tests │ Local Firestore/GCS
│ ~11 tests, ~2s │
├─────────────────────────────────┤
│ E2E Tests │ Full API + Frontend
│ Requires deployed backend │
└─────────────────────────────────┘
```
TESTING.md covers: test types, CI requirements, Playwright usage, mocking guidelines, and coverage enforcement.
## Deployment
@@ -231,10 +207,6 @@ lsof -i :4443 # GCS
./scripts/stop-emulators.sh
```
### Tests Fail with Auth Errors
Ensure `ADMIN_TOKENS` is set in environment or `.env` file.
### Docker Issues
```bash
@@ -245,6 +217,6 @@ docker info
docker pull fsouza/fake-gcs-server
```
### Coverage Below Target
### Test Failures
Run `./scripts/run-tests.sh --coverage` and check `htmlcov/index.html` to see uncovered lines.
See [TESTING.md](TESTING.md#debugging-failed-tests) for debugging test failures, coverage issues, and CI troubleshooting.
+42 -10
View File
@@ -2,6 +2,29 @@
This document defines testing standards, code quality principles, and CI requirements for Nomad Karaoke Gen.
## Critical: All Tests Must Pass
**All tests must pass before merging.** If tests fail:
1. **Fix them** - Don't assume failures are "pre-existing issues"
2. **Investigate** - Understand why the test is failing
3. **Don't dismiss** - If a test fails, either the code or test needs fixing
CI blocks merging if tests fail. If you encounter failures locally, fix them as part of your work.
### Running Tests
```bash
# Run ALL tests (backend + frontend) - installs deps automatically
make test
# Run subsets for faster iteration
make test-backend # Backend only (~2 min)
make test-frontend # Frontend only (~3 min)
```
Dependencies are installed automatically when needed.
## Core Principles
### SOLID Principles
@@ -96,19 +119,25 @@ make emulators-stop
**Locations:**
- `frontend/e2e/regression/` - Mocked API tests (CI-safe, fast)
- `frontend/e2e/production/` - Real backend tests (comprehensive, slower)
- `frontend/e2e/manual/` - Tests requiring manual setup (not in CI)
**Tool:** Playwright
**Regression tests** run on every PR:
- Mock API responses via fixtures
- Test UI behavior in isolation
- ~1-2 min total runtime
- Playwright auto-starts dev server via `webServer` config
- ~3 min total runtime
**Production tests** run on-demand or daily:
- Hit real `https://gen.nomadkaraoke.com` and `https://api.nomadkaraoke.com`
- Full user journeys (20-40 min for happy path)
- Require authentication tokens
**Manual tests** are NOT included in CI:
- Require separate dev servers (e.g., LyricsTranscriber frontend on port 5173)
- See `frontend/e2e/manual/README.md` for setup instructions
## Playwright Usage
### Installation
@@ -353,10 +382,13 @@ karaoke-gen/
│ │ ├── authentication.spec.ts
│ │ ├── karaoke-generation.spec.ts
│ │ └── ...
│ └── production/ # Real backend E2E
│ ├── happy-path-real-user.spec.ts
│ ├── lyrics-review-only.spec.ts
│ └── admin-dashboard.spec.ts
│ ├── production/ # Real backend E2E
│ │ ├── happy-path-real-user.spec.ts
│ │ ├── lyrics-review-only.spec.ts
│ │ └── admin-dashboard.spec.ts
│ └── manual/ # Manual setup required (NOT in CI)
│ ├── README.md # Setup instructions
│ └── lyrics-review-mobile.spec.ts # LyricsTranscriber frontend
├── jest.config.js
├── jest.setup.js
├── playwright.config.ts # Default (regression)
@@ -489,13 +521,13 @@ When adding a new feature:
| Task | Command |
|------|---------|
| All tests | `make test` |
| Package unit | `make test-unit` |
| **All tests (before commit)** | `make test` |
| Backend only | `make test-backend` |
| With emulators | `make test-e2e` |
| Frontend unit | `cd frontend && npm run test:unit` |
| Frontend E2E (regression) | `cd frontend && npm run test:e2e` |
| Frontend only | `make test-frontend` |
| Package unit | `make test-unit` |
| Emulator tests | `make test-e2e` |
| Frontend E2E (production) | `cd frontend && npm run test:e2e:prod` |
| Coverage HTML | `poetry run pytest tests/unit --cov=karaoke_gen --cov-report=html` |
| Install deps | `make install` |
| Start emulators | `make emulators-start` |
| Stop emulators | `make emulators-stop` |
+33
View File
@@ -0,0 +1,33 @@
# Manual E2E Tests
Tests in this directory require **manual setup** before running. They are NOT included in CI or `npm run test:e2e`.
## Why Manual?
These tests target components with separate dev servers that aren't auto-started by Playwright's `webServer` config.
## Tests
### `lyrics-review-mobile.spec.ts`
Tests the **LyricsTranscriber frontend** (Vite, port 5173), not the main karaoke-gen frontend (Next.js, port 3000).
**To run:**
```bash
# Terminal 1: Start LyricsTranscriber frontend dev server
cd lyrics_transcriber_temp/lyrics_transcriber/frontend
yarn dev # Starts on port 5173
# Terminal 2: Start LyricsTranscriber backend (for API)
# This serves the review API on port 8767
poetry run python -c "from lyrics_transcriber.review.server import ReviewServer; ..."
# Terminal 3: Run the test
cd frontend
npx playwright test e2e/manual/lyrics-review-mobile.spec.ts
```
## Future
These tests should be consolidated when LyricsTranscriber is fully integrated into the main karaoke-gen frontend. See CLAUDE.md for project context.
+2 -1
View File
@@ -23,7 +23,8 @@ test.describe('Landing Page', () => {
// Check hero content
await expect(page.locator('h1')).toContainText('Karaoke Video');
await expect(page.locator('nav')).toBeVisible();
await expect(page.getByRole('navigation').getByText('Nomad Karaoke')).toBeVisible();
// Logo is an image with alt text, not visible text
await expect(page.getByRole('navigation').getByAltText('Nomad Karaoke')).toBeVisible();
});
test('displays pricing section with 4 packages', async ({ page }) => {