Files
karaoke-gen/backend/tests/test_test_data_utils.py
Andrew Beveridge ☄️andClaude Opus 4.8 fb9d86d17f fix(e2e): hide test-runner jobs from admin, stop stranding + name drift (#1019)
Follow-up to the investigation of why E2E "piri/dog" jobs started
appearing in the live admin dashboard. Three related fixes (the
transcription-cache cost bug from the same investigation ships
separately in #1017).

## #2 — Hide `e2e-test-runner` jobs from the admin "My Jobs" view
`list_jobs`' `exclude_test` filter uses `is_test_email()`, which only
matched `@inbox.testmail.app`. When the CI post-deploy canary + daily
E2E workflows switched to impersonating the **persistent**
`e2e-test-runner@nomadkaraoke.com` account, their jobs stopped being
filtered and leaked into the admin dashboard (previously every E2E job
ran under an ephemeral testmail user and was hidden).

Fix: add a `TEST_EMAIL_ADDRESSES` allowlist in
`backend/utils/test_data.py` so this internal test account is treated as
test data everywhere `is_test_email()` is used.

## #3 — Safety-net teardown for the happy-path spec
`happy-path-real-user.spec.ts` only deleted its job at **Step 12**,
which runs solely on a fully successful pass. A run that failed/timed
out earlier (e.g. stuck at `awaiting_review`) stranded its job in
production forever. Added a `test.afterEach` that deletes the tracked
job on every attempt (including retries), regardless of where the test
stopped. Idempotent with Step 12.

## #5 — Deterministic E2E song identity (`piri` vs `piri & tommy`)
The guided-flow **AI match-judge** nondeterministically rewrote the
typed artist to its canonical form (`piri` → `piri & tommy`) depending
on the audio-confidence tier and LLM output — a legitimate product
feature, but it produced two differently-named jobs across runs. The
test now stubs `POST /api/catalog/match-judge` to a no-op verdict
echoing the typed values, pinning the job to `piri - dog`. **Product
behavior is unchanged** — only the test is pinned.

## Testing
- New `backend/tests/test_test_data_utils.py` — `is_test_email` /
`is_internal_email` classification incl. `e2e-test-runner`
(case-insensitive), plus real/team accounts.
- Extended the existing `exclude_test` summary-dict test to include the
runner account.
- Frontend spec typechecks (`tsc`) and lints (`eslint`) clean.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-09-17 13:57:57 -04:00

53 lines
1.9 KiB
Python

"""Unit tests for backend.utils.test_data email classification helpers."""
from backend.utils.test_data import (
is_test_email,
is_internal_email,
is_internal_or_test_email,
)
class TestIsTestEmail:
def test_testmail_domain_is_test(self):
assert is_test_email("k1l43.test-123@inbox.testmail.app") is True
def test_e2e_test_runner_account_is_test(self):
# Persistent account impersonated by the CI canary + daily E2E runs.
assert is_test_email("e2e-test-runner@nomadkaraoke.com") is True
def test_e2e_test_runner_is_case_insensitive(self):
assert is_test_email("E2E-Test-Runner@NomadKaraoke.com") is True
def test_regular_team_account_is_not_test(self):
# Other @nomadkaraoke.com accounts are internal, not test data.
assert is_test_email("andrew@nomadkaraoke.com") is False
def test_real_user_is_not_test(self):
assert is_test_email("real@example.com") is False
def test_empty_is_not_test(self):
assert is_test_email("") is False
assert is_test_email(None) is False
class TestIsInternalEmail:
def test_nomadkaraoke_is_internal(self):
assert is_internal_email("andrew@nomadkaraoke.com") is True
def test_e2e_runner_is_also_internal_domain(self):
# It lives on the internal domain; is_internal stays True. The point of
# the TEST_EMAIL_ADDRESSES entry is that is_test_email ALSO returns True
# so the admin dashboard's exclude_test filter hides it.
assert is_internal_email("e2e-test-runner@nomadkaraoke.com") is True
def test_real_user_is_not_internal(self):
assert is_internal_email("real@example.com") is False
class TestIsInternalOrTestEmail:
def test_e2e_runner_matches(self):
assert is_internal_or_test_email("e2e-test-runner@nomadkaraoke.com") is True
def test_real_user_does_not_match(self):
assert is_internal_or_test_email("real@example.com") is False