mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
Follow-up to the investigation of why E2E "piri/dog" jobs started appearing in the live admin dashboard. Three related fixes (the transcription-cache cost bug from the same investigation ships separately in #1017). ## #2 — Hide `e2e-test-runner` jobs from the admin "My Jobs" view `list_jobs`' `exclude_test` filter uses `is_test_email()`, which only matched `@inbox.testmail.app`. When the CI post-deploy canary + daily E2E workflows switched to impersonating the **persistent** `e2e-test-runner@nomadkaraoke.com` account, their jobs stopped being filtered and leaked into the admin dashboard (previously every E2E job ran under an ephemeral testmail user and was hidden). Fix: add a `TEST_EMAIL_ADDRESSES` allowlist in `backend/utils/test_data.py` so this internal test account is treated as test data everywhere `is_test_email()` is used. ## #3 — Safety-net teardown for the happy-path spec `happy-path-real-user.spec.ts` only deleted its job at **Step 12**, which runs solely on a fully successful pass. A run that failed/timed out earlier (e.g. stuck at `awaiting_review`) stranded its job in production forever. Added a `test.afterEach` that deletes the tracked job on every attempt (including retries), regardless of where the test stopped. Idempotent with Step 12. ## #5 — Deterministic E2E song identity (`piri` vs `piri & tommy`) The guided-flow **AI match-judge** nondeterministically rewrote the typed artist to its canonical form (`piri` → `piri & tommy`) depending on the audio-confidence tier and LLM output — a legitimate product feature, but it produced two differently-named jobs across runs. The test now stubs `POST /api/catalog/match-judge` to a no-op verdict echoing the typed values, pinning the job to `piri - dog`. **Product behavior is unchanged** — only the test is pinned. ## Testing - New `backend/tests/test_test_data_utils.py` — `is_test_email` / `is_internal_email` classification incl. `e2e-test-runner` (case-insensitive), plus real/team accounts. - Extended the existing `exclude_test` summary-dict test to include the runner account. - Frontend spec typechecks (`tsc`) and lints (`eslint`) clean. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
53 lines
1.9 KiB
Python
53 lines
1.9 KiB
Python
"""Unit tests for backend.utils.test_data email classification helpers."""
|
|
|
|
from backend.utils.test_data import (
|
|
is_test_email,
|
|
is_internal_email,
|
|
is_internal_or_test_email,
|
|
)
|
|
|
|
|
|
class TestIsTestEmail:
|
|
def test_testmail_domain_is_test(self):
|
|
assert is_test_email("k1l43.test-123@inbox.testmail.app") is True
|
|
|
|
def test_e2e_test_runner_account_is_test(self):
|
|
# Persistent account impersonated by the CI canary + daily E2E runs.
|
|
assert is_test_email("e2e-test-runner@nomadkaraoke.com") is True
|
|
|
|
def test_e2e_test_runner_is_case_insensitive(self):
|
|
assert is_test_email("E2E-Test-Runner@NomadKaraoke.com") is True
|
|
|
|
def test_regular_team_account_is_not_test(self):
|
|
# Other @nomadkaraoke.com accounts are internal, not test data.
|
|
assert is_test_email("andrew@nomadkaraoke.com") is False
|
|
|
|
def test_real_user_is_not_test(self):
|
|
assert is_test_email("real@example.com") is False
|
|
|
|
def test_empty_is_not_test(self):
|
|
assert is_test_email("") is False
|
|
assert is_test_email(None) is False
|
|
|
|
|
|
class TestIsInternalEmail:
|
|
def test_nomadkaraoke_is_internal(self):
|
|
assert is_internal_email("andrew@nomadkaraoke.com") is True
|
|
|
|
def test_e2e_runner_is_also_internal_domain(self):
|
|
# It lives on the internal domain; is_internal stays True. The point of
|
|
# the TEST_EMAIL_ADDRESSES entry is that is_test_email ALSO returns True
|
|
# so the admin dashboard's exclude_test filter hides it.
|
|
assert is_internal_email("e2e-test-runner@nomadkaraoke.com") is True
|
|
|
|
def test_real_user_is_not_internal(self):
|
|
assert is_internal_email("real@example.com") is False
|
|
|
|
|
|
class TestIsInternalOrTestEmail:
|
|
def test_e2e_runner_matches(self):
|
|
assert is_internal_or_test_email("e2e-test-runner@nomadkaraoke.com") is True
|
|
|
|
def test_real_user_does_not_match(self):
|
|
assert is_internal_or_test_email("real@example.com") is False
|