test(e2e): rewrite tenant daily E2E as a faithful frontend journey (#873)

## Why

The daily tenant E2E (`scripts/e2e/tenant_e2e.py`) drove the whole flow
via **direct backend API calls**, bypassing the frontend. Two problems:
1. **Not representative** — it never exercised the tenant portal UI a
real operator uses.
2. **Intermittent stuck-render orphans** (Run #31 vocalstar, #32 singa)
— because it never loaded the lyrics-review page, it never fired
`encoding-worker/warmup`, so the encoder was cold when the render
started and the job orphaned in `rendering_video` (no recovery).
Root-caused this session; the fix is to make the test faithful, which
fires the warmup a real user fires.

## What

Replaces `tenant_e2e.py` with
**`frontend/e2e/production/tenant-happy-path.spec.ts`** — a Playwright
journey mirroring `happy-path-real-user.spec.ts`, run per-tenant via the
existing `e2e-tenant-daily.yml` matrix (vocalstar, singa; `max-parallel:
1`):

login (real magic-link UI) → submit track (tenant form + real audio
upload) → wait for transcription → **open review UI (fires warmup)** →
approve → wait for render → assert tenant distribution (Dropbox present,
no YouTube, downloads, `is_private`) → cleanup.

- **Tenant auth**: added `inbox.testmail.app` to both tenants'
`allowed_email_domains` (setup scripts + live GCS config via
`scripts/e2e/update_tenant_allowlist.py`) so the test signs in through
the real magic-link UI.
- **Test audio**: short 60s clips fetched from private GCS in CI
(`scripts/e2e/fetch_test_audio.sh`) — fast, reliable uploads and quick
renders.
- Workflow reworked to Node/Playwright; `tenant_e2e.py` deleted; version
`0.190.3 → 0.190.4`.

## Findings surfaced (validating the exercise)

- **Browser upload was CORS-blocked in Playwright** by the shared
config's global `x-playwright-test` header (it made the GCS signed-URL
PUT preflight request a header the bucket's CORS doesn't allow).
Verified the same PUT succeeds in a real browser — a **test-harness
artifact, not a product bug**. Worked around by stripping the header for
`storage.googleapis.com` requests. (Note: any prod E2E doing a browser
signed-URL upload is affected by this global header.)

## Verification

Both tenants run **green end-to-end against prod** locally (login →
submit → review → render → distribution; render completed without
orphaning). Reworked workflow dispatched on this branch to confirm CI
wiring (Node setup + private-audio fetch + secrets).

Test-only + CI + tenant-config change. No product code.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

@coderabbitai ignore

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Andrew Beveridge ☄️andClaude Opus 4.8 authored and GitHub committed 2026-07-05 02:33:20 -04:00
1 parent f93d5f6923
commit 6f86b3d901
11 files changed
+973 -244

No files matched your search

+39 -11
View File
@@ -34,10 +34,12 @@ jobs:
- name: Checkout code
uses: actions/checkout@v4
- name: Setup Python
uses: actions/setup-python@v5
- name: Setup Node.js
uses: actions/setup-node@v4
with:
python-version: '3.11'
node-version: '20'
cache: 'npm'
cache-dependency-path: frontend/package-lock.json
- name: Authenticate to GCP
uses: google-github-actions/auth@v2
@@ -46,15 +48,38 @@ jobs:
service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
- name: Install dependencies
run: pip install google-cloud-storage google-cloud-secret-manager
working-directory: frontend
run: npm ci --legacy-peer-deps
- name: Run tenant E2E
id: run_test
env:
KARAOKE_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
- name: Install Playwright browsers
working-directory: frontend
run: npx playwright install --with-deps chromium
- name: Fetch test audio (private GCS)
run: bash scripts/e2e/fetch_test_audio.sh frontend/e2e/fixtures
- name: Ensure testmail.app domain is allowlisted (rate limit)
run: |
echo "=== Tenant E2E: ${{ matrix.tenant }} ==="
python scripts/e2e/tenant_e2e.py ${{ matrix.tenant }} 2>&1 | tee e2e-output.log
curl -sf -X POST \
"https://api.nomadkaraoke.com/api/admin/rate-limits/blocklists/allowlisted-domains" \
-H "X-Admin-Token: ${{ secrets.E2E_ADMIN_TOKEN }}" \
-H "Content-Type: application/json" \
-d '{"domain": "inbox.testmail.app"}' || echo "Warning: allowlist may already exist"
- name: Run tenant E2E (frontend journey)
id: run_test
working-directory: frontend
env:
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
TESTMAIL_NAMESPACE: ${{ secrets.TESTMAIL_NAMESPACE }}
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
TENANT_ID: ${{ matrix.tenant }}
TENANT_PORTAL_URL: https://${{ matrix.tenant }}.nomadkaraoke.com
run: |
echo "=== Tenant E2E (frontend): ${{ matrix.tenant }} ==="
npx playwright test tenant-happy-path.spec.ts \
--config=playwright.production.config.ts --project=chromium \
--reporter=list --timeout=2400000 2>&1 | tee e2e-output.log
echo "exit_code=${PIPESTATUS[0]}" >> "$GITHUB_OUTPUT"
continue-on-error: true
@@ -63,7 +88,10 @@ jobs:
uses: actions/upload-artifact@v4
with:
name: e2e-tenant-${{ matrix.tenant }}-${{ github.run_number }}
path: e2e-output.log
path: |
frontend/e2e-output.log
frontend/test-results/
frontend/playwright-report/
retention-days: 14
- name: Discord notification (failure)
@@ -0,0 +1,87 @@
# Tenant Portal E2E — Rewrite as a Faithful Frontend Journey Test
**Date:** 2026-07-04
**Status:** Design approved, pending spec review → implementation plan
**Author:** Andrew Beveridge + Claude
**Scope:** karaoke-gen (`frontend/e2e/production/`, `.github/workflows/e2e-tenant-daily.yml`, tenant setup scripts)
## Problem
The daily tenant E2E (`scripts/e2e/tenant_e2e.py`, run by `e2e-tenant-daily.yml`) is a **Python script that drives the whole flow via direct backend API calls** — submit, `uploads-complete`, `complete-review`, poll — bypassing the frontend entirely. It was built this way in June 2026 (PR #828) to prove the tenant pipeline worked, not to represent the user experience.
Two consequences:
1. **It is not a representative test.** It never exercises the tenant portal UI a real operator uses, so it cannot catch UX regressions and gives false confidence.
2. **It intermittently fails with stuck-render orphans.** Because it never loads the lyrics-review page, it never fires `POST /api/internal/encoding-worker/warmup/{job_id}` (+ `/heartbeat`) that the real review UI fires on load (`LyricsAnalyzer.tsx`). The encoding-worker VM (terminated at idle) is therefore cold when the render starts; the render worker enters its GCE-connection retry loop, and if the backend Cloud Run instance is recycled mid-retry the job orphans in `rendering_video` with no recovery. Confirmed root cause for runs #31 (vocalstar) and #32 (singa) — see the render-orphan analysis in this session's notes.
The consumer daily test (`happy-path-real-user.spec.ts`, "piri – dog") is the model done right: a real Playwright journey through the frontend that naturally fires the warmup.
## Goal
Replace the API-driven tenant script with a **Playwright frontend-journey test** that drives the tenant portal UI exactly as a real tenant operator would — which makes it representative **and** fires the warmup that prevents the orphan.
## Non-Goals (explicitly out of scope — separate follow-up)
- Deeper render-pipeline reliability hardening for real users: server-side warmup guarantee, pinning the encoding-worker URL for a render's lifetime (the undone 2026-06-15 follow-up), and a backstop recovery scheduler for jobs stuck in `rendering_video`. These protect real users from the rarer orphan and will be tracked as their own effort. This spec relies on the frontend warmup (which real tenant users already fire) to keep the happy path green.
## Design
### Approach
Replace `scripts/e2e/tenant_e2e.py` with a single parametrized Playwright spec that mirrors `happy-path-real-user.spec.ts`, run per-tenant via the existing matrix.
### 1. File layout & workflow
- **New:** `frontend/e2e/production/tenant-happy-path.spec.ts` — parametrized by `TENANT_ID` (`vocalstar` | `singa`) and `TENANT_PORTAL_URL` env vars. Structured like the consumer happy-path test.
- **Reuse helpers:** `email-testing.ts` (magic link via testmail), `auth.ts` — including `clickCompleteSignInGate()` (added earlier this session for the #870 gate), `test-cleanup.ts`.
- **Delete:** `scripts/e2e/tenant_e2e.py`.
- **Rework** `.github/workflows/e2e-tenant-daily.yml`: swap the `python scripts/e2e/tenant_e2e.py` step for `npx playwright test tenant-happy-path.spec.ts --config=playwright.production.config.ts`. Keep the `vocalstar`/`singa` matrix + `max-parallel: 1`. Add the Node/Playwright/testmail setup the consumer daily job already uses (`TESTMAIL_API_KEY`, `TESTMAIL_NAMESPACE`, `E2E_ADMIN_TOKEN`); keep WIF auth only if still needed for cleanup/GCS. Pass `TENANT_ID`/`TENANT_PORTAL_URL` per matrix entry.
### 2. The journey — entirely through the tenant portal UI
On `https://{tenant}.nomadkaraoke.com`:
1. **Sign in** via the real magic-link flow: open AuthDialog → request link to a testmail inbox → poll testmail → navigate the magic link → **Complete Sign-In gate** (`clickCompleteSignInGate`) → land in `/app`. The issued token carries `tenant_id` (verified in June).
2. **Submit track** via the tenant's simplified form: fill unique Artist/Title, upload real mixed + instrumental audio (`#tenant-mixed-audio`, `#tenant-instrumental-audio`), click *Submit Track* → assert *Track Submitted* + capture `[data-testid="created-job-id"]`.
3. **Wait for transcription** by polling the tenant portal job list UI until the *Review Lyrics* link/affordance appears.
4. **Open the review UI and approve** — the crux. Loading the review page fires `warmup` + `heartbeat`, keeping the encoder alive. Proceed through the review → instrumental step and complete it (the UI calls `completeReview(..., 'custom', ...)`; tenant instrumental is pre-supplied → `'custom'`, matching June bug-fix #7).
5. **Wait for render** by polling the UI to completion.
6. **Verify tenant distribution** (same guarantees the old script asserted, read after the UI-driven flow): Dropbox link present; **no** YouTube URL; downloads present (4K/720p/with-vocals + CDG + TXT via `job.file_urls`); job `is_private`; locked theme.
7. **Cleanup** the job (reuse `test-cleanup.ts` / admin delete).
### 3. Auth — allowlist a dedicated E2E email domain
Tenant magic-link login is gated by `tenant_config.auth.allowed_email_domains` (`users.py:287`). Current live values:
- vocalstar: `vocal-star.com`, `vocalstarmusic.com`
- singa: `singa.com`, `nomadkaraoke.com`
CI can only read **testmail.app** inboxes, which are on neither allowlist. Per the design decision, add a **dedicated E2E domain** to both tenants' `allowed_email_domains` so the test logs in exactly like a real operator.
**Domain choice (to finalize in the plan):**
- **Preferred (dedicated, owned):** a domain we control that testmail can receive for — e.g. `e2e.nomadkaraoke.com` via testmail's custom-domain feature (requires an MX record + testmail plan support). Truly dedicated; does not widen the portal to a shared public domain.
- **Fallback (simple):** `inbox.testmail.app`. Zero setup, but shared — anyone with a testmail account could register on the B2B portal. Low real risk (tenant jobs forced `is_private`, no payment on signup), but a wider surface than the owned-domain option.
Apply the change in **both** `scripts/setup-{vocalstar,singa}-tenant.py` **and** the live GCS `tenants/{id}/config.json` (the setup scripts are the source of truth; the live config must be updated to take effect).
### 4. Coverage & test data
- **Both tenants** (`vocalstar`, `singa`) via the matrix, sequential (`max-parallel: 1`) — the shared single encoder can't do two concurrent renders.
- **Real audio pair** (existing `gs://…/e2e-tests/shared/{e2e-mixed,e2e-instrumental}.mp3`, or bundled test assets). **Unique Artist/Title per run** (encoder caches by base name — repeated identical titles collide).
## Open items to resolve during planning/implementation
1. **E2E domain**: confirm dedicated-owned (`e2e.nomadkaraoke.com` via testmail custom domain — check plan/DNS) vs. fallback `inbox.testmail.app`.
2. **Review-approve selectors**: the exact tenant review/instrumental UI affordances that drive `completeReview('custom')` — nail down against the real UI during implementation.
3. **WIF vs testmail workflow setup**: confirm what the reworked workflow still needs from GCP auth (cleanup/GCS) vs. the consumer test's testmail/admin-token setup.
4. **Credits**: none needed — tenant portals do not require credits (separate financial agreement / manual invoicing per tenant). No credit-granting step in the test.
## Testing / verification
- Iterate the spec locally against prod for one tenant first (whichever the finalized E2E domain supports), driving the real portal via Playwright, before wiring the workflow.
- Confirm the warmup fires (review-page load → `warmup`/`heartbeat` 200s) and the render completes without orphaning.
- Then enable the reworked workflow and confirm a clean scheduled/dispatched run for both tenants.
## Risks
- Testmail custom-domain feasibility (for the dedicated-owned option) is unconfirmed — may fall back to `inbox.testmail.app`.
- The underlying render-orphan can still theoretically occur if the encoder is stopped mid-render despite warmup (the deeper reliability follow-up covers this); the frontend warmup makes it unlikely but not impossible. Retain a sane render timeout and clear failure output.
@@ -0,0 +1,503 @@
# Tenant Portal Frontend E2E — Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Replace the API-bypassing `scripts/e2e/tenant_e2e.py` with a Playwright frontend-journey test that drives the tenant portal UI end-to-end, so the daily tenant E2E is representative of the real operator experience and fires the encoding-worker warmup that prevents stuck-render orphans.
**Architecture:** A single parametrized Playwright spec (`tenant-happy-path.spec.ts`), run per-tenant via the existing `e2e-tenant-daily.yml` matrix, that mirrors the consumer `happy-path-real-user.spec.ts`: magic-link login on the tenant subdomain → submit track via the tenant form → wait for transcription → open the review UI (fires `warmup`/`heartbeat`) → approve → wait for render → assert tenant distribution → cleanup.
**Tech Stack:** Playwright (`@playwright/test`), TypeScript, testmail.app (E2E email), GitHub Actions matrix, FastAPI backend (tenant config in GCS).
## Global Constraints
- **Verification model:** These are production E2E tests. A task's "test cycle" is *running the spec (or the built-up portion of it) against production* and observing the segment pass — not unit tests. Use `cd frontend && ./node_modules/.bin/playwright test tenant-happy-path.spec.ts --config=playwright.production.config.ts --project=chromium --reporter=list` with the required env vars set.
- **Required env vars for local runs:** `TESTMAIL_API_KEY`, `TESTMAIL_NAMESPACE`, `E2E_ADMIN_TOKEN` (admin token, `gcloud secrets versions access latest --secret=admin-tokens --project=nomadkaraoke | cut -d, -f1`), `TENANT_ID` (`vocalstar`|`singa`), `TENANT_PORTAL_URL` (`https://{tenant}.nomadkaraoke.com`).
- **testmail address domain is `inbox.testmail.app`** — hardcoded in `frontend/e2e/helpers/email-testing.ts:133`. The tenant allowlist decision (Task 1) is bound to this: allowlisting `inbox.testmail.app` is the minimal path; a truly-owned dedicated domain would additionally require testmail custom-domain support + a helper change (out of scope for this plan — see Task 1 note).
- **Encoder is a single shared worker** — tenants must run sequentially (`max-parallel: 1`), and every run must use a **unique Artist/Title** (encoder caches by base name; repeated titles collide).
- **Tenant distribution invariants** (assert these): Dropbox link present, **no** YouTube URL, downloads from `job.file_urls` present (finals: `lossy_4k_mp4`, `lossy_720p_mp4`, `with_vocals_mp4`; packages: `cdg_zip`, `txt_zip`), job `is_private`, locked theme.
- **No credits needed** — tenant portals do not require credits (manual invoicing per tenant). Do not add a credit-granting step.
- **Reuse existing helpers** — `createEmailHelper`/`isEmailTestingAvailable` (`email-testing.ts`), `clickCompleteSignInGate` (`auth.ts`), cleanup via admin `DELETE /api/admin/jobs/{id}` (`test-cleanup.ts` pattern). Do not reinvent them.
- **Commit style:** end commit messages with `Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>`. Bump `pyproject.toml` version once (Task 7).
## File Structure
- **Create:** `frontend/e2e/production/tenant-happy-path.spec.ts` — the parametrized tenant journey spec. One responsibility: drive one tenant's full portal journey and assert distribution.
- **Modify:** `scripts/setup-vocalstar-tenant.py`, `scripts/setup-singa-tenant.py` — add the E2E domain to `allowed_email_domains`.
- **Create:** `scripts/e2e/update_tenant_allowlist.py` — small idempotent script to apply the `allowed_email_domains` change to the **live** GCS `tenants/{id}/config.json` (setup scripts are source-of-truth; live config must also be updated).
- **Modify:** `.github/workflows/e2e-tenant-daily.yml` — run the Playwright spec per matrix tenant; add Node/Playwright/testmail setup; keep matrix + `max-parallel: 1`.
- **Delete:** `scripts/e2e/tenant_e2e.py`.
- **Modify:** `pyproject.toml` — version bump (Task 7).
---
### Task 1: Allowlist the E2E email domain on both tenants
**Files:**
- Modify: `scripts/setup-vocalstar-tenant.py` (the `allowed_email_domains` list)
- Modify: `scripts/setup-singa-tenant.py` (the `allowed_email_domains` list)
- Create: `scripts/e2e/update_tenant_allowlist.py`
**Note (decision):** Uses `inbox.testmail.app`. The spec's "dedicated owned domain" preference (`e2e.nomadkaraoke.com`) needs testmail custom-domain support + a change to the hardcoded domain in `email-testing.ts`; that is deferred. If the shared-domain surface is later deemed unacceptable, swap the constant here and in the helper. Risk is low: tenant jobs are forced `is_private`, no payment on signup.
**Interfaces:**
- Produces: both tenants' live config `auth.allowed_email_domains` contains `inbox.testmail.app`, so `POST /api/users/auth/magic-link` with a testmail address + `X-Tenant-ID: {tenant}` returns 200 (not domain-rejected).
- [ ] **Step 1: Add the domain to the setup scripts**
In `scripts/setup-vocalstar-tenant.py`, change:
```python
"allowed_email_domains": ["vocal-star.com", "vocalstarmusic.com"],
```
to:
```python
"allowed_email_domains": ["vocal-star.com", "vocalstarmusic.com", "inbox.testmail.app"],
```
In `scripts/setup-singa-tenant.py`, change:
```python
"allowed_email_domains": ["singa.com", "nomadkaraoke.com"],
```
to:
```python
"allowed_email_domains": ["singa.com", "nomadkaraoke.com", "inbox.testmail.app"],
```
- [ ] **Step 2: Write the live-config updater script**
Create `scripts/e2e/update_tenant_allowlist.py`:
```python
#!/usr/bin/env python3
"""Idempotently add inbox.testmail.app to a tenant's allowed_email_domains in
the live GCS config (tenants/{id}/config.json). Setup scripts are the source
of truth, but the live config must be updated to take effect.
Usage: python scripts/e2e/update_tenant_allowlist.py vocalstar singa
"""
import json
import sys
from google.cloud import storage
BUCKET = "karaoke-gen-storage-nomadkaraoke"
E2E_DOMAIN = "inbox.testmail.app"
def update(tenant_id: str) -> None:
client = storage.Client(project="nomadkaraoke")
blob = client.bucket(BUCKET).blob(f"tenants/{tenant_id}/config.json")
cfg = json.loads(blob.download_as_text())
auth = cfg.setdefault("auth", {})
domains = auth.setdefault("allowed_email_domains", [])
if E2E_DOMAIN in domains:
print(f"{tenant_id}: already allowlisted ({domains})")
return
domains.append(E2E_DOMAIN)
blob.upload_from_string(json.dumps(cfg, indent=2), content_type="application/json")
print(f"{tenant_id}: added {E2E_DOMAIN} -> {domains}")
if __name__ == "__main__":
for t in sys.argv[1:] or ["vocalstar", "singa"]:
update(t)
```
- [ ] **Step 3: Apply to live config**
Run: `cd <worktree> && python scripts/e2e/update_tenant_allowlist.py vocalstar singa`
Expected: prints `vocalstar: added inbox.testmail.app -> [...]` and same for singa. (Uses ADC — the read-only SA can read; if it lacks write, run with an admin-capable credential / break-glass per project convention. If write is blocked, STOP and notify.)
- [ ] **Step 4: Verify magic-link is accepted for a testmail address on the tenant portal**
Run:
```bash
curl -s -X POST "https://api.nomadkaraoke.com/api/users/auth/magic-link" \
-H "Content-Type: application/json" -H "X-Tenant-ID: vocalstar" \
-d '{"email": "'"$TESTMAIL_NAMESPACE"'.probe@inbox.testmail.app"}'
```
Expected: HTTP 200 / `{"status": ...}` (NOT a domain-rejection error). Repeat with `X-Tenant-ID: singa`.
- [ ] **Step 5: Commit**
```bash
git add scripts/setup-vocalstar-tenant.py scripts/setup-singa-tenant.py scripts/e2e/update_tenant_allowlist.py
git commit -m "feat(e2e): allowlist testmail domain for tenant magic-link login"
```
---
### Task 2: Scaffold the spec + magic-link login segment
**Files:**
- Create: `frontend/e2e/production/tenant-happy-path.spec.ts`
**Interfaces:**
- Consumes: `createEmailHelper`, `isEmailTestingAvailable` from `../helpers/email-testing`; `clickCompleteSignInGate` from `../helpers/auth`.
- Produces: an authenticated tenant session in `page` (localStorage `karaoke_access_token` set, token carries `tenant_id`), reached entirely via the portal UI.
- [ ] **Step 1: Create the spec with env parametrization + login segment**
```typescript
import { test, expect, Page } from '@playwright/test';
import { createEmailHelper, isEmailTestingAvailable, EmailHelper } from '../helpers/email-testing';
import { clickCompleteSignInGate } from '../helpers/auth';
const TENANT_ID = process.env.TENANT_ID || 'vocalstar';
const PORTAL_URL = process.env.TENANT_PORTAL_URL || `https://${TENANT_ID}.nomadkaraoke.com`;
const API_URL = 'https://api.nomadkaraoke.com';
const ADMIN_TOKEN = process.env.E2E_ADMIN_TOKEN || process.env.KARAOKE_ADMIN_TOKEN || '';
const T = { action: 30_000, transcription: 900_000, render: 1_800_000, full: 3_000_000 };
test.describe(`Tenant Portal Happy Path — ${TENANT_ID}`, () => {
test.describe.configure({ retries: 0 });
let emailHelper: EmailHelper | null = null;
let createdJobId: string | null = null;
test.beforeAll(async () => {
if (isEmailTestingAvailable()) emailHelper = await createEmailHelper();
});
test.afterAll(async () => {
if (createdJobId && ADMIN_TOKEN) {
await fetch(`${API_URL}/api/admin/jobs/${createdJobId}`, {
method: 'DELETE', headers: { 'X-Admin-Token': ADMIN_TOKEN },
}).catch(() => {});
}
});
test(`Full tenant journey: login -> submit -> review -> render -> distribution`, async ({ page, context }) => {
test.skip(!emailHelper, 'Email testing not configured (TESTMAIL_API_KEY/NAMESPACE)');
test.setTimeout(T.full);
// ---- Sign in via magic link on the tenant portal ----
const inbox = await emailHelper!.createInbox();
await page.goto(`${PORTAL_URL}/app`, { waitUntil: 'domcontentloaded', timeout: T.action });
const signInButton = page.getByRole('button', { name: /sign (in|up)/i }).first();
await expect(signInButton).toBeVisible({ timeout: T.action });
await signInButton.click();
const authDialog = page.getByRole('dialog');
await expect(authDialog).toBeVisible({ timeout: T.action });
await authDialog.getByPlaceholder('you@example.com').fill(inbox.emailAddress);
await page.getByRole('button', { name: /send sign-in link/i }).click();
await expect(page.getByText(/check your email/i)).toBeVisible({ timeout: 15_000 });
const email = await emailHelper!.waitForEmail(inbox.id, 60_000);
const magicLink = emailHelper!.extractMagicLink(email);
if (!magicLink) throw new Error('Could not extract magic link from tenant sign-in email');
await page.goto(magicLink);
await clickCompleteSignInGate(page);
await page.waitForURL(/\/app/, { timeout: T.action });
const token = await page.evaluate(() => localStorage.getItem('karaoke_access_token'));
expect(token, 'authenticated token present after magic-link login').toBeTruthy();
console.log(`[${TENANT_ID}] Signed in via magic-link UI`);
});
});
```
- [ ] **Step 2: Run the login segment against prod (singa first — the allowlist from Task 1 is live)**
Run: `cd frontend && ./node_modules/.bin/playwright test tenant-happy-path.spec.ts --config=playwright.production.config.ts --project=chromium --reporter=list`
(with `TENANT_ID=singa TENANT_PORTAL_URL=https://singa.nomadkaraoke.com` + testmail/admin env).
Expected: reaches `/app`, logs `Signed in via magic-link UI`, token present. (The test will continue past login only once later tasks add steps; for now it ends after the login assertion.)
- [ ] **Step 3: Confirm the token carries the tenant_id**
Add a temporary log of the decoded token payload or verify via `GET /api/users/me` with the token that the session is tenant-scoped. Expected: `tenant_id == singa`. Remove the temporary log before commit.
- [ ] **Step 4: Commit**
```bash
git add frontend/e2e/production/tenant-happy-path.spec.ts
git commit -m "feat(e2e): tenant-happy-path spec scaffold + magic-link login segment"
```
---
### Task 3: Submit-track segment
**Files:**
- Modify: `frontend/e2e/production/tenant-happy-path.spec.ts`
**Interfaces:**
- Consumes: authenticated `page` from Task 2.
- Produces: `createdJobId` set to the submitted job's short id; job is transcribing.
**Test audio:** download the shared E2E pair to temp files at test start:
`gs://karaoke-gen-storage-nomadkaraoke/e2e-tests/shared/e2e-mixed.mp3` and `.../e2e-instrumental.mp3`. Fetch via the public/authorized GCS path or a signed URL helper; if not directly fetchable in CI, stage them as repo test assets under `frontend/e2e/fixtures/`. Confirm the fetch path during this task's run.
- [ ] **Step 1: Add the submit segment after login**
Insert before the closing of the test body:
```typescript
// ---- Submit a track via the tenant form ----
const uniqueTitle = `E2E ${TENANT_ID} ${new Date().toISOString().replace(/[:.]/g, '-')}`;
await page.getByLabel('Artist').fill('Nomad E2E');
await page.getByLabel('Title').fill(uniqueTitle);
await page.locator('#tenant-mixed-audio').setInputFiles(mixedAudioPath);
await page.locator('#tenant-instrumental-audio').setInputFiles(instrumentalAudioPath);
await page.getByRole('button', { name: /submit track/i }).click();
await expect(page.getByText(/track submitted/i)).toBeVisible({ timeout: 120_000 });
const jobIdText = await page.locator('[data-testid="created-job-id"]').textContent();
createdJobId = jobIdText?.replace(/^ID:\s*/i, '').trim() || null;
expect(createdJobId, 'created job id captured from tenant submit').toBeTruthy();
console.log(`[${TENANT_ID}] Submitted job ${createdJobId}: ${uniqueTitle}`);
```
Add the audio-staging code in `beforeAll` (or top of test) producing `mixedAudioPath` / `instrumentalAudioPath` (temp files). Use a unique title per run (constraint).
- [ ] **Step 2: Run login+submit against prod (singa)**
Run: same command as Task 2 Step 2.
Expected: `Track Submitted` appears, `createdJobId` logged. The `afterAll` cleanup deletes the job.
- [ ] **Step 3: Commit**
```bash
git add frontend/e2e/production/tenant-happy-path.spec.ts
git commit -m "feat(e2e): tenant submit-track segment via portal form"
```
---
### Task 4: Wait-for-transcription + open-review (warmup) + approve segment
**Files:**
- Modify: `frontend/e2e/production/tenant-happy-path.spec.ts`
**Interfaces:**
- Consumes: `createdJobId`, authenticated `page`.
- Produces: job past review (status left `awaiting_review`), review-page load having fired `warmup`+`heartbeat`.
**Pattern to follow:** `happy-path-real-user.spec.ts` Steps 7–8 (poll job card for a `Review Lyrics` link → open review page in a new tab → the page fires `warmupEncodingWorker(jobId)` on load → proceed to completion). The exact tenant review→approve affordance that calls `completeReview(..., 'custom')` must be confirmed against the live tenant review UI during this task's run (this is the legitimate E2E test cycle). Opening the review page is the essential warmup-firing action.
- [ ] **Step 1: Add poll-for-review-readiness**
```typescript
// ---- Wait until the job is ready for review (poll the portal UI) ----
const jobCard = page.locator(`text=${createdJobId}`).locator('xpath=ancestor::*[self::div][1]');
let reviewLink = null;
const deadline = Date.now() + T.transcription;
while (Date.now() < deadline) {
await page.reload({ waitUntil: 'domcontentloaded' });
const link = page.getByRole('link', { name: /review lyrics/i }).first();
if (await link.isVisible().catch(() => false)) { reviewLink = link; break; }
await page.waitForTimeout(15_000);
}
expect(reviewLink, 'Review Lyrics link appeared before transcription timeout').not.toBeNull();
```
- [ ] **Step 2: Open the review page (fires warmup) and capture the warmup request**
```typescript
const reviewHref = await reviewLink!.getAttribute('href');
const reviewUrl = reviewHref!.startsWith('http') ? reviewHref! : `${PORTAL_URL}${reviewHref}`;
const reviewPage = await context.newPage();
// Assert the warmup fires on review-page load (the whole point of the UI journey)
const warmupReq = reviewPage.waitForRequest(
(r) => r.url().includes('/api/internal/encoding-worker/warmup/'),
{ timeout: 60_000 },
);
await reviewPage.goto(reviewUrl, { waitUntil: 'domcontentloaded' });
await warmupReq;
console.log(`[${TENANT_ID}] Review page fired encoding-worker warmup`);
```
- [ ] **Step 3: Complete the review through the UI**
Follow the happy-path Step 7–8 flow adapted for the tenant (pre-supplied instrumental → `'custom'`). Confirm the exact finalize affordance against the live tenant review UI while running this task; encode the confirmed selectors here. The completion triggers `completeReview(jobId, ..., 'custom', ...)` and moves the job into the render pipeline.
- [ ] **Step 4: Run login→submit→review-approve against prod (singa)**
Run: same command.
Expected: logs the warmup firing, review completes, job leaves `awaiting_review` (verify via a `GET /api/jobs/{id}` status check with the session token or admin token).
- [ ] **Step 5: Commit**
```bash
git add frontend/e2e/production/tenant-happy-path.spec.ts
git commit -m "feat(e2e): tenant review-open (warmup) + approve segment"
```
---
### Task 5: Wait-for-render + distribution assertions
**Files:**
- Modify: `frontend/e2e/production/tenant-happy-path.spec.ts`
**Interfaces:**
- Consumes: `createdJobId`, session token.
- Produces: full green journey for one tenant (singa).
- [ ] **Step 1: Poll to render completion, then assert tenant distribution**
```typescript
// ---- Wait for render to complete, then assert tenant distribution ----
const token2 = await page.evaluate(() => localStorage.getItem('karaoke_access_token'));
const getJob = async () => {
const r = await fetch(`${API_URL}/api/jobs/${createdJobId}`, {
headers: { Authorization: `Bearer ${token2}` },
});
if (!r.ok) throw new Error(`GET job ${createdJobId}: ${r.status}`);
return r.json();
};
const renderDeadline = Date.now() + T.render;
let job: any = null;
while (Date.now() < renderDeadline) {
job = await getJob();
if (job.status === 'complete') break;
if (job.status === 'failed' || job.status === 'error') {
throw new Error(`Render failed: status=${job.status} ${JSON.stringify(job.state_data?.error || '')}`);
}
await new Promise((res) => setTimeout(res, 15_000));
}
expect(job?.status, 'job reached complete before render timeout').toBe('complete');
const fileUrls = job.file_urls || {};
const finals = Object.keys(fileUrls.finals || {});
const packages = Object.keys(fileUrls.packages || {});
expect(job.dropbox_url || job.state_data?.dropbox_url, 'Dropbox link present').toBeTruthy();
expect(job.youtube_url || null, 'no YouTube upload for tenant').toBeFalsy();
for (const f of ['lossy_4k_mp4', 'lossy_720p_mp4', 'with_vocals_mp4']) expect(finals).toContain(f);
for (const p of ['cdg_zip', 'txt_zip']) expect(packages).toContain(p);
expect(job.is_private, 'tenant job is private').toBeTruthy();
console.log(`[${TENANT_ID}] PASSED — complete, Dropbox present, no YouTube, downloads available`);
```
(Confirm the exact field names — `dropbox_url` vs `state_data.dropbox_url`, `is_private` location — against a real completed job's `GET /api/jobs/{id}` payload during this run; the old `tenant_e2e.py` read `file_urls` finals/packages, `dropbox_link`, `youtube_url`.)
- [ ] **Step 2: Run the FULL journey against prod (singa) end-to-end**
Run: same command, allow up to ~30–40 min.
Expected: `[singa] PASSED …`. This is the first full green frontend journey.
- [ ] **Step 3: Commit**
```bash
git add frontend/e2e/production/tenant-happy-path.spec.ts
git commit -m "feat(e2e): tenant render-wait + distribution assertions (full journey green)"
```
---
### Task 6: Confirm vocalstar + robust cleanup
**Files:**
- Modify: `frontend/e2e/production/tenant-happy-path.spec.ts` (only if selector/field differences surface)
- [ ] **Step 1: Run the full journey for vocalstar**
Run: same command with `TENANT_ID=vocalstar TENANT_PORTAL_URL=https://vocalstar.nomadkaraoke.com`.
Expected: `[vocalstar] PASSED …`. Fix any tenant-specific selector/field differences inline (both tenants use the same portal app, so differences should be minimal).
- [ ] **Step 2: Verify cleanup deletes the job for both tenants**
Confirm `afterAll` deleted each test job (`GET /api/jobs/{id}` → 404, or check admin). No orphaned E2E jobs left.
- [ ] **Step 3: Commit (if any changes)**
```bash
git add frontend/e2e/production/tenant-happy-path.spec.ts
git commit -m "fix(e2e): tenant-happy-path passes for both vocalstar and singa"
```
---
### Task 7: Rework the workflow + delete the old script + version bump
**Files:**
- Modify: `.github/workflows/e2e-tenant-daily.yml`
- Delete: `scripts/e2e/tenant_e2e.py`
- Modify: `pyproject.toml`
- [ ] **Step 1: Replace the run step in the workflow**
Replace the Python "Run tenant E2E" step with the Playwright run, keeping the `matrix` (`vocalstar`, `singa`) and `max-parallel: 1`. Add the Node/Playwright/testmail setup that the consumer daily job (`e2e-daily.yml`) already uses. Concretely, the job's steps become:
```yaml
- name: Checkout code
uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with: { node-version: '20', cache: 'npm', cache-dependency-path: frontend/package-lock.json }
- name: Install dependencies
working-directory: frontend
run: npm ci --legacy-peer-deps
- name: Install Playwright browsers
working-directory: frontend
run: npx playwright install --with-deps chromium
- name: Ensure testmail.app domain is allowlisted (rate-limit)
run: |
curl -sf -X POST "https://api.nomadkaraoke.com/api/admin/rate-limits/blocklists/allowlisted-domains" \
-H "X-Admin-Token: ${{ secrets.E2E_ADMIN_TOKEN }}" -H "Content-Type: application/json" \
-d '{"domain": "inbox.testmail.app"}' || echo "Warning: allowlist may already exist"
- name: Run tenant E2E (frontend journey)
id: run_test
working-directory: frontend
env:
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
TESTMAIL_NAMESPACE: ${{ secrets.TESTMAIL_NAMESPACE }}
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
TENANT_ID: ${{ matrix.tenant }}
TENANT_PORTAL_URL: https://${{ matrix.tenant }}.nomadkaraoke.com
run: |
npx playwright test tenant-happy-path.spec.ts \
--config=playwright.production.config.ts --project=chromium \
--reporter=list --timeout=2400000 2>&1 | tee e2e-output.log
echo "exit_code=${PIPESTATUS[0]}" >> "$GITHUB_OUTPUT"
continue-on-error: true
```
Keep the existing "Upload logs", "Discord notification (failure)", "Fail the job", and `notify-email` steps; point the log upload at `frontend/e2e-output.log` and `frontend/test-results/`. Drop the GCP WIF auth + `pip install` steps unless still needed (they were for the Python script's Secret Manager/GCS access — the Playwright test uses `E2E_ADMIN_TOKEN` directly; remove if unused).
- [ ] **Step 2: Delete the old Python script**
```bash
git rm scripts/e2e/tenant_e2e.py
```
- [ ] **Step 3: Bump version**
In `pyproject.toml`, bump the patch version (e.g. `0.190.3` → `0.190.4`).
- [ ] **Step 4: Commit**
```bash
git add .github/workflows/e2e-tenant-daily.yml pyproject.toml
git commit -m "ci(e2e): run tenant daily via Playwright frontend journey; drop API script"
```
---
### Task 8: Final end-to-end verification via the workflow
- [ ] **Step 1: Dispatch the reworked workflow against the branch, off-peak if possible**
Run: `gh workflow run e2e-tenant-daily.yml --repo nomadkaraoke/karaoke-gen --ref <branch>`
(Only the default branch may be dispatchable; if so, this runs after merge. Prefer a low-traffic window given the shared encoder.)
- [ ] **Step 2: Watch both matrix jobs to green**
Confirm `Tenant E2E: vocalstar` and `Tenant E2E: singa` both succeed, and the logs show the warmup firing + render completing (no `rendering_video` orphan). If a job stalls at `rendering_video`, capture logs — that would indicate the warmup did not prevent the orphan and the separate reliability follow-up is needed.
- [ ] **Step 3: Confirm no leftover E2E jobs**
Check admin/job list for orphaned `Nomad E2E` jobs; none should remain.
---
## Self-Review
**Spec coverage:**
- Representative frontend journey (login→submit→review→render→verify) → Tasks 2–6. ✓
- Delete `tenant_e2e.py` / rework workflow → Task 7. ✓
- Allowlist dedicated E2E domain on both tenants (scripts + live config) → Task 1. ✓
- Both tenants, sequential, unique title, real audio → Tasks 3, 6, Global Constraints. ✓
- Distribution invariants → Task 5 + Global Constraints. ✓
- No credits step → Global Constraints. ✓
- Warmup-fires assertion (the crux) → Task 4 Step 2. ✓
- Non-goal (reliability hardening) → explicitly excluded; Task 8 Step 2 flags if warmup proves insufficient. ✓
**Placeholder scan:** Two spots require live-UI confirmation (Task 4 Step 3 review-approve affordance; Task 5 Step 1 exact job field names). These are inherent to E2E authoring against a live UI, not deferred implementation — each names the exact mechanism, the reference (`happy-path` Steps 7–8; old `tenant_e2e.py` fields), and the run in which to confirm. Acceptable; not fixable ahead of a live run without guessing selectors.
**Type/name consistency:** `createdJobId`, `emailHelper`, `PORTAL_URL`, `TENANT_ID`, `reviewUrl`, `getJob()` used consistently across tasks. Helper names (`createEmailHelper`, `waitForEmail`, `extractMagicLink`, `clickCompleteSignInGate`) match the real exports verified in the source.
+1
View File
@@ -23,3 +23,4 @@ messages/.en-snapshot.json
# Generated at build time by scripts/write-version.mjs
public/version.json
e2e/fixtures/*.mp3
@@ -0,0 +1,280 @@
import { test, expect } from '@playwright/test';
import * as path from 'path';
import * as fs from 'fs';
import { createEmailHelper, isEmailTestingAvailable, EmailHelper } from '../helpers/email-testing';
import { clickCompleteSignInGate } from '../helpers/auth';
// Real test audio (mixed + instrumental), gitignored under e2e/fixtures.
// Populate with scripts/e2e/fetch_test_audio.sh (CI + local).
const MIXED_AUDIO = path.resolve(__dirname, '../fixtures/e2e-mixed.mp3');
const INSTRUMENTAL_AUDIO = path.resolve(__dirname, '../fixtures/e2e-instrumental.mp3');
/**
* Tenant Portal Happy Path — Production E2E (frontend journey)
*
* Drives the full tenant operator journey through the tenant portal UI, exactly
* as a real operator would, so it is representative of the user experience AND
* fires the encoding-worker warmup (on review-page load) that keeps the encoder
* alive for the render. Replaces the API-driven scripts/e2e/tenant_e2e.py.
*
* Parametrized per tenant via env:
* TENANT_ID vocalstar | singa (default: vocalstar)
* TENANT_PORTAL_URL https://{tenant}.nomadkaraoke.com
*
* Requires: TESTMAIL_API_KEY, TESTMAIL_NAMESPACE, E2E_ADMIN_TOKEN.
* The tenant must allowlist inbox.testmail.app in auth.allowed_email_domains.
*/
const TENANT_ID = process.env.TENANT_ID || 'vocalstar';
const PORTAL_URL = process.env.TENANT_PORTAL_URL || `https://${TENANT_ID}.nomadkaraoke.com`;
const API_URL = 'https://api.nomadkaraoke.com';
const ADMIN_TOKEN = process.env.E2E_ADMIN_TOKEN || process.env.KARAOKE_ADMIN_TOKEN || '';
const T = {
action: 30_000,
transcription: 900_000, // 15 min to reach review
render: 1_800_000, // 30 min to complete render
full: 3_000_000, // 50 min whole-test budget
};
test.describe(`Tenant Portal Happy Path — ${TENANT_ID}`, () => {
test.describe.configure({ retries: 0 });
let emailHelper: EmailHelper | null = null;
let createdJobId: string | null = null;
test.beforeAll(async () => {
if (isEmailTestingAvailable()) emailHelper = await createEmailHelper();
});
test.afterAll(async () => {
if (createdJobId && ADMIN_TOKEN) {
// DELETE /api/jobs/{id} uses require_auth + ownership; admins bypass.
await fetch(`${API_URL}/api/jobs/${createdJobId}`, {
method: 'DELETE',
headers: { Authorization: `Bearer ${ADMIN_TOKEN}` },
}).catch(() => {});
console.log(`[${TENANT_ID}] Cleaned up job ${createdJobId}`);
}
});
test('Full tenant journey: login -> submit -> review -> render -> distribution', async ({ page }) => {
test.skip(!emailHelper, 'Email testing not configured (TESTMAIL_API_KEY/NAMESPACE)');
test.skip(
!fs.existsSync(MIXED_AUDIO) || !fs.existsSync(INSTRUMENTAL_AUDIO),
'Test audio missing — run scripts/e2e/fetch_test_audio.sh',
);
test.setTimeout(T.full);
// The shared prod config sets a global `x-playwright-test` header on every
// request (playwright.production.config.ts). On the cross-origin PUT to the
// GCS signed URL that header makes the browser's CORS preflight request an
// extra header the bucket's CORS policy (Content-Type only) doesn't allow,
// so the upload fails — a test-harness artifact, not a product bug (verified
// the same PUT succeeds in a real browser). Strip it only for GCS uploads.
await page.route('https://storage.googleapis.com/**', async (route) => {
const headers = { ...route.request().headers() };
delete headers['x-playwright-test'];
await route.continue({ headers });
});
// =====================================================================
// Sign in via the tenant portal magic-link UI
// =====================================================================
const inbox = await emailHelper!.createInbox();
console.log(`[${TENANT_ID}] Test inbox: ${inbox.emailAddress}`);
await page.goto(`${PORTAL_URL}/app`, { waitUntil: 'domcontentloaded', timeout: T.action });
// Tenant landing shows a single "Sign In" button that opens the auth dialog.
const signInTrigger = page.getByRole('button', { name: /sign in/i }).first();
await expect(signInTrigger).toBeVisible({ timeout: T.action });
await signInTrigger.click();
const authDialog = page.getByRole('dialog');
await expect(authDialog).toBeVisible({ timeout: T.action });
await authDialog.getByPlaceholder('you@example.com').fill(inbox.emailAddress);
const sendLinkButton = authDialog.getByRole('button', { name: /send sign-in link/i });
await expect(sendLinkButton).toBeEnabled({ timeout: T.action });
await sendLinkButton.click();
await expect(page.getByText(/check your email/i)).toBeVisible({ timeout: 15_000 });
console.log(`[${TENANT_ID}] Magic-link email requested`);
const email = await emailHelper!.waitForEmail(inbox.id, 60_000);
const magicLink = emailHelper!.extractMagicLink(email);
if (!magicLink) {
console.log(' Email body preview:', (email.body || '').substring(0, 500));
throw new Error('Could not extract magic link from tenant sign-in email');
}
console.log(`[${TENANT_ID}] Magic link received`);
await page.goto(magicLink);
// Verify page gates behind an explicit "Complete Sign-In" click (see #870).
await clickCompleteSignInGate(page);
// New users (incl. first tenant login) see a credit interstitial; returning
// users see "Successfully signed in!". Handle both, then land on /app.
const verifyResult = await Promise.race([
page.getByText(/successfully signed in/i).waitFor({ state: 'visible', timeout: T.action }).then(() => 'success'),
page.getByText(/welcome to nomad karaoke/i).waitFor({ state: 'visible', timeout: T.action }).then(() => 'credits_interstitial'),
page.getByText(/sign-in failed/i).waitFor({ state: 'visible', timeout: T.action }).then(() => 'error'),
]);
if (verifyResult === 'error') {
const detail = await page.locator('.text-muted-foreground').first().textContent().catch(() => null);
throw new Error(`Tenant magic-link verification failed: ${detail ?? 'unknown'}`);
}
if (verifyResult === 'credits_interstitial') {
const startButton = page.getByRole('button', { name: /start creating|go to dashboard|explore the app/i });
await expect(startButton).toBeVisible({ timeout: T.action });
await startButton.click();
}
await page.waitForURL(/\/app/, { timeout: T.action });
const token = await page.evaluate(() => localStorage.getItem('karaoke_access_token'));
expect(token, 'authenticated token present after magic-link login').toBeTruthy();
console.log(`[${TENANT_ID}] Signed in via magic-link UI (${verifyResult})`);
// =====================================================================
// Submit a track through the tenant portal form
// =====================================================================
const uniqueTitle = `E2E ${TENANT_ID} ${new Date().toISOString().replace(/[:.]/g, '-')}`;
await page.getByLabel('Artist').fill('Nomad E2E');
await page.getByLabel('Title').fill(uniqueTitle);
await page.locator('#tenant-mixed-audio').setInputFiles(MIXED_AUDIO);
await page.locator('#tenant-instrumental-audio').setInputFiles(INSTRUMENTAL_AUDIO);
await page.getByRole('button', { name: /submit track/i }).click();
await expect(page.getByText(/track submitted/i)).toBeVisible({ timeout: 120_000 });
const jobIdText = await page.locator('[data-testid="created-job-id"]').textContent();
createdJobId = (jobIdText || '').replace(/^ID:\s*/i, '').trim() || null;
expect(createdJobId, 'created job id captured from tenant submit').toBeTruthy();
console.log(`[${TENANT_ID}] Submitted job ${createdJobId}: ${uniqueTitle}`);
// =====================================================================
// Wait for transcription, then open the review UI (fires warmup)
// =====================================================================
await page.goto(`${PORTAL_URL}/app`, { waitUntil: 'domcontentloaded' });
let reviewLink = null;
const reviewDeadline = Date.now() + T.transcription;
while (Date.now() < reviewDeadline) {
const link = page.getByRole('link', { name: /review lyrics/i }).first();
// waitFor (not instant isVisible) so the client-side job-list fetch has
// time to render each iteration; reload + retry if not ready yet.
try {
await link.waitFor({ state: 'visible', timeout: 15_000 });
reviewLink = link;
break;
} catch {
await page.reload({ waitUntil: 'domcontentloaded' });
}
}
expect(reviewLink, 'Review Lyrics link appeared before transcription timeout').not.toBeNull();
console.log(`[${TENANT_ID}] Job ready for review`);
const reviewHref = await reviewLink!.getAttribute('href');
const reviewUrl = reviewHref!.startsWith('http') ? reviewHref! : `${PORTAL_URL}${reviewHref}`;
// The review page must fire the encoding-worker warmup on load — this is
// what keeps the encoder alive for the render (and what the old API-driven
// test skipped, causing stuck-render orphans).
const warmupReq = page.waitForRequest(
(r) => r.url().includes('/api/internal/encoding-worker/warmup/'),
{ timeout: 60_000 },
);
await page.goto(reviewUrl, { waitUntil: 'domcontentloaded' });
await warmupReq;
console.log(`[${TENANT_ID}] Review page fired encoding-worker warmup`);
// =====================================================================
// Approve the review through the UI (Preview -> Instrumental -> Confirm)
// =====================================================================
// Preview Video (bottom of the review page) — generates a preview using the
// now-warm encoder, then opens a modal with a "Proceed to Instrumental" CTA.
await page.evaluate(() => window.scrollTo(0, document.body.scrollHeight));
const previewBtn = page.getByRole('button', { name: /preview video/i });
await expect(previewBtn).toBeVisible({ timeout: T.action });
await previewBtn.click();
const modal = page.getByRole('dialog');
await expect(modal).toBeVisible({ timeout: T.action });
// Preview generation can take a bit; don't fail the journey if it's slow.
await page.getByText(/generating preview video/i)
.waitFor({ state: 'hidden', timeout: 180_000 })
.catch(() => {});
const proceedBtn = page.getByRole('button', { name: /proceed to instrumental/i });
await expect(proceedBtn).toBeVisible({ timeout: T.action });
await proceedBtn.click();
const sessionToken = await page.evaluate(() => localStorage.getItem('karaoke_access_token'));
const getJob = async () => {
const r = await fetch(`${API_URL}/api/jobs/${createdJobId}`, {
headers: { Authorization: `Bearer ${sessionToken}` },
});
if (!r.ok) throw new Error(`GET job ${createdJobId}: ${r.status}`);
return r.json();
};
// For a pre-supplied (tenant) instrumental there is nothing to pick, so
// "Proceed to Instrumental Review" completes the review directly and the
// instrumental route redirects to /app. For a selectable instrumental an
// InstrumentalSelector (#submit-btn) renders instead. Handle both.
const submitBtn = page.locator('#submit-btn');
let hasInstrumentalStep = false;
try {
// waitFor actually blocks up to the timeout (isVisible would not); if the
// selector never appears the review already completed on "Proceed".
await submitBtn.waitFor({ state: 'visible', timeout: 15_000 });
hasInstrumentalStep = true;
} catch { /* pre-supplied tenant instrumental: review completed directly */ }
if (hasInstrumentalStep) {
const opt = page.locator('.selection-option').first();
if (await opt.isVisible().catch(() => false)) await opt.click();
await submitBtn.click();
console.log(`[${TENANT_ID}] Instrumental confirmed`);
}
// Confirm the review actually advanced (left awaiting_review) before waiting
// on the render — otherwise a silent no-op would masquerade as a render hang.
let left = false;
for (let i = 0; i < 8; i++) {
const s = (await getJob()).status;
if (s !== 'awaiting_review' && s !== 'in_review' && s !== 'reviewing') { left = true; break; }
await new Promise((res) => setTimeout(res, 5_000));
}
expect(left, 'review completed (job left awaiting_review)').toBeTruthy();
console.log(`[${TENANT_ID}] Review approved -> render started`);
// =====================================================================
// Wait for the render to complete, then assert tenant distribution
// =====================================================================
let job: any = null;
const renderDeadline = Date.now() + T.render;
while (Date.now() < renderDeadline) {
job = await getJob();
if (job.status === 'complete') break;
if (job.status === 'failed' || job.status === 'error') {
throw new Error(`Render failed: status=${job.status} ${JSON.stringify(job.state_data?.error || '')}`);
}
await new Promise((res) => setTimeout(res, 15_000));
}
expect(job?.status, 'job reached complete before render timeout').toBe('complete');
console.log(`[${TENANT_ID}] Render complete`);
// Tenant distribution invariants (field paths per the proven tenant flow:
// dropbox_link + youtube_url live in state_data; downloads in file_urls).
const sd = job.state_data || {};
const fileUrls = job.file_urls || {};
const finals = Object.keys(fileUrls.finals || {});
const packages = Object.keys(fileUrls.packages || {});
expect(sd.dropbox_link || job.dropbox_link, 'Dropbox link present').toBeTruthy();
expect(sd.youtube_url || job.youtube_url || null, 'no YouTube upload for tenant').toBeFalsy();
for (const f of ['lossy_4k_mp4', 'lossy_720p_mp4', 'with_vocals_mp4']) expect(finals).toContain(f);
for (const p of ['cdg_zip', 'txt_zip']) expect(packages).toContain(p);
expect(job.is_private, 'tenant job is private').toBeTruthy();
console.log(`[${TENANT_ID}] PASSED — complete, Dropbox present, no YouTube, downloads available`);
});
});
+1 -1
View File
@@ -1,6 +1,6 @@
[tool.poetry]
name = "karaoke-gen"
version = "0.190.3"
version = "0.190.4"
description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens."
authors = ["Andrew Beveridge <andrew@beveridge.uk>"]
license = "MIT"
+11
View File
@@ -0,0 +1,11 @@
#!/usr/bin/env bash
# Download the shared E2E test audio pair (mixed + instrumental) used by the
# tenant frontend E2E. Gitignored — not committed (12MB). Requires gcloud auth
# (WIF in CI, ADC locally) with read access to the storage bucket.
set -euo pipefail
DEST="${1:-frontend/e2e/fixtures}"
SRC="gs://karaoke-gen-storage-nomadkaraoke/e2e-tests/shared/short"
mkdir -p "$DEST"
gcloud storage cp "$SRC/e2e-mixed.mp3" "$DEST/e2e-mixed.mp3"
gcloud storage cp "$SRC/e2e-instrumental.mp3" "$DEST/e2e-instrumental.mp3"
echo "Fetched test audio to $DEST"
-230
View File
@@ -1,230 +0,0 @@
#!/usr/bin/env python3
"""
Production end-to-end health check for a white-label tenant portal.
Drives the exact path a tenant uses: submit a job with X-Tenant-ID + the tenant's own
mixed/instrumental audio, let it transcribe, approve the lyrics review, wait for the
render, and assert the tenant distribution was applied (locked theme, no YouTube, no
Google Drive, Dropbox folder + downloadable outputs). Cleans up the job and its uploads.
Designed to run daily per tenant in GitHub Actions (see .github/workflows/e2e-tenant-daily.yml)
and locally for ad-hoc verification.
Auth / credentials:
- Admin token: env KARAOKE_ADMIN_TOKEN, else Secret Manager (admin-tokens) via ADC.
- Test audio + cleanup: Google Cloud Storage via ADC.
In CI these are provided by Workload Identity Federation (google-github-actions/auth).
Usage:
python scripts/e2e/tenant_e2e.py vocalstar
python scripts/e2e/tenant_e2e.py singa
"""
import os
import sys
import time
import json
import tempfile
import urllib.request
import urllib.error
API = os.environ.get("KARAOKE_API_URL", "https://api.nomadkaraoke.com")
BUCKET = os.environ.get("KARAOKE_GCS_BUCKET", "karaoke-gen-storage-nomadkaraoke")
# Shared test pair (real audio with vocals so transcription has something to align).
TEST_MIXED = f"gs://{BUCKET}/e2e-tests/shared/e2e-mixed.mp3"
TEST_INSTRUMENTAL = f"gs://{BUCKET}/e2e-tests/shared/e2e-instrumental.mp3"
# Clearly-labelled so the daily output is identifiable in the tenant's Dropbox folder.
# Title is tenant-specific: the GCE encoder caches by base_name (artist - title), so two
# tenants running the matrix concurrently must NOT share a base_name or one would receive
# the other's (differently-themed) cached encode.
ARTIST = "Nomad Karaoke"
TITLE = None # set per-tenant below
EXPECTED = {
"vocalstar": {"theme": "vocalstar"},
"singa": {"theme": "singa"},
}
REVIEW_TIMEOUT = int(os.environ.get("E2E_REVIEW_TIMEOUT", "900")) # 15 min to transcription/review
RENDER_TIMEOUT = int(os.environ.get("E2E_RENDER_TIMEOUT", "1800")) # 30 min to full render
class TransientEncoderError(Exception):
"""A known, non-tenant-specific transient failure from the shared GCE encoder."""
def log(msg):
print(msg, flush=True)
def admin_token():
tok = os.environ.get("KARAOKE_ADMIN_TOKEN")
if tok:
return tok.split(",")[0].strip()
from google.cloud import secretmanager
c = secretmanager.SecretManagerServiceClient()
resp = c.access_secret_version(request={
"name": "projects/nomadkaraoke/secrets/admin-tokens/versions/latest"})
return resp.payload.data.decode().split(",")[0].strip()
def gcs_download(gs_uri, local_path):
from google.cloud import storage
bucket, _, blob = gs_uri[5:].partition("/")
storage.Client().bucket(bucket).blob(blob).download_to_filename(local_path)
def gcs_delete_prefix(prefix):
from google.cloud import storage
client = storage.Client()
b = client.bucket(BUCKET)
for blob in b.list_blobs(prefix=prefix):
blob.delete()
TENANT = sys.argv[1] if len(sys.argv) > 1 else os.environ.get("E2E_TENANT", "vocalstar")
if TENANT not in EXPECTED:
log(f"Unknown tenant: {TENANT}")
sys.exit(2)
TITLE = f"E2E Health Check {TENANT}"
TOKEN = admin_token()
def req(method, path, body=None, timeout=180):
url = path if path.startswith("http") else API + path
data = json.dumps(body).encode() if body is not None else None
headers = {"Authorization": f"Bearer {TOKEN}", "X-Tenant-ID": TENANT}
if data is not None:
headers["Content-Type"] = "application/json"
r = urllib.request.Request(url, data=data, headers=headers, method=method)
try:
with urllib.request.urlopen(r, timeout=timeout) as resp:
raw = resp.read().decode()
return resp.status, (json.loads(raw) if raw else {})
except urllib.error.HTTPError as e:
return e.code, {"error": e.read().decode()[:400]}
def put_file(signed_url, local_path, content_type):
with open(local_path, "rb") as f:
data = f.read()
r = urllib.request.Request(signed_url, data=data, headers={"Content-Type": content_type}, method="PUT")
with urllib.request.urlopen(r, timeout=600) as resp:
return resp.status
def poll_until(job_id, statuses, timeout, label):
deadline = time.time() + timeout
last = None
while time.time() < deadline:
st, job = req("GET", f"/api/jobs/{job_id}")
s = job.get("status")
if s != last:
log(f" [{time.strftime('%H:%M:%S')}] status={s} progress={job.get('progress')}")
last = s
if s in statuses:
return job
if s in ("failed", "error"):
msg = job.get("error_message") or ""
# The shared GCE encoder occasionally drops a job mid-encode (VM restart /
# idle-shutdown race) — a pre-existing, non-tenant-specific transient. Surface
# it distinctly so the caller retries the whole job rather than false-alarming.
if "lost contact with worker" in msg or "not found" in msg:
raise TransientEncoderError(f"Transient encoder failure during {label}: {msg}")
raise RuntimeError(f"Job failed during {label}: {msg}")
time.sleep(15)
raise TimeoutError(f"Timed out waiting for {label} (last status={last})")
def run_once(mixed, instrumental):
"""Submit one job, drive it to completion, assert outputs. Cleans up its own job.
Raises TransientEncoderError on a known transient encoder failure (caller may retry)."""
files = [
{"file_type": "audio", "filename": "mixed.mp3", "content_type": "audio/mpeg"},
{"file_type": "existing_instrumental", "filename": "instrumental.mp3", "content_type": "audio/mpeg"},
]
st, resp = req("POST", "/api/jobs/create-with-upload-urls",
{"artist": ARTIST, "title": TITLE, "files": files,
"is_private": True, "existing_instrumental": True})
if st != 200:
raise RuntimeError(f"create failed: {st} {resp}")
job_id = resp["job_id"]
log(f"job_id: {job_id}")
try:
urls = {u["file_type"]: u for u in resp["upload_urls"]}
put_file(urls["audio"]["upload_url"], mixed, urls["audio"]["content_type"])
put_file(urls["existing_instrumental"]["upload_url"], instrumental, urls["existing_instrumental"]["content_type"])
req("POST", f"/api/jobs/{job_id}/uploads-complete", {"uploaded_files": ["audio", "existing_instrumental"]})
poll_until(job_id, ("awaiting_review", "in_review"), REVIEW_TIMEOUT, "review gate")
log("approving review")
st, _ = req("POST", f"/api/jobs/{job_id}/complete-review", {})
if st != 200:
raise RuntimeError(f"complete-review failed: {st}")
job = poll_until(job_id, ("complete",), RENDER_TIMEOUT, "render")
# Assertions: tenant distribution applied end-to-end.
sd = job.get("state_data") or {}
dropbox = sd.get("dropbox_link") or job.get("dropbox_link")
youtube = sd.get("youtube_url") or job.get("youtube_url")
# Portal downloads are driven by job.file_urls (finals/packages), the same field
# the OutputLinks UI reads — not download_urls/output_files.
file_urls = job.get("file_urls") or {}
finals = file_urls.get("finals") or {}
packages = file_urls.get("packages") or {}
has_downloads = bool(
finals.get("lossy_4k_mp4") or finals.get("lossy_720p_mp4")
or finals.get("with_vocals_mp4") or packages.get("cdg_zip") or packages.get("txt_zip")
)
log(f" dropbox_link: {dropbox}")
log(f" youtube_url: {youtube}")
log(f" finals: {sorted(finals.keys())} | packages: {sorted(packages.keys())}")
errors = []
if not dropbox:
errors.append("no dropbox_link (tenant Dropbox distribution did not run)")
if youtube:
errors.append(f"unexpected youtube_url={youtube} (tenant must not publish to YouTube)")
if not has_downloads:
errors.append("no downloadable outputs in file_urls (finals/packages empty)")
if errors:
raise RuntimeError("Assertions failed: " + "; ".join(errors))
log(f"\n✅ {TENANT} E2E PASSED — job completed, Dropbox link present, no YouTube, downloads available")
finally:
# Always clean up the test job and its uploads.
try:
req("DELETE", f"/api/jobs/{job_id}")
gcs_delete_prefix(f"uploads/{job_id}/")
gcs_delete_prefix(f"jobs/{job_id}/")
log(f"cleaned up job {job_id}")
except Exception as e:
log(f"cleanup warning: {e}")
def main():
log(f"=== Tenant E2E health check: {TENANT} ===")
tmp = tempfile.mkdtemp()
mixed = os.path.join(tmp, "mixed.mp3")
instrumental = os.path.join(tmp, "instrumental.mp3")
gcs_download(TEST_MIXED, mixed)
gcs_download(TEST_INSTRUMENTAL, instrumental)
# Retry once on a known transient encoder failure (shared GCE worker dropping a job
# mid-encode) so the daily health check doesn't false-alarm on pre-existing flakiness.
attempts = 2
for attempt in range(1, attempts + 1):
try:
run_once(mixed, instrumental)
return
except TransientEncoderError as e:
log(f"⚠️ attempt {attempt}/{attempts}: {e}")
if attempt == attempts:
log("Transient encoder failure persisted across retries.")
sys.exit(1)
log("Retrying with a fresh job...")
if __name__ == "__main__":
main()
+49
View File
@@ -0,0 +1,49 @@
#!/usr/bin/env python3
"""Idempotently add the E2E email domain to a tenant's allowed_email_domains in
the live GCS config (tenants/{id}/config.json).
The setup-{tenant}-tenant.py scripts are the source of truth, but the live GCS
config must also be updated for the change to take effect (tenant config is
loaded from GCS, not re-derived from the setup scripts).
Usage: python scripts/e2e/update_tenant_allowlist.py vocalstar singa
Requires a write-capable credential (Application Default Credentials of an
account with storage.objects.create/update on the bucket). If ADC is pinned to
a read-only service account via GOOGLE_APPLICATION_CREDENTIALS, run with that
override cleared, e.g.:
env -u GOOGLE_APPLICATION_CREDENTIALS python scripts/e2e/update_tenant_allowlist.py vocalstar singa
"""
import json
import sys
from google.cloud import storage
BUCKET = "karaoke-gen-storage-nomadkaraoke"
E2E_DOMAIN = "inbox.testmail.app"
def update(tenant_id: str) -> None:
client = storage.Client(project="nomadkaraoke")
blob = client.bucket(BUCKET).blob(f"tenants/{tenant_id}/config.json")
cfg = json.loads(blob.download_as_text()) # populates blob.generation
auth = cfg.setdefault("auth", {})
domains = auth.setdefault("allowed_email_domains", [])
if E2E_DOMAIN in domains:
print(f"{tenant_id}: already allowlisted ({domains})")
return
domains.append(E2E_DOMAIN)
# Optimistic concurrency: only write if the object hasn't changed since read,
# so a parallel edit to the tenant config isn't silently clobbered.
blob.upload_from_string(
json.dumps(cfg, indent=2),
content_type="application/json",
if_generation_match=blob.generation,
)
print(f"{tenant_id}: added {E2E_DOMAIN} -> {domains}")
if __name__ == "__main__":
for t in sys.argv[1:] or ["vocalstar", "singa"]:
update(t)
+1 -1
View File
@@ -586,7 +586,7 @@ def create_tenant_config() -> dict:
"youtube_description_template": None,
},
"auth": {
"allowed_email_domains": ["singa.com", "nomadkaraoke.com"],
"allowed_email_domains": ["singa.com", "nomadkaraoke.com", "inbox.testmail.app"],
"require_email_domain": True,
"fixed_token_ids": [],
"sender_email": "singa@nomadkaraoke.com",
+1 -1
View File
@@ -267,7 +267,7 @@ def create_tenant_config() -> dict:
"youtube_description_template": None
},
"auth": {
"allowed_email_domains": ["vocal-star.com", "vocalstarmusic.com"],
"allowed_email_domains": ["vocal-star.com", "vocalstarmusic.com", "inbox.testmail.app"],
"require_email_domain": True,
"fixed_token_ids": [], # Will be populated when tokens are created
"sender_email": "vocalstar@nomadkaraoke.com"