Files
karaoke-gen/backend/tests/test_internal_api.py
T
Andrew Beveridge ☄️andClaude Opus 5.5 4e43449559 fix(backend): stop API event-loop freezes + identify users in degradation telemetry (v0.264.0) (#1120)
## Summary
The "Reconnecting" pill and the "servers unavailable" banner (164 / 49
events, 2026-09-19 → 10-02) were mostly caused by the single API
instance's **event loop freezing for 20–45s**. Every request on the
instance, `/api/health` included, completed at the same moment. The full
evidence and plan are in
`docs/archive/2026-10-03-backend-loop-freezes-telemetry-plan.md`.

### What froze the loop, and the fix for each
- **Screens worker:** it ran as a post-response `async def`
BackgroundTask doing 4K PIL renders, GCS transfers and ffmpeg transcodes
on the loop. Under `--cpu-throttling` it was also CPU-starved. **Fix:**
it now runs inline in its Cloud Tasks request (CPU stays allocated,
failures still return 200 so Cloud Tasks doesn't double-run it), with
the heavy steps in `asyncio.to_thread`. Also offloaded: style load,
timing signals, pre-apply.
- **Matplotlib font cache:** it was rebuilt on every new instance (17/28
episodes). **Fix:** it's pre-built in `Dockerfile.base`
(`MPLCONFIGDIR=/opt/mplconfig`), and pyplot is imported lazily.
- **Hourly stale-review cron:** an `async def` with zero awaits doing
≤1500 reads and SMTP sends. **Fix:** it runs in a thread.
- **166 `async def` routes that never awaited** (sync
Firestore/GCS/email/Gemini on the loop, e.g. magic-link verify).
**Fix:** they're now plain `def`, so they run in FastAPI's threadpool.
The new AST guard `test_no_blocking_async_routes.py` fails CI on new
ones. It also flags sync routes that call
`asyncio.create_task`/`get_running_loop`; that caught `confirm_duration`
and `trigger_youtube_queue_processing`, which stay async.

### Safety net
- `backend/services/loop_watchdog.py`: a heartbeat plus a daemon thread.
Any stall over 1s logs `EVENT_LOOP_STALL` with the loop thread's stack
and a `culprit=file:line`. Stalls of 5s or more are written to
`client_events` as `server_loop_stall`.
- Pulumi: the log metric `backend/event_loop_stalls` plus an alert on ≥2
stalls of ≥10s in 30 min. **This still needs a local `pulumi up`.**
Pulumi is never applied by CI, and the admin gcloud login has expired.

### Telemetry gaps fixed
- **Identity:** `user_email` used to be hard-coded to null. The frontend
now sends its session token, and the server resolves `user_email`,
`is_admin`, `is_internal` and `is_test`. A client-sent email is ignored.
- **New fields:** `device_fingerprint` (the existing FingerprintJS
visitorId), `tab_id`, `episode_id`, and `tenant`.
- **New event:** `banner_recovered`, carrying `duration_ms` and
`peak_status`.
- **Report:** `scripts/client_events_report.py` summarises the data,
excluding admin/internal/test accounts by default.

## Testing
- `make test` passes: backend 5529, plus the emulator suite, karaoke_gen
unit tests, and frontend Jest 1623.
- New tests:
- Watchdog: a deterministic state machine plus detection of a real
blocking call.
- Event-loop responsiveness: a ticker keeps ticking while the heavy
steps run.
  - Screens inline and failure semantics.
  - `client_events` identity and fields.
  - The AST guard.
  - The report summary.
  - Frontend reporter and banner episode tests.
- Local CodeRabbit CLI review: 0 findings.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

@coderabbitai ignore

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 14:17:51 -04:00

453 lines
18 KiB
Python

"""
Unit tests for internal.py API routes using FastAPI TestClient.
Internal routes are used for worker communication within Cloud Run.
These endpoints require admin authentication.
With Cloud Tasks integration, these endpoints include idempotency checks
to prevent duplicate processing on retries.
"""
import os
import pytest
from datetime import datetime, UTC
from unittest.mock import MagicMock, AsyncMock, patch
from fastapi.testclient import TestClient
from backend.models.job import Job, JobStatus
# Set test admin token for auth
os.environ.setdefault('ADMIN_TOKENS', 'test-admin-token')
def _create_mock_job(job_id: str = "test123") -> Job:
"""Create a mock Job object for testing."""
return Job(
job_id=job_id,
status=JobStatus.PROCESSING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
state_data={}, # Empty state_data - no worker progress yet
)
@pytest.fixture
def client():
"""Create TestClient with mocked workers and job manager."""
mock_creds = MagicMock()
mock_creds.universe_domain = 'googleapis.com'
# Mock the worker functions at the internal routes module level
mock_audio = AsyncMock()
mock_lyrics = AsyncMock()
mock_screens = AsyncMock()
mock_video = AsyncMock()
mock_render = AsyncMock()
# Mock JobManager for idempotency checks
mock_job_manager = MagicMock()
mock_job_manager.get_job.return_value = _create_mock_job()
mock_job_manager.update_state_data.return_value = None
with patch('backend.api.routes.internal.process_audio_separation', mock_audio), \
patch('backend.api.routes.internal.process_lyrics_transcription', mock_lyrics), \
patch('backend.api.routes.internal.generate_screens', mock_screens), \
patch('backend.api.routes.internal.generate_video', mock_video), \
patch('backend.api.routes.internal.process_render_video', mock_render), \
patch('backend.api.routes.internal.JobManager', return_value=mock_job_manager), \
patch('backend.services.firestore_service.firestore'), \
patch('backend.services.storage_service.storage'), \
patch('google.auth.default', return_value=(mock_creds, 'test-project')):
from backend.main import app
yield TestClient(app)
@pytest.fixture
def auth_headers():
"""Auth headers for internal API testing."""
return {"Authorization": "Bearer test-admin-token"}
class TestAudioWorkerEndpoint:
"""Tests for POST /api/internal/workers/audio."""
def test_audio_worker_returns_200(self, client, auth_headers):
"""Test audio worker endpoint returns 200 with auth."""
response = client.post(
"/api/internal/workers/audio",
headers=auth_headers,
json={"job_id": "test123"}
)
assert response.status_code == 200
def test_audio_worker_accepts_job_id(self, client, auth_headers):
"""Test audio worker accepts job_id in body."""
response = client.post(
"/api/internal/workers/audio",
headers=auth_headers,
json={"job_id": "any-job-id"}
)
assert response.status_code == 200
def test_audio_worker_requires_job_id(self, client, auth_headers):
"""Test audio worker requires job_id."""
response = client.post(
"/api/internal/workers/audio",
headers=auth_headers,
json={}
)
assert response.status_code == 422
def test_audio_worker_requires_auth(self, client, auth_headers):
"""Test audio worker requires authentication."""
response = client.post(
"/api/internal/workers/audio",
json={"job_id": "test123"}
)
assert response.status_code == 401
class TestLyricsWorkerEndpoint:
"""Tests for POST /api/internal/workers/lyrics."""
def test_lyrics_worker_returns_200(self, client, auth_headers):
"""Test lyrics worker endpoint returns 200 with auth."""
response = client.post(
"/api/internal/workers/lyrics",
headers=auth_headers,
json={"job_id": "test123"}
)
assert response.status_code == 200
def test_lyrics_worker_requires_job_id(self, client, auth_headers):
"""Test lyrics worker requires job_id."""
response = client.post(
"/api/internal/workers/lyrics",
headers=auth_headers,
json={}
)
assert response.status_code == 422
def test_lyrics_worker_requires_auth(self, client, auth_headers):
"""Test lyrics worker requires authentication."""
response = client.post(
"/api/internal/workers/lyrics",
json={"job_id": "test123"}
)
assert response.status_code == 401
class TestScreensWorkerEndpoint:
"""Tests for POST /api/internal/workers/screens."""
def test_screens_worker_returns_200(self, client, auth_headers):
"""Test screens worker endpoint returns 200 with auth."""
response = client.post(
"/api/internal/workers/screens",
headers=auth_headers,
json={"job_id": "test123"}
)
assert response.status_code == 200
def test_screens_worker_requires_job_id(self, client, auth_headers):
"""Test screens worker requires job_id."""
response = client.post(
"/api/internal/workers/screens",
headers=auth_headers,
json={}
)
assert response.status_code == 422
def test_screens_worker_requires_auth(self, client, auth_headers):
"""Test screens worker requires authentication."""
response = client.post(
"/api/internal/workers/screens",
json={"job_id": "test123"}
)
assert response.status_code == 401
class TestScreensWorkerInlineInCloudTasksMode:
"""Prod (Cloud Tasks) runs screens INLINE so CPU stays allocated and the
heavy steps run in worker threads instead of a post-response BackgroundTask."""
def _settings(self, enable_cloud_tasks):
s = MagicMock()
s.enable_cloud_tasks = enable_cloud_tasks
return s
def test_cloud_tasks_mode_awaits_worker_inline(self, client, auth_headers):
import backend.api.routes.internal as internal_mod
with patch('backend.config.get_settings', return_value=self._settings(True)):
internal_mod.generate_screens.reset_mock()
internal_mod.generate_screens.return_value = True
response = client.post(
"/api/internal/workers/screens", headers=auth_headers, json={"job_id": "test123"}
)
assert response.status_code == 200
assert response.json()["status"] == "completed"
internal_mod.generate_screens.assert_awaited_once_with("test123")
def test_cloud_tasks_mode_worker_failure_still_200(self, client, auth_headers):
"""A handled failure must not return non-2xx (Cloud Tasks would re-run it)."""
import backend.api.routes.internal as internal_mod
with patch('backend.config.get_settings', return_value=self._settings(True)):
internal_mod.generate_screens.return_value = False
response = client.post(
"/api/internal/workers/screens", headers=auth_headers, json={"job_id": "test123"}
)
assert response.status_code == 200
assert response.json()["status"] == "failed"
def test_cloud_tasks_mode_worker_exception_still_200(self, client, auth_headers):
import backend.api.routes.internal as internal_mod
with patch('backend.config.get_settings', return_value=self._settings(True)):
internal_mod.generate_screens.side_effect = RuntimeError("boom")
try:
response = client.post(
"/api/internal/workers/screens", headers=auth_headers, json={"job_id": "test123"}
)
finally:
internal_mod.generate_screens.side_effect = None
assert response.status_code == 200
assert response.json()["status"] == "failed"
def test_direct_http_mode_keeps_background_task(self, client, auth_headers):
with patch('backend.config.get_settings', return_value=self._settings(False)):
response = client.post(
"/api/internal/workers/screens", headers=auth_headers, json={"job_id": "test123"}
)
assert response.status_code == 200
assert response.json()["status"] == "started"
class TestVideoWorkerEndpoint:
"""Tests for POST /api/internal/workers/video."""
def test_video_worker_returns_200(self, client, auth_headers):
"""Test video worker endpoint returns 200 with auth."""
response = client.post(
"/api/internal/workers/video",
headers=auth_headers,
json={"job_id": "test123"}
)
assert response.status_code == 200
def test_video_worker_requires_job_id(self, client, auth_headers):
"""Test video worker requires job_id."""
response = client.post(
"/api/internal/workers/video",
headers=auth_headers,
json={}
)
assert response.status_code == 422
def test_video_worker_requires_auth(self, client, auth_headers):
"""Test video worker requires authentication."""
response = client.post(
"/api/internal/workers/video",
json={"job_id": "test123"}
)
assert response.status_code == 401
class TestRenderVideoWorkerEndpoint:
"""Tests for POST /api/internal/workers/render-video."""
def test_render_video_worker_returns_200(self, client, auth_headers):
"""Test render-video worker endpoint returns 200 with auth."""
response = client.post(
"/api/internal/workers/render-video",
headers=auth_headers,
json={"job_id": "test123"}
)
assert response.status_code == 200
def test_render_video_worker_requires_job_id(self, client, auth_headers):
"""Test render-video worker requires job_id."""
response = client.post(
"/api/internal/workers/render-video",
headers=auth_headers,
json={}
)
assert response.status_code == 422
def test_render_video_worker_requires_auth(self, client, auth_headers):
"""Test render-video worker requires authentication."""
response = client.post(
"/api/internal/workers/render-video",
json={"job_id": "test123"}
)
assert response.status_code == 401
class TestWorkerResponseFormat:
"""Tests for worker response format."""
def test_audio_worker_response_contains_status(self, client, auth_headers):
"""Test audio worker response contains status."""
response = client.post(
"/api/internal/workers/audio",
headers=auth_headers,
json={"job_id": "test123"}
)
data = response.json()
assert "status" in data or "message" in data
def test_lyrics_worker_response_contains_status(self, client, auth_headers):
"""Test lyrics worker response contains status."""
response = client.post(
"/api/internal/workers/lyrics",
headers=auth_headers,
json={"job_id": "test123"}
)
data = response.json()
assert "status" in data or "message" in data
class TestWorkerIdempotency:
"""Tests for worker idempotency checks (Cloud Tasks retry support)."""
def test_audio_worker_returns_already_running_when_in_progress(self, auth_headers):
"""Test audio worker returns already_running if worker is already running."""
mock_creds = MagicMock()
mock_creds.universe_domain = 'googleapis.com'
# Create job with audio already running
mock_job = _create_mock_job()
mock_job.state_data = {'audio_progress': {'stage': 'running'}}
mock_job_manager = MagicMock()
mock_job_manager.get_job.return_value = mock_job
with patch('backend.api.routes.internal.process_audio_separation', AsyncMock()), \
patch('backend.api.routes.internal.JobManager', return_value=mock_job_manager), \
patch('backend.services.firestore_service.firestore'), \
patch('backend.services.storage_service.storage'), \
patch('google.auth.default', return_value=(mock_creds, 'test-project')):
from backend.main import app
client = TestClient(app)
response = client.post(
"/api/internal/workers/audio",
headers=auth_headers,
json={"job_id": "test123"}
)
assert response.status_code == 200
data = response.json()
assert data["status"] == "already_running"
def test_audio_worker_returns_already_complete_when_finished(self, auth_headers):
"""Test audio worker returns already_complete if worker already finished."""
mock_creds = MagicMock()
mock_creds.universe_domain = 'googleapis.com'
# Create job with audio already complete
mock_job = _create_mock_job()
mock_job.state_data = {'audio_progress': {'stage': 'complete'}}
mock_job_manager = MagicMock()
mock_job_manager.get_job.return_value = mock_job
with patch('backend.api.routes.internal.process_audio_separation', AsyncMock()), \
patch('backend.api.routes.internal.JobManager', return_value=mock_job_manager), \
patch('backend.services.firestore_service.firestore'), \
patch('backend.services.storage_service.storage'), \
patch('google.auth.default', return_value=(mock_creds, 'test-project')):
from backend.main import app
client = TestClient(app)
response = client.post(
"/api/internal/workers/audio",
headers=auth_headers,
json={"job_id": "test123"}
)
assert response.status_code == 200
data = response.json()
assert data["status"] == "already_complete"
def test_audio_worker_returns_not_found_when_job_missing(self, auth_headers):
"""Test audio worker returns not_found if job doesn't exist."""
mock_creds = MagicMock()
mock_creds.universe_domain = 'googleapis.com'
# Return None for job not found
mock_job_manager = MagicMock()
mock_job_manager.get_job.return_value = None
with patch('backend.api.routes.internal.process_audio_separation', AsyncMock()), \
patch('backend.api.routes.internal.JobManager', return_value=mock_job_manager), \
patch('backend.services.firestore_service.firestore'), \
patch('backend.services.storage_service.storage'), \
patch('google.auth.default', return_value=(mock_creds, 'test-project')):
from backend.main import app
client = TestClient(app)
response = client.post(
"/api/internal/workers/audio",
headers=auth_headers,
json={"job_id": "nonexistent"}
)
assert response.status_code == 200
data = response.json()
assert data["status"] == "not_found"
class TestAutoApprovalEval:
"""POST /api/internal/jobs/{job_id}/auto-approval-eval — admin re-evaluation."""
def test_eval_calls_executor_and_returns_outcome(self, client):
with patch(
'backend.services.auto_approval.executor.maybe_auto_complete_review',
new=AsyncMock(return_value={"outcome": "auto_completed"}),
) as mock_eval:
response = client.post(
"/api/internal/jobs/test123/auto-approval-eval",
headers={"Authorization": "Bearer test-admin-token"},
)
assert response.status_code == 200
data = response.json()
assert data["job_id"] == "test123"
assert data["outcome"] == "auto_completed"
mock_eval.assert_awaited_once_with("test123", trigger="admin_eval")
def test_eval_requires_admin(self, client):
response = client.post("/api/internal/jobs/test123/auto-approval-eval")
assert response.status_code == 401
class TestVocalGapsEndpoint:
"""POST /api/internal/jobs/{job_id}/vocal-gaps — missing-lyrics backfill/inspect."""
def test_stores_by_default(self, client):
with patch(
'backend.services.auto_approval.executor.analyze_job_vocal_gaps',
return_value={"status": "checked", "stored": True},
) as analyze:
response = client.post(
"/api/internal/jobs/test123/vocal-gaps",
headers={"Authorization": "Bearer test-admin-token"},
)
assert response.status_code == 200
assert response.json() == {"job_id": "test123", "status": "checked", "stored": True}
analyze.assert_called_once_with("test123", True)
def test_dry_run_does_not_store(self, client):
with patch(
'backend.services.auto_approval.executor.analyze_job_vocal_gaps',
return_value={"status": "checked", "stored": False},
) as analyze:
client.post(
"/api/internal/jobs/test123/vocal-gaps?dry_run=true",
headers={"Authorization": "Bearer test-admin-token"},
)
analyze.assert_called_once_with("test123", False)
def test_requires_admin(self, client):
response = client.post("/api/internal/jobs/test123/vocal-gaps")
assert response.status_code == 401