mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary The "Reconnecting" pill and the "servers unavailable" banner (164 / 49 events, 2026-09-19 → 10-02) were mostly caused by the single API instance's **event loop freezing for 20–45s**. Every request on the instance, `/api/health` included, completed at the same moment. The full evidence and plan are in `docs/archive/2026-10-03-backend-loop-freezes-telemetry-plan.md`. ### What froze the loop, and the fix for each - **Screens worker:** it ran as a post-response `async def` BackgroundTask doing 4K PIL renders, GCS transfers and ffmpeg transcodes on the loop. Under `--cpu-throttling` it was also CPU-starved. **Fix:** it now runs inline in its Cloud Tasks request (CPU stays allocated, failures still return 200 so Cloud Tasks doesn't double-run it), with the heavy steps in `asyncio.to_thread`. Also offloaded: style load, timing signals, pre-apply. - **Matplotlib font cache:** it was rebuilt on every new instance (17/28 episodes). **Fix:** it's pre-built in `Dockerfile.base` (`MPLCONFIGDIR=/opt/mplconfig`), and pyplot is imported lazily. - **Hourly stale-review cron:** an `async def` with zero awaits doing ≤1500 reads and SMTP sends. **Fix:** it runs in a thread. - **166 `async def` routes that never awaited** (sync Firestore/GCS/email/Gemini on the loop, e.g. magic-link verify). **Fix:** they're now plain `def`, so they run in FastAPI's threadpool. The new AST guard `test_no_blocking_async_routes.py` fails CI on new ones. It also flags sync routes that call `asyncio.create_task`/`get_running_loop`; that caught `confirm_duration` and `trigger_youtube_queue_processing`, which stay async. ### Safety net - `backend/services/loop_watchdog.py`: a heartbeat plus a daemon thread. Any stall over 1s logs `EVENT_LOOP_STALL` with the loop thread's stack and a `culprit=file:line`. Stalls of 5s or more are written to `client_events` as `server_loop_stall`. - Pulumi: the log metric `backend/event_loop_stalls` plus an alert on ≥2 stalls of ≥10s in 30 min. **This still needs a local `pulumi up`.** Pulumi is never applied by CI, and the admin gcloud login has expired. ### Telemetry gaps fixed - **Identity:** `user_email` used to be hard-coded to null. The frontend now sends its session token, and the server resolves `user_email`, `is_admin`, `is_internal` and `is_test`. A client-sent email is ignored. - **New fields:** `device_fingerprint` (the existing FingerprintJS visitorId), `tab_id`, `episode_id`, and `tenant`. - **New event:** `banner_recovered`, carrying `duration_ms` and `peak_status`. - **Report:** `scripts/client_events_report.py` summarises the data, excluding admin/internal/test accounts by default. ## Testing - `make test` passes: backend 5529, plus the emulator suite, karaoke_gen unit tests, and frontend Jest 1623. - New tests: - Watchdog: a deterministic state machine plus detection of a real blocking call. - Event-loop responsiveness: a ticker keeps ticking while the heavy steps run. - Screens inline and failure semantics. - `client_events` identity and fields. - The AST guard. - The report summary. - Frontend reporter and banner episode tests. - Local CodeRabbit CLI review: 0 findings. 🤖 Generated with [Claude Code](https://claude.com/claude-code) @coderabbitai ignore --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
453 lines
18 KiB
Python
453 lines
18 KiB
Python
"""
|
|
Unit tests for internal.py API routes using FastAPI TestClient.
|
|
|
|
Internal routes are used for worker communication within Cloud Run.
|
|
These endpoints require admin authentication.
|
|
|
|
With Cloud Tasks integration, these endpoints include idempotency checks
|
|
to prevent duplicate processing on retries.
|
|
"""
|
|
import os
|
|
import pytest
|
|
from datetime import datetime, UTC
|
|
from unittest.mock import MagicMock, AsyncMock, patch
|
|
from fastapi.testclient import TestClient
|
|
|
|
from backend.models.job import Job, JobStatus
|
|
|
|
|
|
# Set test admin token for auth
|
|
os.environ.setdefault('ADMIN_TOKENS', 'test-admin-token')
|
|
|
|
|
|
def _create_mock_job(job_id: str = "test123") -> Job:
|
|
"""Create a mock Job object for testing."""
|
|
return Job(
|
|
job_id=job_id,
|
|
status=JobStatus.PROCESSING,
|
|
created_at=datetime.now(UTC),
|
|
updated_at=datetime.now(UTC),
|
|
state_data={}, # Empty state_data - no worker progress yet
|
|
)
|
|
|
|
|
|
@pytest.fixture
|
|
def client():
|
|
"""Create TestClient with mocked workers and job manager."""
|
|
mock_creds = MagicMock()
|
|
mock_creds.universe_domain = 'googleapis.com'
|
|
|
|
# Mock the worker functions at the internal routes module level
|
|
mock_audio = AsyncMock()
|
|
mock_lyrics = AsyncMock()
|
|
mock_screens = AsyncMock()
|
|
mock_video = AsyncMock()
|
|
mock_render = AsyncMock()
|
|
|
|
# Mock JobManager for idempotency checks
|
|
mock_job_manager = MagicMock()
|
|
mock_job_manager.get_job.return_value = _create_mock_job()
|
|
mock_job_manager.update_state_data.return_value = None
|
|
|
|
with patch('backend.api.routes.internal.process_audio_separation', mock_audio), \
|
|
patch('backend.api.routes.internal.process_lyrics_transcription', mock_lyrics), \
|
|
patch('backend.api.routes.internal.generate_screens', mock_screens), \
|
|
patch('backend.api.routes.internal.generate_video', mock_video), \
|
|
patch('backend.api.routes.internal.process_render_video', mock_render), \
|
|
patch('backend.api.routes.internal.JobManager', return_value=mock_job_manager), \
|
|
patch('backend.services.firestore_service.firestore'), \
|
|
patch('backend.services.storage_service.storage'), \
|
|
patch('google.auth.default', return_value=(mock_creds, 'test-project')):
|
|
from backend.main import app
|
|
yield TestClient(app)
|
|
|
|
|
|
@pytest.fixture
|
|
def auth_headers():
|
|
"""Auth headers for internal API testing."""
|
|
return {"Authorization": "Bearer test-admin-token"}
|
|
|
|
|
|
class TestAudioWorkerEndpoint:
|
|
"""Tests for POST /api/internal/workers/audio."""
|
|
|
|
def test_audio_worker_returns_200(self, client, auth_headers):
|
|
"""Test audio worker endpoint returns 200 with auth."""
|
|
response = client.post(
|
|
"/api/internal/workers/audio",
|
|
headers=auth_headers,
|
|
json={"job_id": "test123"}
|
|
)
|
|
assert response.status_code == 200
|
|
|
|
def test_audio_worker_accepts_job_id(self, client, auth_headers):
|
|
"""Test audio worker accepts job_id in body."""
|
|
response = client.post(
|
|
"/api/internal/workers/audio",
|
|
headers=auth_headers,
|
|
json={"job_id": "any-job-id"}
|
|
)
|
|
assert response.status_code == 200
|
|
|
|
def test_audio_worker_requires_job_id(self, client, auth_headers):
|
|
"""Test audio worker requires job_id."""
|
|
response = client.post(
|
|
"/api/internal/workers/audio",
|
|
headers=auth_headers,
|
|
json={}
|
|
)
|
|
assert response.status_code == 422
|
|
|
|
def test_audio_worker_requires_auth(self, client, auth_headers):
|
|
"""Test audio worker requires authentication."""
|
|
response = client.post(
|
|
"/api/internal/workers/audio",
|
|
json={"job_id": "test123"}
|
|
)
|
|
assert response.status_code == 401
|
|
|
|
|
|
class TestLyricsWorkerEndpoint:
|
|
"""Tests for POST /api/internal/workers/lyrics."""
|
|
|
|
def test_lyrics_worker_returns_200(self, client, auth_headers):
|
|
"""Test lyrics worker endpoint returns 200 with auth."""
|
|
response = client.post(
|
|
"/api/internal/workers/lyrics",
|
|
headers=auth_headers,
|
|
json={"job_id": "test123"}
|
|
)
|
|
assert response.status_code == 200
|
|
|
|
def test_lyrics_worker_requires_job_id(self, client, auth_headers):
|
|
"""Test lyrics worker requires job_id."""
|
|
response = client.post(
|
|
"/api/internal/workers/lyrics",
|
|
headers=auth_headers,
|
|
json={}
|
|
)
|
|
assert response.status_code == 422
|
|
|
|
def test_lyrics_worker_requires_auth(self, client, auth_headers):
|
|
"""Test lyrics worker requires authentication."""
|
|
response = client.post(
|
|
"/api/internal/workers/lyrics",
|
|
json={"job_id": "test123"}
|
|
)
|
|
assert response.status_code == 401
|
|
|
|
|
|
class TestScreensWorkerEndpoint:
|
|
"""Tests for POST /api/internal/workers/screens."""
|
|
|
|
def test_screens_worker_returns_200(self, client, auth_headers):
|
|
"""Test screens worker endpoint returns 200 with auth."""
|
|
response = client.post(
|
|
"/api/internal/workers/screens",
|
|
headers=auth_headers,
|
|
json={"job_id": "test123"}
|
|
)
|
|
assert response.status_code == 200
|
|
|
|
def test_screens_worker_requires_job_id(self, client, auth_headers):
|
|
"""Test screens worker requires job_id."""
|
|
response = client.post(
|
|
"/api/internal/workers/screens",
|
|
headers=auth_headers,
|
|
json={}
|
|
)
|
|
assert response.status_code == 422
|
|
|
|
def test_screens_worker_requires_auth(self, client, auth_headers):
|
|
"""Test screens worker requires authentication."""
|
|
response = client.post(
|
|
"/api/internal/workers/screens",
|
|
json={"job_id": "test123"}
|
|
)
|
|
assert response.status_code == 401
|
|
|
|
|
|
class TestScreensWorkerInlineInCloudTasksMode:
|
|
"""Prod (Cloud Tasks) runs screens INLINE so CPU stays allocated and the
|
|
heavy steps run in worker threads instead of a post-response BackgroundTask."""
|
|
|
|
def _settings(self, enable_cloud_tasks):
|
|
s = MagicMock()
|
|
s.enable_cloud_tasks = enable_cloud_tasks
|
|
return s
|
|
|
|
def test_cloud_tasks_mode_awaits_worker_inline(self, client, auth_headers):
|
|
import backend.api.routes.internal as internal_mod
|
|
with patch('backend.config.get_settings', return_value=self._settings(True)):
|
|
internal_mod.generate_screens.reset_mock()
|
|
internal_mod.generate_screens.return_value = True
|
|
response = client.post(
|
|
"/api/internal/workers/screens", headers=auth_headers, json={"job_id": "test123"}
|
|
)
|
|
assert response.status_code == 200
|
|
assert response.json()["status"] == "completed"
|
|
internal_mod.generate_screens.assert_awaited_once_with("test123")
|
|
|
|
def test_cloud_tasks_mode_worker_failure_still_200(self, client, auth_headers):
|
|
"""A handled failure must not return non-2xx (Cloud Tasks would re-run it)."""
|
|
import backend.api.routes.internal as internal_mod
|
|
with patch('backend.config.get_settings', return_value=self._settings(True)):
|
|
internal_mod.generate_screens.return_value = False
|
|
response = client.post(
|
|
"/api/internal/workers/screens", headers=auth_headers, json={"job_id": "test123"}
|
|
)
|
|
assert response.status_code == 200
|
|
assert response.json()["status"] == "failed"
|
|
|
|
def test_cloud_tasks_mode_worker_exception_still_200(self, client, auth_headers):
|
|
import backend.api.routes.internal as internal_mod
|
|
with patch('backend.config.get_settings', return_value=self._settings(True)):
|
|
internal_mod.generate_screens.side_effect = RuntimeError("boom")
|
|
try:
|
|
response = client.post(
|
|
"/api/internal/workers/screens", headers=auth_headers, json={"job_id": "test123"}
|
|
)
|
|
finally:
|
|
internal_mod.generate_screens.side_effect = None
|
|
assert response.status_code == 200
|
|
assert response.json()["status"] == "failed"
|
|
|
|
def test_direct_http_mode_keeps_background_task(self, client, auth_headers):
|
|
with patch('backend.config.get_settings', return_value=self._settings(False)):
|
|
response = client.post(
|
|
"/api/internal/workers/screens", headers=auth_headers, json={"job_id": "test123"}
|
|
)
|
|
assert response.status_code == 200
|
|
assert response.json()["status"] == "started"
|
|
|
|
|
|
class TestVideoWorkerEndpoint:
|
|
"""Tests for POST /api/internal/workers/video."""
|
|
|
|
def test_video_worker_returns_200(self, client, auth_headers):
|
|
"""Test video worker endpoint returns 200 with auth."""
|
|
response = client.post(
|
|
"/api/internal/workers/video",
|
|
headers=auth_headers,
|
|
json={"job_id": "test123"}
|
|
)
|
|
assert response.status_code == 200
|
|
|
|
def test_video_worker_requires_job_id(self, client, auth_headers):
|
|
"""Test video worker requires job_id."""
|
|
response = client.post(
|
|
"/api/internal/workers/video",
|
|
headers=auth_headers,
|
|
json={}
|
|
)
|
|
assert response.status_code == 422
|
|
|
|
def test_video_worker_requires_auth(self, client, auth_headers):
|
|
"""Test video worker requires authentication."""
|
|
response = client.post(
|
|
"/api/internal/workers/video",
|
|
json={"job_id": "test123"}
|
|
)
|
|
assert response.status_code == 401
|
|
|
|
|
|
class TestRenderVideoWorkerEndpoint:
|
|
"""Tests for POST /api/internal/workers/render-video."""
|
|
|
|
def test_render_video_worker_returns_200(self, client, auth_headers):
|
|
"""Test render-video worker endpoint returns 200 with auth."""
|
|
response = client.post(
|
|
"/api/internal/workers/render-video",
|
|
headers=auth_headers,
|
|
json={"job_id": "test123"}
|
|
)
|
|
assert response.status_code == 200
|
|
|
|
def test_render_video_worker_requires_job_id(self, client, auth_headers):
|
|
"""Test render-video worker requires job_id."""
|
|
response = client.post(
|
|
"/api/internal/workers/render-video",
|
|
headers=auth_headers,
|
|
json={}
|
|
)
|
|
assert response.status_code == 422
|
|
|
|
def test_render_video_worker_requires_auth(self, client, auth_headers):
|
|
"""Test render-video worker requires authentication."""
|
|
response = client.post(
|
|
"/api/internal/workers/render-video",
|
|
json={"job_id": "test123"}
|
|
)
|
|
assert response.status_code == 401
|
|
|
|
|
|
class TestWorkerResponseFormat:
|
|
"""Tests for worker response format."""
|
|
|
|
def test_audio_worker_response_contains_status(self, client, auth_headers):
|
|
"""Test audio worker response contains status."""
|
|
response = client.post(
|
|
"/api/internal/workers/audio",
|
|
headers=auth_headers,
|
|
json={"job_id": "test123"}
|
|
)
|
|
data = response.json()
|
|
assert "status" in data or "message" in data
|
|
|
|
def test_lyrics_worker_response_contains_status(self, client, auth_headers):
|
|
"""Test lyrics worker response contains status."""
|
|
response = client.post(
|
|
"/api/internal/workers/lyrics",
|
|
headers=auth_headers,
|
|
json={"job_id": "test123"}
|
|
)
|
|
data = response.json()
|
|
assert "status" in data or "message" in data
|
|
|
|
|
|
class TestWorkerIdempotency:
|
|
"""Tests for worker idempotency checks (Cloud Tasks retry support)."""
|
|
|
|
def test_audio_worker_returns_already_running_when_in_progress(self, auth_headers):
|
|
"""Test audio worker returns already_running if worker is already running."""
|
|
mock_creds = MagicMock()
|
|
mock_creds.universe_domain = 'googleapis.com'
|
|
|
|
# Create job with audio already running
|
|
mock_job = _create_mock_job()
|
|
mock_job.state_data = {'audio_progress': {'stage': 'running'}}
|
|
|
|
mock_job_manager = MagicMock()
|
|
mock_job_manager.get_job.return_value = mock_job
|
|
|
|
with patch('backend.api.routes.internal.process_audio_separation', AsyncMock()), \
|
|
patch('backend.api.routes.internal.JobManager', return_value=mock_job_manager), \
|
|
patch('backend.services.firestore_service.firestore'), \
|
|
patch('backend.services.storage_service.storage'), \
|
|
patch('google.auth.default', return_value=(mock_creds, 'test-project')):
|
|
from backend.main import app
|
|
client = TestClient(app)
|
|
|
|
response = client.post(
|
|
"/api/internal/workers/audio",
|
|
headers=auth_headers,
|
|
json={"job_id": "test123"}
|
|
)
|
|
|
|
assert response.status_code == 200
|
|
data = response.json()
|
|
assert data["status"] == "already_running"
|
|
|
|
def test_audio_worker_returns_already_complete_when_finished(self, auth_headers):
|
|
"""Test audio worker returns already_complete if worker already finished."""
|
|
mock_creds = MagicMock()
|
|
mock_creds.universe_domain = 'googleapis.com'
|
|
|
|
# Create job with audio already complete
|
|
mock_job = _create_mock_job()
|
|
mock_job.state_data = {'audio_progress': {'stage': 'complete'}}
|
|
|
|
mock_job_manager = MagicMock()
|
|
mock_job_manager.get_job.return_value = mock_job
|
|
|
|
with patch('backend.api.routes.internal.process_audio_separation', AsyncMock()), \
|
|
patch('backend.api.routes.internal.JobManager', return_value=mock_job_manager), \
|
|
patch('backend.services.firestore_service.firestore'), \
|
|
patch('backend.services.storage_service.storage'), \
|
|
patch('google.auth.default', return_value=(mock_creds, 'test-project')):
|
|
from backend.main import app
|
|
client = TestClient(app)
|
|
|
|
response = client.post(
|
|
"/api/internal/workers/audio",
|
|
headers=auth_headers,
|
|
json={"job_id": "test123"}
|
|
)
|
|
|
|
assert response.status_code == 200
|
|
data = response.json()
|
|
assert data["status"] == "already_complete"
|
|
|
|
def test_audio_worker_returns_not_found_when_job_missing(self, auth_headers):
|
|
"""Test audio worker returns not_found if job doesn't exist."""
|
|
mock_creds = MagicMock()
|
|
mock_creds.universe_domain = 'googleapis.com'
|
|
|
|
# Return None for job not found
|
|
mock_job_manager = MagicMock()
|
|
mock_job_manager.get_job.return_value = None
|
|
|
|
with patch('backend.api.routes.internal.process_audio_separation', AsyncMock()), \
|
|
patch('backend.api.routes.internal.JobManager', return_value=mock_job_manager), \
|
|
patch('backend.services.firestore_service.firestore'), \
|
|
patch('backend.services.storage_service.storage'), \
|
|
patch('google.auth.default', return_value=(mock_creds, 'test-project')):
|
|
from backend.main import app
|
|
client = TestClient(app)
|
|
|
|
response = client.post(
|
|
"/api/internal/workers/audio",
|
|
headers=auth_headers,
|
|
json={"job_id": "nonexistent"}
|
|
)
|
|
|
|
assert response.status_code == 200
|
|
data = response.json()
|
|
assert data["status"] == "not_found"
|
|
|
|
|
|
|
|
class TestAutoApprovalEval:
|
|
"""POST /api/internal/jobs/{job_id}/auto-approval-eval — admin re-evaluation."""
|
|
|
|
def test_eval_calls_executor_and_returns_outcome(self, client):
|
|
with patch(
|
|
'backend.services.auto_approval.executor.maybe_auto_complete_review',
|
|
new=AsyncMock(return_value={"outcome": "auto_completed"}),
|
|
) as mock_eval:
|
|
response = client.post(
|
|
"/api/internal/jobs/test123/auto-approval-eval",
|
|
headers={"Authorization": "Bearer test-admin-token"},
|
|
)
|
|
assert response.status_code == 200
|
|
data = response.json()
|
|
assert data["job_id"] == "test123"
|
|
assert data["outcome"] == "auto_completed"
|
|
mock_eval.assert_awaited_once_with("test123", trigger="admin_eval")
|
|
|
|
def test_eval_requires_admin(self, client):
|
|
response = client.post("/api/internal/jobs/test123/auto-approval-eval")
|
|
assert response.status_code == 401
|
|
|
|
|
|
class TestVocalGapsEndpoint:
|
|
"""POST /api/internal/jobs/{job_id}/vocal-gaps — missing-lyrics backfill/inspect."""
|
|
|
|
def test_stores_by_default(self, client):
|
|
with patch(
|
|
'backend.services.auto_approval.executor.analyze_job_vocal_gaps',
|
|
return_value={"status": "checked", "stored": True},
|
|
) as analyze:
|
|
response = client.post(
|
|
"/api/internal/jobs/test123/vocal-gaps",
|
|
headers={"Authorization": "Bearer test-admin-token"},
|
|
)
|
|
assert response.status_code == 200
|
|
assert response.json() == {"job_id": "test123", "status": "checked", "stored": True}
|
|
analyze.assert_called_once_with("test123", True)
|
|
|
|
def test_dry_run_does_not_store(self, client):
|
|
with patch(
|
|
'backend.services.auto_approval.executor.analyze_job_vocal_gaps',
|
|
return_value={"status": "checked", "stored": False},
|
|
) as analyze:
|
|
client.post(
|
|
"/api/internal/jobs/test123/vocal-gaps?dry_run=true",
|
|
headers={"Authorization": "Bearer test-admin-token"},
|
|
)
|
|
analyze.assert_called_once_with("test123", False)
|
|
|
|
def test_requires_admin(self, client):
|
|
response = client.post("/api/internal/jobs/test123/vocal-gaps")
|
|
assert response.status_code == 401
|