mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary The "Reconnecting" pill and the "servers unavailable" banner (164 / 49 events, 2026-09-19 → 10-02) were mostly caused by the single API instance's **event loop freezing for 20–45s**. Every request on the instance, `/api/health` included, completed at the same moment. The full evidence and plan are in `docs/archive/2026-10-03-backend-loop-freezes-telemetry-plan.md`. ### What froze the loop, and the fix for each - **Screens worker:** it ran as a post-response `async def` BackgroundTask doing 4K PIL renders, GCS transfers and ffmpeg transcodes on the loop. Under `--cpu-throttling` it was also CPU-starved. **Fix:** it now runs inline in its Cloud Tasks request (CPU stays allocated, failures still return 200 so Cloud Tasks doesn't double-run it), with the heavy steps in `asyncio.to_thread`. Also offloaded: style load, timing signals, pre-apply. - **Matplotlib font cache:** it was rebuilt on every new instance (17/28 episodes). **Fix:** it's pre-built in `Dockerfile.base` (`MPLCONFIGDIR=/opt/mplconfig`), and pyplot is imported lazily. - **Hourly stale-review cron:** an `async def` with zero awaits doing ≤1500 reads and SMTP sends. **Fix:** it runs in a thread. - **166 `async def` routes that never awaited** (sync Firestore/GCS/email/Gemini on the loop, e.g. magic-link verify). **Fix:** they're now plain `def`, so they run in FastAPI's threadpool. The new AST guard `test_no_blocking_async_routes.py` fails CI on new ones. It also flags sync routes that call `asyncio.create_task`/`get_running_loop`; that caught `confirm_duration` and `trigger_youtube_queue_processing`, which stay async. ### Safety net - `backend/services/loop_watchdog.py`: a heartbeat plus a daemon thread. Any stall over 1s logs `EVENT_LOOP_STALL` with the loop thread's stack and a `culprit=file:line`. Stalls of 5s or more are written to `client_events` as `server_loop_stall`. - Pulumi: the log metric `backend/event_loop_stalls` plus an alert on ≥2 stalls of ≥10s in 30 min. **This still needs a local `pulumi up`.** Pulumi is never applied by CI, and the admin gcloud login has expired. ### Telemetry gaps fixed - **Identity:** `user_email` used to be hard-coded to null. The frontend now sends its session token, and the server resolves `user_email`, `is_admin`, `is_internal` and `is_test`. A client-sent email is ignored. - **New fields:** `device_fingerprint` (the existing FingerprintJS visitorId), `tab_id`, `episode_id`, and `tenant`. - **New event:** `banner_recovered`, carrying `duration_ms` and `peak_status`. - **Report:** `scripts/client_events_report.py` summarises the data, excluding admin/internal/test accounts by default. ## Testing - `make test` passes: backend 5529, plus the emulator suite, karaoke_gen unit tests, and frontend Jest 1623. - New tests: - Watchdog: a deterministic state machine plus detection of a real blocking call. - Event-loop responsiveness: a ticker keeps ticking while the heavy steps run. - Screens inline and failure semantics. - `client_events` identity and fields. - The AST guard. - The report summary. - Frontend reporter and banner episode tests. - Local CodeRabbit CLI review: 0 findings. 🤖 Generated with [Claude Code](https://claude.com/claude-code) @coderabbitai ignore --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
459 lines
18 KiB
Python
459 lines
18 KiB
Python
"""
|
|
Unit tests for admin feedback endpoint and feedback email notification.
|
|
"""
|
|
import pytest
|
|
from unittest.mock import Mock, patch, MagicMock
|
|
from datetime import datetime
|
|
|
|
from backend.services.email_service import EmailService
|
|
|
|
|
|
class TestAdminFeedbackEndpoint:
|
|
"""Tests for GET /api/admin/feedback endpoint."""
|
|
|
|
def _make_feedback_doc(self, email="user@example.com", overall=4, **kwargs):
|
|
"""Create a mock Firestore feedback document."""
|
|
doc = Mock()
|
|
doc.id = kwargs.get("id", "test-id-1")
|
|
data = {
|
|
"id": doc.id,
|
|
"user_email": email,
|
|
"created_at": kwargs.get("created_at", datetime(2026, 3, 1, 12, 0, 0)),
|
|
"overall_rating": overall,
|
|
"ease_of_use_rating": kwargs.get("ease_of_use", 4),
|
|
"lyrics_accuracy_rating": kwargs.get("lyrics_accuracy", 3),
|
|
"correction_experience_rating": kwargs.get("correction_experience", 4),
|
|
"what_went_well": kwargs.get("what_went_well", "Great app"),
|
|
"what_could_improve": kwargs.get("what_could_improve", "More songs"),
|
|
"additional_comments": kwargs.get("additional_comments", None),
|
|
"would_recommend": kwargs.get("would_recommend", True),
|
|
"would_use_again": kwargs.get("would_use_again", True),
|
|
}
|
|
doc.to_dict.return_value = data
|
|
return doc
|
|
|
|
@patch("backend.api.routes.admin.get_user_service")
|
|
@patch("backend.api.routes.admin.is_test_email")
|
|
def test_list_feedback_returns_items(self, mock_is_test, mock_get_user_svc):
|
|
"""Test that feedback items are returned correctly."""
|
|
from backend.api.routes.admin import list_user_feedback
|
|
|
|
mock_db = Mock()
|
|
mock_user_svc = Mock()
|
|
mock_user_svc.db = mock_db
|
|
mock_get_user_svc.return_value = mock_user_svc
|
|
mock_is_test.return_value = False
|
|
|
|
doc1 = self._make_feedback_doc("alice@example.com", 5, id="id-1")
|
|
doc2 = self._make_feedback_doc("bob@example.com", 3, id="id-2")
|
|
mock_db.collection.return_value.stream.return_value = [doc1, doc2]
|
|
|
|
result = list_user_feedback(
|
|
limit=50, offset=0, search=None, exclude_test=False,
|
|
auth_result=("admin@test.com", "admin", 1),
|
|
)
|
|
|
|
assert result.total == 2
|
|
assert len(result.items) == 2
|
|
assert result.has_more is False
|
|
|
|
@patch("backend.api.routes.admin.get_user_service")
|
|
@patch("backend.api.routes.admin.is_test_email")
|
|
def test_list_feedback_search_filter(self, mock_is_test, mock_get_user_svc):
|
|
"""Test that search filters by email."""
|
|
from backend.api.routes.admin import list_user_feedback
|
|
|
|
mock_db = Mock()
|
|
mock_user_svc = Mock()
|
|
mock_user_svc.db = mock_db
|
|
mock_get_user_svc.return_value = mock_user_svc
|
|
mock_is_test.return_value = False
|
|
|
|
doc1 = self._make_feedback_doc("alice@example.com", 5, id="id-1")
|
|
doc2 = self._make_feedback_doc("bob@example.com", 3, id="id-2")
|
|
mock_db.collection.return_value.stream.return_value = [doc1, doc2]
|
|
|
|
result = list_user_feedback(
|
|
limit=50, offset=0, search="alice", exclude_test=False,
|
|
auth_result=("admin@test.com", "admin", 1),
|
|
)
|
|
|
|
assert result.total == 1
|
|
assert result.items[0].user_email == "alice@example.com"
|
|
|
|
@patch("backend.api.routes.admin.get_user_service")
|
|
@patch("backend.api.routes.admin.is_test_email")
|
|
def test_list_feedback_exclude_test(self, mock_is_test, mock_get_user_svc):
|
|
"""Test that test emails are excluded when exclude_test=True."""
|
|
from backend.api.routes.admin import list_user_feedback
|
|
|
|
mock_db = Mock()
|
|
mock_user_svc = Mock()
|
|
mock_user_svc.db = mock_db
|
|
mock_get_user_svc.return_value = mock_user_svc
|
|
|
|
doc1 = self._make_feedback_doc("alice@example.com", 5, id="id-1")
|
|
doc2 = self._make_feedback_doc("test@inbox.testmail.app", 3, id="id-2")
|
|
mock_db.collection.return_value.stream.return_value = [doc1, doc2]
|
|
|
|
# Make is_test_email return True for the test email
|
|
mock_is_test.side_effect = lambda e: "testmail" in e
|
|
|
|
result = list_user_feedback(
|
|
limit=50, offset=0, search=None, exclude_test=True,
|
|
auth_result=("admin@test.com", "admin", 1),
|
|
)
|
|
|
|
assert result.total == 1
|
|
assert result.items[0].user_email == "alice@example.com"
|
|
|
|
@patch("backend.api.routes.admin.get_user_service")
|
|
@patch("backend.api.routes.admin.is_test_email")
|
|
def test_list_feedback_pagination(self, mock_is_test, mock_get_user_svc):
|
|
"""Test pagination with offset and limit."""
|
|
from backend.api.routes.admin import list_user_feedback
|
|
|
|
mock_db = Mock()
|
|
mock_user_svc = Mock()
|
|
mock_user_svc.db = mock_db
|
|
mock_get_user_svc.return_value = mock_user_svc
|
|
mock_is_test.return_value = False
|
|
|
|
docs = [
|
|
self._make_feedback_doc(f"user{i}@example.com", 4, id=f"id-{i}")
|
|
for i in range(5)
|
|
]
|
|
mock_db.collection.return_value.stream.return_value = docs
|
|
|
|
result = list_user_feedback(
|
|
limit=2, offset=2, search=None, exclude_test=False,
|
|
auth_result=("admin@test.com", "admin", 1),
|
|
)
|
|
|
|
assert result.total == 5
|
|
assert len(result.items) == 2
|
|
assert result.has_more is True
|
|
assert result.offset == 2
|
|
assert result.limit == 2
|
|
|
|
@patch("backend.api.routes.admin.get_user_service")
|
|
@patch("backend.api.routes.admin.is_test_email")
|
|
def test_list_feedback_aggregate_stats(self, mock_is_test, mock_get_user_svc):
|
|
"""Test that aggregate rating averages are computed correctly."""
|
|
from backend.api.routes.admin import list_user_feedback
|
|
|
|
mock_db = Mock()
|
|
mock_user_svc = Mock()
|
|
mock_user_svc.db = mock_db
|
|
mock_get_user_svc.return_value = mock_user_svc
|
|
mock_is_test.return_value = False
|
|
|
|
doc1 = self._make_feedback_doc("a@x.com", 5, id="id-1", ease_of_use=4, lyrics_accuracy=3, correction_experience=2)
|
|
doc2 = self._make_feedback_doc("b@x.com", 3, id="id-2", ease_of_use=2, lyrics_accuracy=5, correction_experience=4)
|
|
mock_db.collection.return_value.stream.return_value = [doc1, doc2]
|
|
|
|
result = list_user_feedback(
|
|
limit=50, offset=0, search=None, exclude_test=False,
|
|
auth_result=("admin@test.com", "admin", 1),
|
|
)
|
|
|
|
assert result.avg_overall_rating == 4.0
|
|
assert result.avg_ease_of_use_rating == 3.0
|
|
assert result.avg_lyrics_accuracy_rating == 4.0
|
|
assert result.avg_correction_experience_rating == 3.0
|
|
|
|
@patch("backend.api.routes.admin.get_user_service")
|
|
@patch("backend.api.routes.admin.is_test_email")
|
|
def test_list_feedback_empty(self, mock_is_test, mock_get_user_svc):
|
|
"""Test empty results."""
|
|
from backend.api.routes.admin import list_user_feedback
|
|
|
|
mock_db = Mock()
|
|
mock_user_svc = Mock()
|
|
mock_user_svc.db = mock_db
|
|
mock_get_user_svc.return_value = mock_user_svc
|
|
|
|
mock_db.collection.return_value.stream.return_value = []
|
|
|
|
result = list_user_feedback(
|
|
limit=50, offset=0, search=None, exclude_test=False,
|
|
auth_result=("admin@test.com", "admin", 1),
|
|
)
|
|
|
|
assert result.total == 0
|
|
assert len(result.items) == 0
|
|
assert result.avg_overall_rating is None
|
|
|
|
|
|
@patch("backend.api.routes.admin.get_user_service")
|
|
@patch("backend.api.routes.admin.is_test_email")
|
|
def test_list_feedback_handles_string_created_at(self, mock_is_test, mock_get_user_svc):
|
|
"""Test that string timestamps (not datetime objects) are handled correctly."""
|
|
from backend.api.routes.admin import list_user_feedback
|
|
|
|
mock_db = Mock()
|
|
mock_user_svc = Mock()
|
|
mock_user_svc.db = mock_db
|
|
mock_get_user_svc.return_value = mock_user_svc
|
|
mock_is_test.return_value = False
|
|
|
|
doc = self._make_feedback_doc("a@x.com", 5, id="id-1", created_at="2026-03-01T12:00:00")
|
|
mock_db.collection.return_value.stream.return_value = [doc]
|
|
|
|
result = list_user_feedback(
|
|
limit=50, offset=0, search=None, exclude_test=False,
|
|
auth_result=("admin@test.com", "admin", 1),
|
|
)
|
|
|
|
assert result.items[0].created_at == "2026-03-01T12:00:00"
|
|
|
|
@patch("backend.api.routes.admin.get_user_service")
|
|
@patch("backend.api.routes.admin.is_test_email")
|
|
def test_list_feedback_handles_none_created_at(self, mock_is_test, mock_get_user_svc):
|
|
"""Test that None created_at is handled correctly."""
|
|
from backend.api.routes.admin import list_user_feedback
|
|
|
|
mock_db = Mock()
|
|
mock_user_svc = Mock()
|
|
mock_user_svc.db = mock_db
|
|
mock_get_user_svc.return_value = mock_user_svc
|
|
mock_is_test.return_value = False
|
|
|
|
doc = self._make_feedback_doc("a@x.com", 5, id="id-1", created_at=None)
|
|
mock_db.collection.return_value.stream.return_value = [doc]
|
|
|
|
result = list_user_feedback(
|
|
limit=50, offset=0, search=None, exclude_test=False,
|
|
auth_result=("admin@test.com", "admin", 1),
|
|
)
|
|
|
|
assert result.items[0].created_at is None
|
|
|
|
|
|
class TestSubmitFeedbackEmailIntegration:
|
|
"""Test that submit_user_feedback correctly calls send_feedback_notification."""
|
|
|
|
@patch("backend.services.email_service.get_email_service")
|
|
@patch("backend.services.auth_service.get_auth_service")
|
|
def test_submit_feedback_sends_admin_notification(self, mock_get_auth, mock_get_email):
|
|
"""Verify the caller (submit_user_feedback) passes correct args to email service."""
|
|
import asyncio
|
|
import uuid as uuid_mod
|
|
from backend.api.routes.users import submit_user_feedback
|
|
from backend.models.user import UserFeedbackRequest
|
|
|
|
# Mock auth
|
|
mock_auth_svc = Mock()
|
|
mock_auth_result = Mock()
|
|
mock_auth_result.is_valid = True
|
|
mock_auth_result.user_email = "alice@example.com"
|
|
mock_auth_svc.validate_token_full.return_value = mock_auth_result
|
|
mock_get_auth.return_value = mock_auth_svc
|
|
|
|
# Mock email service
|
|
mock_email_svc = Mock()
|
|
mock_email_svc.send_feedback_notification.return_value = True
|
|
mock_get_email.return_value = mock_email_svc
|
|
|
|
# Mock user service with a db that has a collection mock
|
|
mock_user_svc = Mock()
|
|
mock_user = Mock()
|
|
mock_user.email = "alice@example.com"
|
|
mock_user.total_jobs_completed = 5
|
|
mock_user.has_submitted_feedback = False
|
|
mock_user_svc.get_or_create_user.return_value = mock_user
|
|
mock_user_svc.update_user.return_value = None
|
|
mock_user_svc.add_credits.return_value = None
|
|
|
|
went_well = "The lyrics synchronization was incredibly accurate and impressive overall"
|
|
could_improve = "Speed could be faster for longer songs in the processing queue"
|
|
|
|
request = UserFeedbackRequest(
|
|
overall_rating=4,
|
|
ease_of_use_rating=3,
|
|
lyrics_accuracy_rating=5,
|
|
correction_experience_rating=2,
|
|
what_went_well=went_well,
|
|
what_could_improve=could_improve,
|
|
additional_comments=None,
|
|
would_recommend=True,
|
|
would_use_again=False,
|
|
)
|
|
|
|
with patch.object(uuid_mod, "uuid4", return_value="fake-uuid"):
|
|
result = (submit_user_feedback(http_request=MagicMock(headers={"accept-language": "en"}), request=request, authorization="Bearer test-token", user_service=mock_user_svc)
|
|
)
|
|
|
|
# Verify email notification was called with matching args
|
|
mock_email_svc.send_feedback_notification.assert_called_once_with(
|
|
user_email="alice@example.com",
|
|
overall_rating=4,
|
|
ease_of_use_rating=3,
|
|
lyrics_accuracy_rating=5,
|
|
correction_experience_rating=2,
|
|
what_went_well=went_well,
|
|
what_could_improve=could_improve,
|
|
additional_comments=None,
|
|
would_recommend=True,
|
|
would_use_again=False,
|
|
)
|
|
|
|
assert result.credits_granted == 1
|
|
|
|
@patch("backend.services.email_service.get_email_service")
|
|
@patch("backend.services.auth_service.get_auth_service")
|
|
def test_submit_feedback_continues_on_email_failure(self, mock_get_auth, mock_get_email):
|
|
"""Verify feedback submission succeeds even if email notification fails."""
|
|
import asyncio
|
|
import uuid as uuid_mod
|
|
from backend.api.routes.users import submit_user_feedback
|
|
from backend.models.user import UserFeedbackRequest
|
|
|
|
mock_auth_svc = Mock()
|
|
mock_auth_result = Mock()
|
|
mock_auth_result.is_valid = True
|
|
mock_auth_result.user_email = "bob@example.com"
|
|
mock_auth_svc.validate_token_full.return_value = mock_auth_result
|
|
mock_get_auth.return_value = mock_auth_svc
|
|
|
|
# Email service throws an exception
|
|
mock_email_svc = Mock()
|
|
mock_email_svc.send_feedback_notification.side_effect = Exception("Postmark down")
|
|
mock_get_email.return_value = mock_email_svc
|
|
|
|
mock_user_svc = Mock()
|
|
mock_user = Mock()
|
|
mock_user.email = "bob@example.com"
|
|
mock_user.total_jobs_completed = 3
|
|
mock_user.has_submitted_feedback = False
|
|
mock_user_svc.get_or_create_user.return_value = mock_user
|
|
|
|
request = UserFeedbackRequest(
|
|
overall_rating=5,
|
|
ease_of_use_rating=5,
|
|
lyrics_accuracy_rating=5,
|
|
correction_experience_rating=5,
|
|
what_went_well="Everything was perfect and I really enjoyed the experience a lot",
|
|
)
|
|
|
|
with patch.object(uuid_mod, "uuid4", return_value="fake-uuid"):
|
|
result = (submit_user_feedback(http_request=MagicMock(headers={"accept-language": "en"}), request=request, authorization="Bearer test-token", user_service=mock_user_svc)
|
|
)
|
|
|
|
# User still gets credits even though email failed
|
|
assert result.status == "success"
|
|
assert result.credits_granted == 1
|
|
|
|
|
|
class TestFeedbackEmailNotification:
|
|
"""Tests for send_feedback_notification email method."""
|
|
|
|
@patch.object(EmailService, '_get_provider')
|
|
def test_send_feedback_notification_calls_provider(self, mock_get_provider):
|
|
"""Test that feedback notification sends email via provider."""
|
|
mock_provider = Mock()
|
|
mock_provider.send_email.return_value = True
|
|
mock_get_provider.return_value = mock_provider
|
|
|
|
service = EmailService()
|
|
result = service.send_feedback_notification(
|
|
user_email="alice@example.com",
|
|
overall_rating=5,
|
|
ease_of_use_rating=4,
|
|
lyrics_accuracy_rating=3,
|
|
correction_experience_rating=4,
|
|
what_went_well="Great lyrics sync",
|
|
what_could_improve="More song variety",
|
|
additional_comments=None,
|
|
would_recommend=True,
|
|
would_use_again=True,
|
|
)
|
|
|
|
assert result is True
|
|
mock_provider.send_email.assert_called_once()
|
|
call_args = mock_provider.send_email.call_args
|
|
|
|
# Check email is sent to admin
|
|
to_email = call_args.kwargs.get("to_email") or call_args[0][0]
|
|
assert to_email == "admin@nomadkaraoke.com"
|
|
|
|
# Check subject contains user email and rating
|
|
subject = call_args.kwargs.get("subject") or call_args[0][1]
|
|
assert "alice@example.com" in subject
|
|
assert "5/5" in subject
|
|
|
|
@patch.object(EmailService, '_get_provider')
|
|
def test_send_feedback_notification_content(self, mock_get_provider):
|
|
"""Test that feedback notification email contains all feedback fields."""
|
|
mock_provider = Mock()
|
|
mock_provider.send_email.return_value = True
|
|
mock_get_provider.return_value = mock_provider
|
|
|
|
service = EmailService()
|
|
service.send_feedback_notification(
|
|
user_email="bob@example.com",
|
|
overall_rating=3,
|
|
ease_of_use_rating=2,
|
|
lyrics_accuracy_rating=4,
|
|
correction_experience_rating=5,
|
|
what_went_well="Easy to use",
|
|
what_could_improve="Better mobile support",
|
|
additional_comments="Love it overall",
|
|
would_recommend=False,
|
|
would_use_again=True,
|
|
)
|
|
|
|
call_args = mock_provider.send_email.call_args
|
|
html_content = call_args.kwargs.get("html_content") or call_args[0][2]
|
|
text_content = call_args.kwargs.get("text_content") or call_args[0][3]
|
|
|
|
# Check HTML contains feedback text
|
|
assert "Easy to use" in html_content
|
|
assert "Better mobile support" in html_content
|
|
assert "Love it overall" in html_content
|
|
assert "bob@example.com" in html_content
|
|
|
|
# Check plain text also contains feedback
|
|
assert "Easy to use" in text_content
|
|
assert "Better mobile support" in text_content
|
|
assert "3/5" in text_content
|
|
|
|
@patch.object(EmailService, '_get_provider')
|
|
@patch.dict("os.environ", {"ADMIN_NOTIFICATION_EMAIL": "custom@admin.com"})
|
|
def test_send_feedback_notification_custom_admin_email(self, mock_get_provider):
|
|
"""Test that admin email can be configured via env var."""
|
|
mock_provider = Mock()
|
|
mock_provider.send_email.return_value = True
|
|
mock_get_provider.return_value = mock_provider
|
|
|
|
service = EmailService()
|
|
service.send_feedback_notification(
|
|
user_email="test@example.com",
|
|
overall_rating=4,
|
|
ease_of_use_rating=4,
|
|
lyrics_accuracy_rating=4,
|
|
correction_experience_rating=4,
|
|
)
|
|
|
|
call_args = mock_provider.send_email.call_args
|
|
to_email = call_args.kwargs.get("to_email") or call_args[0][0]
|
|
assert to_email == "custom@admin.com"
|
|
|
|
@patch.object(EmailService, '_get_provider')
|
|
def test_send_feedback_notification_returns_false_on_failure(self, mock_get_provider):
|
|
"""Test that failure is returned when provider fails."""
|
|
mock_provider = Mock()
|
|
mock_provider.send_email.return_value = False
|
|
mock_get_provider.return_value = mock_provider
|
|
|
|
service = EmailService()
|
|
result = service.send_feedback_notification(
|
|
user_email="test@example.com",
|
|
overall_rating=1,
|
|
ease_of_use_rating=1,
|
|
lyrics_accuracy_rating=1,
|
|
correction_experience_rating=1,
|
|
)
|
|
|
|
assert result is False
|