Files
karaoke-gen/backend/tests/test_credential_manager.py
Andrew Beveridge ☄️andClaude Opus 5.5 70bdd0e693 fix(alerts): start-review 409, per-cause crash patterns, one alert per job failure (v0.290.1) (#1243)
Section 1 of the 2026-11-03 #alerts triage handoff: a stale push on a
finished song got a 500, and the error monitor could stay silent on a
new crash.

Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566980744448843886
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566980746394996879
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566980747905073224
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1565697725280948294
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1565803187334815775
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1565818418072916110
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566225593354420226
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566256141074169987
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567207141994135654
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567252495535841386
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567252497620537415

## What changed

**1a. start-review:** a job not in review gets 409
(`jobs.jobNotInReviewState`) with no transition attempted. Losing the
auto-approver race is a 409 too. `correction-data` has the same race,
and its `except Exception` turned it into a 500; it now returns 409
(`except HTTPException: raise` added). Its status gate still returns
400, as before.

**1b. Crash patterns:** `Exception in ASGI application` and
`request_audit_error` become `Unhandled exception: <Type: message>`,
read from the entry's `exception` field. The handoff's suspicion is
confirmed in `upsert_pattern`: an existing `new` pattern returns
`is_new=False`, so a second, different crash stayed silent. Both log
lines of one crash now share one pattern. The path and method go in the
sample only.

I read the line after the last frame rather than the traceback's last
line. pydantic's last line is a docs link shared by every failure, and
group tracebacks end in a separator.

**1c. One alert per failure:**
- YouTube ids are normalized out of signatures, which also fixes the
false "first time" line in `ops_alerts`.
- `Job <id> failed` (both call sites in `job_manager`) logs WARNING,
since the status write sends the job-failed alert.
- `URL download failed:` is ignored in the monitor, not demoted, because
`file_upload.py` belongs to the section 4 PR.
- `[job:<id>] Download failed` logs WARNING for the expected
`DownloadError`/`FlacfetchServiceError` types. This wasn't named in the
handoff, but it is the same duplicate on the audio-search path.
Unexpected exceptions there stay ERROR with a traceback.

**1d. Suppressions and demotions:**
- Zalo's `zaloJSV2 is not defined`: dropped at `/api/client-errors`,
only with a Zalo UA.
- `python -c` experiments on `audio-separation-job`: ignored, using the
handoff's regex.
- Spotify 403 on `/lyrics`: WARNING with `response.text[:200]`. 401 on
`/lyrics` and 401/403 on `/search` stay ERROR.
- Dropbox token probe: WARNING. A rejected refresh used to log nothing,
so it now logs ERROR, as does having no refresh credentials.
- decide-search health aborts: this was a small change. `should_ignore`
takes an optional `request_url` and matches the exact path
`/api/search/health`.

**1e. Cold start:** `anyio._backends._asyncio` is added to
`_LAZY_ROUTE_IMPORTS`. The prod stack from 10-30 confirms the stall was
in that import. Lifespan also sends one `/api/health` request through
the app in a worker thread, bounded by the existing 30s timeout and
best-effort.

## Testing

- TDD: new tests failed first, then passed.
- Affected suites: 826 passed.
- `backend/tests` (excluding `backend/tests/emulator`): 7088 passed, 8
failed, 29 skipped. All 8 are in `test_emulator_integration.py`, and
that file also shows 8 failures at the base commit a8f9618a. I did not
compare the 8 test names one by one.
- `tests/unit` outside the listed files was not run.
`test_audio_fetcher.py` and `test_remote_cli.py` fail to collect in my
environment; I didn't change either file.
- Read-only replay of real prod entries through the new code: the 11-03
crash gave one specific pattern with `POST
/api/jobs/7f89f884/start-review` in the sample, the decide-search health
aborts and the experiment traceback were ignored.
- Local CodeRabbit review completed with 0 findings.

## Not verified

- Nothing is live, so the alert volume and the cold-start stall haven't
been measured. The warm request has only run in tests, not on a real
Cloud Run cold start.
- A request's first-use imports beyond the anyio backend aren't
individually checked.
- My test environment has no `flacfetch`, so its lazy imports only log a
warning here.
- Zalo: the trigger is one report (11-03).

## Known leftovers

- `validate_state_transition` still logs ERROR before raising, so the
auto-approver race (and the existing in_review race) still emits one
`Invalid state transition` ERROR even though the client now gets a clean
response. The stale-push case no longer reaches it. I left this because
other callers rely on that log.
- A crash that also logs its own ERROR first still gives two patterns,
the ERROR and `Unhandled exception`.
- The first bot block after deploy shows "first time" once, because the
normalized signature is new.
- Old `Exception in ASGI application` and `request_audit_error` patterns
stop recurring and will auto-resolve.
- If the section 4 PR rewords `URL download failed:`, the ignore entry
in `known_issues.py` needs the same wording.

## For Andrew (infra and cost, not changed here)

- Cloud Run startup probe and min-instances for `karaoke-backend`, if
cold-start stalls continue after this ships.
- decide-search's min-0 is why its health pings abort. The suppression
hides those, but real search paths will still alert on the same abort.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVTvqXyL4QyA2EjzQMG

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 23:04:04 +00:00

532 lines
21 KiB
Python

"""
Tests for credential manager and auth endpoints.
"""
import logging
import pytest
import json
from unittest.mock import MagicMock, patch
from datetime import datetime
from backend.services.credential_manager import (
CredentialManager,
CredentialStatus,
CredentialCheckResult,
DeviceAuthInfo,
_destroy_old_secret_versions,
get_credential_manager,
)
class TestCredentialStatus:
"""Tests for CredentialStatus enum."""
def test_status_values(self):
"""Test all status values exist."""
assert CredentialStatus.VALID == "valid"
assert CredentialStatus.EXPIRED == "expired"
assert CredentialStatus.INVALID == "invalid"
assert CredentialStatus.NOT_CONFIGURED == "not_configured"
assert CredentialStatus.ERROR == "error"
class TestCredentialCheckResult:
"""Tests for CredentialCheckResult dataclass."""
def test_create_result(self):
"""Test creating a check result."""
result = CredentialCheckResult(
service="youtube",
status=CredentialStatus.VALID,
message="Credentials are valid",
last_checked=datetime.utcnow()
)
assert result.service == "youtube"
assert result.status == CredentialStatus.VALID
assert result.message == "Credentials are valid"
assert result.expires_at is None
def test_create_result_with_expiry(self):
"""Test creating a check result with expiry."""
expiry = datetime.utcnow()
result = CredentialCheckResult(
service="gdrive",
status=CredentialStatus.VALID,
message="Valid",
last_checked=datetime.utcnow(),
expires_at=expiry
)
assert result.expires_at == expiry
class TestCredentialManager:
"""Tests for CredentialManager class."""
def test_init(self):
"""Test manager initialization."""
manager = CredentialManager()
assert manager._pending_device_auths == {}
def test_check_youtube_not_configured(self):
"""Test YouTube check when not configured."""
with patch('backend.services.credential_manager.get_settings') as mock_get_settings:
mock_settings = MagicMock()
mock_settings.get_secret.return_value = None
mock_get_settings.return_value = mock_settings
manager = CredentialManager()
result = manager.check_youtube_credentials()
assert result.status == CredentialStatus.NOT_CONFIGURED
assert "not configured" in result.message.lower()
def test_check_youtube_invalid_json(self):
"""Test YouTube check with invalid JSON."""
with patch('backend.services.credential_manager.get_settings') as mock_get_settings:
mock_settings = MagicMock()
mock_settings.get_secret.return_value = "not valid json"
mock_get_settings.return_value = mock_settings
manager = CredentialManager()
result = manager.check_youtube_credentials()
assert result.status == CredentialStatus.INVALID
assert "json" in result.message.lower()
def test_check_youtube_missing_fields(self):
"""Test YouTube check with missing required fields."""
# Missing refresh_token and client_secret
creds = json.dumps({"token": "test", "client_id": "test"})
with patch('backend.services.credential_manager.get_settings') as mock_get_settings:
mock_settings = MagicMock()
mock_settings.get_secret.return_value = creds
mock_get_settings.return_value = mock_settings
manager = CredentialManager()
result = manager.check_youtube_credentials()
assert result.status == CredentialStatus.INVALID
assert "missing" in result.message.lower()
def test_check_gdrive_not_configured(self):
"""Test Google Drive check when not configured."""
with patch('backend.services.credential_manager.get_settings') as mock_get_settings:
mock_settings = MagicMock()
mock_settings.get_secret.return_value = None
mock_get_settings.return_value = mock_settings
manager = CredentialManager()
result = manager.check_gdrive_credentials()
assert result.status == CredentialStatus.NOT_CONFIGURED
def test_check_dropbox_not_configured(self):
"""Test Dropbox check when not configured."""
with patch('backend.services.credential_manager.get_settings') as mock_get_settings:
mock_settings = MagicMock()
mock_settings.get_secret.return_value = None
mock_get_settings.return_value = mock_settings
manager = CredentialManager()
result = manager.check_dropbox_credentials()
assert result.status == CredentialStatus.NOT_CONFIGURED
def test_check_dropbox_missing_access_token(self):
"""Test Dropbox check with missing access token."""
creds = json.dumps({"refresh_token": "test"}) # No access_token
with patch('backend.services.credential_manager.get_settings') as mock_get_settings:
mock_settings = MagicMock()
mock_settings.get_secret.return_value = creds
mock_get_settings.return_value = mock_settings
manager = CredentialManager()
result = manager.check_dropbox_credentials()
assert result.status == CredentialStatus.INVALID
assert "access_token" in result.message.lower()
def test_check_all_credentials(self):
"""Test checking all credentials."""
with patch('backend.services.credential_manager.get_settings') as mock_get_settings:
mock_settings = MagicMock()
mock_settings.get_secret.return_value = None
mock_get_settings.return_value = mock_settings
manager = CredentialManager()
results = manager.check_all_credentials()
assert "youtube" in results
assert "gdrive" in results
assert "dropbox" in results
assert all(r.status == CredentialStatus.NOT_CONFIGURED for r in results.values())
def test_test_youtube_api_failure(self):
"""Test YouTube API test failure."""
manager = CredentialManager()
mock_creds = MagicMock()
with patch('googleapiclient.discovery.build', side_effect=Exception("API Error")):
result = manager._test_youtube_api(mock_creds)
assert result is False
def test_test_gdrive_api_failure(self):
"""Test Google Drive API test failure."""
manager = CredentialManager()
mock_creds = MagicMock()
with patch('googleapiclient.discovery.build', side_effect=Exception("API Error")):
result = manager._test_gdrive_api(mock_creds)
assert result is False
def test_test_dropbox_api_failure(self):
"""Test Dropbox API test failure."""
manager = CredentialManager()
with patch('dropbox.Dropbox', side_effect=Exception("API Error")):
result = manager._test_dropbox_api({"access_token": "test"})
assert result is False
def test_expired_dropbox_token_probe_logs_warning_not_error(self, caplog):
"""Short-lived tokens expire constantly and get refreshed; that isn't page-worthy."""
manager = CredentialManager()
with patch('dropbox.Dropbox', side_effect=Exception("expired_access_token")):
with caplog.at_level(logging.INFO, logger="backend.services.credential_manager"):
assert manager._test_dropbox_api({"access_token": "test"}) is False
levels = [r.levelno for r in caplog.records if "Dropbox" in r.message]
assert levels == [logging.WARNING]
def _dropbox_manager(self, creds):
manager = CredentialManager()
manager.settings = MagicMock()
manager.settings.get_secret.return_value = json.dumps(creds)
return manager
FULL_DROPBOX_CREDS = {"access_token": "a", "refresh_token": "r", "app_key": "k", "app_secret": "s"}
def test_dropbox_refresh_success_logs_no_error(self, caplog):
manager = self._dropbox_manager(self.FULL_DROPBOX_CREDS)
ok = MagicMock(status_code=200)
ok.json.return_value = {"access_token": "new"}
with patch('dropbox.Dropbox', side_effect=Exception("expired_access_token")), \
patch('requests.post', return_value=ok), \
patch.object(manager, "_update_dropbox_credentials", return_value=True):
with caplog.at_level(logging.INFO, logger="backend.services.credential_manager"):
result = manager.check_dropbox_credentials()
assert result.status == CredentialStatus.VALID
assert not [r for r in caplog.records if r.levelno >= logging.ERROR]
def test_dropbox_refresh_rejected_logs_error(self, caplog):
manager = self._dropbox_manager(self.FULL_DROPBOX_CREDS)
bad = MagicMock(status_code=400, text='{"error": "invalid_grant"}')
with patch('dropbox.Dropbox', side_effect=Exception("expired_access_token")), \
patch('requests.post', return_value=bad):
with caplog.at_level(logging.INFO, logger="backend.services.credential_manager"):
result = manager.check_dropbox_credentials()
assert result.status == CredentialStatus.INVALID
errors = [r.message for r in caplog.records if r.levelno >= logging.ERROR]
assert len(errors) == 1
assert "400" in errors[0]
assert "invalid_grant" in errors[0]
def test_dropbox_refresh_raising_logs_one_error(self, caplog):
manager = self._dropbox_manager(self.FULL_DROPBOX_CREDS)
with patch('dropbox.Dropbox', side_effect=Exception("expired_access_token")), \
patch('requests.post', side_effect=RuntimeError("network down")):
with caplog.at_level(logging.INFO, logger="backend.services.credential_manager"):
result = manager.check_dropbox_credentials()
assert result.status == CredentialStatus.INVALID
assert len([r for r in caplog.records if r.levelno >= logging.ERROR]) == 1
def test_dropbox_invalid_with_no_refresh_token_logs_error(self, caplog):
manager = self._dropbox_manager({"access_token": "a"})
with patch('dropbox.Dropbox', side_effect=Exception("expired_access_token")):
with caplog.at_level(logging.INFO, logger="backend.services.credential_manager"):
result = manager.check_dropbox_credentials()
assert result.status == CredentialStatus.INVALID
assert len([r for r in caplog.records if r.levelno >= logging.ERROR]) == 1
def test_send_credential_alert_no_webhook(self):
"""Test alert without webhook URL."""
manager = CredentialManager()
result = manager.send_credential_alert([], discord_webhook_url=None)
assert result is False
def test_send_credential_alert_success(self):
"""Test successful alert sending."""
manager = CredentialManager()
mock_response = MagicMock()
mock_response.status_code = 200
invalid_services = [
CredentialCheckResult(
service="youtube",
status=CredentialStatus.INVALID,
message="Test message",
last_checked=datetime.utcnow()
)
]
with patch('requests.post', return_value=mock_response) as mock_post:
result = manager.send_credential_alert(
invalid_services,
discord_webhook_url="https://discord.com/webhook/test"
)
assert result is True
mock_post.assert_called_once()
def test_start_youtube_device_auth(self):
"""Test starting YouTube device auth flow."""
manager = CredentialManager()
mock_response = MagicMock()
mock_response.status_code = 200
mock_response.json.return_value = {
"device_code": "test_device_code",
"user_code": "TEST-CODE",
"verification_uri": "https://google.com/device",
"expires_in": 1800,
"interval": 5
}
with patch('requests.post', return_value=mock_response):
device_info = manager.start_youtube_device_auth(
client_id="test_client",
client_secret="test_secret"
)
assert device_info.device_code == "test_device_code"
assert device_info.user_code == "TEST-CODE"
assert device_info.verification_url == "https://google.com/device"
# Should be stored for polling
key = f"youtube:{device_info.device_code}"
assert key in manager._pending_device_auths
def test_poll_device_auth_no_client_creds(self):
"""Test polling when client credentials are not in Secret Manager."""
manager = CredentialManager()
# Mock get_youtube_client_credentials to return None (no creds)
with patch.object(manager, 'get_youtube_client_credentials', return_value=None):
status, data = manager.poll_device_auth("youtube", "some_code")
assert status == "error"
assert "client credentials not found" in data["message"].lower()
def test_poll_device_auth_expired(self):
"""Test polling with expired device code."""
manager = CredentialManager()
mock_response = MagicMock()
mock_response.status_code = 400
mock_response.json.return_value = {"error": "expired_token"}
with patch.object(manager, 'get_youtube_client_credentials',
return_value={"client_id": "test", "client_secret": "test"}):
with patch('requests.post', return_value=mock_response):
status, data = manager.poll_device_auth("youtube", "expired_code")
assert status == "expired"
assert "expired" in data["message"].lower()
def test_poll_device_auth_pending(self):
"""Test polling when authorization is pending."""
manager = CredentialManager()
mock_response = MagicMock()
mock_response.status_code = 400
mock_response.json.return_value = {"error": "authorization_pending"}
with patch.object(manager, 'get_youtube_client_credentials',
return_value={"client_id": "test", "client_secret": "test"}):
with patch('requests.post', return_value=mock_response):
status, data = manager.poll_device_auth("youtube", "pending_code")
assert status == "pending"
assert "waiting" in data["message"].lower()
def test_poll_device_auth_success(self):
"""Test polling when authorization completes successfully."""
manager = CredentialManager()
mock_response = MagicMock()
mock_response.status_code = 200
mock_response.json.return_value = {
"access_token": "new_access_token",
"refresh_token": "new_refresh_token"
}
with patch.object(manager, 'get_youtube_client_credentials',
return_value={"client_id": "test_id", "client_secret": "test_secret"}):
with patch('requests.post', return_value=mock_response):
with patch.object(manager, '_save_credentials_to_secret', return_value=True):
status, data = manager.poll_device_auth("youtube", "completed_code")
assert status == "complete"
assert data["token"] == "new_access_token"
assert data["refresh_token"] == "new_refresh_token"
assert data["client_id"] == "test_id"
class TestGetCredentialManager:
"""Tests for singleton factory function."""
def test_returns_singleton(self):
"""Test that factory returns same instance."""
# Reset singleton
import backend.services.credential_manager as module
module._credential_manager = None
manager1 = get_credential_manager()
manager2 = get_credential_manager()
assert manager1 is manager2
class TestAuthRoutes:
"""Tests for auth API routes."""
def test_routes_import(self):
"""Test that routes can be imported."""
from backend.api.routes.auth import router
assert router is not None
def test_status_endpoint_exists(self):
"""Test that status endpoint is defined."""
from backend.api.routes.auth import get_credentials_status
assert get_credentials_status is not None
def test_validate_endpoint_exists(self):
"""Test that validate endpoint is defined."""
from backend.api.routes.auth import validate_credentials
assert validate_credentials is not None
def test_device_auth_endpoint_exists(self):
"""Test that device auth endpoints are defined."""
from backend.api.routes.auth import (
start_youtube_device_auth,
poll_youtube_device_auth,
start_gdrive_device_auth,
poll_gdrive_device_auth,
)
assert start_youtube_device_auth is not None
assert poll_youtube_device_auth is not None
assert start_gdrive_device_auth is not None
assert poll_gdrive_device_auth is not None
class TestFileUploadCredentialValidation:
"""Tests for credential validation in file upload."""
def test_credential_manager_imported(self):
"""Test that credential manager is imported in file upload."""
from backend.api.routes.file_upload import get_credential_manager, CredentialStatus
assert get_credential_manager is not None
assert CredentialStatus is not None
class TestDestroyOldSecretVersions:
"""Tests for _destroy_old_secret_versions helper."""
def _make_version(self, name, state, create_time):
v = MagicMock()
v.name = name
v.state = state
v.create_time = create_time
return v
def test_destroys_versions_beyond_keep_count(self):
"""Keep newest N, destroy the rest."""
client = MagicMock()
enabled = MagicMock()
# State enum lookup on the version (v.State.ENABLED) — make .State.ENABLED a sentinel
# and match it on each version's `.state` field.
State = MagicMock()
State.ENABLED = "ENABLED"
def mk(name, ts):
v = MagicMock()
v.name = name
v.state = State.ENABLED
v.State = State # so v.state == v.State.ENABLED resolves correctly
v.create_time = ts
return v
# 7 versions, oldest → newest by create_time
versions = [mk(f"v{i}", i) for i in range(7)]
client.list_secret_versions.return_value = versions
_destroy_old_secret_versions(client, "projects/p/secrets/s", keep=3)
# Should destroy 4 oldest (v0..v3); keep v4, v5, v6
destroyed_names = [
c.kwargs["request"]["name"]
for c in client.destroy_secret_version.call_args_list
]
assert sorted(destroyed_names) == ["v0", "v1", "v2", "v3"]
def test_swallows_list_errors(self):
"""List failure must not raise — refresh path must keep working."""
client = MagicMock()
client.list_secret_versions.side_effect = RuntimeError("boom")
_destroy_old_secret_versions(client, "projects/p/secrets/s", keep=5)
client.destroy_secret_version.assert_not_called()
def test_swallows_per_version_destroy_errors(self):
"""One bad destroy doesn't abort the rest."""
client = MagicMock()
State = MagicMock()
State.ENABLED = "ENABLED"
def mk(name, ts):
v = MagicMock()
v.name = name
v.state = State.ENABLED
v.State = State
v.create_time = ts
return v
versions = [mk(f"v{i}", i) for i in range(7)]
client.list_secret_versions.return_value = versions
# First destroy fails, rest succeed
client.destroy_secret_version.side_effect = [
RuntimeError("nope"), None, None, None,
]
_destroy_old_secret_versions(client, "projects/p/secrets/s", keep=3)
# All 4 destroy calls attempted despite first failing
assert client.destroy_secret_version.call_count == 4
def test_does_nothing_when_below_keep_count(self):
"""If fewer than `keep` enabled versions exist, destroy nothing."""
client = MagicMock()
State = MagicMock()
State.ENABLED = "ENABLED"
def mk(name, ts):
v = MagicMock()
v.name = name
v.state = State.ENABLED
v.State = State
v.create_time = ts
return v
versions = [mk(f"v{i}", i) for i in range(3)]
client.list_secret_versions.return_value = versions
_destroy_old_secret_versions(client, "projects/p/secrets/s", keep=5)
client.destroy_secret_version.assert_not_called()