Files
Andrew Beveridge ☄️andClaude Opus 5.5 8490ab75eb chore(gemini): move every Gemini call from Vertex AI to the Gemini Developer API (#1108)
## Summary
Moves **every** Gemini call in gen off **Vertex AI** (billed to the
`nomadkaraoke` GCP project, ~$85/mo) onto the **Gemini Developer API**
using the `gemini-api-key` Secret Manager secret (an AI Studio key
billed outside this project). Once deployed and verified,
`aiplatform.googleapis.com` / `generativelanguage.googleapis.com` get
disabled in nomadkaraoke.

### Key injection
- `backend/services/gemini_client.get_genai_client(timeout_ms)` — key
from `GEMINI_API_KEY`, else Secret Manager `gemini-api-key` via
`settings.get_secret` (runtime SA already has project-level
`secretAccessor`). No `project`/`location`.
- Service: CI `--set-secrets ... GEMINI_API_KEY=gemini-api-key:latest`.
- Jobs: Pulumi secret env on `lyrics-transcription-job`,
`video-encoding-job` (renders + translations), `nomad-error-monitor` —
**already applied** (targeted `pulumi up`, only that env added). Other
jobs fall back to Secret Manager.
- Local scripts (`frontend/scripts/translate.py`,
`scripts/measure_autocorrect_cost.py`,
`scripts/test_agentic_correction.py`): env, else `gcloud secrets
versions access` (in-process only).
- **Rotation:** add a new secret version; new instances / job executions
pick it up. No code change.

### Degradation (quota / billing / invalid key; per-minute 429s stay
retryable)
| Caller | Behaviour |
|---|---|
| auto-correct | non-retryable 503 → compare mode drops the Gemini leg,
keeps Opus |
| match-judge / free-text / parse-titles / tenant bulk | existing
fallbacks (no suggestion / blanks / regex) |
| credit eval | `pending_review` (fail-closed, unchanged) |
| lyrics translation | no retry; render proceeds without translation,
status recorded |
| custom lyrics | clean 503 "temporarily unavailable…" (+ 5-min timeout
so it can't hang) |
| error-monitor LLM | no retry; alerts sent without AI summary |
| translate.py | exit 2 "Gemini quota/credit exhausted — top up AI
Studio or rotate gemini-api-key" |

`note_gemini_failure()` also sends one Discord ops alert per 6h
(Firestore-throttled across instances via new
`ops_alerts.send_throttled_ops_alert`).

### Infra
- Pulumi: removes `roles/aiplatform.user` from `karaoke-backend` —
**will be applied after this deploys and Vertex traffic is confirmed at
zero**.
- Agentic correction default model `gemini/gemini-3.8-flash`
(`vertexai/` prefix still accepted, routed to the Developer API).
- Version 0.261.1 → 0.262.0. Docs: CLAUDE.md, LESSONS-LEARNED, README,
ARCHITECTURE, API, etc.

## Testing
- New: `backend/tests/services/test_backend_gemini_client.py`
(classifier, key resolution, no-Vertex client kwargs, alert throttle,
per-caller degradation),
`tests/unit/lyrics_transcriber/test_agentic_gemini_provider.py`,
`frontend/scripts/test_gemini_client.py`, lyrics-translation quota test.
- `pytest backend/tests --ignore=backend/tests/emulator` — all pass;
`pytest tests/unit` — 2021 passed.
- Verified `gemini-3.8-flash` is listed on the Developer API and
answered a 5-token test call with the key.
- `translate.py --dry-run` on `frontend/messages` and
`backend/translations` OK.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Updates**
* AI-powered lyric translation, custom lyrics, auto-correction, title
parsing, match judging, and related features now use the Gemini
Developer API.
* AI features use the configured Gemini API key. Quota, billing, or key
issues may show an unavailable message or use an existing fallback
instead of repeated attempts.
* **Documentation**
* Updated product, API, setup, and recovery guidance to reflect Gemini
API key configuration and usage.
* Updated guidance for handling AI service availability and quota
issues.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 21:58:53 -04:00

199 lines
7.9 KiB
Python

"""
Error Monitor Cloud Run Job and Cloud Scheduler resources.
Deploys the error monitor as a Cloud Run Job triggered by Cloud Scheduler
hourly (error checks) and daily at 08:00 UTC (digest report).
"""
import base64
import json
import pulumi
import pulumi_gcp as gcp
from pulumi_gcp import cloudrunv2, cloudscheduler, serviceaccount
from config import PROJECT_ID, REGION, ErrorMonitorConfig
# Empty body for the monitor trigger (Cloud Run Jobs API requires POST body)
_EMPTY_BODY_B64 = base64.b64encode(b"{}").decode()
# Digest trigger body — sets DIGEST_MODE=true via Cloud Run Job override
_DIGEST_BODY_B64 = base64.b64encode(
json.dumps({
"overrides": {
"containerOverrides": [
{
"env": [
{"name": "DIGEST_MODE", "value": "true"},
]
}
]
}
}).encode()
).decode()
def create_error_monitor(
backend_image: pulumi.Input[str],
backend_sa_email: pulumi.Input[str],
depends_on: list = None,
) -> dict:
"""
Create the error monitor Cloud Run Job and Cloud Scheduler triggers.
The error monitor runs as the backend SA (same permissions: Firestore,
Secrets, Logging). A dedicated scheduler SA is created for Cloud Scheduler
OAuth authentication.
Args:
backend_image: Docker image URL for the backend (Artifact Registry).
backend_sa_email: Email of the backend service account (runs the job).
depends_on: Optional list of Pulumi resources to depend on.
Returns:
dict with keys: 'job', 'scheduler_sa', 'monitor_scheduler', 'digest_scheduler'.
"""
job_opts = pulumi.ResourceOptions(depends_on=depends_on) if depends_on else None
# ---- Service Account for Cloud Scheduler ----
# Scheduler SA authenticates when invoking the Cloud Run Job via OAuth token.
scheduler_sa = serviceaccount.Account(
"error-monitor-scheduler-sa",
account_id=ErrorMonitorConfig.SCHEDULER_SA_NAME,
display_name="Error Monitor Scheduler Service Account",
)
# ---- Cloud Run Job ----
error_monitor_job = cloudrunv2.Job(
"error-monitor-job",
name=ErrorMonitorConfig.JOB_NAME,
location=REGION,
template=cloudrunv2.JobTemplateArgs(
template=cloudrunv2.JobTemplateTemplateArgs(
containers=[
cloudrunv2.JobTemplateTemplateContainerArgs(
image=backend_image,
args=["python", "-m", "backend.services.error_monitor.monitor"],
resources=cloudrunv2.JobTemplateTemplateContainerResourcesArgs(
limits={
"cpu": ErrorMonitorConfig.CPU,
"memory": ErrorMonitorConfig.MEMORY,
},
),
envs=[
# GCP project for Firestore, Logging, etc.
cloudrunv2.JobTemplateTemplateContainerEnvArgs(
name="GCP_PROJECT_ID",
value=PROJECT_ID,
),
# Environment tag for log filtering
cloudrunv2.JobTemplateTemplateContainerEnvArgs(
name="ENVIRONMENT",
value="production",
),
# Gemini Developer API key (AI Studio; Vertex AI is
# disabled in this project). Rotate = new secret version.
cloudrunv2.JobTemplateTemplateContainerEnvArgs(
name="GEMINI_API_KEY",
value_source=cloudrunv2.JobTemplateTemplateContainerEnvValueSourceArgs(
secret_key_ref=cloudrunv2.JobTemplateTemplateContainerEnvValueSourceSecretKeyRefArgs(
secret=f"projects/{PROJECT_ID}/secrets/gemini-api-key",
version="latest",
),
),
),
# Discord webhook for alert notifications
cloudrunv2.JobTemplateTemplateContainerEnvArgs(
name="DISCORD_WEBHOOK_URL",
value_source=cloudrunv2.JobTemplateTemplateContainerEnvValueSourceArgs(
secret_key_ref=cloudrunv2.JobTemplateTemplateContainerEnvValueSourceSecretKeyRefArgs(
secret=f"projects/{PROJECT_ID}/secrets/discord-alert-webhook",
version="latest",
),
),
),
],
)
],
service_account=backend_sa_email,
timeout=ErrorMonitorConfig.TIMEOUT,
max_retries=ErrorMonitorConfig.MAX_RETRIES,
),
),
opts=job_opts,
)
# ---- IAM: grant scheduler SA permission to invoke (run) the job ----
error_monitor_job_invoker = cloudrunv2.JobIamMember(
"error-monitor-scheduler-invoker",
project=PROJECT_ID,
location=REGION,
name=error_monitor_job.name,
role="roles/run.invoker",
member=scheduler_sa.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Cloud Run Jobs API endpoint — Cloud Scheduler calls this to trigger a run.
# Format: https://run.googleapis.com/v2/projects/{project}/locations/{region}/jobs/{job}:run
job_run_uri = error_monitor_job.name.apply(
lambda name: (
f"https://run.googleapis.com/v2/projects/{PROJECT_ID}"
f"/locations/{REGION}/jobs/{name}:run"
)
)
# Shared opts: schedulers depend on IAM binding being in place first
scheduler_opts = pulumi.ResourceOptions(depends_on=[error_monitor_job_invoker])
# ---- Cloud Scheduler: hourly monitor ----
monitor_scheduler = cloudscheduler.Job(
"error-monitor-scheduler",
name="error-monitor-trigger",
description="Trigger error monitor Cloud Run Job hourly",
region=REGION,
schedule=ErrorMonitorConfig.MONITOR_SCHEDULE,
time_zone="UTC",
http_target=cloudscheduler.JobHttpTargetArgs(
uri=job_run_uri,
http_method="POST",
body=_EMPTY_BODY_B64,
oauth_token=cloudscheduler.JobHttpTargetOauthTokenArgs(
service_account_email=scheduler_sa.email,
),
),
retry_config=cloudscheduler.JobRetryConfigArgs(
retry_count=0,
),
opts=scheduler_opts,
)
# ---- Cloud Scheduler: daily digest at 08:00 UTC ----
# Passes DIGEST_MODE=true in the request body as a Cloud Run Job override.
digest_scheduler = cloudscheduler.Job(
"error-monitor-digest-scheduler",
name="error-monitor-daily-digest",
description="Trigger error monitor Cloud Run Job daily at 08:00 UTC for digest report",
region=REGION,
schedule=ErrorMonitorConfig.DIGEST_SCHEDULE,
time_zone="UTC",
http_target=cloudscheduler.JobHttpTargetArgs(
uri=job_run_uri,
http_method="POST",
body=_DIGEST_BODY_B64,
oauth_token=cloudscheduler.JobHttpTargetOauthTokenArgs(
service_account_email=scheduler_sa.email,
),
),
retry_config=cloudscheduler.JobRetryConfigArgs(
retry_count=0,
),
opts=scheduler_opts,
)
return {
"job": error_monitor_job,
"scheduler_sa": scheduler_sa,
"monitor_scheduler": monitor_scheduler,
"digest_scheduler": digest_scheduler,
}