Files
karaoke-gen/scripts/measure_autocorrect_cost.py
Andrew Beveridge ☄️andClaude Opus 5.5 8490ab75eb chore(gemini): move every Gemini call from Vertex AI to the Gemini Developer API (#1108)
## Summary
Moves **every** Gemini call in gen off **Vertex AI** (billed to the
`nomadkaraoke` GCP project, ~$85/mo) onto the **Gemini Developer API**
using the `gemini-api-key` Secret Manager secret (an AI Studio key
billed outside this project). Once deployed and verified,
`aiplatform.googleapis.com` / `generativelanguage.googleapis.com` get
disabled in nomadkaraoke.

### Key injection
- `backend/services/gemini_client.get_genai_client(timeout_ms)` — key
from `GEMINI_API_KEY`, else Secret Manager `gemini-api-key` via
`settings.get_secret` (runtime SA already has project-level
`secretAccessor`). No `project`/`location`.
- Service: CI `--set-secrets ... GEMINI_API_KEY=gemini-api-key:latest`.
- Jobs: Pulumi secret env on `lyrics-transcription-job`,
`video-encoding-job` (renders + translations), `nomad-error-monitor` —
**already applied** (targeted `pulumi up`, only that env added). Other
jobs fall back to Secret Manager.
- Local scripts (`frontend/scripts/translate.py`,
`scripts/measure_autocorrect_cost.py`,
`scripts/test_agentic_correction.py`): env, else `gcloud secrets
versions access` (in-process only).
- **Rotation:** add a new secret version; new instances / job executions
pick it up. No code change.

### Degradation (quota / billing / invalid key; per-minute 429s stay
retryable)
| Caller | Behaviour |
|---|---|
| auto-correct | non-retryable 503 → compare mode drops the Gemini leg,
keeps Opus |
| match-judge / free-text / parse-titles / tenant bulk | existing
fallbacks (no suggestion / blanks / regex) |
| credit eval | `pending_review` (fail-closed, unchanged) |
| lyrics translation | no retry; render proceeds without translation,
status recorded |
| custom lyrics | clean 503 "temporarily unavailable…" (+ 5-min timeout
so it can't hang) |
| error-monitor LLM | no retry; alerts sent without AI summary |
| translate.py | exit 2 "Gemini quota/credit exhausted — top up AI
Studio or rotate gemini-api-key" |

`note_gemini_failure()` also sends one Discord ops alert per 6h
(Firestore-throttled across instances via new
`ops_alerts.send_throttled_ops_alert`).

### Infra
- Pulumi: removes `roles/aiplatform.user` from `karaoke-backend` —
**will be applied after this deploys and Vertex traffic is confirmed at
zero**.
- Agentic correction default model `gemini/gemini-3.8-flash`
(`vertexai/` prefix still accepted, routed to the Developer API).
- Version 0.261.1 → 0.262.0. Docs: CLAUDE.md, LESSONS-LEARNED, README,
ARCHITECTURE, API, etc.

## Testing
- New: `backend/tests/services/test_backend_gemini_client.py`
(classifier, key resolution, no-Vertex client kwargs, alert throttle,
per-caller degradation),
`tests/unit/lyrics_transcriber/test_agentic_gemini_provider.py`,
`frontend/scripts/test_gemini_client.py`, lyrics-translation quota test.
- `pytest backend/tests --ignore=backend/tests/emulator` — all pass;
`pytest tests/unit` — 2021 passed.
- Verified `gemini-3.8-flash` is listed on the Developer API and
answered a 5-token test call with the key.
- `translate.py --dry-run` on `frontend/messages` and
`backend/translations` OK.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Updates**
* AI-powered lyric translation, custom lyrics, auto-correction, title
parsing, match judging, and related features now use the Gemini
Developer API.
* AI features use the configured Gemini API key. Quota, billing, or key
issues may show an unavailable message or use an existing fallback
instead of repeated attempts.
* **Documentation**
* Updated product, API, setup, and recovery guidance to reflect Gemini
API key configuration and usage.
* Updated guidance for handling AI service availability and quota
issues.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 21:58:53 -04:00

339 lines
12 KiB
Python

#!/usr/bin/env python3
"""
Measure the real token usage and cost of the AI lyrics auto-correct feature
against production jobs.
For each job it loads the stored corrections (the same transcription +
reference lyrics the review UI sends to ``POST /{job_id}/auto-correct``),
rebuilds the exact prompts, calls each configured model once, and reports
input/output/thinking tokens and estimated USD per model — plus a per-track
and aggregate summary. This is the immediate ground-truth read that the
production usage logging (``auto-correct usage`` / ``auto-correct cost`` lines)
accumulates over time.
Usage:
# Auto-pick the N most recent complete jobs that have corrections
python scripts/measure_autocorrect_cost.py --limit 3
# Measure specific jobs
python scripts/measure_autocorrect_cost.py --jobs abc123 def456
# Single-model instead of the multi-model compare config
python scripts/measure_autocorrect_cost.py --limit 3 --models claude-opus-5-5
Requires:
- ANTHROPIC_API_KEY in the environment (for claude-* models)
- Gemini Developer API key for gemini-* models: GEMINI_API_KEY, else read
from Secret Manager `gemini-api-key` via your gcloud login (in-process only)
- GCP creds for Firestore/GCS:
gcloud auth application-default login (or GOOGLE_APPLICATION_CREDENTIALS)
- GOOGLE_CLOUD_PROJECT=nomadkaraoke (set automatically below if unset)
NOTE: every run makes real, billed model calls. Gemini USD figures use the
ESTIMATED rates in backend/services/auto_correct/pricing.py — token counts are
exact; verify the Gemini rate before trusting its dollar value.
"""
from __future__ import annotations
import argparse
import json
import logging
import os
import sys
import time
from typing import Optional
os.environ.setdefault("GOOGLE_CLOUD_PROJECT", "nomadkaraoke")
# Running "python scripts/foo.py" puts scripts/ on sys.path, not the repo root,
# so the backend.* imports below would fail. Add the repo root explicitly.
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from google.cloud import firestore, storage # noqa: E402
logging.basicConfig(level=logging.WARNING, format="%(levelname)s %(message)s")
logger = logging.getLogger("measure_autocorrect_cost")
logger.setLevel(logging.INFO)
PROJECT = "nomadkaraoke"
GCS_BUCKET = "karaoke-gen-storage-nomadkaraoke"
DEFAULT_MODELS = ["claude-opus-5-5", "gemini-3.8-flash"]
def _ensure_gemini_key() -> None:
"""Populate GEMINI_API_KEY (this process only) from Secret Manager via gcloud."""
if os.environ.get("GEMINI_API_KEY"):
return
repo_root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
sys.path.insert(0, os.path.join(repo_root, "frontend", "scripts"))
from gemini_client import get_api_key # noqa: E402 - scripts helper (gcloud fallback)
os.environ["GEMINI_API_KEY"] = get_api_key()
# ---------------------------------------------------------------------------
# Job + corrections loading
# ---------------------------------------------------------------------------
def _corrections_path(job_data: dict, job_id: str) -> Optional[str]:
"""Mirror review.py: prefer the latest review edits, then the original."""
lyrics = (job_data.get("file_urls") or {}).get("lyrics") or {}
return (
lyrics.get("corrections_updated")
or lyrics.get("corrections")
or f"jobs/{job_id}/lyrics/corrections.json"
)
def fetch_recent_jobs(db: firestore.Client, limit: int) -> list[dict]:
query = (
db.collection("jobs")
.where("status", "==", "complete")
.order_by("created_at", direction=firestore.Query.DESCENDING)
.limit(limit * 4)
)
jobs = []
for doc in query.stream():
data = doc.to_dict()
data["_job_id"] = doc.id
lyrics = (data.get("file_urls") or {}).get("lyrics") or {}
if lyrics.get("corrections") or lyrics.get("corrections_updated"):
jobs.append(data)
if len(jobs) >= limit:
break
return jobs
def fetch_job(db: firestore.Client, job_id: str) -> Optional[dict]:
doc = db.collection("jobs").document(job_id).get()
if not doc.exists:
logger.warning("job %s not found in Firestore", job_id)
return None
data = doc.to_dict()
data["_job_id"] = job_id
return data
def download_corrections(
gcs: storage.Client, path: str
) -> Optional[dict]:
if path.startswith("gs://"):
bucket_name, _, blob_path = path[5:].partition("/")
else:
bucket_name, blob_path = GCS_BUCKET, path
try:
blob = gcs.bucket(bucket_name).blob(blob_path)
if not blob.exists():
return None
return json.loads(blob.download_as_bytes())
except Exception as exc: # noqa: BLE001
logger.warning("failed to read %s: %s", path, exc)
return None
def build_inputs(corrections: dict) -> Optional[tuple[list[dict], dict]]:
"""Return (segments, reference_lyrics) in the shape suggest() expects."""
segments = corrections.get("corrected_segments") or corrections.get("segments") or []
reference_lyrics = corrections.get("reference_lyrics") or {}
if not segments:
logger.warning("no segments in corrections")
return None
if not reference_lyrics:
logger.warning("no reference_lyrics in corrections — auto-correct needs >=1 source")
return None
# The service requires segment + word ids; older jobs may lack them.
for seg in segments:
if not seg.get("id") or any(not w.get("id") for w in (seg.get("words") or [])):
logger.warning("segments missing word/segment ids — skipping job")
return None
return segments, reference_lyrics
# ---------------------------------------------------------------------------
# Measurement
# ---------------------------------------------------------------------------
def measure_job(service, job: dict, gcs, models: list[str]) -> Optional[dict]:
from backend.services.auto_correct.pricing import estimate_cost_usd
from backend.services.auto_correct.prompts import (
build_system_prompt,
build_user_prompt,
)
from backend.services.auto_correct.settings import AutoCorrectSettings
job_id = job["_job_id"]
artist = job.get("artist")
title = job.get("title")
path = _corrections_path(job, job_id)
corrections = download_corrections(gcs, path)
if not corrections:
logger.warning("[%s] no corrections at %s — skipping", job_id, path)
return None
built = build_inputs(corrections)
if not built:
return None
segments, reference_lyrics = built
settings = AutoCorrectSettings()
system_prompt = build_system_prompt(settings)
user_prompt = build_user_prompt(
segments=segments,
reference_lyrics=reference_lyrics,
artist=artist,
title=title,
)
word_count = sum(len(seg.get("words") or []) for seg in segments)
logger.info(
"[%s] %s - %s | %d segments, %d words, %d ref sources",
job_id, artist or "?", title or "?", len(segments), word_count,
len(reference_lyrics),
)
per_model = []
for model in models:
t0 = time.time()
try:
raw, usage = service._call_model(
model, system_prompt, user_prompt, job_id=job_id
)
except Exception as exc: # noqa: BLE001
logger.warning("[%s] model %s failed: %s", job_id, model, exc)
per_model.append({"model": model, "error": str(exc)})
continue
elapsed = time.time() - t0
cost = estimate_cost_usd(model, usage)
n_sugg = len(raw.get("suggestions") or []) if isinstance(raw, dict) else 0
per_model.append({
"model": model,
"usage": usage,
"cost": cost,
"suggestions": n_sugg,
"elapsed": elapsed,
})
if usage:
logger.info(
" %-26s in=%-6d out=%-6d think=%-6s cost=%s (%.1fs, %d suggestions)",
model,
usage.get("input_tokens", 0),
usage.get("output_tokens", 0),
"na" if usage.get("thinking_tokens") is None else usage["thinking_tokens"],
"?" if cost is None else f"${cost:.4f}",
elapsed, n_sugg,
)
else:
logger.info(" %-26s usage unavailable", model)
return {
"job_id": job_id,
"artist": artist,
"title": title,
"word_count": word_count,
"per_model": per_model,
}
def print_report(results: list[dict], models: list[str]) -> None:
print("\n" + "=" * 78)
print("AUTO-CORRECT COST MEASUREMENT")
print("=" * 78)
track_totals = []
model_cost_sums = {m: 0.0 for m in models}
model_cost_known = {m: False for m in models}
for r in results:
track_total = 0.0
any_cost = False
print(f"\n{r['artist'] or '?'} - {r['title'] or '?'} "
f"({r['job_id']}, {r['word_count']} words)")
for pm in r["per_model"]:
if "error" in pm:
print(f" {pm['model']:<26s} ERROR: {pm['error']}")
continue
u = pm["usage"] or {}
cost = pm["cost"]
if cost is not None:
track_total += cost
any_cost = True
model_cost_sums[pm["model"]] += cost
model_cost_known[pm["model"]] = True
print(
f" {pm['model']:<26s} "
f"in={u.get('input_tokens', 0):<6d} "
f"out={u.get('output_tokens', 0):<6d} "
f"think={('na' if u.get('thinking_tokens') is None else u['thinking_tokens'])!s:<6s} "
f"cost={'?' if cost is None else f'${cost:.4f}':<9s} "
f"({pm['suggestions']} suggestions)"
)
print(f" {'TRACK TOTAL':<26s} {'':<26s}"
f"{'$%.4f' % track_total if any_cost else 'unknown'}")
if any_cost:
track_totals.append(track_total)
print("\n" + "-" * 78)
print("AGGREGATE")
print("-" * 78)
for m in models:
if model_cost_known[m]:
avg = model_cost_sums[m] / len(results) if results else 0.0
print(f" {m:<26s} avg ${avg:.4f}/track")
if track_totals:
avg = sum(track_totals) / len(track_totals)
hi = max(track_totals)
print(f"\n Avg total cost/track: ${avg:.4f}")
print(f" Max total cost/track: ${hi:.4f}")
print(f" Jobs measured: {len(track_totals)}")
print(f"\n Projected at 30 jobs/day: ${avg * 30:.2f}/day (~${avg * 30 * 30:.0f}/mo)")
print(f" Projected at 100 jobs/day: ${avg * 100:.2f}/day (~${avg * 100 * 30:.0f}/mo)")
print("\nNOTE: Gemini USD uses ESTIMATED rates (see pricing.py). Token counts are exact.")
print("=" * 78)
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--jobs", nargs="+", help="Specific job IDs to measure")
parser.add_argument("--limit", type=int, default=3,
help="Auto-pick N recent complete jobs (ignored if --jobs given)")
parser.add_argument("--models", nargs="+", default=DEFAULT_MODELS,
help=f"Models to measure (default: {' '.join(DEFAULT_MODELS)})")
args = parser.parse_args()
if any(m.startswith("gemini") for m in args.models):
_ensure_gemini_key()
if any(m.startswith("claude") for m in args.models) and not os.environ.get("ANTHROPIC_API_KEY"):
logger.error("ANTHROPIC_API_KEY is not set — required for claude-* models")
return 2
from backend.services.auto_correct.service import AutoCorrectService
db = firestore.Client(project=PROJECT)
gcs = storage.Client(project=PROJECT)
service = AutoCorrectService()
if args.jobs:
jobs = [j for j in (fetch_job(db, jid) for jid in args.jobs) if j]
else:
jobs = fetch_recent_jobs(db, args.limit)
logger.info("auto-picked %d recent complete jobs with corrections", len(jobs))
if not jobs:
logger.error("no jobs to measure")
return 1
results = []
for job in jobs:
r = measure_job(service, job, gcs, args.models)
if r:
results.append(r)
if not results:
logger.error("no jobs produced measurable results")
return 1
print_report(results, args.models)
return 0
if __name__ == "__main__":
sys.exit(main())