mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary Moves **every** Gemini call in gen off **Vertex AI** (billed to the `nomadkaraoke` GCP project, ~$85/mo) onto the **Gemini Developer API** using the `gemini-api-key` Secret Manager secret (an AI Studio key billed outside this project). Once deployed and verified, `aiplatform.googleapis.com` / `generativelanguage.googleapis.com` get disabled in nomadkaraoke. ### Key injection - `backend/services/gemini_client.get_genai_client(timeout_ms)` — key from `GEMINI_API_KEY`, else Secret Manager `gemini-api-key` via `settings.get_secret` (runtime SA already has project-level `secretAccessor`). No `project`/`location`. - Service: CI `--set-secrets ... GEMINI_API_KEY=gemini-api-key:latest`. - Jobs: Pulumi secret env on `lyrics-transcription-job`, `video-encoding-job` (renders + translations), `nomad-error-monitor` — **already applied** (targeted `pulumi up`, only that env added). Other jobs fall back to Secret Manager. - Local scripts (`frontend/scripts/translate.py`, `scripts/measure_autocorrect_cost.py`, `scripts/test_agentic_correction.py`): env, else `gcloud secrets versions access` (in-process only). - **Rotation:** add a new secret version; new instances / job executions pick it up. No code change. ### Degradation (quota / billing / invalid key; per-minute 429s stay retryable) | Caller | Behaviour | |---|---| | auto-correct | non-retryable 503 → compare mode drops the Gemini leg, keeps Opus | | match-judge / free-text / parse-titles / tenant bulk | existing fallbacks (no suggestion / blanks / regex) | | credit eval | `pending_review` (fail-closed, unchanged) | | lyrics translation | no retry; render proceeds without translation, status recorded | | custom lyrics | clean 503 "temporarily unavailable…" (+ 5-min timeout so it can't hang) | | error-monitor LLM | no retry; alerts sent without AI summary | | translate.py | exit 2 "Gemini quota/credit exhausted — top up AI Studio or rotate gemini-api-key" | `note_gemini_failure()` also sends one Discord ops alert per 6h (Firestore-throttled across instances via new `ops_alerts.send_throttled_ops_alert`). ### Infra - Pulumi: removes `roles/aiplatform.user` from `karaoke-backend` — **will be applied after this deploys and Vertex traffic is confirmed at zero**. - Agentic correction default model `gemini/gemini-3.8-flash` (`vertexai/` prefix still accepted, routed to the Developer API). - Version 0.261.1 → 0.262.0. Docs: CLAUDE.md, LESSONS-LEARNED, README, ARCHITECTURE, API, etc. ## Testing - New: `backend/tests/services/test_backend_gemini_client.py` (classifier, key resolution, no-Vertex client kwargs, alert throttle, per-caller degradation), `tests/unit/lyrics_transcriber/test_agentic_gemini_provider.py`, `frontend/scripts/test_gemini_client.py`, lyrics-translation quota test. - `pytest backend/tests --ignore=backend/tests/emulator` — all pass; `pytest tests/unit` — 2021 passed. - Verified `gemini-3.8-flash` is listed on the Developer API and answered a 5-token test call with the key. - `translate.py --dry-run` on `frontend/messages` and `backend/translations` OK. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Updates** * AI-powered lyric translation, custom lyrics, auto-correction, title parsing, match judging, and related features now use the Gemini Developer API. * AI features use the configured Gemini API key. Quota, billing, or key issues may show an unavailable message or use an existing fallback instead of repeated attempts. * **Documentation** * Updated product, API, setup, and recovery guidance to reflect Gemini API key configuration and usage. * Updated guidance for handling AI service availability and quota issues. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
270 lines
9.5 KiB
Python
Executable File
270 lines
9.5 KiB
Python
Executable File
#!/usr/bin/env python3
|
|
"""
|
|
Test script for the agentic correction workflow.
|
|
|
|
This script allows testing the agentic correction locally without deploying
|
|
to Cloud Run or running a full karaoke job.
|
|
|
|
Usage:
|
|
# Test with the default Gemini model (Developer API; GEMINI_API_KEY, else
|
|
# read from Secret Manager `gemini-api-key` via your gcloud login):
|
|
python scripts/test_agentic_correction.py
|
|
|
|
# Test with a specific model:
|
|
AGENTIC_AI_MODEL=openai/gpt-4o python scripts/test_agentic_correction.py
|
|
AGENTIC_AI_MODEL=anthropic/claude-3-5-sonnet-20241022 python scripts/test_agentic_correction.py
|
|
|
|
# Test with local Ollama:
|
|
PRIVACY_MODE=1 python scripts/test_agentic_correction.py
|
|
|
|
Environment variables:
|
|
GEMINI_API_KEY: Gemini Developer API key (else fetched via gcloud)
|
|
AGENTIC_AI_MODEL: Override the model (e.g., openai/gpt-4o)
|
|
PRIVACY_MODE: Set to 1 to use local Ollama
|
|
LANGFUSE_PUBLIC_KEY: Langfuse public key for tracing
|
|
LANGFUSE_SECRET_KEY: Langfuse secret key for tracing
|
|
"""
|
|
|
|
import os
|
|
import sys
|
|
import json
|
|
import logging
|
|
from typing import List, Dict, Any
|
|
|
|
# Add the project to the path
|
|
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
|
|
|
# Configure logging
|
|
logging.basicConfig(
|
|
level=logging.INFO,
|
|
format="%(asctime)s %(levelname)s %(name)s: %(message)s",
|
|
datefmt="%H:%M:%S"
|
|
)
|
|
logger = logging.getLogger(__name__)
|
|
|
|
# Sample test cases representing different gap categories
|
|
TEST_CASES = [
|
|
{
|
|
"name": "Sound-alike error (wurld -> world)",
|
|
"gap_id": "test_gap_1",
|
|
"gap_words": [
|
|
{"id": "w1", "text": "wurld", "start_time": 0.5, "end_time": 1.0}
|
|
],
|
|
"preceding_words": "hello",
|
|
"following_words": "this is a test",
|
|
"reference_contexts": {
|
|
"genius": "hello world this is a test",
|
|
"lrclib": "Hello world, this is a test"
|
|
},
|
|
"artist": "Test Artist",
|
|
"title": "Test Song"
|
|
},
|
|
{
|
|
"name": "Punctuation difference",
|
|
"gap_id": "test_gap_2",
|
|
"gap_words": [
|
|
{"id": "w2", "text": "Hello", "start_time": 0.0, "end_time": 0.5}
|
|
],
|
|
"preceding_words": "",
|
|
"following_words": "is it me you're looking for",
|
|
"reference_contexts": {
|
|
"genius": "Hello, is it me you're looking for?"
|
|
},
|
|
"artist": "Lionel Richie",
|
|
"title": "Hello"
|
|
},
|
|
{
|
|
"name": "Extra filler word",
|
|
"gap_id": "test_gap_3",
|
|
"gap_words": [
|
|
{"id": "w3", "text": "And", "start_time": 2.0, "end_time": 2.2},
|
|
{"id": "w4", "text": "I", "start_time": 2.2, "end_time": 2.4},
|
|
{"id": "w5", "text": "love", "start_time": 2.4, "end_time": 2.6},
|
|
{"id": "w6", "text": "you", "start_time": 2.6, "end_time": 2.8}
|
|
],
|
|
"preceding_words": "baby",
|
|
"following_words": "so much",
|
|
"reference_contexts": {
|
|
"genius": "baby I love you so much"
|
|
},
|
|
"artist": "Test Artist",
|
|
"title": "Love Song"
|
|
},
|
|
{
|
|
"name": "Complex multi-word error",
|
|
"gap_id": "test_gap_4",
|
|
"gap_words": [
|
|
{"id": "w7", "text": "gonna", "start_time": 3.0, "end_time": 3.2},
|
|
{"id": "w8", "text": "give", "start_time": 3.2, "end_time": 3.4},
|
|
{"id": "w9", "text": "you", "start_time": 3.4, "end_time": 3.6},
|
|
{"id": "w10", "text": "up", "start_time": 3.6, "end_time": 3.8}
|
|
],
|
|
"preceding_words": "never",
|
|
"following_words": "never gonna let you down",
|
|
"reference_contexts": {
|
|
"genius": "Never gonna give you up, never gonna let you down"
|
|
},
|
|
"artist": "Rick Astley",
|
|
"title": "Never Gonna Give You Up"
|
|
}
|
|
]
|
|
|
|
|
|
def run_single_test(test_case: Dict[str, Any], agent) -> Dict[str, Any]:
|
|
"""Run a single test case and return results."""
|
|
logger.info(f"\n{'='*60}")
|
|
logger.info(f"Test: {test_case['name']}")
|
|
logger.info(f"Gap words: {[w['text'] for w in test_case['gap_words']]}")
|
|
logger.info(f"Reference: {list(test_case['reference_contexts'].values())[0][:50]}...")
|
|
logger.info(f"{'='*60}")
|
|
|
|
try:
|
|
proposals = agent.propose_for_gap(
|
|
gap_id=test_case["gap_id"],
|
|
gap_words=test_case["gap_words"],
|
|
preceding_words=test_case["preceding_words"],
|
|
following_words=test_case["following_words"],
|
|
reference_contexts=test_case["reference_contexts"],
|
|
artist=test_case.get("artist"),
|
|
title=test_case.get("title")
|
|
)
|
|
|
|
result = {
|
|
"test_name": test_case["name"],
|
|
"success": True,
|
|
"proposals": []
|
|
}
|
|
|
|
if not proposals:
|
|
logger.info("No proposals returned (gap may be correct)")
|
|
result["proposals"] = []
|
|
else:
|
|
for p in proposals:
|
|
proposal_dict = {
|
|
"action": p.action,
|
|
"word_id": p.word_id,
|
|
"word_ids": p.word_ids,
|
|
"replacement_text": p.replacement_text,
|
|
"confidence": p.confidence,
|
|
"reason": p.reason,
|
|
"gap_category": str(p.gap_category) if p.gap_category else None,
|
|
"requires_human_review": p.requires_human_review
|
|
}
|
|
result["proposals"].append(proposal_dict)
|
|
|
|
logger.info(f"\nProposal:")
|
|
logger.info(f" Action: {p.action}")
|
|
logger.info(f" Word ID(s): {p.word_id or p.word_ids}")
|
|
if p.replacement_text:
|
|
logger.info(f" Replacement: {p.replacement_text}")
|
|
logger.info(f" Confidence: {p.confidence:.2f}")
|
|
logger.info(f" Category: {p.gap_category}")
|
|
logger.info(f" Reason: {p.reason}")
|
|
if p.requires_human_review:
|
|
logger.info(f" ⚠️ Requires human review")
|
|
|
|
return result
|
|
|
|
except Exception as e:
|
|
logger.error(f"Test failed with error: {e}")
|
|
import traceback
|
|
traceback.print_exc()
|
|
return {
|
|
"test_name": test_case["name"],
|
|
"success": False,
|
|
"error": str(e)
|
|
}
|
|
|
|
|
|
def main():
|
|
"""Main entry point."""
|
|
# Check environment
|
|
project_id = os.getenv("GOOGLE_CLOUD_PROJECT") or os.getenv("GCP_PROJECT_ID")
|
|
location = os.getenv("GCP_LOCATION", "global")
|
|
model_override = os.getenv("AGENTIC_AI_MODEL")
|
|
privacy_mode = os.getenv("PRIVACY_MODE", "").lower() in ("1", "true", "yes")
|
|
|
|
logger.info("=" * 60)
|
|
logger.info("AGENTIC CORRECTION TEST SCRIPT")
|
|
logger.info("=" * 60)
|
|
logger.info(f"GCP Project: {project_id or 'not set'}")
|
|
logger.info(f"GCP Location: {location}")
|
|
logger.info(f"Model Override: {model_override or 'none (using default)'}")
|
|
logger.info(f"Privacy Mode: {privacy_mode}")
|
|
logger.info(f"Langfuse: {'enabled' if os.getenv('LANGFUSE_PUBLIC_KEY') else 'disabled'}")
|
|
|
|
# Import after path setup
|
|
try:
|
|
from karaoke_gen.lyrics_transcriber.correction.agentic.agent import AgenticCorrector
|
|
from karaoke_gen.lyrics_transcriber.correction.agentic.router import ModelRouter
|
|
from karaoke_gen.lyrics_transcriber.correction.agentic.providers.config import ProviderConfig
|
|
except ImportError as e:
|
|
logger.error(f"Failed to import agentic correction modules: {e}")
|
|
logger.error("Make sure you're running from the karaoke-gen directory")
|
|
sys.exit(1)
|
|
|
|
# Determine which model to use
|
|
config = ProviderConfig.from_env()
|
|
router = ModelRouter(config)
|
|
model = router.choose_model("unknown", 0.5)
|
|
logger.info(f"Selected model: {model}")
|
|
|
|
# Gemini Developer API key (this process only; never written to disk)
|
|
if model.split("/", 1)[0] in ("gemini", "google", "vertexai") and not (
|
|
os.environ.get("GEMINI_API_KEY") or os.environ.get("GOOGLE_API_KEY")
|
|
):
|
|
sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "frontend", "scripts"))
|
|
from gemini_client import get_api_key
|
|
|
|
os.environ["GEMINI_API_KEY"] = get_api_key()
|
|
config = ProviderConfig.from_env()
|
|
|
|
# Create the agent
|
|
logger.info(f"\nCreating AgenticCorrector with model: {model}")
|
|
try:
|
|
agent = AgenticCorrector.from_model(
|
|
model=model,
|
|
session_id="test_session_local",
|
|
cache_dir=os.path.expanduser("~/lyrics-transcriber-cache")
|
|
)
|
|
logger.info("Agent created successfully!")
|
|
except Exception as e:
|
|
logger.error(f"Failed to create agent: {e}")
|
|
import traceback
|
|
traceback.print_exc()
|
|
sys.exit(1)
|
|
|
|
# Run test cases
|
|
results = []
|
|
for test_case in TEST_CASES:
|
|
result = run_single_test(test_case, agent)
|
|
results.append(result)
|
|
|
|
# Summary
|
|
logger.info("\n" + "=" * 60)
|
|
logger.info("TEST SUMMARY")
|
|
logger.info("=" * 60)
|
|
|
|
passed = sum(1 for r in results if r["success"])
|
|
failed = len(results) - passed
|
|
|
|
for r in results:
|
|
status = "✅ PASS" if r["success"] else "❌ FAIL"
|
|
logger.info(f"{status}: {r['test_name']}")
|
|
if not r["success"]:
|
|
logger.info(f" Error: {r.get('error', 'unknown')}")
|
|
|
|
logger.info(f"\nResults: {passed}/{len(results)} passed")
|
|
|
|
# Output JSON results
|
|
output_file = "agentic_test_results.json"
|
|
with open(output_file, "w") as f:
|
|
json.dump(results, f, indent=2)
|
|
logger.info(f"\nDetailed results saved to: {output_file}")
|
|
|
|
return 0 if failed == 0 else 1
|
|
|
|
|
|
if __name__ == "__main__":
|
|
sys.exit(main())
|