mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
- Add archive doc for LangFuse integration work - Update README with recent changes - Add lesson learned about GCP org policy for SA keys 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
3.2 KiB
3.2 KiB
LangFuse Prompt Management Integration
Date: 2025-12-30 PR: https://github.com/nomadkaraoke/karaoke-gen/pull/140
Summary
Replaced hardcoded LLM prompts in the agentic correction workflow with LangFuse-managed prompts, enabling prompt iteration without redeploying the backend.
What Changed
New Components
-
LangFusePromptService (
lyrics_transcriber/.../prompts/langfuse_prompts.py)- Fetches prompts from LangFuse with hardcoded fallback for local development
- Fetches few-shot examples from LangFuse datasets
- Singleton pattern with fail-fast when LangFuse is configured
-
Migration Script (
scripts/migrate-prompts-to-langfuse.py)- Uploads
gap-classifierprompt template to LangFuse - Uploads 9 few-shot examples as
gap-classifier-examplesdataset - Supports
--dry-run,--force,--verify-onlyflags
- Uploads
-
LangFuse Vertex AI Service Account (Pulumi)
langfuse-vertexai@nomadkaraoke.iam.gserviceaccount.com- Has
roles/aiplatform.userfor calling Gemini models - JSON key generated for LangFuse to use GCP credits
Modified Components
langfuse_integration.py- Addedfetch_prompt()andfetch_dataset()with label fallbackclassifier.py- Now uses LangFusePromptService, hardcoded function renamed tobuild_classification_prompt_hardcoded- Default LangFuse host changed to US region (
https://us.cloud.langfuse.com)
Deleted Components
handlers/llm.py- Legacy LLM handler not being usedhandlers/llm_providers.py- Legacy provider implementations
Configuration
LangFuse Credentials (GCP Secret Manager)
langfuse-public-key- Project: nomadkaraokelangfuse-secret-key- Project: nomadkaraoke
Environment Variables
LANGFUSE_PUBLIC_KEY=pk-lf-...
LANGFUSE_SECRET_KEY=sk-lf-...
LANGFUSE_HOST=https://us.cloud.langfuse.com # Optional, this is the default
LangFuse Resources
- Prompt:
gap-classifier(promote to "production" label for default fetch) - Dataset:
gap-classifier-examples(9 few-shot examples) - Project: nomadkaraoke at https://us.cloud.langfuse.com
How Prompt Fetching Works
- Try to fetch prompt with "production" label (default)
- If not found, fall back to version 1 (for newly created prompts)
- If LangFuse not configured, use hardcoded fallback
This allows:
- New prompts to work immediately after upload
- Production prompts to be explicitly promoted
- Local development without LangFuse credentials
GCP Organization Policy Issue
Creating service account keys was blocked by organization policy iam.disableServiceAccountKeyCreation. Solution:
- Temporarily disable policy at organization level
- Create key via
gcloud iam service-accounts keys create - Re-enable policy immediately
- Delete local key file after pasting into LangFuse
This is a one-time operation. The key is now stored in LangFuse, not in any repo or GCP secret.
Testing
All 676 unit tests pass. The prompt service has dedicated tests in test_langfuse_prompts.py that mock the LangFuse client.
Future Work
- Add more prompts for other agentic workflows
- Set up LangFuse evaluations for prompt quality
- Consider A/B testing different prompt versions