mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
@coderabbitai ignore ## Why LRCLIB now applies stricter rate limits to unregistered clients (https://lrclib.net/docs#implementation-requirements-rate-limiting), and Cloud Run shares egress IPs. On 10-19, prod jobs got intermittent 503s, and job 80e7ffdf lost its LRCLIB lyrics. LRCLIB is our only free source; it supplied lyrics for 165 of 220 jobs in 30 days. Andrew registered a server-side app. ## What - **`karaoke_gen/lyrics_transcriber/lyrics/lrclib_client.py`** (new): - Sends `Lrclib-App-Id` + `Lrclib-Api-Key` from `LRCLIB_APP_ID` / `LRCLIB_SECRET_KEY`. - Applies LRCLIB's rules: every 429 sets a **process-wide cooldown** of `Retry-After` plus up to 30% jitter. The wait is never shortened, and overlapping 429s keep the latest expiry. Nothing is sent until the cooldown has passed. It waits and retries once only when the wait is ≤30s. - Retries 5xx after 1s/2s, honouring a 5xx `Retry-After`. - A **401 on our credentials** logs a WARNING and drops to anonymous access for the process, so a revoked key degrades instead of losing LRCLIB. - `LRCLIBProvider` (both `get` and `search`) uses it. The `quick_version` renderer uses it with `retry=False`, so the fast path gets the auth and the cooldown but never blocks. - **Infra:** - Pulumi secrets `lrclib-app-id` / `lrclib-api-key`. - Env vars on the `lyrics-transcription-job`. - The backend service deploy's `--set-secrets` (`ci.yml`). The karaoke-backend SA already has project-wide `secretAccessor`. ## Deploy order (do NOT merge before steps 1–2) 1. `pulumi up` to create the two secrets and the job env. Needs `/gcp-admin`. 2. Add the secret versions from Andrew's `.envrc` values. 3. Merge. The CI backend deploy references the secrets. ## Testing - New `tests/unit/test_lrclib_client.py` (14 tests): headers; anonymous use; Retry-After with jitter; a long Retry-After is not shortened and blocks further calls; no-retry mode; a second 429; 5xx backoff; 5xx Retry-After; 401 fallback; overlap merge (mutation-checked); provider routes through the client. - 164 passed across the client, `quick_version` and `search_lyrics` tests. - **Live** with the real app key: Dar Williams - Iowa and Wolfmother - Where Eagles Have Been were found. One 503 was recovered by the retry. - Local CodeRabbit, 3 cycles. I fixed the Retry-After cap, the cooldown after a final 429, the `quick_version` 401 path, the overlap merge and 5xx Retry-After. I skipped one false positive about the tests' fake clock (the retry path sleeps, then sends without re-checking the clock). 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Jw21syo4HizJJjYw2mg2Vp --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
114 lines
4.0 KiB
Python
114 lines
4.0 KiB
Python
"""
|
|
Secret Manager resources.
|
|
|
|
Manages all secrets for the application. Secret values are added manually
|
|
via gcloud or the Cloud Console after resources are created.
|
|
"""
|
|
|
|
from pulumi_gcp import secretmanager
|
|
|
|
|
|
def create_secrets() -> dict[str, secretmanager.Secret]:
|
|
"""
|
|
Create all Secret Manager secrets.
|
|
|
|
Returns:
|
|
dict: Dictionary mapping secret names to Secret resources.
|
|
"""
|
|
# Define all secret names
|
|
# Note: Private tracker secrets (red-*, ops-*) are managed by flacfetch repo
|
|
secret_names = [
|
|
# Audio processing
|
|
"audioshake-api-key",
|
|
"genius-api-key",
|
|
# LRCLIB registered server-side app (higher rate limits; lrclib.net developer portal)
|
|
"lrclib-app-id",
|
|
"lrclib-api-key",
|
|
"audio-separator-api-url",
|
|
|
|
# Payment and email
|
|
"stripe-secret-key",
|
|
"stripe-webhook-secret",
|
|
"postmark-server-token",
|
|
|
|
# Observability (Langfuse for LLM tracing)
|
|
"langfuse-public-key",
|
|
"langfuse-secret-key",
|
|
"langfuse-host",
|
|
|
|
# AI auto-correct suggestions (Claude models via Anthropic API)
|
|
"anthropic-api-key",
|
|
|
|
# Service authentication
|
|
"flacfetch-api-key",
|
|
"flacfetch-api-url",
|
|
"encoding-worker-api-key",
|
|
# Capacity-fallback VMs config (JSON list of {vm,zone,ip,machine_type?}).
|
|
# machine_type (e.g. "c4-highcpu-32") drives the shared speed-rank
|
|
# ordering; optional for legacy entries (inferred from the vm name).
|
|
# Stored as a secret because --set-env-vars on Cloud Run is
|
|
# comma-delimited and the JSON value contains commas; the value itself is
|
|
# non-secret (just public IPs of fallback VMs).
|
|
"encoding-worker-fallback-vms",
|
|
|
|
# Notifications
|
|
"pushbullet-api-key",
|
|
|
|
# Internal service URLs
|
|
"gdrive-validator-url",
|
|
|
|
# Web Push (VAPID keys for mobile push notifications)
|
|
"vapid-public-key",
|
|
"vapid-private-key",
|
|
|
|
# GitHub
|
|
"github-runner-pat",
|
|
"github-webhook-secret", # For runner manager webhook verification
|
|
# DR: PAT (repo + read:org) the nightly backup uses to clone every repo
|
|
# under github.com/nomadkaraoke + github.com/beveradb into git bundles on
|
|
# S3, so code survives loss of GitHub access. Value added manually.
|
|
"github-backup-token",
|
|
|
|
# E2E Testing
|
|
"e2e-bypass-key",
|
|
|
|
# KaraokeNerds API
|
|
"karaokenerds-api-key",
|
|
|
|
# Notifications (Discord)
|
|
"discord-alert-webhook",
|
|
|
|
# Divebar on-demand refresh trigger (shared bearer token: kjbox's
|
|
# "Refresh catalog" button -> divebar-lookup `refresh` action). Gates
|
|
# an otherwise-public endpoint that re-runs the mirror/sync/xref
|
|
# scheduler jobs on demand, so it's anti-abuse, not data-sensitive.
|
|
"divebar-refresh-token",
|
|
|
|
# Cloudflare edge origin lock: shared secret Cloudflare injects as the
|
|
# X-Edge-Auth header on proxied requests; the backend edge-auth
|
|
# middleware rejects public requests that lack it (blocks direct-to-
|
|
# origin bypass of the WAF). Value added manually; also provided to
|
|
# Pulumi as `edge:originSecret` for the Cloudflare header transform.
|
|
# See modules/edge_security.py.
|
|
"edge-origin-secret",
|
|
# Cloudflare API token for the admin tenant console: adds/removes each
|
|
# tenant's `<id>.nomadkaraoke.com` Pages custom domain (project
|
|
# karaoke-gen-tenant) + proxied CNAME. Account-owned token scoped to
|
|
# Pages Read/Write (account) + DNS Read/Write (nomadkaraoke.com zone
|
|
# only). Value added manually. See services/tenant_domain_service.py.
|
|
"cloudflare-tenant-domains-token",
|
|
]
|
|
|
|
secrets = {}
|
|
|
|
for secret_id in secret_names:
|
|
secrets[secret_id] = secretmanager.Secret(
|
|
secret_id,
|
|
secret_id=secret_id,
|
|
replication=secretmanager.SecretReplicationArgs(
|
|
auto=secretmanager.SecretReplicationAutoArgs(),
|
|
),
|
|
)
|
|
|
|
return secrets
|