fix: LRCLIB as a registered server-side app; honour Retry-After, retry 5xx (v0.279.2) (#1200)

@coderabbitai ignore

## Why
LRCLIB now applies stricter rate limits to unregistered clients
(https://lrclib.net/docs#implementation-requirements-rate-limiting), and
Cloud Run shares egress IPs. On 10-19, prod jobs got intermittent 503s,
and job 80e7ffdf lost its LRCLIB lyrics. LRCLIB is our only free source;
it supplied lyrics for 165 of 220 jobs in 30 days. Andrew registered a
server-side app.

## What
- **`karaoke_gen/lyrics_transcriber/lyrics/lrclib_client.py`** (new):
- Sends `Lrclib-App-Id` + `Lrclib-Api-Key` from `LRCLIB_APP_ID` /
`LRCLIB_SECRET_KEY`.
- Applies LRCLIB's rules: every 429 sets a **process-wide cooldown** of
`Retry-After` plus up to 30% jitter. The wait is never shortened, and
overlapping 429s keep the latest expiry. Nothing is sent until the
cooldown has passed. It waits and retries once only when the wait is
≤30s.
  - Retries 5xx after 1s/2s, honouring a 5xx `Retry-After`.
- A **401 on our credentials** logs a WARNING and drops to anonymous
access for the process, so a revoked key degrades instead of losing
LRCLIB.
- `LRCLIBProvider` (both `get` and `search`) uses it. The
`quick_version` renderer uses it with `retry=False`, so the fast path
gets the auth and the cooldown but never blocks.
- **Infra:**
  - Pulumi secrets `lrclib-app-id` / `lrclib-api-key`.
  - Env vars on the `lyrics-transcription-job`.
- The backend service deploy's `--set-secrets` (`ci.yml`). The
karaoke-backend SA already has project-wide `secretAccessor`.

## Deploy order (do NOT merge before steps 1–2)
1. `pulumi up` to create the two secrets and the job env. Needs
`/gcp-admin`.
2. Add the secret versions from Andrew's `.envrc` values.
3. Merge. The CI backend deploy references the secrets.

## Testing
- New `tests/unit/test_lrclib_client.py` (14 tests): headers; anonymous
use; Retry-After with jitter; a long Retry-After is not shortened and
blocks further calls; no-retry mode; a second 429; 5xx backoff; 5xx
Retry-After; 401 fallback; overlap merge (mutation-checked); provider
routes through the client.
- 164 passed across the client, `quick_version` and `search_lyrics`
tests.
- **Live** with the real app key: Dar Williams - Iowa and Wolfmother -
Where Eagles Have Been were found. One 503 was recovered by the retry.
- Local CodeRabbit, 3 cycles. I fixed the Retry-After cap, the cooldown
after a final 429, the `quick_version` 401 path, the overlap merge and
5xx Retry-After. I skipped one false positive about the tests' fake
clock (the retry path sleeps, then sends without re-checking the clock).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Jw21syo4HizJJjYw2mg2Vp

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Andrew Beveridge ☄️andClaude Opus 5.5 authored and GitHub committed 2026-10-19 17:33:19 -04:00
1 parent 653801cd04
commit 4286d98dbe
8 files changed
+283 -15

No files matched your search

+1 -1
View File
@@ -2176,7 +2176,7 @@ jobs:
--min-instances 0 \
--clear-vpc-connector \
--set-env-vars "GOOGLE_CLOUD_PROJECT=${{ env.PROJECT_ID }},GCS_BUCKET_NAME=karaoke-gen-storage-${{ env.PROJECT_ID }},FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,KARAOKE_GEN_VERSION=$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,ENABLE_CLOUD_TASKS=true,CLOUD_RUN_SERVICE_URL=https://api.nomadkaraoke.com,GCP_REGION=us-central1,DEFAULT_ENABLE_YOUTUBE_UPLOAD=true,DEFAULT_BRAND_PREFIX=NOMAD,DEFAULT_PRIVATE_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-NonPublished,DEFAULT_PRIVATE_BRAND_PREFIX=NOMADNP,USE_GCE_ENCODING=true,USE_GCE_PREVIEW_ENCODING=true,ENABLE_PUSH_NOTIFICATIONS=true,USE_CLOUD_RUN_JOBS_FOR_VIDEO=true,USE_CLOUD_RUN_JOBS_FOR_RENDER=true,EMAIL_PROVIDER=postmark,AUTO_CORRECT_MODEL=claude-opus-5-5,AUTO_CORRECT_COMPARE_MODELS=claude-opus-5-5;gemini-3.8-flash,AUTO_CORRECT_PROACTIVE_ENABLED=true,AUTO_APPROVAL_BACKING_KEEP_ENABLED=true,COMMUNITY_DAILY_PICK_ENABLED=true,STORAGE_RETENTION_DRY_RUN=false,STORAGE_RETENTION_MAX_JOBS_PER_RUN=50,CONVERSION_EMAILS_ENABLED=true,REVIEW_LAST_CHANCE_ENABLED=true,CONVERSION_OFFER_PERCENT=25,EDGE_AUTH_MODE=enforce,COMMIT_SHA=${{ steps.build-meta.outputs.sha }},PR_NUMBER=${{ steps.build-meta.outputs.pr_number }},PR_TITLE=${{ steps.build-meta.outputs.pr_title }}" \
--set-secrets "ANTHROPIC_API_KEY=anthropic-api-key:latest,GEMINI_API_KEY=gemini-api-key:latest,AUDIOSHAKE_API_TOKEN=audioshake-api-key:latest,GENIUS_API_TOKEN=genius-api-key:latest,SPOTIFY_COOKIE_SP_DC=spotify-cookie:latest,RAPIDAPI_KEY=rapidapi-key:latest,DEFAULT_DISCORD_WEBHOOK_URL=discord-releases-webhook:latest,ADMIN_TOKENS=admin-tokens:latest,RED_API_KEY=red-api-key:latest,RED_API_URL=red-api-url:latest,OPS_API_KEY=ops-api-key:latest,OPS_API_URL=ops-api-url:latest,FLACFETCH_API_URL=flacfetch-api-url:latest,FLACFETCH_API_KEY=flacfetch-api-key:latest,POSTMARK_SERVER_TOKEN=postmark-server-token:latest,STRIPE_SECRET_KEY=stripe-secret-key:latest,STRIPE_WEBHOOK_SECRET=stripe-webhook-secret:latest,LANGFUSE_PUBLIC_KEY=langfuse-public-key:latest,LANGFUSE_SECRET_KEY=langfuse-secret-key:latest,LANGFUSE_HOST=langfuse-host:latest,ENCODING_WORKER_URL=encoding-worker-url:latest,ENCODING_WORKER_API_KEY=encoding-worker-api-key:latest,ENCODING_WORKER_FALLBACK_VMS=encoding-worker-fallback-vms:latest,VAPID_PUBLIC_KEY=vapid-public-key:latest,VAPID_PRIVATE_KEY=vapid-private-key:latest,E2E_BYPASS_KEY=e2e-bypass-key:latest,GDRIVE_VALIDATOR_URL=gdrive-validator-url:latest,EDGE_ORIGIN_SECRET=edge-origin-secret:latest,KARAOKEHUNT_FORWARDER_SECRET=karaokehunt-forwarder-secret:latest,KJBOX_PARTNER_SECRET=kjbox-partner-secret:latest"
--set-secrets "ANTHROPIC_API_KEY=anthropic-api-key:latest,GEMINI_API_KEY=gemini-api-key:latest,AUDIOSHAKE_API_TOKEN=audioshake-api-key:latest,GENIUS_API_TOKEN=genius-api-key:latest,SPOTIFY_COOKIE_SP_DC=spotify-cookie:latest,RAPIDAPI_KEY=rapidapi-key:latest,LRCLIB_APP_ID=lrclib-app-id:latest,LRCLIB_SECRET_KEY=lrclib-api-key:latest,DEFAULT_DISCORD_WEBHOOK_URL=discord-releases-webhook:latest,ADMIN_TOKENS=admin-tokens:latest,RED_API_KEY=red-api-key:latest,RED_API_URL=red-api-url:latest,OPS_API_KEY=ops-api-key:latest,OPS_API_URL=ops-api-url:latest,FLACFETCH_API_URL=flacfetch-api-url:latest,FLACFETCH_API_KEY=flacfetch-api-key:latest,POSTMARK_SERVER_TOKEN=postmark-server-token:latest,STRIPE_SECRET_KEY=stripe-secret-key:latest,STRIPE_WEBHOOK_SECRET=stripe-webhook-secret:latest,LANGFUSE_PUBLIC_KEY=langfuse-public-key:latest,LANGFUSE_SECRET_KEY=langfuse-secret-key:latest,LANGFUSE_HOST=langfuse-host:latest,ENCODING_WORKER_URL=encoding-worker-url:latest,ENCODING_WORKER_API_KEY=encoding-worker-api-key:latest,ENCODING_WORKER_FALLBACK_VMS=encoding-worker-fallback-vms:latest,VAPID_PUBLIC_KEY=vapid-public-key:latest,VAPID_PRIVATE_KEY=vapid-private-key:latest,E2E_BYPASS_KEY=e2e-bypass-key:latest,GDRIVE_VALIDATOR_URL=gdrive-validator-url:latest,EDGE_ORIGIN_SECRET=edge-origin-secret:latest,KARAOKEHUNT_FORWARDER_SECRET=karaokehunt-forwarder-secret:latest,KJBOX_PARTNER_SECRET=kjbox-partner-secret:latest"
# R3/R4 — smoke the tagged candidate revision BEFORE it serves any traffic.
- name: Smoke candidate revision