mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
feat(youtube): TornadoAPI first for YouTube downloads, flacfetch as fallback (v0.291.0) (#1247)
YouTube downloads depended on one logged-in cookie session on one VM. This tries TornadoAPI first and keeps flacfetch as the fallback. Plan: workspace `docs/archive/2026-11-03-tornado-youtube-download-plan.md`. Alert: https://discord.com/channels/1198691101943541770/1265517528608669707/1567241547739570298 Alert: https://discord.com/channels/1198691101943541770/1265517528608669707/1567252495535841386 Alert: https://discord.com/channels/1198691101943541770/1265517528608669707/1567252497620537415 ## Inactive until the secret exists Nothing changes in prod when this merges. The deploy adds `TORNADO_API_KEY` only once secret `tornado-api-key` has a version, and creating it needs `/gcp-admin`. `YOUTUBE_DOWNLOAD_PROVIDER=flacfetch` switches it off. ## What changed - **`tornado_client.py`:** create job, poll with a 180s deadline (cancels on timeout), stream the delivered file, delete it from Tornado. Retries with one idempotency key per download; honours `Retry-After`. - **`YouTubeDownloadService`:** Tornado first for YouTube, then flacfetch. WARNING per fallback, ERROR on 401/402/403, so an outage or lapsed billing is visible. Non-YouTube URLs stay on flacfetch. - **Availability check:** Tornado metadata first, because flacfetch's check rejects valid videos while its cookies are bot-blocked. If the lookup fails, flacfetch decides as before. ## Spike results (real API, 11 jobs) - `audio_only` + `format: opus` gives **packet-identical audio** to flacfetch's real job input (`Tk3ty5ansqs`, stream MD5 match). `audio_codec: copy` or no `format` gives 128k AAC instead, so the request omits it. - `mBJivIsN0n0`, which bot-blocked job 526bf46c, completed in about 5s of processing. - An age-restricted video completed without cookies; a missing video ends in `Warning`. - Billed about 25.6 MB for a 4-minute song. ## Testing - 79 mocked tests pass (`test_tornado_client.py`, `test_youtube_download_service.py`). - Live run through `YouTubeDownloadService` with the GCS upload stubbed, using a URL with a playlist parameter: only the single video was sent. - Local CodeRabbit: all findings fixed; the last small fix wasn't re-reviewed. ## Not verified - **A full pipeline job with this input.** It can only run once the secret exists. Files are Ogg Opus named `.ogg` (flacfetch's were WebM Opus), so the codec is the same but the container is new to this path. Run the daily E2E or one admin job first. - **Deviation from the plan:** the Tornado job id is logged but not saved on the gen job, because a restart loses the in-request download on the flacfetch path too. A restart can orphan one job, which costs a fraction of a cent. - **CI's secret check:** it depends on the deploy service account being able to read the secret; the log line `tornado-api-key wired:` shows the result. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVTvqXyL4QyA2EjzQMG Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
3a563ab3e6
commit
ca05245469
8 files changed
+861
-22
No files matched your search
@@ -2197,6 +2197,14 @@ jobs:
|
||||
# --cpu-throttling (request-based billing) a warm-but-idle non-min
|
||||
# instance costs nothing. Concurrency stays far below the wedge-era 80. --clear-vpc-connector: the
|
||||
# serverless VPC connector was deleted (flacfetch is public now).
|
||||
# YouTube downloads try TornadoAPI first once this secret has a version. Until then the
|
||||
# backend uses flacfetch, and referencing a missing secret would fail the deploy.
|
||||
TORNADO_SECRET=""
|
||||
if gcloud secrets versions describe latest --secret=tornado-api-key --project ${{ env.PROJECT_ID }} >/dev/null 2>&1; then
|
||||
TORNADO_SECRET=",TORNADO_API_KEY=tornado-api-key:latest"
|
||||
fi
|
||||
echo "tornado-api-key wired: ${TORNADO_SECRET:+yes}"
|
||||
|
||||
gcloud run deploy karaoke-backend \
|
||||
--image $IMAGE \
|
||||
--no-traffic \
|
||||
@@ -2215,7 +2223,7 @@ jobs:
|
||||
--min-instances 0 \
|
||||
--clear-vpc-connector \
|
||||
--set-env-vars "GOOGLE_CLOUD_PROJECT=${{ env.PROJECT_ID }},GCS_BUCKET_NAME=karaoke-gen-storage-${{ env.PROJECT_ID }},FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,KARAOKE_GEN_VERSION=$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,ENABLE_CLOUD_TASKS=true,CLOUD_RUN_SERVICE_URL=https://api.nomadkaraoke.com,GCP_REGION=us-central1,DEFAULT_ENABLE_YOUTUBE_UPLOAD=true,DEFAULT_BRAND_PREFIX=NOMAD,DEFAULT_PRIVATE_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-NonPublished,DEFAULT_PRIVATE_BRAND_PREFIX=NOMADNP,USE_GCE_ENCODING=true,USE_GCE_PREVIEW_ENCODING=true,ENABLE_PUSH_NOTIFICATIONS=true,USE_CLOUD_RUN_JOBS_FOR_VIDEO=true,USE_CLOUD_RUN_JOBS_FOR_RENDER=true,EMAIL_PROVIDER=postmark,AUTO_CORRECT_MODEL=claude-opus-5-5,AUTO_CORRECT_COMPARE_MODELS=claude-opus-5-5;gemini-3.8-flash,AUTO_CORRECT_PROACTIVE_ENABLED=true,AUTO_APPROVAL_BACKING_KEEP_ENABLED=true,COMMUNITY_DAILY_PICK_ENABLED=true,STORAGE_RETENTION_DRY_RUN=false,STORAGE_RETENTION_MAX_JOBS_PER_RUN=50,CONVERSION_EMAILS_ENABLED=true,REVIEW_LAST_CHANCE_ENABLED=true,CONVERSION_OFFER_PERCENT=25,YOUTUBE_COMMENT_AUTOREPLY_ENABLED=true,YOUTUBE_COMMENT_AUTOREPLY_START=2026-04-22T00:00:00Z,EDGE_AUTH_MODE=enforce,COMMIT_SHA=${{ steps.build-meta.outputs.sha }},PR_NUMBER=${{ steps.build-meta.outputs.pr_number }},PR_TITLE=${{ steps.build-meta.outputs.pr_title }}" \
|
||||
--set-secrets "ANTHROPIC_API_KEY=anthropic-api-key:latest,GEMINI_API_KEY=gemini-api-key:latest,AUDIOSHAKE_API_TOKEN=audioshake-api-key:latest,GENIUS_API_TOKEN=genius-api-key:latest,SPOTIFY_COOKIE_SP_DC=spotify-cookie:latest,RAPIDAPI_KEY=rapidapi-key:latest,LRCLIB_APP_ID=lrclib-app-id:latest,LRCLIB_SECRET_KEY=lrclib-api-key:latest,DEFAULT_DISCORD_WEBHOOK_URL=discord-releases-webhook:latest,ADMIN_TOKENS=admin-tokens:latest,RED_API_KEY=red-api-key:latest,RED_API_URL=red-api-url:latest,OPS_API_KEY=ops-api-key:latest,OPS_API_URL=ops-api-url:latest,FLACFETCH_API_URL=flacfetch-api-url:latest,FLACFETCH_API_KEY=flacfetch-api-key:latest,POSTMARK_SERVER_TOKEN=postmark-server-token:latest,STRIPE_SECRET_KEY=stripe-secret-key:latest,STRIPE_WEBHOOK_SECRET=stripe-webhook-secret:latest,LANGFUSE_PUBLIC_KEY=langfuse-public-key:latest,LANGFUSE_SECRET_KEY=langfuse-secret-key:latest,LANGFUSE_HOST=langfuse-host:latest,ENCODING_WORKER_URL=encoding-worker-url:latest,ENCODING_WORKER_API_KEY=encoding-worker-api-key:latest,ENCODING_WORKER_FALLBACK_VMS=encoding-worker-fallback-vms:latest,VAPID_PUBLIC_KEY=vapid-public-key:latest,VAPID_PRIVATE_KEY=vapid-private-key:latest,E2E_BYPASS_KEY=e2e-bypass-key:latest,GDRIVE_VALIDATOR_URL=gdrive-validator-url:latest,EDGE_ORIGIN_SECRET=edge-origin-secret:latest,KARAOKEHUNT_FORWARDER_SECRET=karaokehunt-forwarder-secret:latest,KJBOX_PARTNER_SECRET=kjbox-partner-secret:latest,DECIDE_GEN_PARTNER_SECRET=decide-gen-partner-secret:latest,REVENUECAT_WEBHOOK_AUTH=revenuecat-webhook-auth:latest,REVENUECAT_SECRET_API_KEY=revenuecat-secret-api-key:latest,APP_REVIEW_DEMO_CODE=app-review-demo-code:latest"
|
||||
--set-secrets "ANTHROPIC_API_KEY=anthropic-api-key:latest,GEMINI_API_KEY=gemini-api-key:latest,AUDIOSHAKE_API_TOKEN=audioshake-api-key:latest,GENIUS_API_TOKEN=genius-api-key:latest,SPOTIFY_COOKIE_SP_DC=spotify-cookie:latest,RAPIDAPI_KEY=rapidapi-key:latest,LRCLIB_APP_ID=lrclib-app-id:latest,LRCLIB_SECRET_KEY=lrclib-api-key:latest,DEFAULT_DISCORD_WEBHOOK_URL=discord-releases-webhook:latest,ADMIN_TOKENS=admin-tokens:latest,RED_API_KEY=red-api-key:latest,RED_API_URL=red-api-url:latest,OPS_API_KEY=ops-api-key:latest,OPS_API_URL=ops-api-url:latest,FLACFETCH_API_URL=flacfetch-api-url:latest,FLACFETCH_API_KEY=flacfetch-api-key:latest,POSTMARK_SERVER_TOKEN=postmark-server-token:latest,STRIPE_SECRET_KEY=stripe-secret-key:latest,STRIPE_WEBHOOK_SECRET=stripe-webhook-secret:latest,LANGFUSE_PUBLIC_KEY=langfuse-public-key:latest,LANGFUSE_SECRET_KEY=langfuse-secret-key:latest,LANGFUSE_HOST=langfuse-host:latest,ENCODING_WORKER_URL=encoding-worker-url:latest,ENCODING_WORKER_API_KEY=encoding-worker-api-key:latest,ENCODING_WORKER_FALLBACK_VMS=encoding-worker-fallback-vms:latest,VAPID_PUBLIC_KEY=vapid-public-key:latest,VAPID_PRIVATE_KEY=vapid-private-key:latest,E2E_BYPASS_KEY=e2e-bypass-key:latest,GDRIVE_VALIDATOR_URL=gdrive-validator-url:latest,EDGE_ORIGIN_SECRET=edge-origin-secret:latest,KARAOKEHUNT_FORWARDER_SECRET=karaokehunt-forwarder-secret:latest,KJBOX_PARTNER_SECRET=kjbox-partner-secret:latest,DECIDE_GEN_PARTNER_SECRET=decide-gen-partner-secret:latest,REVENUECAT_WEBHOOK_AUTH=revenuecat-webhook-auth:latest,REVENUECAT_SECRET_API_KEY=revenuecat-secret-api-key:latest,APP_REVIEW_DEMO_CODE=app-review-demo-code:latest${TORNADO_SECRET}"
|
||||
|
||||
# R3/R4 — smoke the tagged candidate revision BEFORE it serves any traffic.
|
||||
- name: Smoke candidate revision
|
||||
|
||||
Reference in new issue
Block a user