mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary
Phase 1 of moving encoding to AWS. EC2 is paid for by the AWS Activate
credits (acct 790139457751, expiring 2028-10-31).
- **Plan:** `docs/archive/2026-10-29-aws-encoding-worker-plan.md`.
- **Benchmark** (3 real prod jobs replayed): an EC2 c7a.4xlarge matched
or beat GCE c4d-highcpu-16.
The EC2 box runs the **unchanged** `gce_encoding` worker service and
joins the existing candidate pool as `provider: aws`. The existing
failover, active override, lost-job detection and stockout cooldowns all
apply to it as they are.
## Changes
- **`backend/services/aws_encoding_compute.py` (new):**
`AwsComputeAdapter` looks to the manager like the
`compute_v1.InstancesClient` methods it uses.
- EC2 states map to GCE statuses.
- EC2 capacity errors are added to `CAPACITY_ERROR_CODES`.
- A missing instance raises `NotFound`.
- Controller credentials come from Secret Manager
`aws-encoding-controller-credentials`.
- **`EncodingWorkerManager`:**
- `_compute_for(vm)` sends AWS VMs to the adapter; the GCE paths are
untouched.
- A shared `parse_fallback_vms` passes `provider`/`instance_id`/`region`
through.
- `ranked_candidates()` is used by both the encode path and the warmups.
- `ensure_preferred_running()` behaves **exactly like
`ensure_primary_running` whenever the GCE primary ranks first**, which
is today's case.
- An AWS control-plane fault fails over to the GCE candidates.
- **Ranking:** `SPEED_RANK["c7a"] = 90`, so AWS is the last resort by
default. Firestore `config/encoding-worker.speed_rank_overrides =
{"c7a": 5}` makes it preferred, with no deploy.
- **Isolation from GCE-only tooling:**
- The override now records `active_override_provider`.
- The CI blue-green deploy, `deploy_promote` and the idle Cloud Function
all ignore the AWS worker; it hot-updates its wheel per job.
- A GCE promotion clears any stale provider tag.
- **Worker:**
- Uses `gcloud storage` on AWS, because gsutil can't use WIF credential
files. GCE keeps gsutil.
- `infrastructure/encoding-worker/aws/` adds:
- `provision.sh`, as EC2 user-data;
- `startup-aws.sh`, uploaded by CI;
- `idle-watchdog.py`, an idle self-stop that runs as its own systemd
timer.
- **Infra (GCP), already applied with `/gcp-admin`:**
- The WIF pool `aws-encoding`, plus an AWS provider that only admits the
`nomad-encoding-worker` role, federated to `encoding-worker@`.
- The secret `aws-encoding-controller-credentials`.
- Preview and apply: 4 creates, 288 unchanged.
- **Infra (AWS):** a new Pulumi project `infrastructure/aws` (stack
`nomadkaraoke/prod`):
- the c7a.8xlarge **persistent Spot** worker, which stops on
interruption or shutdown;
- an Elastic IP;
- a security group on 8080 (API key, same as the GCE firewall);
- the worker role with SSM;
- a tag-scoped `nomad-encoding-controller` user.
- Preview: 10 creates. **I apply it after merge**, because the box
provisions from the deployed wheel and boot scripts.
- `boto3` is now an explicit dependency.
- CI also runs the `encoding_worker_idle` function tests, in the job
renamed "Infra - Cloud Function tests".
## Rollout (after merge)
1. **Stage A:** `pulumi up` the AWS stack. Store the controller key in
the secret, then add the worker's entry to
`encoding-worker-fallback-vms`. It's inert unless both c4d and c2d stock
out.
2. **Stage B:** a supervised `speed_rank_overrides {"c7a": 5}` window.
Run real jobs, then compare timings and outputs, including a visual
check of the portrait video.
3. **Stage C:** keep it preferred. After a clean week, delete the c4d
pair.
Phase 2, running distribution (Dropbox/YouTube/Drive uploads) from AWS,
follows in a separate PR.
## Testing
- **New tests:**
- adapter: botocore `Stubber`, covering the state map, capacity and
NotFound handling;
- manager routing and ranking with overrides;
- `ensure_preferred_running` branches;
- failover on an AWS control-plane fault;
- provider tagging;
- deploy_promote exclusions;
- the idle function skipping AWS;
- watchdog decisions;
- the gcs CLI choice.
- **Updated** the candidate and warmup-route tests to the new seams.
- **Full backend + unit suites** have no new failures. The local
emulator and cross-suite failures are identical on main.
- **Local CodeRabbit:** 2 findings, both fixed (failover on AWS faults;
stale provider tag).
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
128 lines
4.9 KiB
Python
128 lines
4.9 KiB
Python
"""
|
|
Secret Manager resources.
|
|
|
|
Manages all secrets for the application. Secret values are added manually
|
|
via gcloud or the Cloud Console after resources are created.
|
|
"""
|
|
|
|
from pulumi_gcp import secretmanager
|
|
|
|
|
|
def create_secrets() -> dict[str, secretmanager.Secret]:
|
|
"""
|
|
Create all Secret Manager secrets.
|
|
|
|
Returns:
|
|
dict: Dictionary mapping secret names to Secret resources.
|
|
"""
|
|
# Define all secret names
|
|
# Note: Private tracker secrets (red-*, ops-*) are managed by flacfetch repo
|
|
secret_names = [
|
|
# Audio processing
|
|
"audioshake-api-key",
|
|
"genius-api-key",
|
|
# LRCLIB registered server-side app (higher rate limits; lrclib.net developer portal)
|
|
"lrclib-app-id",
|
|
"lrclib-api-key",
|
|
"audio-separator-api-url",
|
|
|
|
# Payment and email
|
|
"stripe-secret-key",
|
|
"stripe-webhook-secret",
|
|
"postmark-server-token",
|
|
|
|
# Observability (Langfuse for LLM tracing)
|
|
"langfuse-public-key",
|
|
"langfuse-secret-key",
|
|
"langfuse-host",
|
|
|
|
# AI auto-correct suggestions (Claude models via Anthropic API)
|
|
"anthropic-api-key",
|
|
|
|
# Service authentication
|
|
"flacfetch-api-key",
|
|
"flacfetch-api-url",
|
|
"encoding-worker-api-key",
|
|
# Capacity-fallback VMs config (JSON list of {vm,zone,ip,machine_type?}).
|
|
# machine_type (e.g. "c4-highcpu-32") drives the shared speed-rank
|
|
# ordering; optional for legacy entries (inferred from the vm name).
|
|
# Stored as a secret because --set-env-vars on Cloud Run is
|
|
# comma-delimited and the JSON value contains commas; the value itself is
|
|
# non-secret (just public IPs of fallback VMs).
|
|
"encoding-worker-fallback-vms",
|
|
# AWS encoding worker: IAM user keys ({"access_key_id","secret_access_key"})
|
|
# the backend uses to start/stop/describe the EC2 worker
|
|
# (backend/services/aws_encoding_compute.py). Version added by hand.
|
|
"aws-encoding-controller-credentials",
|
|
|
|
# Notifications
|
|
"pushbullet-api-key",
|
|
|
|
# Internal service URLs
|
|
"gdrive-validator-url",
|
|
|
|
# Web Push (VAPID keys for mobile push notifications)
|
|
"vapid-public-key",
|
|
"vapid-private-key",
|
|
|
|
# GitHub
|
|
"github-runner-pat",
|
|
"github-webhook-secret", # For runner manager webhook verification
|
|
# DR: PAT (repo + read:org) the nightly backup uses to clone every repo
|
|
# under github.com/nomadkaraoke + github.com/beveradb into git bundles on
|
|
# S3, so code survives loss of GitHub access. Value added manually.
|
|
"github-backup-token",
|
|
|
|
# E2E Testing
|
|
"e2e-bypass-key",
|
|
|
|
# KaraokeNerds API
|
|
"karaokenerds-api-key",
|
|
|
|
# Notifications (Discord)
|
|
"discord-alert-webhook",
|
|
|
|
# Divebar on-demand refresh trigger (shared bearer token: kjbox's
|
|
# "Refresh catalog" button -> divebar-lookup `refresh` action). Gates
|
|
# an otherwise-public endpoint that re-runs the mirror/sync/xref
|
|
# scheduler jobs on demand, so it's anti-abuse, not data-sensitive.
|
|
"divebar-refresh-token",
|
|
|
|
# Cloudflare edge origin lock: shared secret Cloudflare injects as the
|
|
# X-Edge-Auth header on proxied requests; the backend edge-auth
|
|
# middleware rejects public requests that lack it (blocks direct-to-
|
|
# origin bypass of the WAF). Value added manually; also provided to
|
|
# Pulumi as `edge:originSecret` for the Cloudflare header transform.
|
|
# See modules/edge_security.py.
|
|
"edge-origin-secret",
|
|
# Cloudflare API token for the admin tenant console: adds/removes each
|
|
# tenant's `<id>.nomadkaraoke.com` Pages custom domain (project
|
|
# karaoke-gen-tenant) + proxied CNAME. Account-owned token scoped to
|
|
# Pages Read/Write (account) + DNS Read/Write (nomadkaraoke.com zone
|
|
# only). Value added manually. See services/tenant_domain_service.py.
|
|
"cloudflare-tenant-domains-token",
|
|
# karaoke-decide's partner secret for POST /api/kjbox/catalog/resolve
|
|
# (header X-Partner-Secret; mounted as DECIDE_GEN_PARTNER_SECRET by the
|
|
# CI deploy step). Value added manually; the same value is mounted into
|
|
# the decide services.
|
|
"decide-gen-partner-secret",
|
|
# OAuth token (youtube + youtube.force-ssl) for the YouTube comment
|
|
# auto-reply bot, separate from the upload token. Minted with the
|
|
# "youtube-comment-bot" Desktop OAuth client; value added manually (see
|
|
# docs/REQUESTS-BOARD.md § YouTube comment auto-reply).
|
|
"youtube-comment-bot-credentials",
|
|
]
|
|
|
|
secrets = {}
|
|
|
|
for secret_id in secret_names:
|
|
secrets[secret_id] = secretmanager.Secret(
|
|
secret_id,
|
|
secret_id=secret_id,
|
|
replication=secretmanager.SecretReplicationArgs(
|
|
auto=secretmanager.SecretReplicationAutoArgs(),
|
|
),
|
|
)
|
|
|
|
return secrets
|