Files
karaoke-gen/infrastructure/modules/secrets.py
T
Andrew Beveridge ☄️andClaude Opus 5.5 de5dfcd50d feat(encoding): AWS EC2 encoding worker as a pool candidate (v0.288.0) (#1236)
## Summary
Phase 1 of moving encoding to AWS. EC2 is paid for by the AWS Activate
credits (acct 790139457751, expiring 2028-10-31).
- **Plan:** `docs/archive/2026-10-29-aws-encoding-worker-plan.md`.
- **Benchmark** (3 real prod jobs replayed): an EC2 c7a.4xlarge matched
or beat GCE c4d-highcpu-16.

The EC2 box runs the **unchanged** `gce_encoding` worker service and
joins the existing candidate pool as `provider: aws`. The existing
failover, active override, lost-job detection and stockout cooldowns all
apply to it as they are.

## Changes
- **`backend/services/aws_encoding_compute.py` (new):**
`AwsComputeAdapter` looks to the manager like the
`compute_v1.InstancesClient` methods it uses.
  - EC2 states map to GCE statuses.
  - EC2 capacity errors are added to `CAPACITY_ERROR_CODES`.
  - A missing instance raises `NotFound`.
- Controller credentials come from Secret Manager
`aws-encoding-controller-credentials`.
- **`EncodingWorkerManager`:**
- `_compute_for(vm)` sends AWS VMs to the adapter; the GCE paths are
untouched.
- A shared `parse_fallback_vms` passes `provider`/`instance_id`/`region`
through.
- `ranked_candidates()` is used by both the encode path and the warmups.
- `ensure_preferred_running()` behaves **exactly like
`ensure_primary_running` whenever the GCE primary ranks first**, which
is today's case.
  - An AWS control-plane fault fails over to the GCE candidates.
- **Ranking:** `SPEED_RANK["c7a"] = 90`, so AWS is the last resort by
default. Firestore `config/encoding-worker.speed_rank_overrides =
{"c7a": 5}` makes it preferred, with no deploy.
- **Isolation from GCE-only tooling:**
  - The override now records `active_override_provider`.
- The CI blue-green deploy, `deploy_promote` and the idle Cloud Function
all ignore the AWS worker; it hot-updates its wheel per job.
  - A GCE promotion clears any stale provider tag.
- **Worker:**
- Uses `gcloud storage` on AWS, because gsutil can't use WIF credential
files. GCE keeps gsutil.
  - `infrastructure/encoding-worker/aws/` adds:
    - `provision.sh`, as EC2 user-data;
    - `startup-aws.sh`, uploaded by CI;
- `idle-watchdog.py`, an idle self-stop that runs as its own systemd
timer.
- **Infra (GCP), already applied with `/gcp-admin`:**
- The WIF pool `aws-encoding`, plus an AWS provider that only admits the
`nomad-encoding-worker` role, federated to `encoding-worker@`.
  - The secret `aws-encoding-controller-credentials`.
  - Preview and apply: 4 creates, 288 unchanged.
- **Infra (AWS):** a new Pulumi project `infrastructure/aws` (stack
`nomadkaraoke/prod`):
- the c7a.8xlarge **persistent Spot** worker, which stops on
interruption or shutdown;
  - an Elastic IP;
  - a security group on 8080 (API key, same as the GCE firewall);
  - the worker role with SSM;
  - a tag-scoped `nomad-encoding-controller` user.
- Preview: 10 creates. **I apply it after merge**, because the box
provisions from the deployed wheel and boot scripts.
- `boto3` is now an explicit dependency.
- CI also runs the `encoding_worker_idle` function tests, in the job
renamed "Infra - Cloud Function tests".

## Rollout (after merge)
1. **Stage A:** `pulumi up` the AWS stack. Store the controller key in
the secret, then add the worker's entry to
`encoding-worker-fallback-vms`. It's inert unless both c4d and c2d stock
out.
2. **Stage B:** a supervised `speed_rank_overrides {"c7a": 5}` window.
Run real jobs, then compare timings and outputs, including a visual
check of the portrait video.
3. **Stage C:** keep it preferred. After a clean week, delete the c4d
pair.

Phase 2, running distribution (Dropbox/YouTube/Drive uploads) from AWS,
follows in a separate PR.

## Testing
- **New tests:**
- adapter: botocore `Stubber`, covering the state map, capacity and
NotFound handling;
  - manager routing and ranking with overrides;
  - `ensure_preferred_running` branches;
  - failover on an AWS control-plane fault;
  - provider tagging;
  - deploy_promote exclusions;
  - the idle function skipping AWS;
  - watchdog decisions;
  - the gcs CLI choice.
- **Updated** the candidate and warmup-route tests to the new seams.
- **Full backend + unit suites** have no new failures. The local
emulator and cross-suite failures are identical on main.
- **Local CodeRabbit:** 2 findings, both fixed (failover on AWS faults;
stale provider tag).

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 17:42:35 -04:00

128 lines
4.9 KiB
Python

"""
Secret Manager resources.
Manages all secrets for the application. Secret values are added manually
via gcloud or the Cloud Console after resources are created.
"""
from pulumi_gcp import secretmanager
def create_secrets() -> dict[str, secretmanager.Secret]:
"""
Create all Secret Manager secrets.
Returns:
dict: Dictionary mapping secret names to Secret resources.
"""
# Define all secret names
# Note: Private tracker secrets (red-*, ops-*) are managed by flacfetch repo
secret_names = [
# Audio processing
"audioshake-api-key",
"genius-api-key",
# LRCLIB registered server-side app (higher rate limits; lrclib.net developer portal)
"lrclib-app-id",
"lrclib-api-key",
"audio-separator-api-url",
# Payment and email
"stripe-secret-key",
"stripe-webhook-secret",
"postmark-server-token",
# Observability (Langfuse for LLM tracing)
"langfuse-public-key",
"langfuse-secret-key",
"langfuse-host",
# AI auto-correct suggestions (Claude models via Anthropic API)
"anthropic-api-key",
# Service authentication
"flacfetch-api-key",
"flacfetch-api-url",
"encoding-worker-api-key",
# Capacity-fallback VMs config (JSON list of {vm,zone,ip,machine_type?}).
# machine_type (e.g. "c4-highcpu-32") drives the shared speed-rank
# ordering; optional for legacy entries (inferred from the vm name).
# Stored as a secret because --set-env-vars on Cloud Run is
# comma-delimited and the JSON value contains commas; the value itself is
# non-secret (just public IPs of fallback VMs).
"encoding-worker-fallback-vms",
# AWS encoding worker: IAM user keys ({"access_key_id","secret_access_key"})
# the backend uses to start/stop/describe the EC2 worker
# (backend/services/aws_encoding_compute.py). Version added by hand.
"aws-encoding-controller-credentials",
# Notifications
"pushbullet-api-key",
# Internal service URLs
"gdrive-validator-url",
# Web Push (VAPID keys for mobile push notifications)
"vapid-public-key",
"vapid-private-key",
# GitHub
"github-runner-pat",
"github-webhook-secret", # For runner manager webhook verification
# DR: PAT (repo + read:org) the nightly backup uses to clone every repo
# under github.com/nomadkaraoke + github.com/beveradb into git bundles on
# S3, so code survives loss of GitHub access. Value added manually.
"github-backup-token",
# E2E Testing
"e2e-bypass-key",
# KaraokeNerds API
"karaokenerds-api-key",
# Notifications (Discord)
"discord-alert-webhook",
# Divebar on-demand refresh trigger (shared bearer token: kjbox's
# "Refresh catalog" button -> divebar-lookup `refresh` action). Gates
# an otherwise-public endpoint that re-runs the mirror/sync/xref
# scheduler jobs on demand, so it's anti-abuse, not data-sensitive.
"divebar-refresh-token",
# Cloudflare edge origin lock: shared secret Cloudflare injects as the
# X-Edge-Auth header on proxied requests; the backend edge-auth
# middleware rejects public requests that lack it (blocks direct-to-
# origin bypass of the WAF). Value added manually; also provided to
# Pulumi as `edge:originSecret` for the Cloudflare header transform.
# See modules/edge_security.py.
"edge-origin-secret",
# Cloudflare API token for the admin tenant console: adds/removes each
# tenant's `<id>.nomadkaraoke.com` Pages custom domain (project
# karaoke-gen-tenant) + proxied CNAME. Account-owned token scoped to
# Pages Read/Write (account) + DNS Read/Write (nomadkaraoke.com zone
# only). Value added manually. See services/tenant_domain_service.py.
"cloudflare-tenant-domains-token",
# karaoke-decide's partner secret for POST /api/kjbox/catalog/resolve
# (header X-Partner-Secret; mounted as DECIDE_GEN_PARTNER_SECRET by the
# CI deploy step). Value added manually; the same value is mounted into
# the decide services.
"decide-gen-partner-secret",
# OAuth token (youtube + youtube.force-ssl) for the YouTube comment
# auto-reply bot, separate from the upload token. Minted with the
# "youtube-comment-bot" Desktop OAuth client; value added manually (see
# docs/REQUESTS-BOARD.md § YouTube comment auto-reply).
"youtube-comment-bot-credentials",
]
secrets = {}
for secret_id in secret_names:
secrets[secret_id] = secretmanager.Secret(
secret_id,
secret_id=secret_id,
replication=secretmanager.SecretReplicationArgs(
auto=secretmanager.SecretReplicationAutoArgs(),
),
)
return secrets