mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary Phase 2 of moving encoding to AWS; Phase 1 was #1236 and #1239. Design and rationale are in `docs/archive/2026-10-29-aws-encoding-worker-plan.md` § Phase 2. When Firestore `config/encoding-worker.aws_video_worker` is true **and** the AWS worker ranks first, `trigger_video_worker` runs the video worker **on the AWS box** instead of starting the Cloud Run job `video-encoding-job`. - **Uploads move to AWS:** the Dropbox/YouTube/Drive uploads (~187 GB/mo, ~$18/mo of Cloud Run internet egress) leave from AWS, on credits. - **No Cloud Run CPU** for the job (~$11/mo). - **Off by default.** Any failure falls back to the Cloud Run job, so this path can't drop a job. ## Changes - **`POST /run-worker` (worker):** - Spawns `python -m backend.workers.video_worker --job-id …` with the box's video-worker env. - Status is reported via `/status/{id}`. - A run counts in `/health` `active_jobs` from the moment it's accepted, so the idle watchdog won't stop the box mid-run. - Returns 503 unless the env file exists. - `job_id` is pattern-validated. - Any startup failure marks the run failed. - **`backend/services/local_encoded_outputs.py`:** a same-box handoff. - When the co-located video worker asks (`stash_local_outputs`, set only where `LOCAL_ENCODED_OUTPUTS_DIR` exists), the encoder keeps local copies of the finals, keyed by blob path. - The video worker takes those instead of downloading them back out of GCS. Without this, the move would *add* ~400 MB/job of GCS egress. - Copies are made off the event loop and purged after 6 h. - No change on GCE or Cloud Run. - **`video-worker-env.py` (box, every boot):** - Mirrors the **live** `video-encoding-job` env, with secrets resolved, so nothing is hand-copied and nothing drifts. - Runs as the job's identity, `karaoke-backend@`, via a second WIF binding (Pulumi `aws-encoding-worker-wif-backend-user`). - On failure it writes no env file, so `/run-worker` answers 503 and the Cloud Run job is used. - **`WorkerService._trigger_video_worker_on_aws`:** checks the flag and that the AWS worker ranks first, ensures it's running, waits for health, then POSTs `/run-worker`. Returns False on any doubt. - **Also:** `startup-aws.sh` creates the backend credential config and runs the env helper; CI uploads `video-worker-env.py`; `worker_code_paths.txt` now gates `local_encoded_outputs.py`. ## Rollout (needs `/gcp-admin`) 1. **Before merge:** targeted `pulumi up` of the backend WIF binding. It's additive. 2. **After stage B of Phase 1** (AWS worker preferred, timings verified): stop/start the box once so the new `startup-aws.sh` writes the env, then set `aws_video_worker: true`. Set it to `false` to roll back. ## Testing - **27 new tests:** - handoff stash/take/purge, path-escape guard, and the per-request stash flag; - `/run-worker`: disabled, bad module, dedupe, active immediately, spawn args/env, failure log tail, wheel-not-ready, safe start, unsafe ids; - env builder; - the trigger's AWS branch: off, not first, happy path, didn't start, refusals, errors. - **Full backend + unit suites:** no new failures (the 13 local failures are identical on main). - **Local CodeRabbit:** 5 findings, all fixed. The code-paths gate test also caught a missing worker path, now fixed. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
77 lines
3.2 KiB
Python
77 lines
3.2 KiB
Python
"""Workload Identity Federation for the AWS encoding worker.
|
|
|
|
The AWS encoding worker (EC2 in Nomad's AWS account) authenticates to GCP as
|
|
the existing ``encoding-worker`` service account with no keys: the instance's
|
|
IAM role ``nomad-encoding-worker`` is exchanged for a short-lived GCP token via
|
|
this pool/provider. The box's credential config is generated from the provider
|
|
name in infrastructure/encoding-worker/aws/provision.sh.
|
|
"""
|
|
|
|
import pulumi
|
|
import pulumi_gcp as gcp
|
|
|
|
from config import PROJECT_ID
|
|
|
|
AWS_ACCOUNT_ID = "790139457751"
|
|
AWS_WORKER_ROLE = "nomad-encoding-worker"
|
|
POOL_ID = "aws-encoding"
|
|
PROVIDER_ID = f"aws-{AWS_ACCOUNT_ID}"
|
|
|
|
|
|
def create_aws_encoding_wif(
|
|
encoding_worker_sa: gcp.serviceaccount.Account,
|
|
backend_sa: gcp.serviceaccount.Account,
|
|
) -> dict:
|
|
pool = gcp.iam.WorkloadIdentityPool(
|
|
"aws-encoding-wif-pool",
|
|
workload_identity_pool_id=POOL_ID,
|
|
display_name="AWS encoding worker",
|
|
description="EC2 encoding worker in Nomad's AWS account (karaoke-gen)",
|
|
)
|
|
provider = gcp.iam.WorkloadIdentityPoolProvider(
|
|
"aws-encoding-wif-provider",
|
|
workload_identity_pool_id=pool.workload_identity_pool_id,
|
|
workload_identity_pool_provider_id=PROVIDER_ID,
|
|
display_name="Nomad AWS account",
|
|
aws=gcp.iam.WorkloadIdentityPoolProviderAwsArgs(account_id=AWS_ACCOUNT_ID),
|
|
attribute_mapping={
|
|
"google.subject": "assertion.arn",
|
|
"attribute.aws_role": (
|
|
"assertion.arn.contains('assumed-role') ? "
|
|
"assertion.arn.extract('{account_arn}assumed-role/') + 'assumed-role/' + "
|
|
"assertion.arn.extract('assumed-role/{role_name}/') : assertion.arn"
|
|
),
|
|
},
|
|
# Only the worker's role in Nomad's account may federate.
|
|
attribute_condition=(
|
|
f"attribute.aws_role == 'arn:aws:sts::{AWS_ACCOUNT_ID}:assumed-role/{AWS_WORKER_ROLE}'"
|
|
),
|
|
)
|
|
binding = gcp.serviceaccount.IAMMember(
|
|
"aws-encoding-worker-wif-user",
|
|
service_account_id=encoding_worker_sa.name,
|
|
role="roles/iam.workloadIdentityUser",
|
|
member=pool.name.apply(
|
|
lambda name: (
|
|
f"principalSet://iam.googleapis.com/{name}/attribute.aws_role/"
|
|
f"arn:aws:sts::{AWS_ACCOUNT_ID}:assumed-role/{AWS_WORKER_ROLE}"
|
|
)
|
|
),
|
|
)
|
|
# The video worker can also run on the box (/run-worker): it uses the Cloud
|
|
# Run video-encoding-job's identity, karaoke-backend@, for exact permission
|
|
# parity (see infrastructure/encoding-worker/aws/video-worker-env.py).
|
|
backend_binding = gcp.serviceaccount.IAMMember(
|
|
"aws-encoding-worker-wif-backend-user",
|
|
service_account_id=backend_sa.name,
|
|
role="roles/iam.workloadIdentityUser",
|
|
member=pool.name.apply(
|
|
lambda name: (
|
|
f"principalSet://iam.googleapis.com/{name}/attribute.aws_role/"
|
|
f"arn:aws:sts::{AWS_ACCOUNT_ID}:assumed-role/{AWS_WORKER_ROLE}"
|
|
)
|
|
),
|
|
)
|
|
pulumi.export("aws_encoding_wif_provider", provider.name)
|
|
return {"pool": pool, "provider": provider, "binding": binding, "backend_binding": backend_binding}
|