Files
karaoke-gen/infrastructure/modules/aws_encoding_wif.py
T
Andrew Beveridge ☄️andClaude Opus 5.5 78264fa23a feat(encoding): run the video worker (distribution) on the AWS box (v0.289.0) (#1240)
## Summary
Phase 2 of moving encoding to AWS; Phase 1 was #1236 and #1239. Design
and rationale are in
`docs/archive/2026-10-29-aws-encoding-worker-plan.md` § Phase 2.

When Firestore `config/encoding-worker.aws_video_worker` is true **and**
the AWS worker ranks first, `trigger_video_worker` runs the video worker
**on the AWS box** instead of starting the Cloud Run job
`video-encoding-job`.
- **Uploads move to AWS:** the Dropbox/YouTube/Drive uploads (~187
GB/mo, ~$18/mo of Cloud Run internet egress) leave from AWS, on credits.
- **No Cloud Run CPU** for the job (~$11/mo).
- **Off by default.** Any failure falls back to the Cloud Run job, so
this path can't drop a job.

## Changes
- **`POST /run-worker` (worker):**
- Spawns `python -m backend.workers.video_worker --job-id …` with the
box's video-worker env.
  - Status is reported via `/status/{id}`.
- A run counts in `/health` `active_jobs` from the moment it's accepted,
so the idle watchdog won't stop the box mid-run.
  - Returns 503 unless the env file exists.
  - `job_id` is pattern-validated.
  - Any startup failure marks the run failed.
- **`backend/services/local_encoded_outputs.py`:** a same-box handoff.
- When the co-located video worker asks (`stash_local_outputs`, set only
where `LOCAL_ENCODED_OUTPUTS_DIR` exists), the encoder keeps local
copies of the finals, keyed by blob path.
- The video worker takes those instead of downloading them back out of
GCS. Without this, the move would *add* ~400 MB/job of GCS egress.
  - Copies are made off the event loop and purged after 6 h.
  - No change on GCE or Cloud Run.
- **`video-worker-env.py` (box, every boot):**
- Mirrors the **live** `video-encoding-job` env, with secrets resolved,
so nothing is hand-copied and nothing drifts.
- Runs as the job's identity, `karaoke-backend@`, via a second WIF
binding (Pulumi `aws-encoding-worker-wif-backend-user`).
- On failure it writes no env file, so `/run-worker` answers 503 and the
Cloud Run job is used.
- **`WorkerService._trigger_video_worker_on_aws`:** checks the flag and
that the AWS worker ranks first, ensures it's running, waits for health,
then POSTs `/run-worker`. Returns False on any doubt.
- **Also:** `startup-aws.sh` creates the backend credential config and
runs the env helper; CI uploads `video-worker-env.py`;
`worker_code_paths.txt` now gates `local_encoded_outputs.py`.

## Rollout (needs `/gcp-admin`)
1. **Before merge:** targeted `pulumi up` of the backend WIF binding.
It's additive.
2. **After stage B of Phase 1** (AWS worker preferred, timings
verified): stop/start the box once so the new `startup-aws.sh` writes
the env, then set `aws_video_worker: true`. Set it to `false` to roll
back.

## Testing
- **27 new tests:**
- handoff stash/take/purge, path-escape guard, and the per-request stash
flag;
- `/run-worker`: disabled, bad module, dedupe, active immediately, spawn
args/env, failure log tail, wheel-not-ready, safe start, unsafe ids;
  - env builder;
- the trigger's AWS branch: off, not first, happy path, didn't start,
refusals, errors.
- **Full backend + unit suites:** no new failures (the 13 local failures
are identical on main).
- **Local CodeRabbit:** 5 findings, all fixed. The code-paths gate test
also caught a missing worker path, now fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 15:02:42 -04:00

77 lines
3.2 KiB
Python

"""Workload Identity Federation for the AWS encoding worker.
The AWS encoding worker (EC2 in Nomad's AWS account) authenticates to GCP as
the existing ``encoding-worker`` service account with no keys: the instance's
IAM role ``nomad-encoding-worker`` is exchanged for a short-lived GCP token via
this pool/provider. The box's credential config is generated from the provider
name in infrastructure/encoding-worker/aws/provision.sh.
"""
import pulumi
import pulumi_gcp as gcp
from config import PROJECT_ID
AWS_ACCOUNT_ID = "790139457751"
AWS_WORKER_ROLE = "nomad-encoding-worker"
POOL_ID = "aws-encoding"
PROVIDER_ID = f"aws-{AWS_ACCOUNT_ID}"
def create_aws_encoding_wif(
encoding_worker_sa: gcp.serviceaccount.Account,
backend_sa: gcp.serviceaccount.Account,
) -> dict:
pool = gcp.iam.WorkloadIdentityPool(
"aws-encoding-wif-pool",
workload_identity_pool_id=POOL_ID,
display_name="AWS encoding worker",
description="EC2 encoding worker in Nomad's AWS account (karaoke-gen)",
)
provider = gcp.iam.WorkloadIdentityPoolProvider(
"aws-encoding-wif-provider",
workload_identity_pool_id=pool.workload_identity_pool_id,
workload_identity_pool_provider_id=PROVIDER_ID,
display_name="Nomad AWS account",
aws=gcp.iam.WorkloadIdentityPoolProviderAwsArgs(account_id=AWS_ACCOUNT_ID),
attribute_mapping={
"google.subject": "assertion.arn",
"attribute.aws_role": (
"assertion.arn.contains('assumed-role') ? "
"assertion.arn.extract('{account_arn}assumed-role/') + 'assumed-role/' + "
"assertion.arn.extract('assumed-role/{role_name}/') : assertion.arn"
),
},
# Only the worker's role in Nomad's account may federate.
attribute_condition=(
f"attribute.aws_role == 'arn:aws:sts::{AWS_ACCOUNT_ID}:assumed-role/{AWS_WORKER_ROLE}'"
),
)
binding = gcp.serviceaccount.IAMMember(
"aws-encoding-worker-wif-user",
service_account_id=encoding_worker_sa.name,
role="roles/iam.workloadIdentityUser",
member=pool.name.apply(
lambda name: (
f"principalSet://iam.googleapis.com/{name}/attribute.aws_role/"
f"arn:aws:sts::{AWS_ACCOUNT_ID}:assumed-role/{AWS_WORKER_ROLE}"
)
),
)
# The video worker can also run on the box (/run-worker): it uses the Cloud
# Run video-encoding-job's identity, karaoke-backend@, for exact permission
# parity (see infrastructure/encoding-worker/aws/video-worker-env.py).
backend_binding = gcp.serviceaccount.IAMMember(
"aws-encoding-worker-wif-backend-user",
service_account_id=backend_sa.name,
role="roles/iam.workloadIdentityUser",
member=pool.name.apply(
lambda name: (
f"principalSet://iam.googleapis.com/{name}/attribute.aws_role/"
f"arn:aws:sts::{AWS_ACCOUNT_ID}:assumed-role/{AWS_WORKER_ROLE}"
)
),
)
pulumi.export("aws_encoding_wif_provider", provider.name)
return {"pool": pool, "provider": provider, "binding": binding, "backend_binding": backend_binding}