Files
Andrew Beveridge ☄️andClaude Opus 5.5 a1cbe1c0d4 feat(encoding): retire the GCE c4d pair; run on AWS + c2df only (0.293.0) (#1255)
## Why
Since 2026-10-31 the AWS EC2 worker (`aws-encoding-worker-1`) has
handled every encode and distribution job. 27 jobs went to it, with no
Cloud Run fallbacks, no `encoding_stuck`, and nothing encoding-related
in #alerts. The c4d Spot pair (`encoding-worker-a`/`-b`) now only costs
~$8/mo of Hyperdisk. `encoding-worker-fallback-c2df` stays as the GCP
fallback.

## What
Deleting the pair needs code changes first, because several paths
assumed it exists:
- **`encoding_worker_manager`**
  - The primary is optional: an empty `primary_vm` means none.
- `ranked_candidates` skips a missing primary and always sets
`is_primary`, so with no primary a successful start records the AWS/c2df
override. Previously the legacy index-0 rule would have cleared it.
- `ensure_primary_running` and `ensure_preferred_running` start a ranked
candidate. If there are no candidates they raise `NO_CANDIDATES`.
- `demote_active_worker` looks up the override's `machine_type`, so the
AWS box can now be cooled down. It then clears the override so the retry
goes elsewhere.
- **`encoding_service`**
  - When `active_url` is empty, it targets the top-ranked candidate.
- It also invalidates its cached URL after a single-candidate fallback.
- **CI deploy (`ci.yml` + `deploy_promote.py`)**
- While AWS serves, a c2df green is validate-only: it writes
`validated_version`, keeps the AWS override, and stops the green. Before
this, it would have replaced the AWS override with c2df.
- The worker-change gate now diffs against `validated_version`, because
`primary_version` stops moving.
- An in-place restart of a serving override first waits for it to drain
(15-min cap). If it is still busy, the restart is skipped.
- **Idle function:** a missing VM is logged as a WARNING returning
`NOT_FOUND`, not an ERROR.
- **Error monitor:** watches c2df instead of a/b.
- **Pulumi:** drops the a/b VMs and static IPs. The prod preview shows
exactly 4 deletes. A separate, unrelated
`youtube-comment-autoreply-scheduler` drift shows up too; it will not be
applied.
- **Docs:** CLAUDE.md, ARCHITECTURE, DEVELOPMENT and TROUBLESHOOTING.

## Rollout (after merge + deploy)
1. Back up `config/encoding-worker`, then set `primary_vm`,
`primary_ip`, `secondary_vm` and `secondary_ip` to `""`.
2. Apply `pulumi destroy --target` on the 4 a/b URNs only.
3. Redeploy the idle function with `deploy.sh`.
4. Verify: one prod encode on AWS, warmup, and the logs.

**Rollback:** in Firestore, delete `speed_rank_overrides` and set
`aws_video_worker=false`. Restoring the pair would need a revert plus
`pulumi up`, which gives new static IPs.

## Testing
- New `backend/tests/test_encoding_worker_no_primary.py` (13 tests),
plus new deploy_promote, idle-function and error-monitor tests.
- Full backend suite: 7188 passed. The 8 `test_emulator_integration`
failures also fail on main, because of a local emulator from another
session.
- CodeRabbit CLI: no findings. An independent review found 2 issues,
both fixed: the demote retry path, and the in-place restart drain.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVTkatGnrYFhCzq4PTe

@coderabbitai ignore

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 18:50:54 -05:00
..

Encoding Worker Deployment

This directory contains the scripts for the GCE encoding worker's immutable deployment pattern.

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                    PACKER IMAGE (rarely changes)                │
├─────────────────────────────────────────────────────────────────┤
│ - Python 3.13, FFmpeg, fonts (static dependencies)             │
│ - bootstrap.sh (downloads startup.sh from GCS)                 │
│ - systemd service definition                                    │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼ downloads on every service start
┌─────────────────────────────────────────────────────────────────┐
│                    GCS (CI-managed, always current)             │
├─────────────────────────────────────────────────────────────────┤
│ gs://karaoke-gen-storage-nomadkaraoke/                         │
│ ├── encoding-worker/                                            │
│ │   ├── startup.sh      ← Main logic (this file)               │
│ │   └── version.txt     ← Expected version number              │
│ └── wheels/                                                     │
│     └── karaoke_gen-current.whl  ← Latest wheel (fixed path)   │
└─────────────────────────────────────────────────────────────────┘

Files

File Location Purpose
bootstrap.sh Baked in Packer image Downloads and runs startup.sh
startup.sh Deployed to GCS by CI Main startup logic
version.txt Written to GCS by CI Expected version for verification

Deployment Flow

  1. CI builds wheel → uploads to wheels/karaoke_gen-current.whl
  2. CI uploads startup.sh → encoding-worker/startup.sh
  3. CI writes version → encoding-worker/version.txt
  4. CI restarts service → triggers bootstrap.sh
  5. bootstrap.sh downloads fresh startup.sh from GCS
  6. startup.sh downloads wheel, installs, verifies version
  7. CI verifies health endpoint returns correct version
  8. CI FAILS if version mismatch

Why This Pattern?

Before (problematic)

  • startup.sh baked into Packer image
  • Required image rebuild + VM recreation for logic changes
  • 51 versioned wheels, sorting could fail
  • CI warned on version mismatch but continued

After (robust)

  • startup.sh downloaded from GCS on every start
  • Logic changes via CI, no image rebuild needed
  • Fixed path karaoke_gen-current.whl, no sorting
  • CI FAILS on version mismatch

Modifying Startup Logic

  1. Edit infrastructure/encoding-worker/startup.sh
  2. Push to main
  3. CI automatically deploys to GCS
  4. Service restart picks up new script

No Packer image rebuild needed!

When to Rebuild Packer Image

Only rebuild when changing:

  • Python version
  • FFmpeg version
  • System packages/fonts
  • bootstrap.sh (should be extremely rare)

Manual Operations

# Check current version on VM
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
  --tunnel-through-iap \
  --command="curl -s http://localhost:8080/health | jq '.wheel_version'"

# View startup logs
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
  --tunnel-through-iap \
  --command="sudo journalctl -u encoding-worker -n 50"

# Force service restart (triggers fresh startup.sh download)
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
  --tunnel-through-iap \
  --command="sudo systemctl restart encoding-worker"

Troubleshooting

Version mismatch after deployment

  1. Check CI logs for GCS upload success
  2. Check /var/log/encoding-worker-startup.log on VM
  3. Verify version.txt in GCS matches expected

Service won't start

  1. Check bootstrap.sh can reach GCS: gsutil ls gs://karaoke-gen-storage-nomadkaraoke/
  2. Check Secret Manager access for API key
  3. Check startup.sh logs: cat /var/log/encoding-worker-startup.log