Compare commits

...
Author SHA1 Message Date
Andrew BeveridgeandClaude Opus 4.8 0539479281 docs: address CodeRabbit review on #913
- handoff: made self-contained (dropped agent-memory refs); reframed around the
  confirmed static finding that `ffmpeg_threads` is DEAD CONFIG (video_worker sets
  it but EncodingConfig/LocalEncodingService never read it, and the finalization
  ffmpeg commands omit -threads → default all-core), so it's a narrower
  efficiency+cleanup task, not "fix an 8-core cap"; strengthened measurement
  guidance to process-level (ffmpeg argv, per-thread CPU/RSS, per-format timings)
  and bound the encode+probes to one observed worker IP; fixed a wrong file
  pointer (backend/services/local_encoding_service.py, not the wheel).
- plan doc: c2df/n4db are now provisioned (secret v5, all 8 fallbacks live) — the
  autonomous retry loop finished them 2026-08-16; replaced the ellipsis
  `--target …` with full executable URNs.
- ARCHITECTURE: deploy-flow steps now cover the fallback-green/active_override
  branch, override clearing, drain+stop, and the no-green in-place-restart path.
- TROUBLESHOOTING: replaced the obsolete `encoding-worker` ssh host with
  active-worker (primary_vm / active_override_vm) guidance.
- GCP-COST-OPTIMIZATION: recommendation now matches the updated state (stay
  on-demand + idle-shutdown; Spot only as a future experiment).

Docs-only; no version bump.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-16 14:08:34 -04:00
Andrew BeveridgeandClaude Opus 4.8 b7c4edc7fc docs: reflect multi-instance-type encode pool + add finalization-CPU handoff
Bring all live docs in line with the v0.195.0 encode-worker mechanism (ranked
6-family pool, shared encoding_worker_preference ordering, capacity_state
cooldown, dep-baked image) and add a handoff for the next investigation.

- ARCHITECTURE.md: rewrite the "Encoding Worker" section — primary pair + 6-family
  fallback pool, shared preference module (runtime + deploy), capacity_state,
  is_primary override routing, self-sufficient dep-baked image; update the topology
  diagram + intro.
- TROUBLESHOOTING.md: capacity runbook now lists all 6 families / 8 fallbacks +
  cooldown; capacity-probe loop covers the 4 new VMs; deploy green-selection points
  at the ranked pool; "No module named" section notes the dep-bake largely fixed it
  (+ the create-time-stockout caveat).
- DEVELOPMENT.md + CLAUDE.md: fix "two VMs" framing; CLAUDE.md ssh example uses a
  real VM name (there is no VM literally named `encoding-worker`).
- GCP-COST-OPTIMIZATION.md: correct the stale "c4d running 24/7 ~$700/mo" current
  state (now on-demand pool + idle-shutdown, largely the shipped opportunities).
- LESSONS-LEARNED.md: mark the Aug-2026 2-family diversification lesson as
  superseded by the v0.195.0 pool (preserve history, redirect to current).
- NEW docs/archive/2026-08-16-finalization-cpu-efficiency-handoff.md: grounded
  handoff to investigate whether prod finalization is CPU-underutilized
  (ffmpeg_threads:8 on 32-vCPU workers), with repro + constraints.

Docs-only; no version bump.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-16 13:47:05 -04:00
8 changed files with 311 additions and 45 deletions

No files matched your search

+9 -4
View File
@@ -145,13 +145,18 @@ EOF
```
### GCE Encoding Worker
The worker is a c4d blue-green primary pair (`encoding-worker-a`/`-b`, us-central1-c)
plus 8 stopped fallback VMs across 6 machine families for stockout resilience
(`encoding-worker-fallback-{a,b,n2c,n2f,c4a,n4db,c2df,n2da}`). The currently-serving
VM is `active_override_vm` in `config/encoding-worker` if set, else `primary_vm`.
Use that VM name + its zone below (there is no VM literally named `encoding-worker`):
```bash
# Check health (includes wheel_version)
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
# Check health (includes wheel_version). Example uses the primary pair's VM.
gcloud compute ssh encoding-worker-a --zone=us-central1-c --project=nomadkaraoke \
--command="curl -s http://localhost:8080/health"
# Restart service (pick up new wheel after a deploy)
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
# Restart the service (pick up new worker-side code after a deploy)
gcloud compute ssh encoding-worker-a --zone=us-central1-c --project=nomadkaraoke \
--command="sudo systemctl restart encoding-worker"
```
+87 -20
View File
@@ -38,10 +38,13 @@
│ (jobs) │ │ (files) │ │ (API keys) │ │ Worker (*) │
└──────────┘ └──────────┘ └──────────────┘ └──────────────┘
(*) GCE Encoding Worker: c4d-highcpu-32 VM with AMD EPYC 9B45 (Turin) for
high-performance FFmpeg encoding (4.92x faster than previous c4-standard-8).
(*) GCE Encoding Worker: a ranked pool of 6 x86_64 32-vCPU highcpu machine
families (c4d/c4/n4d/c2d/n2d/n2). c4d-highcpu-32 (AMD EPYC Turin) is the
fastest and stays the primary/preferred pick; the other families are
stockout-resilience fallbacks selected fastest-first with a capacity cooldown.
Used for both final video encoding and preview video generation.
Uses immutable deployment pattern - see infrastructure/encoding-worker/README.md.
Uses immutable deployment pattern - see infrastructure/encoding-worker/README.md
and the "Encoding Worker" section below.
```
### Error Monitor
@@ -292,33 +295,82 @@ karaoke-gen shares a GCP project (`nomadkaraoke`) with karaoke-decide, but uses
**Worker Logs**: Stored in subcollection (`jobs/{job_id}/logs`) instead of embedded array to avoid Firestore's 1MB document size limit. TTL policy auto-deletes logs after 30 days. Feature flag: `USE_LOG_SUBCOLLECTION` (default: true). See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#firestore-document-1mb-limit-with-embedded-arrays).
## Encoding Worker Blue-Green Deployment
## Encoding Worker: blue-green primary pair + multi-family fallback pool
The GCE encoding worker uses a blue-green deployment pattern with two named VMs (`encoding-worker-a` and `encoding-worker-b`).
The GCE encoding worker is a **blue-green primary pair** (`encoding-worker-a`/`-b`,
both c4d-highcpu-32 in us-central1-c) plus a **ranked pool of stockout-resilience
fallback VMs** spanning 6 x86_64 machine families (see "Multi-instance-type pool"
below). **10 VMs total**, all TERMINATED by default and started on demand.
### Architecture
```text
Frontend → Backend (Cloud Run) → Firestore config → Primary VM (encoding-worker-a or b)
↑
Frontend → Backend (Cloud Run) → Firestore config → active worker (primary, or a
↑ capacity-fallback "override")
Cloud Scheduler (5 min) → Cloud Function → checks idle + stops VMs
↑
CI (GitHub Actions) → deploys to secondary → health check → swap Firestore → stop old
CI (GitHub Actions) → selects a fresh "green" (ranked pool) → health + encode test
→ promote (swap primary OR set active_override) → stop old
```
### Multi-instance-type pool (stockout resilience, v0.195.0)
A single machine family can hit a region-wide `ZONE_RESOURCE_POOL_EXHAUSTED`
stockout across every zone at once. To survive that, the fallback fleet spans
**6 families** (c4d, c4, n4d, c2d, n2d, n2 — all `-highcpu-32`, ≥32 GB) across
zones a/b/c/f: `encoding-worker-fallback-a`/`-b` (c4d), `-n2c`/`-n2f` (n2),
`-c4a` (c4), `-n4db` (n4d), `-c2df` (c2d), `-n2da` (n2d).
**Candidate ordering is one shared pure module**,
`backend/services/encoding_worker_preference.py::ordered_candidates(pool, capacity_state)`,
used by BOTH runtime selection and deploy green selection so they can't drift:
- **Fastest-first** by `SPEED_RANK` (c4d preferred whenever it can start).
- **Availability-aware**: a family that recently hit a capacity/stockout error is
recorded in the Firestore `capacity_state` map and **demoted for a 15-min
cooldown**, then re-probed — so we stop wasting ~2 min probing a dead family but
snap back to c4d the moment capacity returns.
The fallback fleet is defined in `infrastructure/config.py::EncodingWorkerConfig.FALLBACKS`
and published to the backend/idle-fn via the `encoding-worker-fallback-vms` secret
(entries: `{vm, zone, ip, machine_type}`).
### Key Components
**Two VMs (a/b)**: Both VMs are TERMINATED by default. One is designated "primary" and serves all encoding traffic. The secondary VM exists solely to receive new deployments before they go live.
**Primary pair + fallback fleet**: All VMs TERMINATED by default. The primary pair
(a/b) does blue-green deploys; the 8 fallbacks absorb capacity stockouts. Runtime
warmup (`encoding_worker_manager.ensure_any_running`) tries the ranked candidates
in order, falling through on stockout and recording an `active_override` when it
lands on a fallback.
**Firestore config doc** (`config/encoding-worker`): Single source of truth for routing. Fields: `primary` (a or b), `ip_a`, `ip_b`, `deploy_in_progress` flag, `activity_a`/`activity_b` timestamps.
**Firestore config doc** (`config/encoding-worker`): source of truth for routing.
Fields: `primary_vm`/`primary_ip`/`primary_version`, `secondary_vm`/…, the
capacity-fallback pointer `active_override_vm`/`_ip`/`_zone`/`_version` (set when a
fallback is serving), `deploy_in_progress`, `last_activity_at`, and the
`capacity_state` map (`{"<machine_type>@<zone>": last_stockout_iso}`). `active_url`
= override IP if set, else primary IP.
**Backend routing** (`encoding_interface.py`): Reads the primary IP from the Firestore config doc with a 30-second TTL cache. All encoding requests route to the current primary.
**Backend routing** (`encoding_interface.py`): reads the active worker IP from the
config doc with a 30-second TTL cache.
**JIT startup**: When a user enters the lyrics review page, the backend starts the primary VM on demand. This gives ~60 seconds of natural warmup time before encoding is needed, avoiding both 24/7 costs and noticeable cold-start latency for the user.
**JIT startup**: entering the lyrics-review page starts the worker on demand
(~60 s natural warmup), avoiding 24/7 cost and cold-start latency.
**Idle auto-shutdown**: A Cloud Function (triggered every 5 minutes by Cloud Scheduler) checks the `activity_*` timestamps in Firestore and stops any VM that has been idle for more than 15 minutes. This is external to the VM — if the encoding worker code itself has a bug, the Cloud Function still shuts it down.
**Idle auto-shutdown**: a Cloud Function (every 5 min) stops idle VMs (primary,
secondary, and every fallback in the secret) based on Firestore activity — external
to the VM so a sick worker still gets stopped.
**Blue-green deploys**: CI builds a new wheel, deploys it to the secondary VM (not the live primary), runs a real encode test against the secondary, then atomically swaps the `primary` field in Firestore to point to the secondary. The old primary is stopped after the swap. If the health check fails, the primary is never updated.
**Capacity-aware deploys**: CI selects a FRESH "green" from the ranked pool via
`deploy_promote.select_green_candidates` (the same preference module) — the c4d
secondary if it can start, else a fallback family, never the currently-serving
override. After a real encode test on the green, it promotes: a c4d-secondary green
does the classic primary swap (and clears any stale override); a fallback green
becomes the `active_override`. The retired VM is drained + stopped; a failed health
check never promotes. Override routing keys off an explicit `is_primary` flag, not
list position. See `infrastructure/encoding-worker/deploy_promote.py`.
**Self-sufficient image**: the Packer image (`infrastructure/packer/scripts/provision.sh`)
bakes the full karaoke-gen dependency tree with **CPU-only torch**, so a fresh VM of
any family boots ready to encode (no first-job dependency install).
### Seeding the Config
@@ -327,17 +379,32 @@ Initial setup or reset:
python scripts/seed-encoding-worker-config.py <ip-a> <ip-b>
```
### Deploy Flow
### Deploy Flow (capacity-aware, v0.194.2+)
```text
1. CI builds new karaoke-gen wheel + pushes to GCS
2. CI starts secondary VM, waits for /health to respond
3. CI runs real encode test against secondary VM
4. If test passes: CI writes secondary name to Firestore `primary` field
5. CI stops the old primary VM
6. If test fails: CI stops secondary VM, primary unchanged
2. CI picks a FRESH "green" from the ranked pool (select_green_candidates →
ordered_candidates): the c4d secondary if it can start, else the fastest
available fallback family — never the currently-serving active_override.
3. CI starts the green, waits for /health, runs a real encode test against it.
4. If the test FAILS: CI stops the green; primary/override unchanged (no promote).
5. If the test PASSES: CI promotes the green:
- green is the c4d secondary → swap it to `primary_vm` (classic blue-green)
AND clear any stale `active_override_*` so
traffic returns to the fresh c4d primary.
- green is a fallback family → set it as `active_override_*` (the serving
worker); leave primary/secondary so c4d
resumes as primary when its capacity returns.
6. CI drains (waits active_jobs==0, bounded) + stops the retired VM — never the
promoted green.
7. Last resort (no separate green can start, e.g. deep stockout): in-place restart
of the current serving override — a brief blip, covered by client auto-resubmit.
```
Override routing keys off an explicit `is_primary` flag (not list position), so a
fallback ranked ahead of a stocked-out primary is still routed correctly. Pure
decision logic + unit tests: `infrastructure/encoding-worker/deploy_promote.py`.
## Video Worker Orchestrator
The Video Worker uses an orchestrator pattern to ensure all features work regardless of encoding backend (local Cloud Run or GCE VM).
+1 -1
View File
@@ -209,7 +209,7 @@ TESTMAIL_NAMESPACE=... # testmail.app namespace
## Encoding Worker Operations
The encoding worker uses a blue-green deployment with two GCE VMs (`encoding-worker-a`, `encoding-worker-b`). See [ARCHITECTURE.md](ARCHITECTURE.md#encoding-worker-blue-green-deployment) for the full design.
The encoding worker is a blue-green primary pair (`encoding-worker-a`/`-b`, c4d) plus a ranked pool of 8 stopped fallback VMs spanning 6 machine families (c4d/c4/n4d/c2d/n2d/n2) for stockout resilience — 10 VMs total. Candidate ordering (runtime + deploy) is the shared `backend/services/encoding_worker_preference.py`. See [ARCHITECTURE.md](ARCHITECTURE.md#encoding-worker-blue-green-primary-pair--multi-family-fallback-pool) for the full design.
### Seed Firestore config (initial setup or reset)
+18 -11
View File
@@ -79,19 +79,26 @@ The `concurrency=1` setting may have been added deliberately for performance rea
### 🔍 GCE Encoding Worker
**Current State:**
- c4d-highcpu-32 VM running 24/7
- Cost: ~$700/month
**Current State (updated v0.195.0 — the opportunities below are largely SHIPPED):**
- No longer a single 24/7 VM. It is a pool of **on-demand** VMs: a c4d-highcpu-32
blue-green primary pair + 8 stopped fallbacks across 6 machine families
(c4d/c4/n4d/c2d/n2d/n2) for stockout resilience.
- **Idle auto-shutdown is live** (JIT start on the lyrics-review page; a Cloud
Function stops idle VMs after 15 min). When idle, cost is just boot disks
(~$10/VM/mo); compute is billed only while encoding — so the historical
"~$700/mo running 24/7" no longer applies.
**Opportunities:**
1. **Spot Instance:** Could save 60-91% (~$420-630/month savings)
- Risk: May be preempted during video encoding jobs
- Mitigation: Implement job restart logic
2. **Auto-scaling:** Start/stop based on video queue depth
- Could scale to zero when no jobs queued
- Startup time: ~2-3 minutes (acceptable for async processing)
**Opportunities (status):**
1. **Spot Instance:** Could save 60-91% — NOT adopted; on-demand chosen so a
fallback can always be *started* when needed (Spot preemption would undermine
the anti-stockout pool during the industry-wide crunch). Kept as a future lever.
2. **Auto-scaling / scale-to-zero:** ✅ IMPLEMENTED via JIT start + idle-shutdown
Cloud Function (start/stop on demand rather than queue-depth-based).
**Recommendation:** Test Spot instance first (lower risk), then consider auto-scaling
**Recommendation:** Stay on-demand (already scales to zero via idle-shutdown, which
captured most of the savings). Do NOT move to Spot for now — Spot preemption plus
the industry-wide stockout would undermine the anti-stockout fallback pool. Revisit
Spot only as a future experiment once capacity/resiliency is validated.
### 🔧 Artifact Registry Cleanup (Quick Win)
+10
View File
@@ -406,6 +406,16 @@ When two sequential human review steps can be combined into one, do it. We origi
### Machine-family diversification (August 2026)
> **Superseded by v0.195.0** — the "Multi-instance-type encode pool" lesson at the
> top of this file. The 2-family fallback fleet (c4d + n2) described below was
> broadened to a **ranked 6-family pool** (c4d/c4/n4d/c2d/n2d/n2), and runtime +
> deploy selection now route through the shared
> `backend/services/encoding_worker_preference.py` (fastest-first + 15-min capacity
> cooldown), not "primary first, fallbacks in declared order". The incident + root
> lesson below remain accurate history; the "Fix (Option B)" and "Architecture
> summary" specifics (2 families / 6 VMs / declared-order iteration) are outdated —
> see the v0.195.0 lesson for the current mechanism.
**Incident 2026-08-12:** `c4d-highcpu-32` hit a **region-wide `ZONE_RESOURCE_POOL_EXHAUSTED` stockout across `us-central1-a`, `-b` AND `-c` simultaneously.** Because the primary pair *and* both fallbacks were all the same machine family, every lane in `ensure_any_running` failed at once. Effects:
- **Preview** (`POST /api/review/{id}/preview-video`) fell through to slow local Cloud-Run encoding (~130–160 s), which **exceeds Cloudflare's ~100 s edge timeout** → browser got a **524 / "NetworkError when attempting to fetch resource"** even though the backend actually returned 200. The 524 and the NetworkError are the same event; backend logs showing 200 are misleading because the client was already disconnected.
- **Render** parked in `RENDER_PENDING_CAPACITY` and auto-retried (safe only if capacity returns within 24 h).
+19 -8
View File
@@ -64,7 +64,7 @@ Since v0.184.2, in-flight status polls are also **pinned to the worker that acce
Since **v0.194.0**, a mid-run job loss is also **auto-recovered**: the encoding worker processes heavy renders/encodes **one at a time** (`heavy_executor`, `ENCODING_HEAVY_CONCURRENCY=1`) so it can no longer OOM from concurrent 4K encodes (the trigger for the 2026-08-15 Arctic Monkeys batch failure — 3 concurrent encodes OOM-killed the 32 GB fallback worker and restarted it, wiping its in-memory jobs). If the worker *does* restart mid-render (deploy/crash), `wait_for_completion` now raises `EncodingJobLostError` and the render/encode worker **resubmits the job under a fresh `<id>_retry_<hex8>`** (bounded by `ENCODING_RESUBMIT_MAX=2`) instead of failing. To confirm serialization is live on a worker: `curl -s localhost:8080/health` shows `queue_length` growing while `active_jobs` stays 1 under load. **Note:** `main.py` ships in the wheel but the running uvicorn process only picks up worker-side changes (executor split, `queue_position`) on a **fresh boot / service restart** — `ensure_latest_wheel` alone does not reload the app process.
Since **v0.194.2**, the CI deploy handles that restart automatically even during a c4d Spot stockout. The old blue-green only targeted the c4d primary/secondary; when both were down it rolled back and **never refreshed the serving n2 fallback** (recorded as `active_override_vm`), so worker-side changes didn't reach prod until a manual restart (`primary_version` in the config doc went stale). The deploy is now **capacity-aware** (`infrastructure/encoding-worker/deploy_promote.py`): it validates the new wheel on a *fresh* green worker — the c4d secondary if it starts, else a different n2 fallback (never the current override, so it keeps serving) — then **promotes** the green (primary/secondary swap for a c4d green, or sets `active_override` for a fallback green) and drains+stops the retired worker. A c4d green also **clears a stale override** so traffic returns to the fresh primary. Zero-downtime, and it works while c4d is exhausted. Last resort if no separate green can start: an in-place restart of the serving override (brief blip, covered by auto-resubmit).
Since **v0.194.2**, the CI deploy handles that restart automatically even during a c4d Spot stockout. The old blue-green only targeted the c4d primary/secondary; when both were down it rolled back and **never refreshed the serving n2 fallback** (recorded as `active_override_vm`), so worker-side changes didn't reach prod until a manual restart (`primary_version` in the config doc went stale). The deploy is now **capacity-aware** (`infrastructure/encoding-worker/deploy_promote.py`): it validates the new wheel on a *fresh* green worker selected from the **ranked 6-family pool** (`select_green_candidates` → the shared `encoding_worker_preference.ordered_candidates` — fastest-first with capacity cooldown; the c4d secondary if it can start, else the fastest available fallback family, never the current override so it keeps serving) — then **promotes** the green (primary/secondary swap for a c4d green, or sets `active_override` for a fallback green) and drains+stops the retired worker. A c4d green also **clears a stale override** so traffic returns to the fresh primary. Zero-downtime, and it works while c4d is exhausted. Last resort if no separate green can start: an in-place restart of the serving override (brief blip, covered by auto-resubmit).
---
@@ -94,13 +94,18 @@ The script is idempotent — re-running it skips regeneration if the CDG ZIP alr
**Cause:** Worker picks up a new deploy but doesn't restart automatically.
Target the **currently-serving** VM — `active_override_vm` in `config/encoding-worker`
if set, else `primary_vm` (with its zone). There is no VM literally named
`encoding-worker`; the examples below use the primary `encoding-worker-a` — swap in
the active VM/zone if a fallback is serving.
```bash
# Check current wheel version
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
# Check current wheel version (example: the primary a in us-central1-c)
gcloud compute ssh encoding-worker-a --zone=us-central1-c --project=nomadkaraoke \
--command="curl -s http://localhost:8080/health | python3 -m json.tool"
# Restart to pick up latest wheel
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
gcloud compute ssh encoding-worker-a --zone=us-central1-c --project=nomadkaraoke \
--command="sudo systemctl restart encoding-worker"
```
@@ -110,7 +115,9 @@ gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
**Symptoms:** A render fails with `OutputGenerator not available: No module named 'tenacity'. The karaoke-gen wheel must be installed. Check that ensure_latest_wheel() succeeded.` (or another missing package). Retries hit the same VM and fail identically.
**Cause:** A freshly-provisioned encoding-worker VM comes up with only a handful of packages baked into the Packer image, and boot installs the wheel with `--no-deps`. The full karaoke-gen dependency tree (torch, langchain, tenacity, …) is installed lazily at the first job by `ensure_latest_wheel()`. If that install fails or partially completes, the venv is left importable-but-incomplete and the render dies on the first missing import. First seen 2026-08-13 (job 233b9536) on a fresh `n2-highcpu-32` fallback added in #903.
**Largely fixed in v0.195.0 (dep-bake):** the Packer image now bakes the **full** karaoke-gen dependency tree (with CPU-only torch) into the venv at build time, so a freshly-provisioned VM of any family boots self-sufficient — no first-job dependency install. This failure mode should now only appear on a VM whose boot disk predates the dep-bake image, or a venv left partial by an interrupted repair. The `ensure_latest_wheel()` runtime backstop below still exists for those cases.
**Original cause (pre-v0.195.0):** A freshly-provisioned VM came up with only a handful of packages baked in, and boot installed the wheel with `--no-deps`. The full dependency tree (torch, langchain, tenacity, …) was installed lazily at the first job by `ensure_latest_wheel()`; a failed/partial install left the venv importable-but-incomplete and the render died on the first missing import. First seen 2026-08-13 (job 233b9536) on a fresh `n2-highcpu-32` fallback added in #903.
**Fixed in v0.192.7:** `ensure_latest_wheel()` now downloads only the single latest wheel (was: copying the entire ~6 GiB `wheels/` dir every job with a 60 s timeout), retries the install (3× at 900 s), and **verifies the render/encode import chain** before returning — a clean `pip` exit is no longer trusted. Callers fail the job with a clear retryable error instead of proceeding. Fresh workers pick this up via the wheel-at-boot.
@@ -281,7 +288,7 @@ gcloud compute instances describe encoding-worker-a \
**Background — what this state means:** GCE returned `ZONE_RESOURCE_POOL_EXHAUSTED` or transient `503 SERVICE_UNAVAILABLE` from `compute.instances.start` on every encoding-worker VM. The render worker parks the job in `RENDER_PENDING_CAPACITY` instead of failing it; Cloud Scheduler retries it every 5 min via `/api/internal/retry-pending-render-jobs`. Hard timeout is 24 hours (then transitions to `failed` with a clear permanent-failure message).
The fallback fleet is diversified by **machine family** (since incident 2026-08-12): `c4d-highcpu-32` primaries in `us-central1-c` + `c4d` fallbacks in `-a`/`-b` + `n2-highcpu-32` fallbacks (`encoding-worker-fallback-n2c`/`-n2f`) in `-c`/`-f`. The n2 pool is much deeper, so a full `c4d` region-wide stockout (which took out a/b/c at once on 2026-08-12) should now still find n2 capacity. If you see EVERY candidate (c4d **and** n2) return stockout, that's a genuinely severe regional event — wait, or add a cross-region fallback.
The fallback fleet is diversified across **6 machine families** (broadened to the full pool in v0.195.0): `c4d-highcpu-32` primaries (`encoding-worker-a`/`-b`) in `us-central1-c`, plus 8 stopped fallbacks — `c4d` (`-fallback-a`/`-b`, zones a/b), `n2` (`-n2c`/`-n2f`, zones c/f), `c4` (`-c4a`, zone a), `n4d` (`-n4db`, zone b), `c2d` (`-c2df`, zone f), `n2d` (`-n2da`, zone a). Candidates are tried **fastest-first with a 15-min capacity cooldown** (a family that stocks out is demoted, then re-probed) — see `backend/services/encoding_worker_preference.py`. A single-family region-wide stockout (which took out c4d a/b/c at once on 2026-08-12) now still finds capacity in the 5 other families. If you see EVERY family return stockout, that's a genuinely severe regional event — wait, or add a cross-region fallback. NOTE: creating a *new* fallback VM (or re-creating a deleted one) still needs a one-time boot allocation, so a deep enough crunch can even block provisioning — a `pulumi up` may show the missing VM as "to create / errored" until capacity returns.
**Related symptom — preview 524 / "NetworkError":** the same capacity exhaustion also makes `POST /api/review/{id}/preview-video` fall back to slow local encoding (~130–160 s), which exceeds Cloudflare's ~100 s edge timeout → the browser shows a **524** or Firefox *"NetworkError when attempting to fetch resource"* even though the backend returns 200. If a user reports the review preview failing, check for a concurrent stockout; the already-rendered preview mp4 (if any) is fetchable via `GET /api/review/{id}/preview-video/{hash}` (302 → signed GCS URL) with a Bearer admin token. Completing the review does **not** require the preview.
@@ -309,14 +316,18 @@ gcloud logging read 'protoPayload.methodName:"instances.start" AND "ZONE_RESOURC
--project=nomadkaraoke --freshness=30m --limit=5 \
--format='value(timestamp, protoPayload.resourceName)'
# Try starting a fallback in each family; if BOTH c4d and n2 refuse, it's severe — wait.
# Try starting a fallback in each of the 6 families; if ALL refuse, it's severe — wait.
# Synchronous (no --async) so the start op surfaces stockout inline, and we capture
# gcloud's own exit status rather than piping (a pipe would return sed's status).
for VM_ZONE in \
"encoding-worker-fallback-a:us-central1-a" \
"encoding-worker-fallback-b:us-central1-b" \
"encoding-worker-fallback-n2c:us-central1-c" \
"encoding-worker-fallback-n2f:us-central1-f"; do
"encoding-worker-fallback-n2f:us-central1-f" \
"encoding-worker-fallback-c4a:us-central1-a" \
"encoding-worker-fallback-n4db:us-central1-b" \
"encoding-worker-fallback-c2df:us-central1-f" \
"encoding-worker-fallback-n2da:us-central1-a"; do
VM="${VM_ZONE%%:*}"; ZONE="${VM_ZONE##*:}"
if OUT=$(gcloud compute instances start "$VM" --zone="$ZONE" --project=nomadkaraoke 2>&1); then
echo "$VM: OK (started — remember to stop it)"
@@ -2,7 +2,32 @@
**Date:** 2026-08-15
**Branch:** `feat/sess-20260815-1915-encoding-instance-type-diversity`
**Status:** IMPLEMENTED (v0.195.0) — code + IaC + Packer landed; operator rollout steps below still pending.
**Status:** SHIPPED (v0.195.0 #911 + v0.195.1 #912) — code + IaC + Packer merged & rolled out.
## Rollout outcome (2026-08-16)
- ✅ Backend 0.195.1 live on Cloud Run; encode path healthy (c4d primary `encoding-worker-a`
RUNNING 0.195.1 after c4d capacity returned to us-central1-c). `capacity_state` field live.
- ✅ Encoding-worker image rebuilt with the CPU-torch dep-bake; **validated end-to-end** — a fresh
`c4a` booted healthy in ~24s and encoded a preview in <6s (no first-job dep install).
- ✅ **All 8 fallback VMs provisioned** (stopped), so the full 6-family pool is live:
`-a`/`-b` (c4d), `-n2c`/`-n2f` (n2), `-c4a` (c4), `-n2da` (n2d), `-c2df` (c2d), `-n4db` (n4d).
Pulumi state consistent (`c4a`, orphaned by a killed apply, was `pulumi import`ed + reconciled).
Secret at **version 5** (8 entries, all with `machine_type`).
- ⏳→✅ **`c2df` + `n4db` were initially deferred** (created stopped VM still needs a one-time boot
allocation, so the 2026-08-15 stockout in zones f/b blocked provisioning). An autonomous
10-minute retry loop finished them on **2026-08-16 ~07:48** when zones f/b left stockout, and
added both to the secret (v5). **Historical note for a future crunch:** if a fallback VM is
missing/deleted, re-create it with the FULL Pulumi URNs (not an ellipsis) — e.g.
`cd infrastructure && pulumi up --yes --target 'urn:pulumi:prod::karaoke-gen-infrastructure::gcp:compute/instance:Instance::encoding-worker-fallback-c2df' --target 'urn:pulumi:prod::karaoke-gen-infrastructure::gcp:compute/instance:Instance::encoding-worker-fallback-n4db'`
(use the write-token env from the rollout notes) — then add it to the `encoding-worker-fallback-vms`
secret with its `machine_type`, but **never add a VM to the secret before it exists** (a
non-existent candidate → `get_vm_status` 404 → breaks warmup).
- Not done: measured `SPEED_RANK` benchmark run (arch-seeded; a 2026-08-16 benchmark was too noisy
to apply — see the CPU-platform-variance findings); MIG; reservations.
---
**Context:** memory `project_gen_encoding_instance_type_diversity` + `_worker_serialization` + `_machine_family_diversity` + `_wheel_deps_missing`
## Goal
@@ -0,0 +1,141 @@
# Handoff: Investigate prod finalization CPU efficiency / utilization — 2026-08-16
**Status:** OPEN investigation (not started). Written for a fresh Claude session —
self-contained; all facts you need are inlined below or in the linked repo docs.
**Origin:** surfaced by the encode-worker multi-instance-type benchmark
(`docs/archive/2026-08-15-encoding-instance-type-diversity-plan.md`; benchmark
baselines inlined under "Reproducing the benchmark" below).
## The question
Prod video **finalization** (the `/encode` path that produces
`mp4_4k_lossless + mp4_4k_lossy + mp4_720p`) — is it using the 32-vCPU
`*-highcpu-32` workers efficiently? The trigger was the stale
`"ffmpeg_threads": 8` (with a comment referencing the retired `c4-standard-8`)
in the job config, which looked like it might cap ffmpeg to 8 of 32 cores.
**Static analysis has already answered the first-order question (do NOT re-derive
it — verify then move on):** `ffmpeg_threads` is **dead config**. It is set at
`backend/workers/video_worker.py:131` but **no code reads it** (grep for consumers
returns nothing; `run_encoding()`'s `EncodingConfig(...)` at
`backend/services/gce_encoding/main.py:1111` never passes it), and the actual
finalization ffmpeg commands in `backend/services/local_encoding_service.py`
(`encode_all_formats` → `encode_lossless_mp4`/`encode_lossy_mp4`/`encode_720p`,
around lines 343 / 396 / 471) carry **no `-threads` flag at all** → ffmpeg falls
back to its default (`-threads 0`, i.e. auto ≈ all cores). So finalization is
**almost certainly NOT capped at 8 cores**; the "leaving 75% idle" fear is likely
unfounded. (Separate, unrelated paths that DO hardcode threads — don't confuse
them: `main.py:372` uses `-threads 8` in the **preview** `/encode-preview`
480×270 path; `local_preview_encoding_service.py:310` uses `-threads 0`.)
So this is now a **narrower efficiency + cleanup** task, not a "fix the cap" task.
## What to determine (in order)
1. **Confirm the static finding empirically** (x264's `-threads 0` "auto" is a
heuristic — it does NOT always scale to all cores; historically it caps frame
threads, with diminishing returns past ~16). Drive a real finalization and
capture **process-level** evidence, not just host-wide load:
- the **deployed ffmpeg argv** for each format (`ps -eo pid,args | grep ffmpeg`
on the worker mid-encode, or add a log line) — confirm no `-threads` and see
what x264 chose;
- **per-process / per-thread** CPU% and RSS (`top -H -p <ffmpeg-pid>`,
`pidstat -t -p <pid> 2`), NOT `mpstat -P ALL` alone — host-wide utilization
can't tell an 8-thread cap from serial `encode_all_formats` stages or
unrelated work;
- **per-format stage timings** (which of lossless/lossy/720p dominates).
Run all of this against the SAME worker you drive the encode to (see the
binding note in "Reproducing the benchmark").
2. **Check whether the 3 formats encode SERIALLY or in PARALLEL** in
`encode_all_formats` (`backend/services/local_encoding_service.py`). If serial,
wall-time ≈ sum of the three even if each saturates the cores — the real lever
may be **overlapping independent format encodes** rather than more threads. But
see the RAM constraint below before parallelizing.
3. **Decide the fix (likely small) and validate it:**
- **Cleanup regardless:** remove the dead `ffmpeg_threads` field + stale comment
at `backend/workers/video_worker.py:131` (it misleads; that is the "documentation-
only" close-out CodeRabbit flagged). If you'd rather *use* it, wire it through
`EncodingConfig` → the `local_encoding_service.py` ffmpeg commands as an explicit
`-threads`; otherwise delete it.
- If step 1 shows real idle cores (x264 auto under-scales), the win is an explicit
high `-threads` (or `-x264-params threads=N`) or overlapping formats — validate
with a before/after re-benchmark on the SAME input+VM.
- If step 1 shows cores already saturated, close this out as "already efficient;
removed dead config" and stop.
## Hard constraints — do NOT regress these
- **Heavy-lane serialization stays.** `ENCODING_HEAVY_CONCURRENCY=1` (one heavy
encode at a time per worker) exists because 3 **concurrent** 4K encodes OOM-killed
the 32 GB worker (incident 2026-08-15; full write-up in `docs/LESSONS-LEARNED.md`
→ "Concurrent encodes OOM-killed the worker → lost renders").
Raising *threads within a single encode* is **orthogonal** to that and does not
reintroduce concurrency — but it DOES raise a single encode's **peak RAM** (more
parallel ffmpeg slices / parallel format encodes). Watch RSS on a 32 GB floor VM
(n2/n2d fallbacks are 32 GB; c4d/c4/n4d are 60–64 GB). A single 4K lossless encode
was ~18 GB; leave headroom.
- **Lossless 4K is the heavy one** (`pcm_s16le`, huge intermediate) — profile that
format specifically.
## Reproducing the benchmark (for before/after numbers)
The benchmark used a real completed job's inputs as a canonical `/encode` input.
It was deleted after the run (`gs://…/bench/` cleaned up). To recreate:
```bash
J=gs://karaoke-gen-storage-nomadkaraoke/jobs/ffc17eb7 # "Glen Campbell - A Better Place"
I=gs://karaoke-gen-storage-nomadkaraoke/bench/encode-input
gsutil -q cp "$J/videos/with_vocals.mkv" "$I/videos/with_vocals.mkv" # 119 MB rendered 4K karaoke video
gsutil -q cp "$J/screens/title.png" "$I/screens/title.png"
gsutil -q cp "$J/screens/end.png" "$I/screens/end.png"
gsutil -q cp "$J/stems/instrumental_clean.flac" "$I/stems/instrumental_clean.flac"
gsutil -q cp "$J/style/style_params.json" "$I/style/style_params.json"
gsutil -q cp "$J/lyrics/karaoke.ass" "$I/lyrics/karaoke.ass"
```
**Bind the encode to the worker you observe.** Pick ONE stopped fallback (e.g.
`encoding-worker-fallback-c4a`, us-central1-a), start it, and send BOTH the
`/encode` POST and the `/status` polls **directly to that worker's external IP**
(`http://<IP>:8080/…`, not through the backend/`api.nomadkaraoke.com` — the
backend may route to a different active worker). SSH the SAME VM for the
process-level probes in step 1, so the CPU/RSS you measure is the encode you drove.
Get the IP: `gcloud compute instances describe <vm> --zone=<z>
--format='value(networkInterfaces[0].accessConfigs[0].natIP)'`.
`POST http://<IP>:8080/encode` body (`X-API-Key` from
`gcloud secrets versions access latest --secret=encoding-worker-api-key`):
```json
{"job_id":"cpu-probe-1","input_gcs_path":"gs://…/bench/encode-input/",
"output_gcs_path":"gs://…/bench/out/probe/",
"encoding_config":{"formats":["mp4_4k_lossless","mp4_4k_lossy","mp4_720p"],
"base_name":"Glen Campbell - A Better Place","artist":"Glen Campbell",
"title":"A Better Place","instrumental_selection":"clean",
"existing_instrumental":null}}
```
(`ffmpeg_threads` deliberately omitted from the body above — it is dead config;
see "The question".) Poll `GET http://<IP>:8080/status/{job_id}` to `complete`. Baseline medians from 2026-08-16 (full
3-format finalization of this ~4-min song; the runs sent `ffmpeg_threads:8` in the
body but the worker ignored it, so these reflect ffmpeg's default all-core threading):
c4-Emerald ≈ **139s**, n2d-Milan ≈ **139s**, n2-CascadeLake ≈ **247s**, n2-IceLake
≈ **92s** (c4d unmeasured — could not start a 2nd c4d in the stockout). The huge
n2 spread — **92s vs 247s for the identical `n2-highcpu-32` SKU** — is because n2
spans Intel Ice Lake (fast) and Cascade Lake (slow) and GCE assigns either; that
intra-family variance exceeds the between-family differences. Separate actionable
finding: pin `min_cpu_platform` (e.g. "Intel Ice Lake" / "AMD Milan") on n2/n2d
fallbacks for predictable speed, or rank them lower to reflect the gamble. Also
note the counterintuitive result that c4 (newest Intel Emerald) ≈ n2d (Milan) and
did NOT beat n2 Ice Lake for this lossless-4K-heavy workload — so "newest = fastest"
does not hold here; measure, don't assume.
**Operational hygiene:** benchmark VMs are stopped fallbacks — start, probe, then
STOP them (never the live-serving primary; the idle-shutdown fn also handles it).
Delete `gs://…/bench/**` when done. Auth for VM start/stop: `gcloud` as
`admin@nomadkaraoke.com`.
## Expected outcome
Either: (a) confirm CPU is already saturated (`ffmpeg_threads` is dead config) →
close this out and just fix the stale comment; or (b) confirm an ~2–4× idle-core
gap → raise threads / parallelize formats, validate the speedup and RAM headroom,
ship it. Bump `pyproject.toml` version; if the fix is in the wheel's
`LocalEncodingService`, it ships via the normal wheel → `ensure_latest_wheel` path.