mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
0539479281 | ||
|
|
b7c4edc7fc |
No files matched your search
@@ -145,13 +145,18 @@ EOF
|
||||
```
|
||||
|
||||
### GCE Encoding Worker
|
||||
The worker is a c4d blue-green primary pair (`encoding-worker-a`/`-b`, us-central1-c)
|
||||
plus 8 stopped fallback VMs across 6 machine families for stockout resilience
|
||||
(`encoding-worker-fallback-{a,b,n2c,n2f,c4a,n4db,c2df,n2da}`). The currently-serving
|
||||
VM is `active_override_vm` in `config/encoding-worker` if set, else `primary_vm`.
|
||||
Use that VM name + its zone below (there is no VM literally named `encoding-worker`):
|
||||
```bash
|
||||
# Check health (includes wheel_version)
|
||||
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
|
||||
# Check health (includes wheel_version). Example uses the primary pair's VM.
|
||||
gcloud compute ssh encoding-worker-a --zone=us-central1-c --project=nomadkaraoke \
|
||||
--command="curl -s http://localhost:8080/health"
|
||||
|
||||
# Restart service (pick up new wheel after a deploy)
|
||||
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
|
||||
# Restart the service (pick up new worker-side code after a deploy)
|
||||
gcloud compute ssh encoding-worker-a --zone=us-central1-c --project=nomadkaraoke \
|
||||
--command="sudo systemctl restart encoding-worker"
|
||||
```
|
||||
|
||||
|
||||
+87
-20
@@ -38,10 +38,13 @@
|
||||
│ (jobs) │ │ (files) │ │ (API keys) │ │ Worker (*) │
|
||||
└──────────┘ └──────────┘ └──────────────┘ └──────────────┘
|
||||
|
||||
(*) GCE Encoding Worker: c4d-highcpu-32 VM with AMD EPYC 9B45 (Turin) for
|
||||
high-performance FFmpeg encoding (4.92x faster than previous c4-standard-8).
|
||||
(*) GCE Encoding Worker: a ranked pool of 6 x86_64 32-vCPU highcpu machine
|
||||
families (c4d/c4/n4d/c2d/n2d/n2). c4d-highcpu-32 (AMD EPYC Turin) is the
|
||||
fastest and stays the primary/preferred pick; the other families are
|
||||
stockout-resilience fallbacks selected fastest-first with a capacity cooldown.
|
||||
Used for both final video encoding and preview video generation.
|
||||
Uses immutable deployment pattern - see infrastructure/encoding-worker/README.md.
|
||||
Uses immutable deployment pattern - see infrastructure/encoding-worker/README.md
|
||||
and the "Encoding Worker" section below.
|
||||
```
|
||||
|
||||
### Error Monitor
|
||||
@@ -292,33 +295,82 @@ karaoke-gen shares a GCP project (`nomadkaraoke`) with karaoke-decide, but uses
|
||||
|
||||
**Worker Logs**: Stored in subcollection (`jobs/{job_id}/logs`) instead of embedded array to avoid Firestore's 1MB document size limit. TTL policy auto-deletes logs after 30 days. Feature flag: `USE_LOG_SUBCOLLECTION` (default: true). See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#firestore-document-1mb-limit-with-embedded-arrays).
|
||||
|
||||
## Encoding Worker Blue-Green Deployment
|
||||
## Encoding Worker: blue-green primary pair + multi-family fallback pool
|
||||
|
||||
The GCE encoding worker uses a blue-green deployment pattern with two named VMs (`encoding-worker-a` and `encoding-worker-b`).
|
||||
The GCE encoding worker is a **blue-green primary pair** (`encoding-worker-a`/`-b`,
|
||||
both c4d-highcpu-32 in us-central1-c) plus a **ranked pool of stockout-resilience
|
||||
fallback VMs** spanning 6 x86_64 machine families (see "Multi-instance-type pool"
|
||||
below). **10 VMs total**, all TERMINATED by default and started on demand.
|
||||
|
||||
### Architecture
|
||||
|
||||
```text
|
||||
Frontend → Backend (Cloud Run) → Firestore config → Primary VM (encoding-worker-a or b)
|
||||
↑
|
||||
Frontend → Backend (Cloud Run) → Firestore config → active worker (primary, or a
|
||||
↑ capacity-fallback "override")
|
||||
Cloud Scheduler (5 min) → Cloud Function → checks idle + stops VMs
|
||||
↑
|
||||
CI (GitHub Actions) → deploys to secondary → health check → swap Firestore → stop old
|
||||
CI (GitHub Actions) → selects a fresh "green" (ranked pool) → health + encode test
|
||||
→ promote (swap primary OR set active_override) → stop old
|
||||
```
|
||||
|
||||
### Multi-instance-type pool (stockout resilience, v0.195.0)
|
||||
|
||||
A single machine family can hit a region-wide `ZONE_RESOURCE_POOL_EXHAUSTED`
|
||||
stockout across every zone at once. To survive that, the fallback fleet spans
|
||||
**6 families** (c4d, c4, n4d, c2d, n2d, n2 — all `-highcpu-32`, ≥32 GB) across
|
||||
zones a/b/c/f: `encoding-worker-fallback-a`/`-b` (c4d), `-n2c`/`-n2f` (n2),
|
||||
`-c4a` (c4), `-n4db` (n4d), `-c2df` (c2d), `-n2da` (n2d).
|
||||
|
||||
**Candidate ordering is one shared pure module**,
|
||||
`backend/services/encoding_worker_preference.py::ordered_candidates(pool, capacity_state)`,
|
||||
used by BOTH runtime selection and deploy green selection so they can't drift:
|
||||
- **Fastest-first** by `SPEED_RANK` (c4d preferred whenever it can start).
|
||||
- **Availability-aware**: a family that recently hit a capacity/stockout error is
|
||||
recorded in the Firestore `capacity_state` map and **demoted for a 15-min
|
||||
cooldown**, then re-probed — so we stop wasting ~2 min probing a dead family but
|
||||
snap back to c4d the moment capacity returns.
|
||||
|
||||
The fallback fleet is defined in `infrastructure/config.py::EncodingWorkerConfig.FALLBACKS`
|
||||
and published to the backend/idle-fn via the `encoding-worker-fallback-vms` secret
|
||||
(entries: `{vm, zone, ip, machine_type}`).
|
||||
|
||||
### Key Components
|
||||
|
||||
**Two VMs (a/b)**: Both VMs are TERMINATED by default. One is designated "primary" and serves all encoding traffic. The secondary VM exists solely to receive new deployments before they go live.
|
||||
**Primary pair + fallback fleet**: All VMs TERMINATED by default. The primary pair
|
||||
(a/b) does blue-green deploys; the 8 fallbacks absorb capacity stockouts. Runtime
|
||||
warmup (`encoding_worker_manager.ensure_any_running`) tries the ranked candidates
|
||||
in order, falling through on stockout and recording an `active_override` when it
|
||||
lands on a fallback.
|
||||
|
||||
**Firestore config doc** (`config/encoding-worker`): Single source of truth for routing. Fields: `primary` (a or b), `ip_a`, `ip_b`, `deploy_in_progress` flag, `activity_a`/`activity_b` timestamps.
|
||||
**Firestore config doc** (`config/encoding-worker`): source of truth for routing.
|
||||
Fields: `primary_vm`/`primary_ip`/`primary_version`, `secondary_vm`/…, the
|
||||
capacity-fallback pointer `active_override_vm`/`_ip`/`_zone`/`_version` (set when a
|
||||
fallback is serving), `deploy_in_progress`, `last_activity_at`, and the
|
||||
`capacity_state` map (`{"<machine_type>@<zone>": last_stockout_iso}`). `active_url`
|
||||
= override IP if set, else primary IP.
|
||||
|
||||
**Backend routing** (`encoding_interface.py`): Reads the primary IP from the Firestore config doc with a 30-second TTL cache. All encoding requests route to the current primary.
|
||||
**Backend routing** (`encoding_interface.py`): reads the active worker IP from the
|
||||
config doc with a 30-second TTL cache.
|
||||
|
||||
**JIT startup**: When a user enters the lyrics review page, the backend starts the primary VM on demand. This gives ~60 seconds of natural warmup time before encoding is needed, avoiding both 24/7 costs and noticeable cold-start latency for the user.
|
||||
**JIT startup**: entering the lyrics-review page starts the worker on demand
|
||||
(~60 s natural warmup), avoiding 24/7 cost and cold-start latency.
|
||||
|
||||
**Idle auto-shutdown**: A Cloud Function (triggered every 5 minutes by Cloud Scheduler) checks the `activity_*` timestamps in Firestore and stops any VM that has been idle for more than 15 minutes. This is external to the VM — if the encoding worker code itself has a bug, the Cloud Function still shuts it down.
|
||||
**Idle auto-shutdown**: a Cloud Function (every 5 min) stops idle VMs (primary,
|
||||
secondary, and every fallback in the secret) based on Firestore activity — external
|
||||
to the VM so a sick worker still gets stopped.
|
||||
|
||||
**Blue-green deploys**: CI builds a new wheel, deploys it to the secondary VM (not the live primary), runs a real encode test against the secondary, then atomically swaps the `primary` field in Firestore to point to the secondary. The old primary is stopped after the swap. If the health check fails, the primary is never updated.
|
||||
**Capacity-aware deploys**: CI selects a FRESH "green" from the ranked pool via
|
||||
`deploy_promote.select_green_candidates` (the same preference module) — the c4d
|
||||
secondary if it can start, else a fallback family, never the currently-serving
|
||||
override. After a real encode test on the green, it promotes: a c4d-secondary green
|
||||
does the classic primary swap (and clears any stale override); a fallback green
|
||||
becomes the `active_override`. The retired VM is drained + stopped; a failed health
|
||||
check never promotes. Override routing keys off an explicit `is_primary` flag, not
|
||||
list position. See `infrastructure/encoding-worker/deploy_promote.py`.
|
||||
|
||||
**Self-sufficient image**: the Packer image (`infrastructure/packer/scripts/provision.sh`)
|
||||
bakes the full karaoke-gen dependency tree with **CPU-only torch**, so a fresh VM of
|
||||
any family boots ready to encode (no first-job dependency install).
|
||||
|
||||
### Seeding the Config
|
||||
|
||||
@@ -327,17 +379,32 @@ Initial setup or reset:
|
||||
python scripts/seed-encoding-worker-config.py <ip-a> <ip-b>
|
||||
```
|
||||
|
||||
### Deploy Flow
|
||||
### Deploy Flow (capacity-aware, v0.194.2+)
|
||||
|
||||
```text
|
||||
1. CI builds new karaoke-gen wheel + pushes to GCS
|
||||
2. CI starts secondary VM, waits for /health to respond
|
||||
3. CI runs real encode test against secondary VM
|
||||
4. If test passes: CI writes secondary name to Firestore `primary` field
|
||||
5. CI stops the old primary VM
|
||||
6. If test fails: CI stops secondary VM, primary unchanged
|
||||
2. CI picks a FRESH "green" from the ranked pool (select_green_candidates →
|
||||
ordered_candidates): the c4d secondary if it can start, else the fastest
|
||||
available fallback family — never the currently-serving active_override.
|
||||
3. CI starts the green, waits for /health, runs a real encode test against it.
|
||||
4. If the test FAILS: CI stops the green; primary/override unchanged (no promote).
|
||||
5. If the test PASSES: CI promotes the green:
|
||||
- green is the c4d secondary → swap it to `primary_vm` (classic blue-green)
|
||||
AND clear any stale `active_override_*` so
|
||||
traffic returns to the fresh c4d primary.
|
||||
- green is a fallback family → set it as `active_override_*` (the serving
|
||||
worker); leave primary/secondary so c4d
|
||||
resumes as primary when its capacity returns.
|
||||
6. CI drains (waits active_jobs==0, bounded) + stops the retired VM — never the
|
||||
promoted green.
|
||||
7. Last resort (no separate green can start, e.g. deep stockout): in-place restart
|
||||
of the current serving override — a brief blip, covered by client auto-resubmit.
|
||||
```
|
||||
|
||||
Override routing keys off an explicit `is_primary` flag (not list position), so a
|
||||
fallback ranked ahead of a stocked-out primary is still routed correctly. Pure
|
||||
decision logic + unit tests: `infrastructure/encoding-worker/deploy_promote.py`.
|
||||
|
||||
## Video Worker Orchestrator
|
||||
|
||||
The Video Worker uses an orchestrator pattern to ensure all features work regardless of encoding backend (local Cloud Run or GCE VM).
|
||||
|
||||
+1
-1
@@ -209,7 +209,7 @@ TESTMAIL_NAMESPACE=... # testmail.app namespace
|
||||
|
||||
## Encoding Worker Operations
|
||||
|
||||
The encoding worker uses a blue-green deployment with two GCE VMs (`encoding-worker-a`, `encoding-worker-b`). See [ARCHITECTURE.md](ARCHITECTURE.md#encoding-worker-blue-green-deployment) for the full design.
|
||||
The encoding worker is a blue-green primary pair (`encoding-worker-a`/`-b`, c4d) plus a ranked pool of 8 stopped fallback VMs spanning 6 machine families (c4d/c4/n4d/c2d/n2d/n2) for stockout resilience — 10 VMs total. Candidate ordering (runtime + deploy) is the shared `backend/services/encoding_worker_preference.py`. See [ARCHITECTURE.md](ARCHITECTURE.md#encoding-worker-blue-green-primary-pair--multi-family-fallback-pool) for the full design.
|
||||
|
||||
### Seed Firestore config (initial setup or reset)
|
||||
|
||||
|
||||
@@ -79,19 +79,26 @@ The `concurrency=1` setting may have been added deliberately for performance rea
|
||||
|
||||
### 🔍 GCE Encoding Worker
|
||||
|
||||
**Current State:**
|
||||
- c4d-highcpu-32 VM running 24/7
|
||||
- Cost: ~$700/month
|
||||
**Current State (updated v0.195.0 — the opportunities below are largely SHIPPED):**
|
||||
- No longer a single 24/7 VM. It is a pool of **on-demand** VMs: a c4d-highcpu-32
|
||||
blue-green primary pair + 8 stopped fallbacks across 6 machine families
|
||||
(c4d/c4/n4d/c2d/n2d/n2) for stockout resilience.
|
||||
- **Idle auto-shutdown is live** (JIT start on the lyrics-review page; a Cloud
|
||||
Function stops idle VMs after 15 min). When idle, cost is just boot disks
|
||||
(~$10/VM/mo); compute is billed only while encoding — so the historical
|
||||
"~$700/mo running 24/7" no longer applies.
|
||||
|
||||
**Opportunities:**
|
||||
1. **Spot Instance:** Could save 60-91% (~$420-630/month savings)
|
||||
- Risk: May be preempted during video encoding jobs
|
||||
- Mitigation: Implement job restart logic
|
||||
2. **Auto-scaling:** Start/stop based on video queue depth
|
||||
- Could scale to zero when no jobs queued
|
||||
- Startup time: ~2-3 minutes (acceptable for async processing)
|
||||
**Opportunities (status):**
|
||||
1. **Spot Instance:** Could save 60-91% — NOT adopted; on-demand chosen so a
|
||||
fallback can always be *started* when needed (Spot preemption would undermine
|
||||
the anti-stockout pool during the industry-wide crunch). Kept as a future lever.
|
||||
2. **Auto-scaling / scale-to-zero:** ✅ IMPLEMENTED via JIT start + idle-shutdown
|
||||
Cloud Function (start/stop on demand rather than queue-depth-based).
|
||||
|
||||
**Recommendation:** Test Spot instance first (lower risk), then consider auto-scaling
|
||||
**Recommendation:** Stay on-demand (already scales to zero via idle-shutdown, which
|
||||
captured most of the savings). Do NOT move to Spot for now — Spot preemption plus
|
||||
the industry-wide stockout would undermine the anti-stockout fallback pool. Revisit
|
||||
Spot only as a future experiment once capacity/resiliency is validated.
|
||||
|
||||
### 🔧 Artifact Registry Cleanup (Quick Win)
|
||||
|
||||
|
||||
@@ -406,6 +406,16 @@ When two sequential human review steps can be combined into one, do it. We origi
|
||||
|
||||
### Machine-family diversification (August 2026)
|
||||
|
||||
> **Superseded by v0.195.0** — the "Multi-instance-type encode pool" lesson at the
|
||||
> top of this file. The 2-family fallback fleet (c4d + n2) described below was
|
||||
> broadened to a **ranked 6-family pool** (c4d/c4/n4d/c2d/n2d/n2), and runtime +
|
||||
> deploy selection now route through the shared
|
||||
> `backend/services/encoding_worker_preference.py` (fastest-first + 15-min capacity
|
||||
> cooldown), not "primary first, fallbacks in declared order". The incident + root
|
||||
> lesson below remain accurate history; the "Fix (Option B)" and "Architecture
|
||||
> summary" specifics (2 families / 6 VMs / declared-order iteration) are outdated —
|
||||
> see the v0.195.0 lesson for the current mechanism.
|
||||
|
||||
**Incident 2026-08-12:** `c4d-highcpu-32` hit a **region-wide `ZONE_RESOURCE_POOL_EXHAUSTED` stockout across `us-central1-a`, `-b` AND `-c` simultaneously.** Because the primary pair *and* both fallbacks were all the same machine family, every lane in `ensure_any_running` failed at once. Effects:
|
||||
- **Preview** (`POST /api/review/{id}/preview-video`) fell through to slow local Cloud-Run encoding (~130–160 s), which **exceeds Cloudflare's ~100 s edge timeout** → browser got a **524 / "NetworkError when attempting to fetch resource"** even though the backend actually returned 200. The 524 and the NetworkError are the same event; backend logs showing 200 are misleading because the client was already disconnected.
|
||||
- **Render** parked in `RENDER_PENDING_CAPACITY` and auto-retried (safe only if capacity returns within 24 h).
|
||||
|
||||
+19
-8
@@ -64,7 +64,7 @@ Since v0.184.2, in-flight status polls are also **pinned to the worker that acce
|
||||
|
||||
Since **v0.194.0**, a mid-run job loss is also **auto-recovered**: the encoding worker processes heavy renders/encodes **one at a time** (`heavy_executor`, `ENCODING_HEAVY_CONCURRENCY=1`) so it can no longer OOM from concurrent 4K encodes (the trigger for the 2026-08-15 Arctic Monkeys batch failure — 3 concurrent encodes OOM-killed the 32 GB fallback worker and restarted it, wiping its in-memory jobs). If the worker *does* restart mid-render (deploy/crash), `wait_for_completion` now raises `EncodingJobLostError` and the render/encode worker **resubmits the job under a fresh `<id>_retry_<hex8>`** (bounded by `ENCODING_RESUBMIT_MAX=2`) instead of failing. To confirm serialization is live on a worker: `curl -s localhost:8080/health` shows `queue_length` growing while `active_jobs` stays 1 under load. **Note:** `main.py` ships in the wheel but the running uvicorn process only picks up worker-side changes (executor split, `queue_position`) on a **fresh boot / service restart** — `ensure_latest_wheel` alone does not reload the app process.
|
||||
|
||||
Since **v0.194.2**, the CI deploy handles that restart automatically even during a c4d Spot stockout. The old blue-green only targeted the c4d primary/secondary; when both were down it rolled back and **never refreshed the serving n2 fallback** (recorded as `active_override_vm`), so worker-side changes didn't reach prod until a manual restart (`primary_version` in the config doc went stale). The deploy is now **capacity-aware** (`infrastructure/encoding-worker/deploy_promote.py`): it validates the new wheel on a *fresh* green worker — the c4d secondary if it starts, else a different n2 fallback (never the current override, so it keeps serving) — then **promotes** the green (primary/secondary swap for a c4d green, or sets `active_override` for a fallback green) and drains+stops the retired worker. A c4d green also **clears a stale override** so traffic returns to the fresh primary. Zero-downtime, and it works while c4d is exhausted. Last resort if no separate green can start: an in-place restart of the serving override (brief blip, covered by auto-resubmit).
|
||||
Since **v0.194.2**, the CI deploy handles that restart automatically even during a c4d Spot stockout. The old blue-green only targeted the c4d primary/secondary; when both were down it rolled back and **never refreshed the serving n2 fallback** (recorded as `active_override_vm`), so worker-side changes didn't reach prod until a manual restart (`primary_version` in the config doc went stale). The deploy is now **capacity-aware** (`infrastructure/encoding-worker/deploy_promote.py`): it validates the new wheel on a *fresh* green worker selected from the **ranked 6-family pool** (`select_green_candidates` → the shared `encoding_worker_preference.ordered_candidates` — fastest-first with capacity cooldown; the c4d secondary if it can start, else the fastest available fallback family, never the current override so it keeps serving) — then **promotes** the green (primary/secondary swap for a c4d green, or sets `active_override` for a fallback green) and drains+stops the retired worker. A c4d green also **clears a stale override** so traffic returns to the fresh primary. Zero-downtime, and it works while c4d is exhausted. Last resort if no separate green can start: an in-place restart of the serving override (brief blip, covered by auto-resubmit).
|
||||
|
||||
---
|
||||
|
||||
@@ -94,13 +94,18 @@ The script is idempotent — re-running it skips regeneration if the CDG ZIP alr
|
||||
|
||||
**Cause:** Worker picks up a new deploy but doesn't restart automatically.
|
||||
|
||||
Target the **currently-serving** VM — `active_override_vm` in `config/encoding-worker`
|
||||
if set, else `primary_vm` (with its zone). There is no VM literally named
|
||||
`encoding-worker`; the examples below use the primary `encoding-worker-a` — swap in
|
||||
the active VM/zone if a fallback is serving.
|
||||
|
||||
```bash
|
||||
# Check current wheel version
|
||||
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
|
||||
# Check current wheel version (example: the primary a in us-central1-c)
|
||||
gcloud compute ssh encoding-worker-a --zone=us-central1-c --project=nomadkaraoke \
|
||||
--command="curl -s http://localhost:8080/health | python3 -m json.tool"
|
||||
|
||||
# Restart to pick up latest wheel
|
||||
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
|
||||
gcloud compute ssh encoding-worker-a --zone=us-central1-c --project=nomadkaraoke \
|
||||
--command="sudo systemctl restart encoding-worker"
|
||||
```
|
||||
|
||||
@@ -110,7 +115,9 @@ gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
|
||||
|
||||
**Symptoms:** A render fails with `OutputGenerator not available: No module named 'tenacity'. The karaoke-gen wheel must be installed. Check that ensure_latest_wheel() succeeded.` (or another missing package). Retries hit the same VM and fail identically.
|
||||
|
||||
**Cause:** A freshly-provisioned encoding-worker VM comes up with only a handful of packages baked into the Packer image, and boot installs the wheel with `--no-deps`. The full karaoke-gen dependency tree (torch, langchain, tenacity, …) is installed lazily at the first job by `ensure_latest_wheel()`. If that install fails or partially completes, the venv is left importable-but-incomplete and the render dies on the first missing import. First seen 2026-08-13 (job 233b9536) on a fresh `n2-highcpu-32` fallback added in #903.
|
||||
**Largely fixed in v0.195.0 (dep-bake):** the Packer image now bakes the **full** karaoke-gen dependency tree (with CPU-only torch) into the venv at build time, so a freshly-provisioned VM of any family boots self-sufficient — no first-job dependency install. This failure mode should now only appear on a VM whose boot disk predates the dep-bake image, or a venv left partial by an interrupted repair. The `ensure_latest_wheel()` runtime backstop below still exists for those cases.
|
||||
|
||||
**Original cause (pre-v0.195.0):** A freshly-provisioned VM came up with only a handful of packages baked in, and boot installed the wheel with `--no-deps`. The full dependency tree (torch, langchain, tenacity, …) was installed lazily at the first job by `ensure_latest_wheel()`; a failed/partial install left the venv importable-but-incomplete and the render died on the first missing import. First seen 2026-08-13 (job 233b9536) on a fresh `n2-highcpu-32` fallback added in #903.
|
||||
|
||||
**Fixed in v0.192.7:** `ensure_latest_wheel()` now downloads only the single latest wheel (was: copying the entire ~6 GiB `wheels/` dir every job with a 60 s timeout), retries the install (3× at 900 s), and **verifies the render/encode import chain** before returning — a clean `pip` exit is no longer trusted. Callers fail the job with a clear retryable error instead of proceeding. Fresh workers pick this up via the wheel-at-boot.
|
||||
|
||||
@@ -281,7 +288,7 @@ gcloud compute instances describe encoding-worker-a \
|
||||
|
||||
**Background — what this state means:** GCE returned `ZONE_RESOURCE_POOL_EXHAUSTED` or transient `503 SERVICE_UNAVAILABLE` from `compute.instances.start` on every encoding-worker VM. The render worker parks the job in `RENDER_PENDING_CAPACITY` instead of failing it; Cloud Scheduler retries it every 5 min via `/api/internal/retry-pending-render-jobs`. Hard timeout is 24 hours (then transitions to `failed` with a clear permanent-failure message).
|
||||
|
||||
The fallback fleet is diversified by **machine family** (since incident 2026-08-12): `c4d-highcpu-32` primaries in `us-central1-c` + `c4d` fallbacks in `-a`/`-b` + `n2-highcpu-32` fallbacks (`encoding-worker-fallback-n2c`/`-n2f`) in `-c`/`-f`. The n2 pool is much deeper, so a full `c4d` region-wide stockout (which took out a/b/c at once on 2026-08-12) should now still find n2 capacity. If you see EVERY candidate (c4d **and** n2) return stockout, that's a genuinely severe regional event — wait, or add a cross-region fallback.
|
||||
The fallback fleet is diversified across **6 machine families** (broadened to the full pool in v0.195.0): `c4d-highcpu-32` primaries (`encoding-worker-a`/`-b`) in `us-central1-c`, plus 8 stopped fallbacks — `c4d` (`-fallback-a`/`-b`, zones a/b), `n2` (`-n2c`/`-n2f`, zones c/f), `c4` (`-c4a`, zone a), `n4d` (`-n4db`, zone b), `c2d` (`-c2df`, zone f), `n2d` (`-n2da`, zone a). Candidates are tried **fastest-first with a 15-min capacity cooldown** (a family that stocks out is demoted, then re-probed) — see `backend/services/encoding_worker_preference.py`. A single-family region-wide stockout (which took out c4d a/b/c at once on 2026-08-12) now still finds capacity in the 5 other families. If you see EVERY family return stockout, that's a genuinely severe regional event — wait, or add a cross-region fallback. NOTE: creating a *new* fallback VM (or re-creating a deleted one) still needs a one-time boot allocation, so a deep enough crunch can even block provisioning — a `pulumi up` may show the missing VM as "to create / errored" until capacity returns.
|
||||
|
||||
**Related symptom — preview 524 / "NetworkError":** the same capacity exhaustion also makes `POST /api/review/{id}/preview-video` fall back to slow local encoding (~130–160 s), which exceeds Cloudflare's ~100 s edge timeout → the browser shows a **524** or Firefox *"NetworkError when attempting to fetch resource"* even though the backend returns 200. If a user reports the review preview failing, check for a concurrent stockout; the already-rendered preview mp4 (if any) is fetchable via `GET /api/review/{id}/preview-video/{hash}` (302 → signed GCS URL) with a Bearer admin token. Completing the review does **not** require the preview.
|
||||
|
||||
@@ -309,14 +316,18 @@ gcloud logging read 'protoPayload.methodName:"instances.start" AND "ZONE_RESOURC
|
||||
--project=nomadkaraoke --freshness=30m --limit=5 \
|
||||
--format='value(timestamp, protoPayload.resourceName)'
|
||||
|
||||
# Try starting a fallback in each family; if BOTH c4d and n2 refuse, it's severe — wait.
|
||||
# Try starting a fallback in each of the 6 families; if ALL refuse, it's severe — wait.
|
||||
# Synchronous (no --async) so the start op surfaces stockout inline, and we capture
|
||||
# gcloud's own exit status rather than piping (a pipe would return sed's status).
|
||||
for VM_ZONE in \
|
||||
"encoding-worker-fallback-a:us-central1-a" \
|
||||
"encoding-worker-fallback-b:us-central1-b" \
|
||||
"encoding-worker-fallback-n2c:us-central1-c" \
|
||||
"encoding-worker-fallback-n2f:us-central1-f"; do
|
||||
"encoding-worker-fallback-n2f:us-central1-f" \
|
||||
"encoding-worker-fallback-c4a:us-central1-a" \
|
||||
"encoding-worker-fallback-n4db:us-central1-b" \
|
||||
"encoding-worker-fallback-c2df:us-central1-f" \
|
||||
"encoding-worker-fallback-n2da:us-central1-a"; do
|
||||
VM="${VM_ZONE%%:*}"; ZONE="${VM_ZONE##*:}"
|
||||
if OUT=$(gcloud compute instances start "$VM" --zone="$ZONE" --project=nomadkaraoke 2>&1); then
|
||||
echo "$VM: OK (started — remember to stop it)"
|
||||
|
||||
@@ -2,7 +2,32 @@
|
||||
|
||||
**Date:** 2026-08-15
|
||||
**Branch:** `feat/sess-20260815-1915-encoding-instance-type-diversity`
|
||||
**Status:** IMPLEMENTED (v0.195.0) — code + IaC + Packer landed; operator rollout steps below still pending.
|
||||
**Status:** SHIPPED (v0.195.0 #911 + v0.195.1 #912) — code + IaC + Packer merged & rolled out.
|
||||
|
||||
## Rollout outcome (2026-08-16)
|
||||
|
||||
- ✅ Backend 0.195.1 live on Cloud Run; encode path healthy (c4d primary `encoding-worker-a`
|
||||
RUNNING 0.195.1 after c4d capacity returned to us-central1-c). `capacity_state` field live.
|
||||
- ✅ Encoding-worker image rebuilt with the CPU-torch dep-bake; **validated end-to-end** — a fresh
|
||||
`c4a` booted healthy in ~24s and encoded a preview in <6s (no first-job dep install).
|
||||
- ✅ **All 8 fallback VMs provisioned** (stopped), so the full 6-family pool is live:
|
||||
`-a`/`-b` (c4d), `-n2c`/`-n2f` (n2), `-c4a` (c4), `-n2da` (n2d), `-c2df` (c2d), `-n4db` (n4d).
|
||||
Pulumi state consistent (`c4a`, orphaned by a killed apply, was `pulumi import`ed + reconciled).
|
||||
Secret at **version 5** (8 entries, all with `machine_type`).
|
||||
- ⏳→✅ **`c2df` + `n4db` were initially deferred** (created stopped VM still needs a one-time boot
|
||||
allocation, so the 2026-08-15 stockout in zones f/b blocked provisioning). An autonomous
|
||||
10-minute retry loop finished them on **2026-08-16 ~07:48** when zones f/b left stockout, and
|
||||
added both to the secret (v5). **Historical note for a future crunch:** if a fallback VM is
|
||||
missing/deleted, re-create it with the FULL Pulumi URNs (not an ellipsis) — e.g.
|
||||
`cd infrastructure && pulumi up --yes --target 'urn:pulumi:prod::karaoke-gen-infrastructure::gcp:compute/instance:Instance::encoding-worker-fallback-c2df' --target 'urn:pulumi:prod::karaoke-gen-infrastructure::gcp:compute/instance:Instance::encoding-worker-fallback-n4db'`
|
||||
(use the write-token env from the rollout notes) — then add it to the `encoding-worker-fallback-vms`
|
||||
secret with its `machine_type`, but **never add a VM to the secret before it exists** (a
|
||||
non-existent candidate → `get_vm_status` 404 → breaks warmup).
|
||||
- Not done: measured `SPEED_RANK` benchmark run (arch-seeded; a 2026-08-16 benchmark was too noisy
|
||||
to apply — see the CPU-platform-variance findings); MIG; reservations.
|
||||
|
||||
---
|
||||
|
||||
**Context:** memory `project_gen_encoding_instance_type_diversity` + `_worker_serialization` + `_machine_family_diversity` + `_wheel_deps_missing`
|
||||
|
||||
## Goal
|
||||
|
||||
@@ -0,0 +1,141 @@
|
||||
# Handoff: Investigate prod finalization CPU efficiency / utilization — 2026-08-16
|
||||
|
||||
**Status:** OPEN investigation (not started). Written for a fresh Claude session —
|
||||
self-contained; all facts you need are inlined below or in the linked repo docs.
|
||||
**Origin:** surfaced by the encode-worker multi-instance-type benchmark
|
||||
(`docs/archive/2026-08-15-encoding-instance-type-diversity-plan.md`; benchmark
|
||||
baselines inlined under "Reproducing the benchmark" below).
|
||||
|
||||
## The question
|
||||
|
||||
Prod video **finalization** (the `/encode` path that produces
|
||||
`mp4_4k_lossless + mp4_4k_lossy + mp4_720p`) — is it using the 32-vCPU
|
||||
`*-highcpu-32` workers efficiently? The trigger was the stale
|
||||
`"ffmpeg_threads": 8` (with a comment referencing the retired `c4-standard-8`)
|
||||
in the job config, which looked like it might cap ffmpeg to 8 of 32 cores.
|
||||
|
||||
**Static analysis has already answered the first-order question (do NOT re-derive
|
||||
it — verify then move on):** `ffmpeg_threads` is **dead config**. It is set at
|
||||
`backend/workers/video_worker.py:131` but **no code reads it** (grep for consumers
|
||||
returns nothing; `run_encoding()`'s `EncodingConfig(...)` at
|
||||
`backend/services/gce_encoding/main.py:1111` never passes it), and the actual
|
||||
finalization ffmpeg commands in `backend/services/local_encoding_service.py`
|
||||
(`encode_all_formats` → `encode_lossless_mp4`/`encode_lossy_mp4`/`encode_720p`,
|
||||
around lines 343 / 396 / 471) carry **no `-threads` flag at all** → ffmpeg falls
|
||||
back to its default (`-threads 0`, i.e. auto ≈ all cores). So finalization is
|
||||
**almost certainly NOT capped at 8 cores**; the "leaving 75% idle" fear is likely
|
||||
unfounded. (Separate, unrelated paths that DO hardcode threads — don't confuse
|
||||
them: `main.py:372` uses `-threads 8` in the **preview** `/encode-preview`
|
||||
480×270 path; `local_preview_encoding_service.py:310` uses `-threads 0`.)
|
||||
|
||||
So this is now a **narrower efficiency + cleanup** task, not a "fix the cap" task.
|
||||
|
||||
## What to determine (in order)
|
||||
|
||||
1. **Confirm the static finding empirically** (x264's `-threads 0` "auto" is a
|
||||
heuristic — it does NOT always scale to all cores; historically it caps frame
|
||||
threads, with diminishing returns past ~16). Drive a real finalization and
|
||||
capture **process-level** evidence, not just host-wide load:
|
||||
- the **deployed ffmpeg argv** for each format (`ps -eo pid,args | grep ffmpeg`
|
||||
on the worker mid-encode, or add a log line) — confirm no `-threads` and see
|
||||
what x264 chose;
|
||||
- **per-process / per-thread** CPU% and RSS (`top -H -p <ffmpeg-pid>`,
|
||||
`pidstat -t -p <pid> 2`), NOT `mpstat -P ALL` alone — host-wide utilization
|
||||
can't tell an 8-thread cap from serial `encode_all_formats` stages or
|
||||
unrelated work;
|
||||
- **per-format stage timings** (which of lossless/lossy/720p dominates).
|
||||
Run all of this against the SAME worker you drive the encode to (see the
|
||||
binding note in "Reproducing the benchmark").
|
||||
|
||||
2. **Check whether the 3 formats encode SERIALLY or in PARALLEL** in
|
||||
`encode_all_formats` (`backend/services/local_encoding_service.py`). If serial,
|
||||
wall-time ≈ sum of the three even if each saturates the cores — the real lever
|
||||
may be **overlapping independent format encodes** rather than more threads. But
|
||||
see the RAM constraint below before parallelizing.
|
||||
|
||||
3. **Decide the fix (likely small) and validate it:**
|
||||
- **Cleanup regardless:** remove the dead `ffmpeg_threads` field + stale comment
|
||||
at `backend/workers/video_worker.py:131` (it misleads; that is the "documentation-
|
||||
only" close-out CodeRabbit flagged). If you'd rather *use* it, wire it through
|
||||
`EncodingConfig` → the `local_encoding_service.py` ffmpeg commands as an explicit
|
||||
`-threads`; otherwise delete it.
|
||||
- If step 1 shows real idle cores (x264 auto under-scales), the win is an explicit
|
||||
high `-threads` (or `-x264-params threads=N`) or overlapping formats — validate
|
||||
with a before/after re-benchmark on the SAME input+VM.
|
||||
- If step 1 shows cores already saturated, close this out as "already efficient;
|
||||
removed dead config" and stop.
|
||||
|
||||
## Hard constraints — do NOT regress these
|
||||
|
||||
- **Heavy-lane serialization stays.** `ENCODING_HEAVY_CONCURRENCY=1` (one heavy
|
||||
encode at a time per worker) exists because 3 **concurrent** 4K encodes OOM-killed
|
||||
the 32 GB worker (incident 2026-08-15; full write-up in `docs/LESSONS-LEARNED.md`
|
||||
→ "Concurrent encodes OOM-killed the worker → lost renders").
|
||||
Raising *threads within a single encode* is **orthogonal** to that and does not
|
||||
reintroduce concurrency — but it DOES raise a single encode's **peak RAM** (more
|
||||
parallel ffmpeg slices / parallel format encodes). Watch RSS on a 32 GB floor VM
|
||||
(n2/n2d fallbacks are 32 GB; c4d/c4/n4d are 60–64 GB). A single 4K lossless encode
|
||||
was ~18 GB; leave headroom.
|
||||
- **Lossless 4K is the heavy one** (`pcm_s16le`, huge intermediate) — profile that
|
||||
format specifically.
|
||||
|
||||
## Reproducing the benchmark (for before/after numbers)
|
||||
|
||||
The benchmark used a real completed job's inputs as a canonical `/encode` input.
|
||||
It was deleted after the run (`gs://…/bench/` cleaned up). To recreate:
|
||||
```bash
|
||||
J=gs://karaoke-gen-storage-nomadkaraoke/jobs/ffc17eb7 # "Glen Campbell - A Better Place"
|
||||
I=gs://karaoke-gen-storage-nomadkaraoke/bench/encode-input
|
||||
gsutil -q cp "$J/videos/with_vocals.mkv" "$I/videos/with_vocals.mkv" # 119 MB rendered 4K karaoke video
|
||||
gsutil -q cp "$J/screens/title.png" "$I/screens/title.png"
|
||||
gsutil -q cp "$J/screens/end.png" "$I/screens/end.png"
|
||||
gsutil -q cp "$J/stems/instrumental_clean.flac" "$I/stems/instrumental_clean.flac"
|
||||
gsutil -q cp "$J/style/style_params.json" "$I/style/style_params.json"
|
||||
gsutil -q cp "$J/lyrics/karaoke.ass" "$I/lyrics/karaoke.ass"
|
||||
```
|
||||
**Bind the encode to the worker you observe.** Pick ONE stopped fallback (e.g.
|
||||
`encoding-worker-fallback-c4a`, us-central1-a), start it, and send BOTH the
|
||||
`/encode` POST and the `/status` polls **directly to that worker's external IP**
|
||||
(`http://<IP>:8080/…`, not through the backend/`api.nomadkaraoke.com` — the
|
||||
backend may route to a different active worker). SSH the SAME VM for the
|
||||
process-level probes in step 1, so the CPU/RSS you measure is the encode you drove.
|
||||
Get the IP: `gcloud compute instances describe <vm> --zone=<z>
|
||||
--format='value(networkInterfaces[0].accessConfigs[0].natIP)'`.
|
||||
|
||||
`POST http://<IP>:8080/encode` body (`X-API-Key` from
|
||||
`gcloud secrets versions access latest --secret=encoding-worker-api-key`):
|
||||
```json
|
||||
{"job_id":"cpu-probe-1","input_gcs_path":"gs://…/bench/encode-input/",
|
||||
"output_gcs_path":"gs://…/bench/out/probe/",
|
||||
"encoding_config":{"formats":["mp4_4k_lossless","mp4_4k_lossy","mp4_720p"],
|
||||
"base_name":"Glen Campbell - A Better Place","artist":"Glen Campbell",
|
||||
"title":"A Better Place","instrumental_selection":"clean",
|
||||
"existing_instrumental":null}}
|
||||
```
|
||||
(`ffmpeg_threads` deliberately omitted from the body above — it is dead config;
|
||||
see "The question".) Poll `GET http://<IP>:8080/status/{job_id}` to `complete`. Baseline medians from 2026-08-16 (full
|
||||
3-format finalization of this ~4-min song; the runs sent `ffmpeg_threads:8` in the
|
||||
body but the worker ignored it, so these reflect ffmpeg's default all-core threading):
|
||||
c4-Emerald ≈ **139s**, n2d-Milan ≈ **139s**, n2-CascadeLake ≈ **247s**, n2-IceLake
|
||||
≈ **92s** (c4d unmeasured — could not start a 2nd c4d in the stockout). The huge
|
||||
n2 spread — **92s vs 247s for the identical `n2-highcpu-32` SKU** — is because n2
|
||||
spans Intel Ice Lake (fast) and Cascade Lake (slow) and GCE assigns either; that
|
||||
intra-family variance exceeds the between-family differences. Separate actionable
|
||||
finding: pin `min_cpu_platform` (e.g. "Intel Ice Lake" / "AMD Milan") on n2/n2d
|
||||
fallbacks for predictable speed, or rank them lower to reflect the gamble. Also
|
||||
note the counterintuitive result that c4 (newest Intel Emerald) ≈ n2d (Milan) and
|
||||
did NOT beat n2 Ice Lake for this lossless-4K-heavy workload — so "newest = fastest"
|
||||
does not hold here; measure, don't assume.
|
||||
|
||||
**Operational hygiene:** benchmark VMs are stopped fallbacks — start, probe, then
|
||||
STOP them (never the live-serving primary; the idle-shutdown fn also handles it).
|
||||
Delete `gs://…/bench/**` when done. Auth for VM start/stop: `gcloud` as
|
||||
`admin@nomadkaraoke.com`.
|
||||
|
||||
## Expected outcome
|
||||
|
||||
Either: (a) confirm CPU is already saturated (`ffmpeg_threads` is dead config) →
|
||||
close this out and just fix the stale comment; or (b) confirm an ~2–4× idle-core
|
||||
gap → raise threads / parallelize formats, validate the speedup and RAM headroom,
|
||||
ship it. Bump `pyproject.toml` version; if the fix is in the wheel's
|
||||
`LocalEncodingService`, it ships via the normal wheel → `ensure_latest_wheel` path.
|
||||
Reference in new issue
Block a user