mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
main
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
de5dfcd50d |
feat(encoding): AWS EC2 encoding worker as a pool candidate (v0.288.0) (#1236)
## Summary
Phase 1 of moving encoding to AWS. EC2 is paid for by the AWS Activate
credits (acct 790139457751, expiring 2028-10-31).
- **Plan:** `docs/archive/2026-10-29-aws-encoding-worker-plan.md`.
- **Benchmark** (3 real prod jobs replayed): an EC2 c7a.4xlarge matched
or beat GCE c4d-highcpu-16.
The EC2 box runs the **unchanged** `gce_encoding` worker service and
joins the existing candidate pool as `provider: aws`. The existing
failover, active override, lost-job detection and stockout cooldowns all
apply to it as they are.
## Changes
- **`backend/services/aws_encoding_compute.py` (new):**
`AwsComputeAdapter` looks to the manager like the
`compute_v1.InstancesClient` methods it uses.
- EC2 states map to GCE statuses.
- EC2 capacity errors are added to `CAPACITY_ERROR_CODES`.
- A missing instance raises `NotFound`.
- Controller credentials come from Secret Manager
`aws-encoding-controller-credentials`.
- **`EncodingWorkerManager`:**
- `_compute_for(vm)` sends AWS VMs to the adapter; the GCE paths are
untouched.
- A shared `parse_fallback_vms` passes `provider`/`instance_id`/`region`
through.
- `ranked_candidates()` is used by both the encode path and the warmups.
- `ensure_preferred_running()` behaves **exactly like
`ensure_primary_running` whenever the GCE primary ranks first**, which
is today's case.
- An AWS control-plane fault fails over to the GCE candidates.
- **Ranking:** `SPEED_RANK["c7a"] = 90`, so AWS is the last resort by
default. Firestore `config/encoding-worker.speed_rank_overrides =
{"c7a": 5}` makes it preferred, with no deploy.
- **Isolation from GCE-only tooling:**
- The override now records `active_override_provider`.
- The CI blue-green deploy, `deploy_promote` and the idle Cloud Function
all ignore the AWS worker; it hot-updates its wheel per job.
- A GCE promotion clears any stale provider tag.
- **Worker:**
- Uses `gcloud storage` on AWS, because gsutil can't use WIF credential
files. GCE keeps gsutil.
- `infrastructure/encoding-worker/aws/` adds:
- `provision.sh`, as EC2 user-data;
- `startup-aws.sh`, uploaded by CI;
- `idle-watchdog.py`, an idle self-stop that runs as its own systemd
timer.
- **Infra (GCP), already applied with `/gcp-admin`:**
- The WIF pool `aws-encoding`, plus an AWS provider that only admits the
`nomad-encoding-worker` role, federated to `encoding-worker@`.
- The secret `aws-encoding-controller-credentials`.
- Preview and apply: 4 creates, 288 unchanged.
- **Infra (AWS):** a new Pulumi project `infrastructure/aws` (stack
`nomadkaraoke/prod`):
- the c7a.8xlarge **persistent Spot** worker, which stops on
interruption or shutdown;
- an Elastic IP;
- a security group on 8080 (API key, same as the GCE firewall);
- the worker role with SSM;
- a tag-scoped `nomad-encoding-controller` user.
- Preview: 10 creates. **I apply it after merge**, because the box
provisions from the deployed wheel and boot scripts.
- `boto3` is now an explicit dependency.
- CI also runs the `encoding_worker_idle` function tests, in the job
renamed "Infra - Cloud Function tests".
## Rollout (after merge)
1. **Stage A:** `pulumi up` the AWS stack. Store the controller key in
the secret, then add the worker's entry to
`encoding-worker-fallback-vms`. It's inert unless both c4d and c2d stock
out.
2. **Stage B:** a supervised `speed_rank_overrides {"c7a": 5}` window.
Run real jobs, then compare timings and outputs, including a visual
check of the portrait video.
3. **Stage C:** keep it preferred. After a clean week, delete the c4d
pair.
Phase 2, running distribution (Dropbox/YouTube/Drive uploads) from AWS,
follows in a separate PR.
## Testing
- **New tests:**
- adapter: botocore `Stubber`, covering the state map, capacity and
NotFound handling;
- manager routing and ranking with overrides;
- `ensure_preferred_running` branches;
- failover on an AWS control-plane fault;
- provider tagging;
- deploy_promote exclusions;
- the idle function skipping AWS;
- watchdog decisions;
- the gcs CLI choice.
- **Updated** the candidate and warmup-route tests to the new seams.
- **Full backend + unit suites** have no new failures. The local
emulator and cross-suite failures are identical on main.
- **Local CodeRabbit:** 2 findings, both fixed (failover on AWS faults;
stale provider tag).
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
d39dfd6778 |
Encoding worker infra/auth failures: retry instead of terminal-failing the job (#972)
## Why Prod alert burst (2026-08-31): a paying customer's job (`6452888e`) was **terminally failed** — its render worker error was: > Encoding job 6452888e failed: Failed to retrieve `https://metadata.google.internal/.../service-accounts/default/` … **Compute Engine Metadata server unavailable** … `SSLCertVerificationError` Root cause: the c4d primary pair was in a zonal stockout (503 on start), so the render fell back to `encoding-worker-fallback-c4a`. That fallback VM **booted and accepted the job** but then couldn't reach the GCE metadata server to fetch its service-account token, so it couldn't touch GCS and died mid-encode. This surfaced as a bare `RuntimeError`, hit `render_video_worker`'s generic `except Exception` → `fail_job`, and lost the job with **no retry** — even though the exact same work succeeds on any healthy worker. The same alert burst also included the daily E2E happy-path failure (the encoder being offline made the preview step flaky) and a benign `match-judge AI call failed; returning no-suggestion` red-error page. All three are addressed here. ## What **1. Encoding infra/auth failures are now recoverable (primary fix)** - `encoding_errors.py`: `is_worker_infra_error()` classifier (metadata / SSL-cert / missing-credentials markers, deliberately specific so a real ffmpeg/input error never matches) + new `EncodingWorkerInfraError` — a **subclass of `EncodingWorkerStartError`** so it flows through the existing park-for-auto-retry path with zero changes to the render worker's `except` (and the final-encode Cloud Run Job retries too). - `encoding_service.wait_for_completion()`: on a worker-reported infra/auth failure, **demote the broken worker + invalidate the URL cache**, then raise the recoverable typed error instead of `RuntimeError`. - `encoding_worker_manager.demote_active_worker()`: records a capacity-state cooldown for the serving fallback's `machine_type@zone` and clears `active_override`, so the retry re-selects a *different, healthy* VM. Scoped to **fallbacks only** — it never family-demotes the fast c4d primary on a rare blip. **2. match-judge noise** — the two graceful-degradation `logger.exception` calls become `logger.warning(exc_info=True)`, so a transient Vertex/Gemini blip that safely returns no-suggestion stops paging as a red "new error pattern". **3. Happy-path E2E robustness** — tolerate slow encoder cold-starts at the preview step: scope the alert check to the modal (kills the spurious empty-`[role=alert]` "Preview error:" from run #151) and recover the "Proceed to Instrumental" button via reload+reopen. The final assertion stays authoritative, so genuinely broken generation still fails the smoke test. ## Testing - New unit coverage: classifier positive/negative + subclass (`test_encoding_errors.py`), `demote_active_worker` (fallback demote+clear, primary no-op, never-raises), `wait_for_completion` infra classification (+ demote/invalidate, and no-manager path), and render worker **parks** (not fails) on `EncodingWorkerInfraError`. - `python -m pytest` across the encoding / worker / match-judge suites: **177 passed** locally. ## Not in this PR (infra, needs Andrew / GCP write) - **Fault A**: `encoding-worker-a` (c4d, us-central1-c) is chronically 503 on start (zonal stockout) — that's why the "Encoder offline" footer showed in the E2E. This change makes the fallback path robust to it, but the underlying c4d capacity/zone choice is an infra decision (secret/VM change; my GCP access is read-only). Worth deciding whether to repin the primary family/zone or promote a fallback. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Detects infrastructure and authentication failures on encoding workers. * Demotes affected workers and retries jobs using alternate capacity when available. * **Bug Fixes** * Prevents recoverable worker failures from becoming terminal job errors. * Reduces misleading error-level logging for gracefully handled match-judge failures. * Improves preview-flow recovery by waiting for actions to become enabled. * **Documentation** * Clarified that retries prefer, but do not guarantee, a different worker. * **Chores** * Updated the application version to 0.217.0. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
|
51dbf1cbfd |
fix(encoding): serialize heavy encodes (OOM-proof) + auto-resubmit lost renders (#908)
## Problem Three Arctic Monkeys renders (`a3f340a2`, `374dec26`, `590630c0`) failed within ~2 min on 2026-08-15 with `lost contact with worker after 5 consecutive poll failures: Encoding job … not found`. **Root cause (confirmed from live forensics):** Both 60 GB c4d primary encoding workers were down (Spot stockout), so all render traffic funneled onto the single 32 GB `n2-highcpu-32` fallback. The worker ran `ThreadPoolExecutor(max_workers=4)`, so the batch put **3 concurrent 4K ffmpeg encodes** (~30 GB) on it → **kernel OOM-killer fired** → `encoding-worker` service restarted → in-memory `jobs` registry wiped → every poll `404`'d → all three failed. ffmpeg already saturates all cores for one job, so the parallelism bought **no throughput**, only multiplied peak RAM. ## Fix **Worker — serialize the heavy lane** (`backend/services/gce_encoding/main.py`) - Split the single pool into `heavy_executor` (`max_workers=ENCODING_HEAVY_CONCURRENCY`, default **1**, clamped to `[1,4]`) for renders/encodes and `light_executor` (previews + wheel installs). One heavy ffmpeg at a time = OOM-proof on any machine type; extras queue as `pending`. - `/status` now exposes `queue_position` (+ `restart_failure_code`); jobs tagged by `kind`. **Client + workers — auto-resubmit a lost job** - `wait_for_completion` raises the typed `EncodingJobLostError` on a **404-after-the-job-was-seen** (a pinned poll can only 404 if the worker lost it) or a terminal failure carrying `restart_failure_code`. 404 is now a typed `EncodingJobNotFoundError` (no message parsing). - Time spent `pending` counts against a generous `queue_timeout`, not the per-run `timeout`. - `run_with_lost_job_resubmit` resubmits a lost job as `<id>_retry_<hex8>` (bounded, `ENCODING_RESUBMIT_MAX=2`), wired into `render_video_worker`, `video_worker`, and `video_worker_orchestrator` (incl. the stale-cache path). Idempotent by deterministic output paths (overwrite, no dupes). - `GCEEncodingBackend.encode` re-raises the lost error instead of flattening it to `success=False`, so the resubmit actually fires. ## Testing - New tests: worker lane serialization + `queue_position`; concurrency-clamp matrix; client lost-job detection (404-after-seen, restart code, pending-not-timeout); resubmit wrapper (fresh id, bounded, non-retry of other errors); backend re-raise; preview queue-timeout cap. - Full backend suite: **3766 passed, 139 skipped**. - Local CodeRabbit review completed; feedback addressed (typed 404, bounded env parsing, preview queue cap, idempotency doc). ## Deploy note `main.py` ships in the wheel, but the running uvicorn process only picks up worker-side changes on a **service restart** — the CI deploy / blue-green swap handles this. Post-deploy verification: re-render the 3 failed jobs **concurrently** (the exact OOM trigger) and confirm no OOM (`journalctl -k`), `NRestarts` doesn't increase, and all three complete. @coderabbitai ignore --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
|
20fae2c579 |
Encoding worker capacity resilience: real errors, auto-retry, multi-zone failover (#748)
## Summary
Investigation of two failed jobs (`fbc651be`, `bee150fd`) revealed both
failed at the GCE encoding-worker render stage with empty error messages
(`Video render failed: `). Root cause: the GCP `compute.instances.start`
API was returning `ZONE_RESOURCE_POOL_EXHAUSTED` (us-central1-c out of
`c4d-highcpu-32` capacity), but the application's fire-and-forget call
swallowed the error. Users saw 7-minute timeouts with no useful message,
and there was no mechanism to retry once capacity returned.
This PR delivers all three phases of the capacity-resilience plan
([docs/archive/2026-05-05-encoding-worker-capacity-resilience-plan.md](docs/archive/2026-05-05-encoding-worker-capacity-resilience-plan.md)):
### Phase 1 — Surface real errors (commit `56e5968e`)
- New `EncodingWorkerCapacityError` / `EncodingWorkerStartError` typed
exceptions
- `EncodingWorkerManager.start_vm` now waits for the operation result
and raises the right typed error on `ZONE_RESOURCE_POOL_EXHAUSTED` /
`STOCKOUT` / `QUOTA_EXCEEDED`
- Capacity errors propagate through the warmup-fallback path; the HTTP
retry loop bails out fast (instead of hammering an unreachable VM for 7
min)
- Render worker catches capacity errors and parks the job in new state
`RENDER_PENDING_CAPACITY` with a user-friendly message: *"Encoding
capacity is temporarily unavailable. Your job will retry automatically —
no action needed. Most jobs recover within 5–30 minutes."*
- Repeat capacity hits accumulate `attempt_count` while preserving
`first_seen_at`
- `repr()` fallback in the generic exception handler so empty-string
errors no longer produce blank `Video render failed: ` messages
### Phase 2 — Auto-retry scheduler (commit `9d6d2161`)
- New endpoint `POST /api/internal/retry-pending-render-jobs`
(admin-auth)
- Cloud Scheduler fires it every 5 min — picks the oldest
`RENDER_PENDING_CAPACITY` job, clears its error state, re-triggers the
render worker
- 24h hard deadline: jobs older than that fail permanently with a clear
"retry manually or contact support" message
- Processes 1 job per tick (the GCE worker is single-tenant per render)
### Phase 3 — Multi-zone failover (commit `6d6b1376`)
- IaC: new fallback VMs `encoding-worker-fallback-a` (us-central1-a) and
`encoding-worker-fallback-f` (us-central1-f). All four zones support
`c4d-highcpu-32`. Provisioned stopped — ~$10/mo idle each
- New `EncodingWorkerManager.ensure_any_running(candidates)` iterates
VMs in order, catching capacity errors per-zone, falling through to the
next; returns the active VM info
- New `active_override_*` fields on the Firestore config doc track the
active fallback URL; `EncodingWorkerConfig.active_url` returns the
override if set, else primary; `EncodingService._get_worker_url`
consults `active_url` so in-flight requests after fallback hit the right
VM
- Gated by env var `ENCODING_WORKER_FALLBACK_VMS` (JSON list) — empty
default = single-zone behavior unchanged
## Operational rollout
Phases 1 & 2 ship as code only — no manual steps. Phase 3 needs:
1. `pulumi up` from `infrastructure/` — provisions the 2 fallback VMs
(stopped)
2. Capture Pulumi-output IPs (`encoding_worker_fallback_a_ip`,
`encoding_worker_fallback_f_ip`)
3. Set env var on `karaoke-backend` Cloud Run:
```
ENCODING_WORKER_FALLBACK_VMS=[{"vm":"encoding-worker-fallback-a","zone":"us-central1-a","ip":"<ip-a>"},{"vm":"encoding-worker-fallback-f","zone":"us-central1-f","ip":"<ip-f>"}]
```
4. Until step 3 is done the application uses single-zone behavior;
fallback VMs are inert
Plan doc:
[docs/archive/2026-05-05-encoding-worker-capacity-resilience-plan.md](docs/archive/2026-05-05-encoding-worker-capacity-resilience-plan.md)
## Test plan
- [x] Unit: `EncodingWorkerManager.start_vm` raises
`EncodingWorkerCapacityError` for `ZONE_RESOURCE_POOL_EXHAUSTED`,
generic `EncodingWorkerStartError` for other codes, surfaces `result()`
failures
- [x] Unit: `_warmup_encoding_worker_fallback` propagates capacity
errors, swallows generic warmup failures
- [x] Unit: `_request_with_retry` aborts immediately on capacity error
from warmup (no 7-min retry loop)
- [x] Unit: render worker transitions to `RENDER_PENDING_CAPACITY` on
capacity error, NOT `FAILED`; preserves `first_seen_at`; increments
`attempt_count`
- [x] Unit: render worker non-capacity exceptions still fail with a
useful message (no more blank `Video render failed: `)
- [x] Unit: `/api/internal/retry-pending-render-jobs` retries 1 job per
tick (oldest first), times out 24h+ jobs, no-op on empty queue, clears
error state before retry
- [x] Unit: `ensure_any_running` happy path, falls through on capacity
error, raises when all candidates exhausted, clears stale override on
primary success
- [x] Unit: `EncodingWorkerConfig.active_url` returns override if set,
else primary
- [x] Full backend suite passes (3114 tests; 5 pre-existing failures in
`test_internal_api.py` are unrelated and present on `main`)
- [ ] Manual prod test: trigger a render during a confirmed capacity
outage, observe job parks in `RENDER_PENDING_CAPACITY`, scheduler picks
it up next tick, completes when capacity returns
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|