mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary Two bugs, both hit by job `82f48738` (a private → public visibility change) on 2026-10-12. ### 1. Encoding worker VMs boot into an unrecoverable restart loop If gcloud's first call (the API key fetch in `startup.sh`) happens before DHCP, gcloud caches "not on GCE" in `/root/.config/gcloud/gce`. Every later gcloud call in that boot then has no credentials, so the API key comes back empty, the v0.266.4 self-heal refuses to serve, and systemd re-runs `startup.sh` against the same poisoned cache until the VM idle-stops. worker-b was up 20:30–20:42 and never answered /health, so the render failed with `Cannot connect to host 34.10.189.118:8080`. The same thing happened on 10-09 (`f4310aea`): it was fixed by hand and the durable fix was noted but never shipped. - `startup.sh`: `rm -f /root/.config/gcloud/gce`, wait up to 60s for the metadata server, retry the secret fetch 6× (clearing the cache each time). CI uploads this script on every deploy and bootstrap downloads it on every service start, so existing VM disks pick it up. - Packer `startup-fallback.sh`: clears the cache too (new images). ### 2. Retry after a failed change-to-public 404s forever `change_to_public` deletes screens, `videos/with_vocals.mkv` and the finals but left their `file_urls`. `/retry` then saw `videos.with_vocals` + `instrumental_selection`, skipped the render and 404'd on the deleted MKV. It now drops those refs, as `ThemeRerenderService` already does. ## Testing - New `backend/tests/test_encoding_worker_startup_script.py` and `test_change_visibility.py::test_drops_file_urls_for_deleted_artifacts` - `pytest backend/tests/test_encoding_worker_startup_script.py backend/tests/test_encoding_worker_self_heal.py backend/tests/test_encoding_worker_code_paths.py backend/tests/test_change_visibility.py`: all pass 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01JUFVxMZME6ydzkmLvyuXRo <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * When a private job is made public, references to deleted screen and video files are cleared before processing resumes, preventing stale files from appearing. * Improved encoding worker startup reliability by retrying API-key retrieval when the metadata service or secret store is temporarily unavailable. <!-- end of auto-generated comment: release notes by coderabbit.ai --> @coderabbitai ignore --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
63 lines
2.4 KiB
Python
63 lines
2.4 KiB
Python
"""Guards for the encoding-worker boot scripts' gcloud on-GCE cache handling.
|
|
|
|
A gcloud call made before DHCP finishes writes "False" to /root/.config/gcloud/gce,
|
|
so every later gcloud call in that boot has no credentials. The worker then had
|
|
no API key, self-heal refused to serve, and systemd re-ran startup.sh against the
|
|
same poisoned cache until the VM idle-stopped (jobs f4310aea 2026-10-09,
|
|
82f48738 2026-10-12). The scripts must drop that cache before fetching the key.
|
|
"""
|
|
import re
|
|
import subprocess
|
|
from pathlib import Path
|
|
|
|
REPO = Path(__file__).resolve().parents[2]
|
|
STARTUP = REPO / "infrastructure" / "encoding-worker" / "startup.sh"
|
|
PROVISION = REPO / "infrastructure" / "packer" / "scripts" / "provision.sh"
|
|
|
|
CACHE_RM = "rm -f /root/.config/gcloud/gce"
|
|
SECRET_FETCH = "gcloud secrets versions access latest --secret=encoding-worker-api-key"
|
|
|
|
|
|
def test_startup_script_is_valid_bash():
|
|
subprocess.run(["bash", "-n", str(STARTUP)], check=True)
|
|
|
|
|
|
def test_startup_clears_gce_cache_before_first_secret_fetch():
|
|
text = STARTUP.read_text()
|
|
assert CACHE_RM in text
|
|
assert text.index(CACHE_RM) < text.index(SECRET_FETCH)
|
|
|
|
|
|
def test_startup_waits_for_metadata_server_before_fetch():
|
|
text = STARTUP.read_text()
|
|
metadata_probe = text.index("computeMetadata/v1/instance/id")
|
|
assert metadata_probe < text.index(SECRET_FETCH)
|
|
|
|
|
|
def test_startup_retries_secret_fetch():
|
|
text = STARTUP.read_text()
|
|
loops = re.findall(r"for attempt in \$\(seq 1 \d+\); do\n(.*?)\ndone", text, re.S)
|
|
assert any(SECRET_FETCH in body and CACHE_RM in body for body in loops)
|
|
|
|
|
|
def test_packer_fallback_clears_gce_cache_before_secret_fetch():
|
|
text = PROVISION.read_text()
|
|
fallback = text[text.index("startup-fallback.sh << 'FALLBACK'"):]
|
|
assert CACHE_RM in fallback
|
|
assert fallback.index(CACHE_RM) < fallback.index(SECRET_FETCH)
|
|
|
|
|
|
def test_startup_metadata_probe_has_total_timeout():
|
|
text = STARTUP.read_text()
|
|
lines = text.splitlines()
|
|
url_idx = next(i for i, l in enumerate(lines) if "computeMetadata/v1/instance/id" in l)
|
|
# The curl flags sit on the line before the URL continuation.
|
|
assert "--max-time" in lines[url_idx - 1] + lines[url_idx]
|
|
|
|
|
|
def test_startup_exits_nonzero_without_api_key():
|
|
text = STARTUP.read_text()
|
|
block = text[text.index('if [ -z "$ENCODING_API_KEY" ]; then'):]
|
|
block = block[: block.index("fi")]
|
|
assert "exit 1" in block
|