Files
karaoke-gen/backend/tests/test_encoding_worker_startup_script.py
Andrew Beveridge ☄️andClaude Opus 5.5 77bec3327c fix: encoding-worker boot loop (poisoned gcloud cache) + stale file_urls after change-to-public (v0.268.3) (#1138)
## Summary
Two bugs, both hit by job `82f48738` (a private → public visibility
change) on 2026-10-12.

### 1. Encoding worker VMs boot into an unrecoverable restart loop
If gcloud's first call (the API key fetch in `startup.sh`) happens
before DHCP, gcloud caches "not on GCE" in `/root/.config/gcloud/gce`.
Every later gcloud call in that boot then has no credentials, so the API
key comes back empty, the v0.266.4 self-heal refuses to serve, and
systemd re-runs `startup.sh` against the same poisoned cache until the
VM idle-stops. worker-b was up 20:30–20:42 and never answered /health,
so the render failed with `Cannot connect to host 34.10.189.118:8080`.
The same thing happened on 10-09 (`f4310aea`): it was fixed by hand and
the durable fix was noted but never shipped.

- `startup.sh`: `rm -f /root/.config/gcloud/gce`, wait up to 60s for the
metadata server, retry the secret fetch 6× (clearing the cache each
time). CI uploads this script on every deploy and bootstrap downloads it
on every service start, so existing VM disks pick it up.
- Packer `startup-fallback.sh`: clears the cache too (new images).

### 2. Retry after a failed change-to-public 404s forever
`change_to_public` deletes screens, `videos/with_vocals.mkv` and the
finals but left their `file_urls`. `/retry` then saw
`videos.with_vocals` + `instrumental_selection`, skipped the render and
404'd on the deleted MKV. It now drops those refs, as
`ThemeRerenderService` already does.

## Testing
- New `backend/tests/test_encoding_worker_startup_script.py` and
`test_change_visibility.py::test_drops_file_urls_for_deleted_artifacts`
- `pytest backend/tests/test_encoding_worker_startup_script.py
backend/tests/test_encoding_worker_self_heal.py
backend/tests/test_encoding_worker_code_paths.py
backend/tests/test_change_visibility.py`: all pass

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01JUFVxMZME6ydzkmLvyuXRo

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* When a private job is made public, references to deleted screen and
video files are cleared before processing resumes, preventing stale
files from appearing.
* Improved encoding worker startup reliability by retrying API-key
retrieval when the metadata service or secret store is temporarily
unavailable.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

@coderabbitai ignore

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 17:29:43 -04:00

63 lines
2.4 KiB
Python

"""Guards for the encoding-worker boot scripts' gcloud on-GCE cache handling.
A gcloud call made before DHCP finishes writes "False" to /root/.config/gcloud/gce,
so every later gcloud call in that boot has no credentials. The worker then had
no API key, self-heal refused to serve, and systemd re-ran startup.sh against the
same poisoned cache until the VM idle-stopped (jobs f4310aea 2026-10-09,
82f48738 2026-10-12). The scripts must drop that cache before fetching the key.
"""
import re
import subprocess
from pathlib import Path
REPO = Path(__file__).resolve().parents[2]
STARTUP = REPO / "infrastructure" / "encoding-worker" / "startup.sh"
PROVISION = REPO / "infrastructure" / "packer" / "scripts" / "provision.sh"
CACHE_RM = "rm -f /root/.config/gcloud/gce"
SECRET_FETCH = "gcloud secrets versions access latest --secret=encoding-worker-api-key"
def test_startup_script_is_valid_bash():
subprocess.run(["bash", "-n", str(STARTUP)], check=True)
def test_startup_clears_gce_cache_before_first_secret_fetch():
text = STARTUP.read_text()
assert CACHE_RM in text
assert text.index(CACHE_RM) < text.index(SECRET_FETCH)
def test_startup_waits_for_metadata_server_before_fetch():
text = STARTUP.read_text()
metadata_probe = text.index("computeMetadata/v1/instance/id")
assert metadata_probe < text.index(SECRET_FETCH)
def test_startup_retries_secret_fetch():
text = STARTUP.read_text()
loops = re.findall(r"for attempt in \$\(seq 1 \d+\); do\n(.*?)\ndone", text, re.S)
assert any(SECRET_FETCH in body and CACHE_RM in body for body in loops)
def test_packer_fallback_clears_gce_cache_before_secret_fetch():
text = PROVISION.read_text()
fallback = text[text.index("startup-fallback.sh << 'FALLBACK'"):]
assert CACHE_RM in fallback
assert fallback.index(CACHE_RM) < fallback.index(SECRET_FETCH)
def test_startup_metadata_probe_has_total_timeout():
text = STARTUP.read_text()
lines = text.splitlines()
url_idx = next(i for i, l in enumerate(lines) if "computeMetadata/v1/instance/id" in l)
# The curl flags sit on the line before the URL continuation.
assert "--max-time" in lines[url_idx - 1] + lines[url_idx]
def test_startup_exits_nonzero_without_api_key():
text = STARTUP.read_text()
block = text[text.index('if [ -z "$ENCODING_API_KEY" ]; then'):]
block = block[: block.index("fi")]
assert "exit 1" in block