mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary Two bugs, both hit by job `82f48738` (a private → public visibility change) on 2026-10-12. ### 1. Encoding worker VMs boot into an unrecoverable restart loop If gcloud's first call (the API key fetch in `startup.sh`) happens before DHCP, gcloud caches "not on GCE" in `/root/.config/gcloud/gce`. Every later gcloud call in that boot then has no credentials, so the API key comes back empty, the v0.266.4 self-heal refuses to serve, and systemd re-runs `startup.sh` against the same poisoned cache until the VM idle-stops. worker-b was up 20:30–20:42 and never answered /health, so the render failed with `Cannot connect to host 34.10.189.118:8080`. The same thing happened on 10-09 (`f4310aea`): it was fixed by hand and the durable fix was noted but never shipped. - `startup.sh`: `rm -f /root/.config/gcloud/gce`, wait up to 60s for the metadata server, retry the secret fetch 6× (clearing the cache each time). CI uploads this script on every deploy and bootstrap downloads it on every service start, so existing VM disks pick it up. - Packer `startup-fallback.sh`: clears the cache too (new images). ### 2. Retry after a failed change-to-public 404s forever `change_to_public` deletes screens, `videos/with_vocals.mkv` and the finals but left their `file_urls`. `/retry` then saw `videos.with_vocals` + `instrumental_selection`, skipped the render and 404'd on the deleted MKV. It now drops those refs, as `ThemeRerenderService` already does. ## Testing - New `backend/tests/test_encoding_worker_startup_script.py` and `test_change_visibility.py::test_drops_file_urls_for_deleted_artifacts` - `pytest backend/tests/test_encoding_worker_startup_script.py backend/tests/test_encoding_worker_self_heal.py backend/tests/test_encoding_worker_code_paths.py backend/tests/test_change_visibility.py`: all pass 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01JUFVxMZME6ydzkmLvyuXRo <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * When a private job is made public, references to deleted screen and video files are cleared before processing resumes, preventing stale files from appearing. * Improved encoding worker startup reliability by retrying API-key retrieval when the metadata service or secret store is temporarily unavailable. <!-- end of auto-generated comment: release notes by coderabbit.ai --> @coderabbitai ignore --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
180 lines
7.5 KiB
Bash
180 lines
7.5 KiB
Bash
#!/bin/bash
|
|
# Encoding Worker Startup Script
|
|
#
|
|
# This script is deployed to GCS and downloaded by bootstrap.sh on every service start.
|
|
# It handles:
|
|
# 1. Fetching API key from Secret Manager
|
|
# 2. Downloading the current wheel (fixed path, no version sorting!)
|
|
# 3. Installing the wheel
|
|
# 4. Verifying installed version matches expected version
|
|
# 5. Failing fast if version mismatch (don't start with wrong code)
|
|
#
|
|
# CI uploads this script to: gs://karaoke-gen-storage-nomadkaraoke/encoding-worker/startup.sh
|
|
# CI also uploads:
|
|
# - gs://karaoke-gen-storage-nomadkaraoke/wheels/karaoke_gen-current.whl (latest wheel)
|
|
# - gs://karaoke-gen-storage-nomadkaraoke/encoding-worker/version.txt (expected version)
|
|
|
|
set -e
|
|
|
|
BUCKET="gs://karaoke-gen-storage-nomadkaraoke"
|
|
WORKER_DIR="/opt/encoding-worker"
|
|
VENV="${WORKER_DIR}/venv"
|
|
TMP_DIR="/tmp/encoding-worker-startup"
|
|
LOG_FILE="/var/log/encoding-worker-startup.log"
|
|
|
|
exec > >(tee -a "$LOG_FILE") 2>&1
|
|
|
|
echo "=============================================="
|
|
echo "Encoding Worker Startup: $(date)"
|
|
echo "=============================================="
|
|
|
|
# Create temp directory for downloads
|
|
rm -rf "${TMP_DIR}"
|
|
mkdir -p "${TMP_DIR}"
|
|
|
|
# 1. Fetch API key from Secret Manager
|
|
#
|
|
# Boot race (2026-10-09, 2026-10-12): if gcloud's first call runs before DHCP
|
|
# brings up the metadata server, gcloud writes "False" to its on-GCE cache
|
|
# (/root/.config/gcloud/gce) and every later gcloud call in this boot runs
|
|
# with no credentials -> empty API key -> the worker's self-heal refuses to
|
|
# serve -> systemd re-runs this script, which hits the same poisoned cache,
|
|
# forever (until the VM idle-stops). So: drop the cache, wait for the metadata
|
|
# server, and retry the fetch instead of giving up after one attempt.
|
|
echo "[1/5] Fetching API key from Secret Manager..."
|
|
rm -f /root/.config/gcloud/gce "${HOME:-/root}/.config/gcloud/gce"
|
|
for attempt in $(seq 1 30); do
|
|
if curl -sf --connect-timeout 2 --max-time 3 -H "Metadata-Flavor: Google" \
|
|
"http://metadata.google.internal/computeMetadata/v1/instance/id" >/dev/null 2>&1; then
|
|
break
|
|
fi
|
|
echo "Metadata server not reachable yet (attempt ${attempt}/30), waiting 2s..."
|
|
sleep 2
|
|
done
|
|
|
|
ENCODING_API_KEY=""
|
|
for attempt in $(seq 1 6); do
|
|
ENCODING_API_KEY=$(gcloud secrets versions access latest --secret=encoding-worker-api-key 2>/dev/null || echo "")
|
|
[ -n "$ENCODING_API_KEY" ] && break
|
|
echo "API key fetch failed (attempt ${attempt}/6), retrying in 5s..."
|
|
rm -f /root/.config/gcloud/gce "${HOME:-/root}/.config/gcloud/gce"
|
|
sleep 5
|
|
done
|
|
|
|
if [ -z "$ENCODING_API_KEY" ]; then
|
|
# The worker refuses to serve without a key (self_heal boot check), so fail
|
|
# here and let systemd re-run the bootstrap rather than start broken.
|
|
echo "FATAL: No API key found in Secret Manager; exiting so systemd retries"
|
|
exit 1
|
|
fi
|
|
|
|
# Read this VM's GCE instance name so the encoding worker can tag its
|
|
# Firestore-persisted job state with vm_name. This lets each VM only see
|
|
# its own jobs at startup (different fallback VMs have separate state).
|
|
# Falls back to the OS hostname (which on GCE is the instance name anyway)
|
|
# if the metadata server isn't reachable for any reason.
|
|
VM_NAME=$(curl -s --connect-timeout 2 -H "Metadata-Flavor: Google" \
|
|
"http://metadata.google.internal/computeMetadata/v1/instance/name" 2>/dev/null \
|
|
|| hostname)
|
|
echo "VM name: ${VM_NAME}"
|
|
|
|
# Write environment file for systemd.
|
|
#
|
|
# GCE_METADATA_MTLS_MODE=none forces HTTP metadata server access on port 80.
|
|
# google-auth >=2.40 uses mTLS over HTTPS (port 443) by default if
|
|
# /run/google-mds-mtls/ certs exist on the VM, which is the case on newer
|
|
# GCE images. The mTLS handshake has been observed to fail with
|
|
# `SSL CERTIFICATE_VERIFY_FAILED` on freshly-provisioned encoding-worker-
|
|
# fallback-* VMs (2026-05-06), preventing the worker from refreshing its
|
|
# service-account token. Forcing the standard HTTP path avoids the issue
|
|
# entirely while still using the GCE metadata server for auth.
|
|
cat > "${WORKER_DIR}/env" <<EOF
|
|
ENCODING_API_KEY=${ENCODING_API_KEY}
|
|
GCE_METADATA_MTLS_MODE=none
|
|
ENCODING_WORKER_VM_NAME=${VM_NAME}
|
|
EOF
|
|
chmod 600 "${WORKER_DIR}/env"
|
|
|
|
# 2. Download expected version manifest
|
|
echo "[2/5] Downloading version manifest..."
|
|
if ! gsutil cp "${BUCKET}/encoding-worker/version.txt" "${TMP_DIR}/version.txt" 2>&1; then
|
|
echo "WARNING: Could not download version.txt - skipping version verification"
|
|
EXPECTED_VERSION=""
|
|
else
|
|
EXPECTED_VERSION=$(cat "${TMP_DIR}/version.txt" | tr -d '[:space:]')
|
|
echo "Expected version: ${EXPECTED_VERSION}"
|
|
fi
|
|
|
|
# 3. Download current wheel (FIXED PATH - no sorting!)
|
|
echo "[3/5] Downloading current wheel..."
|
|
|
|
# pip requires properly named wheel files (PEP 427)
|
|
# We need to rename the wheel to a valid name based on the version
|
|
if [ -n "$EXPECTED_VERSION" ]; then
|
|
WHEEL_NAME="karaoke_gen-${EXPECTED_VERSION}-py3-none-any.whl"
|
|
else
|
|
# Fallback name if no version.txt
|
|
WHEEL_NAME="karaoke_gen-0.0.0-py3-none-any.whl"
|
|
fi
|
|
WHEEL_PATH="${TMP_DIR}/${WHEEL_NAME}"
|
|
|
|
if ! gsutil cp "${BUCKET}/wheels/karaoke_gen-current.whl" "${WHEEL_PATH}" 2>&1; then
|
|
echo "ERROR: Failed to download karaoke_gen-current.whl"
|
|
echo "Checking for any available wheel as fallback..."
|
|
|
|
# Fallback: try to get the latest versioned wheel (with proper version sort)
|
|
LATEST_WHEEL=$(gsutil ls "${BUCKET}/wheels/karaoke_gen-*.whl" 2>/dev/null | grep -v 'current' | sort -V | tail -1 || echo "")
|
|
|
|
if [ -n "$LATEST_WHEEL" ]; then
|
|
echo "Fallback: Using ${LATEST_WHEEL}"
|
|
gsutil cp "${LATEST_WHEEL}" "${WHEEL_PATH}"
|
|
else
|
|
echo "FATAL: No wheel available in GCS"
|
|
exit 1
|
|
fi
|
|
fi
|
|
|
|
echo "Downloaded wheel as: ${WHEEL_NAME}"
|
|
|
|
# 4. Install the wheel (code only)
|
|
# Use --no-deps at boot because full dependency resolution on the complex dep
|
|
# graph (langchain, CUDA, etc.) can hit pip's "resolution-too-deep" error and
|
|
# crash-loop the service before it can serve health checks (see #587).
|
|
#
|
|
# The full karaoke-gen dependency tree (torch [CPU-only], langchain, tenacity,
|
|
# ...) is now BAKED INTO THE IMAGE at build time (provision.sh), so this --no-deps
|
|
# code upgrade just fast-forwards the app to the newest wheel on an already-
|
|
# complete venv. ensure_latest_wheel() in backend/services/gce_encoding/main.py
|
|
# remains as a runtime backstop (installs WITH deps + verifies imports) for the
|
|
# rare case of an image predating the dep-bake or a partial venv.
|
|
echo "[4/5] Installing wheel..."
|
|
"${VENV}/bin/pip" install --upgrade --quiet --force-reinstall --no-deps "${WHEEL_PATH}"
|
|
|
|
# 5. Verify installed version
|
|
echo "[5/5] Verifying installed version..."
|
|
INSTALLED_VERSION=$("${VENV}/bin/pip" show karaoke-gen 2>/dev/null | grep "^Version:" | cut -d' ' -f2 | tr -d '[:space:]')
|
|
echo "Installed version: ${INSTALLED_VERSION}"
|
|
|
|
if [ -n "$EXPECTED_VERSION" ]; then
|
|
if [ "${INSTALLED_VERSION}" != "${EXPECTED_VERSION}" ]; then
|
|
echo "=============================================="
|
|
echo "FATAL: VERSION MISMATCH!"
|
|
echo " Expected: ${EXPECTED_VERSION}"
|
|
echo " Installed: ${INSTALLED_VERSION}"
|
|
echo "=============================================="
|
|
echo "Refusing to start with incorrect version."
|
|
echo "This indicates a deployment issue - check CI logs."
|
|
exit 1
|
|
fi
|
|
echo "Version verified successfully!"
|
|
else
|
|
echo "WARNING: Version verification skipped (no version.txt)"
|
|
fi
|
|
|
|
# Cleanup
|
|
rm -rf "${TMP_DIR}"
|
|
|
|
echo "=============================================="
|
|
echo "Startup complete: karaoke-gen v${INSTALLED_VERSION}"
|
|
echo "=============================================="
|