mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary Fixes three cutover bugs that left the ephemeral runner dispatcher silently broken since 2026-05-17T04:56Z: 1. **`on_host_maintenance="TERMINATE"` set unconditionally** — e2 machine types reject TERMINATE unless preemptible, so 100% of general/build VM creates failed. Fix: only set TERMINATE on GPU (n1) families where it's required because attached GPUs can't live-migrate. 3 regression tests added in `test_ephemeral.py`. 2. **GPU `disk_size_gb=150` but image is 200GB** — GCE rejects boot disks smaller than the source image. Fix: raise GPU disk to 200GB. 3. **Runner user has no passwordless sudo** — workflow steps like `sudo apt-get install -y google-cloud-cli-firestore-emulator` (backend-emulator-tests) and `sudo apt-get install -y ffmpeg` (package-*, GPU integration tests) failed with "a terminal is required to read the password". The legacy GHA runner VMs had NOPASSWD sudo. Fix: add `/etc/sudoers.d/runner` in the image provision script. Fixes 1 + 2 are dispatcher code (already applied via `pulumi up --target ...:runner-manager-source ...:runner-manager-function` from this branch). Fix 3 requires a new image build (triggered: build-runner-images.yml on this branch). ## Why the trivial file touches `backend/main.py` and `karaoke_gen/__init__.py` carry one-line comments to flip the `backend` and `package` `dorny/paths-filter` outputs so this PR exercises **all five PR-triggered self-hosted jobs** end-to-end on ephemeral runners. Verification PR (paired): nomadkaraoke/python-audio-separator#286. ## Test plan - [x] `pytest test_ephemeral.py` — 26 passed locally (3 new scheduling tests) - [x] Pulumi applied locally (both iterations, function update verified) - [x] First ephemeral general VM create succeeded (4 RUNNING; previously 100% failure) - [ ] Image rebuild completes for general+build+gpu with new sudo provision - [ ] PR CI passes with all 5 self-hosted jobs running on the new image - [ ] python-audio-separator#286 GPU integration tests pass on the new GPU image - [ ] After merge: `deploy-backend` succeeds on ephemeral build runner; prod /api/health/detailed reports new version - [ ] Phase 4 decommission of 7 legacy VMs+disks (separate PR, ~$220/mo saving) ## Context - `karaoke-gen/docs/EPHEMERAL-GHA-RUNNERS.md` - `docs/archive/2026-05-16-ephemeral-gha-runners-plan.md` (workspace) - Memory: `project_cost_sprint_may2026.md` @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
470 lines
21 KiB
Bash
Executable File
470 lines
21 KiB
Bash
Executable File
#!/bin/bash
|
|
# Image-time provisioning for ephemeral GitHub Actions runner images.
|
|
# Builds one of three variants:
|
|
# general — Docker, Python 3.13, Node 20, Java 21, FFmpeg, gcloud, Poetry, runner binary
|
|
# build — Same as general + pre-pulled Docker base images for deploy-backend
|
|
# gpu — NVIDIA driver/CUDA, Python 3.13, FFmpeg+libsamplerate, gcloud, Poetry,
|
|
# runner binary, audio-separator models (~14GB)
|
|
#
|
|
# Runs on a fresh Debian 12 VM as root. Idempotent (safe to re-run).
|
|
# Signals completion by writing /opt/runner-image-ready.
|
|
|
|
set -euo pipefail
|
|
|
|
# Redirect ALL output to BOTH the on-VM log file AND the GCE serial console.
|
|
# The serial console is readable via `gcloud compute instances
|
|
# get-serial-port-output` with no SSH/IAP dependency, which is the only
|
|
# diagnostic channel that has worked reliably for us when IAP tunnel issues
|
|
# break SSH. Without this, a script crash mid-provision is invisible until
|
|
# someone can get a working SSH session, which can take many retries.
|
|
mkdir -p /var/log
|
|
# /dev/ttyS0 is the GCE serial console (read via `gcloud compute instances
|
|
# get-serial-port-output --port=1`). Use a single tee with multiple targets
|
|
# so a failure on one (e.g. ttyS0 unwritable in some test env) doesn't break
|
|
# the other; tee will print an error and continue with the remaining files.
|
|
exec > >(tee /var/log/runner-image-provision.log /dev/ttyS0 2>/dev/null) 2>&1
|
|
|
|
# ERR trap: print the failed command and line number to stdout (which is
|
|
# also serial-console-routed) so any non-zero exit leaves a clear marker.
|
|
on_err() {
|
|
local rc=$?
|
|
local line=${BASH_LINENO[0]:-?}
|
|
local cmd="${BASH_COMMAND:-?}"
|
|
echo
|
|
echo "########################################################################"
|
|
echo "### runner-image-provision FAILED"
|
|
echo "### exit=$rc line=$line cmd=$cmd"
|
|
echo "### variant=${VARIANT:-?} ts=$(date -u +%FT%TZ)"
|
|
echo "########################################################################"
|
|
# Drop a marker on disk so the GHA workflow can `gcloud compute scp`
|
|
# this back if SSH happens to work — but the serial console output
|
|
# above is the primary diagnostic.
|
|
printf '{"variant":"%s","rc":%d,"line":%s,"cmd":"%s","ts":"%s"}\n' \
|
|
"${VARIANT:-unknown}" "$rc" "$line" "${cmd//\"/\\\"}" "$(date -u +%FT%TZ)" \
|
|
> /opt/runner-image-failed 2>/dev/null || true
|
|
}
|
|
trap on_err ERR
|
|
|
|
# Phase checkpoint helper — prominent on serial console.
|
|
ck() {
|
|
echo
|
|
echo "================================================================"
|
|
echo "### runner-image: $* ($(date -u +%FT%TZ))"
|
|
echo "================================================================"
|
|
}
|
|
|
|
# Resolve the variant in priority order:
|
|
# 1. positional arg (interactive runs / tests)
|
|
# 2. RUNNER_IMAGE_VARIANT env var
|
|
# 3. GCE instance metadata key `image-variant` (workflow sets this)
|
|
VARIANT="${1:-${RUNNER_IMAGE_VARIANT:-}}"
|
|
if [[ -z "$VARIANT" ]]; then
|
|
VARIANT=$(curl -sf -H "Metadata-Flavor: Google" \
|
|
"http://metadata.google.internal/computeMetadata/v1/instance/attributes/image-variant" \
|
|
2>/dev/null || true)
|
|
fi
|
|
|
|
if [[ -z "$VARIANT" ]]; then
|
|
echo "ERROR: variant must be provided as \$1, \$RUNNER_IMAGE_VARIANT, or instance metadata 'image-variant'" >&2
|
|
echo "Valid variants: general | build | gpu" >&2
|
|
exit 2
|
|
fi
|
|
|
|
case "$VARIANT" in
|
|
general|build|gpu) ;;
|
|
*) echo "ERROR: unknown variant '$VARIANT'" >&2; exit 2 ;;
|
|
esac
|
|
|
|
ck "starting variant=$VARIANT"
|
|
|
|
export DEBIAN_FRONTEND=noninteractive
|
|
|
|
# ==================== Pacify apt-daily race ====================
|
|
# Fresh Debian VMs auto-start `apt-daily.service` / `apt-daily-upgrade.service`
|
|
# in the background. These hold /var/lib/apt/lists/lock and /var/lib/dpkg/lock,
|
|
# and any concurrent `apt-get` (especially the GHA runner's own
|
|
# installdependencies.sh later in this script) fails with:
|
|
# E: Could not get lock /var/lib/apt/lists/lock. It is held by process N (apt-get)
|
|
# Disable the timers + services up front, then wait for any in-flight apt run
|
|
# to release its locks. Without this we hit a race ~30% of the time.
|
|
ck "phase: pacify apt-daily"
|
|
systemctl stop apt-daily.timer apt-daily-upgrade.timer 2>/dev/null || true
|
|
systemctl disable apt-daily.timer apt-daily-upgrade.timer 2>/dev/null || true
|
|
systemctl mask apt-daily.service apt-daily-upgrade.service 2>/dev/null || true
|
|
# Wait for any apt-get already in flight to release its lock (up to 2 min).
|
|
for i in $(seq 1 60); do
|
|
if ! fuser /var/lib/apt/lists/lock /var/lib/dpkg/lock-frontend /var/lib/dpkg/lock 2>/dev/null; then
|
|
break
|
|
fi
|
|
echo " apt locked by another process, waiting 2s..."
|
|
sleep 2
|
|
done
|
|
|
|
# ==================== Network ====================
|
|
ck "phase: network (apt IPv4)"
|
|
# Cloud NAT doesn't support IPv6 — force apt to IPv4 to avoid hangs.
|
|
echo 'Acquire::ForceIPv4 "true";' > /etc/apt/apt.conf.d/99force-ipv4
|
|
|
|
# ==================== Helpers ====================
|
|
install_gpg_key() {
|
|
local url="$1" dest="$2"
|
|
[[ -f "$dest" ]] && return 0
|
|
curl -fsSL "$url" | gpg --batch --dearmor -o "$dest"
|
|
}
|
|
|
|
retry() {
|
|
local n=0 max="${RETRIES:-3}" delay="${RETRY_DELAY:-5}"
|
|
until "$@"; do
|
|
n=$((n+1))
|
|
if (( n >= max )); then return 1; fi
|
|
echo "Retry $n/$max after failure: $*"
|
|
sleep "$delay"
|
|
done
|
|
}
|
|
|
|
# ==================== Base packages (all variants) ====================
|
|
ck "phase: apt base packages"
|
|
apt-get update
|
|
apt-get install -y \
|
|
curl git jq unzip \
|
|
build-essential libssl-dev libffi-dev \
|
|
software-properties-common apt-transport-https ca-certificates gnupg lsb-release
|
|
|
|
# ==================== Variant: GPU — kernel + NVIDIA + CUDA ====================
|
|
if [[ "$VARIANT" == "gpu" ]]; then
|
|
ck "phase: gpu kernel + nvidia + cuda"
|
|
RUNNING_KERNEL="$(uname -r)"
|
|
REBOOT_FLAG="/var/lib/gpu-image-kernel-upgraded"
|
|
echo "Running kernel: $RUNNING_KERNEL"
|
|
|
|
if ! apt-cache show "linux-headers-$RUNNING_KERNEL" &>/dev/null; then
|
|
if [[ -f "$REBOOT_FLAG" ]]; then
|
|
echo "ERROR: kernel headers still unavailable after upgrade+reboot" >&2
|
|
exit 1
|
|
fi
|
|
echo "Headers for $RUNNING_KERNEL unavailable — upgrading kernel and rebooting"
|
|
apt-get install -y linux-image-cloud-amd64 linux-headers-cloud-amd64
|
|
touch "$REBOOT_FLAG"
|
|
reboot
|
|
exit 0
|
|
fi
|
|
rm -f "$REBOOT_FLAG"
|
|
|
|
echo "Installing kernel headers for $RUNNING_KERNEL"
|
|
apt-get install -y "linux-headers-$RUNNING_KERNEL" dkms
|
|
|
|
# Strip stale kernel headers
|
|
for old in /usr/src/linux-headers-*; do
|
|
ver="$(basename "$old" | sed 's/^linux-headers-//')"
|
|
[[ "$ver" == "$RUNNING_KERNEL" || "$ver" == *common* ]] && continue
|
|
pkg="linux-headers-$ver"
|
|
if dpkg -l "$pkg" 2>/dev/null | grep -q '^ii'; then
|
|
apt-get remove -y "$pkg" 2>/dev/null || true
|
|
fi
|
|
done
|
|
|
|
if ! command -v nvidia-smi &>/dev/null; then
|
|
echo "Installing NVIDIA driver + CUDA 12.4"
|
|
install_gpg_key \
|
|
"https://developer.download.nvidia.com/compute/cuda/repos/debian12/x86_64/3bf863cc.pub" \
|
|
"/usr/share/keyrings/cuda-archive-keyring.gpg"
|
|
echo "deb [signed-by=/usr/share/keyrings/cuda-archive-keyring.gpg] https://developer.download.nvidia.com/compute/cuda/repos/debian12/x86_64/ /" \
|
|
> /etc/apt/sources.list.d/cuda.list
|
|
apt-get update
|
|
apt-get install -y cuda-drivers cuda-toolkit-12-4
|
|
fi
|
|
fi
|
|
|
|
# ==================== Docker (general + build only) ====================
|
|
if [[ "$VARIANT" != "gpu" ]]; then
|
|
ck "phase: docker"
|
|
if ! command -v docker &>/dev/null; then
|
|
echo "Installing Docker"
|
|
install_gpg_key \
|
|
"https://download.docker.com/linux/debian/gpg" \
|
|
"/usr/share/keyrings/docker-archive-keyring.gpg"
|
|
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian $(lsb_release -cs) stable" \
|
|
> /etc/apt/sources.list.d/docker.list
|
|
apt-get update
|
|
apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
|
|
fi
|
|
systemctl enable docker
|
|
fi
|
|
|
|
# ==================== Python 3.13 ====================
|
|
# Use pyenv for ALL variants. Compiling Python 3.13 from source by hand
|
|
# proved fragile (3.13.0 hit an install-time ensurepip bug; 3.13.2 without
|
|
# --enable-optimizations produced a broken _socket extension that crashed
|
|
# any urllib-using script with `SystemError: error return without exception
|
|
# set`). pyenv has been hardened against these issues over many releases.
|
|
ck "phase: python 3.13"
|
|
if [[ ! -f /opt/pyenv/versions/3.13.0/bin/python3.13 ]]; then
|
|
echo "Installing Python 3.13 via pyenv"
|
|
apt-get install -y make zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev \
|
|
wget llvm libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev liblzma-dev
|
|
[[ -d /opt/pyenv ]] || { export PYENV_ROOT=/opt/pyenv; curl https://pyenv.run | bash; }
|
|
export PATH="/opt/pyenv/bin:$PATH"
|
|
eval "$(/opt/pyenv/bin/pyenv init -)"
|
|
/opt/pyenv/bin/pyenv install 3.13.0
|
|
/opt/pyenv/bin/pyenv global 3.13.0
|
|
ln -sf /opt/pyenv/shims/python3.13 /usr/local/bin/python3.13
|
|
ln -sf /opt/pyenv/shims/python3 /usr/local/bin/python3
|
|
ln -sf /opt/pyenv/shims/pip3 /usr/local/bin/pip3
|
|
fi
|
|
PYTHON_PREFIX="/opt/pyenv/versions/3.13.0"
|
|
cat > /etc/profile.d/pyenv.sh <<'EOF'
|
|
export PYENV_ROOT=/opt/pyenv
|
|
export PATH="$PYENV_ROOT/bin:$PATH"
|
|
eval "$(pyenv init -)"
|
|
EOF
|
|
|
|
# ==================== Node.js 20 + Java 21 (general/build only) ====================
|
|
if [[ "$VARIANT" != "gpu" ]]; then
|
|
ck "phase: node 20 + java 21"
|
|
if ! command -v node &>/dev/null || ! node --version | grep -q "^v20"; then
|
|
echo "Installing Node.js 20"
|
|
retry curl -fsSL --retry 3 --retry-delay 5 https://deb.nodesource.com/setup_20.x -o /tmp/nodesource.sh
|
|
bash /tmp/nodesource.sh
|
|
apt-get install -y nodejs
|
|
rm -f /tmp/nodesource.sh
|
|
fi
|
|
|
|
if ! java -version 2>&1 | grep -q '"21\.'; then
|
|
echo "Installing Java 21 (Temurin)"
|
|
install_gpg_key \
|
|
"https://packages.adoptium.net/artifactory/api/gpg/key/public" \
|
|
"/usr/share/keyrings/adoptium.gpg"
|
|
echo "deb [signed-by=/usr/share/keyrings/adoptium.gpg] https://packages.adoptium.net/artifactory/deb $(lsb_release -cs) main" \
|
|
> /etc/apt/sources.list.d/adoptium.list
|
|
apt-get update
|
|
apt-get install -y temurin-21-jdk
|
|
fi
|
|
fi
|
|
|
|
# ==================== FFmpeg (all variants) ====================
|
|
ck "phase: ffmpeg"
|
|
if [[ "$VARIANT" == "gpu" ]]; then
|
|
apt-get install -y ffmpeg libsamplerate0 libsamplerate-dev
|
|
else
|
|
apt-get install -y ffmpeg
|
|
fi
|
|
|
|
# ==================== Poetry + gcloud (all variants) ====================
|
|
ck "phase: poetry + gcloud"
|
|
if ! command -v poetry &>/dev/null; then
|
|
echo "Installing Poetry"
|
|
# Install to /opt/poetry on every variant so the runner user can access
|
|
# poetry's binaries — /root/.local is mode 700 and shimming through there
|
|
# works only for jobs that run as root (we now have a runner user).
|
|
export POETRY_HOME="/opt/poetry"
|
|
curl -sSL https://install.python-poetry.org | python3 -
|
|
ln -sf /opt/poetry/bin/poetry /usr/local/bin/poetry
|
|
fi
|
|
|
|
if ! command -v gcloud &>/dev/null; then
|
|
echo "Installing Google Cloud SDK"
|
|
install_gpg_key \
|
|
"https://packages.cloud.google.com/apt/doc/apt-key.gpg" \
|
|
"/usr/share/keyrings/cloud.google.gpg"
|
|
echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" \
|
|
> /etc/apt/sources.list.d/google-cloud-sdk.list
|
|
apt-get update
|
|
if [[ "$VARIANT" != "gpu" ]]; then
|
|
apt-get install -y google-cloud-cli google-cloud-cli-firestore-emulator
|
|
else
|
|
apt-get install -y google-cloud-cli
|
|
fi
|
|
fi
|
|
|
|
# ==================== Runner user ====================
|
|
ck "phase: runner user"
|
|
useradd -m -s /bin/bash runner 2>/dev/null || true
|
|
if [[ "$VARIANT" != "gpu" ]]; then
|
|
usermod -aG docker runner
|
|
fi
|
|
|
|
# Many CI workflows install OS packages with `sudo apt-get …`. The legacy
|
|
# GHA runner VMs had passwordless sudo for the runner user; without it,
|
|
# steps like `sudo apt-get install -y google-cloud-cli-firestore-emulator`
|
|
# in backend-emulator-tests fail with "a terminal is required to read the
|
|
# password". Match the legacy behaviour.
|
|
mkdir -p /etc/sudoers.d
|
|
echo "runner ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/runner
|
|
chmod 0440 /etc/sudoers.d/runner
|
|
visudo -cf /etc/sudoers.d/runner
|
|
|
|
# ==================== GitHub Actions runner binary ====================
|
|
ck "phase: github actions runner binary"
|
|
RUNNER_VERSION="2.332.0"
|
|
RUNNER_DIR="/home/runner/actions-runner"
|
|
mkdir -p "$RUNNER_DIR"
|
|
|
|
CURRENT_VERSION=""
|
|
if [[ -f "$RUNNER_DIR/bin/Runner.Listener" ]]; then
|
|
CURRENT_VERSION="$("$RUNNER_DIR/bin/Runner.Listener" --version 2>/dev/null || echo unknown)"
|
|
fi
|
|
if [[ "$CURRENT_VERSION" != "$RUNNER_VERSION" ]]; then
|
|
echo "Downloading runner v$RUNNER_VERSION"
|
|
cd "$RUNNER_DIR"
|
|
curl -sLO "https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz"
|
|
tar xzf "actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz"
|
|
rm -f "actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz"
|
|
fi
|
|
chown -R runner:runner "$RUNNER_DIR"
|
|
|
|
# Pre-install runner dependencies (avoids first-run apt cost).
|
|
# installdependencies.sh is idempotent.
|
|
if [[ -x "$RUNNER_DIR/bin/installdependencies.sh" ]]; then
|
|
"$RUNNER_DIR/bin/installdependencies.sh"
|
|
fi
|
|
|
|
# Convenience symlink so external tooling (runbooks, ad-hoc SSH) can refer to
|
|
# the runner installation by a stable path. The startup script in ephemeral.py
|
|
# cds to $RUNNER_DIR directly and doesn't rely on this.
|
|
ln -sfn "$RUNNER_DIR" /opt/actions-runner
|
|
|
|
# ==================== Docker base-image pre-pull (build variant only) ====================
|
|
if [[ "$VARIANT" == "build" ]]; then
|
|
ck "phase: docker base-image pre-pull"
|
|
echo "Pre-pulling Docker base images for deploy-backend"
|
|
systemctl start docker
|
|
# Image-build VM uses the gha-runner-image-builder@ SA which has artifactregistry.reader.
|
|
gcloud auth configure-docker us-central1-docker.pkg.dev --quiet || true
|
|
gcloud auth configure-docker us-east4-docker.pkg.dev --quiet || true
|
|
docker pull us-central1-docker.pkg.dev/nomadkaraoke/karaoke-repo/karaoke-backend-base:latest || true
|
|
docker pull us-central1-docker.pkg.dev/nomadkaraoke/karaoke-repo/karaoke-backend:latest || true
|
|
docker pull fsouza/fake-gcs-server:latest || true
|
|
fi
|
|
|
|
# ==================== Audio-separator models (GPU only) ====================
|
|
# ~14GB of model weights baked into the image so jobs don't re-download.
|
|
if [[ "$VARIANT" == "gpu" ]]; then
|
|
ck "phase: audio-separator models (~14GB)"
|
|
MODEL_DIR="/opt/audio-separator-models"
|
|
mkdir -p "$MODEL_DIR"
|
|
BASE_URL="https://github.com/TRvlvr/model_repo/releases/download/all_public_uvr_models"
|
|
CONFIG_URL="https://raw.githubusercontent.com/TRvlvr/application_data/main/mdx_model_data/mdx_c_configs"
|
|
META_URL="https://raw.githubusercontent.com/TRvlvr/application_data/main"
|
|
DEMUCS_URL="https://dl.fbaipublicfiles.com/demucs/hybrid_transformer"
|
|
FALLBACK_URL="https://github.com/nomadkaraoke/python-audio-separator/releases/download/model-configs"
|
|
|
|
dl() {
|
|
local url="$1" dest="$2" fb="${3:-}"
|
|
[[ -s "$dest" ]] && return 0
|
|
echo " → $(basename "$dest")"
|
|
if curl -fSL --retry 3 --connect-timeout 30 -o "$dest" "$url" 2>/dev/null; then return 0; fi
|
|
if [[ -n "$fb" ]]; then
|
|
curl -fSL --retry 3 --connect-timeout 30 -o "$dest" "$fb" || return 1
|
|
fi
|
|
}
|
|
|
|
# Metadata
|
|
dl "$META_URL/filelists/download_checks.json" "$MODEL_DIR/download_checks.json"
|
|
dl "$META_URL/vr_model_data/model_data_new.json" "$MODEL_DIR/vr_model_data.json"
|
|
dl "$META_URL/mdx_model_data/model_data_new.json" "$MODEL_DIR/mdx_model_data.json"
|
|
|
|
# Core integration test models (8)
|
|
dl "$BASE_URL/mel_band_roformer_karaoke_aufr33_viperx_sdr_10.1956.ckpt" "$MODEL_DIR/mel_band_roformer_karaoke_aufr33_viperx_sdr_10.1956.ckpt"
|
|
dl "$FALLBACK_URL/mel_band_roformer_karaoke_aufr33_viperx_sdr_10.1956_config.yaml" "$MODEL_DIR/mel_band_roformer_karaoke_aufr33_viperx_sdr_10.1956_config.yaml"
|
|
dl "$BASE_URL/kuielab_b_vocals.onnx" "$MODEL_DIR/kuielab_b_vocals.onnx"
|
|
dl "$BASE_URL/MGM_MAIN_v4.pth" "$MODEL_DIR/MGM_MAIN_v4.pth"
|
|
dl "$BASE_URL/UVR-MDX-NET-Inst_HQ_4.onnx" "$MODEL_DIR/UVR-MDX-NET-Inst_HQ_4.onnx"
|
|
dl "$BASE_URL/2_HP-UVR.pth" "$MODEL_DIR/2_HP-UVR.pth"
|
|
dl "$BASE_URL/htdemucs_6s.yaml" "$MODEL_DIR/htdemucs_6s.yaml"
|
|
dl "$DEMUCS_URL/5c90dfd2-34c22ccb.th" "$MODEL_DIR/5c90dfd2-34c22ccb.th"
|
|
dl "$BASE_URL/model_bs_roformer_ep_937_sdr_10.5309.ckpt" "$MODEL_DIR/model_bs_roformer_ep_937_sdr_10.5309.ckpt"
|
|
dl "$CONFIG_URL/model_bs_roformer_ep_937_sdr_10.5309.yaml" "$MODEL_DIR/model_bs_roformer_ep_937_sdr_10.5309.yaml"
|
|
dl "$BASE_URL/model_bs_roformer_ep_317_sdr_12.9755.ckpt" "$MODEL_DIR/model_bs_roformer_ep_317_sdr_12.9755.ckpt"
|
|
dl "$CONFIG_URL/model_bs_roformer_ep_317_sdr_12.9755.yaml" "$MODEL_DIR/model_bs_roformer_ep_317_sdr_12.9755.yaml"
|
|
|
|
# Ensemble preset models (15, ~12GB)
|
|
for m in \
|
|
bs_roformer_instrumental_resurrection_unwa.ckpt \
|
|
bs_roformer_vocals_resurrection_unwa.ckpt \
|
|
bs_roformer_vocals_revive_v2_unwa.ckpt \
|
|
bs_roformer_vocals_revive_v3e_unwa.ckpt \
|
|
mel_band_roformer_instrumental_becruily.ckpt \
|
|
mel_band_roformer_instrumental_fv7z_gabox.ckpt \
|
|
mel_band_roformer_instrumental_instv8_gabox.ckpt \
|
|
mel_band_roformer_karaoke_becruily.ckpt \
|
|
mel_band_roformer_karaoke_gabox_v2.ckpt \
|
|
mel_band_roformer_kim_ft2_bleedless_unwa.ckpt \
|
|
mel_band_roformer_vocals_becruily.ckpt \
|
|
mel_band_roformer_vocals_fv4_gabox.ckpt \
|
|
melband_roformer_big_beta6x.ckpt \
|
|
melband_roformer_inst_v1e_plus.ckpt \
|
|
UVR-MDX-NET-Inst_HQ_5.onnx; do
|
|
dl "$FALLBACK_URL/$m" "$MODEL_DIR/$m"
|
|
done
|
|
for c in \
|
|
config_bs_roformer_instrumental_resurrection_unwa.yaml \
|
|
config_bs_roformer_vocals_resurrection_unwa.yaml \
|
|
config_bs_roformer_vocals_revive_unwa.yaml \
|
|
config_mel_band_roformer_instrumental_becruily.yaml \
|
|
config_mel_band_roformer_instrumental_gabox.yaml \
|
|
config_mel_band_roformer_karaoke_becruily.yaml \
|
|
config_mel_band_roformer_karaoke_gabox.yaml \
|
|
config_mel_band_roformer_kim_ft_unwa.yaml \
|
|
config_mel_band_roformer_vocals_becruily.yaml \
|
|
config_mel_band_roformer_vocals_gabox.yaml \
|
|
config_melbandroformer_big_beta6x.yaml \
|
|
config_melbandroformer_inst.yaml; do
|
|
dl "$FALLBACK_URL/$c" "$MODEL_DIR/$c"
|
|
done
|
|
|
|
# Multi-stem test models
|
|
dl "$BASE_URL/17_HP-Wind_Inst-UVR.pth" "$MODEL_DIR/17_HP-Wind_Inst-UVR.pth"
|
|
dl "$FALLBACK_URL/MDX23C-DrumSep-aufr33-jarredou.ckpt" "$MODEL_DIR/MDX23C-DrumSep-aufr33-jarredou.ckpt"
|
|
dl "$FALLBACK_URL/dereverb_mel_band_roformer_anvuew_sdr_19.1729.ckpt" "$MODEL_DIR/dereverb_mel_band_roformer_anvuew_sdr_19.1729.ckpt"
|
|
dl "$FALLBACK_URL/config_dereverb_mel_band_roformer_anvuew.yaml" "$MODEL_DIR/config_dereverb_mel_band_roformer_anvuew.yaml"
|
|
|
|
chown -R runner:runner "$MODEL_DIR"
|
|
echo "Audio-separator models: $(du -sh "$MODEL_DIR" | cut -f1)"
|
|
fi
|
|
|
|
# ==================== Tool cache for setup-python action ====================
|
|
ck "phase: tool cache for setup-python"
|
|
# setup-python@v5 looks for prebuilt Python in $RUNNER_TOOL_CACHE/Python/<ver>/x64
|
|
TOOL_CACHE="/home/runner/actions-runner/_work/_tool"
|
|
mkdir -p "$TOOL_CACHE/Python/3.13.0/x64"
|
|
if [[ -d "$PYTHON_PREFIX" && ! -f "$TOOL_CACHE/Python/3.13.0/x64.complete" ]]; then
|
|
cp -r "$PYTHON_PREFIX"/* "$TOOL_CACHE/Python/3.13.0/x64/"
|
|
(cd "$TOOL_CACHE/Python/3.13.0/x64/bin"; [[ -f python ]] || ln -s python3.13 python; [[ -f python3 ]] || ln -s python3.13 python3)
|
|
touch "$TOOL_CACHE/Python/3.13.0/x64.complete"
|
|
fi
|
|
chown -R runner:runner /home/runner/actions-runner/_work
|
|
|
|
# ==================== Docker disk-pressure cleanup cron ====================
|
|
# Carried over from the legacy startup script. Useful even on ephemeral VMs
|
|
# because deploy-backend can fill the disk during long Docker builds.
|
|
if [[ "$VARIANT" != "gpu" ]]; then
|
|
cat > /etc/cron.hourly/docker-cleanup <<'EOF'
|
|
#!/bin/bash
|
|
exec 1> >(logger -t docker-cleanup) 2>&1
|
|
THRESHOLD=70
|
|
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
|
|
if (( USAGE > THRESHOLD )); then
|
|
docker system prune -af
|
|
docker builder prune -af
|
|
fi
|
|
EOF
|
|
chmod +x /etc/cron.hourly/docker-cleanup
|
|
fi
|
|
|
|
# ==================== Cleanup before imaging ====================
|
|
ck "phase: cleanup before imaging"
|
|
apt-get clean
|
|
rm -rf /var/lib/apt/lists/*
|
|
journalctl --vacuum-time=1s 2>/dev/null || true
|
|
|
|
# Remove SSH host keys — regenerated on first boot of cloned VMs
|
|
rm -f /etc/ssh/ssh_host_*
|
|
|
|
# Mark image ready (the workflow polls for this file via gcloud ssh, but
|
|
# the primary completion signal is now the "READY" line on the serial
|
|
# console below).
|
|
date -u +%FT%TZ > /opt/runner-image-ready
|
|
echo "$VARIANT" > /opt/runner-image-variant
|
|
|
|
ck "READY: variant=$VARIANT ts=$(date -u +%FT%TZ)"
|