Files
karaoke-gen/backend/Dockerfile.gpu-base
T
Andrew Beveridge ☄️andClaude Opus 5.5 0d0605d7b5 fix: final encode waits for stem separation instead of stranding the job; pin audio-separator 0.44.2 (v0.278.11) (#1191)
## Problem
The daily E2E Stage 2 failed on 2026-10-17 (run #216) and 2026-10-18
(#217). When a customer finishes review **before** stem separation does,
`video-encoding-job` fails its prerequisites (`Missing instrumental:
instrumental_clean`) 3× within seconds. The job then sits at
`instrumental_selected` ("Starting final encode") forever: it's never
marked failed, nothing alerts, and separation finishing doesn't
re-trigger the encode.

The race became common when the GPU image's unpinned `pip install
"audio-separator[gpu]"` picked up **0.47.0** on the 10-17 rebuild. On
L4, the mel-band RoFormer **karaoke** models run 3–5× slower than on
0.44.2, so total separation went from ~5 to ~10–16 min:

| model (~3-min track) | 0.44.2 | 0.47.0 |
|---|---|---|
| karaoke_aufr33_viperx | ~25s | 75s |
| karaoke_gabox_v2 | ~26s | 149s |
| karaoke_becruily | ~27s | 157s |
| instrumental fv7z / resurrection | ~50–60s | ~62s |

The model YAMLs still say `num_overlap: 8, batch_size: 1`, so this isn't
the #299 YAML-defaults change; the cause is somewhere in 0.45–0.47's
RoFormer inference changes. That root cause is tracked separately.

## Fix
- **Pin** `audio-separator[gpu]==0.44.2` in
`backend/Dockerfile.gpu-base` (`ARG AUDIO_SEPARATOR_VERSION`). Changing
this file triggers the GPU base image rebuild in CI.
- **Video worker parks instead of failing.** When the selected stem is
missing *and* separation is still running (`audio_complete` false, valid
status), it sets `state_data.video_waiting_for_stems`, puts a
user-visible message ("Waiting for audio separation to finish; the final
video starts automatically") and exits 0, so Cloud Run doesn't retry. It
re-checks after parking, and an atomic Firestore test-and-clear
(`claim_job_state_flag`) means exactly one of the video worker and the
audio worker proceeds if the stems land mid-park.
- **Audio worker resumes it.** After `mark_audio_complete`,
`resume_video_if_waiting_for_stems` re-triggers a parked encode. The
fake-workers pipeline does the same, so the local stack matches prod.
- **Backstop in `recover_stuck_jobs`.** `instrumental_selected` for more
than 10 min with the stem present → re-trigger the video worker (covers
a lost resume or lost dispatch). More than 60 min with separation never
finishing → `mark_job_failed` (`stage=video_waiting_for_stems`), so the
job fails loudly. Jobs older than 24h are never revived. (No jobs are in
`instrumental_selected` in prod right now.)

## Testing
- New `backend/tests/test_video_waits_for_stems.py` (16 tests): pending
detection (including that a wrong status never parks), park / continue /
claim-race outcomes, and resume (flag claimed / no flag / job moved on).
- `test_recover_stuck_jobs_repark.py`: 5 new sweep tests (re-trigger,
fresh, still separating, never-ready → fail, >24h ignored).
- Full `backend/tests` (excluding emulator): 6455 passed.
- Local CodeRabbit review: 0 findings.

## Verify after deploy
- The GPU job logs `Separator version 0.44.2`, and karaoke-model
separation is back to ~20–30s.
- The next daily E2E Stage 2 passes.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_019pmG2XAgtUguEJWLFpkKHN

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 02:21:52 -04:00

145 lines
6.5 KiB
Docker

# GPU base image with CUDA, PyTorch, audio-separator, and pre-downloaded models
# This image is rebuilt only when deps or models change (infrequently).
#
# Build with: gcloud builds submit --config=backend/cloudbuild-gpu-base.yaml --project=nomadkaraoke --timeout=3600s
# Or manually: docker build -f backend/Dockerfile.gpu-base -t karaoke-backend-gpu-base .
#
# Layer caching strategy:
# 1. System deps + fonts + ffmpeg (rarely change, always cached)
# 2. CUDA library path configuration
# 3. pyproject.toml + package stubs (changes with dep updates)
# 4. pip install karaoke-gen deps + audio-separator[gpu] (slow, cached unless pyproject.toml changes)
# 5. Source code copy (changes every rebuild, but AFTER deps are installed)
# 6. spaCy model download (cached unless pip install layer changes)
# 7. Pre-downloaded ensemble preset models (~1-1.5 GB, cached unless pip layer changes)
FROM nvidia/cuda:12.6.3-runtime-ubuntu22.04
# Prevent interactive prompts during package installation
ENV DEBIAN_FRONTEND=noninteractive
# Install Python 3.11 from deadsnakes PPA and system dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
software-properties-common \
&& add-apt-repository -y ppa:deadsnakes/ppa \
&& apt-get update && apt-get install -y --no-install-recommends \
# Python 3.11
python3.11 \
python3.11-dev \
python3.11-venv \
# Audio libraries
libsndfile1 \
libsox-dev \
sox \
# Build tools
build-essential \
# Utilities
curl \
xz-utils \
&& rm -rf /var/lib/apt/lists/* \
&& python3.11 --version
# Install comprehensive font support for international lyrics and symbols
# Noto fonts provide full Unicode coverage including:
# - Latin, Greek, Cyrillic (core)
# - CJK: Chinese, Japanese, Korean (~150MB)
# - Arabic, Hebrew, Thai, Hindi, and other scripts (extra)
# - Color emoji (limited support in libass, but available)
# - Musical symbols (♪ ♫ etc.)
# Note: This adds ~200-300MB to the image but ensures all languages render correctly
RUN apt-get update && apt-get install -y --no-install-recommends \
fonts-noto-core \
fonts-noto-cjk \
fonts-noto-extra \
fonts-noto-color-emoji \
fontconfig \
&& fc-cache -f \
&& rm -rf /var/lib/apt/lists/*
# Install FFmpeg from Ubuntu package
# The audio worker only uses FFmpeg for format conversion and padding, not
# encoding-intensive tasks, so the Debian package is sufficient. The static
# build from johnvansickle.com is used for the encoding worker instead.
RUN apt-get update && apt-get install -y ffmpeg && rm -rf /var/lib/apt/lists/*
# Set Python 3.11 as default and install pip
RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.11 1 \
&& update-alternatives --install /usr/bin/python python /usr/bin/python3.11 1 \
&& curl -sS https://bootstrap.pypa.io/get-pip.py | python3.11 \
&& python3 -m pip install --no-cache-dir --upgrade pip setuptools wheel
# Set up CUDA library paths so GPU-accelerated packages can find CUDA libs at runtime
RUN echo '/usr/local/cuda/lib64' >> /etc/ld.so.conf.d/cuda.conf && ldconfig
# Environment configuration for CUDA and Python
ENV LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH \
PATH=/usr/local/cuda/bin:$PATH \
PYTHONUNBUFFERED=1 \
MODEL_DIR=/models
# Set working directory
WORKDIR /app
# Copy ONLY dependency specification files first (changes less often than source code)
COPY pyproject.toml README.md LICENSE /app/
# Create minimal package stubs so pip can resolve the editable install
# pip install -e . needs package directories to exist but doesn't validate content
RUN mkdir -p /app/karaoke_gen /app/lyrics_transcriber_temp /app/backend && \
touch /app/karaoke_gen/__init__.py /app/lyrics_transcriber_temp/__init__.py /app/backend/__init__.py
# Install all karaoke-gen Python dependencies (the slow part)
# BuildKit cache mount preserves downloaded wheels across builds
RUN --mount=type=cache,target=/root/.cache/pip \
pip install --upgrade pip && \
pip install -e .
# Install audio-separator with GPU support (PyTorch + CUDA + onnxruntime-gpu)
# This is separate from pip install -e . to keep layer caching granular
#
# IMPORTANT: Install PyTorch cu126 FIRST, then audio-separator[gpu].
# The default audio-separator[gpu] pulls PyTorch cu130, but Cloud Run L4 GPUs
# have NVIDIA driver 570 which only supports up to CUDA 12.8. PyTorch cu130
# fails with "NVIDIA driver is too old". PyTorch cu126 works with 12.8 drivers.
# Installing torch first means audio-separator[gpu] sees it as already satisfied.
#
# PINNED to an exact version: an unpinned install silently picked up 0.47.0 on the
# 2026-10-17 rebuild, and on L4 its mel-band RoFormer karaoke models run 3-5x slower
# than 0.44.2 (gabox_v2 ~26s -> 149s, becruily ~27s -> 157s for a ~3-min track;
# YAML overlap/batch are unchanged, so it is in the 0.45-0.47 inference changes).
# Total separation went from ~5 to ~10-16 min and the daily E2E started failing.
# Only bump this after timing a real job on the new version (see BACKLOG).
ARG AUDIO_SEPARATOR_VERSION=0.44.2
RUN --mount=type=cache,target=/root/.cache/pip \
pip install torch==2.6.0+cu126 torchvision==0.21.0+cu126 --index-url https://download.pytorch.org/whl/cu126 && \
pip install "audio-separator[gpu]==${AUDIO_SEPARATOR_VERSION}"
# Now copy actual source code (AFTER deps installed, so code changes don't
# invalidate the expensive pip install layer in registry cache)
COPY karaoke_gen /app/karaoke_gen
COPY lyrics_transcriber_temp /app/lyrics_transcriber_temp
COPY backend /app/backend
# Download spaCy language model for lyrics transcription
# Must be done at build time, not runtime (Cloud Run network restrictions)
RUN python -m spacy download en_core_web_sm
# Create models directory and pre-download ensemble preset models
# Baking models into the image eliminates cold-start model download latency.
# Total model size: ~1-1.5 GB for instrumental_clean + karaoke presets.
RUN mkdir -p /models
COPY backend/scripts/download_models.py /tmp/download_models.py
RUN python /tmp/download_models.py /models && \
rm /tmp/download_models.py && \
ls -lh /models/
# Label with build info
ARG BUILD_DATE
ARG BUILD_ID
ARG BASE_CONTENT_HASH
LABEL org.opencontainers.image.created="${BUILD_DATE}"
LABEL org.opencontainers.image.revision="${BUILD_ID}"
LABEL org.opencontainers.image.title="karaoke-backend-gpu-base"
LABEL org.opencontainers.image.description="GPU base image with CUDA, PyTorch, audio-separator, and pre-downloaded models for karaoke-backend audio worker"
LABEL base.content.hash="${BASE_CONTENT_HASH}"