mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Problem The daily E2E Stage 2 failed on 2026-10-17 (run #216) and 2026-10-18 (#217). When a customer finishes review **before** stem separation does, `video-encoding-job` fails its prerequisites (`Missing instrumental: instrumental_clean`) 3× within seconds. The job then sits at `instrumental_selected` ("Starting final encode") forever: it's never marked failed, nothing alerts, and separation finishing doesn't re-trigger the encode. The race became common when the GPU image's unpinned `pip install "audio-separator[gpu]"` picked up **0.47.0** on the 10-17 rebuild. On L4, the mel-band RoFormer **karaoke** models run 3–5× slower than on 0.44.2, so total separation went from ~5 to ~10–16 min: | model (~3-min track) | 0.44.2 | 0.47.0 | |---|---|---| | karaoke_aufr33_viperx | ~25s | 75s | | karaoke_gabox_v2 | ~26s | 149s | | karaoke_becruily | ~27s | 157s | | instrumental fv7z / resurrection | ~50–60s | ~62s | The model YAMLs still say `num_overlap: 8, batch_size: 1`, so this isn't the #299 YAML-defaults change; the cause is somewhere in 0.45–0.47's RoFormer inference changes. That root cause is tracked separately. ## Fix - **Pin** `audio-separator[gpu]==0.44.2` in `backend/Dockerfile.gpu-base` (`ARG AUDIO_SEPARATOR_VERSION`). Changing this file triggers the GPU base image rebuild in CI. - **Video worker parks instead of failing.** When the selected stem is missing *and* separation is still running (`audio_complete` false, valid status), it sets `state_data.video_waiting_for_stems`, puts a user-visible message ("Waiting for audio separation to finish; the final video starts automatically") and exits 0, so Cloud Run doesn't retry. It re-checks after parking, and an atomic Firestore test-and-clear (`claim_job_state_flag`) means exactly one of the video worker and the audio worker proceeds if the stems land mid-park. - **Audio worker resumes it.** After `mark_audio_complete`, `resume_video_if_waiting_for_stems` re-triggers a parked encode. The fake-workers pipeline does the same, so the local stack matches prod. - **Backstop in `recover_stuck_jobs`.** `instrumental_selected` for more than 10 min with the stem present → re-trigger the video worker (covers a lost resume or lost dispatch). More than 60 min with separation never finishing → `mark_job_failed` (`stage=video_waiting_for_stems`), so the job fails loudly. Jobs older than 24h are never revived. (No jobs are in `instrumental_selected` in prod right now.) ## Testing - New `backend/tests/test_video_waits_for_stems.py` (16 tests): pending detection (including that a wrong status never parks), park / continue / claim-race outcomes, and resume (flag claimed / no flag / job moved on). - `test_recover_stuck_jobs_repark.py`: 5 new sweep tests (re-trigger, fresh, still separating, never-ready → fail, >24h ignored). - Full `backend/tests` (excluding emulator): 6455 passed. - Local CodeRabbit review: 0 findings. ## Verify after deploy - The GPU job logs `Separator version 0.44.2`, and karaoke-model separation is back to ~20–30s. - The next daily E2E Stage 2 passes. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_019pmG2XAgtUguEJWLFpkKHN Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
145 lines
6.5 KiB
Docker
145 lines
6.5 KiB
Docker
# GPU base image with CUDA, PyTorch, audio-separator, and pre-downloaded models
|
|
# This image is rebuilt only when deps or models change (infrequently).
|
|
#
|
|
# Build with: gcloud builds submit --config=backend/cloudbuild-gpu-base.yaml --project=nomadkaraoke --timeout=3600s
|
|
# Or manually: docker build -f backend/Dockerfile.gpu-base -t karaoke-backend-gpu-base .
|
|
#
|
|
# Layer caching strategy:
|
|
# 1. System deps + fonts + ffmpeg (rarely change, always cached)
|
|
# 2. CUDA library path configuration
|
|
# 3. pyproject.toml + package stubs (changes with dep updates)
|
|
# 4. pip install karaoke-gen deps + audio-separator[gpu] (slow, cached unless pyproject.toml changes)
|
|
# 5. Source code copy (changes every rebuild, but AFTER deps are installed)
|
|
# 6. spaCy model download (cached unless pip install layer changes)
|
|
# 7. Pre-downloaded ensemble preset models (~1-1.5 GB, cached unless pip layer changes)
|
|
|
|
FROM nvidia/cuda:12.6.3-runtime-ubuntu22.04
|
|
|
|
# Prevent interactive prompts during package installation
|
|
ENV DEBIAN_FRONTEND=noninteractive
|
|
|
|
# Install Python 3.11 from deadsnakes PPA and system dependencies
|
|
RUN apt-get update && apt-get install -y --no-install-recommends \
|
|
software-properties-common \
|
|
&& add-apt-repository -y ppa:deadsnakes/ppa \
|
|
&& apt-get update && apt-get install -y --no-install-recommends \
|
|
# Python 3.11
|
|
python3.11 \
|
|
python3.11-dev \
|
|
python3.11-venv \
|
|
# Audio libraries
|
|
libsndfile1 \
|
|
libsox-dev \
|
|
sox \
|
|
# Build tools
|
|
build-essential \
|
|
# Utilities
|
|
curl \
|
|
xz-utils \
|
|
&& rm -rf /var/lib/apt/lists/* \
|
|
&& python3.11 --version
|
|
|
|
# Install comprehensive font support for international lyrics and symbols
|
|
# Noto fonts provide full Unicode coverage including:
|
|
# - Latin, Greek, Cyrillic (core)
|
|
# - CJK: Chinese, Japanese, Korean (~150MB)
|
|
# - Arabic, Hebrew, Thai, Hindi, and other scripts (extra)
|
|
# - Color emoji (limited support in libass, but available)
|
|
# - Musical symbols (♪ ♫ etc.)
|
|
# Note: This adds ~200-300MB to the image but ensures all languages render correctly
|
|
RUN apt-get update && apt-get install -y --no-install-recommends \
|
|
fonts-noto-core \
|
|
fonts-noto-cjk \
|
|
fonts-noto-extra \
|
|
fonts-noto-color-emoji \
|
|
fontconfig \
|
|
&& fc-cache -f \
|
|
&& rm -rf /var/lib/apt/lists/*
|
|
|
|
# Install FFmpeg from Ubuntu package
|
|
# The audio worker only uses FFmpeg for format conversion and padding, not
|
|
# encoding-intensive tasks, so the Debian package is sufficient. The static
|
|
# build from johnvansickle.com is used for the encoding worker instead.
|
|
RUN apt-get update && apt-get install -y ffmpeg && rm -rf /var/lib/apt/lists/*
|
|
|
|
# Set Python 3.11 as default and install pip
|
|
RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.11 1 \
|
|
&& update-alternatives --install /usr/bin/python python /usr/bin/python3.11 1 \
|
|
&& curl -sS https://bootstrap.pypa.io/get-pip.py | python3.11 \
|
|
&& python3 -m pip install --no-cache-dir --upgrade pip setuptools wheel
|
|
|
|
# Set up CUDA library paths so GPU-accelerated packages can find CUDA libs at runtime
|
|
RUN echo '/usr/local/cuda/lib64' >> /etc/ld.so.conf.d/cuda.conf && ldconfig
|
|
|
|
# Environment configuration for CUDA and Python
|
|
ENV LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH \
|
|
PATH=/usr/local/cuda/bin:$PATH \
|
|
PYTHONUNBUFFERED=1 \
|
|
MODEL_DIR=/models
|
|
|
|
# Set working directory
|
|
WORKDIR /app
|
|
|
|
# Copy ONLY dependency specification files first (changes less often than source code)
|
|
COPY pyproject.toml README.md LICENSE /app/
|
|
|
|
# Create minimal package stubs so pip can resolve the editable install
|
|
# pip install -e . needs package directories to exist but doesn't validate content
|
|
RUN mkdir -p /app/karaoke_gen /app/lyrics_transcriber_temp /app/backend && \
|
|
touch /app/karaoke_gen/__init__.py /app/lyrics_transcriber_temp/__init__.py /app/backend/__init__.py
|
|
|
|
# Install all karaoke-gen Python dependencies (the slow part)
|
|
# BuildKit cache mount preserves downloaded wheels across builds
|
|
RUN --mount=type=cache,target=/root/.cache/pip \
|
|
pip install --upgrade pip && \
|
|
pip install -e .
|
|
|
|
# Install audio-separator with GPU support (PyTorch + CUDA + onnxruntime-gpu)
|
|
# This is separate from pip install -e . to keep layer caching granular
|
|
#
|
|
# IMPORTANT: Install PyTorch cu126 FIRST, then audio-separator[gpu].
|
|
# The default audio-separator[gpu] pulls PyTorch cu130, but Cloud Run L4 GPUs
|
|
# have NVIDIA driver 570 which only supports up to CUDA 12.8. PyTorch cu130
|
|
# fails with "NVIDIA driver is too old". PyTorch cu126 works with 12.8 drivers.
|
|
# Installing torch first means audio-separator[gpu] sees it as already satisfied.
|
|
#
|
|
# PINNED to an exact version: an unpinned install silently picked up 0.47.0 on the
|
|
# 2026-10-17 rebuild, and on L4 its mel-band RoFormer karaoke models run 3-5x slower
|
|
# than 0.44.2 (gabox_v2 ~26s -> 149s, becruily ~27s -> 157s for a ~3-min track;
|
|
# YAML overlap/batch are unchanged, so it is in the 0.45-0.47 inference changes).
|
|
# Total separation went from ~5 to ~10-16 min and the daily E2E started failing.
|
|
# Only bump this after timing a real job on the new version (see BACKLOG).
|
|
ARG AUDIO_SEPARATOR_VERSION=0.44.2
|
|
RUN --mount=type=cache,target=/root/.cache/pip \
|
|
pip install torch==2.6.0+cu126 torchvision==0.21.0+cu126 --index-url https://download.pytorch.org/whl/cu126 && \
|
|
pip install "audio-separator[gpu]==${AUDIO_SEPARATOR_VERSION}"
|
|
|
|
# Now copy actual source code (AFTER deps installed, so code changes don't
|
|
# invalidate the expensive pip install layer in registry cache)
|
|
COPY karaoke_gen /app/karaoke_gen
|
|
COPY lyrics_transcriber_temp /app/lyrics_transcriber_temp
|
|
COPY backend /app/backend
|
|
|
|
# Download spaCy language model for lyrics transcription
|
|
# Must be done at build time, not runtime (Cloud Run network restrictions)
|
|
RUN python -m spacy download en_core_web_sm
|
|
|
|
# Create models directory and pre-download ensemble preset models
|
|
# Baking models into the image eliminates cold-start model download latency.
|
|
# Total model size: ~1-1.5 GB for instrumental_clean + karaoke presets.
|
|
RUN mkdir -p /models
|
|
COPY backend/scripts/download_models.py /tmp/download_models.py
|
|
RUN python /tmp/download_models.py /models && \
|
|
rm /tmp/download_models.py && \
|
|
ls -lh /models/
|
|
|
|
# Label with build info
|
|
ARG BUILD_DATE
|
|
ARG BUILD_ID
|
|
ARG BASE_CONTENT_HASH
|
|
LABEL org.opencontainers.image.created="${BUILD_DATE}"
|
|
LABEL org.opencontainers.image.revision="${BUILD_ID}"
|
|
LABEL org.opencontainers.image.title="karaoke-backend-gpu-base"
|
|
LABEL org.opencontainers.image.description="GPU base image with CUDA, PyTorch, audio-separator, and pre-downloaded models for karaoke-backend audio worker"
|
|
LABEL base.content.hash="${BASE_CONTENT_HASH}"
|