Files
karaoke-gen/backend/Dockerfile.gpu-base
T
Andrew Beveridge ☄️andClaude Opus 4.6 c4ea0e8fb4 fix: GPU audio separation — pin PyTorch cu126 + missing load_model() (#613)
## Summary
- Pin PyTorch to `cu126` in GPU Dockerfile — Cloud Run L4 GPUs have
driver 570 (CUDA 12.8 max), but `audio-separator[gpu]` pulls PyTorch
`cu130` which fails with "driver too old"
- Add missing `separator.load_model()` before `separator.separate()` for
ensemble presets in `audio_processor.py` — the Separator API requires
this even for presets
- Remove `AUDIO_SEPARATOR_API_URL` from Service env vars (cleanup — Job
has its own `MODEL_DIR`)

## Validation
Tested on GCE L4 GPU VM:
- `torch==2.6.0+cu126` + `audio-separator[gpu]` 0.44.1 → CUDA detected,
GPU acceleration enabled
- Full ensemble separation (`instrumental_clean` preset) completed in
**26.6s** on real audio
- Docker image build validation in progress on separate L4 VM

## Test plan
- [ ] CI passes (GPU image rebuilds due to Dockerfile change)
- [ ] Trigger audio separation job after deploy
- [ ] Verify Cloud Run Job logs show "CUDA is available" and "local GPU"
mode
- [ ] Verify stems are correct

@coderabbitai ignore

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-26 23:18:32 +00:00

137 lines
6.0 KiB
Docker

# GPU base image with CUDA, PyTorch, audio-separator, and pre-downloaded models
# This image is rebuilt only when deps or models change (infrequently).
#
# Build with: gcloud builds submit --config=backend/cloudbuild-gpu-base.yaml --project=nomadkaraoke --timeout=3600s
# Or manually: docker build -f backend/Dockerfile.gpu-base -t karaoke-backend-gpu-base .
#
# Layer caching strategy:
# 1. System deps + fonts + ffmpeg (rarely change, always cached)
# 2. CUDA library path configuration
# 3. pyproject.toml + package stubs (changes with dep updates)
# 4. pip install karaoke-gen deps + audio-separator[gpu] (slow, cached unless pyproject.toml changes)
# 5. Source code copy (changes every rebuild, but AFTER deps are installed)
# 6. spaCy model download (cached unless pip install layer changes)
# 7. Pre-downloaded ensemble preset models (~1-1.5 GB, cached unless pip layer changes)
FROM nvidia/cuda:12.6.3-runtime-ubuntu22.04
# Prevent interactive prompts during package installation
ENV DEBIAN_FRONTEND=noninteractive
# Install Python 3.11 from deadsnakes PPA and system dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
software-properties-common \
&& add-apt-repository -y ppa:deadsnakes/ppa \
&& apt-get update && apt-get install -y --no-install-recommends \
# Python 3.11
python3.11 \
python3.11-dev \
python3.11-venv \
# Audio libraries
libsndfile1 \
libsox-dev \
sox \
# Build tools
build-essential \
# Utilities
curl \
xz-utils \
&& rm -rf /var/lib/apt/lists/* \
&& python3.11 --version
# Install comprehensive font support for international lyrics and symbols
# Noto fonts provide full Unicode coverage including:
# - Latin, Greek, Cyrillic (core)
# - CJK: Chinese, Japanese, Korean (~150MB)
# - Arabic, Hebrew, Thai, Hindi, and other scripts (extra)
# - Color emoji (limited support in libass, but available)
# - Musical symbols (♪ ♫ etc.)
# Note: This adds ~200-300MB to the image but ensures all languages render correctly
RUN apt-get update && apt-get install -y --no-install-recommends \
fonts-noto-core \
fonts-noto-cjk \
fonts-noto-extra \
fonts-noto-color-emoji \
fontconfig \
&& fc-cache -f \
&& rm -rf /var/lib/apt/lists/*
# Install FFmpeg from Ubuntu package
# The audio worker only uses FFmpeg for format conversion and padding, not
# encoding-intensive tasks, so the Debian package is sufficient. The static
# build from johnvansickle.com is used for the encoding worker instead.
RUN apt-get update && apt-get install -y ffmpeg && rm -rf /var/lib/apt/lists/*
# Set Python 3.11 as default and install pip
RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.11 1 \
&& update-alternatives --install /usr/bin/python python /usr/bin/python3.11 1 \
&& curl -sS https://bootstrap.pypa.io/get-pip.py | python3.11 \
&& python3 -m pip install --no-cache-dir --upgrade pip setuptools wheel
# Set up CUDA library paths so GPU-accelerated packages can find CUDA libs at runtime
RUN echo '/usr/local/cuda/lib64' >> /etc/ld.so.conf.d/cuda.conf && ldconfig
# Environment configuration for CUDA and Python
ENV LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH \
PATH=/usr/local/cuda/bin:$PATH \
PYTHONUNBUFFERED=1 \
MODEL_DIR=/models
# Set working directory
WORKDIR /app
# Copy ONLY dependency specification files first (changes less often than source code)
COPY pyproject.toml README.md LICENSE /app/
# Create minimal package stubs so pip can resolve the editable install
# pip install -e . needs package directories to exist but doesn't validate content
RUN mkdir -p /app/karaoke_gen /app/lyrics_transcriber_temp /app/backend && \
touch /app/karaoke_gen/__init__.py /app/lyrics_transcriber_temp/__init__.py /app/backend/__init__.py
# Install all karaoke-gen Python dependencies (the slow part)
# BuildKit cache mount preserves downloaded wheels across builds
RUN --mount=type=cache,target=/root/.cache/pip \
pip install --upgrade pip && \
pip install -e .
# Install audio-separator with GPU support (PyTorch + CUDA + onnxruntime-gpu)
# This is separate from pip install -e . to keep layer caching granular
#
# IMPORTANT: Install PyTorch cu126 FIRST, then audio-separator[gpu].
# The default audio-separator[gpu] pulls PyTorch cu130, but Cloud Run L4 GPUs
# have NVIDIA driver 570 which only supports up to CUDA 12.8. PyTorch cu130
# fails with "NVIDIA driver is too old". PyTorch cu126 works with 12.8 drivers.
# Installing torch first means audio-separator[gpu] sees it as already satisfied.
RUN --mount=type=cache,target=/root/.cache/pip \
pip install torch==2.6.0+cu126 torchvision==0.21.0+cu126 --index-url https://download.pytorch.org/whl/cu126 && \
pip install "audio-separator[gpu]"
# Now copy actual source code (AFTER deps installed, so code changes don't
# invalidate the expensive pip install layer in registry cache)
COPY karaoke_gen /app/karaoke_gen
COPY lyrics_transcriber_temp /app/lyrics_transcriber_temp
COPY backend /app/backend
# Download spaCy language model for lyrics transcription
# Must be done at build time, not runtime (Cloud Run network restrictions)
RUN python -m spacy download en_core_web_sm
# Create models directory and pre-download ensemble preset models
# Baking models into the image eliminates cold-start model download latency.
# Total model size: ~1-1.5 GB for instrumental_clean + karaoke presets.
RUN mkdir -p /models
COPY backend/scripts/download_models.py /tmp/download_models.py
RUN python /tmp/download_models.py /models && \
rm /tmp/download_models.py && \
ls -lh /models/
# Label with build info
ARG BUILD_DATE
ARG BUILD_ID
ARG BASE_CONTENT_HASH
LABEL org.opencontainers.image.created="${BUILD_DATE}"
LABEL org.opencontainers.image.revision="${BUILD_ID}"
LABEL org.opencontainers.image.title="karaoke-backend-gpu-base"
LABEL org.opencontainers.image.description="GPU base image with CUDA, PyTorch, audio-separator, and pre-downloaded models for karaoke-backend audio worker"
LABEL base.content.hash="${BASE_CONTENT_HASH}"