mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
@coderabbitai ignore ## Why BACKLOG "audio-separator 0.45–0.47 regression: mel-band RoFormer karaoke models 3–5× slower on L4". The root cause is **python-audio-separator #299 (0.47.0)**, not #298. #299 made RoFormer `overlap` a *window count* that defaults to each model YAML's `num_overlap`. Before that, `overlap` was effectively an 8-second step, which meant almost no overlap. The karaoke models' YAMLs set `num_overlap` 4/8, so a 3-minute track went from 23 forward passes to 87 (aufr33) and 124 (gabox_v2, becruily). That matches the measured 25→75s, 26→149s and 27→157s. 0.45.0 and 0.46.0 still had the old step formula. python-audio-separator #313 (released as **0.47.1**) keeps #299. It adds a per-model forward-pass budget test on every PR and a GPU per-pass timing test (T4: 1.21 s/pass), plus a README note. Andrew chose overlap=2 for gen: smoother chunk seams than 0.44's no-overlap, at about 1.5× the 0.44 compute instead of 4–6×. ## What - `karaoke_gen/audio_processor.py`: `SEPARATOR_MDXC_PARAMS` (overlap=2) goes to all four `Separator(...)` constructions (stage 1, stage 2 karaoke ensemble, single-model paths) **only when audio-separator ≥ 0.47**. On older versions `overlap=2` meant a 2-second step, which is 4× slower, so they keep their defaults. Ensembles load each model through `load_model`, so the param reaches every model in the preset. - `backend/Dockerfile.gpu-base`: `AUDIO_SEPARATOR_VERSION=0.47.1`, with the comment rewritten to explain the root cause. 0.47.1 needs `torch>=2.3` on Linux, so the pre-installed `torch==2.6.0+cu126` (required by the L4 driver) is kept. - The poetry lock stays on 0.44.1. The version-aware params keep CI and CPU images on the old semantics, and there's no torch churn. ## Expected cost per ~3-min track (forward passes) | model | 0.44.2 | 0.47 YAML default | 0.47 + overlap=2 | |---|---|---|---| | karaoke_aufr33 | 23 | 87 | 44 | | karaoke_gabox_v2 | 23 | 124 | 32 | | karaoke_becruily | 23 | 124 | 32 | | instrumental fv7z / resurrection | 23 | 32 / 21 | 32 / 21 | ## Verification plan (before /backlog-finish) After the deploy, time the separation of the daily E2E or a real job on the audio-separation-job logs. I expect roughly 1.5× the 0.44.2 karaoke-stage times (~40–50s per karaoke model rather than 75–157s). If it's worse, revert the pin to 0.44.2. That's one line. ## Testing - `tests/unit/test_audio_processor_local_gpu.py`: version gating (0.47.x/0.48/1.0 → overlap 2; 0.44/0.46 → defaults; unknown → defaults; fresh copy), and both stages' constructors receive `mdxc_params.overlap == 2`. - `make test`: the backend unit suite passes. The only failures are 8 `test_emulator_integration.py` tests, caused by my local environment: another session's emulator setup is on :8080/:4443, and they fail identically on untouched main. CI is green on main. - Local CodeRabbit: 0 findings. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Jw21syo4HizJJjYw2mg2Vp --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
145 lines
6.5 KiB
Docker
145 lines
6.5 KiB
Docker
# GPU base image with CUDA, PyTorch, audio-separator, and pre-downloaded models
|
|
# This image is rebuilt only when deps or models change (infrequently).
|
|
#
|
|
# Build with: gcloud builds submit --config=backend/cloudbuild-gpu-base.yaml --project=nomadkaraoke --timeout=3600s
|
|
# Or manually: docker build -f backend/Dockerfile.gpu-base -t karaoke-backend-gpu-base .
|
|
#
|
|
# Layer caching strategy:
|
|
# 1. System deps + fonts + ffmpeg (rarely change, always cached)
|
|
# 2. CUDA library path configuration
|
|
# 3. pyproject.toml + package stubs (changes with dep updates)
|
|
# 4. pip install karaoke-gen deps + audio-separator[gpu] (slow, cached unless pyproject.toml changes)
|
|
# 5. Source code copy (changes every rebuild, but AFTER deps are installed)
|
|
# 6. spaCy model download (cached unless pip install layer changes)
|
|
# 7. Pre-downloaded ensemble preset models (~1-1.5 GB, cached unless pip layer changes)
|
|
|
|
FROM nvidia/cuda:12.6.3-runtime-ubuntu22.04
|
|
|
|
# Prevent interactive prompts during package installation
|
|
ENV DEBIAN_FRONTEND=noninteractive
|
|
|
|
# Install Python 3.11 from deadsnakes PPA and system dependencies
|
|
RUN apt-get update && apt-get install -y --no-install-recommends \
|
|
software-properties-common \
|
|
&& add-apt-repository -y ppa:deadsnakes/ppa \
|
|
&& apt-get update && apt-get install -y --no-install-recommends \
|
|
# Python 3.11
|
|
python3.11 \
|
|
python3.11-dev \
|
|
python3.11-venv \
|
|
# Audio libraries
|
|
libsndfile1 \
|
|
libsox-dev \
|
|
sox \
|
|
# Build tools
|
|
build-essential \
|
|
# Utilities
|
|
curl \
|
|
xz-utils \
|
|
&& rm -rf /var/lib/apt/lists/* \
|
|
&& python3.11 --version
|
|
|
|
# Install comprehensive font support for international lyrics and symbols
|
|
# Noto fonts provide full Unicode coverage including:
|
|
# - Latin, Greek, Cyrillic (core)
|
|
# - CJK: Chinese, Japanese, Korean (~150MB)
|
|
# - Arabic, Hebrew, Thai, Hindi, and other scripts (extra)
|
|
# - Color emoji (limited support in libass, but available)
|
|
# - Musical symbols (♪ ♫ etc.)
|
|
# Note: This adds ~200-300MB to the image but ensures all languages render correctly
|
|
RUN apt-get update && apt-get install -y --no-install-recommends \
|
|
fonts-noto-core \
|
|
fonts-noto-cjk \
|
|
fonts-noto-extra \
|
|
fonts-noto-color-emoji \
|
|
fontconfig \
|
|
&& fc-cache -f \
|
|
&& rm -rf /var/lib/apt/lists/*
|
|
|
|
# Install FFmpeg from Ubuntu package
|
|
# The audio worker only uses FFmpeg for format conversion and padding, not
|
|
# encoding-intensive tasks, so the Debian package is sufficient. The static
|
|
# build from johnvansickle.com is used for the encoding worker instead.
|
|
RUN apt-get update && apt-get install -y ffmpeg && rm -rf /var/lib/apt/lists/*
|
|
|
|
# Set Python 3.11 as default and install pip
|
|
RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.11 1 \
|
|
&& update-alternatives --install /usr/bin/python python /usr/bin/python3.11 1 \
|
|
&& curl -sS https://bootstrap.pypa.io/get-pip.py | python3.11 \
|
|
&& python3 -m pip install --no-cache-dir --upgrade pip setuptools wheel
|
|
|
|
# Set up CUDA library paths so GPU-accelerated packages can find CUDA libs at runtime
|
|
RUN echo '/usr/local/cuda/lib64' >> /etc/ld.so.conf.d/cuda.conf && ldconfig
|
|
|
|
# Environment configuration for CUDA and Python
|
|
ENV LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH \
|
|
PATH=/usr/local/cuda/bin:$PATH \
|
|
PYTHONUNBUFFERED=1 \
|
|
MODEL_DIR=/models
|
|
|
|
# Set working directory
|
|
WORKDIR /app
|
|
|
|
# Copy ONLY dependency specification files first (changes less often than source code)
|
|
COPY pyproject.toml README.md LICENSE /app/
|
|
|
|
# Create minimal package stubs so pip can resolve the editable install
|
|
# pip install -e . needs package directories to exist but doesn't validate content
|
|
RUN mkdir -p /app/karaoke_gen /app/lyrics_transcriber_temp /app/backend && \
|
|
touch /app/karaoke_gen/__init__.py /app/lyrics_transcriber_temp/__init__.py /app/backend/__init__.py
|
|
|
|
# Install all karaoke-gen Python dependencies (the slow part)
|
|
# BuildKit cache mount preserves downloaded wheels across builds
|
|
RUN --mount=type=cache,target=/root/.cache/pip \
|
|
pip install --upgrade pip && \
|
|
pip install -e .
|
|
|
|
# Install audio-separator with GPU support (PyTorch + CUDA + onnxruntime-gpu)
|
|
# This is separate from pip install -e . to keep layer caching granular
|
|
#
|
|
# IMPORTANT: Install PyTorch cu126 FIRST, then audio-separator[gpu].
|
|
# The default audio-separator[gpu] pulls PyTorch cu130, but Cloud Run L4 GPUs
|
|
# have NVIDIA driver 570 which only supports up to CUDA 12.8. PyTorch cu130
|
|
# fails with "NVIDIA driver is too old". PyTorch cu126 works with 12.8 drivers.
|
|
# Installing torch first means audio-separator[gpu] sees it as already satisfied.
|
|
#
|
|
# PINNED to an exact version: an unpinned install silently picked up 0.47.0 on the
|
|
# 2026-10-17 rebuild and separation went from ~5 to ~10-16 min. Cause: 0.47 (#299) made
|
|
# RoFormer `overlap` a window count defaulting to each model YAML's num_overlap (4-8 for
|
|
# the karaoke models = 4-6x the forward passes of 0.44's fixed 8 s step). karaoke_gen's
|
|
# audio_processor now passes overlap=2 on >= 0.47 (SEPARATOR_MDXC_PARAMS). Time a real
|
|
# job before bumping this, and read the audio-separator release notes for compute changes.
|
|
ARG AUDIO_SEPARATOR_VERSION=0.47.1
|
|
RUN --mount=type=cache,target=/root/.cache/pip \
|
|
pip install torch==2.6.0+cu126 torchvision==0.21.0+cu126 --index-url https://download.pytorch.org/whl/cu126 && \
|
|
pip install "audio-separator[gpu]==${AUDIO_SEPARATOR_VERSION}"
|
|
|
|
# Now copy actual source code (AFTER deps installed, so code changes don't
|
|
# invalidate the expensive pip install layer in registry cache)
|
|
COPY karaoke_gen /app/karaoke_gen
|
|
COPY lyrics_transcriber_temp /app/lyrics_transcriber_temp
|
|
COPY backend /app/backend
|
|
|
|
# Download spaCy language model for lyrics transcription
|
|
# Must be done at build time, not runtime (Cloud Run network restrictions)
|
|
RUN python -m spacy download en_core_web_sm
|
|
|
|
# Create models directory and pre-download ensemble preset models
|
|
# Baking models into the image eliminates cold-start model download latency.
|
|
# Total model size: ~1-1.5 GB for instrumental_clean + karaoke presets.
|
|
RUN mkdir -p /models
|
|
COPY backend/scripts/download_models.py /tmp/download_models.py
|
|
RUN python /tmp/download_models.py /models && \
|
|
rm /tmp/download_models.py && \
|
|
ls -lh /models/
|
|
|
|
# Label with build info
|
|
ARG BUILD_DATE
|
|
ARG BUILD_ID
|
|
ARG BASE_CONTENT_HASH
|
|
LABEL org.opencontainers.image.created="${BUILD_DATE}"
|
|
LABEL org.opencontainers.image.revision="${BUILD_ID}"
|
|
LABEL org.opencontainers.image.title="karaoke-backend-gpu-base"
|
|
LABEL org.opencontainers.image.description="GPU base image with CUDA, PyTorch, audio-separator, and pre-downloaded models for karaoke-backend audio worker"
|
|
LABEL base.content.hash="${BASE_CONTENT_HASH}"
|