feat: Upgrade GCE encoding worker to Python 3.13 (#221)

## Summary

Upgrades the GCE encoding worker VM from Debian's default Python 3.11 to
Python 3.13 built from source, aligning it with CI and Cloud Run Python
version.

## Changes

- Build Python 3.13.1 from source with `--enable-optimizations
--with-lto` for best performance
- Install to `/opt/python313` on the Debian 12 VM
- Create virtual environment using Python 3.13 instead of system Python
3.11
- Add build dependencies (build-essential, libssl-dev, zlib1g-dev, etc.)
- Remove `python3-pip` and `python3-venv` apt packages (no longer
needed)
- Update docs/README.md with recent changes entry

## Testing

- [x] Pulumi preview shows only encoding-worker VM replacement
(expected)
- [x] Backend tests pass (`make test` - 1006 passed)
- [x] Frontend tests pass (`npm run test:all` - 65 passed)
- [x] Python and bash syntax validation pass

## Review

- [x] Local CodeRabbit review completed
- [x] No actionable issues found

## Notes

- The VM will be **replaced** (not updated in-place) when this deploys
- Python build from source takes ~5-10 minutes during VM startup
- Brief encoding downtime expected during deployment

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
Andrew Beveridge ☄️andClaude authored and GitHub committed 2026-01-06 10:43:47 -05:00
1 parent bd26d742a2
commit 07cde54b3f
2 files changed
+23 -4

No files matched your search

+2
View File
@@ -36,6 +36,8 @@
## Recent Changes
- **GCE Encoding Worker Python 3.13** (2026-01-06): Upgraded GCE encoding worker from Debian's Python 3.11 to Python 3.13 built from source. Uses dedicated virtual environment at `/opt/encoding-worker/venv`, removing the need for `--break-system-packages`. Aligns the encoding worker with CI/Cloud Run Python version.
- **GCE Encoding Unified with LocalEncodingService** (2026-01-06): GCE encoding worker now uses the same `LocalEncodingService` as the local CLI, eliminating duplicated encoding logic. Output files now have proper names (`Artist - Title (Final Karaoke Lossless 4k).mp4` instead of `output_4k_lossless.mp4`) and include title/end screen concatenation. Wheel deployed to GCS; worker installs at job start for hot updates without VM restart. Removed all fallback logic - single code path for consistent output across CLI, Cloud Run, and GCE. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#unify-encoding-logic-with-gcs-wheel-deployment).
- **Cloud Distribution Matches Local CLI** (2026-01-06): Fixed cloud distribution (Dropbox uploads) to match local CLI output structure. The `lyrics/` folder now includes: proper filenames (`(Lyrics Corrections).json` instead of `(Karaoke).json`), all reference lyrics files (`(Lyrics Genius).txt`, `(Lyrics Spotify).txt`, etc.), `(With Vocals).mkv`, and `previews/` subfolder with preview ASS files. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#cloud-distribution-must-match-local-cli-output-structure).
+21 -4
View File
@@ -1209,9 +1209,12 @@ set -e
exec > >(tee /var/log/encoding-worker-startup.log) 2>&1
echo "Starting encoding worker setup at $(date)"
# Install dependencies
# Install dependencies (including Python build dependencies)
apt-get update
apt-get install -y python3-pip python3-venv docker.io curl git xz-utils
apt-get install -y docker.io curl git xz-utils \\
build-essential libssl-dev zlib1g-dev libbz2-dev \\
libreadline-dev libsqlite3-dev libncursesw5-dev \\
tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev
# Install fonts for ASS subtitle rendering (libass uses fontconfig)
# - fonts-noto: Noto Sans (default karaoke font)
@@ -1234,12 +1237,26 @@ chmod +x /usr/local/bin/ffmpeg /usr/local/bin/ffprobe
rm -rf /tmp/ffmpeg*
ffmpeg -version
# Build and install Python 3.13 from source
echo "Building Python 3.13 from source..."
PYTHON_VERSION="3.13.1"
cd /tmp
curl -L "https://www.python.org/ftp/python/${PYTHON_VERSION}/Python-${PYTHON_VERSION}.tar.xz" -o python.tar.xz
tar -xf python.tar.xz
cd "Python-${PYTHON_VERSION}"
./configure --prefix=/opt/python313 --enable-optimizations --with-lto
make -j$(nproc)
make install
rm -rf /tmp/python.tar.xz /tmp/Python-${PYTHON_VERSION}
/opt/python313/bin/python3.13 --version
echo "Python 3.13 installed at /opt/python313"
# Create working directory
mkdir -p /opt/encoding-worker
cd /opt/encoding-worker
# Create Python virtual environment
python3 -m venv venv
# Create Python virtual environment using Python 3.13
/opt/python313/bin/python3.13 -m venv venv
source venv/bin/activate
# Install Python dependencies