## Summary
- Refactor monolithic `infrastructure/__main__.py` from **2,602 lines to
339 lines** (87% reduction)
- Split resources into organized, purpose-specific modules following
SOLID principles
- Extract VM startup scripts to separate `.sh` files for easier
maintenance
- Extract embedded Python encoding worker app to proper package
## Changes
### New Module Structure
```
infrastructure/
├── __main__.py # Entry point (339 lines, was 2,602)
├── config.py # Centralized configuration
├── modules/ # Core GCP resources
│ ├── database.py # Firestore + indexes
│ ├── storage.py # GCS bucket + lifecycle
│ ├── artifact_registry.py # Docker repository
│ ├── secrets.py # Secret Manager secrets
│ ├── cloud_tasks.py # Task queues
│ ├── cloud_run.py # Domain mapping, Cloud Run Jobs
│ ├── monitoring.py # Alert policies
│ └── iam/ # Service accounts by purpose
├── compute/ # VM-based resources
│ ├── flacfetch_vm.py
│ ├── encoding_worker_vm.py
│ ├── github_runners.py
│ └── startup_scripts/ # External .sh files
└── docs/
└── PHASE5-PACKER-IMAGE-PLAN.md # Future optimization plan
```
### Extracted Code
- **Startup scripts**: Moved from embedded strings to separate `.sh`
files
- **GCE encoding service**: Extracted ~600-line Python app to
`backend/services/gce_encoding/`
### Documentation
- Updated `infrastructure/README.md` with new structure
- Added `docs/PHASE5-PACKER-IMAGE-PLAN.md` for future Packer image
optimization
## Verification
```bash
pulumi preview --diff
# Result: 120 resources unchanged
# Only 10 GitHub runner replacements (expected - startup script improvements from main)
```
## Test Plan
- [x] `pulumi preview` shows no unintended infrastructure changes
- [x] All module imports resolve correctly
- [x] Startup scripts produce identical output when read from files
## Notes
- Phase 5 (Packer custom images) documented but not implemented -
reduces encoding worker startup from ~10min to ~30sec
- `deploy.sh` for gdrive_validator kept - handles source upload to GCS
which Pulumi doesn't do automatically
@coderabbitai ignore
---
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
9.9 KiB
Phase 5: Encoding Worker Custom Machine Image
Problem Statement
The encoding worker VM currently takes ~10 minutes to become ready after startup because the startup script:
- Builds Python 3.13 from source (
./configure && make -j$(nproc)) - ~7 minutes - Downloads and installs FFmpeg static build - ~1 minute
- Installs system packages and fonts - ~1 minute
- Creates Python venv and installs dependencies - ~1 minute
This slow startup affects:
- VM restarts: Any maintenance event or restart requires waiting 10 minutes
- Scaling: Can't quickly add capacity if needed
- Development: Testing infrastructure changes is slow
Solution: Pre-baked Machine Image with Packer
Create a custom GCE machine image using Packer that pre-installs all dependencies. The startup script then only needs to:
- Fetch API key from Secret Manager (~5 seconds)
- Download latest karaoke-gen wheel from GCS (~10 seconds)
- Start the systemd service (~5 seconds)
Expected startup time: ~30 seconds (vs current ~10 minutes)
Implementation Plan
1. Create Packer Directory Structure
infrastructure/packer/
├── encoding-worker.pkr.hcl # Packer template
├── scripts/
│ └── provision.sh # Provisioning script
└── README.md # Usage documentation
2. Packer Template (encoding-worker.pkr.hcl)
packer {
required_plugins {
googlecompute = {
source = "github.com/hashicorp/googlecompute"
version = "~> 1.1"
}
}
}
variable "project_id" {
type = string
default = "nomadkaraoke"
}
variable "zone" {
type = string
default = "us-central1-a"
}
variable "python_version" {
type = string
default = "3.13.1"
}
source "googlecompute" "encoding-worker" {
project_id = var.project_id
zone = var.zone
source_image_family = "debian-12"
source_image_project_id = "debian-cloud"
# Build on same machine type as production for consistency
machine_type = "c4-standard-8"
disk_size = 100
disk_type = "pd-ssd" # Use SSD for faster builds
image_name = "encoding-worker-{{timestamp}}"
image_family = "encoding-worker"
image_description = "Encoding worker with Python ${var.python_version}, FFmpeg, and fonts pre-installed"
ssh_username = "packer"
# Tags for firewall rules during build
tags = ["packer-build"]
}
build {
sources = ["source.googlecompute.encoding-worker"]
# Upload provisioning script
provisioner "file" {
source = "scripts/provision.sh"
destination = "/tmp/provision.sh"
}
# Run provisioning
provisioner "shell" {
inline = [
"chmod +x /tmp/provision.sh",
"sudo PYTHON_VERSION=${var.python_version} /tmp/provision.sh"
]
}
# Clean up for smaller image
provisioner "shell" {
inline = [
"sudo apt-get clean",
"sudo rm -rf /var/lib/apt/lists/*",
"sudo rm -rf /tmp/*",
"sudo rm -rf /var/tmp/*"
]
}
}
3. Provisioning Script (scripts/provision.sh)
#!/bin/bash
# Provision encoding worker image with all dependencies pre-installed
set -e
echo "=== Starting encoding worker image provisioning ==="
# Install system packages
apt-get update
apt-get install -y \
docker.io \
curl \
git \
xz-utils \
build-essential \
libssl-dev \
zlib1g-dev \
libbz2-dev \
libreadline-dev \
libsqlite3-dev \
libncursesw5-dev \
tk-dev \
libxml2-dev \
libxmlsec1-dev \
libffi-dev \
liblzma-dev
# Install fonts for ASS subtitle rendering
apt-get install -y fonts-noto fonts-noto-cjk fontconfig
fc-cache -fv
# Enable Docker
systemctl enable docker
# Install optimized static FFmpeg build (John Van Sickle)
echo "Installing FFmpeg..."
curl -L https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz -o /tmp/ffmpeg.tar.xz
tar -xf /tmp/ffmpeg.tar.xz -C /tmp
cp /tmp/ffmpeg-*-amd64-static/ffmpeg /usr/local/bin/
cp /tmp/ffmpeg-*-amd64-static/ffprobe /usr/local/bin/
chmod +x /usr/local/bin/ffmpeg /usr/local/bin/ffprobe
rm -rf /tmp/ffmpeg*
ffmpeg -version
# Build and install Python from source
echo "Building Python ${PYTHON_VERSION}..."
cd /tmp
curl -L "https://www.python.org/ftp/python/${PYTHON_VERSION}/Python-${PYTHON_VERSION}.tar.xz" -o python.tar.xz
tar -xf python.tar.xz
cd "Python-${PYTHON_VERSION}"
./configure --prefix=/opt/python313 --enable-optimizations --with-lto
make -j$(nproc)
make install
rm -rf /tmp/python.tar.xz /tmp/Python-${PYTHON_VERSION}
/opt/python313/bin/python3.13 --version
echo "Python ${PYTHON_VERSION} installed at /opt/python313"
# Create working directory and venv
mkdir -p /opt/encoding-worker
cd /opt/encoding-worker
/opt/python313/bin/python3.13 -m venv venv
# Install base Python dependencies (wheel will add more at runtime)
source venv/bin/activate
pip install --upgrade pip
pip install fastapi uvicorn google-cloud-storage aiofiles aiohttp
# Create systemd service template (API key filled at runtime)
cat > /etc/systemd/system/encoding-worker.service << 'SYSTEMD'
[Unit]
Description=Encoding Worker HTTP API Service
After=network.target
[Service]
Type=simple
User=root
WorkingDirectory=/opt/encoding-worker
ExecStartPre=/opt/encoding-worker/startup.sh
ExecStart=/opt/encoding-worker/venv/bin/uvicorn backend.services.gce_encoding.main:app --host 0.0.0.0 --port 8080
Restart=always
RestartSec=10
Environment="GOOGLE_CLOUD_PROJECT=nomadkaraoke"
EnvironmentFile=/opt/encoding-worker/env
[Install]
WantedBy=multi-user.target
SYSTEMD
# Create startup helper script (runs before service starts)
cat > /opt/encoding-worker/startup.sh << 'STARTUP'
#!/bin/bash
set -e
# Fetch API key from Secret Manager
ENCODING_API_KEY=$(gcloud secrets versions access latest --secret=encoding-worker-api-key 2>/dev/null || echo "")
echo "ENCODING_API_KEY=${ENCODING_API_KEY}" > /opt/encoding-worker/env
# Download and install latest karaoke-gen wheel
gsutil cp "gs://karaoke-gen-storage-nomadkaraoke/wheels/karaoke_gen-*.whl" /tmp/ 2>/dev/null || true
WHEEL_FILE=$(ls -t /tmp/karaoke_gen-*.whl 2>/dev/null | head -1)
if [ -n "$WHEEL_FILE" ]; then
/opt/encoding-worker/venv/bin/pip install --upgrade --quiet "$WHEEL_FILE"
fi
STARTUP
chmod +x /opt/encoding-worker/startup.sh
# Enable service (will start on boot)
systemctl daemon-reload
systemctl enable encoding-worker
echo "=== Provisioning complete ==="
4. Update Pulumi to Use Custom Image
In infrastructure/compute/encoding_worker_vm.py, change:
# Before (current)
boot_disk=compute.InstanceBootDiskArgs(
initialize_params=compute.InstanceBootDiskInitializeParamsArgs(
image="debian-cloud/debian-12",
size=DiskSizes.ENCODING_WORKER,
type="hyperdisk-balanced",
),
),
# After (with custom image)
boot_disk=compute.InstanceBootDiskArgs(
initialize_params=compute.InstanceBootDiskInitializeParamsArgs(
image=f"projects/{PROJECT_ID}/global/images/family/encoding-worker",
size=DiskSizes.ENCODING_WORKER,
type="hyperdisk-balanced",
),
),
5. Simplified Startup Script
Replace the 700-line encoding_worker.sh with a minimal script:
#!/bin/bash
# Encoding worker startup script (for custom image)
# Most dependencies are pre-installed in the image.
# This script just starts the service.
set -e
exec > >(tee /var/log/encoding-worker-startup.log) 2>&1
echo "Starting encoding worker at $(date)"
# Start Docker (in case it's not running)
systemctl start docker
# Start the encoding worker service
# The service's ExecStartPre handles:
# - Fetching API key from Secret Manager
# - Installing latest karaoke-gen wheel
systemctl start encoding-worker
echo "Encoding worker started at $(date)"
6. CI Integration (Optional)
Add a GitHub Actions workflow to rebuild the image when Packer files change:
# .github/workflows/build-encoding-worker-image.yml
name: Build Encoding Worker Image
on:
push:
branches: [main]
paths:
- 'infrastructure/packer/**'
workflow_dispatch: # Manual trigger
jobs:
build:
runs-on: ubuntu-latest
permissions:
contents: read
id-token: write
steps:
- uses: actions/checkout@v4
- name: Authenticate to Google Cloud
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ secrets.WIF_PROVIDER }}
service_account: github-actions-deployer@nomadkaraoke.iam.gserviceaccount.com
- name: Setup Packer
uses: hashicorp/setup-packer@main
with:
version: latest
- name: Build Image
working-directory: infrastructure/packer
run: |
packer init encoding-worker.pkr.hcl
packer build encoding-worker.pkr.hcl
Execution Steps
-
Create the Packer files in
infrastructure/packer/ -
Build the image locally (one-time, or via CI):
cd infrastructure/packer packer init encoding-worker.pkr.hcl packer build encoding-worker.pkr.hcl -
Update Pulumi to use the new image family
-
Simplify the startup script to just start the service
-
Run
pulumi upto deploy with the new image -
Verify the VM starts in ~30 seconds instead of ~10 minutes
Rollback Plan
If issues occur:
- Change the image back to
debian-cloud/debian-12in Pulumi - Restore the full startup script
- Run
pulumi up
The old startup script is preserved in git history and can be restored.
Benefits
| Metric | Before | After |
|---|---|---|
| Startup time | ~10 min | ~30 sec |
| Python build | Every start | Never (pre-baked) |
| FFmpeg install | Every start | Never (pre-baked) |
| Image rebuild needed | N/A | Only when deps change |
Maintenance
Rebuild the image when:
- Python version needs updating
- FFmpeg version needs updating
- New system packages required
- Font packages change
The karaoke-gen wheel is still downloaded at runtime, so application code updates don't require image rebuilds.