Compare commits

..
Author SHA1 Message Date
Andrew BeveridgeandClaude Opus 4.5 7b5a150e81 docs: add self-hosted runner documentation
- Update README.md with CI/CD runner status
- Add lessons learned for self-hosted runner setup
- Create archive doc with full implementation details

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-28 12:05:27 -05:00
4 changed files with 121 additions and 57 deletions

No files matched your search

+13
View File
@@ -126,6 +126,19 @@ Migrated from Cloud Build to GitHub Actions for:
- Better visibility
- Faster feedback
### Self-Hosted Runner on GCP
GitHub-hosted runners have ~14GB disk, which caused failures during Docker builds. Solution: self-hosted runner on GCP with 200GB SSD.
**Key learnings:**
1. **GPG in startup scripts**: Always use `gpg --batch` for non-interactive execution
2. **setup-python on self-hosted**: Needs Python in tool cache at `$RUNNER/_work/_tool/Python/<version>/x64/` with `x64.complete` marker
3. **Debian vs Ubuntu**: PPAs don't work on Debian; use upstream repos (Temurin, NodeSource)
4. **pyenv for Python**: More reliable than source builds or distro packages for specific versions
See `docs/archive/2025-12-28-self-hosted-github-runner.md` for full details.
## Performance Observations
| Operation | Duration |
+1
View File
@@ -20,6 +20,7 @@
| Multi-format encoding | Working |
| Token-based auth | Working |
| Payment flow | Not started |
| CI/CD self-hosted runner | Working (GCP) |
## Known Issues
@@ -0,0 +1,80 @@
# Self-Hosted GitHub Actions Runner on GCP
**Date**: 2025-12-28
**Status**: Deployed and operational
## Problem
GitHub-hosted runners have ~14GB disk space. CI jobs were failing with "No space left on device" during:
- Docker builds (multi-stage, large dependencies)
- Emulator tests (Docker + Java + Firestore/GCS emulators)
- Package builds with poetry
Workarounds like `jlumbroso/free-disk-space` action helped but weren't reliable.
## Solution
Deployed a self-hosted runner on GCP with 200GB SSD via Pulumi IaC.
### Infrastructure Added
| Resource | Purpose |
|----------|---------|
| `github-runner` VM | e2-standard-4, 200GB SSD, Debian 12 |
| `github-runner` service account | Minimal permissions for runner |
| `github-runner-pat` secret | PAT for runner registration |
| IAM bindings | Artifact Registry reader, Secret Manager accessor, logging |
### Jobs Moved to Self-Hosted
- `package-build-test`
- `backend-unit-tests`
- `backend-emulator-tests`
- `deploy-backend`
These use `runs-on: [self-hosted, linux, gcp]` with labels.
## Issues Encountered
### 1. GPG tty errors in startup script
**Problem**: `gpg --dearmor` failed with "cannot open /dev/tty" in non-interactive startup.
**Fix**: Add `--batch` flag to all GPG commands.
### 2. setup-python can't find Python 3.13 on Debian 12
**Problem**: `actions/setup-python@v5` couldn't find pre-built Python 3.13 for Debian 12 on self-hosted runners.
**Fix**:
1. Install Python 3.13 via pyenv on the runner
2. Copy to tool cache: `_work/_tool/Python/3.13.0/x64/`
3. Create marker file: `x64.complete`
### 3. Java 21 not in Debian repos
**Problem**: `openjdk-21-jdk` not available in Debian 12 stable.
**Fix**: Use Temurin (Adoptium) repository directly.
## Key Files
- `infrastructure/__main__.py` - Runner VM and IAM setup
- `infrastructure/README.md` - Setup instructions
- `.github/workflows/ci.yml` - Jobs using self-hosted runner
## Lessons for Future
1. **Self-hosted runners need explicit tool setup** - Unlike GitHub-hosted, tools aren't pre-installed in expected locations
2. **setup-python tool cache location**: `$RUNNER_DIR/_work/_tool/Python/<version>/x64/`
3. **GPG in scripts**: Always use `--batch` for non-interactive execution
4. **Debian vs Ubuntu**: Many GitHub Actions assume Ubuntu; PPAs don't work on Debian
## Cost
~$50/month for e2-standard-4 running 24/7 (covered by GCP free credits).
## PRs
- #95: Initial self-hosted runner setup
- #97: Make startup script reproducible (pyenv, tool cache, GPG fixes)
+27 -57
View File
@@ -1255,7 +1255,7 @@ apt-get install -y \\
# ==================== Docker ====================
echo "Installing Docker..."
curl -fsSL https://download.docker.com/linux/debian/gpg | gpg --batch --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
curl -fsSL https://download.docker.com/linux/debian/gpg | gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian $(lsb_release -cs) stable" > /etc/apt/sources.list.d/docker.list
apt-get update
apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
@@ -1264,38 +1264,23 @@ apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin do
systemctl enable docker
systemctl start docker
# ==================== Python 3.13 via pyenv ====================
# setup-python action on self-hosted runners needs Python in the tool cache
echo "Installing Python 3.13 via pyenv..."
# Install pyenv dependencies
apt-get install -y make build-essential libssl-dev zlib1g-dev \\
libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm \\
libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev
# Install pyenv system-wide
export PYENV_ROOT=/opt/pyenv
curl https://pyenv.run | bash
# Add to system profile for future sessions
cat > /etc/profile.d/pyenv.sh << 'PYENV_PROFILE'
export PYENV_ROOT=/opt/pyenv
export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"
PYENV_PROFILE
# Source pyenv for current session
export PATH="/opt/pyenv/bin:$PATH"
eval "$(/opt/pyenv/bin/pyenv init -)"
# Install Python 3.13
/opt/pyenv/bin/pyenv install 3.13.0
/opt/pyenv/bin/pyenv global 3.13.0
# Create symlinks for system-wide access
ln -sf /opt/pyenv/shims/python3.13 /usr/local/bin/python3.13
ln -sf /opt/pyenv/shims/python3 /usr/local/bin/python3
ln -sf /opt/pyenv/shims/pip3 /usr/local/bin/pip3
# ==================== Python 3.13 ====================
echo "Installing Python 3.13..."
add-apt-repository -y ppa:deadsnakes/ppa || true
apt-get update
apt-get install -y python3.13 python3.13-venv python3.13-dev python3-pip || {
# Fallback: build from source if PPA not available
echo "PPA not available, installing Python 3.13 from source..."
apt-get install -y build-essential zlib1g-dev libncurses5-dev libgdbm-dev libnss3-dev libreadline-dev libffi-dev libsqlite3-dev wget libbz2-dev
cd /tmp
wget https://www.python.org/ftp/python/3.13.0/Python-3.13.0.tgz
tar -xf Python-3.13.0.tgz
cd Python-3.13.0
./configure --enable-optimizations
make -j $(nproc)
make altinstall
ln -sf /usr/local/bin/python3.13 /usr/bin/python3.13
}
# ==================== Node.js 20 ====================
echo "Installing Node.js 20..."
@@ -1303,12 +1288,14 @@ curl -fsSL https://deb.nodesource.com/setup_20.x | bash -
apt-get install -y nodejs
# ==================== Java 21 (for Firestore emulator) ====================
echo "Installing Java 21 via Temurin..."
# Use Temurin (Adoptium) - more reliable than Debian repos
curl -fsSL https://packages.adoptium.net/artifactory/api/gpg/key/public | gpg --batch --dearmor -o /usr/share/keyrings/adoptium.gpg
echo "deb [signed-by=/usr/share/keyrings/adoptium.gpg] https://packages.adoptium.net/artifactory/deb $(lsb_release -cs) main" > /etc/apt/sources.list.d/adoptium.list
apt-get update
apt-get install -y temurin-21-jdk
echo "Installing Java 21..."
apt-get install -y openjdk-21-jdk || {
# Fallback: use Temurin
curl -fsSL https://packages.adoptium.net/artifactory/api/gpg/key/public | gpg --dearmor -o /usr/share/keyrings/adoptium.gpg
echo "deb [signed-by=/usr/share/keyrings/adoptium.gpg] https://packages.adoptium.net/artifactory/deb $(lsb_release -cs) main" > /etc/apt/sources.list.d/adoptium.list
apt-get update
apt-get install -y temurin-21-jdk
}
# ==================== FFmpeg ====================
echo "Installing FFmpeg..."
@@ -1321,7 +1308,7 @@ ln -sf /root/.local/bin/poetry /usr/local/bin/poetry
# ==================== Google Cloud SDK ====================
echo "Installing Google Cloud SDK..."
curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | gpg --batch --dearmor -o /usr/share/keyrings/cloud.google.gpg
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | gpg --dearmor -o /usr/share/keyrings/cloud.google.gpg
echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" > /etc/apt/sources.list.d/google-cloud-sdk.list
apt-get update
apt-get install -y google-cloud-cli google-cloud-cli-firestore-emulator
@@ -1388,23 +1375,6 @@ echo "Installing runner service..."
echo "GitHub Actions runner setup complete at $(date)"
echo "Runner registered with labels: self-hosted,linux,x64,gcp,large-disk"
# ==================== Setup Python in tool cache ====================
# setup-python action looks for Python in RUNNER_TOOL_CACHE/_tool/Python
echo "Setting up Python 3.13 in tool cache for setup-python action..."
TOOL_CACHE="/home/runner/actions-runner/_work/_tool"
mkdir -p $TOOL_CACHE/Python/3.13.0/x64
# Copy pyenv Python to tool cache
cp -r /opt/pyenv/versions/3.13.0/* $TOOL_CACHE/Python/3.13.0/x64/
# Create marker file that setup-python looks for
touch $TOOL_CACHE/Python/3.13.0/x64.complete
# Fix permissions
chown -R runner:runner $TOOL_CACHE
echo "Python 3.13 added to tool cache"
# ==================== Docker cleanup cron ====================
# Prevent disk from filling up with old Docker images
echo "Setting up Docker cleanup cron..."