From f582236d18611ea2c7fa9ff64e4f4d2bd0713aef Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Andrew=20Beveridge=20=E2=98=84=EF=B8=8F?= Date: Thu, 19 Mar 2026 15:29:58 -0400 Subject: [PATCH] fix: prevent flaky "docker not found" in CI emulator tests (#543) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Summary - Fix race condition where GHA runner systemd service auto-starts on VM boot and accepts CI jobs before Docker is available - Add defense-in-depth Docker readiness check in the emulator test workflow ## Changes - **`infrastructure/compute/startup_scripts/github_runner.sh`** — Stop the runner service at the very top of the startup script, before any package installations, preventing premature job acceptance - **`.github/workflows/ci.yml`** — Add a "Wait for Docker to be available" step with 60s retry loop in `backend-emulator-tests` job - **`docs/SELF-HOSTED-RUNNERS.md`** — Document Bug 8 in the Historical Bugs section ## Root Cause When runner VMs are stopped/started by the runner_manager, the GHA runner systemd service (from `svc.sh install` on a previous boot) auto-starts immediately on boot. The GCE startup script runs concurrently but only stops/re-registers the runner service after installing Docker and other dependencies. During this window, the runner can accept jobs that fail because Docker isn't ready yet. ## Testing - [x] Infrastructure-only change — no unit tests applicable - [x] Verified startup script logic is correct (subshell for cd, idempotent) - [x] CI workflow change adds graceful retry, not a breaking change ## Review - [x] Local CodeRabbit review completed (no findings) - [x] Documentation updated ## Note The startup script change will trigger Pulumi to detect a metadata change on runner VMs. Run `pulumi up` locally before merging per project convention. @coderabbitai ignore --- 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 (1M context) --- .github/workflows/ci.yml | 19 ++++++++++++++++++- docs/SELF-HOSTED-RUNNERS.md | 8 ++++++++ .../compute/startup_scripts/github_runner.sh | 10 ++++++++++ 3 files changed, 36 insertions(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 23976cd9..e489f770 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -628,7 +628,24 @@ jobs: - name: Install dependencies run: poetry install - - name: Start Docker for GCS emulator + - name: Wait for Docker to be available + run: | + # Self-hosted runners may still be initializing Docker on boot. + # Wait up to 60s for the docker command and daemon to be ready. + for i in $(seq 1 12); do + if command -v docker &>/dev/null && docker info &>/dev/null 2>&1; then + echo "Docker is ready" + exit 0 + fi + echo "Waiting for Docker... (attempt $i/12)" + sleep 5 + done + echo "ERROR: Docker not available after 60s" + echo "PATH=$PATH" + which docker 2>&1 || echo "docker binary not found" + exit 1 + + - name: Pull GCS emulator image run: docker pull fsouza/fake-gcs-server:latest - name: Run emulator integration tests diff --git a/docs/SELF-HOSTED-RUNNERS.md b/docs/SELF-HOSTED-RUNNERS.md index 5049784c..fd8137e5 100644 --- a/docs/SELF-HOSTED-RUNNERS.md +++ b/docs/SELF-HOSTED-RUNNERS.md @@ -165,6 +165,14 @@ The initial runner manager implementation had several bugs that prevented idle s **Fix:** Only update the specific runner identified by `runner_name` in the webhook payload. Removed the `in_progress` handler entirely. +### Bug 8: Flaky "docker: command not found" (Exit Code 127) + +**Problem:** Backend Emulator Integration Tests intermittently failed with `docker: command not found` (exit code 127). Re-running the job always succeeded. Root cause: When a stopped VM restarts, the GHA runner systemd service (installed via `svc.sh install` on a previous boot) auto-starts immediately, accepting CI jobs before the GCE startup script has finished setting up Docker and other dependencies. + +**Fix (two-part):** +1. **Startup script:** Stop the runner service at the very top of the script, before any package installations, preventing premature job acceptance. +2. **CI workflow:** Added a Docker readiness check step with 60-second retry loop in `backend-emulator-tests`, providing defense-in-depth. + ## Troubleshooting ### Runners Not Starting diff --git a/infrastructure/compute/startup_scripts/github_runner.sh b/infrastructure/compute/startup_scripts/github_runner.sh index a83d7f32..703ec1f8 100644 --- a/infrastructure/compute/startup_scripts/github_runner.sh +++ b/infrastructure/compute/startup_scripts/github_runner.sh @@ -7,6 +7,16 @@ exec > >(tee /var/log/github-runner-startup.log) 2>&1 echo "Starting GitHub Actions runner setup at $(date)" +# ==================== Stop existing runner service FIRST ==================== +# CRITICAL: The runner service from a previous boot auto-starts via systemd, +# which means it can accept CI jobs before Docker/Python/Java are ready. +# Stop it immediately to prevent flaky "docker: command not found" errors. +RUNNER_DIR="/home/runner/actions-runner" +if [ -f "$RUNNER_DIR/svc.sh" ]; then + echo "Stopping existing runner service to prevent premature job acceptance..." + (cd "$RUNNER_DIR" && ./svc.sh stop 2>/dev/null) || true +fi + # ==================== Network ==================== # Force apt to use IPv4 only — Cloud NAT doesn't support IPv6 echo 'Acquire::ForceIPv4 "true";' > /etc/apt/apt.conf.d/99force-ipv4