diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 23976cd9..e489f770 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -628,7 +628,24 @@ jobs: - name: Install dependencies run: poetry install - - name: Start Docker for GCS emulator + - name: Wait for Docker to be available + run: | + # Self-hosted runners may still be initializing Docker on boot. + # Wait up to 60s for the docker command and daemon to be ready. + for i in $(seq 1 12); do + if command -v docker &>/dev/null && docker info &>/dev/null 2>&1; then + echo "Docker is ready" + exit 0 + fi + echo "Waiting for Docker... (attempt $i/12)" + sleep 5 + done + echo "ERROR: Docker not available after 60s" + echo "PATH=$PATH" + which docker 2>&1 || echo "docker binary not found" + exit 1 + + - name: Pull GCS emulator image run: docker pull fsouza/fake-gcs-server:latest - name: Run emulator integration tests diff --git a/docs/SELF-HOSTED-RUNNERS.md b/docs/SELF-HOSTED-RUNNERS.md index 5049784c..fd8137e5 100644 --- a/docs/SELF-HOSTED-RUNNERS.md +++ b/docs/SELF-HOSTED-RUNNERS.md @@ -165,6 +165,14 @@ The initial runner manager implementation had several bugs that prevented idle s **Fix:** Only update the specific runner identified by `runner_name` in the webhook payload. Removed the `in_progress` handler entirely. +### Bug 8: Flaky "docker: command not found" (Exit Code 127) + +**Problem:** Backend Emulator Integration Tests intermittently failed with `docker: command not found` (exit code 127). Re-running the job always succeeded. Root cause: When a stopped VM restarts, the GHA runner systemd service (installed via `svc.sh install` on a previous boot) auto-starts immediately, accepting CI jobs before the GCE startup script has finished setting up Docker and other dependencies. + +**Fix (two-part):** +1. **Startup script:** Stop the runner service at the very top of the script, before any package installations, preventing premature job acceptance. +2. **CI workflow:** Added a Docker readiness check step with 60-second retry loop in `backend-emulator-tests`, providing defense-in-depth. + ## Troubleshooting ### Runners Not Starting diff --git a/infrastructure/compute/startup_scripts/github_runner.sh b/infrastructure/compute/startup_scripts/github_runner.sh index a83d7f32..703ec1f8 100644 --- a/infrastructure/compute/startup_scripts/github_runner.sh +++ b/infrastructure/compute/startup_scripts/github_runner.sh @@ -7,6 +7,16 @@ exec > >(tee /var/log/github-runner-startup.log) 2>&1 echo "Starting GitHub Actions runner setup at $(date)" +# ==================== Stop existing runner service FIRST ==================== +# CRITICAL: The runner service from a previous boot auto-starts via systemd, +# which means it can accept CI jobs before Docker/Python/Java are ready. +# Stop it immediately to prevent flaky "docker: command not found" errors. +RUNNER_DIR="/home/runner/actions-runner" +if [ -f "$RUNNER_DIR/svc.sh" ]; then + echo "Stopping existing runner service to prevent premature job acceptance..." + (cd "$RUNNER_DIR" && ./svc.sh stop 2>/dev/null) || true +fi + # ==================== Network ==================== # Force apt to use IPv4 only — Cloud NAT doesn't support IPv6 echo 'Acquire::ForceIPv4 "true";' > /etc/apt/apt.conf.d/99force-ipv4