fix: prevent flaky "docker not found" in CI emulator tests (#543)

## Summary

- Fix race condition where GHA runner systemd service auto-starts on VM
boot and accepts CI jobs before Docker is available
- Add defense-in-depth Docker readiness check in the emulator test
workflow

## Changes

- **`infrastructure/compute/startup_scripts/github_runner.sh`** — Stop
the runner service at the very top of the startup script, before any
package installations, preventing premature job acceptance
- **`.github/workflows/ci.yml`** — Add a "Wait for Docker to be
available" step with 60s retry loop in `backend-emulator-tests` job
- **`docs/SELF-HOSTED-RUNNERS.md`** — Document Bug 8 in the Historical
Bugs section

## Root Cause

When runner VMs are stopped/started by the runner_manager, the GHA
runner systemd service (from `svc.sh install` on a previous boot)
auto-starts immediately on boot. The GCE startup script runs
concurrently but only stops/re-registers the runner service after
installing Docker and other dependencies. During this window, the runner
can accept jobs that fail because Docker isn't ready yet.

## Testing

- [x] Infrastructure-only change — no unit tests applicable
- [x] Verified startup script logic is correct (subshell for cd,
idempotent)
- [x] CI workflow change adds graceful retry, not a breaking change

## Review

- [x] Local CodeRabbit review completed (no findings)
- [x] Documentation updated

## Note

The startup script change will trigger Pulumi to detect a metadata
change on runner VMs. Run `pulumi up` locally before merging per project
convention.

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Andrew Beveridge ☄️andClaude Opus 4.6 authored and GitHub committed 2026-03-19 15:29:58 -04:00
1 parent b8239dc691
commit f582236d18
3 files changed
+36 -1

No files matched your search

+18 -1
View File
@@ -628,7 +628,24 @@ jobs:
- name: Install dependencies
run: poetry install
- name: Start Docker for GCS emulator
- name: Wait for Docker to be available
run: |
# Self-hosted runners may still be initializing Docker on boot.
# Wait up to 60s for the docker command and daemon to be ready.
for i in $(seq 1 12); do
if command -v docker &>/dev/null && docker info &>/dev/null 2>&1; then
echo "Docker is ready"
exit 0
fi
echo "Waiting for Docker... (attempt $i/12)"
sleep 5
done
echo "ERROR: Docker not available after 60s"
echo "PATH=$PATH"
which docker 2>&1 || echo "docker binary not found"
exit 1
- name: Pull GCS emulator image
run: docker pull fsouza/fake-gcs-server:latest
- name: Run emulator integration tests
+8
View File
@@ -165,6 +165,14 @@ The initial runner manager implementation had several bugs that prevented idle s
**Fix:** Only update the specific runner identified by `runner_name` in the webhook payload. Removed the `in_progress` handler entirely.
### Bug 8: Flaky "docker: command not found" (Exit Code 127)
**Problem:** Backend Emulator Integration Tests intermittently failed with `docker: command not found` (exit code 127). Re-running the job always succeeded. Root cause: When a stopped VM restarts, the GHA runner systemd service (installed via `svc.sh install` on a previous boot) auto-starts immediately, accepting CI jobs before the GCE startup script has finished setting up Docker and other dependencies.
**Fix (two-part):**
1. **Startup script:** Stop the runner service at the very top of the script, before any package installations, preventing premature job acceptance.
2. **CI workflow:** Added a Docker readiness check step with 60-second retry loop in `backend-emulator-tests`, providing defense-in-depth.
## Troubleshooting
### Runners Not Starting
@@ -7,6 +7,16 @@
exec > >(tee /var/log/github-runner-startup.log) 2>&1
echo "Starting GitHub Actions runner setup at $(date)"
# ==================== Stop existing runner service FIRST ====================
# CRITICAL: The runner service from a previous boot auto-starts via systemd,
# which means it can accept CI jobs before Docker/Python/Java are ready.
# Stop it immediately to prevent flaky "docker: command not found" errors.
RUNNER_DIR="/home/runner/actions-runner"
if [ -f "$RUNNER_DIR/svc.sh" ]; then
echo "Stopping existing runner service to prevent premature job acceptance..."
(cd "$RUNNER_DIR" && ./svc.sh stop 2>/dev/null) || true
fi
# ==================== Network ====================
# Force apt to use IPv4 only — Cloud NAT doesn't support IPv6
echo 'Acquire::ForceIPv4 "true";' > /etc/apt/apt.conf.d/99force-ipv4