mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
fix: prevent flaky "docker not found" in CI emulator tests (#543)
## Summary - Fix race condition where GHA runner systemd service auto-starts on VM boot and accepts CI jobs before Docker is available - Add defense-in-depth Docker readiness check in the emulator test workflow ## Changes - **`infrastructure/compute/startup_scripts/github_runner.sh`** — Stop the runner service at the very top of the startup script, before any package installations, preventing premature job acceptance - **`.github/workflows/ci.yml`** — Add a "Wait for Docker to be available" step with 60s retry loop in `backend-emulator-tests` job - **`docs/SELF-HOSTED-RUNNERS.md`** — Document Bug 8 in the Historical Bugs section ## Root Cause When runner VMs are stopped/started by the runner_manager, the GHA runner systemd service (from `svc.sh install` on a previous boot) auto-starts immediately on boot. The GCE startup script runs concurrently but only stops/re-registers the runner service after installing Docker and other dependencies. During this window, the runner can accept jobs that fail because Docker isn't ready yet. ## Testing - [x] Infrastructure-only change — no unit tests applicable - [x] Verified startup script logic is correct (subshell for cd, idempotent) - [x] CI workflow change adds graceful retry, not a breaking change ## Review - [x] Local CodeRabbit review completed (no findings) - [x] Documentation updated ## Note The startup script change will trigger Pulumi to detect a metadata change on runner VMs. Run `pulumi up` locally before merging per project convention. @coderabbitai ignore --- 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
b8239dc691
commit
f582236d18
3 files changed
+36
-1
No files matched your search
@@ -628,7 +628,24 @@ jobs:
|
||||
- name: Install dependencies
|
||||
run: poetry install
|
||||
|
||||
- name: Start Docker for GCS emulator
|
||||
- name: Wait for Docker to be available
|
||||
run: |
|
||||
# Self-hosted runners may still be initializing Docker on boot.
|
||||
# Wait up to 60s for the docker command and daemon to be ready.
|
||||
for i in $(seq 1 12); do
|
||||
if command -v docker &>/dev/null && docker info &>/dev/null 2>&1; then
|
||||
echo "Docker is ready"
|
||||
exit 0
|
||||
fi
|
||||
echo "Waiting for Docker... (attempt $i/12)"
|
||||
sleep 5
|
||||
done
|
||||
echo "ERROR: Docker not available after 60s"
|
||||
echo "PATH=$PATH"
|
||||
which docker 2>&1 || echo "docker binary not found"
|
||||
exit 1
|
||||
|
||||
- name: Pull GCS emulator image
|
||||
run: docker pull fsouza/fake-gcs-server:latest
|
||||
|
||||
- name: Run emulator integration tests
|
||||
|
||||
@@ -165,6 +165,14 @@ The initial runner manager implementation had several bugs that prevented idle s
|
||||
|
||||
**Fix:** Only update the specific runner identified by `runner_name` in the webhook payload. Removed the `in_progress` handler entirely.
|
||||
|
||||
### Bug 8: Flaky "docker: command not found" (Exit Code 127)
|
||||
|
||||
**Problem:** Backend Emulator Integration Tests intermittently failed with `docker: command not found` (exit code 127). Re-running the job always succeeded. Root cause: When a stopped VM restarts, the GHA runner systemd service (installed via `svc.sh install` on a previous boot) auto-starts immediately, accepting CI jobs before the GCE startup script has finished setting up Docker and other dependencies.
|
||||
|
||||
**Fix (two-part):**
|
||||
1. **Startup script:** Stop the runner service at the very top of the script, before any package installations, preventing premature job acceptance.
|
||||
2. **CI workflow:** Added a Docker readiness check step with 60-second retry loop in `backend-emulator-tests`, providing defense-in-depth.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Runners Not Starting
|
||||
|
||||
@@ -7,6 +7,16 @@
|
||||
exec > >(tee /var/log/github-runner-startup.log) 2>&1
|
||||
echo "Starting GitHub Actions runner setup at $(date)"
|
||||
|
||||
# ==================== Stop existing runner service FIRST ====================
|
||||
# CRITICAL: The runner service from a previous boot auto-starts via systemd,
|
||||
# which means it can accept CI jobs before Docker/Python/Java are ready.
|
||||
# Stop it immediately to prevent flaky "docker: command not found" errors.
|
||||
RUNNER_DIR="/home/runner/actions-runner"
|
||||
if [ -f "$RUNNER_DIR/svc.sh" ]; then
|
||||
echo "Stopping existing runner service to prevent premature job acceptance..."
|
||||
(cd "$RUNNER_DIR" && ./svc.sh stop 2>/dev/null) || true
|
||||
fi
|
||||
|
||||
# ==================== Network ====================
|
||||
# Force apt to use IPv4 only — Cloud NAT doesn't support IPv6
|
||||
echo 'Acquire::ForceIPv4 "true";' > /etc/apt/apt.conf.d/99force-ipv4
|
||||
|
||||
Reference in new issue
Block a user