ci: reliable R1 canary version-gate + retry (stop false pages) (#988)

Follow-up to the incident-hardening initiative — fixes the false
"Post-deploy canary FAILED (v0.222.3)" page from the #987 deploy.

**Root cause:** the R1 version-change gate did a single un-retried curl
to the raw Cloud Run **origin** `/api/health/detailed`, which came back
empty (cold start on the scaled-to-zero prior revision and/or origin
edge-auth on the detailed endpoint). It logged `Currently-serving
version: <unknown>` and **fail-opened → ran the canary on a no-bump
deploy**, which then flake-failed on a review-page timing wait and
paged.

**Fixes:**
1. **Reliable gate** — read the currently-serving version via the
**edge** (`https://api.nomadkaraoke.com/api/health/detailed`, warm +
Cloudflare-fronted, verified to return the version) with retries. A
no-bump deploy now correctly **skips** the canary.
2. **Retry the canary** — `--retries=2` so a single flaky UI wait
doesn't page; only a failure surviving retries is real. (The
happy-path's review-page waits are inherently flaky; the identical spec
passed on the 0.223.2 deploy.)

No app code changed → no version bump. YAML validated.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Andrew Beveridge ☄️andClaude Opus 4.8 authored and GitHub committed 2026-09-12 23:22:10 +00:00
1 parent 62843f0286
commit d93dbddd48
1 file changed
+21 -9
+21 -9
View File
@@ -1158,22 +1158,29 @@ jobs:
id: version_changed
run: |
NEW_VERSION="${{ steps.version_check.outputs.version }}"
SERVICE_URL=$(gcloud run services describe karaoke-backend \
--region us-central1 \
--format='value(status.url)' 2>/dev/null || echo "")
# Read the currently-serving version via the EDGE (api.nomadkaraoke.com),
# NOT the raw Cloud Run origin URL, and retry. The single un-retried
# origin probe returned empty on #987 (cold start on the scaled-to-zero
# prior revision and/or origin edge-auth on the detailed endpoint), so
# the gate fail-opened and ran the canary on a no-bump deploy — which
# then flake-failed and paged. The edge URL is warm (real traffic) and
# Cloudflare-fronted, so it returns the version reliably.
CURRENT_VERSION=""
if [ -n "$SERVICE_URL" ]; then
CURRENT_VERSION=$(curl -sf --connect-timeout 5 --max-time 20 \
"$SERVICE_URL/api/health/detailed" | jq -r '.version // empty' 2>/dev/null || echo "")
fi
for attempt in 1 2 3 4; do
CURRENT_VERSION=$(curl -sf --connect-timeout 5 --max-time 25 \
"https://api.nomadkaraoke.com/api/health/detailed" | jq -r '.version // empty' 2>/dev/null || echo "")
[ -n "$CURRENT_VERSION" ] && break
echo " version probe attempt $attempt/4 empty; retrying…"
sleep 5
done
echo "Currently-serving version: ${CURRENT_VERSION:-<unknown>}"
echo "New version: $NEW_VERSION"
if [ -n "$CURRENT_VERSION" ] && [ "$CURRENT_VERSION" = "$NEW_VERSION" ]; then
echo "changed=false" >> "$GITHUB_OUTPUT"
echo "↩️ Backend version unchanged — post-deploy canary will be SKIPPED."
else
# Unknown current version (first deploy / unreachable) → run the
# canary to be safe rather than skip a potentially real change.
# Unknown current version (first deploy / unreachable after retries) →
# run the canary to be safe rather than skip a potentially real change.
echo "changed=true" >> "$GITHUB_OUTPUT"
echo "✅ Backend version changed (or current unknown) — post-deploy canary will RUN."
fi
@@ -2210,10 +2217,15 @@ jobs:
echo "=== Post-deploy canary: happy path against production ==="
echo "Deployed version: ${{ needs.deploy-backend.outputs.version }}"
echo "Time: $(date)"
# --retries=2: the happy-path has inherently flaky UI waits (e.g. the
# review page's "proceed to instrumental" enable), and this canary
# PAGES on failure — a single flaky attempt must not cry wolf. Only a
# failure that survives 2 retries is treated as real.
npx playwright test happy-path-real-user.spec.ts \
--config=playwright.production.config.ts \
--project=chromium \
--reporter=list \
--retries=2 \
--timeout=3600000 \
2>&1 | tee test-output-canary.log
# Capture Playwright's real exit code, not tee's (0). Matches