ci: remove post-deploy canary + never email test accounts on stale review (v0.245.2) (#1068)

## Summary
- Removes the `deploy-verify` job ("Deploy - Post-Deploy Canary (Happy
Path)") from `ci.yml`, plus the `Detect backend version change` step /
`version_changed` output that only gated it.
- `stale_review_processor`: jobs owned by automated-test accounts
(`is_test_email` — `e2e-test-runner@nomadkaraoke.com`,
`@inbox.testmail.app`) no longer get the 24h reminder, review-expired,
or duration-confirm-expired emails. They are still auto-expired at 48h
so stranded test jobs don't linger.
- Updates comments + `docs/TROUBLESHOOTING.md` / `docs/TESTING.md` that
referenced the canary. Version → 0.245.2.

## Why
The canary created a real production job (piri – dog, as
`e2e-test-runner@nomadkaraoke.com`) on every backend version bump.
`ci.yml` uses `cancel-in-progress: true`, so back-to-back merges
cancelled it mid-run — the process is killed before Playwright's
`afterEach` cleanup runs, stranding the job in `awaiting_review`. The
stale-review processor then emailed the test account (which forwards to
Andrew's inbox) at 24h/48h (e.g. job `adb8f393` from cancelled run
36288893657; earlier `a5f84cc3`, `10b532e2`). The canary was overkill;
the daily E2E and the traffic-gated candidate-revision smoke remain, and
the email guard backstops the daily E2E if it ever strands a job.

## Test plan
- [x] New unit tests: test-account reminder is silent (+ still flagged),
review expiry + duration-confirm expiry cancel without email
- [x] `pytest backend/tests --ignore=backend/tests/emulator` — 4883
passed
- [x] `actionlint` — no new findings

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* CI no longer runs a post-deployment production canary or sends its
test results and failure notifications. Candidate-revision smoke testing
remains in place.
* Production real-job verification is handled manually or by the daily
E2E run.
* **Documentation**
* Testing and rollback guidance now reflects the updated verification
and recovery steps. Daily E2E failures are identified as a reason to
roll back.
* **Bug Fixes**
* Automated test accounts no longer receive stale-review reminder or
expiry emails; existing expiry and cancellation handling remains in
place.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Andrew Beveridge ☄️andClaude Opus 5.5 authored and GitHub committed 2026-09-27 23:37:45 -04:00
1 parent 5b2a0f8c3e
commit 8a5f754efd
8 files changed
+72 -188

No files matched your search

+5 -173
View File
@@ -1095,9 +1095,6 @@ jobs:
outputs:
base_rebuilt: ${{ steps.check-base.outputs.needs_rebuild }}
version: ${{ steps.version_check.outputs.version }}
# R1: gates the post-deploy canary — "true" only when the backend version
# actually changed (see "Detect backend version change" step).
version_changed: ${{ steps.version_changed.outputs.changed }}
env:
REGISTRY: us-central1-docker.pkg.dev
PROJECT_ID: nomadkaraoke
@@ -1153,43 +1150,6 @@ jobs:
echo "📋 Current version: $VERSION"
echo "🚀 Deploying on every push to main"
# R1 gate — only run the post-deploy functional canary when the BACKEND
# version actually changed. A frontend-/docs-/infra-only merge still runs
# deploy-backend (image rebuild) but ships an identical version; a real
# karaoke generation is slow + resource-heavy, so we short-circuit the
# canary in that case. We compare the version we're about to ship against
# the version currently served by production.
- name: Detect backend version change
id: version_changed
run: |
NEW_VERSION="${{ steps.version_check.outputs.version }}"
# Read the currently-serving version via the EDGE (api.nomadkaraoke.com),
# NOT the raw Cloud Run origin URL, and retry. The single un-retried
# origin probe returned empty on #987 (cold start on the scaled-to-zero
# prior revision and/or origin edge-auth on the detailed endpoint), so
# the gate fail-opened and ran the canary on a no-bump deploy — which
# then flake-failed and paged. The edge URL is warm (real traffic) and
# Cloudflare-fronted, so it returns the version reliably.
CURRENT_VERSION=""
for attempt in 1 2 3 4; do
CURRENT_VERSION=$(curl -sf --connect-timeout 5 --max-time 25 \
"https://api.nomadkaraoke.com/api/health/detailed" | jq -r '.version // empty' 2>/dev/null || echo "")
[ -n "$CURRENT_VERSION" ] && break
echo " version probe attempt $attempt/4 empty; retrying…"
sleep 5
done
echo "Currently-serving version: ${CURRENT_VERSION:-<unknown>}"
echo "New version: $NEW_VERSION"
if [ -n "$CURRENT_VERSION" ] && [ "$CURRENT_VERSION" = "$NEW_VERSION" ]; then
echo "changed=false" >> "$GITHUB_OUTPUT"
echo "↩️ Backend version unchanged — post-deploy canary will be SKIPPED."
else
# Unknown current version (first deploy / unreachable after retries) →
# run the canary to be safe rather than skip a potentially real change.
echo "changed=true" >> "$GITHUB_OUTPUT"
echo "✅ Backend version changed (or current unknown) — post-deploy canary will RUN."
fi
- name: Capture build metadata
id: build-meta
run: |
@@ -2057,8 +2017,8 @@ jobs:
# smoke fails we simply don't migrate: traffic stays on the prior revision,
# so a bad revision serves 0% instead of 100% (auto-revert by default).
# NOTE: this protects only the Cloud Run *service*. The Cloud Run *jobs*
# updated below cannot traffic-split, so they still rely on R1's real-job
# post-deploy canary + the codified `scripts/rollback.sh` (R2).
# updated below cannot traffic-split, so they rely on the daily E2E run +
# the codified `scripts/rollback.sh` (R2).
- name: Deploy to Cloud Run (candidate revision, no traffic)
id: deploy_candidate
run: |
@@ -2161,9 +2121,9 @@ jobs:
- name: Update Cloud Run Jobs with new image
# NOTE: Cloud Run *Jobs* cannot traffic-split or canary — this is an
# image-only, all-or-nothing update picked up on each job's next
# invocation. There is no per-revision smoke here; the jobs rely on R1's
# post-deploy real-job canary and the codified `scripts/rollback.sh` (R2)
# for recovery if a bad image reaches them.
# invocation. There is no per-revision smoke here; the jobs rely on the
# daily E2E run and the codified `scripts/rollback.sh` (R2) for recovery
# if a bad image reaches them.
run: |
IMAGE="${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:${{ github.sha }}"
GPU_IMAGE="${{ env.GPU_REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.GPU_REPOSITORY }}/${{ env.GPU_APP_IMAGE_NAME }}:${{ github.sha }}"
@@ -2253,134 +2213,6 @@ jobs:
echo "📊 Built on self-hosted GCP runner (200GB SSD, no disk space issues)"
echo " Runner: gcp-runner with 4 vCPU, 16GB RAM"
# ============================================
# R1 — POST-DEPLOY FUNCTIONAL CANARY (version-change-gated)
# Runs the real production happy-path (admin-credit variant, no Stripe) after
# a backend deploy — but ONLY when the backend version actually changed
# (deploy-backend.outputs.version_changed == 'true'). A frontend-/docs-/
# infra-only merge ships an identical backend version and short-circuits here,
# so we don't burn encode-VM time + a real Drive/YouTube publish on a no-op.
# On failure: the workflow fails loudly (Discord) — this is where an operator
# invokes `scripts/rollback.sh` (R2) to revert the service + jobs.
# ============================================
deploy-verify:
name: "Deploy - Post-Deploy Canary (Happy Path)"
needs: [deploy-backend]
runs-on: ubuntu-latest
timeout-minutes: 90
if: >-
github.ref == 'refs/heads/main' && github.event_name == 'push' &&
needs.deploy-backend.result == 'success' &&
needs.deploy-backend.outputs.version_changed == 'true'
steps:
- uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
cache-dependency-path: frontend/package-lock.json
- name: Install dependencies
working-directory: frontend
run: npm ci --legacy-peer-deps
- name: Install Playwright browsers
working-directory: frontend
run: npx playwright install --with-deps chromium
- name: Create test-results directory
working-directory: frontend
run: mkdir -p test-results
- name: Ensure testmail.app domain is allowlisted
run: |
curl -sf -X POST \
"https://api.nomadkaraoke.com/api/admin/rate-limits/blocklists/allowlisted-domains" \
-H "X-Admin-Token: ${{ secrets.E2E_ADMIN_TOKEN }}" \
-H "Content-Type: application/json" \
-d '{"domain": "inbox.testmail.app"}' || echo "Warning: allowlist may already exist"
# Admin-credit variant: impersonate the E2E test runner (which holds
# admin-granted credits) instead of going through Stripe. Same token-refresh
# pattern as e2e-daily.yml's use_test_token path.
- name: Refresh E2E session token
id: refresh_token
run: |
RESPONSE=$(curl -sf --connect-timeout 5 --max-time 30 -X POST \
"https://api.nomadkaraoke.com/api/admin/users/e2e-test-runner@nomadkaraoke.com/impersonate" \
-H "X-Admin-Token: ${{ secrets.E2E_ADMIN_TOKEN }}" \
-H "Content-Type: application/json")
TOKEN=$(echo "$RESPONSE" | python3 -c 'import json,sys; print(json.load(sys.stdin)["session_token"])')
echo "::add-mask::$TOKEN"
echo "token=$TOKEN" >> $GITHUB_OUTPUT
- name: Run post-deploy happy-path canary
id: run_test
working-directory: frontend
env:
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
TESTMAIL_NAMESPACE: ${{ secrets.TESTMAIL_NAMESPACE }}
E2E_TEST_TOKEN: ${{ steps.refresh_token.outputs.token }}
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
E2E_BYPASS_KEY: ${{ secrets.E2E_BYPASS_KEY }}
run: |
echo "=== Post-deploy canary: happy path against production ==="
echo "Deployed version: ${{ needs.deploy-backend.outputs.version }}"
echo "Time: $(date)"
# --retries=2: the happy-path has inherently flaky UI waits (e.g. the
# review page's "proceed to instrumental" enable), and this canary
# PAGES on failure — a single flaky attempt must not cry wolf. Only a
# failure that survives 2 retries is treated as real.
npx playwright test happy-path-real-user.spec.ts \
--config=playwright.production.config.ts \
--project=chromium \
--reporter=list \
--retries=2 \
--timeout=3600000 \
2>&1 | tee test-output-canary.log
# Capture Playwright's real exit code, not tee's (0). Matches
# e2e-daily.yml's rationale (grepping the log is fragile).
echo "exit_code=${PIPESTATUS[0]}" >> "$GITHUB_OUTPUT"
continue-on-error: true
- name: Check canary result
id: test_result
run: |
if [ "${{ steps.run_test.outputs.exit_code }}" = "0" ]; then
echo "result=success" >> "$GITHUB_OUTPUT"
else
echo "result=failure" >> "$GITHUB_OUTPUT"
fi
- name: Upload canary artifacts
if: always()
uses: actions/upload-artifact@v4
with:
name: deploy-verify-canary-${{ github.run_number }}
path: |
frontend/test-results/
frontend/playwright-report/
frontend/test-output-canary.log
retention-days: 30
- name: Discord notification (canary failed)
if: steps.test_result.outputs.result == 'failure'
run: |
curl -s -H "Content-Type: application/json" \
-d "{\"embeds\": [{\"title\": \"🚨 Post-deploy canary FAILED (v${{ needs.deploy-backend.outputs.version }})\", \"description\": \"The production happy-path failed AFTER deploying v${{ needs.deploy-backend.outputs.version }}. The service already migrated 100% traffic to the new revision — consider rolling back with scripts/rollback.sh.\n\n**Run:** #${{ github.run_number }}\n**Details:** ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}\", \"color\": 15158332}]}" \
"${{ secrets.DISCORD_ALERT_WEBHOOK_URL }}" || echo "Discord notify failed (non-fatal)"
- name: Fail if canary failed
if: steps.test_result.outputs.result == 'failure'
run: |
echo "::error::Post-deploy canary failed for v${{ needs.deploy-backend.outputs.version }}."
echo "Roll back now with:"
echo " scripts/rollback.sh --list # find the previous good revision + version"
echo " scripts/rollback.sh --to-revision <PREV_REV> --image-version <PREV_VERSION>"
exit 1
# NOTE: Pulumi infrastructure deploys are done locally, not in CI.
# CI runners run on the infrastructure Pulumi manages, creating a circular
# dependency. Always run `pulumi up` locally before merging infra PRs.