mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
ci: remove post-deploy canary + never email test accounts on stale review (v0.245.2) (#1068)
## Summary
- Removes the `deploy-verify` job ("Deploy - Post-Deploy Canary (Happy
Path)") from `ci.yml`, plus the `Detect backend version change` step /
`version_changed` output that only gated it.
- `stale_review_processor`: jobs owned by automated-test accounts
(`is_test_email` — `e2e-test-runner@nomadkaraoke.com`,
`@inbox.testmail.app`) no longer get the 24h reminder, review-expired,
or duration-confirm-expired emails. They are still auto-expired at 48h
so stranded test jobs don't linger.
- Updates comments + `docs/TROUBLESHOOTING.md` / `docs/TESTING.md` that
referenced the canary. Version → 0.245.2.
## Why
The canary created a real production job (piri – dog, as
`e2e-test-runner@nomadkaraoke.com`) on every backend version bump.
`ci.yml` uses `cancel-in-progress: true`, so back-to-back merges
cancelled it mid-run — the process is killed before Playwright's
`afterEach` cleanup runs, stranding the job in `awaiting_review`. The
stale-review processor then emailed the test account (which forwards to
Andrew's inbox) at 24h/48h (e.g. job `adb8f393` from cancelled run
36288893657; earlier `a5f84cc3`, `10b532e2`). The canary was overkill;
the daily E2E and the traffic-gated candidate-revision smoke remain, and
the email guard backstops the daily E2E if it ever strands a job.
## Test plan
- [x] New unit tests: test-account reminder is silent (+ still flagged),
review expiry + duration-confirm expiry cancel without email
- [x] `pytest backend/tests --ignore=backend/tests/emulator` — 4883
passed
- [x] `actionlint` — no new findings
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* CI no longer runs a post-deployment production canary or sends its
test results and failure notifications. Candidate-revision smoke testing
remains in place.
* Production real-job verification is handled manually or by the daily
E2E run.
* **Documentation**
* Testing and rollback guidance now reflects the updated verification
and recovery steps. Daily E2E failures are identified as a reason to
roll back.
* **Bug Fixes**
* Automated test accounts no longer receive stale-review reminder or
expiry emails; existing expiry and cancellation handling remains in
place.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
5b2a0f8c3e
commit
8a5f754efd
8 files changed
+72
-188
No files matched your search
+5
-173
@@ -1095,9 +1095,6 @@ jobs:
|
||||
outputs:
|
||||
base_rebuilt: ${{ steps.check-base.outputs.needs_rebuild }}
|
||||
version: ${{ steps.version_check.outputs.version }}
|
||||
# R1: gates the post-deploy canary — "true" only when the backend version
|
||||
# actually changed (see "Detect backend version change" step).
|
||||
version_changed: ${{ steps.version_changed.outputs.changed }}
|
||||
env:
|
||||
REGISTRY: us-central1-docker.pkg.dev
|
||||
PROJECT_ID: nomadkaraoke
|
||||
@@ -1153,43 +1150,6 @@ jobs:
|
||||
echo "📋 Current version: $VERSION"
|
||||
echo "🚀 Deploying on every push to main"
|
||||
|
||||
# R1 gate — only run the post-deploy functional canary when the BACKEND
|
||||
# version actually changed. A frontend-/docs-/infra-only merge still runs
|
||||
# deploy-backend (image rebuild) but ships an identical version; a real
|
||||
# karaoke generation is slow + resource-heavy, so we short-circuit the
|
||||
# canary in that case. We compare the version we're about to ship against
|
||||
# the version currently served by production.
|
||||
- name: Detect backend version change
|
||||
id: version_changed
|
||||
run: |
|
||||
NEW_VERSION="${{ steps.version_check.outputs.version }}"
|
||||
# Read the currently-serving version via the EDGE (api.nomadkaraoke.com),
|
||||
# NOT the raw Cloud Run origin URL, and retry. The single un-retried
|
||||
# origin probe returned empty on #987 (cold start on the scaled-to-zero
|
||||
# prior revision and/or origin edge-auth on the detailed endpoint), so
|
||||
# the gate fail-opened and ran the canary on a no-bump deploy — which
|
||||
# then flake-failed and paged. The edge URL is warm (real traffic) and
|
||||
# Cloudflare-fronted, so it returns the version reliably.
|
||||
CURRENT_VERSION=""
|
||||
for attempt in 1 2 3 4; do
|
||||
CURRENT_VERSION=$(curl -sf --connect-timeout 5 --max-time 25 \
|
||||
"https://api.nomadkaraoke.com/api/health/detailed" | jq -r '.version // empty' 2>/dev/null || echo "")
|
||||
[ -n "$CURRENT_VERSION" ] && break
|
||||
echo " version probe attempt $attempt/4 empty; retrying…"
|
||||
sleep 5
|
||||
done
|
||||
echo "Currently-serving version: ${CURRENT_VERSION:-<unknown>}"
|
||||
echo "New version: $NEW_VERSION"
|
||||
if [ -n "$CURRENT_VERSION" ] && [ "$CURRENT_VERSION" = "$NEW_VERSION" ]; then
|
||||
echo "changed=false" >> "$GITHUB_OUTPUT"
|
||||
echo "↩️ Backend version unchanged — post-deploy canary will be SKIPPED."
|
||||
else
|
||||
# Unknown current version (first deploy / unreachable after retries) →
|
||||
# run the canary to be safe rather than skip a potentially real change.
|
||||
echo "changed=true" >> "$GITHUB_OUTPUT"
|
||||
echo "✅ Backend version changed (or current unknown) — post-deploy canary will RUN."
|
||||
fi
|
||||
|
||||
- name: Capture build metadata
|
||||
id: build-meta
|
||||
run: |
|
||||
@@ -2057,8 +2017,8 @@ jobs:
|
||||
# smoke fails we simply don't migrate: traffic stays on the prior revision,
|
||||
# so a bad revision serves 0% instead of 100% (auto-revert by default).
|
||||
# NOTE: this protects only the Cloud Run *service*. The Cloud Run *jobs*
|
||||
# updated below cannot traffic-split, so they still rely on R1's real-job
|
||||
# post-deploy canary + the codified `scripts/rollback.sh` (R2).
|
||||
# updated below cannot traffic-split, so they rely on the daily E2E run +
|
||||
# the codified `scripts/rollback.sh` (R2).
|
||||
- name: Deploy to Cloud Run (candidate revision, no traffic)
|
||||
id: deploy_candidate
|
||||
run: |
|
||||
@@ -2161,9 +2121,9 @@ jobs:
|
||||
- name: Update Cloud Run Jobs with new image
|
||||
# NOTE: Cloud Run *Jobs* cannot traffic-split or canary — this is an
|
||||
# image-only, all-or-nothing update picked up on each job's next
|
||||
# invocation. There is no per-revision smoke here; the jobs rely on R1's
|
||||
# post-deploy real-job canary and the codified `scripts/rollback.sh` (R2)
|
||||
# for recovery if a bad image reaches them.
|
||||
# invocation. There is no per-revision smoke here; the jobs rely on the
|
||||
# daily E2E run and the codified `scripts/rollback.sh` (R2) for recovery
|
||||
# if a bad image reaches them.
|
||||
run: |
|
||||
IMAGE="${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:${{ github.sha }}"
|
||||
GPU_IMAGE="${{ env.GPU_REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.GPU_REPOSITORY }}/${{ env.GPU_APP_IMAGE_NAME }}:${{ github.sha }}"
|
||||
@@ -2253,134 +2213,6 @@ jobs:
|
||||
echo "📊 Built on self-hosted GCP runner (200GB SSD, no disk space issues)"
|
||||
echo " Runner: gcp-runner with 4 vCPU, 16GB RAM"
|
||||
|
||||
# ============================================
|
||||
# R1 — POST-DEPLOY FUNCTIONAL CANARY (version-change-gated)
|
||||
# Runs the real production happy-path (admin-credit variant, no Stripe) after
|
||||
# a backend deploy — but ONLY when the backend version actually changed
|
||||
# (deploy-backend.outputs.version_changed == 'true'). A frontend-/docs-/
|
||||
# infra-only merge ships an identical backend version and short-circuits here,
|
||||
# so we don't burn encode-VM time + a real Drive/YouTube publish on a no-op.
|
||||
# On failure: the workflow fails loudly (Discord) — this is where an operator
|
||||
# invokes `scripts/rollback.sh` (R2) to revert the service + jobs.
|
||||
# ============================================
|
||||
deploy-verify:
|
||||
name: "Deploy - Post-Deploy Canary (Happy Path)"
|
||||
needs: [deploy-backend]
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 90
|
||||
if: >-
|
||||
github.ref == 'refs/heads/main' && github.event_name == 'push' &&
|
||||
needs.deploy-backend.result == 'success' &&
|
||||
needs.deploy-backend.outputs.version_changed == 'true'
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '20'
|
||||
cache: 'npm'
|
||||
cache-dependency-path: frontend/package-lock.json
|
||||
|
||||
- name: Install dependencies
|
||||
working-directory: frontend
|
||||
run: npm ci --legacy-peer-deps
|
||||
|
||||
- name: Install Playwright browsers
|
||||
working-directory: frontend
|
||||
run: npx playwright install --with-deps chromium
|
||||
|
||||
- name: Create test-results directory
|
||||
working-directory: frontend
|
||||
run: mkdir -p test-results
|
||||
|
||||
- name: Ensure testmail.app domain is allowlisted
|
||||
run: |
|
||||
curl -sf -X POST \
|
||||
"https://api.nomadkaraoke.com/api/admin/rate-limits/blocklists/allowlisted-domains" \
|
||||
-H "X-Admin-Token: ${{ secrets.E2E_ADMIN_TOKEN }}" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"domain": "inbox.testmail.app"}' || echo "Warning: allowlist may already exist"
|
||||
|
||||
# Admin-credit variant: impersonate the E2E test runner (which holds
|
||||
# admin-granted credits) instead of going through Stripe. Same token-refresh
|
||||
# pattern as e2e-daily.yml's use_test_token path.
|
||||
- name: Refresh E2E session token
|
||||
id: refresh_token
|
||||
run: |
|
||||
RESPONSE=$(curl -sf --connect-timeout 5 --max-time 30 -X POST \
|
||||
"https://api.nomadkaraoke.com/api/admin/users/e2e-test-runner@nomadkaraoke.com/impersonate" \
|
||||
-H "X-Admin-Token: ${{ secrets.E2E_ADMIN_TOKEN }}" \
|
||||
-H "Content-Type: application/json")
|
||||
TOKEN=$(echo "$RESPONSE" | python3 -c 'import json,sys; print(json.load(sys.stdin)["session_token"])')
|
||||
echo "::add-mask::$TOKEN"
|
||||
echo "token=$TOKEN" >> $GITHUB_OUTPUT
|
||||
|
||||
- name: Run post-deploy happy-path canary
|
||||
id: run_test
|
||||
working-directory: frontend
|
||||
env:
|
||||
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
|
||||
TESTMAIL_NAMESPACE: ${{ secrets.TESTMAIL_NAMESPACE }}
|
||||
E2E_TEST_TOKEN: ${{ steps.refresh_token.outputs.token }}
|
||||
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
|
||||
E2E_BYPASS_KEY: ${{ secrets.E2E_BYPASS_KEY }}
|
||||
run: |
|
||||
echo "=== Post-deploy canary: happy path against production ==="
|
||||
echo "Deployed version: ${{ needs.deploy-backend.outputs.version }}"
|
||||
echo "Time: $(date)"
|
||||
# --retries=2: the happy-path has inherently flaky UI waits (e.g. the
|
||||
# review page's "proceed to instrumental" enable), and this canary
|
||||
# PAGES on failure — a single flaky attempt must not cry wolf. Only a
|
||||
# failure that survives 2 retries is treated as real.
|
||||
npx playwright test happy-path-real-user.spec.ts \
|
||||
--config=playwright.production.config.ts \
|
||||
--project=chromium \
|
||||
--reporter=list \
|
||||
--retries=2 \
|
||||
--timeout=3600000 \
|
||||
2>&1 | tee test-output-canary.log
|
||||
# Capture Playwright's real exit code, not tee's (0). Matches
|
||||
# e2e-daily.yml's rationale (grepping the log is fragile).
|
||||
echo "exit_code=${PIPESTATUS[0]}" >> "$GITHUB_OUTPUT"
|
||||
continue-on-error: true
|
||||
|
||||
- name: Check canary result
|
||||
id: test_result
|
||||
run: |
|
||||
if [ "${{ steps.run_test.outputs.exit_code }}" = "0" ]; then
|
||||
echo "result=success" >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
echo "result=failure" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
|
||||
- name: Upload canary artifacts
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: deploy-verify-canary-${{ github.run_number }}
|
||||
path: |
|
||||
frontend/test-results/
|
||||
frontend/playwright-report/
|
||||
frontend/test-output-canary.log
|
||||
retention-days: 30
|
||||
|
||||
- name: Discord notification (canary failed)
|
||||
if: steps.test_result.outputs.result == 'failure'
|
||||
run: |
|
||||
curl -s -H "Content-Type: application/json" \
|
||||
-d "{\"embeds\": [{\"title\": \"🚨 Post-deploy canary FAILED (v${{ needs.deploy-backend.outputs.version }})\", \"description\": \"The production happy-path failed AFTER deploying v${{ needs.deploy-backend.outputs.version }}. The service already migrated 100% traffic to the new revision — consider rolling back with scripts/rollback.sh.\n\n**Run:** #${{ github.run_number }}\n**Details:** ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}\", \"color\": 15158332}]}" \
|
||||
"${{ secrets.DISCORD_ALERT_WEBHOOK_URL }}" || echo "Discord notify failed (non-fatal)"
|
||||
|
||||
- name: Fail if canary failed
|
||||
if: steps.test_result.outputs.result == 'failure'
|
||||
run: |
|
||||
echo "::error::Post-deploy canary failed for v${{ needs.deploy-backend.outputs.version }}."
|
||||
echo "Roll back now with:"
|
||||
echo " scripts/rollback.sh --list # find the previous good revision + version"
|
||||
echo " scripts/rollback.sh --to-revision <PREV_REV> --image-version <PREV_VERSION>"
|
||||
exit 1
|
||||
|
||||
# NOTE: Pulumi infrastructure deploys are done locally, not in CI.
|
||||
# CI runners run on the infrastructure Pulumi manages, creating a circular
|
||||
# dependency. Always run `pulumi up` locally before merging infra PRs.
|
||||
|
||||
Reference in new issue
Block a user