Files
Andrew Beveridge ☄️andClaude Opus 5.5 5c41be1d3b fix(error-monitor): ignore runner-manager's other Cloud Run abort wording (#1252)
## Summary
On 2026-11-04 at 10:00Z a PR opened on python-audio-separator queued a
burst of `workflow_job` webhooks. 22 of 54 deliveries got a 500 from
Cloud Run while `github-runner-manager` scaled up. The error monitor
alerted because Cloud Run used a second wording, "aborted after waiting
too long for an available instance", which the existing ignore pattern
didn't match.

There was no impact: all self-hosted jobs in the affected runs got
runners within about 3.5 minutes and passed.

## Changes
- **Error monitor:** match both Cloud Run abort wordings for
`github-runner-manager`, but only on **webhook** requests (the bare
function URL). Cloud Scheduler ticks (`?action=check_idle`) and entries
with no URL still alert. `_SERVICE_URL_IGNORE_PATTERNS` now accepts a
predicate on the split URL as well as an exact path.
- **Runner manager (Cloud Function):** the ignore relies on the 5-minute
stalled-job tick catching dropped webhooks, so that tick's own failures
now log as structured `severity: ERROR` and page. That covers a
redispatch crash, a repo skipped in the scan, a job given up at 60 min,
and a failed re-dispatch. Before this, they were severity-less prints
that nothing alerted on. Both changes came from CodeRabbit feedback.
- Corrected the pattern's reason: GitHub did not redeliver these
(`redelivery: false`).
- Version 0.292.1; merged latest main.

## ⚠️ Deploy
`infrastructure/functions/runner_manager/` only goes live with a local
`pulumi up` (CI never runs Pulumi). Apply it **before** merging, so the
safety net's alerting is live before the webhook-abort ignore ships with
the backend deploy.

## Testing
- `tests/unit/services/error_monitor/`: 452 passed (webhook,
scheduler-tick, no-URL and other-service cases for both wordings).
- `infrastructure/functions/runner_manager`: 71 passed. The 4 updated
tests fail against main's `ephemeral.py`.
- Full `make test` not run locally; relying on CI.

Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567479055887114363

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVVTKcWCcDArTQrjS82

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* Runner-manager webhook errors are filtered more precisely. Recognized
abort messages are ignored only for matching webhook requests;
scheduler-tick errors, requests with other URLs, and requests without a
URL continue to trigger alerts.
* Runner-manager job-processing failures are recorded as structured
error logs, including stalled-job scans and dispatch failures. Give-up
logs now report the configured limit rather than the job’s current time
in the queue.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 19:58:46 -05:00
..