fix: 16-tab burst tripped the Cloudflare rate limit — raise + scope it, survive edge blocks gracefully (#1033)

## Summary

Root-cause fix for Andrew's failed 16-tab reload test (2026-09-22
21:37Z), which the new degradation telemetry captured end-to-end
(`client_events`: banner_reconnecting ×6, banner_unavailable ×2 with
failed probes, waveform_slow with peaks-fallback).

**Diagnosis:** the origin was completely healthy during the test — zero
backend errors, 3 instances serving — but only ~207 API requests reached
it (16 tabs should have sent 2–3× that; only 9 `correction-data`
arrived). The missing traffic was blocked by **our own Cloudflare rate
limit (50 req/10s/IP)**: 16 tabs × ~10 API calls + CORS preflights ≈
hundreds of edge requests in seconds. CF block pages have no CORS
headers, so cross-origin fetches fail opaquely — producing "Access
denied / Failed to load job details" (blocked job fetch), "temporarily
unavailable" (blocked correction-data, 3 fast retries all inside the 10s
block), and outage banners (the `/api/health` probes were themselves
rate-limited, falsely "confirming" an outage).

## Changes

**Edge (Pulumi `infrastructure/`)**
- `RATE_LIMIT_REQUESTS` 50 → **200 per 10s/IP** — still flood control
(20 req/s sustained blocks), but a legitimate many-tab power user
passes.
- Rate-limit expression now **excludes OPTIONS preflights**
(double-counting; a blocked preflight kills the real request) and
**`/api/health`** (the frontend's connectivity probe must reflect origin
health — never edge-block state — or every tab shows the outage banner
for a healthy origin).

**Frontend**
- `429` treated as transient; retry honors `Retry-After` or waits out
the full 10s mitigation window (fast retries fail *and* prolong the
block).
- Network-failure retry backoff 0.6s/1.5s → **2s/9s**, so the final
attempt can land after a 10s block expires (blocked cross-origin
requests surface as network errors).
- Review-page access check: `BackendUnavailableError` now shows
**"Trouble reaching our servers" + Try again** instead of the misleading
**"Access denied"** — unreachability is not an authorization verdict.

## Testing

- Frontend: 2 new 429 tests (waits out window; honors Retry-After),
network-backoff tests updated to the new schedule with
probe-interference handled — **1309 passed**. Backend suite (untouched
code): **4518 passed**.
- Post-merge: Pulumi applies via CI (local `pulumi up` unavailable to
the agent — read-only ADC can't write Pulumi state; flagged below). Will
verify the ruleset values via the Cloudflare API afterwards and re-run
the review load test.

## ⚠️ Deploy note

`infrastructure/` changes normally get a local `pulumi up` before merge;
the agent's read-only ADC can't write Pulumi state, so this relies on
the CI apply on merge (idempotent single-ruleset update — low
partial-state risk). Andrew: if you prefer, run `pulumi up` locally
before merging.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Andrew Beveridge ☄️andClaude Fable 5 authored and GitHub committed 2026-09-22 17:58:39 -04:00
1 parent 758d74f44e
commit 925ec79c56
6 files changed
+135 -19

No files matched your search

+49 -7
View File
@@ -14,11 +14,12 @@ import {
STALL_UNAVAILABLE_MS,
} from '@/lib/backend-status'
function mockResponse(status: number, body: unknown = {}): Response {
function mockResponse(status: number, body: unknown = {}, headers: Record<string, string> = {}): Response {
return {
ok: status >= 200 && status < 300,
status,
json: async () => body,
headers: { get: (name: string) => headers[name.toLowerCase()] ?? null },
} as unknown as Response
}
@@ -67,18 +68,23 @@ describe('apiFetch', () => {
expect(getBackendStatus()).toBe('online')
})
it('retries a GET on repeated network failure, then throws BackendUnavailableError', async () => {
it('retries a GET on repeated network failure with spaced backoff, then throws BackendUnavailableError', async () => {
jest.useFakeTimers()
fetchMock.mockRejectedValue(new TypeError('Failed to fetch'))
// Network failures back off 2s then 9s: an edge rate-limit block surfaces
// as an opaque network error (no CORS on the block page) and lasts 10s —
// the final attempt must be able to land after the block expires.
const p = apiFetch('/api/jobs/abc')
const assertion = expect(p).rejects.toBeInstanceOf(BackendUnavailableError)
await jest.advanceTimersByTimeAsync(600)
await jest.advanceTimersByTimeAsync(1500)
await jest.advanceTimersByTimeAsync(2_000)
await jest.advanceTimersByTimeAsync(9_000)
await assertion
expect(fetchMock).toHaveBeenCalledTimes(3)
// A fast total failure (~2s) is NOT a stall, so the banner never appears.
// The 11s in-flight span crosses the stall threshold, so a health probe
// fires through the same fetch mock — count only the real API attempts.
const apiCalls = fetchMock.mock.calls.filter((c) => !String(c[0]).includes('/api/health'))
expect(apiCalls).toHaveLength(3)
expect(getBackendStatus()).toBe('online')
})
@@ -96,6 +102,42 @@ describe('apiFetch', () => {
expect(getBackendStatus()).toBe('online')
})
it('waits out the edge rate-limit window on 429, then retries and succeeds', async () => {
jest.useFakeTimers()
// Route by URL: the >10s in-flight span triggers a health probe through the
// same fetch mock, which must not consume the queued API responses.
const apiResponses = [mockResponse(429), mockResponse(200, { ok: true })]
fetchMock.mockImplementation((url: unknown) => {
if (String(url).includes('/api/health')) return Promise.resolve(mockResponse(200))
return Promise.resolve(apiResponses.shift() ?? mockResponse(200, { ok: true }))
})
const apiCalls = () => fetchMock.mock.calls.filter((c) => !String(c[0]).includes('/api/health')).length
const p = apiFetch('/api/jobs/abc')
// Must NOT retry inside the 10s mitigation window (that both fails and
// prolongs the block) — no second API attempt for the first 10s...
await jest.advanceTimersByTimeAsync(10_000)
expect(apiCalls()).toBe(1)
// ...then the retry fires just past the window.
await jest.advanceTimersByTimeAsync(1_000)
const res = await p
expect(res.status).toBe(200)
expect(apiCalls()).toBe(2)
})
it('honors Retry-After on 429 when the edge provides it', async () => {
jest.useFakeTimers()
fetchMock
.mockResolvedValueOnce(mockResponse(429, undefined, { 'retry-after': '3' }))
.mockResolvedValueOnce(mockResponse(200, { ok: true }))
const p = apiFetch('/api/jobs/abc')
await jest.advanceTimersByTimeAsync(3_600)
const res = await p
expect(res.status).toBe(200)
expect(fetchMock).toHaveBeenCalledTimes(2)
})
it('recovers transparently when a retried GET succeeds on a later attempt', async () => {
jest.useFakeTimers()
fetchMock
@@ -103,7 +145,7 @@ describe('apiFetch', () => {
.mockResolvedValueOnce(mockResponse(200, { ok: true }))
const p = apiFetch('/api/jobs/abc')
await jest.advanceTimersByTimeAsync(600)
await jest.advanceTimersByTimeAsync(2_000)
const res = await p
expect(res.status).toBe(200)
@@ -3,7 +3,7 @@
import { useEffect, useState, useCallback, useRef } from "react"
import { reportDegradationEvent } from "@/lib/degradation-events"
import { useAuth } from "@/lib/auth"
import { api, Job, createLyricsReviewApiClient, lyricsReviewApi } from "@/lib/api"
import { api, BackendUnavailableError, Job, createLyricsReviewApiClient, lyricsReviewApi } from "@/lib/api"
import { Spinner } from "@/components/ui/spinner"
import { Button } from "@/components/ui/button"
import { ArrowLeft, AlertCircle } from "lucide-react"
@@ -25,6 +25,7 @@ type AccessState =
| { status: "loading" }
| { status: "not_authenticated" }
| { status: "not_authorized"; reason: string }
| { status: "backend_unavailable" }
| { status: "job_not_found" }
| { status: "wrong_state"; currentState: string; expectedStates: string[] }
| { status: "invalid_route" }
@@ -87,6 +88,7 @@ export function JobRouterClient() {
const { user, isLoading: authLoading, hasHydrated } = useAuth()
const [accessState, setAccessState] = useState<AccessState>({ status: "loading" })
const [accessRetryNonce, setAccessRetryNonce] = useState(0)
useEffect(() => {
async function checkAccess() {
@@ -152,6 +154,13 @@ export function JobRouterClient() {
// Job not found or API error
if (error && typeof error === "object" && "status" in error && error.status === 404) {
setAccessState({ status: "job_not_found" })
} else if (error instanceof BackendUnavailableError) {
// The job fetch never got a real answer (origin blip, or the request
// was blocked at the edge — e.g. a many-tab burst tripping the rate
// limit). This is NOT an authorization verdict; showing "Access
// denied" here (as we used to) told users they lacked permission for
// their own jobs. Offer a retry instead.
setAccessState({ status: "backend_unavailable" })
} else {
setAccessState({
status: "not_authorized",
@@ -162,7 +171,7 @@ export function JobRouterClient() {
}
checkAccess()
}, [inLocalMode, jobId, routeType, user, authLoading, hasHydrated, isReplay])
}, [inLocalMode, jobId, routeType, user, authLoading, hasHydrated, isReplay, accessRetryNonce])
// Loading state
if (accessState.status === "loading") {
@@ -215,6 +224,32 @@ export function JobRouterClient() {
)
}
// Backend unreachable while checking access — a retryable condition, not a verdict
if (accessState.status === "backend_unavailable") {
return (
<div className="flex items-center justify-center min-h-screen">
<div className="text-center max-w-md p-6">
<AlertCircle className="w-12 h-12 mx-auto mb-4 text-amber-500" />
<h1 className="text-xl font-semibold mb-2">Trouble reaching our servers</h1>
<p className="text-muted-foreground mb-4">
This is usually temporary — your job is safe. Please try again in a moment.
</p>
<div className="flex items-center justify-center gap-3">
<Button onClick={() => { setAccessState({ status: "loading" }); setAccessRetryNonce((n) => n + 1) }}>
Try again
</Button>
<Button variant="outline" asChild>
<Link href="/app">
<ArrowLeft className="w-4 h-4 mr-2" />
Back to dashboard
</Link>
</Button>
</div>
</div>
</div>
)
}
// Not authorized
if (accessState.status === "not_authorized") {
return (
+25 -4
View File
@@ -516,9 +516,18 @@ const HARD_TIMEOUT_MS = 45_000;
const GET_MAX_ATTEMPTS = 3;
/** Backoff between GET retries (ms), indexed by prior-attempt number. */
const RETRY_BACKOFF_MS = [600, 1500];
/** Backoff for network-level failures (fetch rejected). A cross-origin request
* blocked by Cloudflare's rate limit surfaces as an opaque network error (its
* block page has no CORS headers), and the mitigation window is 10s — quick
* 0.6/1.5s retries all land inside the block AND add fuel to the count. Space
* the retries so the last attempt can land after a block window has expired. */
const NETWORK_RETRY_BACKOFF_MS = [2_000, 9_000];
/** Edge/origin status codes that mean "backend briefly unreachable", not a real
* application error: standard 502/503/504 plus Cloudflare's origin-reachability 52x. */
const TRANSIENT_STATUS = new Set([502, 503, 504, 520, 521, 522, 523, 524]);
* application error: standard 502/503/504 plus Cloudflare's origin-reachability
* 52x, and 429 (edge rate-limit block, when the response is readable). */
const TRANSIENT_STATUS = new Set([429, 502, 503, 504, 520, 521, 522, 523, 524]);
/** Wait before retrying a 429: at least the edge mitigation window. */
const RATE_LIMITED_RETRY_MS = 10_500;
const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));
@@ -658,7 +667,7 @@ export async function apiFetch(
if (callerSignal?.aborted) throw err;
lastError = err;
if (attempt < maxAttempts) {
await sleep(RETRY_BACKOFF_MS[Math.min(attempt - 1, RETRY_BACKOFF_MS.length - 1)]);
await sleep(NETWORK_RETRY_BACKOFF_MS[Math.min(attempt - 1, NETWORK_RETRY_BACKOFF_MS.length - 1)]);
continue;
}
throw new BackendUnavailableError(err);
@@ -667,7 +676,19 @@ export async function apiFetch(
if (TRANSIENT_STATUS.has(response.status)) {
if (allowRetry && attempt < maxAttempts) {
await sleep(RETRY_BACKOFF_MS[Math.min(attempt - 1, RETRY_BACKOFF_MS.length - 1)]);
if (response.status === 429) {
// Rate-limited at the edge: honor Retry-After when present, else
// wait out the full mitigation window — retrying sooner both fails
// and prolongs the block.
const retryAfterS = Number(response.headers.get('retry-after'));
await sleep(
Number.isFinite(retryAfterS) && retryAfterS > 0
? retryAfterS * 1000 + 500
: RATE_LIMITED_RETRY_MS,
);
} else {
await sleep(RETRY_BACKOFF_MS[Math.min(attempt - 1, RETRY_BACKOFF_MS.length - 1)]);
}
continue;
}
if (isGet) {
+11 -3
View File
@@ -182,9 +182,17 @@ class CloudflareConfig:
# Free plan constraints (enforced by the Cloudflare API): period must be 10s,
# and mitigation_timeout must equal the period (10s). A single flood control
# rule — the WAF path block (below) is the primary defense against the
# scanner class; this catches volumetric abuse. ~50 req / 10s = 5 req/s per
# IP, comfortably above legit page-load bursts to a control-plane API.
RATE_LIMIT_REQUESTS = 50
# scanner class; this catches volumetric abuse.
#
# Sized for the legitimate worst case from ONE IP (2026-09-22 incident): an
# admin reloading 16 review tabs at once fires ~10 API calls per tab within
# seconds. At the old 50/10s the edge blocked a chunk of them — and because
# CF block pages carry no CORS headers, the browser saw opaque network
# failures ("Access denied" / "temporarily unavailable" screens) while the
# origin was fully healthy. 200/10s (20 req/s/IP) still stops volumetric
# abuse; OPTIONS preflights and /api/health are excluded from counting (see
# create_rate_limit_ruleset).
RATE_LIMIT_REQUESTS = 200
RATE_LIMIT_PERIOD_SECONDS = 10
RATE_LIMIT_MITIGATION_SECONDS = 10
+12 -2
View File
@@ -250,10 +250,20 @@ def create_waf_rulesets(zone_id: str, hosts: list[str]) -> dict[str, cloudflare.
# --------------------------------------------------------------------------- #
def create_rate_limit_ruleset(zone_id: str, hosts: list[str]) -> cloudflare.Ruleset:
host_expr = _host_match_expression(hosts)
# Don't rate-limit OIDC-authed scheduler hits to /api/internal/*.
# Exclusions from rate-limit counting:
# - /api/internal/*: OIDC-authed scheduler hits.
# - OPTIONS: CORS preflights double every browser API call's edge count, and
# a blocked preflight kills the real request even when it would have
# passed. Preflights are cheap and carry no abuse value.
# - /api/health: the frontend's connectivity probe must reflect ORIGIN
# health. If probes get rate-limited alongside a burst, every open tab
# shows the outage banner for an origin that is perfectly healthy
# (2026-09-22 16-tab incident).
expression = (
f'{host_expr} and '
f'(not starts_with(http.request.uri.path, "{CloudflareConfig.INTERNAL_PATH_PREFIX}"))'
f'(not starts_with(http.request.uri.path, "{CloudflareConfig.INTERNAL_PATH_PREFIX}")) and '
f'(http.request.method ne "OPTIONS") and '
f'(http.request.uri.path ne "/api/health")'
)
return cloudflare.Ruleset(
"edge-ratelimit",
+1 -1
View File
@@ -1,6 +1,6 @@
[tool.poetry]
name = "karaoke-gen"
version = "0.232.1"
version = "0.233.0"
description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens."
authors = ["Andrew Beveridge <andrew@beveridge.uk>"]
license = "MIT"