mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
Section 1 of the 2026-11-03 #alerts triage handoff: a stale push on a
finished song got a 500, and the error monitor could stay silent on a
new crash.
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566980744448843886
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566980746394996879
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566980747905073224
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1565697725280948294
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1565803187334815775
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1565818418072916110
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566225593354420226
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566256141074169987
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567207141994135654
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567252495535841386
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567252497620537415
## What changed
**1a. start-review:** a job not in review gets 409
(`jobs.jobNotInReviewState`) with no transition attempted. Losing the
auto-approver race is a 409 too. `correction-data` has the same race,
and its `except Exception` turned it into a 500; it now returns 409
(`except HTTPException: raise` added). Its status gate still returns
400, as before.
**1b. Crash patterns:** `Exception in ASGI application` and
`request_audit_error` become `Unhandled exception: <Type: message>`,
read from the entry's `exception` field. The handoff's suspicion is
confirmed in `upsert_pattern`: an existing `new` pattern returns
`is_new=False`, so a second, different crash stayed silent. Both log
lines of one crash now share one pattern. The path and method go in the
sample only.
I read the line after the last frame rather than the traceback's last
line. pydantic's last line is a docs link shared by every failure, and
group tracebacks end in a separator.
**1c. One alert per failure:**
- YouTube ids are normalized out of signatures, which also fixes the
false "first time" line in `ops_alerts`.
- `Job <id> failed` (both call sites in `job_manager`) logs WARNING,
since the status write sends the job-failed alert.
- `URL download failed:` is ignored in the monitor, not demoted, because
`file_upload.py` belongs to the section 4 PR.
- `[job:<id>] Download failed` logs WARNING for the expected
`DownloadError`/`FlacfetchServiceError` types. This wasn't named in the
handoff, but it is the same duplicate on the audio-search path.
Unexpected exceptions there stay ERROR with a traceback.
**1d. Suppressions and demotions:**
- Zalo's `zaloJSV2 is not defined`: dropped at `/api/client-errors`,
only with a Zalo UA.
- `python -c` experiments on `audio-separation-job`: ignored, using the
handoff's regex.
- Spotify 403 on `/lyrics`: WARNING with `response.text[:200]`. 401 on
`/lyrics` and 401/403 on `/search` stay ERROR.
- Dropbox token probe: WARNING. A rejected refresh used to log nothing,
so it now logs ERROR, as does having no refresh credentials.
- decide-search health aborts: this was a small change. `should_ignore`
takes an optional `request_url` and matches the exact path
`/api/search/health`.
**1e. Cold start:** `anyio._backends._asyncio` is added to
`_LAZY_ROUTE_IMPORTS`. The prod stack from 10-30 confirms the stall was
in that import. Lifespan also sends one `/api/health` request through
the app in a worker thread, bounded by the existing 30s timeout and
best-effort.
## Testing
- TDD: new tests failed first, then passed.
- Affected suites: 826 passed.
- `backend/tests` (excluding `backend/tests/emulator`): 7088 passed, 8
failed, 29 skipped. All 8 are in `test_emulator_integration.py`, and
that file also shows 8 failures at the base commit a8f9618a. I did not
compare the 8 test names one by one.
- `tests/unit` outside the listed files was not run.
`test_audio_fetcher.py` and `test_remote_cli.py` fail to collect in my
environment; I didn't change either file.
- Read-only replay of real prod entries through the new code: the 11-03
crash gave one specific pattern with `POST
/api/jobs/7f89f884/start-review` in the sample, the decide-search health
aborts and the experiment traceback were ignored.
- Local CodeRabbit review completed with 0 findings.
## Not verified
- Nothing is live, so the alert volume and the cold-start stall haven't
been measured. The warm request has only run in tests, not on a real
Cloud Run cold start.
- A request's first-use imports beyond the anyio backend aren't
individually checked.
- My test environment has no `flacfetch`, so its lazy imports only log a
warning here.
- Zalo: the trigger is one report (11-03).
## Known leftovers
- `validate_state_transition` still logs ERROR before raising, so the
auto-approver race (and the existing in_review race) still emits one
`Invalid state transition` ERROR even though the client now gets a clean
response. The stale-push case no longer reaches it. I left this because
other callers rely on that log.
- A crash that also logs its own ERROR first still gives two patterns,
the ERROR and `Unhandled exception`.
- The first bot block after deploy shows "first time" once, because the
normalized signature is new.
- Old `Exception in ASGI application` and `request_audit_error` patterns
stop recurring and will auto-resolve.
- If the section 4 PR rewords `URL download failed:`, the ignore entry
in `known_issues.py` needs the same wording.
## For Andrew (infra and cost, not changed here)
- Cloud Run startup probe and min-instances for `karaoke-backend`, if
cold-start stalls continue after this ships.
- decide-search's min-0 is why its health pings abort. The suppression
hides those, but real search paths will still alert on the same abort.
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_012RYLVTvqXyL4QyA2EjzQMG
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Unit Tests for KaraokePrep
This directory contains unit tests for the KaraokePrep class.
Test Files
The tests are organized into the following files:
test_initialization.py: Tests for initialization and basic functionalitytest_file_operations.py: Tests for file operations (copy, download, convert)test_metadata.py: Tests for metadata extraction and parsingtest_lyrics.py: Tests for lyrics processingtest_audio.py: Tests for audio processing and separationtest_video.py: Tests for video creation and processingtest_async.py: Tests for async methods
Running the Tests
To run the tests, you need to have the test dependencies installed:
pip install -r requirements-test.txt
Then you can run the tests with:
python run_tests.py
This will run all the tests and generate a coverage report.
Checking Test Files
If you want to check if the test files are valid Python code without running the tests, you can use:
python check_tests.py
This will check if all the test files can be imported without errors.
Test Coverage
The tests aim to cover all the functionality of the KaraokePrep class. The coverage report will show you which parts of the code are covered by the tests.