Compare commits

...
Author SHA1 Message Date
Andrew BeveridgeandClaude Opus 4.5 c2c4c47bc3 docs: Add lessons learned for GCE encoding response format handling
- Add section about external service response format mismatches
- Document defensive type checking patterns for API integration
- Add recent changes entry for GCE encoding fixes and worker logs plan

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-04 19:28:59 -05:00
2 changed files with 50 additions and 0 deletions

No files matched your search

+48
View File
@@ -1038,6 +1038,54 @@ result = await orchestrator.run()
See `docs/archive/2026-01-04-video-worker-orchestrator-refactor.md` for full details.
### External Service Response Format Mismatches
**Problem**: The GCE encoding worker returned responses in unexpected formats, causing `'list' object has no attribute 'get'` errors that were hard to debug through the async pipeline.
**Issues encountered**:
1. **Status endpoint returning list instead of dict**: The `/status/{job_id}` endpoint sometimes returned a list `[{...}]` instead of a dict `{...}`. Calling `.get()` on the list caused failures.
2. **output_files as list of paths instead of dict**: The worker returned:
```json
"output_files": ["jobs/id/finals/output_4k_lossless.mp4", "jobs/id/finals/output_720p.mp4"]
```
But the backend expected:
```json
"output_files": {"mp4_4k_lossless": "path/...", "mp4_720p": "path/..."}
```
**Symptoms**:
- Job fails late in pipeline (during encoding completion handling)
- Error message is cryptic (`'list' object has no attribute 'get'`)
- Hard to reproduce locally (depends on GCE worker state)
- Multiple similar errors from different root causes
**Solution**: Add defensive type checking for all external service responses:
```python
# Always check response type before accessing
if isinstance(status, list):
logger.warning(f"GCE returned list instead of dict: {status}")
status = status[0] if status and isinstance(status[0], dict) else {}
if not isinstance(status, dict):
status = {}
# Convert list of paths to expected dict format
if isinstance(output_files, list):
output_dict = {}
for path in output_files:
filename = path.split("/")[-1]
if "4k_lossless" in filename:
output_dict["mp4_4k_lossless"] = path
# ... etc
output_files = output_dict
```
**Key insight**: When integrating with external services, never assume the response format. Add explicit type checks and conversions, with logging when unexpected formats are encountered. This is especially important for services you don't control (internal microservices, third-party APIs).
**Debugging tip**: When hitting format errors in async pipelines, check the raw response from the external service directly (curl with API key) before assuming the bug is in your code.
## What We'd Do Differently
1. **Add Pydantic model fields test first** - Would have caught the silent field issue immediately
+2
View File
@@ -36,6 +36,8 @@
## Recent Changes
- **GCE Encoding Response Fixes** (2026-01-04): Fixed multiple response format mismatches with GCE encoding worker. The worker returns `output_files` as a list of paths, not a dict with format keys - added conversion logic. Also added defensive type checking for status responses that could be lists. Created worker logs rearchitecture plan to address Firestore 1MB document limit (logs will move to subcollection). See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#external-service-response-format-mismatches).
- **Video Worker Orchestrator** (2026-01-04): Major refactor to unify video generation pipeline. Created VideoWorkerOrchestrator that coordinates all stages (packaging, encoding, distribution, notifications) regardless of encoding backend (GCE or local). Fixes issue where GCE encoding path bypassed YouTube upload, Discord notifications, and CDG/TXT packaging. Feature flag `USE_NEW_ORCHESTRATOR` (default: true) enables rollback. 139 new tests across 6 new service modules. See [ARCHITECTURE.md](ARCHITECTURE.md#video-worker-orchestrator) and [archive/2026-01-04-video-worker-orchestrator-refactor.md](archive/2026-01-04-video-worker-orchestrator-refactor.md).
- **Full Unicode Font Support** (2026-01-03): Fixed rendering of musical symbols (♪) and added comprehensive international font support to Docker base image. Installed Noto fonts covering Latin, CJK (Chinese/Japanese/Korean), Arabic, Hebrew, Thai, and other scripts. Changed default karaoke font from Arial to Noto Sans. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#fonts-in-docker-for-video-rendering).