Compare commits

...
Author SHA1 Message Date
Andrew BeveridgeandClaude Opus 4.5 c2c4c47bc3 docs: Add lessons learned for GCE encoding response format handling
- Add section about external service response format mismatches
- Document defensive type checking patterns for API integration
- Add recent changes entry for GCE encoding fixes and worker logs plan

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-04 19:28:59 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 44c44da95e fix: Handle GCE worker output_files as list of paths (#193)
## Summary
Fix the actual root cause of `'list' object has no attribute 'get'`
error in GCE encoding.

## Problem
The GCE encoding worker returns `output_files` as a **list of paths**:
```json
"output_files": [
    "jobs/501258e1/finals/output_4k_lossless.mp4",
    "jobs/501258e1/finals/output_4k_lossy.mp4",
    "jobs/501258e1/finals/output_720p.mp4"
]
```

But the code expected a **dict** with format keys:
```json
{"mp4_4k_lossless": "path/...", "mp4_4k_lossy": "path/...", ...}
```

This caused the error when calling `.get()` on the list.

## Solution
Added conversion logic to map the list of paths to the expected dict
format by matching filename patterns (e.g., `output_4k_lossless.mp4` ->
`mp4_4k_lossless`).

## Testing
- [x] Verified GCE worker response format
- [ ] Deploy and test with job 501258e1

## Notes
PR #191 fixed a similar issue in `wait_for_completion` where the entire
`status` response could be a list. This PR fixes a different issue where
`output_files` within the response is a list.

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-04 23:23:21 +00:00
Andrew Beveridge ☄️andClaude Opus 4.5 16dc451bdf docs: Add API auth guide and worker logs rearchitecture plan (#192)
## Summary
- Add API authentication documentation to CLAUDE.md for future agents
- Add comprehensive plan for rearchitecting worker logs storage

## Changes

### CLAUDE.md
- New "API Authentication (for Agents)" section covering:
  - Internal worker endpoints with X-Admin-Token
  - Firestore direct access
  - GCE encoding worker SSH commands
  - Common pitfalls to avoid

### .claude/plans/worker-logs-rearchitecture.plan.md
- Plan to move worker_logs from embedded array to Firestore
subcollection
- Fixes 1MB document size limit issue (job 501258e1 hit 1.26 MB)
- Includes TTL policy for automatic cleanup (30 days)

## Testing
- [x] Documentation review
- No code changes

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-04 22:37:17 +00:00
Andrew Beveridge ☄️andClaude Opus 4.5 278690b373 fix: Handle list response in encoding service wait_for_completion (#191)
## Summary
- Fix `'list' object has no attribute 'get'` error in GCE encoding
wait_for_completion
- The GCE encoding worker sometimes returns a list instead of a dict
from the /status endpoint
- PR #186 added handling in encode(), but same issue occurs in
wait_for_completion()

## Changes
- Added defensive handling to convert list responses to dicts in
wait_for_completion()

## Testing
- [x] Code review
- [ ] Deploy and test with job 501258e1

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-04 22:35:55 +00:00
7 changed files with 335 additions and 2 deletions

No files matched your search

@@ -0,0 +1,206 @@
# Plan: Worker Logs Rearchitecture
**Created:** 2026-01-04
**Branch:** TBD (use `/new-worktree worker-logs-subcollection`)
**Status:** Draft
## Overview
Move `worker_logs` from embedded array in job documents to a Firestore subcollection to avoid the 1MB document size limit. The current approach caused job 501258e1 to fail when logs reached 1.26 MB (98.6% of document size).
## Problem Statement
- Firestore documents have a 1 MB size limit
- `worker_logs` is stored as an array field within job documents
- Long-running jobs (especially video encoding) can generate thousands of log entries
- When the limit is exceeded, job updates fail with: `Document cannot be written because its size exceeds the maximum allowed size`
## Requirements
- [ ] Logs must remain associated with their job
- [ ] Logs should be queryable by job_id, worker type, timestamp
- [ ] Old logs should auto-cleanup (TTL: 30 days)
- [ ] Existing API endpoint (`GET /api/jobs/{id}/logs`) must continue to work
- [ ] Frontend must continue to display logs without changes
- [ ] Migration path for existing jobs (optional - can just start fresh)
## Technical Approach
### Option A: Subcollection (Recommended)
Store logs in `jobs/{job_id}/logs/{log_id}` subcollection.
**Pros:**
- Natural association with job (delete job = delete logs with cascading)
- Each log entry is its own document (no size limit concerns)
- Can add TTL via Firestore TTL policies
- Efficient queries with composite indexes
**Cons:**
- More Firestore read operations for paginated log fetching
- Slightly more complex write logic
### Option B: Separate Collection
Store logs in `job_logs` collection with `job_id` field.
**Pros:**
- Can query across all jobs
- Simpler TTL management
**Cons:**
- Orphaned logs if job deleted (need cleanup logic)
- Extra index needed for job_id queries
### Decision: Option A (Subcollection)
## Implementation Steps
### Phase 1: Backend Changes
1. [ ] **Create log entry model** - `backend/models/log.py`
- `LogEntry` dataclass with: id, job_id, timestamp, level, worker, message, metadata
- Keep backward compatible with existing log format
2. [ ] **Update FirestoreService** - `backend/services/firestore_service.py`
- Add `append_log_to_subcollection(job_id, log_entry)` method
- Add `get_logs_from_subcollection(job_id, limit, since_timestamp, worker)` method
- Keep old `append_worker_log` temporarily for migration period
3. [ ] **Update JobManager** - `backend/services/job_manager.py`
- Modify `add_worker_log()` to write to subcollection instead of array
- Modify `get_worker_logs()` to read from subcollection
- Add migration flag/env var to switch between old and new
4. [ ] **Update jobs API** - `backend/api/routes/jobs.py`
- `GET /api/jobs/{id}/logs` - Update to read from subcollection
- Response format stays the same (backward compatible)
5. [ ] **Add TTL configuration** - `infrastructure/__main__.py`
- Configure Firestore TTL policy for logs subcollection (30 days)
### Phase 2: Testing
6. [ ] **Unit tests** - `backend/tests/unit/test_log_subcollection.py`
- Test write to subcollection
- Test read with pagination
- Test filtering by worker type
- Test TTL field is set correctly
7. [ ] **Integration tests** - `backend/tests/emulator/test_worker_logs_subcollection.py`
- Test concurrent log writes from multiple workers
- Test large log volumes (1000+ entries)
- Test log retrieval performance
### Phase 3: Deployment & Migration
8. [ ] **Deploy with feature flag**
- Add `USE_LOG_SUBCOLLECTION=true` env var
- Deploy to production with flag OFF initially
9. [ ] **Enable for new jobs**
- Turn on flag for new jobs
- Monitor for issues
10. [ ] **Optional: Migrate existing jobs**
- Script to move existing `worker_logs` arrays to subcollections
- Can skip this - old jobs will use old format, new jobs use new
### Phase 4: Cleanup
11. [ ] **Remove old code path**
- Remove `worker_logs` field from job model
- Remove old `append_worker_log` method
- Remove feature flag
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/models/log.py` | Create | LogEntry dataclass |
| `backend/services/firestore_service.py` | Modify | Add subcollection methods |
| `backend/services/job_manager.py` | Modify | Switch to subcollection |
| `backend/api/routes/jobs.py` | Modify | Update logs endpoint |
| `backend/models/job.py` | Modify | Make worker_logs optional/deprecated |
| `backend/tests/unit/test_log_subcollection.py` | Create | Unit tests |
| `backend/tests/emulator/test_worker_logs_subcollection.py` | Create | Integration tests |
| `infrastructure/__main__.py` | Modify | Add TTL policy (if supported) |
## Log Entry Schema
```python
@dataclass
class LogEntry:
id: str # Auto-generated UUID
job_id: str
timestamp: datetime
level: str # "info", "warning", "error", "debug"
worker: str # "audio", "lyrics", "video", "render", "screens"
message: str
metadata: Optional[Dict[str, Any]] = None
ttl_expiry: datetime # For Firestore TTL (timestamp + 30 days)
```
## Firestore Structure
```
jobs/
{job_id}/
... (job fields, no more worker_logs array)
logs/ # subcollection
{log_id}/
timestamp: datetime
level: string
worker: string
message: string
metadata: map (optional)
ttl_expiry: datetime
```
## API Response (Unchanged)
```json
{
"logs": [
{
"timestamp": "2026-01-04T12:00:00Z",
"level": "info",
"worker": "video",
"message": "Starting video generation"
}
],
"total": 150,
"has_more": true
}
```
## Testing Strategy
- **Unit tests**: Mock Firestore, test business logic
- **Emulator tests**: Use Firestore emulator for integration tests
- **Load test**: Generate 5000+ logs for a single job, verify no size issues
- **Manual test**: Run a full job through the pipeline, verify logs appear
## Rollback Plan
1. Set `USE_LOG_SUBCOLLECTION=false` env var
2. Redeploy backend
3. New logs will go back to embedded array
4. Subcollection logs will remain but won't be read
## Open Questions
- [ ] Should we paginate logs in the API? (Currently returns all)
- [ ] What's the optimal TTL period? (30 days proposed)
- [ ] Should we index by timestamp for efficient range queries?
## Performance Considerations
- Reading logs: One query vs reading embedded array (slightly slower)
- Writing logs: Similar cost (one document write either way)
- Storage: Similar (array elements vs subcollection documents)
- Deletion: Subcollection deletes require batch delete or callable
## References
- [Firestore TTL](https://cloud.google.com/firestore/docs/ttl)
- [Firestore Subcollections](https://firebase.google.com/docs/firestore/data-model#subcollections)
- [Firestore Limits](https://firebase.google.com/docs/firestore/quotas#limits)
+46
View File
@@ -52,6 +52,52 @@ cd frontend && npm run test:all 2>&1 | tail -n 200
- `gcloud` CLI for reading/debugging only
- Stop and notify user on auth issues
## API Authentication (for Agents)
When you need to call backend APIs programmatically:
### Internal Worker Endpoints
Use the `X-Admin-Token` header with the token from Secret Manager:
```bash
# Trigger video worker
curl -X POST "https://api.nomadkaraoke.com/api/internal/workers/video" \
-H "X-Admin-Token: $(gcloud secrets versions access latest --secret=admin-tokens --project=nomadkaraoke)" \
-H "Content-Type: application/json" \
-d '{"job_id": "YOUR_JOB_ID"}'
# Other internal endpoints follow the same pattern
```
### Firestore Direct Access
```bash
# Set the correct project
export GOOGLE_CLOUD_PROJECT=nomadkaraoke
# Use Python with google-cloud-firestore
python3 << 'EOF'
import os
os.environ['GOOGLE_CLOUD_PROJECT'] = 'nomadkaraoke'
from google.cloud import firestore
db = firestore.Client(project='nomadkaraoke')
# Query/update documents...
EOF
```
### GCE Encoding Worker
```bash
# SSH to restart service (clears in-memory job queue)
gcloud compute ssh encoding-worker --zone=us-central1-a --project=nomadkaraoke \
--command="sudo systemctl restart encoding-worker"
# Check health
gcloud compute ssh encoding-worker --zone=us-central1-a --project=nomadkaraoke \
--command="curl -s http://localhost:8080/health"
```
### What Doesn't Work
- `gcloud auth print-identity-token` - Wrong token type for internal endpoints
- Cloud Tasks without proper service account - Gets 401 from Cloud Run
## Documentation Maintenance
### Structure
+23 -1
View File
@@ -349,7 +349,29 @@ class GCEEncodingBackend(EncodingBackend):
if not isinstance(result, dict):
self.logger.error(f"Unexpected GCE result type: {type(result)}")
result = {}
output_files = result.get("output_files", {})
raw_output_files = result.get("output_files", {})
# Convert output_files from list of paths to dict if needed
# GCE worker returns list like: ["path/output_4k_lossless.mp4", "path/output_720p.mp4"]
# We need dict like: {"mp4_4k_lossless": "path/...", "mp4_720p": "path/..."}
if isinstance(raw_output_files, list):
output_files = {}
for path in raw_output_files:
if not isinstance(path, str):
continue
filename = path.split("/")[-1] if "/" in path else path
# Map filename patterns to output format keys
if "4k_lossless" in filename:
output_files["mp4_4k_lossless"] = path
elif "4k_lossy" in filename:
output_files["mp4_4k_lossy"] = path
elif "720p" in filename:
output_files["mp4_720p"] = path
elif filename.endswith(".mkv"):
output_files["mkv_4k"] = path
self.logger.info(f"Converted output_files list to dict: {output_files}")
else:
output_files = raw_output_files if isinstance(raw_output_files, dict) else {}
return EncodingOutput(
success=True,
+9
View File
@@ -177,6 +177,15 @@ class EncodingService:
raise TimeoutError(f"Encoding job {job_id} timed out after {timeout}s")
status = await self.get_job_status(job_id)
# Handle case where GCE worker returns a list instead of dict
if isinstance(status, list):
logger.warning(f"[job:{job_id}] GCE returned list instead of dict: {status}")
status = status[0] if status and isinstance(status[0], dict) else {}
if not isinstance(status, dict):
logger.error(f"[job:{job_id}] Unexpected status type: {type(status)}")
status = {}
job_status = status.get("status", "unknown")
progress = status.get("progress", 0)
+48
View File
@@ -1038,6 +1038,54 @@ result = await orchestrator.run()
See `docs/archive/2026-01-04-video-worker-orchestrator-refactor.md` for full details.
### External Service Response Format Mismatches
**Problem**: The GCE encoding worker returned responses in unexpected formats, causing `'list' object has no attribute 'get'` errors that were hard to debug through the async pipeline.
**Issues encountered**:
1. **Status endpoint returning list instead of dict**: The `/status/{job_id}` endpoint sometimes returned a list `[{...}]` instead of a dict `{...}`. Calling `.get()` on the list caused failures.
2. **output_files as list of paths instead of dict**: The worker returned:
```json
"output_files": ["jobs/id/finals/output_4k_lossless.mp4", "jobs/id/finals/output_720p.mp4"]
```
But the backend expected:
```json
"output_files": {"mp4_4k_lossless": "path/...", "mp4_720p": "path/..."}
```
**Symptoms**:
- Job fails late in pipeline (during encoding completion handling)
- Error message is cryptic (`'list' object has no attribute 'get'`)
- Hard to reproduce locally (depends on GCE worker state)
- Multiple similar errors from different root causes
**Solution**: Add defensive type checking for all external service responses:
```python
# Always check response type before accessing
if isinstance(status, list):
logger.warning(f"GCE returned list instead of dict: {status}")
status = status[0] if status and isinstance(status[0], dict) else {}
if not isinstance(status, dict):
status = {}
# Convert list of paths to expected dict format
if isinstance(output_files, list):
output_dict = {}
for path in output_files:
filename = path.split("/")[-1]
if "4k_lossless" in filename:
output_dict["mp4_4k_lossless"] = path
# ... etc
output_files = output_dict
```
**Key insight**: When integrating with external services, never assume the response format. Add explicit type checks and conversions, with logging when unexpected formats are encountered. This is especially important for services you don't control (internal microservices, third-party APIs).
**Debugging tip**: When hitting format errors in async pipelines, check the raw response from the external service directly (curl with API key) before assuming the bug is in your code.
## What We'd Do Differently
1. **Add Pydantic model fields test first** - Would have caught the silent field issue immediately
+2
View File
@@ -36,6 +36,8 @@
## Recent Changes
- **GCE Encoding Response Fixes** (2026-01-04): Fixed multiple response format mismatches with GCE encoding worker. The worker returns `output_files` as a list of paths, not a dict with format keys - added conversion logic. Also added defensive type checking for status responses that could be lists. Created worker logs rearchitecture plan to address Firestore 1MB document limit (logs will move to subcollection). See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#external-service-response-format-mismatches).
- **Video Worker Orchestrator** (2026-01-04): Major refactor to unify video generation pipeline. Created VideoWorkerOrchestrator that coordinates all stages (packaging, encoding, distribution, notifications) regardless of encoding backend (GCE or local). Fixes issue where GCE encoding path bypassed YouTube upload, Discord notifications, and CDG/TXT packaging. Feature flag `USE_NEW_ORCHESTRATOR` (default: true) enables rollback. 139 new tests across 6 new service modules. See [ARCHITECTURE.md](ARCHITECTURE.md#video-worker-orchestrator) and [archive/2026-01-04-video-worker-orchestrator-refactor.md](archive/2026-01-04-video-worker-orchestrator-refactor.md).
- **Full Unicode Font Support** (2026-01-03): Fixed rendering of musical symbols (♪) and added comprehensive international font support to Docker base image. Installed Noto fonts covering Latin, CJK (Chinese/Japanese/Korean), Arabic, Hebrew, Thai, and other scripts. Changed default karaoke font from Arial to Noto Sans. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#fonts-in-docker-for-video-rendering).
+1 -1
View File
@@ -1,6 +1,6 @@
[tool.poetry]
name = "karaoke-gen"
version = "0.89.4"
version = "0.89.6"
description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens."
authors = ["Andrew Beveridge <andrew@beveridge.uk>"]
license = "MIT"