Files
karaoke-gen/.claude/plans/worker-logs-rearchitecture.plan.md
Andrew Beveridge ☄️andClaude ae85ec0a09 feat: Move worker logs to Firestore subcollection to avoid 1MB limit (#195)
## Summary

Move `worker_logs` from embedded array in job documents to a Firestore
subcollection (`jobs/{job_id}/logs`) to avoid the 1MB document size
limit. This fixes job failures when logs exceed the limit during
long-running operations like video encoding.

**Problem:** Job `501258e1` failed when logs reached 1.26MB (2,547 log
entries), representing 98.6% of the maximum document size.

## Changes

### New Files
- `backend/models/worker_log.py` - WorkerLogEntry dataclass with factory
methods, serialization (to_dict, to_legacy_dict, from_dict)
- `backend/tests/test_worker_log_subcollection.py` - 24 unit tests
- `backend/tests/emulator/test_worker_logs_subcollection.py` - 11
integration tests using Firestore emulator

### Modified Files
- `backend/services/firestore_service.py` - Added subcollection methods
(append, get, count, delete)
- `backend/services/job_manager.py` - Use subcollection with fallback to
legacy embedded arrays
- `backend/api/routes/jobs.py` - Updated logs endpoint to use new
methods
- `backend/config.py` - Added `USE_LOG_SUBCOLLECTION` feature flag
(default: true)
- `infrastructure/__main__.py` - Added TTL policy (30 days) and
composite index for logs subcollection
- `pyproject.toml` - Version bump 0.89.6 → 0.90.0

### Documentation
- `docs/README.md` - Added to Recent Changes
- `docs/ARCHITECTURE.md` - Added logs subcollection to Firestore
Collections table
- `docs/LESSONS-LEARNED.md` - Added "Firestore Document 1MB Limit with
Embedded Arrays" section

## Testing

- [x] 24 unit tests pass
(`backend/tests/test_worker_log_subcollection.py`)
- [x] 11 emulator integration tests pass
(`backend/tests/emulator/test_worker_logs_subcollection.py`)
- [x] All 68 existing emulator tests pass

## Review

- [x] CodeRabbit CLI review completed locally
- [x] Review feedback addressed (timestamp formatting fix)

## Rollback Plan

Set `USE_LOG_SUBCOLLECTION=false` environment variable and redeploy. New
logs will go back to embedded array; subcollection logs will remain but
won't be read.

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-04 20:14:25 -05:00

7.7 KiB

Plan: Worker Logs Rearchitecture

Created: 2026-01-04 Branch: feat/sess-20260104-1931-logs-research Status: Implementation Complete - Pending PR Merge & Deployment

Overview

Move worker_logs from embedded array in job documents to a Firestore subcollection to avoid the 1MB document size limit. The current approach caused job 501258e1 to fail when logs reached 1.26 MB (98.6% of document size).

Problem Statement

  • Firestore documents have a 1 MB size limit
  • worker_logs is stored as an array field within job documents
  • Long-running jobs (especially video encoding) can generate thousands of log entries
  • When the limit is exceeded, job updates fail with: Document cannot be written because its size exceeds the maximum allowed size

Requirements

  • Logs must remain associated with their job
  • Logs should be queryable by job_id, worker type, timestamp
  • Old logs should auto-cleanup (TTL: 30 days)
  • Existing API endpoint (GET /api/jobs/{id}/logs) must continue to work
  • Frontend must continue to display logs without changes
  • Migration path for existing jobs (optional - can just start fresh)

Technical Approach

Store logs in jobs/{job_id}/logs/{log_id} subcollection.

Pros:

  • Natural association with job (delete job = delete logs with cascading)
  • Each log entry is its own document (no size limit concerns)
  • Can add TTL via Firestore TTL policies
  • Efficient queries with composite indexes

Cons:

  • More Firestore read operations for paginated log fetching
  • Slightly more complex write logic

Option B: Separate Collection

Store logs in job_logs collection with job_id field.

Pros:

  • Can query across all jobs
  • Simpler TTL management

Cons:

  • Orphaned logs if job deleted (need cleanup logic)
  • Extra index needed for job_id queries

Decision: Option A (Subcollection)

Implementation Steps

Phase 1: Backend Changes

  1. Create log entry model - backend/models/worker_log.py

    • WorkerLogEntry dataclass with: id, job_id, timestamp, level, worker, message, metadata, ttl_expiry
    • Factory method create() for easy instantiation
    • to_dict() for Firestore storage
    • to_legacy_dict() for API compatibility
    • from_dict() for Firestore retrieval
  2. Update FirestoreService - backend/services/firestore_service.py

    • Added append_log_to_subcollection(job_id, log_entry) method
    • Added get_logs_from_subcollection(job_id, limit, since_timestamp, worker, offset) method
    • Added get_logs_count_from_subcollection(job_id) method
    • Added delete_logs_subcollection(job_id, batch_size) method
    • Old append_worker_log marked as deprecated
  3. Update JobManager - backend/services/job_manager.py

    • Modified append_worker_log() to use subcollection when USE_LOG_SUBCOLLECTION=true
    • Modified get_worker_logs() to read from subcollection with fallback to embedded array
    • Added get_worker_logs_count() for total count
    • Updated delete_job() to clean up logs subcollection
  4. Update jobs API - backend/api/routes/jobs.py

    • GET /api/jobs/{id}/logs - Updated to use new methods
    • Response format stays the same (backward compatible)
    • Fixed next_index calculation
  5. Add feature flag - backend/config.py

    • Added USE_LOG_SUBCOLLECTION environment variable (default: true)
  6. Add TTL configuration - infrastructure/__main__.py

    • Added Firestore TTL policy for logs subcollection (30 days)
    • Added composite index for worker + timestamp queries

Phase 2: Testing

  1. Unit tests - backend/tests/test_worker_log_subcollection.py

    • 24 tests covering:
      • WorkerLogEntry model creation and conversion
      • FirestoreService subcollection methods
      • JobManager logging with feature flag
      • Edge cases (unicode, long messages, empty messages)
  2. Integration tests - backend/tests/emulator/test_worker_logs_subcollection.py

    • 11 tests using real Firestore emulator:
      • Single and sequential writes
      • TTL field verification
      • Concurrent writes (no race conditions)
      • Large volume (500 logs)
      • Worker filtering
      • Timestamp ordering
      • Subcollection deletion
      • Concurrent workers interleaved
      • Job deletion cleanup
      • Comparison with embedded array

Phase 3: Deployment & Migration

  1. Deploy with feature flag

    • Add USE_LOG_SUBCOLLECTION=true env var (default is true)
    • Deploy to production
  2. Monitor

    • Watch for errors in Cloud Logging
    • Verify logs appear in frontend
  3. Backward compatibility

    • Old jobs with embedded arrays still work (fallback in get_worker_logs)
    • No migration needed - new jobs use subcollection, old jobs use array

Phase 4: Cleanup (Future)

  1. Remove old code path (after confidence period)
    • Remove worker_logs field from job model
    • Remove old append_worker_log method
    • Remove feature flag

Files Created/Modified

File Action Description
backend/models/worker_log.py Created WorkerLogEntry dataclass
backend/services/firestore_service.py Modified Added subcollection methods
backend/services/job_manager.py Modified Switch to subcollection with fallback
backend/api/routes/jobs.py Modified Updated logs endpoint
backend/config.py Modified Added USE_LOG_SUBCOLLECTION setting
backend/tests/test_worker_log_subcollection.py Created Unit tests (24 tests)
backend/tests/emulator/test_worker_logs_subcollection.py Created Integration tests (11 tests)
infrastructure/__main__.py Modified Added TTL policy and index

Log Entry Schema

@dataclass
class WorkerLogEntry:
    timestamp: datetime
    level: str  # "DEBUG", "INFO", "WARNING", "ERROR"
    worker: str  # "audio", "lyrics", "video", "render", "screens", "distribution"
    message: str
    id: str  # Auto-generated UUID
    job_id: str
    ttl_expiry: datetime  # timestamp + 30 days
    metadata: Optional[Dict[str, Any]] = None

Firestore Structure

jobs/
  {job_id}/
    ... (job fields, worker_logs still present for old jobs)
    logs/  # subcollection for new jobs
      {log_id}/
        id: string
        job_id: string
        timestamp: datetime
        level: string
        worker: string
        message: string
        metadata: map (optional)
        ttl_expiry: datetime

API Response (Unchanged)

{
  "logs": [
    {
      "timestamp": "2026-01-04T12:00:00Z",
      "level": "INFO",
      "worker": "video",
      "message": "Starting video generation"
    }
  ],
  "next_index": 42,
  "total_logs": 150
}

Testing Results

All tests pass:

  • 24 unit tests in test_worker_log_subcollection.py
  • 11 emulator integration tests in test_worker_logs_subcollection.py
  • 68 total tests in emulator suite pass

Rollback Plan

  1. Set USE_LOG_SUBCOLLECTION=false env var
  2. Redeploy backend
  3. New logs will go back to embedded array
  4. Subcollection logs will remain but won't be read

Answers to Open Questions

  • Should we paginate logs in the API? - Yes, added offset parameter support
  • What's the optimal TTL period? - 30 days (configurable via DEFAULT_LOG_TTL_DAYS)
  • Should we index by timestamp for efficient range queries? - Yes, added composite index (worker + timestamp)

References