mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
feat: Move worker logs to Firestore subcollection to avoid 1MB limit (#195)
## Summary
Move `worker_logs` from embedded array in job documents to a Firestore
subcollection (`jobs/{job_id}/logs`) to avoid the 1MB document size
limit. This fixes job failures when logs exceed the limit during
long-running operations like video encoding.
**Problem:** Job `501258e1` failed when logs reached 1.26MB (2,547 log
entries), representing 98.6% of the maximum document size.
## Changes
### New Files
- `backend/models/worker_log.py` - WorkerLogEntry dataclass with factory
methods, serialization (to_dict, to_legacy_dict, from_dict)
- `backend/tests/test_worker_log_subcollection.py` - 24 unit tests
- `backend/tests/emulator/test_worker_logs_subcollection.py` - 11
integration tests using Firestore emulator
### Modified Files
- `backend/services/firestore_service.py` - Added subcollection methods
(append, get, count, delete)
- `backend/services/job_manager.py` - Use subcollection with fallback to
legacy embedded arrays
- `backend/api/routes/jobs.py` - Updated logs endpoint to use new
methods
- `backend/config.py` - Added `USE_LOG_SUBCOLLECTION` feature flag
(default: true)
- `infrastructure/__main__.py` - Added TTL policy (30 days) and
composite index for logs subcollection
- `pyproject.toml` - Version bump 0.89.6 → 0.90.0
### Documentation
- `docs/README.md` - Added to Recent Changes
- `docs/ARCHITECTURE.md` - Added logs subcollection to Firestore
Collections table
- `docs/LESSONS-LEARNED.md` - Added "Firestore Document 1MB Limit with
Embedded Arrays" section
## Testing
- [x] 24 unit tests pass
(`backend/tests/test_worker_log_subcollection.py`)
- [x] 11 emulator integration tests pass
(`backend/tests/emulator/test_worker_logs_subcollection.py`)
- [x] All 68 existing emulator tests pass
## Review
- [x] CodeRabbit CLI review completed locally
- [x] Review feedback addressed (timestamp formatting fix)
## Rollback Plan
Set `USE_LOG_SUBCOLLECTION=false` environment variable and redeploy. New
logs will go back to embedded array; subcollection logs will remain but
won't be read.
@coderabbitai ignore
---
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
1 parent
a2282357da
commit
ae85ec0a09
13 files changed
+1565
-114
No files matched your search
@@ -1,8 +1,8 @@
|
||||
# Plan: Worker Logs Rearchitecture
|
||||
|
||||
**Created:** 2026-01-04
|
||||
**Branch:** TBD (use `/new-worktree worker-logs-subcollection`)
|
||||
**Status:** Draft
|
||||
**Branch:** `feat/sess-20260104-1931-logs-research`
|
||||
**Status:** Implementation Complete - Pending PR Merge & Deployment
|
||||
|
||||
## Overview
|
||||
|
||||
@@ -17,12 +17,12 @@ Move `worker_logs` from embedded array in job documents to a Firestore subcollec
|
||||
|
||||
## Requirements
|
||||
|
||||
- [ ] Logs must remain associated with their job
|
||||
- [ ] Logs should be queryable by job_id, worker type, timestamp
|
||||
- [ ] Old logs should auto-cleanup (TTL: 30 days)
|
||||
- [ ] Existing API endpoint (`GET /api/jobs/{id}/logs`) must continue to work
|
||||
- [ ] Frontend must continue to display logs without changes
|
||||
- [ ] Migration path for existing jobs (optional - can just start fresh)
|
||||
- [x] Logs must remain associated with their job
|
||||
- [x] Logs should be queryable by job_id, worker type, timestamp
|
||||
- [x] Old logs should auto-cleanup (TTL: 30 days)
|
||||
- [x] Existing API endpoint (`GET /api/jobs/{id}/logs`) must continue to work
|
||||
- [x] Frontend must continue to display logs without changes
|
||||
- [x] Migration path for existing jobs (optional - can just start fresh)
|
||||
|
||||
## Technical Approach
|
||||
|
||||
@@ -56,87 +56,107 @@ Store logs in `job_logs` collection with `job_id` field.
|
||||
|
||||
### Phase 1: Backend Changes
|
||||
|
||||
1. [ ] **Create log entry model** - `backend/models/log.py`
|
||||
- `LogEntry` dataclass with: id, job_id, timestamp, level, worker, message, metadata
|
||||
- Keep backward compatible with existing log format
|
||||
1. [x] **Create log entry model** - `backend/models/worker_log.py`
|
||||
- `WorkerLogEntry` dataclass with: id, job_id, timestamp, level, worker, message, metadata, ttl_expiry
|
||||
- Factory method `create()` for easy instantiation
|
||||
- `to_dict()` for Firestore storage
|
||||
- `to_legacy_dict()` for API compatibility
|
||||
- `from_dict()` for Firestore retrieval
|
||||
|
||||
2. [ ] **Update FirestoreService** - `backend/services/firestore_service.py`
|
||||
- Add `append_log_to_subcollection(job_id, log_entry)` method
|
||||
- Add `get_logs_from_subcollection(job_id, limit, since_timestamp, worker)` method
|
||||
- Keep old `append_worker_log` temporarily for migration period
|
||||
2. [x] **Update FirestoreService** - `backend/services/firestore_service.py`
|
||||
- Added `append_log_to_subcollection(job_id, log_entry)` method
|
||||
- Added `get_logs_from_subcollection(job_id, limit, since_timestamp, worker, offset)` method
|
||||
- Added `get_logs_count_from_subcollection(job_id)` method
|
||||
- Added `delete_logs_subcollection(job_id, batch_size)` method
|
||||
- Old `append_worker_log` marked as deprecated
|
||||
|
||||
3. [ ] **Update JobManager** - `backend/services/job_manager.py`
|
||||
- Modify `add_worker_log()` to write to subcollection instead of array
|
||||
- Modify `get_worker_logs()` to read from subcollection
|
||||
- Add migration flag/env var to switch between old and new
|
||||
3. [x] **Update JobManager** - `backend/services/job_manager.py`
|
||||
- Modified `append_worker_log()` to use subcollection when `USE_LOG_SUBCOLLECTION=true`
|
||||
- Modified `get_worker_logs()` to read from subcollection with fallback to embedded array
|
||||
- Added `get_worker_logs_count()` for total count
|
||||
- Updated `delete_job()` to clean up logs subcollection
|
||||
|
||||
4. [ ] **Update jobs API** - `backend/api/routes/jobs.py`
|
||||
- `GET /api/jobs/{id}/logs` - Update to read from subcollection
|
||||
4. [x] **Update jobs API** - `backend/api/routes/jobs.py`
|
||||
- `GET /api/jobs/{id}/logs` - Updated to use new methods
|
||||
- Response format stays the same (backward compatible)
|
||||
- Fixed `next_index` calculation
|
||||
|
||||
5. [ ] **Add TTL configuration** - `infrastructure/__main__.py`
|
||||
- Configure Firestore TTL policy for logs subcollection (30 days)
|
||||
5. [x] **Add feature flag** - `backend/config.py`
|
||||
- Added `USE_LOG_SUBCOLLECTION` environment variable (default: true)
|
||||
|
||||
6. [x] **Add TTL configuration** - `infrastructure/__main__.py`
|
||||
- Added Firestore TTL policy for logs subcollection (30 days)
|
||||
- Added composite index for worker + timestamp queries
|
||||
|
||||
### Phase 2: Testing
|
||||
|
||||
6. [ ] **Unit tests** - `backend/tests/unit/test_log_subcollection.py`
|
||||
- Test write to subcollection
|
||||
- Test read with pagination
|
||||
- Test filtering by worker type
|
||||
- Test TTL field is set correctly
|
||||
7. [x] **Unit tests** - `backend/tests/test_worker_log_subcollection.py`
|
||||
- 24 tests covering:
|
||||
- WorkerLogEntry model creation and conversion
|
||||
- FirestoreService subcollection methods
|
||||
- JobManager logging with feature flag
|
||||
- Edge cases (unicode, long messages, empty messages)
|
||||
|
||||
7. [ ] **Integration tests** - `backend/tests/emulator/test_worker_logs_subcollection.py`
|
||||
- Test concurrent log writes from multiple workers
|
||||
- Test large log volumes (1000+ entries)
|
||||
- Test log retrieval performance
|
||||
8. [x] **Integration tests** - `backend/tests/emulator/test_worker_logs_subcollection.py`
|
||||
- 11 tests using real Firestore emulator:
|
||||
- Single and sequential writes
|
||||
- TTL field verification
|
||||
- Concurrent writes (no race conditions)
|
||||
- Large volume (500 logs)
|
||||
- Worker filtering
|
||||
- Timestamp ordering
|
||||
- Subcollection deletion
|
||||
- Concurrent workers interleaved
|
||||
- Job deletion cleanup
|
||||
- Comparison with embedded array
|
||||
|
||||
### Phase 3: Deployment & Migration
|
||||
|
||||
8. [ ] **Deploy with feature flag**
|
||||
- Add `USE_LOG_SUBCOLLECTION=true` env var
|
||||
- Deploy to production with flag OFF initially
|
||||
9. [ ] **Deploy with feature flag**
|
||||
- Add `USE_LOG_SUBCOLLECTION=true` env var (default is true)
|
||||
- Deploy to production
|
||||
|
||||
9. [ ] **Enable for new jobs**
|
||||
- Turn on flag for new jobs
|
||||
- Monitor for issues
|
||||
10. [ ] **Monitor**
|
||||
- Watch for errors in Cloud Logging
|
||||
- Verify logs appear in frontend
|
||||
|
||||
10. [ ] **Optional: Migrate existing jobs**
|
||||
- Script to move existing `worker_logs` arrays to subcollections
|
||||
- Can skip this - old jobs will use old format, new jobs use new
|
||||
11. [x] **Backward compatibility**
|
||||
- Old jobs with embedded arrays still work (fallback in get_worker_logs)
|
||||
- No migration needed - new jobs use subcollection, old jobs use array
|
||||
|
||||
### Phase 4: Cleanup
|
||||
### Phase 4: Cleanup (Future)
|
||||
|
||||
11. [ ] **Remove old code path**
|
||||
12. [ ] **Remove old code path** (after confidence period)
|
||||
- Remove `worker_logs` field from job model
|
||||
- Remove old `append_worker_log` method
|
||||
- Remove feature flag
|
||||
|
||||
## Files to Create/Modify
|
||||
## Files Created/Modified
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/models/log.py` | Create | LogEntry dataclass |
|
||||
| `backend/services/firestore_service.py` | Modify | Add subcollection methods |
|
||||
| `backend/services/job_manager.py` | Modify | Switch to subcollection |
|
||||
| `backend/api/routes/jobs.py` | Modify | Update logs endpoint |
|
||||
| `backend/models/job.py` | Modify | Make worker_logs optional/deprecated |
|
||||
| `backend/tests/unit/test_log_subcollection.py` | Create | Unit tests |
|
||||
| `backend/tests/emulator/test_worker_logs_subcollection.py` | Create | Integration tests |
|
||||
| `infrastructure/__main__.py` | Modify | Add TTL policy (if supported) |
|
||||
| `backend/models/worker_log.py` | Created | WorkerLogEntry dataclass |
|
||||
| `backend/services/firestore_service.py` | Modified | Added subcollection methods |
|
||||
| `backend/services/job_manager.py` | Modified | Switch to subcollection with fallback |
|
||||
| `backend/api/routes/jobs.py` | Modified | Updated logs endpoint |
|
||||
| `backend/config.py` | Modified | Added USE_LOG_SUBCOLLECTION setting |
|
||||
| `backend/tests/test_worker_log_subcollection.py` | Created | Unit tests (24 tests) |
|
||||
| `backend/tests/emulator/test_worker_logs_subcollection.py` | Created | Integration tests (11 tests) |
|
||||
| `infrastructure/__main__.py` | Modified | Added TTL policy and index |
|
||||
|
||||
## Log Entry Schema
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class LogEntry:
|
||||
class WorkerLogEntry:
|
||||
timestamp: datetime
|
||||
level: str # "DEBUG", "INFO", "WARNING", "ERROR"
|
||||
worker: str # "audio", "lyrics", "video", "render", "screens", "distribution"
|
||||
message: str
|
||||
id: str # Auto-generated UUID
|
||||
job_id: str
|
||||
timestamp: datetime
|
||||
level: str # "info", "warning", "error", "debug"
|
||||
worker: str # "audio", "lyrics", "video", "render", "screens"
|
||||
message: str
|
||||
ttl_expiry: datetime # timestamp + 30 days
|
||||
metadata: Optional[Dict[str, Any]] = None
|
||||
ttl_expiry: datetime # For Firestore TTL (timestamp + 30 days)
|
||||
```
|
||||
|
||||
## Firestore Structure
|
||||
@@ -144,9 +164,11 @@ class LogEntry:
|
||||
```
|
||||
jobs/
|
||||
{job_id}/
|
||||
... (job fields, no more worker_logs array)
|
||||
logs/ # subcollection
|
||||
... (job fields, worker_logs still present for old jobs)
|
||||
logs/ # subcollection for new jobs
|
||||
{log_id}/
|
||||
id: string
|
||||
job_id: string
|
||||
timestamp: datetime
|
||||
level: string
|
||||
worker: string
|
||||
@@ -162,22 +184,22 @@ jobs/
|
||||
"logs": [
|
||||
{
|
||||
"timestamp": "2026-01-04T12:00:00Z",
|
||||
"level": "info",
|
||||
"level": "INFO",
|
||||
"worker": "video",
|
||||
"message": "Starting video generation"
|
||||
}
|
||||
],
|
||||
"total": 150,
|
||||
"has_more": true
|
||||
"next_index": 42,
|
||||
"total_logs": 150
|
||||
}
|
||||
```
|
||||
|
||||
## Testing Strategy
|
||||
## Testing Results
|
||||
|
||||
- **Unit tests**: Mock Firestore, test business logic
|
||||
- **Emulator tests**: Use Firestore emulator for integration tests
|
||||
- **Load test**: Generate 5000+ logs for a single job, verify no size issues
|
||||
- **Manual test**: Run a full job through the pipeline, verify logs appear
|
||||
All tests pass:
|
||||
- 24 unit tests in `test_worker_log_subcollection.py`
|
||||
- 11 emulator integration tests in `test_worker_logs_subcollection.py`
|
||||
- 68 total tests in emulator suite pass
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
@@ -186,21 +208,15 @@ jobs/
|
||||
3. New logs will go back to embedded array
|
||||
4. Subcollection logs will remain but won't be read
|
||||
|
||||
## Open Questions
|
||||
## Answers to Open Questions
|
||||
|
||||
- [ ] Should we paginate logs in the API? (Currently returns all)
|
||||
- [ ] What's the optimal TTL period? (30 days proposed)
|
||||
- [ ] Should we index by timestamp for efficient range queries?
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
- Reading logs: One query vs reading embedded array (slightly slower)
|
||||
- Writing logs: Similar cost (one document write either way)
|
||||
- Storage: Similar (array elements vs subcollection documents)
|
||||
- Deletion: Subcollection deletes require batch delete or callable
|
||||
- [x] **Should we paginate logs in the API?** - Yes, added `offset` parameter support
|
||||
- [x] **What's the optimal TTL period?** - 30 days (configurable via DEFAULT_LOG_TTL_DAYS)
|
||||
- [x] **Should we index by timestamp for efficient range queries?** - Yes, added composite index (worker + timestamp)
|
||||
|
||||
## References
|
||||
|
||||
- [Firestore TTL](https://cloud.google.com/firestore/docs/ttl)
|
||||
- [Firestore Subcollections](https://firebase.google.com/docs/firestore/data-model#subcollections)
|
||||
- [Firestore Limits](https://firebase.google.com/docs/firestore/quotas#limits)
|
||||
- [Pulumi Firestore Field](https://www.pulumi.com/registry/packages/gcp/api-docs/firestore/field/)
|
||||
@@ -1452,10 +1452,14 @@ async def get_worker_logs(
|
||||
This endpoint returns worker logs stored in Firestore.
|
||||
Use `since_index` for efficient polling (returns only new logs).
|
||||
|
||||
Logs are stored in a subcollection (jobs/{job_id}/logs) to avoid
|
||||
the 1MB document size limit. Older jobs may have logs in an embedded
|
||||
array (worker_logs field) - this endpoint handles both transparently.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
since_index: Return only logs after this index (for pagination/polling)
|
||||
worker: Filter by worker name (audio, lyrics, screens, video, render)
|
||||
worker: Filter by worker name (audio, lyrics, screens, video, render, distribution)
|
||||
|
||||
Returns:
|
||||
{
|
||||
@@ -1473,11 +1477,11 @@ async def get_worker_logs(
|
||||
raise HTTPException(status_code=403, detail="You don't have permission to access logs for this job")
|
||||
|
||||
logs = job_manager.get_worker_logs(job_id, since_index=since_index, worker=worker)
|
||||
total = len(job.worker_logs) if job.worker_logs else 0
|
||||
total = job_manager.get_worker_logs_count(job_id)
|
||||
|
||||
return {
|
||||
"logs": logs,
|
||||
"next_index": total,
|
||||
"next_index": since_index + len(logs),
|
||||
"total_logs": total
|
||||
}
|
||||
|
||||
|
||||
@@ -71,6 +71,12 @@ class Settings(BaseSettings):
|
||||
|
||||
# Storage paths
|
||||
temp_dir: str = os.getenv("TEMP_DIR", "/tmp/karaoke-gen")
|
||||
|
||||
# Worker logs storage mode
|
||||
# When enabled, worker logs are stored in a Firestore subcollection (jobs/{job_id}/logs)
|
||||
# instead of an embedded array. This avoids the 1MB document size limit.
|
||||
# Default is true for new deployments.
|
||||
use_log_subcollection: bool = os.getenv("USE_LOG_SUBCOLLECTION", "true").lower() in ("true", "1", "yes")
|
||||
|
||||
# Flacfetch remote service (for torrent downloads)
|
||||
# When configured, audio search uses the remote flacfetch HTTP API instead of local flacfetch.
|
||||
|
||||
@@ -0,0 +1,164 @@
|
||||
"""
|
||||
Worker log entry model for subcollection storage.
|
||||
|
||||
This module defines the LogEntry model for storing worker logs in a Firestore
|
||||
subcollection instead of an embedded array. This avoids the 1MB document size
|
||||
limit that caused job 501258e1 to fail when logs reached 1.26 MB.
|
||||
"""
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime, timezone, timedelta
|
||||
from typing import Optional, Dict, Any
|
||||
import uuid
|
||||
|
||||
|
||||
# Default TTL for log entries (30 days)
|
||||
DEFAULT_LOG_TTL_DAYS = 30
|
||||
|
||||
|
||||
@dataclass
|
||||
class WorkerLogEntry:
|
||||
"""
|
||||
Worker log entry for Firestore subcollection storage.
|
||||
|
||||
Stored at: jobs/{job_id}/logs/{log_id}
|
||||
|
||||
This model is separate from the legacy LogEntry (in job.py) which is
|
||||
stored as an embedded array in the job document. This new model supports
|
||||
the subcollection approach with TTL and richer metadata.
|
||||
"""
|
||||
# Core fields (required)
|
||||
timestamp: datetime
|
||||
level: str # "DEBUG", "INFO", "WARNING", "ERROR"
|
||||
worker: str # "audio", "lyrics", "screens", "video", "render", "distribution"
|
||||
message: str
|
||||
|
||||
# Identifiers
|
||||
id: str = field(default_factory=lambda: str(uuid.uuid4()))
|
||||
job_id: str = "" # Set when writing to Firestore
|
||||
|
||||
# TTL for automatic cleanup
|
||||
ttl_expiry: datetime = field(
|
||||
default_factory=lambda: datetime.now(timezone.utc) + timedelta(days=DEFAULT_LOG_TTL_DAYS)
|
||||
)
|
||||
|
||||
# Optional metadata for debugging
|
||||
metadata: Optional[Dict[str, Any]] = None
|
||||
|
||||
@classmethod
|
||||
def create(
|
||||
cls,
|
||||
job_id: str,
|
||||
worker: str,
|
||||
level: str,
|
||||
message: str,
|
||||
metadata: Optional[Dict[str, Any]] = None,
|
||||
ttl_days: int = DEFAULT_LOG_TTL_DAYS
|
||||
) -> "WorkerLogEntry":
|
||||
"""
|
||||
Factory method to create a new log entry.
|
||||
|
||||
Args:
|
||||
job_id: Job ID this log belongs to
|
||||
worker: Worker name (audio, lyrics, screens, video, render, distribution)
|
||||
level: Log level (DEBUG, INFO, WARNING, ERROR)
|
||||
message: Log message (truncated to 1000 chars)
|
||||
metadata: Optional additional metadata
|
||||
ttl_days: Days until log expires (default 30)
|
||||
|
||||
Returns:
|
||||
New WorkerLogEntry instance
|
||||
"""
|
||||
now = datetime.now(timezone.utc)
|
||||
return cls(
|
||||
id=str(uuid.uuid4()),
|
||||
job_id=job_id,
|
||||
timestamp=now,
|
||||
level=level.upper(),
|
||||
worker=worker,
|
||||
message=message[:1000], # Truncate long messages
|
||||
metadata=metadata,
|
||||
ttl_expiry=now + timedelta(days=ttl_days)
|
||||
)
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Convert to dictionary for Firestore storage.
|
||||
|
||||
Returns:
|
||||
Dictionary representation for Firestore
|
||||
"""
|
||||
result = {
|
||||
"id": self.id,
|
||||
"job_id": self.job_id,
|
||||
"timestamp": self.timestamp,
|
||||
"level": self.level,
|
||||
"worker": self.worker,
|
||||
"message": self.message,
|
||||
"ttl_expiry": self.ttl_expiry
|
||||
}
|
||||
if self.metadata:
|
||||
result["metadata"] = self.metadata
|
||||
return result
|
||||
|
||||
def to_legacy_dict(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Convert to legacy format for backward compatibility with API responses.
|
||||
|
||||
The existing API returns logs in this format:
|
||||
{"timestamp": "...", "level": "INFO", "worker": "audio", "message": "..."}
|
||||
|
||||
Returns:
|
||||
Dictionary in legacy format
|
||||
"""
|
||||
# Format timestamp: use "Z" suffix for UTC, isoformat() for others
|
||||
if self.timestamp.tzinfo is None:
|
||||
# Naive datetime - use as-is
|
||||
timestamp_str = self.timestamp.isoformat()
|
||||
elif self.timestamp.utcoffset() == timedelta(0):
|
||||
# UTC datetime - replace +00:00 with Z for cleaner format
|
||||
timestamp_str = self.timestamp.replace(tzinfo=None).isoformat() + "Z"
|
||||
else:
|
||||
# Non-UTC timezone-aware - use full isoformat with offset
|
||||
timestamp_str = self.timestamp.isoformat()
|
||||
|
||||
return {
|
||||
"timestamp": timestamp_str,
|
||||
"level": self.level,
|
||||
"worker": self.worker,
|
||||
"message": self.message
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, data: Dict[str, Any]) -> "WorkerLogEntry":
|
||||
"""
|
||||
Create from Firestore document data.
|
||||
|
||||
Args:
|
||||
data: Dictionary from Firestore document
|
||||
|
||||
Returns:
|
||||
WorkerLogEntry instance
|
||||
"""
|
||||
# Handle timestamp conversion (Firestore returns datetime objects)
|
||||
timestamp = data.get("timestamp")
|
||||
if isinstance(timestamp, str):
|
||||
# Parse ISO format string
|
||||
timestamp = datetime.fromisoformat(timestamp.rstrip("Z")).replace(tzinfo=timezone.utc)
|
||||
|
||||
ttl_expiry = data.get("ttl_expiry")
|
||||
if isinstance(ttl_expiry, str):
|
||||
ttl_expiry = datetime.fromisoformat(ttl_expiry.rstrip("Z")).replace(tzinfo=timezone.utc)
|
||||
elif ttl_expiry is None:
|
||||
# Default TTL if not set
|
||||
ttl_expiry = datetime.now(timezone.utc) + timedelta(days=DEFAULT_LOG_TTL_DAYS)
|
||||
|
||||
return cls(
|
||||
id=data.get("id", str(uuid.uuid4())),
|
||||
job_id=data.get("job_id", ""),
|
||||
timestamp=timestamp or datetime.now(timezone.utc),
|
||||
level=data.get("level", "INFO"),
|
||||
worker=data.get("worker", "unknown"),
|
||||
message=data.get("message", ""),
|
||||
metadata=data.get("metadata"),
|
||||
ttl_expiry=ttl_expiry
|
||||
)
|
||||
@@ -9,6 +9,7 @@ from google.cloud.firestore_v1 import FieldFilter
|
||||
|
||||
from backend.config import settings
|
||||
from backend.models.job import Job, JobStatus, TimelineEvent
|
||||
from backend.models.worker_log import WorkerLogEntry
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -245,10 +246,13 @@ class FirestoreService:
|
||||
def append_worker_log(self, job_id: str, log_entry: Dict[str, Any]) -> None:
|
||||
"""
|
||||
Atomically append a log entry to worker_logs using ArrayUnion.
|
||||
|
||||
|
||||
This avoids the race condition of read-modify-write when multiple
|
||||
workers are logging concurrently.
|
||||
|
||||
|
||||
DEPRECATED: Use append_log_to_subcollection() instead to avoid
|
||||
the 1MB document size limit.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
log_entry: Log entry dict with timestamp, level, worker, message
|
||||
@@ -263,6 +267,165 @@ class FirestoreService:
|
||||
except Exception as e:
|
||||
# Log but don't raise - logging shouldn't break workers
|
||||
logger.debug(f"Error appending worker log for job {job_id}: {e}")
|
||||
|
||||
# ============================================
|
||||
# Worker Log Subcollection Methods
|
||||
# ============================================
|
||||
# These methods store logs in a subcollection (jobs/{job_id}/logs)
|
||||
# instead of an embedded array to avoid the 1MB document size limit.
|
||||
|
||||
def append_log_to_subcollection(self, job_id: str, log_entry: WorkerLogEntry) -> None:
|
||||
"""
|
||||
Append a log entry to the logs subcollection.
|
||||
|
||||
Stores logs at: jobs/{job_id}/logs/{log_id}
|
||||
|
||||
This approach avoids the 1MB document size limit by storing each
|
||||
log entry as a separate document in a subcollection.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
log_entry: WorkerLogEntry instance
|
||||
"""
|
||||
try:
|
||||
# Ensure job_id is set on the log entry
|
||||
log_entry.job_id = job_id
|
||||
|
||||
# Get subcollection reference
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
|
||||
# Add document with auto-generated ID or use log_entry.id
|
||||
doc_ref = logs_ref.document(log_entry.id)
|
||||
doc_ref.set(log_entry.to_dict())
|
||||
|
||||
# Don't log every append - too spammy
|
||||
except Exception as e:
|
||||
# Log but don't raise - logging shouldn't break workers
|
||||
logger.debug(f"Error appending log to subcollection for job {job_id}: {e}")
|
||||
|
||||
def get_logs_from_subcollection(
|
||||
self,
|
||||
job_id: str,
|
||||
limit: int = 500,
|
||||
since_timestamp: Optional[datetime] = None,
|
||||
worker: Optional[str] = None,
|
||||
offset: int = 0
|
||||
) -> List[WorkerLogEntry]:
|
||||
"""
|
||||
Get log entries from the logs subcollection.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
limit: Maximum number of logs to return (default 500)
|
||||
since_timestamp: Return only logs after this timestamp
|
||||
worker: Filter by worker name (optional)
|
||||
offset: Number of logs to skip (for pagination)
|
||||
|
||||
Returns:
|
||||
List of WorkerLogEntry instances, ordered by timestamp ascending
|
||||
"""
|
||||
try:
|
||||
# Get subcollection reference
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
|
||||
# Build query
|
||||
query = logs_ref.order_by("timestamp", direction=firestore.Query.ASCENDING)
|
||||
|
||||
if since_timestamp:
|
||||
query = query.where(filter=FieldFilter("timestamp", ">", since_timestamp))
|
||||
|
||||
if worker:
|
||||
query = query.where(filter=FieldFilter("worker", "==", worker))
|
||||
|
||||
# Apply offset and limit
|
||||
if offset > 0:
|
||||
query = query.offset(offset)
|
||||
|
||||
query = query.limit(limit)
|
||||
|
||||
# Execute query
|
||||
docs = query.stream()
|
||||
logs = [WorkerLogEntry.from_dict(doc.to_dict()) for doc in docs]
|
||||
|
||||
return logs
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error getting logs from subcollection for job {job_id}: {e}")
|
||||
return []
|
||||
|
||||
def get_logs_count_from_subcollection(self, job_id: str) -> int:
|
||||
"""
|
||||
Get the total count of log entries in the subcollection.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
|
||||
Returns:
|
||||
Total count of log entries
|
||||
"""
|
||||
try:
|
||||
# Get subcollection reference
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
|
||||
# Use aggregation query for efficient counting
|
||||
count_query = logs_ref.count()
|
||||
result = count_query.get()
|
||||
|
||||
# Result is a list of AggregationResult, we want the first one's count
|
||||
if result and len(result) > 0:
|
||||
return result[0][0].value
|
||||
return 0
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error counting logs for job {job_id}: {e}")
|
||||
return 0
|
||||
|
||||
def delete_logs_subcollection(self, job_id: str, batch_size: int = 500) -> int:
|
||||
"""
|
||||
Delete all log entries in the logs subcollection.
|
||||
|
||||
This is used when deleting a job to clean up its logs.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
batch_size: Number of documents to delete per batch
|
||||
|
||||
Returns:
|
||||
Number of logs deleted
|
||||
"""
|
||||
try:
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
deleted_count = 0
|
||||
|
||||
while True:
|
||||
# Get a batch of documents
|
||||
docs = logs_ref.limit(batch_size).stream()
|
||||
deleted_in_batch = 0
|
||||
|
||||
# Delete in a batch
|
||||
batch = self.db.batch()
|
||||
for doc in docs:
|
||||
batch.delete(doc.reference)
|
||||
deleted_in_batch += 1
|
||||
|
||||
if deleted_in_batch == 0:
|
||||
break
|
||||
|
||||
batch.commit()
|
||||
deleted_count += deleted_in_batch
|
||||
|
||||
# If we deleted less than batch_size, we're done
|
||||
if deleted_in_batch < batch_size:
|
||||
break
|
||||
|
||||
if deleted_count > 0:
|
||||
logger.info(f"Deleted {deleted_count} logs for job {job_id}")
|
||||
|
||||
return deleted_count
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error deleting logs subcollection for job {job_id}: {e}")
|
||||
return 0
|
||||
|
||||
# ============================================
|
||||
# Token Management Methods
|
||||
|
||||
@@ -13,7 +13,9 @@ import uuid
|
||||
from datetime import datetime
|
||||
from typing import Optional, Dict, Any, List
|
||||
|
||||
from backend.config import settings
|
||||
from backend.models.job import Job, JobStatus, JobCreate, STATE_TRANSITIONS
|
||||
from backend.models.worker_log import WorkerLogEntry
|
||||
from backend.services.firestore_service import FirestoreService
|
||||
from backend.services.storage_service import StorageService
|
||||
|
||||
@@ -224,7 +226,7 @@ class JobManager:
|
||||
return urls
|
||||
|
||||
def delete_job(self, job_id: str, delete_files: bool = True) -> None:
|
||||
"""Delete a job and optionally its files."""
|
||||
"""Delete a job, its files, and its logs subcollection."""
|
||||
if delete_files:
|
||||
job = self.get_job(job_id)
|
||||
if job and job.output_files:
|
||||
@@ -233,7 +235,15 @@ class JobManager:
|
||||
self.storage.delete_file(gcs_path)
|
||||
except Exception as e:
|
||||
logger.error(f"Error deleting file {gcs_path}: {e}")
|
||||
|
||||
|
||||
# Delete logs subcollection first (must be done before deleting parent doc)
|
||||
try:
|
||||
deleted_logs = self.firestore.delete_logs_subcollection(job_id)
|
||||
if deleted_logs > 0:
|
||||
logger.info(f"Deleted {deleted_logs} log entries for job {job_id}")
|
||||
except Exception as e:
|
||||
logger.warning(f"Error deleting logs subcollection for job {job_id}: {e}")
|
||||
|
||||
self.firestore.delete_job(job_id)
|
||||
logger.info(f"Deleted job {job_id}")
|
||||
|
||||
@@ -579,33 +589,45 @@ class JobManager:
|
||||
worker: str,
|
||||
level: str,
|
||||
message: str,
|
||||
max_logs: int = 500
|
||||
max_logs: int = 500,
|
||||
metadata: Optional[Dict[str, Any]] = None
|
||||
) -> None:
|
||||
"""
|
||||
Append a log entry to the job's worker_logs atomically.
|
||||
|
||||
Uses Firestore ArrayUnion for atomic append to avoid race conditions
|
||||
when multiple workers (audio + lyrics) are logging concurrently.
|
||||
|
||||
Note: max_logs is not enforced per-append (would require transactions).
|
||||
Logs may grow beyond max_logs; trim periodically if needed.
|
||||
|
||||
Append a log entry to the job's logs.
|
||||
|
||||
By default (USE_LOG_SUBCOLLECTION=true), logs are stored in a Firestore
|
||||
subcollection (jobs/{job_id}/logs) to avoid the 1MB document size limit.
|
||||
|
||||
For backward compatibility (USE_LOG_SUBCOLLECTION=false), logs are stored
|
||||
in the embedded worker_logs array using atomic ArrayUnion.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
worker: Worker name (audio, lyrics, screens, video, render)
|
||||
worker: Worker name (audio, lyrics, screens, video, render, distribution)
|
||||
level: Log level (DEBUG, INFO, WARNING, ERROR)
|
||||
message: Log message
|
||||
max_logs: Not used (kept for API compatibility)
|
||||
metadata: Optional additional metadata dict
|
||||
"""
|
||||
log_entry = {
|
||||
'timestamp': datetime.utcnow().isoformat() + 'Z',
|
||||
'level': level,
|
||||
'worker': worker,
|
||||
'message': message[:1000] # Truncate long messages
|
||||
}
|
||||
|
||||
# Use atomic ArrayUnion to avoid race conditions
|
||||
self.firestore.append_worker_log(job_id, log_entry)
|
||||
if settings.use_log_subcollection:
|
||||
# New subcollection approach - avoids 1MB limit
|
||||
log_entry = WorkerLogEntry.create(
|
||||
job_id=job_id,
|
||||
worker=worker,
|
||||
level=level,
|
||||
message=message,
|
||||
metadata=metadata
|
||||
)
|
||||
self.firestore.append_log_to_subcollection(job_id, log_entry)
|
||||
else:
|
||||
# Legacy embedded array approach
|
||||
log_entry = {
|
||||
'timestamp': datetime.utcnow().isoformat() + 'Z',
|
||||
'level': level,
|
||||
'worker': worker,
|
||||
'message': message[:1000] # Truncate long messages
|
||||
}
|
||||
self.firestore.append_worker_log(job_id, log_entry)
|
||||
|
||||
def get_worker_logs(
|
||||
self,
|
||||
@@ -615,28 +637,68 @@ class JobManager:
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Get worker logs for a job, optionally filtered by worker and index.
|
||||
|
||||
|
||||
By default (USE_LOG_SUBCOLLECTION=true), logs are read from the
|
||||
subcollection. Falls back to embedded array for older jobs.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
since_index: Return only logs after this index (for pagination)
|
||||
worker: Filter by worker name (optional)
|
||||
|
||||
|
||||
Returns:
|
||||
List of log entries as dicts
|
||||
List of log entries as dicts (in legacy format for API compatibility)
|
||||
"""
|
||||
if settings.use_log_subcollection:
|
||||
# Try subcollection first
|
||||
subcollection_logs = self.firestore.get_logs_from_subcollection(
|
||||
job_id=job_id,
|
||||
offset=since_index,
|
||||
worker=worker,
|
||||
limit=500
|
||||
)
|
||||
if subcollection_logs:
|
||||
# Convert to legacy format for API compatibility
|
||||
return [log.to_legacy_dict() for log in subcollection_logs]
|
||||
# Fall through to check embedded array for older jobs
|
||||
|
||||
# Embedded array approach (legacy jobs or fallback)
|
||||
job = self.get_job(job_id)
|
||||
if not job or not job.worker_logs:
|
||||
return []
|
||||
|
||||
|
||||
logs = [log.dict() if hasattr(log, 'dict') else log for log in job.worker_logs]
|
||||
|
||||
|
||||
# Filter by index
|
||||
if since_index > 0:
|
||||
logs = logs[since_index:]
|
||||
|
||||
|
||||
# Filter by worker
|
||||
if worker:
|
||||
logs = [log for log in logs if log.get('worker') == worker]
|
||||
|
||||
|
||||
return logs
|
||||
|
||||
def get_worker_logs_count(self, job_id: str) -> int:
|
||||
"""
|
||||
Get the total count of worker logs for a job.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
|
||||
Returns:
|
||||
Total count of logs
|
||||
"""
|
||||
if settings.use_log_subcollection:
|
||||
# Try subcollection first
|
||||
count = self.firestore.get_logs_count_from_subcollection(job_id)
|
||||
if count > 0:
|
||||
return count
|
||||
# Fall through to check embedded array
|
||||
|
||||
# Embedded array (legacy jobs or fallback)
|
||||
job = self.get_job(job_id)
|
||||
if not job or not job.worker_logs:
|
||||
return 0
|
||||
return len(job.worker_logs)
|
||||
|
||||
@@ -0,0 +1,443 @@
|
||||
"""
|
||||
Integration tests for worker log subcollection storage.
|
||||
|
||||
Tests the subcollection approach using real Firestore emulator to verify:
|
||||
- Logs are stored in subcollection (jobs/{job_id}/logs)
|
||||
- TTL expiry field is set correctly
|
||||
- Large log volumes work without 1MB limit issues
|
||||
- Concurrent writes are handled correctly
|
||||
- Logs can be queried efficiently
|
||||
|
||||
Run with: ./scripts/run-emulator-tests.sh
|
||||
"""
|
||||
import pytest
|
||||
import time
|
||||
import requests
|
||||
import os
|
||||
import threading
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
from datetime import datetime, timezone, timedelta
|
||||
|
||||
|
||||
def emulators_running() -> bool:
|
||||
"""Check if GCP emulators are running."""
|
||||
try:
|
||||
requests.get("http://127.0.0.1:8080", timeout=1)
|
||||
return True
|
||||
except (requests.exceptions.ConnectionError, requests.exceptions.Timeout):
|
||||
return False
|
||||
|
||||
|
||||
# Skip all tests in this module if emulators aren't running
|
||||
pytestmark = pytest.mark.skipif(
|
||||
not emulators_running(),
|
||||
reason="GCP emulators not running. Start with: scripts/start-emulators.sh"
|
||||
)
|
||||
|
||||
# Set up environment for emulator
|
||||
os.environ["FIRESTORE_EMULATOR_HOST"] = "127.0.0.1:8080"
|
||||
os.environ["GOOGLE_CLOUD_PROJECT"] = "test-project"
|
||||
|
||||
|
||||
class TestWorkerLogsSubcollectionDirect:
|
||||
"""Direct Firestore tests for worker logs subcollection."""
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def setup_firestore(self):
|
||||
"""Set up Firestore client for each test."""
|
||||
from google.cloud import firestore
|
||||
self.db = firestore.Client(project="test-project")
|
||||
self.collection = "test-subcollection-jobs"
|
||||
yield
|
||||
|
||||
def _create_test_job(self):
|
||||
"""Create a test job document."""
|
||||
job_id = f"test-sub-{int(time.time() * 1000)}"
|
||||
doc_ref = self.db.collection(self.collection).document(job_id)
|
||||
doc_ref.set({
|
||||
"job_id": job_id,
|
||||
"status": "pending",
|
||||
"created_at": datetime.now(timezone.utc),
|
||||
"worker_logs": [] # Legacy field - kept empty for new jobs
|
||||
})
|
||||
return job_id
|
||||
|
||||
def _append_log_to_subcollection(self, job_id: str, worker: str, message: str, ttl_days: int = 30):
|
||||
"""Add log to subcollection at jobs/{job_id}/logs."""
|
||||
import uuid
|
||||
log_id = str(uuid.uuid4())
|
||||
now = datetime.now(timezone.utc)
|
||||
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
logs_ref.document(log_id).set({
|
||||
"id": log_id,
|
||||
"job_id": job_id,
|
||||
"timestamp": now,
|
||||
"level": "INFO",
|
||||
"worker": worker,
|
||||
"message": message,
|
||||
"ttl_expiry": now + timedelta(days=ttl_days)
|
||||
})
|
||||
return log_id
|
||||
|
||||
def _get_logs_from_subcollection(self, job_id: str, worker: str = None, limit: int = 500):
|
||||
"""Get logs from subcollection."""
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
query = logs_ref.order_by("timestamp")
|
||||
|
||||
if worker:
|
||||
from google.cloud.firestore_v1 import FieldFilter
|
||||
query = query.where(filter=FieldFilter("worker", "==", worker))
|
||||
|
||||
query = query.limit(limit)
|
||||
|
||||
return [doc.to_dict() for doc in query.stream()]
|
||||
|
||||
def _count_logs_in_subcollection(self, job_id: str) -> int:
|
||||
"""Count logs in subcollection."""
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
count_query = logs_ref.count()
|
||||
result = count_query.get()
|
||||
if result and len(result) > 0:
|
||||
return result[0][0].value
|
||||
return 0
|
||||
|
||||
def _delete_logs_subcollection(self, job_id: str, batch_size: int = 100) -> int:
|
||||
"""Delete all logs in subcollection."""
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
deleted_count = 0
|
||||
|
||||
while True:
|
||||
docs = list(logs_ref.limit(batch_size).stream())
|
||||
if not docs:
|
||||
break
|
||||
|
||||
batch = self.db.batch()
|
||||
for doc in docs:
|
||||
batch.delete(doc.reference)
|
||||
deleted_count += 1
|
||||
|
||||
batch.commit()
|
||||
|
||||
return deleted_count
|
||||
|
||||
def test_subcollection_single_write(self):
|
||||
"""Test writing single log to subcollection."""
|
||||
job_id = self._create_test_job()
|
||||
|
||||
log_id = self._append_log_to_subcollection(job_id, "test", "Single log message")
|
||||
|
||||
time.sleep(0.1)
|
||||
logs = self._get_logs_from_subcollection(job_id)
|
||||
|
||||
assert len(logs) == 1
|
||||
assert logs[0]["message"] == "Single log message"
|
||||
assert logs[0]["worker"] == "test"
|
||||
assert logs[0]["id"] == log_id
|
||||
print("Single write to subcollection works")
|
||||
|
||||
def test_subcollection_ttl_field_is_set(self):
|
||||
"""Test TTL expiry field is set correctly."""
|
||||
job_id = self._create_test_job()
|
||||
|
||||
self._append_log_to_subcollection(job_id, "test", "Log with TTL", ttl_days=7)
|
||||
|
||||
time.sleep(0.1)
|
||||
logs = self._get_logs_from_subcollection(job_id)
|
||||
|
||||
assert len(logs) == 1
|
||||
ttl_expiry = logs[0]["ttl_expiry"]
|
||||
assert ttl_expiry is not None
|
||||
|
||||
# TTL should be ~7 days from now
|
||||
expected_ttl = datetime.now(timezone.utc) + timedelta(days=7)
|
||||
if hasattr(ttl_expiry, 'timestamp'):
|
||||
# Firestore datetime object
|
||||
diff = abs((ttl_expiry.replace(tzinfo=timezone.utc) - expected_ttl).total_seconds())
|
||||
else:
|
||||
diff = 0 # If already datetime
|
||||
assert diff < 60, f"TTL expiry should be ~7 days from now, diff was {diff}s"
|
||||
print("TTL field is set correctly")
|
||||
|
||||
def test_subcollection_sequential_writes(self):
|
||||
"""Test multiple sequential writes to subcollection."""
|
||||
job_id = self._create_test_job()
|
||||
|
||||
for i in range(20):
|
||||
self._append_log_to_subcollection(job_id, "test", f"Log {i}")
|
||||
|
||||
time.sleep(0.2)
|
||||
logs = self._get_logs_from_subcollection(job_id)
|
||||
|
||||
assert len(logs) == 20, f"Expected 20 logs, got {len(logs)}"
|
||||
print(f"Sequential writes: {len(logs)} logs created")
|
||||
|
||||
def test_subcollection_concurrent_writes(self):
|
||||
"""Test concurrent writes to subcollection - no race conditions."""
|
||||
job_id = self._create_test_job()
|
||||
num_writes = 50
|
||||
|
||||
def write_log(index):
|
||||
self._append_log_to_subcollection(job_id, "worker", f"Concurrent Log {index}")
|
||||
|
||||
with ThreadPoolExecutor(max_workers=10) as executor:
|
||||
futures = [executor.submit(write_log, i) for i in range(num_writes)]
|
||||
for future in as_completed(futures):
|
||||
future.result()
|
||||
|
||||
time.sleep(0.5)
|
||||
logs = self._get_logs_from_subcollection(job_id)
|
||||
|
||||
assert len(logs) == num_writes, f"Expected {num_writes} logs, got {len(logs)}"
|
||||
print(f"Concurrent writes: All {num_writes} logs preserved")
|
||||
|
||||
def test_subcollection_large_volume(self):
|
||||
"""Test large volume of logs - demonstrates no 1MB limit."""
|
||||
job_id = self._create_test_job()
|
||||
num_logs = 500 # Would exceed 1MB in embedded array
|
||||
|
||||
# Write logs in batches
|
||||
batch_size = 50
|
||||
for batch_start in range(0, num_logs, batch_size):
|
||||
with ThreadPoolExecutor(max_workers=10) as executor:
|
||||
futures = []
|
||||
for i in range(batch_start, min(batch_start + batch_size, num_logs)):
|
||||
futures.append(executor.submit(
|
||||
self._append_log_to_subcollection,
|
||||
job_id,
|
||||
"stress",
|
||||
f"Large volume test log {i} - " + "x" * 500 # ~500 bytes per log
|
||||
))
|
||||
for future in as_completed(futures):
|
||||
future.result()
|
||||
|
||||
time.sleep(1)
|
||||
count = self._count_logs_in_subcollection(job_id)
|
||||
|
||||
assert count == num_logs, f"Expected {num_logs} logs, got {count}"
|
||||
print(f"Large volume: {count} logs created (~{count * 500 / 1024:.1f}KB would exceed 1MB if embedded)")
|
||||
|
||||
def test_subcollection_filter_by_worker(self):
|
||||
"""Test filtering logs by worker type."""
|
||||
job_id = self._create_test_job()
|
||||
|
||||
# Add logs from different workers
|
||||
for i in range(10):
|
||||
self._append_log_to_subcollection(job_id, "audio", f"Audio log {i}")
|
||||
self._append_log_to_subcollection(job_id, "lyrics", f"Lyrics log {i}")
|
||||
self._append_log_to_subcollection(job_id, "video", f"Video log {i}")
|
||||
|
||||
time.sleep(0.3)
|
||||
|
||||
# Query by worker
|
||||
audio_logs = self._get_logs_from_subcollection(job_id, worker="audio")
|
||||
lyrics_logs = self._get_logs_from_subcollection(job_id, worker="lyrics")
|
||||
all_logs = self._get_logs_from_subcollection(job_id)
|
||||
|
||||
assert len(audio_logs) == 10, f"Expected 10 audio logs, got {len(audio_logs)}"
|
||||
assert len(lyrics_logs) == 10, f"Expected 10 lyrics logs, got {len(lyrics_logs)}"
|
||||
assert len(all_logs) == 30, f"Expected 30 total logs, got {len(all_logs)}"
|
||||
print("Worker filtering works correctly")
|
||||
|
||||
def test_subcollection_ordered_by_timestamp(self):
|
||||
"""Test logs are returned in timestamp order."""
|
||||
job_id = self._create_test_job()
|
||||
|
||||
# Add logs with slight delays to ensure different timestamps
|
||||
for i in range(5):
|
||||
self._append_log_to_subcollection(job_id, "test", f"Log {i}")
|
||||
time.sleep(0.02)
|
||||
|
||||
logs = self._get_logs_from_subcollection(job_id)
|
||||
|
||||
# Verify order (timestamps should be ascending)
|
||||
for i in range(len(logs) - 1):
|
||||
assert logs[i]["timestamp"] <= logs[i + 1]["timestamp"], \
|
||||
f"Logs not in order at index {i}"
|
||||
print("Logs are ordered by timestamp")
|
||||
|
||||
def test_subcollection_delete_all_logs(self):
|
||||
"""Test deleting all logs in subcollection."""
|
||||
job_id = self._create_test_job()
|
||||
|
||||
# Add logs
|
||||
for i in range(25):
|
||||
self._append_log_to_subcollection(job_id, "test", f"Log {i}")
|
||||
|
||||
time.sleep(0.2)
|
||||
count_before = self._count_logs_in_subcollection(job_id)
|
||||
assert count_before == 25
|
||||
|
||||
# Delete all logs
|
||||
deleted = self._delete_logs_subcollection(job_id)
|
||||
|
||||
time.sleep(0.2)
|
||||
count_after = self._count_logs_in_subcollection(job_id)
|
||||
|
||||
assert deleted == 25, f"Expected to delete 25 logs, deleted {deleted}"
|
||||
assert count_after == 0, f"Expected 0 logs after delete, got {count_after}"
|
||||
print(f"Deleted {deleted} logs from subcollection")
|
||||
|
||||
def test_subcollection_concurrent_workers_interleaved(self):
|
||||
"""Test simulating audio and lyrics workers writing concurrently."""
|
||||
job_id = self._create_test_job()
|
||||
logs_per_worker = 20
|
||||
|
||||
def audio_worker():
|
||||
for i in range(logs_per_worker):
|
||||
self._append_log_to_subcollection(job_id, "audio", f"Audio processing step {i}")
|
||||
time.sleep(0.005)
|
||||
|
||||
def lyrics_worker():
|
||||
for i in range(logs_per_worker):
|
||||
self._append_log_to_subcollection(job_id, "lyrics", f"Lyrics processing step {i}")
|
||||
time.sleep(0.005)
|
||||
|
||||
def video_worker():
|
||||
for i in range(logs_per_worker):
|
||||
self._append_log_to_subcollection(job_id, "video", f"Video encoding step {i}")
|
||||
time.sleep(0.005)
|
||||
|
||||
# Start all workers
|
||||
threads = [
|
||||
threading.Thread(target=audio_worker),
|
||||
threading.Thread(target=lyrics_worker),
|
||||
threading.Thread(target=video_worker)
|
||||
]
|
||||
|
||||
for t in threads:
|
||||
t.start()
|
||||
for t in threads:
|
||||
t.join()
|
||||
|
||||
time.sleep(0.5)
|
||||
|
||||
audio_logs = self._get_logs_from_subcollection(job_id, worker="audio")
|
||||
lyrics_logs = self._get_logs_from_subcollection(job_id, worker="lyrics")
|
||||
video_logs = self._get_logs_from_subcollection(job_id, worker="video")
|
||||
total = self._count_logs_in_subcollection(job_id)
|
||||
|
||||
assert len(audio_logs) == logs_per_worker
|
||||
assert len(lyrics_logs) == logs_per_worker
|
||||
assert len(video_logs) == logs_per_worker
|
||||
assert total == logs_per_worker * 3
|
||||
|
||||
print(f"Concurrent workers: {len(audio_logs)} audio + {len(lyrics_logs)} lyrics + {len(video_logs)} video = {total}")
|
||||
|
||||
def test_subcollection_job_deletion_cleans_up_logs(self):
|
||||
"""Test that deleting job document doesn't orphan logs."""
|
||||
job_id = self._create_test_job()
|
||||
|
||||
# Add logs
|
||||
for i in range(10):
|
||||
self._append_log_to_subcollection(job_id, "test", f"Log {i}")
|
||||
|
||||
time.sleep(0.2)
|
||||
assert self._count_logs_in_subcollection(job_id) == 10
|
||||
|
||||
# Delete logs first (must happen before parent doc deletion in real code)
|
||||
deleted = self._delete_logs_subcollection(job_id)
|
||||
assert deleted == 10
|
||||
|
||||
# Now delete job document
|
||||
self.db.collection(self.collection).document(job_id).delete()
|
||||
|
||||
# Verify subcollection is empty (Firestore doesn't cascade delete subcollections)
|
||||
time.sleep(0.2)
|
||||
assert self._count_logs_in_subcollection(job_id) == 0
|
||||
print("Job deletion with log cleanup works correctly")
|
||||
|
||||
|
||||
class TestSubcollectionVsEmbeddedArray:
|
||||
"""Compare subcollection approach with embedded array."""
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def setup_firestore(self):
|
||||
"""Set up Firestore client for each test."""
|
||||
from google.cloud import firestore
|
||||
self.db = firestore.Client(project="test-project")
|
||||
self.collection = "test-comparison-jobs"
|
||||
yield
|
||||
|
||||
def _create_test_job(self, use_array: bool):
|
||||
"""Create test job."""
|
||||
job_id = f"test-cmp-{int(time.time() * 1000)}-{'arr' if use_array else 'sub'}"
|
||||
doc_ref = self.db.collection(self.collection).document(job_id)
|
||||
doc_ref.set({
|
||||
"job_id": job_id,
|
||||
"status": "pending",
|
||||
"created_at": datetime.now(timezone.utc),
|
||||
"worker_logs": []
|
||||
})
|
||||
return job_id
|
||||
|
||||
def _append_log_array(self, job_id: str, message: str):
|
||||
"""Add log to embedded array."""
|
||||
from google.cloud import firestore
|
||||
doc_ref = self.db.collection(self.collection).document(job_id)
|
||||
doc_ref.update({
|
||||
"worker_logs": firestore.ArrayUnion([{
|
||||
"timestamp": datetime.now(timezone.utc).isoformat() + "Z",
|
||||
"level": "INFO",
|
||||
"worker": "test",
|
||||
"message": message
|
||||
}])
|
||||
})
|
||||
|
||||
def _append_log_subcollection(self, job_id: str, message: str):
|
||||
"""Add log to subcollection."""
|
||||
import uuid
|
||||
now = datetime.now(timezone.utc)
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
logs_ref.document(str(uuid.uuid4())).set({
|
||||
"timestamp": now,
|
||||
"level": "INFO",
|
||||
"worker": "test",
|
||||
"message": message,
|
||||
"ttl_expiry": now + timedelta(days=30)
|
||||
})
|
||||
|
||||
def test_document_size_comparison(self):
|
||||
"""Compare document sizes between approaches."""
|
||||
array_job_id = self._create_test_job(use_array=True)
|
||||
sub_job_id = self._create_test_job(use_array=False)
|
||||
|
||||
num_logs = 100
|
||||
large_message = "x" * 5000 # 5KB per log
|
||||
|
||||
# Add logs to both
|
||||
print("\nAdding logs to both approaches...")
|
||||
for i in range(num_logs):
|
||||
self._append_log_array(array_job_id, f"Array log {i}: {large_message}")
|
||||
self._append_log_subcollection(sub_job_id, f"Sub log {i}: {large_message}")
|
||||
|
||||
time.sleep(0.5)
|
||||
|
||||
# Get document sizes (approximate via raw data)
|
||||
array_doc = self.db.collection(self.collection).document(array_job_id).get()
|
||||
sub_doc = self.db.collection(self.collection).document(sub_job_id).get()
|
||||
|
||||
array_data = array_doc.to_dict()
|
||||
sub_data = sub_doc.to_dict()
|
||||
|
||||
array_logs = array_data.get("worker_logs", [])
|
||||
sub_logs = sub_data.get("worker_logs", [])
|
||||
|
||||
print(f"\nEmbedded array: {len(array_logs)} logs in document")
|
||||
print(f"Subcollection: {len(sub_logs)} logs in document (0 expected)")
|
||||
|
||||
# Embedded array should have all logs in document
|
||||
assert len(array_logs) == num_logs
|
||||
|
||||
# Subcollection should have empty array in main document
|
||||
assert len(sub_logs) == 0
|
||||
|
||||
# Subcollection logs are in subcollection
|
||||
sub_count = self.db.collection(self.collection).document(sub_job_id).collection("logs").count().get()[0][0].value
|
||||
assert sub_count == num_logs
|
||||
|
||||
print(f"Subcollection logs stored separately: {sub_count}")
|
||||
print("Subcollection approach keeps main document small")
|
||||
|
||||
|
||||
print("Worker logs subcollection integration tests ready")
|
||||
@@ -0,0 +1,509 @@
|
||||
"""
|
||||
Unit tests for worker log subcollection functionality.
|
||||
|
||||
Tests the WorkerLogEntry model, FirestoreService subcollection methods,
|
||||
and JobManager log operations with the feature flag.
|
||||
"""
|
||||
import pytest
|
||||
from unittest.mock import Mock, MagicMock, patch
|
||||
from datetime import datetime, timezone, timedelta
|
||||
import uuid
|
||||
|
||||
# Mock Firestore before imports
|
||||
import sys
|
||||
sys.modules['google.cloud.firestore'] = MagicMock()
|
||||
sys.modules['google.cloud.firestore_v1'] = MagicMock()
|
||||
|
||||
from backend.models.worker_log import WorkerLogEntry, DEFAULT_LOG_TTL_DAYS
|
||||
|
||||
|
||||
class TestWorkerLogEntry:
|
||||
"""Tests for WorkerLogEntry dataclass."""
|
||||
|
||||
def test_create_log_entry(self):
|
||||
"""Test creating a log entry with factory method."""
|
||||
entry = WorkerLogEntry.create(
|
||||
job_id="job123",
|
||||
worker="audio",
|
||||
level="INFO",
|
||||
message="Test message"
|
||||
)
|
||||
|
||||
assert entry.job_id == "job123"
|
||||
assert entry.worker == "audio"
|
||||
assert entry.level == "INFO"
|
||||
assert entry.message == "Test message"
|
||||
assert entry.id is not None
|
||||
assert entry.timestamp is not None
|
||||
assert entry.ttl_expiry is not None
|
||||
# TTL should be ~30 days from now
|
||||
expected_ttl = datetime.now(timezone.utc) + timedelta(days=DEFAULT_LOG_TTL_DAYS)
|
||||
assert abs((entry.ttl_expiry - expected_ttl).total_seconds()) < 5
|
||||
|
||||
def test_create_log_entry_truncates_long_message(self):
|
||||
"""Test that messages longer than 1000 chars are truncated."""
|
||||
long_message = "x" * 2000
|
||||
entry = WorkerLogEntry.create(
|
||||
job_id="job123",
|
||||
worker="audio",
|
||||
level="INFO",
|
||||
message=long_message
|
||||
)
|
||||
|
||||
assert len(entry.message) == 1000
|
||||
|
||||
def test_create_log_entry_normalizes_level(self):
|
||||
"""Test that log level is normalized to uppercase."""
|
||||
entry = WorkerLogEntry.create(
|
||||
job_id="job123",
|
||||
worker="audio",
|
||||
level="info",
|
||||
message="Test"
|
||||
)
|
||||
|
||||
assert entry.level == "INFO"
|
||||
|
||||
def test_create_log_entry_custom_ttl(self):
|
||||
"""Test creating entry with custom TTL."""
|
||||
entry = WorkerLogEntry.create(
|
||||
job_id="job123",
|
||||
worker="audio",
|
||||
level="INFO",
|
||||
message="Test",
|
||||
ttl_days=7
|
||||
)
|
||||
|
||||
expected_ttl = datetime.now(timezone.utc) + timedelta(days=7)
|
||||
assert abs((entry.ttl_expiry - expected_ttl).total_seconds()) < 5
|
||||
|
||||
def test_create_log_entry_with_metadata(self):
|
||||
"""Test creating entry with metadata."""
|
||||
metadata = {"file_size": 1024, "duration": 300}
|
||||
entry = WorkerLogEntry.create(
|
||||
job_id="job123",
|
||||
worker="audio",
|
||||
level="INFO",
|
||||
message="Test",
|
||||
metadata=metadata
|
||||
)
|
||||
|
||||
assert entry.metadata == metadata
|
||||
|
||||
def test_to_dict(self):
|
||||
"""Test converting entry to dict for Firestore."""
|
||||
entry = WorkerLogEntry.create(
|
||||
job_id="job123",
|
||||
worker="audio",
|
||||
level="INFO",
|
||||
message="Test message",
|
||||
metadata={"key": "value"}
|
||||
)
|
||||
|
||||
d = entry.to_dict()
|
||||
|
||||
assert d["job_id"] == "job123"
|
||||
assert d["worker"] == "audio"
|
||||
assert d["level"] == "INFO"
|
||||
assert d["message"] == "Test message"
|
||||
assert d["metadata"] == {"key": "value"}
|
||||
assert "id" in d
|
||||
assert "timestamp" in d
|
||||
assert "ttl_expiry" in d
|
||||
|
||||
def test_to_dict_without_metadata(self):
|
||||
"""Test to_dict doesn't include metadata when None."""
|
||||
entry = WorkerLogEntry.create(
|
||||
job_id="job123",
|
||||
worker="audio",
|
||||
level="INFO",
|
||||
message="Test"
|
||||
)
|
||||
|
||||
d = entry.to_dict()
|
||||
assert "metadata" not in d
|
||||
|
||||
def test_to_legacy_dict(self):
|
||||
"""Test converting to legacy format for API compatibility."""
|
||||
entry = WorkerLogEntry.create(
|
||||
job_id="job123",
|
||||
worker="audio",
|
||||
level="INFO",
|
||||
message="Test message"
|
||||
)
|
||||
|
||||
d = entry.to_legacy_dict()
|
||||
|
||||
assert "timestamp" in d
|
||||
assert d["level"] == "INFO"
|
||||
assert d["worker"] == "audio"
|
||||
assert d["message"] == "Test message"
|
||||
# Should not include id, job_id, ttl_expiry, metadata
|
||||
assert "id" not in d
|
||||
assert "job_id" not in d
|
||||
assert "ttl_expiry" not in d
|
||||
|
||||
def test_from_dict(self):
|
||||
"""Test creating entry from Firestore document."""
|
||||
timestamp = datetime.now(timezone.utc)
|
||||
ttl_expiry = timestamp + timedelta(days=30)
|
||||
|
||||
data = {
|
||||
"id": "log123",
|
||||
"job_id": "job123",
|
||||
"timestamp": timestamp,
|
||||
"level": "WARNING",
|
||||
"worker": "video",
|
||||
"message": "Warning message",
|
||||
"metadata": {"error_code": 500},
|
||||
"ttl_expiry": ttl_expiry
|
||||
}
|
||||
|
||||
entry = WorkerLogEntry.from_dict(data)
|
||||
|
||||
assert entry.id == "log123"
|
||||
assert entry.job_id == "job123"
|
||||
assert entry.level == "WARNING"
|
||||
assert entry.worker == "video"
|
||||
assert entry.message == "Warning message"
|
||||
assert entry.metadata == {"error_code": 500}
|
||||
|
||||
def test_from_dict_with_iso_strings(self):
|
||||
"""Test creating entry from dict with ISO format strings."""
|
||||
data = {
|
||||
"timestamp": "2026-01-04T12:00:00Z",
|
||||
"level": "INFO",
|
||||
"worker": "audio",
|
||||
"message": "Test"
|
||||
}
|
||||
|
||||
entry = WorkerLogEntry.from_dict(data)
|
||||
|
||||
assert entry.timestamp.tzinfo == timezone.utc
|
||||
assert entry.message == "Test"
|
||||
|
||||
def test_from_dict_missing_fields(self):
|
||||
"""Test creating entry with missing fields uses defaults."""
|
||||
data = {
|
||||
"message": "Minimal log"
|
||||
}
|
||||
|
||||
entry = WorkerLogEntry.from_dict(data)
|
||||
|
||||
assert entry.message == "Minimal log"
|
||||
assert entry.level == "INFO" # Default
|
||||
assert entry.worker == "unknown" # Default
|
||||
assert entry.id is not None # Generated
|
||||
|
||||
|
||||
class TestFirestoreServiceSubcollection:
|
||||
"""Tests for FirestoreService log subcollection methods."""
|
||||
|
||||
@pytest.fixture
|
||||
def mock_db(self):
|
||||
"""Create mock Firestore database."""
|
||||
return MagicMock()
|
||||
|
||||
@pytest.fixture
|
||||
def firestore_service(self, mock_db):
|
||||
"""Create FirestoreService with mocked DB."""
|
||||
with patch('backend.services.firestore_service.firestore') as mock_firestore:
|
||||
mock_firestore.Client.return_value = mock_db
|
||||
from backend.services.firestore_service import FirestoreService
|
||||
service = FirestoreService()
|
||||
service.db = mock_db
|
||||
return service
|
||||
|
||||
def test_append_log_to_subcollection(self, firestore_service, mock_db):
|
||||
"""Test appending log to subcollection."""
|
||||
entry = WorkerLogEntry.create(
|
||||
job_id="job123",
|
||||
worker="audio",
|
||||
level="INFO",
|
||||
message="Test message"
|
||||
)
|
||||
|
||||
# Setup mock chain
|
||||
mock_collection = MagicMock()
|
||||
mock_doc = MagicMock()
|
||||
mock_logs_ref = MagicMock()
|
||||
mock_log_doc = MagicMock()
|
||||
|
||||
mock_db.collection.return_value = mock_collection
|
||||
mock_collection.document.return_value = mock_doc
|
||||
mock_doc.collection.return_value = mock_logs_ref
|
||||
mock_logs_ref.document.return_value = mock_log_doc
|
||||
|
||||
firestore_service.append_log_to_subcollection("job123", entry)
|
||||
|
||||
# Verify correct path: jobs/{job_id}/logs/{log_id}
|
||||
mock_db.collection.assert_called_with("jobs")
|
||||
mock_collection.document.assert_called_with("job123")
|
||||
mock_doc.collection.assert_called_with("logs")
|
||||
mock_logs_ref.document.assert_called_with(entry.id)
|
||||
mock_log_doc.set.assert_called_once()
|
||||
|
||||
def test_get_logs_from_subcollection(self, firestore_service, mock_db):
|
||||
"""Test getting logs from subcollection."""
|
||||
# Setup mock chain
|
||||
mock_collection = MagicMock()
|
||||
mock_doc = MagicMock()
|
||||
mock_logs_ref = MagicMock()
|
||||
mock_query = MagicMock()
|
||||
|
||||
mock_db.collection.return_value = mock_collection
|
||||
mock_collection.document.return_value = mock_doc
|
||||
mock_doc.collection.return_value = mock_logs_ref
|
||||
mock_logs_ref.order_by.return_value = mock_query
|
||||
mock_query.limit.return_value = mock_query
|
||||
|
||||
# Mock document data
|
||||
mock_doc1 = MagicMock()
|
||||
mock_doc1.to_dict.return_value = {
|
||||
"timestamp": datetime.now(timezone.utc),
|
||||
"level": "INFO",
|
||||
"worker": "audio",
|
||||
"message": "Log 1"
|
||||
}
|
||||
|
||||
mock_query.stream.return_value = iter([mock_doc1])
|
||||
|
||||
logs = firestore_service.get_logs_from_subcollection("job123")
|
||||
|
||||
assert len(logs) == 1
|
||||
assert logs[0].message == "Log 1"
|
||||
mock_logs_ref.order_by.assert_called_once()
|
||||
|
||||
def test_get_logs_from_subcollection_with_worker_filter(self, firestore_service, mock_db):
|
||||
"""Test filtering logs by worker."""
|
||||
mock_collection = MagicMock()
|
||||
mock_doc = MagicMock()
|
||||
mock_logs_ref = MagicMock()
|
||||
mock_query = MagicMock()
|
||||
|
||||
mock_db.collection.return_value = mock_collection
|
||||
mock_collection.document.return_value = mock_doc
|
||||
mock_doc.collection.return_value = mock_logs_ref
|
||||
mock_logs_ref.order_by.return_value = mock_query
|
||||
mock_query.where.return_value = mock_query
|
||||
mock_query.limit.return_value = mock_query
|
||||
mock_query.stream.return_value = iter([])
|
||||
|
||||
firestore_service.get_logs_from_subcollection("job123", worker="audio")
|
||||
|
||||
# Verify where clause was added
|
||||
mock_query.where.assert_called()
|
||||
|
||||
def test_delete_logs_subcollection(self, firestore_service, mock_db):
|
||||
"""Test deleting all logs in subcollection."""
|
||||
mock_collection = MagicMock()
|
||||
mock_doc = MagicMock()
|
||||
mock_logs_ref = MagicMock()
|
||||
mock_batch = MagicMock()
|
||||
|
||||
mock_db.collection.return_value = mock_collection
|
||||
mock_collection.document.return_value = mock_doc
|
||||
mock_doc.collection.return_value = mock_logs_ref
|
||||
mock_db.batch.return_value = mock_batch
|
||||
|
||||
# First call returns 2 docs, second call returns empty
|
||||
mock_log_doc1 = MagicMock()
|
||||
mock_log_doc2 = MagicMock()
|
||||
|
||||
call_count = [0]
|
||||
def stream_side_effect(*args, **kwargs):
|
||||
call_count[0] += 1
|
||||
if call_count[0] == 1:
|
||||
return iter([mock_log_doc1, mock_log_doc2])
|
||||
return iter([])
|
||||
|
||||
mock_logs_ref.limit.return_value = mock_logs_ref
|
||||
mock_logs_ref.stream.side_effect = stream_side_effect
|
||||
|
||||
deleted = firestore_service.delete_logs_subcollection("job123")
|
||||
|
||||
assert deleted == 2
|
||||
mock_batch.commit.assert_called()
|
||||
|
||||
|
||||
class TestJobManagerLogging:
|
||||
"""Tests for JobManager log methods with feature flag."""
|
||||
|
||||
@pytest.fixture
|
||||
def mock_firestore_service(self):
|
||||
"""Mock FirestoreService."""
|
||||
with patch('backend.services.job_manager.FirestoreService') as mock:
|
||||
service = MagicMock()
|
||||
mock.return_value = service
|
||||
yield service
|
||||
|
||||
@pytest.fixture
|
||||
def mock_storage_service(self):
|
||||
"""Mock StorageService."""
|
||||
with patch('backend.services.job_manager.StorageService') as mock:
|
||||
service = MagicMock()
|
||||
mock.return_value = service
|
||||
yield service
|
||||
|
||||
def test_append_worker_log_uses_subcollection_when_enabled(
|
||||
self, mock_firestore_service, mock_storage_service
|
||||
):
|
||||
"""Test that logs go to subcollection when feature is enabled."""
|
||||
with patch('backend.services.job_manager.settings') as mock_settings:
|
||||
mock_settings.use_log_subcollection = True
|
||||
|
||||
from backend.services.job_manager import JobManager
|
||||
manager = JobManager()
|
||||
|
||||
manager.append_worker_log(
|
||||
job_id="job123",
|
||||
worker="audio",
|
||||
level="INFO",
|
||||
message="Test message"
|
||||
)
|
||||
|
||||
# Should call subcollection method
|
||||
mock_firestore_service.append_log_to_subcollection.assert_called_once()
|
||||
# Should NOT call legacy method
|
||||
mock_firestore_service.append_worker_log.assert_not_called()
|
||||
|
||||
def test_append_worker_log_uses_array_when_disabled(
|
||||
self, mock_firestore_service, mock_storage_service
|
||||
):
|
||||
"""Test that logs go to embedded array when feature is disabled."""
|
||||
with patch('backend.services.job_manager.settings') as mock_settings:
|
||||
mock_settings.use_log_subcollection = False
|
||||
|
||||
from backend.services.job_manager import JobManager
|
||||
manager = JobManager()
|
||||
|
||||
manager.append_worker_log(
|
||||
job_id="job123",
|
||||
worker="audio",
|
||||
level="INFO",
|
||||
message="Test message"
|
||||
)
|
||||
|
||||
# Should call legacy method
|
||||
mock_firestore_service.append_worker_log.assert_called_once()
|
||||
# Should NOT call subcollection method
|
||||
mock_firestore_service.append_log_to_subcollection.assert_not_called()
|
||||
|
||||
def test_get_worker_logs_from_subcollection(
|
||||
self, mock_firestore_service, mock_storage_service
|
||||
):
|
||||
"""Test getting logs from subcollection."""
|
||||
with patch('backend.services.job_manager.settings') as mock_settings:
|
||||
mock_settings.use_log_subcollection = True
|
||||
|
||||
entry = WorkerLogEntry.create(
|
||||
job_id="job123",
|
||||
worker="audio",
|
||||
level="INFO",
|
||||
message="Test"
|
||||
)
|
||||
mock_firestore_service.get_logs_from_subcollection.return_value = [entry]
|
||||
|
||||
from backend.services.job_manager import JobManager
|
||||
manager = JobManager()
|
||||
|
||||
logs = manager.get_worker_logs("job123")
|
||||
|
||||
assert len(logs) == 1
|
||||
assert logs[0]["message"] == "Test"
|
||||
mock_firestore_service.get_logs_from_subcollection.assert_called_once()
|
||||
|
||||
def test_get_worker_logs_falls_back_to_array(
|
||||
self, mock_firestore_service, mock_storage_service
|
||||
):
|
||||
"""Test fallback to embedded array for legacy jobs."""
|
||||
with patch('backend.services.job_manager.settings') as mock_settings:
|
||||
mock_settings.use_log_subcollection = True
|
||||
|
||||
# Subcollection returns empty
|
||||
mock_firestore_service.get_logs_from_subcollection.return_value = []
|
||||
|
||||
# Mock job with legacy logs
|
||||
from backend.models.job import Job, JobStatus
|
||||
mock_job = Job(
|
||||
job_id="job123",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
worker_logs=[
|
||||
{"timestamp": "2026-01-04T12:00:00Z", "level": "INFO", "worker": "audio", "message": "Legacy log"}
|
||||
]
|
||||
)
|
||||
mock_firestore_service.get_job.return_value = mock_job
|
||||
|
||||
from backend.services.job_manager import JobManager
|
||||
manager = JobManager()
|
||||
|
||||
logs = manager.get_worker_logs("job123")
|
||||
|
||||
assert len(logs) == 1
|
||||
assert logs[0]["message"] == "Legacy log"
|
||||
|
||||
def test_delete_job_deletes_logs_subcollection(
|
||||
self, mock_firestore_service, mock_storage_service
|
||||
):
|
||||
"""Test that deleting a job also deletes its logs subcollection."""
|
||||
mock_firestore_service.get_job.return_value = None
|
||||
mock_firestore_service.delete_logs_subcollection.return_value = 5
|
||||
|
||||
from backend.services.job_manager import JobManager
|
||||
manager = JobManager()
|
||||
|
||||
manager.delete_job("job123", delete_files=False)
|
||||
|
||||
mock_firestore_service.delete_logs_subcollection.assert_called_once_with("job123")
|
||||
mock_firestore_service.delete_job.assert_called_once_with("job123")
|
||||
|
||||
|
||||
class TestWorkerLogEntryEdgeCases:
|
||||
"""Edge case tests for WorkerLogEntry."""
|
||||
|
||||
def test_create_with_empty_message(self):
|
||||
"""Test creating entry with empty message."""
|
||||
entry = WorkerLogEntry.create(
|
||||
job_id="job123",
|
||||
worker="audio",
|
||||
level="INFO",
|
||||
message=""
|
||||
)
|
||||
assert entry.message == ""
|
||||
|
||||
def test_create_with_unicode_message(self):
|
||||
"""Test creating entry with Unicode characters."""
|
||||
entry = WorkerLogEntry.create(
|
||||
job_id="job123",
|
||||
worker="audio",
|
||||
level="INFO",
|
||||
message="Processing song: 日本語の曲 - アーティスト"
|
||||
)
|
||||
assert "日本語" in entry.message
|
||||
|
||||
def test_create_with_newlines_in_message(self):
|
||||
"""Test creating entry with newlines."""
|
||||
entry = WorkerLogEntry.create(
|
||||
job_id="job123",
|
||||
worker="audio",
|
||||
level="ERROR",
|
||||
message="Error occurred:\nLine 1\nLine 2"
|
||||
)
|
||||
assert "\n" in entry.message
|
||||
|
||||
def test_from_dict_handles_missing_ttl_expiry(self):
|
||||
"""Test from_dict creates default TTL when missing."""
|
||||
data = {
|
||||
"timestamp": datetime.now(timezone.utc),
|
||||
"level": "INFO",
|
||||
"worker": "audio",
|
||||
"message": "Test"
|
||||
}
|
||||
|
||||
entry = WorkerLogEntry.from_dict(data)
|
||||
|
||||
assert entry.ttl_expiry is not None
|
||||
# Should be ~30 days from now
|
||||
expected = datetime.now(timezone.utc) + timedelta(days=30)
|
||||
assert abs((entry.ttl_expiry - expected).total_seconds()) < 10
|
||||
@@ -163,12 +163,15 @@ karaoke-gen shares a GCP project (`nomadkaraoke`) with karaoke-decide, but uses
|
||||
|------------|---------|------------|
|
||||
| `gen_users` | karaoke-gen user accounts | email, credits, role, is_active |
|
||||
| `jobs` | Karaoke generation jobs | job_id, user_email, status, state_data |
|
||||
| `jobs/{job_id}/logs` | Worker log entries (subcollection) | timestamp, level, worker, message, ttl_expiry |
|
||||
| `sessions` | Magic link auth sessions | user_email, token, expires_at |
|
||||
| `magic_links` | Passwordless auth tokens | email, token, expires_at, used |
|
||||
| `beta_feedback` | Beta program feedback | user_email, ratings, comments |
|
||||
|
||||
**Note**: The `users` collection in the same Firestore instance belongs to karaoke-decide (different schema: user_id, is_guest, quiz_* fields). Don't use it for karaoke-gen.
|
||||
|
||||
**Worker Logs**: Stored in subcollection (`jobs/{job_id}/logs`) instead of embedded array to avoid Firestore's 1MB document size limit. TTL policy auto-deletes logs after 30 days. Feature flag: `USE_LOG_SUBCOLLECTION` (default: true). See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#firestore-document-1mb-limit-with-embedded-arrays).
|
||||
|
||||
## Video Worker Orchestrator
|
||||
|
||||
The Video Worker uses an orchestrator pattern to ensure all features work regardless of encoding backend (local Cloud Run or GCE VM).
|
||||
|
||||
@@ -1038,6 +1038,47 @@ result = await orchestrator.run()
|
||||
|
||||
See `docs/archive/2026-01-04-video-worker-orchestrator-refactor.md` for full details.
|
||||
|
||||
### Firestore Document 1MB Limit with Embedded Arrays
|
||||
|
||||
**Problem**: Firestore documents have a 1MB size limit. Storing logs as an embedded array (`worker_logs: [{...}, {...}, ...]`) within job documents caused job failures when logs exceeded the limit during long-running operations like video encoding.
|
||||
|
||||
**Example**: Job `501258e1` failed when logs reached 1.26MB (2,547 log entries), representing 98.6% of the maximum document size. The job update failed with:
|
||||
```
|
||||
Document cannot be written because its size exceeds the maximum allowed size
|
||||
```
|
||||
|
||||
**Root cause**: Video encoding jobs generate thousands of log entries (FFmpeg output, progress updates). Each log entry is ~500 bytes. 2,000+ entries easily exceed 1MB.
|
||||
|
||||
**Solution**: Store logs in a Firestore subcollection instead of embedded array:
|
||||
|
||||
```python
|
||||
# BAD - embedded array hits 1MB limit
|
||||
doc_ref.update({
|
||||
"worker_logs": firestore.ArrayUnion([{"message": "..."}])
|
||||
})
|
||||
|
||||
# GOOD - subcollection has no size limit per parent doc
|
||||
logs_ref = db.collection("jobs").document(job_id).collection("logs")
|
||||
logs_ref.document(log_id).set({
|
||||
"timestamp": now,
|
||||
"level": "INFO",
|
||||
"worker": "video",
|
||||
"message": "...",
|
||||
"ttl_expiry": now + timedelta(days=30)
|
||||
})
|
||||
```
|
||||
|
||||
**Key benefits of subcollection approach**:
|
||||
1. Each log entry is its own document (no size limit concerns)
|
||||
2. Natural association with parent job (path: `jobs/{job_id}/logs/{log_id}`)
|
||||
3. Can add TTL via Firestore TTL policies for automatic cleanup
|
||||
4. Efficient queries with composite indexes (worker + timestamp)
|
||||
5. Concurrent workers can write without race conditions (unlike ArrayUnion which can conflict)
|
||||
|
||||
**Migration strategy**: Used feature flag (`USE_LOG_SUBCOLLECTION=true`) for rollback capability. New jobs use subcollection; old jobs with embedded arrays continue to work via fallback read logic.
|
||||
|
||||
**Lesson**: Never use embedded arrays for unbounded data. If an array can grow indefinitely based on operation duration or user input, use a subcollection instead. The 1MB limit applies to the entire document, including all fields.
|
||||
|
||||
### External Service Response Format Mismatches
|
||||
|
||||
**Problem**: The GCE encoding worker returned responses in unexpected formats, causing `'list' object has no attribute 'get'` errors that were hard to debug through the async pipeline.
|
||||
|
||||
+3
-1
@@ -36,7 +36,9 @@
|
||||
|
||||
## Recent Changes
|
||||
|
||||
- **GCE Encoding Response Fixes** (2026-01-04): Fixed multiple response format mismatches with GCE encoding worker. The worker returns `output_files` as a list of paths, not a dict with format keys - added conversion logic. Also added defensive type checking for status responses that could be lists. Created worker logs rearchitecture plan to address Firestore 1MB document limit (logs will move to subcollection). See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#external-service-response-format-mismatches).
|
||||
- **Worker Logs Subcollection** (2026-01-04): Moved `worker_logs` from embedded array in job documents to Firestore subcollection (`jobs/{job_id}/logs`). Fixes job failures when logs exceed 1MB (job 501258e1 had 1.26MB of logs). New logs stored with 30-day TTL via Firestore TTL policy. Feature flag `USE_LOG_SUBCOLLECTION=true` (default). See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#firestore-document-1mb-limit-with-embedded-arrays).
|
||||
|
||||
- **GCE Encoding Response Fixes** (2026-01-04): Fixed multiple response format mismatches with GCE encoding worker. The worker returns `output_files` as a list of paths, not a dict with format keys - added conversion logic. Also added defensive type checking for status responses that could be lists. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#external-service-response-format-mismatches).
|
||||
|
||||
- **Video Worker Orchestrator** (2026-01-04): Major refactor to unify video generation pipeline. Created VideoWorkerOrchestrator that coordinates all stages (packaging, encoding, distribution, notifications) regardless of encoding backend (GCE or local). Fixes issue where GCE encoding path bypassed YouTube upload, Discord notifications, and CDG/TXT packaging. Feature flag `USE_NEW_ORCHESTRATOR` (default: true) enables rollback. 139 new tests across 6 new service modules. See [ARCHITECTURE.md](ARCHITECTURE.md#video-worker-orchestrator) and [archive/2026-01-04-video-worker-orchestrator-refactor.md](archive/2026-01-04-video-worker-orchestrator-refactor.md).
|
||||
|
||||
|
||||
@@ -144,6 +144,42 @@ firestore_index_gen_users_active_email = firestore.Index(
|
||||
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
|
||||
)
|
||||
|
||||
# =============================================================================
|
||||
# Firestore TTL Field Configuration
|
||||
# TTL (Time-to-Live) policies automatically delete documents after expiration.
|
||||
# =============================================================================
|
||||
|
||||
# Worker logs TTL: logs subcollection documents expire after 30 days
|
||||
# Collection path for subcollections: "jobs/{job_id}/logs"
|
||||
# When using subcollection groups, use just the subcollection name "logs"
|
||||
firestore_field_logs_ttl = firestore.Field(
|
||||
"firestore-field-logs-ttl",
|
||||
project=project_id,
|
||||
database=firestore_db.name,
|
||||
collection="logs", # Subcollection name (applies to all jobs/{job_id}/logs)
|
||||
field="ttl_expiry",
|
||||
ttl_config={}, # Empty block enables TTL based on ttl_expiry field
|
||||
index_config={}, # Disable indexing on TTL field
|
||||
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
|
||||
)
|
||||
|
||||
# =============================================================================
|
||||
# Firestore Composite Indexes for Logs Subcollection
|
||||
# =============================================================================
|
||||
|
||||
# Logs: query by worker, order by timestamp (for filtering logs by worker type)
|
||||
firestore_index_logs_worker_timestamp = firestore.Index(
|
||||
"firestore-index-logs-worker-timestamp",
|
||||
project=project_id,
|
||||
database=firestore_db.name,
|
||||
collection="logs", # Subcollection name
|
||||
fields=[
|
||||
firestore.IndexFieldArgs(field_path="worker", order="ASCENDING"),
|
||||
firestore.IndexFieldArgs(field_path="timestamp", order="ASCENDING"),
|
||||
],
|
||||
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
|
||||
)
|
||||
|
||||
# Create Cloud Storage Bucket
|
||||
bucket = storage.Bucket(
|
||||
"karaoke-storage",
|
||||
@@ -811,6 +847,8 @@ pulumi.export("firestore_index_gen_users_active_created", firestore_index_gen_us
|
||||
pulumi.export("firestore_index_gen_users_active_login", firestore_index_gen_users_active_login.name)
|
||||
pulumi.export("firestore_index_gen_users_active_credits", firestore_index_gen_users_active_credits.name)
|
||||
pulumi.export("firestore_index_gen_users_active_email", firestore_index_gen_users_active_email.name)
|
||||
pulumi.export("firestore_field_logs_ttl", firestore_field_logs_ttl.name)
|
||||
pulumi.export("firestore_index_logs_worker_timestamp", firestore_index_logs_worker_timestamp.name)
|
||||
pulumi.export("service_account_email", service_account.email)
|
||||
pulumi.export("artifact_repo_url", artifact_repo.name.apply(
|
||||
lambda name: f"us-central1-docker.pkg.dev/{project_id}/karaoke-repo"
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
[tool.poetry]
|
||||
name = "karaoke-gen"
|
||||
version = "0.89.6"
|
||||
version = "0.90.0"
|
||||
description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens."
|
||||
authors = ["Andrew Beveridge <andrew@beveridge.uk>"]
|
||||
license = "MIT"
|
||||
|
||||
Reference in new issue
Block a user