Compare commits

...
Author SHA1 Message Date
Andrew Beveridge d0a657610b Added admin API script 2026-01-04 20:32:46 -05:00
Andrew Beveridge ☄️andClaude ae85ec0a09 feat: Move worker logs to Firestore subcollection to avoid 1MB limit (#195)
## Summary

Move `worker_logs` from embedded array in job documents to a Firestore
subcollection (`jobs/{job_id}/logs`) to avoid the 1MB document size
limit. This fixes job failures when logs exceed the limit during
long-running operations like video encoding.

**Problem:** Job `501258e1` failed when logs reached 1.26MB (2,547 log
entries), representing 98.6% of the maximum document size.

## Changes

### New Files
- `backend/models/worker_log.py` - WorkerLogEntry dataclass with factory
methods, serialization (to_dict, to_legacy_dict, from_dict)
- `backend/tests/test_worker_log_subcollection.py` - 24 unit tests
- `backend/tests/emulator/test_worker_logs_subcollection.py` - 11
integration tests using Firestore emulator

### Modified Files
- `backend/services/firestore_service.py` - Added subcollection methods
(append, get, count, delete)
- `backend/services/job_manager.py` - Use subcollection with fallback to
legacy embedded arrays
- `backend/api/routes/jobs.py` - Updated logs endpoint to use new
methods
- `backend/config.py` - Added `USE_LOG_SUBCOLLECTION` feature flag
(default: true)
- `infrastructure/__main__.py` - Added TTL policy (30 days) and
composite index for logs subcollection
- `pyproject.toml` - Version bump 0.89.6 → 0.90.0

### Documentation
- `docs/README.md` - Added to Recent Changes
- `docs/ARCHITECTURE.md` - Added logs subcollection to Firestore
Collections table
- `docs/LESSONS-LEARNED.md` - Added "Firestore Document 1MB Limit with
Embedded Arrays" section

## Testing

- [x] 24 unit tests pass
(`backend/tests/test_worker_log_subcollection.py`)
- [x] 11 emulator integration tests pass
(`backend/tests/emulator/test_worker_logs_subcollection.py`)
- [x] All 68 existing emulator tests pass

## Review

- [x] CodeRabbit CLI review completed locally
- [x] Review feedback addressed (timestamp formatting fix)

## Rollback Plan

Set `USE_LOG_SUBCOLLECTION=false` environment variable and redeploy. New
logs will go back to embedded array; subcollection logs will remain but
won't be read.

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-04 20:14:25 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 a2282357da docs: Add lessons learned for GCE encoding response format handling (#194)
## Summary
Document lessons learned from debugging GCE encoding response format
issues.

## Changes
- Add "External Service Response Format Mismatches" section to
LESSONS-LEARNED.md
- Document defensive type checking patterns for API integration
- Add recent changes entry for GCE encoding fixes (PRs #186-193)
- Reference the worker logs rearchitecture plan

## Context
This session fixed multiple response format bugs in the GCE encoding
integration:
- Status endpoint returning list instead of dict
- output_files as list of paths instead of dict with format keys
- Created plan for worker logs to use Firestore subcollection (avoid 1MB
limit)

These lessons are valuable for future debugging of external service
integrations.

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-05 00:29:23 +00:00
Andrew Beveridge ☄️andClaude Opus 4.5 44c44da95e fix: Handle GCE worker output_files as list of paths (#193)
## Summary
Fix the actual root cause of `'list' object has no attribute 'get'`
error in GCE encoding.

## Problem
The GCE encoding worker returns `output_files` as a **list of paths**:
```json
"output_files": [
    "jobs/501258e1/finals/output_4k_lossless.mp4",
    "jobs/501258e1/finals/output_4k_lossy.mp4",
    "jobs/501258e1/finals/output_720p.mp4"
]
```

But the code expected a **dict** with format keys:
```json
{"mp4_4k_lossless": "path/...", "mp4_4k_lossy": "path/...", ...}
```

This caused the error when calling `.get()` on the list.

## Solution
Added conversion logic to map the list of paths to the expected dict
format by matching filename patterns (e.g., `output_4k_lossless.mp4` ->
`mp4_4k_lossless`).

## Testing
- [x] Verified GCE worker response format
- [ ] Deploy and test with job 501258e1

## Notes
PR #191 fixed a similar issue in `wait_for_completion` where the entire
`status` response could be a list. This PR fixes a different issue where
`output_files` within the response is a list.

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-04 23:23:21 +00:00
Andrew Beveridge ☄️andClaude Opus 4.5 16dc451bdf docs: Add API auth guide and worker logs rearchitecture plan (#192)
## Summary
- Add API authentication documentation to CLAUDE.md for future agents
- Add comprehensive plan for rearchitecting worker logs storage

## Changes

### CLAUDE.md
- New "API Authentication (for Agents)" section covering:
  - Internal worker endpoints with X-Admin-Token
  - Firestore direct access
  - GCE encoding worker SSH commands
  - Common pitfalls to avoid

### .claude/plans/worker-logs-rearchitecture.plan.md
- Plan to move worker_logs from embedded array to Firestore
subcollection
- Fixes 1MB document size limit issue (job 501258e1 hit 1.26 MB)
- Includes TTL policy for automatic cleanup (30 days)

## Testing
- [x] Documentation review
- No code changes

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-04 22:37:17 +00:00
Andrew Beveridge ☄️andClaude Opus 4.5 278690b373 fix: Handle list response in encoding service wait_for_completion (#191)
## Summary
- Fix `'list' object has no attribute 'get'` error in GCE encoding
wait_for_completion
- The GCE encoding worker sometimes returns a list instead of a dict
from the /status endpoint
- PR #186 added handling in encode(), but same issue occurs in
wait_for_completion()

## Changes
- Added defensive handling to convert list responses to dicts in
wait_for_completion()

## Testing
- [x] Code review
- [ ] Deploy and test with job 501258e1

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-04 22:35:55 +00:00
17 changed files with 1860 additions and 35 deletions

No files matched your search

@@ -0,0 +1,222 @@
# Plan: Worker Logs Rearchitecture
**Created:** 2026-01-04
**Branch:** `feat/sess-20260104-1931-logs-research`
**Status:** Implementation Complete - Pending PR Merge & Deployment
## Overview
Move `worker_logs` from embedded array in job documents to a Firestore subcollection to avoid the 1MB document size limit. The current approach caused job 501258e1 to fail when logs reached 1.26 MB (98.6% of document size).
## Problem Statement
- Firestore documents have a 1 MB size limit
- `worker_logs` is stored as an array field within job documents
- Long-running jobs (especially video encoding) can generate thousands of log entries
- When the limit is exceeded, job updates fail with: `Document cannot be written because its size exceeds the maximum allowed size`
## Requirements
- [x] Logs must remain associated with their job
- [x] Logs should be queryable by job_id, worker type, timestamp
- [x] Old logs should auto-cleanup (TTL: 30 days)
- [x] Existing API endpoint (`GET /api/jobs/{id}/logs`) must continue to work
- [x] Frontend must continue to display logs without changes
- [x] Migration path for existing jobs (optional - can just start fresh)
## Technical Approach
### Option A: Subcollection (Recommended)
Store logs in `jobs/{job_id}/logs/{log_id}` subcollection.
**Pros:**
- Natural association with job (delete job = delete logs with cascading)
- Each log entry is its own document (no size limit concerns)
- Can add TTL via Firestore TTL policies
- Efficient queries with composite indexes
**Cons:**
- More Firestore read operations for paginated log fetching
- Slightly more complex write logic
### Option B: Separate Collection
Store logs in `job_logs` collection with `job_id` field.
**Pros:**
- Can query across all jobs
- Simpler TTL management
**Cons:**
- Orphaned logs if job deleted (need cleanup logic)
- Extra index needed for job_id queries
### Decision: Option A (Subcollection)
## Implementation Steps
### Phase 1: Backend Changes
1. [x] **Create log entry model** - `backend/models/worker_log.py`
- `WorkerLogEntry` dataclass with: id, job_id, timestamp, level, worker, message, metadata, ttl_expiry
- Factory method `create()` for easy instantiation
- `to_dict()` for Firestore storage
- `to_legacy_dict()` for API compatibility
- `from_dict()` for Firestore retrieval
2. [x] **Update FirestoreService** - `backend/services/firestore_service.py`
- Added `append_log_to_subcollection(job_id, log_entry)` method
- Added `get_logs_from_subcollection(job_id, limit, since_timestamp, worker, offset)` method
- Added `get_logs_count_from_subcollection(job_id)` method
- Added `delete_logs_subcollection(job_id, batch_size)` method
- Old `append_worker_log` marked as deprecated
3. [x] **Update JobManager** - `backend/services/job_manager.py`
- Modified `append_worker_log()` to use subcollection when `USE_LOG_SUBCOLLECTION=true`
- Modified `get_worker_logs()` to read from subcollection with fallback to embedded array
- Added `get_worker_logs_count()` for total count
- Updated `delete_job()` to clean up logs subcollection
4. [x] **Update jobs API** - `backend/api/routes/jobs.py`
- `GET /api/jobs/{id}/logs` - Updated to use new methods
- Response format stays the same (backward compatible)
- Fixed `next_index` calculation
5. [x] **Add feature flag** - `backend/config.py`
- Added `USE_LOG_SUBCOLLECTION` environment variable (default: true)
6. [x] **Add TTL configuration** - `infrastructure/__main__.py`
- Added Firestore TTL policy for logs subcollection (30 days)
- Added composite index for worker + timestamp queries
### Phase 2: Testing
7. [x] **Unit tests** - `backend/tests/test_worker_log_subcollection.py`
- 24 tests covering:
- WorkerLogEntry model creation and conversion
- FirestoreService subcollection methods
- JobManager logging with feature flag
- Edge cases (unicode, long messages, empty messages)
8. [x] **Integration tests** - `backend/tests/emulator/test_worker_logs_subcollection.py`
- 11 tests using real Firestore emulator:
- Single and sequential writes
- TTL field verification
- Concurrent writes (no race conditions)
- Large volume (500 logs)
- Worker filtering
- Timestamp ordering
- Subcollection deletion
- Concurrent workers interleaved
- Job deletion cleanup
- Comparison with embedded array
### Phase 3: Deployment & Migration
9. [ ] **Deploy with feature flag**
- Add `USE_LOG_SUBCOLLECTION=true` env var (default is true)
- Deploy to production
10. [ ] **Monitor**
- Watch for errors in Cloud Logging
- Verify logs appear in frontend
11. [x] **Backward compatibility**
- Old jobs with embedded arrays still work (fallback in get_worker_logs)
- No migration needed - new jobs use subcollection, old jobs use array
### Phase 4: Cleanup (Future)
12. [ ] **Remove old code path** (after confidence period)
- Remove `worker_logs` field from job model
- Remove old `append_worker_log` method
- Remove feature flag
## Files Created/Modified
| File | Action | Description |
|------|--------|-------------|
| `backend/models/worker_log.py` | Created | WorkerLogEntry dataclass |
| `backend/services/firestore_service.py` | Modified | Added subcollection methods |
| `backend/services/job_manager.py` | Modified | Switch to subcollection with fallback |
| `backend/api/routes/jobs.py` | Modified | Updated logs endpoint |
| `backend/config.py` | Modified | Added USE_LOG_SUBCOLLECTION setting |
| `backend/tests/test_worker_log_subcollection.py` | Created | Unit tests (24 tests) |
| `backend/tests/emulator/test_worker_logs_subcollection.py` | Created | Integration tests (11 tests) |
| `infrastructure/__main__.py` | Modified | Added TTL policy and index |
## Log Entry Schema
```python
@dataclass
class WorkerLogEntry:
timestamp: datetime
level: str # "DEBUG", "INFO", "WARNING", "ERROR"
worker: str # "audio", "lyrics", "video", "render", "screens", "distribution"
message: str
id: str # Auto-generated UUID
job_id: str
ttl_expiry: datetime # timestamp + 30 days
metadata: Optional[Dict[str, Any]] = None
```
## Firestore Structure
```
jobs/
{job_id}/
... (job fields, worker_logs still present for old jobs)
logs/ # subcollection for new jobs
{log_id}/
id: string
job_id: string
timestamp: datetime
level: string
worker: string
message: string
metadata: map (optional)
ttl_expiry: datetime
```
## API Response (Unchanged)
```json
{
"logs": [
{
"timestamp": "2026-01-04T12:00:00Z",
"level": "INFO",
"worker": "video",
"message": "Starting video generation"
}
],
"next_index": 42,
"total_logs": 150
}
```
## Testing Results
All tests pass:
- 24 unit tests in `test_worker_log_subcollection.py`
- 11 emulator integration tests in `test_worker_logs_subcollection.py`
- 68 total tests in emulator suite pass
## Rollback Plan
1. Set `USE_LOG_SUBCOLLECTION=false` env var
2. Redeploy backend
3. New logs will go back to embedded array
4. Subcollection logs will remain but won't be read
## Answers to Open Questions
- [x] **Should we paginate logs in the API?** - Yes, added `offset` parameter support
- [x] **What's the optimal TTL period?** - 30 days (configurable via DEFAULT_LOG_TTL_DAYS)
- [x] **Should we index by timestamp for efficient range queries?** - Yes, added composite index (worker + timestamp)
## References
- [Firestore TTL](https://cloud.google.com/firestore/docs/ttl)
- [Firestore Subcollections](https://firebase.google.com/docs/firestore/data-model#subcollections)
- [Firestore Limits](https://firebase.google.com/docs/firestore/quotas#limits)
- [Pulumi Firestore Field](https://www.pulumi.com/registry/packages/gcp/api-docs/firestore/field/)
+46
View File
@@ -52,6 +52,52 @@ cd frontend && npm run test:all 2>&1 | tail -n 200
- `gcloud` CLI for reading/debugging only
- Stop and notify user on auth issues
## API Authentication (for Agents)
When you need to call backend APIs programmatically:
### Internal Worker Endpoints
Use the `X-Admin-Token` header with the token from Secret Manager:
```bash
# Trigger video worker
curl -X POST "https://api.nomadkaraoke.com/api/internal/workers/video" \
-H "X-Admin-Token: $(gcloud secrets versions access latest --secret=admin-tokens --project=nomadkaraoke)" \
-H "Content-Type: application/json" \
-d '{"job_id": "YOUR_JOB_ID"}'
# Other internal endpoints follow the same pattern
```
### Firestore Direct Access
```bash
# Set the correct project
export GOOGLE_CLOUD_PROJECT=nomadkaraoke
# Use Python with google-cloud-firestore
python3 << 'EOF'
import os
os.environ['GOOGLE_CLOUD_PROJECT'] = 'nomadkaraoke'
from google.cloud import firestore
db = firestore.Client(project='nomadkaraoke')
# Query/update documents...
EOF
```
### GCE Encoding Worker
```bash
# SSH to restart service (clears in-memory job queue)
gcloud compute ssh encoding-worker --zone=us-central1-a --project=nomadkaraoke \
--command="sudo systemctl restart encoding-worker"
# Check health
gcloud compute ssh encoding-worker --zone=us-central1-a --project=nomadkaraoke \
--command="curl -s http://localhost:8080/health"
```
### What Doesn't Work
- `gcloud auth print-identity-token` - Wrong token type for internal endpoints
- Cloud Tasks without proper service account - Gets 401 from Cloud Run
## Documentation Maintenance
### Structure
+7 -3
View File
@@ -1452,10 +1452,14 @@ async def get_worker_logs(
This endpoint returns worker logs stored in Firestore.
Use `since_index` for efficient polling (returns only new logs).
Logs are stored in a subcollection (jobs/{job_id}/logs) to avoid
the 1MB document size limit. Older jobs may have logs in an embedded
array (worker_logs field) - this endpoint handles both transparently.
Args:
job_id: Job ID
since_index: Return only logs after this index (for pagination/polling)
worker: Filter by worker name (audio, lyrics, screens, video, render)
worker: Filter by worker name (audio, lyrics, screens, video, render, distribution)
Returns:
{
@@ -1473,11 +1477,11 @@ async def get_worker_logs(
raise HTTPException(status_code=403, detail="You don't have permission to access logs for this job")
logs = job_manager.get_worker_logs(job_id, since_index=since_index, worker=worker)
total = len(job.worker_logs) if job.worker_logs else 0
total = job_manager.get_worker_logs_count(job_id)
return {
"logs": logs,
"next_index": total,
"next_index": since_index + len(logs),
"total_logs": total
}
+6
View File
@@ -71,6 +71,12 @@ class Settings(BaseSettings):
# Storage paths
temp_dir: str = os.getenv("TEMP_DIR", "/tmp/karaoke-gen")
# Worker logs storage mode
# When enabled, worker logs are stored in a Firestore subcollection (jobs/{job_id}/logs)
# instead of an embedded array. This avoids the 1MB document size limit.
# Default is true for new deployments.
use_log_subcollection: bool = os.getenv("USE_LOG_SUBCOLLECTION", "true").lower() in ("true", "1", "yes")
# Flacfetch remote service (for torrent downloads)
# When configured, audio search uses the remote flacfetch HTTP API instead of local flacfetch.
+164
View File
@@ -0,0 +1,164 @@
"""
Worker log entry model for subcollection storage.
This module defines the LogEntry model for storing worker logs in a Firestore
subcollection instead of an embedded array. This avoids the 1MB document size
limit that caused job 501258e1 to fail when logs reached 1.26 MB.
"""
from dataclasses import dataclass, field
from datetime import datetime, timezone, timedelta
from typing import Optional, Dict, Any
import uuid
# Default TTL for log entries (30 days)
DEFAULT_LOG_TTL_DAYS = 30
@dataclass
class WorkerLogEntry:
"""
Worker log entry for Firestore subcollection storage.
Stored at: jobs/{job_id}/logs/{log_id}
This model is separate from the legacy LogEntry (in job.py) which is
stored as an embedded array in the job document. This new model supports
the subcollection approach with TTL and richer metadata.
"""
# Core fields (required)
timestamp: datetime
level: str # "DEBUG", "INFO", "WARNING", "ERROR"
worker: str # "audio", "lyrics", "screens", "video", "render", "distribution"
message: str
# Identifiers
id: str = field(default_factory=lambda: str(uuid.uuid4()))
job_id: str = "" # Set when writing to Firestore
# TTL for automatic cleanup
ttl_expiry: datetime = field(
default_factory=lambda: datetime.now(timezone.utc) + timedelta(days=DEFAULT_LOG_TTL_DAYS)
)
# Optional metadata for debugging
metadata: Optional[Dict[str, Any]] = None
@classmethod
def create(
cls,
job_id: str,
worker: str,
level: str,
message: str,
metadata: Optional[Dict[str, Any]] = None,
ttl_days: int = DEFAULT_LOG_TTL_DAYS
) -> "WorkerLogEntry":
"""
Factory method to create a new log entry.
Args:
job_id: Job ID this log belongs to
worker: Worker name (audio, lyrics, screens, video, render, distribution)
level: Log level (DEBUG, INFO, WARNING, ERROR)
message: Log message (truncated to 1000 chars)
metadata: Optional additional metadata
ttl_days: Days until log expires (default 30)
Returns:
New WorkerLogEntry instance
"""
now = datetime.now(timezone.utc)
return cls(
id=str(uuid.uuid4()),
job_id=job_id,
timestamp=now,
level=level.upper(),
worker=worker,
message=message[:1000], # Truncate long messages
metadata=metadata,
ttl_expiry=now + timedelta(days=ttl_days)
)
def to_dict(self) -> Dict[str, Any]:
"""
Convert to dictionary for Firestore storage.
Returns:
Dictionary representation for Firestore
"""
result = {
"id": self.id,
"job_id": self.job_id,
"timestamp": self.timestamp,
"level": self.level,
"worker": self.worker,
"message": self.message,
"ttl_expiry": self.ttl_expiry
}
if self.metadata:
result["metadata"] = self.metadata
return result
def to_legacy_dict(self) -> Dict[str, Any]:
"""
Convert to legacy format for backward compatibility with API responses.
The existing API returns logs in this format:
{"timestamp": "...", "level": "INFO", "worker": "audio", "message": "..."}
Returns:
Dictionary in legacy format
"""
# Format timestamp: use "Z" suffix for UTC, isoformat() for others
if self.timestamp.tzinfo is None:
# Naive datetime - use as-is
timestamp_str = self.timestamp.isoformat()
elif self.timestamp.utcoffset() == timedelta(0):
# UTC datetime - replace +00:00 with Z for cleaner format
timestamp_str = self.timestamp.replace(tzinfo=None).isoformat() + "Z"
else:
# Non-UTC timezone-aware - use full isoformat with offset
timestamp_str = self.timestamp.isoformat()
return {
"timestamp": timestamp_str,
"level": self.level,
"worker": self.worker,
"message": self.message
}
@classmethod
def from_dict(cls, data: Dict[str, Any]) -> "WorkerLogEntry":
"""
Create from Firestore document data.
Args:
data: Dictionary from Firestore document
Returns:
WorkerLogEntry instance
"""
# Handle timestamp conversion (Firestore returns datetime objects)
timestamp = data.get("timestamp")
if isinstance(timestamp, str):
# Parse ISO format string
timestamp = datetime.fromisoformat(timestamp.rstrip("Z")).replace(tzinfo=timezone.utc)
ttl_expiry = data.get("ttl_expiry")
if isinstance(ttl_expiry, str):
ttl_expiry = datetime.fromisoformat(ttl_expiry.rstrip("Z")).replace(tzinfo=timezone.utc)
elif ttl_expiry is None:
# Default TTL if not set
ttl_expiry = datetime.now(timezone.utc) + timedelta(days=DEFAULT_LOG_TTL_DAYS)
return cls(
id=data.get("id", str(uuid.uuid4())),
job_id=data.get("job_id", ""),
timestamp=timestamp or datetime.now(timezone.utc),
level=data.get("level", "INFO"),
worker=data.get("worker", "unknown"),
message=data.get("message", ""),
metadata=data.get("metadata"),
ttl_expiry=ttl_expiry
)
+23 -1
View File
@@ -349,7 +349,29 @@ class GCEEncodingBackend(EncodingBackend):
if not isinstance(result, dict):
self.logger.error(f"Unexpected GCE result type: {type(result)}")
result = {}
output_files = result.get("output_files", {})
raw_output_files = result.get("output_files", {})
# Convert output_files from list of paths to dict if needed
# GCE worker returns list like: ["path/output_4k_lossless.mp4", "path/output_720p.mp4"]
# We need dict like: {"mp4_4k_lossless": "path/...", "mp4_720p": "path/..."}
if isinstance(raw_output_files, list):
output_files = {}
for path in raw_output_files:
if not isinstance(path, str):
continue
filename = path.split("/")[-1] if "/" in path else path
# Map filename patterns to output format keys
if "4k_lossless" in filename:
output_files["mp4_4k_lossless"] = path
elif "4k_lossy" in filename:
output_files["mp4_4k_lossy"] = path
elif "720p" in filename:
output_files["mp4_720p"] = path
elif filename.endswith(".mkv"):
output_files["mkv_4k"] = path
self.logger.info(f"Converted output_files list to dict: {output_files}")
else:
output_files = raw_output_files if isinstance(raw_output_files, dict) else {}
return EncodingOutput(
success=True,
+9
View File
@@ -177,6 +177,15 @@ class EncodingService:
raise TimeoutError(f"Encoding job {job_id} timed out after {timeout}s")
status = await self.get_job_status(job_id)
# Handle case where GCE worker returns a list instead of dict
if isinstance(status, list):
logger.warning(f"[job:{job_id}] GCE returned list instead of dict: {status}")
status = status[0] if status and isinstance(status[0], dict) else {}
if not isinstance(status, dict):
logger.error(f"[job:{job_id}] Unexpected status type: {type(status)}")
status = {}
job_status = status.get("status", "unknown")
progress = status.get("progress", 0)
+165 -2
View File
@@ -9,6 +9,7 @@ from google.cloud.firestore_v1 import FieldFilter
from backend.config import settings
from backend.models.job import Job, JobStatus, TimelineEvent
from backend.models.worker_log import WorkerLogEntry
logger = logging.getLogger(__name__)
@@ -245,10 +246,13 @@ class FirestoreService:
def append_worker_log(self, job_id: str, log_entry: Dict[str, Any]) -> None:
"""
Atomically append a log entry to worker_logs using ArrayUnion.
This avoids the race condition of read-modify-write when multiple
workers are logging concurrently.
DEPRECATED: Use append_log_to_subcollection() instead to avoid
the 1MB document size limit.
Args:
job_id: Job ID
log_entry: Log entry dict with timestamp, level, worker, message
@@ -263,6 +267,165 @@ class FirestoreService:
except Exception as e:
# Log but don't raise - logging shouldn't break workers
logger.debug(f"Error appending worker log for job {job_id}: {e}")
# ============================================
# Worker Log Subcollection Methods
# ============================================
# These methods store logs in a subcollection (jobs/{job_id}/logs)
# instead of an embedded array to avoid the 1MB document size limit.
def append_log_to_subcollection(self, job_id: str, log_entry: WorkerLogEntry) -> None:
"""
Append a log entry to the logs subcollection.
Stores logs at: jobs/{job_id}/logs/{log_id}
This approach avoids the 1MB document size limit by storing each
log entry as a separate document in a subcollection.
Args:
job_id: Job ID
log_entry: WorkerLogEntry instance
"""
try:
# Ensure job_id is set on the log entry
log_entry.job_id = job_id
# Get subcollection reference
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
# Add document with auto-generated ID or use log_entry.id
doc_ref = logs_ref.document(log_entry.id)
doc_ref.set(log_entry.to_dict())
# Don't log every append - too spammy
except Exception as e:
# Log but don't raise - logging shouldn't break workers
logger.debug(f"Error appending log to subcollection for job {job_id}: {e}")
def get_logs_from_subcollection(
self,
job_id: str,
limit: int = 500,
since_timestamp: Optional[datetime] = None,
worker: Optional[str] = None,
offset: int = 0
) -> List[WorkerLogEntry]:
"""
Get log entries from the logs subcollection.
Args:
job_id: Job ID
limit: Maximum number of logs to return (default 500)
since_timestamp: Return only logs after this timestamp
worker: Filter by worker name (optional)
offset: Number of logs to skip (for pagination)
Returns:
List of WorkerLogEntry instances, ordered by timestamp ascending
"""
try:
# Get subcollection reference
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
# Build query
query = logs_ref.order_by("timestamp", direction=firestore.Query.ASCENDING)
if since_timestamp:
query = query.where(filter=FieldFilter("timestamp", ">", since_timestamp))
if worker:
query = query.where(filter=FieldFilter("worker", "==", worker))
# Apply offset and limit
if offset > 0:
query = query.offset(offset)
query = query.limit(limit)
# Execute query
docs = query.stream()
logs = [WorkerLogEntry.from_dict(doc.to_dict()) for doc in docs]
return logs
except Exception as e:
logger.error(f"Error getting logs from subcollection for job {job_id}: {e}")
return []
def get_logs_count_from_subcollection(self, job_id: str) -> int:
"""
Get the total count of log entries in the subcollection.
Args:
job_id: Job ID
Returns:
Total count of log entries
"""
try:
# Get subcollection reference
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
# Use aggregation query for efficient counting
count_query = logs_ref.count()
result = count_query.get()
# Result is a list of AggregationResult, we want the first one's count
if result and len(result) > 0:
return result[0][0].value
return 0
except Exception as e:
logger.error(f"Error counting logs for job {job_id}: {e}")
return 0
def delete_logs_subcollection(self, job_id: str, batch_size: int = 500) -> int:
"""
Delete all log entries in the logs subcollection.
This is used when deleting a job to clean up its logs.
Args:
job_id: Job ID
batch_size: Number of documents to delete per batch
Returns:
Number of logs deleted
"""
try:
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
deleted_count = 0
while True:
# Get a batch of documents
docs = logs_ref.limit(batch_size).stream()
deleted_in_batch = 0
# Delete in a batch
batch = self.db.batch()
for doc in docs:
batch.delete(doc.reference)
deleted_in_batch += 1
if deleted_in_batch == 0:
break
batch.commit()
deleted_count += deleted_in_batch
# If we deleted less than batch_size, we're done
if deleted_in_batch < batch_size:
break
if deleted_count > 0:
logger.info(f"Deleted {deleted_count} logs for job {job_id}")
return deleted_count
except Exception as e:
logger.error(f"Error deleting logs subcollection for job {job_id}: {e}")
return 0
# ============================================
# Token Management Methods
+90 -28
View File
@@ -13,7 +13,9 @@ import uuid
from datetime import datetime
from typing import Optional, Dict, Any, List
from backend.config import settings
from backend.models.job import Job, JobStatus, JobCreate, STATE_TRANSITIONS
from backend.models.worker_log import WorkerLogEntry
from backend.services.firestore_service import FirestoreService
from backend.services.storage_service import StorageService
@@ -224,7 +226,7 @@ class JobManager:
return urls
def delete_job(self, job_id: str, delete_files: bool = True) -> None:
"""Delete a job and optionally its files."""
"""Delete a job, its files, and its logs subcollection."""
if delete_files:
job = self.get_job(job_id)
if job and job.output_files:
@@ -233,7 +235,15 @@ class JobManager:
self.storage.delete_file(gcs_path)
except Exception as e:
logger.error(f"Error deleting file {gcs_path}: {e}")
# Delete logs subcollection first (must be done before deleting parent doc)
try:
deleted_logs = self.firestore.delete_logs_subcollection(job_id)
if deleted_logs > 0:
logger.info(f"Deleted {deleted_logs} log entries for job {job_id}")
except Exception as e:
logger.warning(f"Error deleting logs subcollection for job {job_id}: {e}")
self.firestore.delete_job(job_id)
logger.info(f"Deleted job {job_id}")
@@ -579,33 +589,45 @@ class JobManager:
worker: str,
level: str,
message: str,
max_logs: int = 500
max_logs: int = 500,
metadata: Optional[Dict[str, Any]] = None
) -> None:
"""
Append a log entry to the job's worker_logs atomically.
Uses Firestore ArrayUnion for atomic append to avoid race conditions
when multiple workers (audio + lyrics) are logging concurrently.
Note: max_logs is not enforced per-append (would require transactions).
Logs may grow beyond max_logs; trim periodically if needed.
Append a log entry to the job's logs.
By default (USE_LOG_SUBCOLLECTION=true), logs are stored in a Firestore
subcollection (jobs/{job_id}/logs) to avoid the 1MB document size limit.
For backward compatibility (USE_LOG_SUBCOLLECTION=false), logs are stored
in the embedded worker_logs array using atomic ArrayUnion.
Args:
job_id: Job ID
worker: Worker name (audio, lyrics, screens, video, render)
worker: Worker name (audio, lyrics, screens, video, render, distribution)
level: Log level (DEBUG, INFO, WARNING, ERROR)
message: Log message
max_logs: Not used (kept for API compatibility)
metadata: Optional additional metadata dict
"""
log_entry = {
'timestamp': datetime.utcnow().isoformat() + 'Z',
'level': level,
'worker': worker,
'message': message[:1000] # Truncate long messages
}
# Use atomic ArrayUnion to avoid race conditions
self.firestore.append_worker_log(job_id, log_entry)
if settings.use_log_subcollection:
# New subcollection approach - avoids 1MB limit
log_entry = WorkerLogEntry.create(
job_id=job_id,
worker=worker,
level=level,
message=message,
metadata=metadata
)
self.firestore.append_log_to_subcollection(job_id, log_entry)
else:
# Legacy embedded array approach
log_entry = {
'timestamp': datetime.utcnow().isoformat() + 'Z',
'level': level,
'worker': worker,
'message': message[:1000] # Truncate long messages
}
self.firestore.append_worker_log(job_id, log_entry)
def get_worker_logs(
self,
@@ -615,28 +637,68 @@ class JobManager:
) -> List[Dict[str, Any]]:
"""
Get worker logs for a job, optionally filtered by worker and index.
By default (USE_LOG_SUBCOLLECTION=true), logs are read from the
subcollection. Falls back to embedded array for older jobs.
Args:
job_id: Job ID
since_index: Return only logs after this index (for pagination)
worker: Filter by worker name (optional)
Returns:
List of log entries as dicts
List of log entries as dicts (in legacy format for API compatibility)
"""
if settings.use_log_subcollection:
# Try subcollection first
subcollection_logs = self.firestore.get_logs_from_subcollection(
job_id=job_id,
offset=since_index,
worker=worker,
limit=500
)
if subcollection_logs:
# Convert to legacy format for API compatibility
return [log.to_legacy_dict() for log in subcollection_logs]
# Fall through to check embedded array for older jobs
# Embedded array approach (legacy jobs or fallback)
job = self.get_job(job_id)
if not job or not job.worker_logs:
return []
logs = [log.dict() if hasattr(log, 'dict') else log for log in job.worker_logs]
# Filter by index
if since_index > 0:
logs = logs[since_index:]
# Filter by worker
if worker:
logs = [log for log in logs if log.get('worker') == worker]
return logs
def get_worker_logs_count(self, job_id: str) -> int:
"""
Get the total count of worker logs for a job.
Args:
job_id: Job ID
Returns:
Total count of logs
"""
if settings.use_log_subcollection:
# Try subcollection first
count = self.firestore.get_logs_count_from_subcollection(job_id)
if count > 0:
return count
# Fall through to check embedded array
# Embedded array (legacy jobs or fallback)
job = self.get_job(job_id)
if not job or not job.worker_logs:
return 0
return len(job.worker_logs)
@@ -0,0 +1,443 @@
"""
Integration tests for worker log subcollection storage.
Tests the subcollection approach using real Firestore emulator to verify:
- Logs are stored in subcollection (jobs/{job_id}/logs)
- TTL expiry field is set correctly
- Large log volumes work without 1MB limit issues
- Concurrent writes are handled correctly
- Logs can be queried efficiently
Run with: ./scripts/run-emulator-tests.sh
"""
import pytest
import time
import requests
import os
import threading
from concurrent.futures import ThreadPoolExecutor, as_completed
from datetime import datetime, timezone, timedelta
def emulators_running() -> bool:
"""Check if GCP emulators are running."""
try:
requests.get("http://127.0.0.1:8080", timeout=1)
return True
except (requests.exceptions.ConnectionError, requests.exceptions.Timeout):
return False
# Skip all tests in this module if emulators aren't running
pytestmark = pytest.mark.skipif(
not emulators_running(),
reason="GCP emulators not running. Start with: scripts/start-emulators.sh"
)
# Set up environment for emulator
os.environ["FIRESTORE_EMULATOR_HOST"] = "127.0.0.1:8080"
os.environ["GOOGLE_CLOUD_PROJECT"] = "test-project"
class TestWorkerLogsSubcollectionDirect:
"""Direct Firestore tests for worker logs subcollection."""
@pytest.fixture(autouse=True)
def setup_firestore(self):
"""Set up Firestore client for each test."""
from google.cloud import firestore
self.db = firestore.Client(project="test-project")
self.collection = "test-subcollection-jobs"
yield
def _create_test_job(self):
"""Create a test job document."""
job_id = f"test-sub-{int(time.time() * 1000)}"
doc_ref = self.db.collection(self.collection).document(job_id)
doc_ref.set({
"job_id": job_id,
"status": "pending",
"created_at": datetime.now(timezone.utc),
"worker_logs": [] # Legacy field - kept empty for new jobs
})
return job_id
def _append_log_to_subcollection(self, job_id: str, worker: str, message: str, ttl_days: int = 30):
"""Add log to subcollection at jobs/{job_id}/logs."""
import uuid
log_id = str(uuid.uuid4())
now = datetime.now(timezone.utc)
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
logs_ref.document(log_id).set({
"id": log_id,
"job_id": job_id,
"timestamp": now,
"level": "INFO",
"worker": worker,
"message": message,
"ttl_expiry": now + timedelta(days=ttl_days)
})
return log_id
def _get_logs_from_subcollection(self, job_id: str, worker: str = None, limit: int = 500):
"""Get logs from subcollection."""
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
query = logs_ref.order_by("timestamp")
if worker:
from google.cloud.firestore_v1 import FieldFilter
query = query.where(filter=FieldFilter("worker", "==", worker))
query = query.limit(limit)
return [doc.to_dict() for doc in query.stream()]
def _count_logs_in_subcollection(self, job_id: str) -> int:
"""Count logs in subcollection."""
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
count_query = logs_ref.count()
result = count_query.get()
if result and len(result) > 0:
return result[0][0].value
return 0
def _delete_logs_subcollection(self, job_id: str, batch_size: int = 100) -> int:
"""Delete all logs in subcollection."""
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
deleted_count = 0
while True:
docs = list(logs_ref.limit(batch_size).stream())
if not docs:
break
batch = self.db.batch()
for doc in docs:
batch.delete(doc.reference)
deleted_count += 1
batch.commit()
return deleted_count
def test_subcollection_single_write(self):
"""Test writing single log to subcollection."""
job_id = self._create_test_job()
log_id = self._append_log_to_subcollection(job_id, "test", "Single log message")
time.sleep(0.1)
logs = self._get_logs_from_subcollection(job_id)
assert len(logs) == 1
assert logs[0]["message"] == "Single log message"
assert logs[0]["worker"] == "test"
assert logs[0]["id"] == log_id
print("Single write to subcollection works")
def test_subcollection_ttl_field_is_set(self):
"""Test TTL expiry field is set correctly."""
job_id = self._create_test_job()
self._append_log_to_subcollection(job_id, "test", "Log with TTL", ttl_days=7)
time.sleep(0.1)
logs = self._get_logs_from_subcollection(job_id)
assert len(logs) == 1
ttl_expiry = logs[0]["ttl_expiry"]
assert ttl_expiry is not None
# TTL should be ~7 days from now
expected_ttl = datetime.now(timezone.utc) + timedelta(days=7)
if hasattr(ttl_expiry, 'timestamp'):
# Firestore datetime object
diff = abs((ttl_expiry.replace(tzinfo=timezone.utc) - expected_ttl).total_seconds())
else:
diff = 0 # If already datetime
assert diff < 60, f"TTL expiry should be ~7 days from now, diff was {diff}s"
print("TTL field is set correctly")
def test_subcollection_sequential_writes(self):
"""Test multiple sequential writes to subcollection."""
job_id = self._create_test_job()
for i in range(20):
self._append_log_to_subcollection(job_id, "test", f"Log {i}")
time.sleep(0.2)
logs = self._get_logs_from_subcollection(job_id)
assert len(logs) == 20, f"Expected 20 logs, got {len(logs)}"
print(f"Sequential writes: {len(logs)} logs created")
def test_subcollection_concurrent_writes(self):
"""Test concurrent writes to subcollection - no race conditions."""
job_id = self._create_test_job()
num_writes = 50
def write_log(index):
self._append_log_to_subcollection(job_id, "worker", f"Concurrent Log {index}")
with ThreadPoolExecutor(max_workers=10) as executor:
futures = [executor.submit(write_log, i) for i in range(num_writes)]
for future in as_completed(futures):
future.result()
time.sleep(0.5)
logs = self._get_logs_from_subcollection(job_id)
assert len(logs) == num_writes, f"Expected {num_writes} logs, got {len(logs)}"
print(f"Concurrent writes: All {num_writes} logs preserved")
def test_subcollection_large_volume(self):
"""Test large volume of logs - demonstrates no 1MB limit."""
job_id = self._create_test_job()
num_logs = 500 # Would exceed 1MB in embedded array
# Write logs in batches
batch_size = 50
for batch_start in range(0, num_logs, batch_size):
with ThreadPoolExecutor(max_workers=10) as executor:
futures = []
for i in range(batch_start, min(batch_start + batch_size, num_logs)):
futures.append(executor.submit(
self._append_log_to_subcollection,
job_id,
"stress",
f"Large volume test log {i} - " + "x" * 500 # ~500 bytes per log
))
for future in as_completed(futures):
future.result()
time.sleep(1)
count = self._count_logs_in_subcollection(job_id)
assert count == num_logs, f"Expected {num_logs} logs, got {count}"
print(f"Large volume: {count} logs created (~{count * 500 / 1024:.1f}KB would exceed 1MB if embedded)")
def test_subcollection_filter_by_worker(self):
"""Test filtering logs by worker type."""
job_id = self._create_test_job()
# Add logs from different workers
for i in range(10):
self._append_log_to_subcollection(job_id, "audio", f"Audio log {i}")
self._append_log_to_subcollection(job_id, "lyrics", f"Lyrics log {i}")
self._append_log_to_subcollection(job_id, "video", f"Video log {i}")
time.sleep(0.3)
# Query by worker
audio_logs = self._get_logs_from_subcollection(job_id, worker="audio")
lyrics_logs = self._get_logs_from_subcollection(job_id, worker="lyrics")
all_logs = self._get_logs_from_subcollection(job_id)
assert len(audio_logs) == 10, f"Expected 10 audio logs, got {len(audio_logs)}"
assert len(lyrics_logs) == 10, f"Expected 10 lyrics logs, got {len(lyrics_logs)}"
assert len(all_logs) == 30, f"Expected 30 total logs, got {len(all_logs)}"
print("Worker filtering works correctly")
def test_subcollection_ordered_by_timestamp(self):
"""Test logs are returned in timestamp order."""
job_id = self._create_test_job()
# Add logs with slight delays to ensure different timestamps
for i in range(5):
self._append_log_to_subcollection(job_id, "test", f"Log {i}")
time.sleep(0.02)
logs = self._get_logs_from_subcollection(job_id)
# Verify order (timestamps should be ascending)
for i in range(len(logs) - 1):
assert logs[i]["timestamp"] <= logs[i + 1]["timestamp"], \
f"Logs not in order at index {i}"
print("Logs are ordered by timestamp")
def test_subcollection_delete_all_logs(self):
"""Test deleting all logs in subcollection."""
job_id = self._create_test_job()
# Add logs
for i in range(25):
self._append_log_to_subcollection(job_id, "test", f"Log {i}")
time.sleep(0.2)
count_before = self._count_logs_in_subcollection(job_id)
assert count_before == 25
# Delete all logs
deleted = self._delete_logs_subcollection(job_id)
time.sleep(0.2)
count_after = self._count_logs_in_subcollection(job_id)
assert deleted == 25, f"Expected to delete 25 logs, deleted {deleted}"
assert count_after == 0, f"Expected 0 logs after delete, got {count_after}"
print(f"Deleted {deleted} logs from subcollection")
def test_subcollection_concurrent_workers_interleaved(self):
"""Test simulating audio and lyrics workers writing concurrently."""
job_id = self._create_test_job()
logs_per_worker = 20
def audio_worker():
for i in range(logs_per_worker):
self._append_log_to_subcollection(job_id, "audio", f"Audio processing step {i}")
time.sleep(0.005)
def lyrics_worker():
for i in range(logs_per_worker):
self._append_log_to_subcollection(job_id, "lyrics", f"Lyrics processing step {i}")
time.sleep(0.005)
def video_worker():
for i in range(logs_per_worker):
self._append_log_to_subcollection(job_id, "video", f"Video encoding step {i}")
time.sleep(0.005)
# Start all workers
threads = [
threading.Thread(target=audio_worker),
threading.Thread(target=lyrics_worker),
threading.Thread(target=video_worker)
]
for t in threads:
t.start()
for t in threads:
t.join()
time.sleep(0.5)
audio_logs = self._get_logs_from_subcollection(job_id, worker="audio")
lyrics_logs = self._get_logs_from_subcollection(job_id, worker="lyrics")
video_logs = self._get_logs_from_subcollection(job_id, worker="video")
total = self._count_logs_in_subcollection(job_id)
assert len(audio_logs) == logs_per_worker
assert len(lyrics_logs) == logs_per_worker
assert len(video_logs) == logs_per_worker
assert total == logs_per_worker * 3
print(f"Concurrent workers: {len(audio_logs)} audio + {len(lyrics_logs)} lyrics + {len(video_logs)} video = {total}")
def test_subcollection_job_deletion_cleans_up_logs(self):
"""Test that deleting job document doesn't orphan logs."""
job_id = self._create_test_job()
# Add logs
for i in range(10):
self._append_log_to_subcollection(job_id, "test", f"Log {i}")
time.sleep(0.2)
assert self._count_logs_in_subcollection(job_id) == 10
# Delete logs first (must happen before parent doc deletion in real code)
deleted = self._delete_logs_subcollection(job_id)
assert deleted == 10
# Now delete job document
self.db.collection(self.collection).document(job_id).delete()
# Verify subcollection is empty (Firestore doesn't cascade delete subcollections)
time.sleep(0.2)
assert self._count_logs_in_subcollection(job_id) == 0
print("Job deletion with log cleanup works correctly")
class TestSubcollectionVsEmbeddedArray:
"""Compare subcollection approach with embedded array."""
@pytest.fixture(autouse=True)
def setup_firestore(self):
"""Set up Firestore client for each test."""
from google.cloud import firestore
self.db = firestore.Client(project="test-project")
self.collection = "test-comparison-jobs"
yield
def _create_test_job(self, use_array: bool):
"""Create test job."""
job_id = f"test-cmp-{int(time.time() * 1000)}-{'arr' if use_array else 'sub'}"
doc_ref = self.db.collection(self.collection).document(job_id)
doc_ref.set({
"job_id": job_id,
"status": "pending",
"created_at": datetime.now(timezone.utc),
"worker_logs": []
})
return job_id
def _append_log_array(self, job_id: str, message: str):
"""Add log to embedded array."""
from google.cloud import firestore
doc_ref = self.db.collection(self.collection).document(job_id)
doc_ref.update({
"worker_logs": firestore.ArrayUnion([{
"timestamp": datetime.now(timezone.utc).isoformat() + "Z",
"level": "INFO",
"worker": "test",
"message": message
}])
})
def _append_log_subcollection(self, job_id: str, message: str):
"""Add log to subcollection."""
import uuid
now = datetime.now(timezone.utc)
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
logs_ref.document(str(uuid.uuid4())).set({
"timestamp": now,
"level": "INFO",
"worker": "test",
"message": message,
"ttl_expiry": now + timedelta(days=30)
})
def test_document_size_comparison(self):
"""Compare document sizes between approaches."""
array_job_id = self._create_test_job(use_array=True)
sub_job_id = self._create_test_job(use_array=False)
num_logs = 100
large_message = "x" * 5000 # 5KB per log
# Add logs to both
print("\nAdding logs to both approaches...")
for i in range(num_logs):
self._append_log_array(array_job_id, f"Array log {i}: {large_message}")
self._append_log_subcollection(sub_job_id, f"Sub log {i}: {large_message}")
time.sleep(0.5)
# Get document sizes (approximate via raw data)
array_doc = self.db.collection(self.collection).document(array_job_id).get()
sub_doc = self.db.collection(self.collection).document(sub_job_id).get()
array_data = array_doc.to_dict()
sub_data = sub_doc.to_dict()
array_logs = array_data.get("worker_logs", [])
sub_logs = sub_data.get("worker_logs", [])
print(f"\nEmbedded array: {len(array_logs)} logs in document")
print(f"Subcollection: {len(sub_logs)} logs in document (0 expected)")
# Embedded array should have all logs in document
assert len(array_logs) == num_logs
# Subcollection should have empty array in main document
assert len(sub_logs) == 0
# Subcollection logs are in subcollection
sub_count = self.db.collection(self.collection).document(sub_job_id).collection("logs").count().get()[0][0].value
assert sub_count == num_logs
print(f"Subcollection logs stored separately: {sub_count}")
print("Subcollection approach keeps main document small")
print("Worker logs subcollection integration tests ready")
@@ -0,0 +1,509 @@
"""
Unit tests for worker log subcollection functionality.
Tests the WorkerLogEntry model, FirestoreService subcollection methods,
and JobManager log operations with the feature flag.
"""
import pytest
from unittest.mock import Mock, MagicMock, patch
from datetime import datetime, timezone, timedelta
import uuid
# Mock Firestore before imports
import sys
sys.modules['google.cloud.firestore'] = MagicMock()
sys.modules['google.cloud.firestore_v1'] = MagicMock()
from backend.models.worker_log import WorkerLogEntry, DEFAULT_LOG_TTL_DAYS
class TestWorkerLogEntry:
"""Tests for WorkerLogEntry dataclass."""
def test_create_log_entry(self):
"""Test creating a log entry with factory method."""
entry = WorkerLogEntry.create(
job_id="job123",
worker="audio",
level="INFO",
message="Test message"
)
assert entry.job_id == "job123"
assert entry.worker == "audio"
assert entry.level == "INFO"
assert entry.message == "Test message"
assert entry.id is not None
assert entry.timestamp is not None
assert entry.ttl_expiry is not None
# TTL should be ~30 days from now
expected_ttl = datetime.now(timezone.utc) + timedelta(days=DEFAULT_LOG_TTL_DAYS)
assert abs((entry.ttl_expiry - expected_ttl).total_seconds()) < 5
def test_create_log_entry_truncates_long_message(self):
"""Test that messages longer than 1000 chars are truncated."""
long_message = "x" * 2000
entry = WorkerLogEntry.create(
job_id="job123",
worker="audio",
level="INFO",
message=long_message
)
assert len(entry.message) == 1000
def test_create_log_entry_normalizes_level(self):
"""Test that log level is normalized to uppercase."""
entry = WorkerLogEntry.create(
job_id="job123",
worker="audio",
level="info",
message="Test"
)
assert entry.level == "INFO"
def test_create_log_entry_custom_ttl(self):
"""Test creating entry with custom TTL."""
entry = WorkerLogEntry.create(
job_id="job123",
worker="audio",
level="INFO",
message="Test",
ttl_days=7
)
expected_ttl = datetime.now(timezone.utc) + timedelta(days=7)
assert abs((entry.ttl_expiry - expected_ttl).total_seconds()) < 5
def test_create_log_entry_with_metadata(self):
"""Test creating entry with metadata."""
metadata = {"file_size": 1024, "duration": 300}
entry = WorkerLogEntry.create(
job_id="job123",
worker="audio",
level="INFO",
message="Test",
metadata=metadata
)
assert entry.metadata == metadata
def test_to_dict(self):
"""Test converting entry to dict for Firestore."""
entry = WorkerLogEntry.create(
job_id="job123",
worker="audio",
level="INFO",
message="Test message",
metadata={"key": "value"}
)
d = entry.to_dict()
assert d["job_id"] == "job123"
assert d["worker"] == "audio"
assert d["level"] == "INFO"
assert d["message"] == "Test message"
assert d["metadata"] == {"key": "value"}
assert "id" in d
assert "timestamp" in d
assert "ttl_expiry" in d
def test_to_dict_without_metadata(self):
"""Test to_dict doesn't include metadata when None."""
entry = WorkerLogEntry.create(
job_id="job123",
worker="audio",
level="INFO",
message="Test"
)
d = entry.to_dict()
assert "metadata" not in d
def test_to_legacy_dict(self):
"""Test converting to legacy format for API compatibility."""
entry = WorkerLogEntry.create(
job_id="job123",
worker="audio",
level="INFO",
message="Test message"
)
d = entry.to_legacy_dict()
assert "timestamp" in d
assert d["level"] == "INFO"
assert d["worker"] == "audio"
assert d["message"] == "Test message"
# Should not include id, job_id, ttl_expiry, metadata
assert "id" not in d
assert "job_id" not in d
assert "ttl_expiry" not in d
def test_from_dict(self):
"""Test creating entry from Firestore document."""
timestamp = datetime.now(timezone.utc)
ttl_expiry = timestamp + timedelta(days=30)
data = {
"id": "log123",
"job_id": "job123",
"timestamp": timestamp,
"level": "WARNING",
"worker": "video",
"message": "Warning message",
"metadata": {"error_code": 500},
"ttl_expiry": ttl_expiry
}
entry = WorkerLogEntry.from_dict(data)
assert entry.id == "log123"
assert entry.job_id == "job123"
assert entry.level == "WARNING"
assert entry.worker == "video"
assert entry.message == "Warning message"
assert entry.metadata == {"error_code": 500}
def test_from_dict_with_iso_strings(self):
"""Test creating entry from dict with ISO format strings."""
data = {
"timestamp": "2026-01-04T12:00:00Z",
"level": "INFO",
"worker": "audio",
"message": "Test"
}
entry = WorkerLogEntry.from_dict(data)
assert entry.timestamp.tzinfo == timezone.utc
assert entry.message == "Test"
def test_from_dict_missing_fields(self):
"""Test creating entry with missing fields uses defaults."""
data = {
"message": "Minimal log"
}
entry = WorkerLogEntry.from_dict(data)
assert entry.message == "Minimal log"
assert entry.level == "INFO" # Default
assert entry.worker == "unknown" # Default
assert entry.id is not None # Generated
class TestFirestoreServiceSubcollection:
"""Tests for FirestoreService log subcollection methods."""
@pytest.fixture
def mock_db(self):
"""Create mock Firestore database."""
return MagicMock()
@pytest.fixture
def firestore_service(self, mock_db):
"""Create FirestoreService with mocked DB."""
with patch('backend.services.firestore_service.firestore') as mock_firestore:
mock_firestore.Client.return_value = mock_db
from backend.services.firestore_service import FirestoreService
service = FirestoreService()
service.db = mock_db
return service
def test_append_log_to_subcollection(self, firestore_service, mock_db):
"""Test appending log to subcollection."""
entry = WorkerLogEntry.create(
job_id="job123",
worker="audio",
level="INFO",
message="Test message"
)
# Setup mock chain
mock_collection = MagicMock()
mock_doc = MagicMock()
mock_logs_ref = MagicMock()
mock_log_doc = MagicMock()
mock_db.collection.return_value = mock_collection
mock_collection.document.return_value = mock_doc
mock_doc.collection.return_value = mock_logs_ref
mock_logs_ref.document.return_value = mock_log_doc
firestore_service.append_log_to_subcollection("job123", entry)
# Verify correct path: jobs/{job_id}/logs/{log_id}
mock_db.collection.assert_called_with("jobs")
mock_collection.document.assert_called_with("job123")
mock_doc.collection.assert_called_with("logs")
mock_logs_ref.document.assert_called_with(entry.id)
mock_log_doc.set.assert_called_once()
def test_get_logs_from_subcollection(self, firestore_service, mock_db):
"""Test getting logs from subcollection."""
# Setup mock chain
mock_collection = MagicMock()
mock_doc = MagicMock()
mock_logs_ref = MagicMock()
mock_query = MagicMock()
mock_db.collection.return_value = mock_collection
mock_collection.document.return_value = mock_doc
mock_doc.collection.return_value = mock_logs_ref
mock_logs_ref.order_by.return_value = mock_query
mock_query.limit.return_value = mock_query
# Mock document data
mock_doc1 = MagicMock()
mock_doc1.to_dict.return_value = {
"timestamp": datetime.now(timezone.utc),
"level": "INFO",
"worker": "audio",
"message": "Log 1"
}
mock_query.stream.return_value = iter([mock_doc1])
logs = firestore_service.get_logs_from_subcollection("job123")
assert len(logs) == 1
assert logs[0].message == "Log 1"
mock_logs_ref.order_by.assert_called_once()
def test_get_logs_from_subcollection_with_worker_filter(self, firestore_service, mock_db):
"""Test filtering logs by worker."""
mock_collection = MagicMock()
mock_doc = MagicMock()
mock_logs_ref = MagicMock()
mock_query = MagicMock()
mock_db.collection.return_value = mock_collection
mock_collection.document.return_value = mock_doc
mock_doc.collection.return_value = mock_logs_ref
mock_logs_ref.order_by.return_value = mock_query
mock_query.where.return_value = mock_query
mock_query.limit.return_value = mock_query
mock_query.stream.return_value = iter([])
firestore_service.get_logs_from_subcollection("job123", worker="audio")
# Verify where clause was added
mock_query.where.assert_called()
def test_delete_logs_subcollection(self, firestore_service, mock_db):
"""Test deleting all logs in subcollection."""
mock_collection = MagicMock()
mock_doc = MagicMock()
mock_logs_ref = MagicMock()
mock_batch = MagicMock()
mock_db.collection.return_value = mock_collection
mock_collection.document.return_value = mock_doc
mock_doc.collection.return_value = mock_logs_ref
mock_db.batch.return_value = mock_batch
# First call returns 2 docs, second call returns empty
mock_log_doc1 = MagicMock()
mock_log_doc2 = MagicMock()
call_count = [0]
def stream_side_effect(*args, **kwargs):
call_count[0] += 1
if call_count[0] == 1:
return iter([mock_log_doc1, mock_log_doc2])
return iter([])
mock_logs_ref.limit.return_value = mock_logs_ref
mock_logs_ref.stream.side_effect = stream_side_effect
deleted = firestore_service.delete_logs_subcollection("job123")
assert deleted == 2
mock_batch.commit.assert_called()
class TestJobManagerLogging:
"""Tests for JobManager log methods with feature flag."""
@pytest.fixture
def mock_firestore_service(self):
"""Mock FirestoreService."""
with patch('backend.services.job_manager.FirestoreService') as mock:
service = MagicMock()
mock.return_value = service
yield service
@pytest.fixture
def mock_storage_service(self):
"""Mock StorageService."""
with patch('backend.services.job_manager.StorageService') as mock:
service = MagicMock()
mock.return_value = service
yield service
def test_append_worker_log_uses_subcollection_when_enabled(
self, mock_firestore_service, mock_storage_service
):
"""Test that logs go to subcollection when feature is enabled."""
with patch('backend.services.job_manager.settings') as mock_settings:
mock_settings.use_log_subcollection = True
from backend.services.job_manager import JobManager
manager = JobManager()
manager.append_worker_log(
job_id="job123",
worker="audio",
level="INFO",
message="Test message"
)
# Should call subcollection method
mock_firestore_service.append_log_to_subcollection.assert_called_once()
# Should NOT call legacy method
mock_firestore_service.append_worker_log.assert_not_called()
def test_append_worker_log_uses_array_when_disabled(
self, mock_firestore_service, mock_storage_service
):
"""Test that logs go to embedded array when feature is disabled."""
with patch('backend.services.job_manager.settings') as mock_settings:
mock_settings.use_log_subcollection = False
from backend.services.job_manager import JobManager
manager = JobManager()
manager.append_worker_log(
job_id="job123",
worker="audio",
level="INFO",
message="Test message"
)
# Should call legacy method
mock_firestore_service.append_worker_log.assert_called_once()
# Should NOT call subcollection method
mock_firestore_service.append_log_to_subcollection.assert_not_called()
def test_get_worker_logs_from_subcollection(
self, mock_firestore_service, mock_storage_service
):
"""Test getting logs from subcollection."""
with patch('backend.services.job_manager.settings') as mock_settings:
mock_settings.use_log_subcollection = True
entry = WorkerLogEntry.create(
job_id="job123",
worker="audio",
level="INFO",
message="Test"
)
mock_firestore_service.get_logs_from_subcollection.return_value = [entry]
from backend.services.job_manager import JobManager
manager = JobManager()
logs = manager.get_worker_logs("job123")
assert len(logs) == 1
assert logs[0]["message"] == "Test"
mock_firestore_service.get_logs_from_subcollection.assert_called_once()
def test_get_worker_logs_falls_back_to_array(
self, mock_firestore_service, mock_storage_service
):
"""Test fallback to embedded array for legacy jobs."""
with patch('backend.services.job_manager.settings') as mock_settings:
mock_settings.use_log_subcollection = True
# Subcollection returns empty
mock_firestore_service.get_logs_from_subcollection.return_value = []
# Mock job with legacy logs
from backend.models.job import Job, JobStatus
mock_job = Job(
job_id="job123",
status=JobStatus.PENDING,
created_at=datetime.now(timezone.utc),
updated_at=datetime.now(timezone.utc),
worker_logs=[
{"timestamp": "2026-01-04T12:00:00Z", "level": "INFO", "worker": "audio", "message": "Legacy log"}
]
)
mock_firestore_service.get_job.return_value = mock_job
from backend.services.job_manager import JobManager
manager = JobManager()
logs = manager.get_worker_logs("job123")
assert len(logs) == 1
assert logs[0]["message"] == "Legacy log"
def test_delete_job_deletes_logs_subcollection(
self, mock_firestore_service, mock_storage_service
):
"""Test that deleting a job also deletes its logs subcollection."""
mock_firestore_service.get_job.return_value = None
mock_firestore_service.delete_logs_subcollection.return_value = 5
from backend.services.job_manager import JobManager
manager = JobManager()
manager.delete_job("job123", delete_files=False)
mock_firestore_service.delete_logs_subcollection.assert_called_once_with("job123")
mock_firestore_service.delete_job.assert_called_once_with("job123")
class TestWorkerLogEntryEdgeCases:
"""Edge case tests for WorkerLogEntry."""
def test_create_with_empty_message(self):
"""Test creating entry with empty message."""
entry = WorkerLogEntry.create(
job_id="job123",
worker="audio",
level="INFO",
message=""
)
assert entry.message == ""
def test_create_with_unicode_message(self):
"""Test creating entry with Unicode characters."""
entry = WorkerLogEntry.create(
job_id="job123",
worker="audio",
level="INFO",
message="Processing song: 日本語の曲 - アーティスト"
)
assert "日本語" in entry.message
def test_create_with_newlines_in_message(self):
"""Test creating entry with newlines."""
entry = WorkerLogEntry.create(
job_id="job123",
worker="audio",
level="ERROR",
message="Error occurred:\nLine 1\nLine 2"
)
assert "\n" in entry.message
def test_from_dict_handles_missing_ttl_expiry(self):
"""Test from_dict creates default TTL when missing."""
data = {
"timestamp": datetime.now(timezone.utc),
"level": "INFO",
"worker": "audio",
"message": "Test"
}
entry = WorkerLogEntry.from_dict(data)
assert entry.ttl_expiry is not None
# Should be ~30 days from now
expected = datetime.now(timezone.utc) + timedelta(days=30)
assert abs((entry.ttl_expiry - expected).total_seconds()) < 10
+3
View File
@@ -163,12 +163,15 @@ karaoke-gen shares a GCP project (`nomadkaraoke`) with karaoke-decide, but uses
|------------|---------|------------|
| `gen_users` | karaoke-gen user accounts | email, credits, role, is_active |
| `jobs` | Karaoke generation jobs | job_id, user_email, status, state_data |
| `jobs/{job_id}/logs` | Worker log entries (subcollection) | timestamp, level, worker, message, ttl_expiry |
| `sessions` | Magic link auth sessions | user_email, token, expires_at |
| `magic_links` | Passwordless auth tokens | email, token, expires_at, used |
| `beta_feedback` | Beta program feedback | user_email, ratings, comments |
**Note**: The `users` collection in the same Firestore instance belongs to karaoke-decide (different schema: user_id, is_guest, quiz_* fields). Don't use it for karaoke-gen.
**Worker Logs**: Stored in subcollection (`jobs/{job_id}/logs`) instead of embedded array to avoid Firestore's 1MB document size limit. TTL policy auto-deletes logs after 30 days. Feature flag: `USE_LOG_SUBCOLLECTION` (default: true). See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#firestore-document-1mb-limit-with-embedded-arrays).
## Video Worker Orchestrator
The Video Worker uses an orchestrator pattern to ensure all features work regardless of encoding backend (local Cloud Run or GCE VM).
+89
View File
@@ -1038,6 +1038,95 @@ result = await orchestrator.run()
See `docs/archive/2026-01-04-video-worker-orchestrator-refactor.md` for full details.
### Firestore Document 1MB Limit with Embedded Arrays
**Problem**: Firestore documents have a 1MB size limit. Storing logs as an embedded array (`worker_logs: [{...}, {...}, ...]`) within job documents caused job failures when logs exceeded the limit during long-running operations like video encoding.
**Example**: Job `501258e1` failed when logs reached 1.26MB (2,547 log entries), representing 98.6% of the maximum document size. The job update failed with:
```
Document cannot be written because its size exceeds the maximum allowed size
```
**Root cause**: Video encoding jobs generate thousands of log entries (FFmpeg output, progress updates). Each log entry is ~500 bytes. 2,000+ entries easily exceed 1MB.
**Solution**: Store logs in a Firestore subcollection instead of embedded array:
```python
# BAD - embedded array hits 1MB limit
doc_ref.update({
"worker_logs": firestore.ArrayUnion([{"message": "..."}])
})
# GOOD - subcollection has no size limit per parent doc
logs_ref = db.collection("jobs").document(job_id).collection("logs")
logs_ref.document(log_id).set({
"timestamp": now,
"level": "INFO",
"worker": "video",
"message": "...",
"ttl_expiry": now + timedelta(days=30)
})
```
**Key benefits of subcollection approach**:
1. Each log entry is its own document (no size limit concerns)
2. Natural association with parent job (path: `jobs/{job_id}/logs/{log_id}`)
3. Can add TTL via Firestore TTL policies for automatic cleanup
4. Efficient queries with composite indexes (worker + timestamp)
5. Concurrent workers can write without race conditions (unlike ArrayUnion which can conflict)
**Migration strategy**: Used feature flag (`USE_LOG_SUBCOLLECTION=true`) for rollback capability. New jobs use subcollection; old jobs with embedded arrays continue to work via fallback read logic.
**Lesson**: Never use embedded arrays for unbounded data. If an array can grow indefinitely based on operation duration or user input, use a subcollection instead. The 1MB limit applies to the entire document, including all fields.
### External Service Response Format Mismatches
**Problem**: The GCE encoding worker returned responses in unexpected formats, causing `'list' object has no attribute 'get'` errors that were hard to debug through the async pipeline.
**Issues encountered**:
1. **Status endpoint returning list instead of dict**: The `/status/{job_id}` endpoint sometimes returned a list `[{...}]` instead of a dict `{...}`. Calling `.get()` on the list caused failures.
2. **output_files as list of paths instead of dict**: The worker returned:
```json
"output_files": ["jobs/id/finals/output_4k_lossless.mp4", "jobs/id/finals/output_720p.mp4"]
```
But the backend expected:
```json
"output_files": {"mp4_4k_lossless": "path/...", "mp4_720p": "path/..."}
```
**Symptoms**:
- Job fails late in pipeline (during encoding completion handling)
- Error message is cryptic (`'list' object has no attribute 'get'`)
- Hard to reproduce locally (depends on GCE worker state)
- Multiple similar errors from different root causes
**Solution**: Add defensive type checking for all external service responses:
```python
# Always check response type before accessing
if isinstance(status, list):
logger.warning(f"GCE returned list instead of dict: {status}")
status = status[0] if status and isinstance(status[0], dict) else {}
if not isinstance(status, dict):
status = {}
# Convert list of paths to expected dict format
if isinstance(output_files, list):
output_dict = {}
for path in output_files:
filename = path.split("/")[-1]
if "4k_lossless" in filename:
output_dict["mp4_4k_lossless"] = path
# ... etc
output_files = output_dict
```
**Key insight**: When integrating with external services, never assume the response format. Add explicit type checks and conversions, with logging when unexpected formats are encountered. This is especially important for services you don't control (internal microservices, third-party APIs).
**Debugging tip**: When hitting format errors in async pipelines, check the raw response from the external service directly (curl with API key) before assuming the bug is in your code.
## What We'd Do Differently
1. **Add Pydantic model fields test first** - Would have caught the silent field issue immediately
+4
View File
@@ -36,6 +36,10 @@
## Recent Changes
- **Worker Logs Subcollection** (2026-01-04): Moved `worker_logs` from embedded array in job documents to Firestore subcollection (`jobs/{job_id}/logs`). Fixes job failures when logs exceed 1MB (job 501258e1 had 1.26MB of logs). New logs stored with 30-day TTL via Firestore TTL policy. Feature flag `USE_LOG_SUBCOLLECTION=true` (default). See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#firestore-document-1mb-limit-with-embedded-arrays).
- **GCE Encoding Response Fixes** (2026-01-04): Fixed multiple response format mismatches with GCE encoding worker. The worker returns `output_files` as a list of paths, not a dict with format keys - added conversion logic. Also added defensive type checking for status responses that could be lists. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#external-service-response-format-mismatches).
- **Video Worker Orchestrator** (2026-01-04): Major refactor to unify video generation pipeline. Created VideoWorkerOrchestrator that coordinates all stages (packaging, encoding, distribution, notifications) regardless of encoding backend (GCE or local). Fixes issue where GCE encoding path bypassed YouTube upload, Discord notifications, and CDG/TXT packaging. Feature flag `USE_NEW_ORCHESTRATOR` (default: true) enables rollback. 139 new tests across 6 new service modules. See [ARCHITECTURE.md](ARCHITECTURE.md#video-worker-orchestrator) and [archive/2026-01-04-video-worker-orchestrator-refactor.md](archive/2026-01-04-video-worker-orchestrator-refactor.md).
- **Full Unicode Font Support** (2026-01-03): Fixed rendering of musical symbols (♪) and added comprehensive international font support to Docker base image. Installed Noto fonts covering Latin, CJK (Chinese/Japanese/Korean), Arabic, Hebrew, Thai, and other scripts. Changed default karaoke font from Arial to Noto Sans. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#fonts-in-docker-for-video-rendering).
+38
View File
@@ -144,6 +144,42 @@ firestore_index_gen_users_active_email = firestore.Index(
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
)
# =============================================================================
# Firestore TTL Field Configuration
# TTL (Time-to-Live) policies automatically delete documents after expiration.
# =============================================================================
# Worker logs TTL: logs subcollection documents expire after 30 days
# Collection path for subcollections: "jobs/{job_id}/logs"
# When using subcollection groups, use just the subcollection name "logs"
firestore_field_logs_ttl = firestore.Field(
"firestore-field-logs-ttl",
project=project_id,
database=firestore_db.name,
collection="logs", # Subcollection name (applies to all jobs/{job_id}/logs)
field="ttl_expiry",
ttl_config={}, # Empty block enables TTL based on ttl_expiry field
index_config={}, # Disable indexing on TTL field
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
)
# =============================================================================
# Firestore Composite Indexes for Logs Subcollection
# =============================================================================
# Logs: query by worker, order by timestamp (for filtering logs by worker type)
firestore_index_logs_worker_timestamp = firestore.Index(
"firestore-index-logs-worker-timestamp",
project=project_id,
database=firestore_db.name,
collection="logs", # Subcollection name
fields=[
firestore.IndexFieldArgs(field_path="worker", order="ASCENDING"),
firestore.IndexFieldArgs(field_path="timestamp", order="ASCENDING"),
],
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
)
# Create Cloud Storage Bucket
bucket = storage.Bucket(
"karaoke-storage",
@@ -811,6 +847,8 @@ pulumi.export("firestore_index_gen_users_active_created", firestore_index_gen_us
pulumi.export("firestore_index_gen_users_active_login", firestore_index_gen_users_active_login.name)
pulumi.export("firestore_index_gen_users_active_credits", firestore_index_gen_users_active_credits.name)
pulumi.export("firestore_index_gen_users_active_email", firestore_index_gen_users_active_email.name)
pulumi.export("firestore_field_logs_ttl", firestore_field_logs_ttl.name)
pulumi.export("firestore_index_logs_worker_timestamp", firestore_index_logs_worker_timestamp.name)
pulumi.export("service_account_email", service_account.email)
pulumi.export("artifact_repo_url", artifact_repo.name.apply(
lambda name: f"us-central1-docker.pkg.dev/{project_id}/karaoke-repo"
+1 -1
View File
@@ -1,6 +1,6 @@
[tool.poetry]
name = "karaoke-gen"
version = "0.89.4"
version = "0.90.0"
description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens."
authors = ["Andrew Beveridge <andrew@beveridge.uk>"]
license = "MIT"
+41
View File
@@ -0,0 +1,41 @@
#!/bin/bash
# Admin API helper script
# Usage: ./scripts/admin_api.sh <endpoint> [method] [data]
#
# Examples:
# ./scripts/admin_api.sh /api/jobs # List all jobs (admin sees all)
# ./scripts/admin_api.sh /api/jobs?limit=10 # List 10 jobs
# ./scripts/admin_api.sh /api/jobs/abc123 # Get specific job
# ./scripts/admin_api.sh /api/jobs/abc123/retry POST # Retry a job
# ./scripts/admin_api.sh /api/jobs/abc123/select-instrumental POST '{"selection":"clean"}'
set -e
API_URL="${KARAOKE_API_URL:-https://api.nomadkaraoke.com}"
# Get admin token from Secret Manager if not set
if [ -z "$KARAOKE_ADMIN_TOKEN" ]; then
KARAOKE_ADMIN_TOKEN=$(gcloud secrets versions access latest --secret=admin-tokens --project=nomadkaraoke 2>/dev/null)
fi
if [ -z "$KARAOKE_ADMIN_TOKEN" ]; then
echo "Error: Could not get admin token. Run 'gcloud auth login' first." >&2
exit 1
fi
ENDPOINT="${1:-/api/admin/jobs}"
METHOD="${2:-GET}"
DATA="$3"
if [ "$METHOD" = "GET" ]; then
curl -s "${API_URL}${ENDPOINT}" \
-H "Authorization: Bearer $KARAOKE_ADMIN_TOKEN" | jq .
elif [ -n "$DATA" ]; then
curl -s -X "$METHOD" "${API_URL}${ENDPOINT}" \
-H "Authorization: Bearer $KARAOKE_ADMIN_TOKEN" \
-H "Content-Type: application/json" \
-d "$DATA" | jq .
else
curl -s -X "$METHOD" "${API_URL}${ENDPOINT}" \
-H "Authorization: Bearer $KARAOKE_ADMIN_TOKEN" | jq .
fi