Improve job observability with timeline metadata and log_to_job (#508)

@coderabbitai ignore

## Summary
- Add `metadata` field to `TimelineEvent` for structured output details
(brand codes, YouTube video IDs, Dropbox paths, GDrive file IDs)
- Add `log_to_job()` helper for API endpoints to write to Firestore job
log subcollections
- Plumb `timeline_metadata` through `job_manager.transition_to_state()`
and `firestore_service.update_job_status()`
- Capture output details in completion timeline events for full
traceability
- Log edit/delete cleanup operations to both Cloud Logging and Firestore
job logs
- Snapshot previous outputs before cleanup for audit trail
- Enhance distribution logging with video_id, folder path, and file IDs

## Context
After shipping "edit completed tracks," tracing edited jobs was
difficult because timeline events lacked output details, cleanup
operations weren't logged to Firestore, and previous outputs were
overwritten with no history. These changes make the job's Firestore
document self-documenting — any agent or human can read the timeline and
understand the full lifecycle.

## Test plan
- [x] 7 new unit tests for `TimelineEvent.metadata` and `log_to_job()`
helper
- [x] All 103 backend tests pass
- [ ] Verify in production after deploy: edit a test job and confirm
timeline metadata appears in Firestore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Andrew Beveridge ☄️andClaude Opus 4.6 authored and GitHub committed 2026-03-07 23:08:17 -05:00
1 parent e04337fa5d
commit 9d8bf3a952
12 files changed
+427 -20

No files matched your search

@@ -0,0 +1,149 @@
# Plan: Improve Job Observability
**Created:** 2026-03-07
**Branch:** feat/sess-20260307-2216-improve-job-observability
**Status:** Draft
## Overview
After shipping the "edit completed tracks" feature, we found it very difficult to trace what actually happened to edited jobs. Timeline events are too sparse (no output details), cleanup operations aren't logged to Cloud Logging or Firestore, state_data gets fully overwritten during re-processing (losing history of previous outputs), and API endpoints don't write to the job's Firestore log subcollection.
The goal is to make it trivially easy for future agents and humans to reconstruct the complete lifecycle of any job by reading its timeline and Firestore logs.
## Problem Analysis
### What we found investigating two edited jobs:
1. **Timeline events only have `status`, `timestamp`, `progress`, `message`** — no output details like YouTube video IDs, brand codes, Dropbox links, or GDrive file IDs
2. **Edit endpoint cleanup results never logged** — the `cleanup_results` dict is built and returned in the HTTP response but never logged to Cloud Logging or the job's Firestore log subcollection
3. **Distribution details incomplete in logs** — YouTube logs the URL but not video_id separately; Dropbox doesn't log folder path; GDrive doesn't log file IDs
4. **`state_data` fully overwritten on re-processing** — `video_worker.py:342-354` does `{**job.state_data, ...}`, so any `DELETE_FIELD` operations from the edit endpoint get lost when old keys are re-merged from the in-memory copy
5. **`outputs_deleted_at` cleared by video worker** — intentionally, but loses audit trail
6. **No Firestore job logs from API endpoints** — edit/admin endpoints only log to Cloud Run stdout, not to `jobs/{job_id}/logs/`
## Requirements
- [ ] Add `metadata` dict to `TimelineEvent` for structured output details
- [ ] Log all cleanup operations (success AND failure) to both Cloud Logging and Firestore job logs
- [ ] Log distribution output details (YouTube video ID, Dropbox path, brand code, GDrive file IDs) in timeline events and Firestore logs
- [ ] Preserve previous output details in timeline when a job is edited (snapshot before cleanup)
- [ ] Add helper to write Firestore job logs from API endpoints (not just workers)
- [ ] Ensure edit/admin delete endpoints write to Firestore job logs
- [ ] No breaking changes to existing timeline format (additive only)
## Technical Approach
### 1. Add `metadata` field to `TimelineEvent`
The `TimelineEvent` model gains an optional `metadata: Optional[Dict[str, Any]]` field. This allows any timeline event to carry structured data without changing the schema. The field is additive — existing events without metadata continue to work.
Key events that gain metadata:
- **`complete`**: `{youtube_url, youtube_video_id, dropbox_link, brand_code, gdrive_file_ids}`
- **Edit initiated**: `{previous_outputs: {youtube_url, dropbox_link, brand_code, gdrive_files}, cleanup_results: {...}}`
- **Outputs deleted**: `{deleted_outputs: {...}, cleanup_results: {...}}`
### 2. Write Firestore job logs from API endpoints
Create a simple helper function that API route handlers can call to append logs to the job's Firestore subcollection. Workers already have `JobLogAdapter` — we just need a lightweight equivalent for API routes:
```python
def log_to_job(job_id: str, worker: str, level: str, message: str, metadata: dict = None):
"""Write a log entry to the job's Firestore log subcollection."""
entry = WorkerLogEntry.create(job_id, worker, level, message, metadata)
FirestoreService().append_log_to_subcollection(job_id, entry)
```
### 3. Enhance distribution logging in video_worker_orchestrator
After each distribution upload succeeds, log the details with structured metadata:
- YouTube: video_id, video_url, channel
- Dropbox: folder_path, sharing_link
- GDrive: file_ids dict
- Brand code: code, prefix, allocated vs recycled
### 4. Snapshot previous outputs on edit
Before the edit endpoint clears state_data distribution keys, capture the current values in the timeline event's metadata. This preserves the history permanently in the timeline array (which is append-only via `ArrayUnion`).
### 5. Log cleanup results from edit/admin endpoints
After cleanup runs, log each step's result to both Cloud Logging and the Firestore job log subcollection with structured metadata.
## Implementation Steps
### Step 1: Add `metadata` to `TimelineEvent` model
- [ ] Add `metadata: Optional[Dict[str, Any]] = None` to `TimelineEvent` in `backend/models/job.py`
- [ ] No migration needed — Firestore is schemaless, existing events just won't have the field
### Step 2: Create `log_to_job` helper
- [ ] Add `log_to_job()` function in `backend/services/firestore_service.py` (or a small utility module)
- [ ] Takes `job_id`, `worker`, `level`, `message`, optional `metadata`
- [ ] Wraps `WorkerLogEntry.create()` + `append_log_to_subcollection()`
- [ ] Catches and suppresses errors (logging should never break the caller)
### Step 3: Update `transition_to_state` to accept metadata
- [ ] Add optional `timeline_metadata: Optional[Dict[str, Any]] = None` parameter to `job_manager.transition_to_state()`
- [ ] Pass through to `update_job_status()` → `TimelineEvent` construction
- [ ] Update `firestore_service.update_job_status()` to accept and pass `timeline_metadata`
### Step 4: Enhance edit endpoint logging
- [ ] Before cleanup: snapshot current distribution outputs (`youtube_url`, `dropbox_link`, `brand_code`, `gdrive_files`) into `previous_outputs` dict
- [ ] After each cleanup step: log result to Cloud Logging AND call `log_to_job()`
- [ ] After all cleanup: include `previous_outputs` and `cleanup_results` in the timeline event metadata
- [ ] The "Track edit initiated" timeline event gains metadata: `{initiated_by, previous_outputs, cleanup_results}`
### Step 5: Enhance admin delete-outputs endpoint logging
- [ ] Same pattern as edit endpoint: snapshot outputs, log each cleanup step, include in timeline metadata
- [ ] The "Outputs deleted by admin" timeline event gains metadata: `{deleted_by, deleted_outputs, cleanup_results}`
### Step 6: Enhance distribution logging in video_worker_orchestrator
- [ ] YouTube upload success: log video_id and video_url with metadata `{video_id, video_url}`
- [ ] Dropbox upload success: log folder_path and sharing_link with metadata `{folder_path, sharing_link}`
- [ ] GDrive upload success: log file_ids with metadata `{file_ids: {...}}`
- [ ] Brand code allocation: log with metadata `{brand_code, source: "allocated"|"recycled"}`
### Step 7: Add output details to `complete` timeline event
- [ ] When `video_worker.py` calls `transition_to_state(COMPLETE)`, pass `timeline_metadata` with distribution output details
- [ ] Include: `youtube_url`, `youtube_video_id`, `dropbox_link`, `brand_code`, `gdrive_file_ids`, `edit_count`
### Step 8: Tests
- [ ] Unit test: `TimelineEvent` with metadata serializes correctly
- [ ] Unit test: `log_to_job` writes to subcollection
- [ ] Unit test: edit endpoint includes `previous_outputs` and `cleanup_results` in timeline event
- [ ] Unit test: `complete` transition includes output metadata
- [ ] Verify existing tests still pass
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/models/job.py` | Modify | Add `metadata` field to `TimelineEvent` |
| `backend/services/firestore_service.py` | Modify | Add `log_to_job()` helper, update `update_job_status()` to accept timeline metadata |
| `backend/services/job_manager.py` | Modify | Add `timeline_metadata` param to `transition_to_state()` |
| `backend/api/routes/jobs.py` | Modify | Enhance edit endpoint with cleanup logging, output snapshots, timeline metadata |
| `backend/api/routes/admin.py` | Modify | Enhance delete-outputs endpoint similarly |
| `backend/workers/video_worker_orchestrator.py` | Modify | Add structured metadata to distribution log messages |
| `backend/workers/video_worker.py` | Modify | Pass output metadata in `complete` transition (both orchestrator and legacy paths) |
| `backend/tests/test_edit_completed_track.py` | Modify | Verify timeline metadata in edit tests |
| `backend/tests/test_observability.py` | Create | Tests for TimelineEvent metadata and log_to_job |
## Testing Strategy
- **Unit tests**: TimelineEvent metadata serialization, log_to_job helper, edit endpoint timeline metadata assertions
- **Existing test verification**: `make test` must pass — changes are additive so existing tests should be unaffected
- **No E2E tests needed**: This is purely backend logging/observability — no UI changes
## What This Does NOT Include
- **Dedicated admin UI for viewing job logs** — existing admin page already shows timeline
- **Log aggregation/search UI** — Cloud Logging and Firestore viewer already serve this purpose
- **Changing state_data overwrite behavior** — the `{**job.state_data, ...}` pattern in video_worker.py is intentional; instead we preserve history in timeline events
- **Changing outputs_deleted_at clearing** — video worker clearing it after re-processing is correct behavior; the edit history lives in timeline metadata instead
## Rollback Plan
All changes are additive (new optional field on TimelineEvent, additional log writes). Rolling back just means deploying the previous version — no data migration needed.
## Open Questions
None — the approach is straightforward and low-risk.
+25 -1
View File
@@ -22,6 +22,7 @@ from backend.services.flacfetch_client import get_flacfetch_client, FlacfetchSer
from backend.services.storage_service import StorageService
from backend.services.audio_search_service import get_audio_search_service, NoResultsError, AudioSearchError
from backend.models.job import JobStatus
from backend.services.firestore_service import log_to_job
from backend.utils.test_data import is_test_email
from karaoke_gen.utils import sanitize_filename
@@ -1591,6 +1592,19 @@ async def delete_job_outputs(
"gdrive": {"status": "skipped", "reason": "no gdrive_files in state_data"},
}
# Snapshot current outputs before cleanup (preserved in timeline metadata)
deleted_outputs = {}
if state_data.get('youtube_url'):
deleted_outputs['youtube_url'] = state_data['youtube_url']
if state_data.get('dropbox_link'):
deleted_outputs['dropbox_link'] = state_data['dropbox_link']
if state_data.get('brand_code'):
deleted_outputs['brand_code'] = state_data['brand_code']
if state_data.get('gdrive_files'):
deleted_outputs['gdrive_files'] = state_data['gdrive_files']
log_to_job(job_id, "admin", "INFO", f"Output deletion initiated by {admin_email}", {"deleted_outputs": deleted_outputs})
# Clean up YouTube
youtube_url = state_data.get('youtube_url')
if youtube_url:
@@ -1717,11 +1731,17 @@ async def delete_job_outputs(
update_payload[f"state_data.{key}"] = DELETE_FIELD
cleared_keys.append(key)
# Add timeline event
# Add timeline event with structured metadata
timeline_event = {
"status": job.status, # Keep current status
"timestamp": deletion_timestamp.isoformat(),
"message": f"Outputs deleted by admin ({admin_email})",
"metadata": {
"action": "outputs_deleted",
"deleted_by": admin_email,
"deleted_outputs": deleted_outputs,
"cleanup_results": results,
},
}
update_payload["timeline"] = ArrayUnion([timeline_event])
@@ -1755,6 +1775,10 @@ async def delete_job_outputs(
f"Brand Code: {brand_code_status}. "
f"Cleared state_data keys: {cleared_keys}"
)
log_to_job(job_id, "admin", "INFO", f"Outputs deleted by {admin_email}: {overall_status}", {
"cleanup_results": results,
"cleared_keys": cleared_keys,
})
return DeleteOutputsResponse(
status=overall_status,
+29 -2
View File
@@ -40,7 +40,7 @@ from backend.services.metrics import metrics
from backend.middleware.tenant import get_tenant_from_request
from backend.utils.test_data import is_test_email
from backend.exceptions import InsufficientCreditsError
from backend.services.firestore_service import FirestoreService
from backend.services.firestore_service import FirestoreService, log_to_job
from backend.models.requests import ChangeVisibilityRequest, ChangeVisibilityResponse
from pydantic import BaseModel, Field, validator
@@ -272,6 +272,19 @@ async def edit_completed_track(
state_data = job.state_data or {}
cleanup_results = {}
# Snapshot current outputs before cleanup (preserved in timeline metadata)
previous_outputs = {}
if state_data.get('youtube_url'):
previous_outputs['youtube_url'] = state_data['youtube_url']
if state_data.get('dropbox_link'):
previous_outputs['dropbox_link'] = state_data['dropbox_link']
if state_data.get('brand_code'):
previous_outputs['brand_code'] = state_data['brand_code']
if state_data.get('gdrive_files'):
previous_outputs['gdrive_files'] = state_data['gdrive_files']
log_to_job(job_id, "edit", "INFO", f"Edit initiated by {user_email}", {"previous_outputs": previous_outputs})
# --- Phase 1: Clean up distributed outputs ---
# Delete YouTube video
@@ -301,6 +314,7 @@ async def edit_completed_track(
cleanup_results["youtube"] = {"status": "error", "error": str(e)}
else:
cleanup_results["youtube"] = {"status": "skipped", "reason": "no youtube_url"}
log_to_job(job_id, "edit", "INFO", f"YouTube cleanup: {cleanup_results['youtube'].get('status')}", cleanup_results["youtube"])
# Delete Dropbox folder
brand_code = state_data.get('brand_code')
@@ -322,6 +336,7 @@ async def edit_completed_track(
cleanup_results["dropbox"] = {"status": "error", "error": str(e)}
else:
cleanup_results["dropbox"] = {"status": "skipped", "reason": "no brand_code or dropbox_path"}
log_to_job(job_id, "edit", "INFO", f"Dropbox cleanup: {cleanup_results['dropbox'].get('status')}", cleanup_results["dropbox"])
# Delete Google Drive files
gdrive_files = state_data.get('gdrive_files')
@@ -341,6 +356,7 @@ async def edit_completed_track(
cleanup_results["gdrive"] = {"status": "error", "error": str(e)}
else:
cleanup_results["gdrive"] = {"status": "skipped", "reason": "no gdrive_files"}
log_to_job(job_id, "edit", "INFO", f"GDrive cleanup: {cleanup_results['gdrive'].get('status')}", cleanup_results["gdrive"])
# Delete GCS finals folder
try:
@@ -427,12 +443,19 @@ async def edit_completed_track(
if "screens_progress" in state_data:
update_payload[f"state_data.screens_progress"] = DELETE_FIELD
# Add timeline event
# Add timeline event with structured metadata
edit_number = (job.edit_count if hasattr(job, 'edit_count') else 0) + 1
timeline_event = {
"status": "complete",
"timestamp": now.isoformat(),
"message": f"Track edit initiated by {user_email} (edit #{edit_number})",
"metadata": {
"action": "edit_initiated",
"initiated_by": user_email,
"edit_number": edit_number,
"previous_outputs": previous_outputs,
"cleanup_results": cleanup_results,
},
}
update_payload["timeline"] = ArrayUnion([timeline_event])
@@ -466,6 +489,10 @@ async def edit_completed_track(
f"Dropbox={cleanup_results.get('dropbox', {}).get('status')}, "
f"GDrive={cleanup_results.get('gdrive', {}).get('status')}"
)
log_to_job(job_id, "edit", "INFO", f"Edit #{edit_number} complete: reopened for review", {
"cleanup_results": cleanup_results,
"metadata_updated": metadata_updated,
})
return EditTrackResponse(
status="success",
+1
View File
@@ -179,6 +179,7 @@ class TimelineEvent(BaseModel):
timestamp: str
progress: Optional[int] = None
message: Optional[str] = None
metadata: Optional[Dict[str, Any]] = None
class LogEntry(BaseModel):
+36 -2
View File
@@ -75,18 +75,20 @@ class FirestoreService:
status: JobStatus,
progress: Optional[int] = None,
message: Optional[str] = None,
timeline_metadata: Optional[Dict[str, Any]] = None,
**additional_fields
) -> None:
"""Update job status and add timeline event."""
try:
doc_ref = self.db.collection(self.collection).document(job_id)
# Create timeline event
timeline_event = TimelineEvent(
status=status.value,
timestamp=datetime.utcnow().isoformat(),
progress=progress,
message=message
message=message,
metadata=timeline_metadata,
)
# Prepare updates
@@ -710,3 +712,35 @@ def get_firestore_client() -> firestore.Client:
_firestore_client = firestore.Client(project=settings.google_cloud_project)
return _firestore_client
_log_service: Optional["FirestoreService"] = None
def _get_log_service() -> "FirestoreService":
"""Return a cached FirestoreService instance for log_to_job."""
global _log_service
if _log_service is None:
_log_service = FirestoreService()
return _log_service
def log_to_job(job_id: str, worker: str, level: str, message: str, metadata: Optional[Dict[str, Any]] = None) -> None:
"""
Write a log entry to a job's Firestore log subcollection.
This is a convenience function for API endpoints that need to log
to a job's log history (workers already have JobLogAdapter for this).
Args:
job_id: Job ID to log to
worker: Worker/source name (e.g., "api", "edit", "admin")
level: Log level (INFO, WARNING, ERROR)
message: Log message
metadata: Optional structured metadata dict
"""
try:
entry = WorkerLogEntry.create(job_id, worker, level, message, metadata)
_get_log_service().append_log_to_subcollection(job_id, entry)
except Exception as e:
logger.debug(f"Failed to write job log for {job_id}: {e}")
+11 -7
View File
@@ -413,7 +413,8 @@ class JobManager:
progress: Optional[int] = None,
message: Optional[str] = None,
state_data_updates: Optional[Dict[str, Any]] = None,
raise_on_invalid: bool = True
raise_on_invalid: bool = True,
timeline_metadata: Optional[Dict[str, Any]] = None,
) -> bool:
"""
Transition job to new state with validation.
@@ -426,6 +427,7 @@ class JobManager:
state_data_updates: Updates to state_data field
raise_on_invalid: If True (default), raise InvalidStateTransitionError
on invalid transitions. If False, return False silently.
timeline_metadata: Optional structured metadata to attach to the timeline event
Returns:
True if transition succeeded, False if failed (only when raise_on_invalid=False)
@@ -435,15 +437,15 @@ class JobManager:
"""
if not self.validate_state_transition(job_id, new_status, raise_on_invalid=raise_on_invalid):
return False
updates = {
'status': new_status,
'updated_at': datetime.utcnow()
}
if progress is not None:
updates['progress'] = progress
# Generate review token when entering AWAITING_REVIEW state
# Tokens don't expire - they're job-scoped so low risk, and natural expiry happens when job completes
if new_status == JobStatus.AWAITING_REVIEW:
@@ -452,14 +454,14 @@ class JobManager:
updates['review_token'] = review_token
updates['review_token_expires_at'] = None # No expiry - token is job-scoped
logger.info(f"Generated review token for job {job_id} (no expiry)")
# If we have state_data_updates, merge them with existing state_data
merged_state_data = None
if state_data_updates:
job = self.get_job(job_id)
if job:
merged_state_data = {**job.state_data, **state_data_updates}
# Update job status (includes timeline event), passing state_data if present
if merged_state_data is not None:
self.update_job_status(
@@ -467,6 +469,7 @@ class JobManager:
status=new_status,
progress=progress,
message=message,
timeline_metadata=timeline_metadata,
state_data=merged_state_data
)
else:
@@ -474,7 +477,8 @@ class JobManager:
job_id=job_id,
status=new_status,
progress=progress,
message=message
message=message,
timeline_metadata=timeline_metadata,
)
# Apply review token update separately if generated
+132
View File
@@ -0,0 +1,132 @@
"""
Tests for job observability improvements.
Covers:
- TimelineEvent metadata field
- log_to_job helper function
- Timeline metadata in state transitions
"""
import pytest
from unittest.mock import patch, MagicMock
from datetime import datetime, timezone
from backend.models.job import TimelineEvent
class TestTimelineEventMetadata:
"""Tests for the metadata field on TimelineEvent."""
def test_timeline_event_without_metadata(self):
"""Existing events without metadata still work."""
event = TimelineEvent(
status="complete",
timestamp="2026-03-07T12:00:00",
progress=100,
message="Done!",
)
data = event.model_dump(mode='json')
assert data["status"] == "complete"
assert data["metadata"] is None
def test_timeline_event_with_metadata(self):
"""Events can include structured metadata."""
metadata = {
"action": "completed",
"brand_code": "NOMAD-1234",
"youtube_url": "https://youtube.com/watch?v=abc123",
"gdrive_file_ids": {"mp4": "file1", "cdg": "file2"},
}
event = TimelineEvent(
status="complete",
timestamp="2026-03-07T12:00:00",
progress=100,
message="Karaoke generation complete!",
metadata=metadata,
)
data = event.model_dump(mode='json')
assert data["metadata"]["brand_code"] == "NOMAD-1234"
assert data["metadata"]["youtube_url"] == "https://youtube.com/watch?v=abc123"
assert data["metadata"]["gdrive_file_ids"]["mp4"] == "file1"
def test_timeline_event_with_edit_metadata(self):
"""Edit-initiated events include previous outputs and cleanup results."""
metadata = {
"action": "edit_initiated",
"initiated_by": "user@example.com",
"edit_number": 1,
"previous_outputs": {
"youtube_url": "https://youtube.com/watch?v=old123",
"brand_code": "NOMAD-0042",
"dropbox_link": "https://dropbox.com/sh/old",
},
"cleanup_results": {
"youtube": {"status": "success", "video_id": "old123"},
"dropbox": {"status": "success", "path": "/Karaoke/NOMAD-0042"},
"gdrive": {"status": "skipped", "reason": "no gdrive_files"},
},
}
event = TimelineEvent(
status="complete",
timestamp="2026-03-07T12:00:00",
message="Track edit initiated by user@example.com (edit #1)",
metadata=metadata,
)
data = event.model_dump(mode='json')
assert data["metadata"]["previous_outputs"]["youtube_url"] == "https://youtube.com/watch?v=old123"
assert data["metadata"]["cleanup_results"]["youtube"]["status"] == "success"
def test_timeline_event_metadata_roundtrip(self):
"""Metadata survives serialization/deserialization."""
metadata = {"brand_code": "NOMAD-1234", "nested": {"key": "value"}}
event = TimelineEvent(
status="complete",
timestamp="2026-03-07T12:00:00",
metadata=metadata,
)
data = event.model_dump(mode='json')
restored = TimelineEvent(**data)
assert restored.metadata == metadata
class TestLogToJob:
"""Tests for the log_to_job helper function."""
@patch('backend.services.firestore_service._get_log_service')
def test_log_to_job_writes_to_subcollection(self, mock_get_service):
"""log_to_job creates a WorkerLogEntry and writes to Firestore."""
mock_service = MagicMock()
mock_get_service.return_value = mock_service
from backend.services.firestore_service import log_to_job
log_to_job("job123", "edit", "INFO", "Edit started", {"key": "value"})
mock_service.append_log_to_subcollection.assert_called_once()
call_args = mock_service.append_log_to_subcollection.call_args
assert call_args[0][0] == "job123"
entry = call_args[0][1]
assert entry.worker == "edit"
assert entry.level == "INFO"
assert entry.message == "Edit started"
assert entry.metadata == {"key": "value"}
@patch('backend.services.firestore_service._get_log_service')
def test_log_to_job_suppresses_errors(self, mock_get_service):
"""log_to_job should never raise — logging failures must be silent."""
mock_get_service.side_effect = Exception("Firestore down")
from backend.services.firestore_service import log_to_job
# Should not raise
log_to_job("job123", "edit", "INFO", "Edit started")
@patch('backend.services.firestore_service._get_log_service')
def test_log_to_job_without_metadata(self, mock_get_service):
"""log_to_job works without metadata."""
mock_service = MagicMock()
mock_get_service.return_value = mock_service
from backend.services.firestore_service import log_to_job
log_to_job("job123", "admin", "WARNING", "Something happened")
mock_service.append_log_to_subcollection.assert_called_once()
entry = mock_service.append_log_to_subcollection.call_args[0][1]
assert entry.metadata is None
+27 -4
View File
@@ -364,14 +364,27 @@ async def generate_video_orchestrated(job_id: str) -> bool:
)
# Mark job as complete (triggers completion email with youtube_url now available)
duration = time.time() - start_time
completion_metadata = {
"action": "completed",
"brand_code": result.brand_code,
"youtube_url": result.youtube_url,
"youtube_upload_queued": result.youtube_upload_queued,
"dropbox_link": result.dropbox_link,
"gdrive_file_ids": result.gdrive_files,
"edit_count": getattr(job, 'edit_count', 0),
"duration_seconds": round(duration, 1),
}
if result.distribution_warnings:
completion_metadata["distribution_warnings"] = result.distribution_warnings
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.COMPLETE,
progress=100,
message="Karaoke generation complete!"
message="Karaoke generation complete!",
timeline_metadata=completion_metadata,
)
duration = time.time() - start_time
root_span.set_attribute("duration_seconds", duration)
root_span.set_attribute("brand_code", result.brand_code or '')
logger.info(f"[job:{job_id}] WORKER_END worker=video orchestrator=true status=success duration={duration:.1f}s")
@@ -859,14 +872,24 @@ async def generate_video_legacy(job_id: str) -> bool:
})
# Mark job as complete (triggers completion email with youtube_url now available)
duration = time.time() - start_time
completion_metadata = {
"action": "completed",
"brand_code": result.get('brand_code'),
"youtube_url": result.get('youtube_url'),
"dropbox_link": result.get('dropbox_link'),
"gdrive_file_ids": result.get('gdrive_files'),
"edit_count": getattr(job, 'edit_count', 0),
"duration_seconds": round(duration, 1),
}
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.COMPLETE,
progress=100,
message="Karaoke generation complete!"
message="Karaoke generation complete!",
timeline_metadata=completion_metadata,
)
duration = time.time() - start_time
root_span.set_attribute("duration_seconds", duration)
root_span.set_attribute("brand_code", result.get('brand_code', ''))
logger.info(f"[job:{job_id}] WORKER_END worker=video status=success duration={duration:.1f}s")
+3 -3
View File
@@ -638,7 +638,7 @@ class VideoWorkerOrchestrator:
if video_url:
self.result.youtube_url = video_url
self.job_log.info(f"Uploaded to YouTube: {video_url}")
self.job_log.info(f"Uploaded to YouTube: {video_url} (video_id={video_id})")
# Record upload in pending buffer (bridges ~7min GCP monitoring delay)
try:
@@ -715,7 +715,7 @@ class VideoWorkerOrchestrator:
except Exception as e:
self.job_log.warning(f"Failed to create Dropbox sharing link: {e}")
self.job_log.info("Dropbox upload complete")
self.job_log.info(f"Dropbox upload complete: {remote_folder}")
except Exception as e:
self.job_log.error(f"Dropbox upload failed: {e}")
@@ -752,7 +752,7 @@ class VideoWorkerOrchestrator:
)
self.result.gdrive_files = uploaded
self.job_log.info(f"Google Drive upload complete: {len(uploaded)} files")
self.job_log.info(f"Google Drive upload complete: {len(uploaded)} files (ids={uploaded})")
except Exception as e:
self.job_log.error(f"Google Drive upload failed: {e}")
+11
View File
@@ -885,3 +885,14 @@ Even with idempotent endpoints (PR #413) and basic retry logic (PR #242), jobs c
**Key insight:** When investigating "what did the client send?", Cloud Trace request size can serve as a forensic tool even without request body logging. Compact JSON serialization (`JSON.stringify` with no spaces) is deterministic, so byte-level size matching is reliable.
### Timeline Metadata for Job Traceability (2026-03-07)
**Problem:** After shipping the "edit completed tracks" feature, tracing what happened to edited jobs was very difficult. Timeline events only had `status`, `timestamp`, `progress`, `message` — no output details. Cleanup operations (YouTube deletion, Dropbox removal) weren't logged to Firestore. Previous outputs were overwritten in `state_data` with no history. API endpoints (edit, admin-delete) logged to Cloud Run stdout but not to the job's Firestore log subcollection.
**Solution:**
1. **`TimelineEvent.metadata`** — Optional `Dict[str, Any]` field on timeline events. Completion events now capture `brand_code`, `youtube_url`, `dropbox_link`, `gdrive_file_ids`, `duration_seconds`. Edit/delete events capture `previous_outputs` and `cleanup_results`.
2. **`log_to_job()` helper** — Module-level function in `firestore_service.py` that writes to the job's Firestore log subcollection. Wraps errors silently so logging never breaks the caller. Used by API endpoints that previously only logged to stdout.
3. **Plumbed `timeline_metadata`** through `job_manager.transition_to_state()` → `firestore_service.update_job_status()`.
**Key insight:** The job's Firestore document (timeline + log subcollection) should be the single source of truth for "what happened to this job." Cloud Logging has retention limits and requires knowing which service to search. Timeline metadata makes the job self-documenting — any agent or human can read the timeline and understand the full lifecycle without cross-referencing logs.
+2
View File
@@ -45,6 +45,8 @@
## Recent Changes
- **Job Observability Improvements** (2026-03-07): Enhanced timeline events with a `metadata` field to capture structured output details (brand codes, YouTube video IDs, Dropbox paths, GDrive file IDs) at key moments — job completion, edit initiation, and admin output deletion. Added `log_to_job()` helper for API endpoints to write to Firestore job log subcollections (previously only workers logged there). Edit and admin-delete endpoints now snapshot previous outputs before cleanup and log each cleanup step. Distribution logging enhanced with video IDs, folder paths, and file IDs. See [plans/improve-job-observability.plan.md](../.claude/plans/improve-job-observability.plan.md).
- **Edit Completed Tracks** (2026-03-06): Users can now edit completed karaoke tracks — correcting lyrics, changing instrumental selection, or updating artist/title metadata — without consuming additional credits. Clicking "Edit" on a completed job opens a confirmation modal (with optional artist/title editing), then cleans up all distributed outputs (YouTube video deleted, Dropbox folder removed, GDrive files deleted, GCS finals purged, brand code recycled), resets the job to `awaiting_review`, and redirects to the existing review screen. After re-review, the standard pipeline re-renders and re-distributes with a fresh brand code. New endpoint: `POST /api/jobs/{job_id}/edit`. New Job model field: `edit_count`. State machine updated: `COMPLETE → AWAITING_REVIEW` transition allowed. See [archive/2026-03-06-edit-completed-tracks-plan.md](archive/2026-03-06-edit-completed-tracks-plan.md) and [API.md](API.md#edit-completed-track).
- **Change Visibility** (2026-03-06): Users can now toggle completed jobs between public and private via a "Make Public"/"Make Private" button on the job card. Two distinct flows: Public→Private is fast (~1-2 min, redistributes existing finals), Private→Public is slow (~15-30 min, resets styles to Nomad theme, regenerates screens, re-renders video). Confirmation modal explains the implications. Tenant jobs excluded. See [archive/2026-03-06-change-visibility-plan.md](archive/2026-03-06-change-visibility-plan.md) and [API.md](API.md#change-visibility).
+1 -1
View File
@@ -1,6 +1,6 @@
[tool.poetry]
name = "karaoke-gen"
version = "0.137.1"
version = "0.137.2"
description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens."
authors = ["Andrew Beveridge <andrew@beveridge.uk>"]
license = "MIT"