diff --git a/.claude/plans/improve-job-observability.plan.md b/.claude/plans/improve-job-observability.plan.md new file mode 100644 index 00000000..e883d266 --- /dev/null +++ b/.claude/plans/improve-job-observability.plan.md @@ -0,0 +1,149 @@ +# Plan: Improve Job Observability + +**Created:** 2026-03-07 +**Branch:** feat/sess-20260307-2216-improve-job-observability +**Status:** Draft + +## Overview + +After shipping the "edit completed tracks" feature, we found it very difficult to trace what actually happened to edited jobs. Timeline events are too sparse (no output details), cleanup operations aren't logged to Cloud Logging or Firestore, state_data gets fully overwritten during re-processing (losing history of previous outputs), and API endpoints don't write to the job's Firestore log subcollection. + +The goal is to make it trivially easy for future agents and humans to reconstruct the complete lifecycle of any job by reading its timeline and Firestore logs. + +## Problem Analysis + +### What we found investigating two edited jobs: + +1. **Timeline events only have `status`, `timestamp`, `progress`, `message`** — no output details like YouTube video IDs, brand codes, Dropbox links, or GDrive file IDs +2. **Edit endpoint cleanup results never logged** — the `cleanup_results` dict is built and returned in the HTTP response but never logged to Cloud Logging or the job's Firestore log subcollection +3. **Distribution details incomplete in logs** — YouTube logs the URL but not video_id separately; Dropbox doesn't log folder path; GDrive doesn't log file IDs +4. **`state_data` fully overwritten on re-processing** — `video_worker.py:342-354` does `{**job.state_data, ...}`, so any `DELETE_FIELD` operations from the edit endpoint get lost when old keys are re-merged from the in-memory copy +5. **`outputs_deleted_at` cleared by video worker** — intentionally, but loses audit trail +6. **No Firestore job logs from API endpoints** — edit/admin endpoints only log to Cloud Run stdout, not to `jobs/{job_id}/logs/` + +## Requirements + +- [ ] Add `metadata` dict to `TimelineEvent` for structured output details +- [ ] Log all cleanup operations (success AND failure) to both Cloud Logging and Firestore job logs +- [ ] Log distribution output details (YouTube video ID, Dropbox path, brand code, GDrive file IDs) in timeline events and Firestore logs +- [ ] Preserve previous output details in timeline when a job is edited (snapshot before cleanup) +- [ ] Add helper to write Firestore job logs from API endpoints (not just workers) +- [ ] Ensure edit/admin delete endpoints write to Firestore job logs +- [ ] No breaking changes to existing timeline format (additive only) + +## Technical Approach + +### 1. Add `metadata` field to `TimelineEvent` + +The `TimelineEvent` model gains an optional `metadata: Optional[Dict[str, Any]]` field. This allows any timeline event to carry structured data without changing the schema. The field is additive — existing events without metadata continue to work. + +Key events that gain metadata: +- **`complete`**: `{youtube_url, youtube_video_id, dropbox_link, brand_code, gdrive_file_ids}` +- **Edit initiated**: `{previous_outputs: {youtube_url, dropbox_link, brand_code, gdrive_files}, cleanup_results: {...}}` +- **Outputs deleted**: `{deleted_outputs: {...}, cleanup_results: {...}}` + +### 2. Write Firestore job logs from API endpoints + +Create a simple helper function that API route handlers can call to append logs to the job's Firestore subcollection. Workers already have `JobLogAdapter` — we just need a lightweight equivalent for API routes: + +```python +def log_to_job(job_id: str, worker: str, level: str, message: str, metadata: dict = None): + """Write a log entry to the job's Firestore log subcollection.""" + entry = WorkerLogEntry.create(job_id, worker, level, message, metadata) + FirestoreService().append_log_to_subcollection(job_id, entry) +``` + +### 3. Enhance distribution logging in video_worker_orchestrator + +After each distribution upload succeeds, log the details with structured metadata: +- YouTube: video_id, video_url, channel +- Dropbox: folder_path, sharing_link +- GDrive: file_ids dict +- Brand code: code, prefix, allocated vs recycled + +### 4. Snapshot previous outputs on edit + +Before the edit endpoint clears state_data distribution keys, capture the current values in the timeline event's metadata. This preserves the history permanently in the timeline array (which is append-only via `ArrayUnion`). + +### 5. Log cleanup results from edit/admin endpoints + +After cleanup runs, log each step's result to both Cloud Logging and the Firestore job log subcollection with structured metadata. + +## Implementation Steps + +### Step 1: Add `metadata` to `TimelineEvent` model +- [ ] Add `metadata: Optional[Dict[str, Any]] = None` to `TimelineEvent` in `backend/models/job.py` +- [ ] No migration needed — Firestore is schemaless, existing events just won't have the field + +### Step 2: Create `log_to_job` helper +- [ ] Add `log_to_job()` function in `backend/services/firestore_service.py` (or a small utility module) +- [ ] Takes `job_id`, `worker`, `level`, `message`, optional `metadata` +- [ ] Wraps `WorkerLogEntry.create()` + `append_log_to_subcollection()` +- [ ] Catches and suppresses errors (logging should never break the caller) + +### Step 3: Update `transition_to_state` to accept metadata +- [ ] Add optional `timeline_metadata: Optional[Dict[str, Any]] = None` parameter to `job_manager.transition_to_state()` +- [ ] Pass through to `update_job_status()` → `TimelineEvent` construction +- [ ] Update `firestore_service.update_job_status()` to accept and pass `timeline_metadata` + +### Step 4: Enhance edit endpoint logging +- [ ] Before cleanup: snapshot current distribution outputs (`youtube_url`, `dropbox_link`, `brand_code`, `gdrive_files`) into `previous_outputs` dict +- [ ] After each cleanup step: log result to Cloud Logging AND call `log_to_job()` +- [ ] After all cleanup: include `previous_outputs` and `cleanup_results` in the timeline event metadata +- [ ] The "Track edit initiated" timeline event gains metadata: `{initiated_by, previous_outputs, cleanup_results}` + +### Step 5: Enhance admin delete-outputs endpoint logging +- [ ] Same pattern as edit endpoint: snapshot outputs, log each cleanup step, include in timeline metadata +- [ ] The "Outputs deleted by admin" timeline event gains metadata: `{deleted_by, deleted_outputs, cleanup_results}` + +### Step 6: Enhance distribution logging in video_worker_orchestrator +- [ ] YouTube upload success: log video_id and video_url with metadata `{video_id, video_url}` +- [ ] Dropbox upload success: log folder_path and sharing_link with metadata `{folder_path, sharing_link}` +- [ ] GDrive upload success: log file_ids with metadata `{file_ids: {...}}` +- [ ] Brand code allocation: log with metadata `{brand_code, source: "allocated"|"recycled"}` + +### Step 7: Add output details to `complete` timeline event +- [ ] When `video_worker.py` calls `transition_to_state(COMPLETE)`, pass `timeline_metadata` with distribution output details +- [ ] Include: `youtube_url`, `youtube_video_id`, `dropbox_link`, `brand_code`, `gdrive_file_ids`, `edit_count` + +### Step 8: Tests +- [ ] Unit test: `TimelineEvent` with metadata serializes correctly +- [ ] Unit test: `log_to_job` writes to subcollection +- [ ] Unit test: edit endpoint includes `previous_outputs` and `cleanup_results` in timeline event +- [ ] Unit test: `complete` transition includes output metadata +- [ ] Verify existing tests still pass + +## Files to Create/Modify + +| File | Action | Description | +|------|--------|-------------| +| `backend/models/job.py` | Modify | Add `metadata` field to `TimelineEvent` | +| `backend/services/firestore_service.py` | Modify | Add `log_to_job()` helper, update `update_job_status()` to accept timeline metadata | +| `backend/services/job_manager.py` | Modify | Add `timeline_metadata` param to `transition_to_state()` | +| `backend/api/routes/jobs.py` | Modify | Enhance edit endpoint with cleanup logging, output snapshots, timeline metadata | +| `backend/api/routes/admin.py` | Modify | Enhance delete-outputs endpoint similarly | +| `backend/workers/video_worker_orchestrator.py` | Modify | Add structured metadata to distribution log messages | +| `backend/workers/video_worker.py` | Modify | Pass output metadata in `complete` transition (both orchestrator and legacy paths) | +| `backend/tests/test_edit_completed_track.py` | Modify | Verify timeline metadata in edit tests | +| `backend/tests/test_observability.py` | Create | Tests for TimelineEvent metadata and log_to_job | + +## Testing Strategy + +- **Unit tests**: TimelineEvent metadata serialization, log_to_job helper, edit endpoint timeline metadata assertions +- **Existing test verification**: `make test` must pass — changes are additive so existing tests should be unaffected +- **No E2E tests needed**: This is purely backend logging/observability — no UI changes + +## What This Does NOT Include + +- **Dedicated admin UI for viewing job logs** — existing admin page already shows timeline +- **Log aggregation/search UI** — Cloud Logging and Firestore viewer already serve this purpose +- **Changing state_data overwrite behavior** — the `{**job.state_data, ...}` pattern in video_worker.py is intentional; instead we preserve history in timeline events +- **Changing outputs_deleted_at clearing** — video worker clearing it after re-processing is correct behavior; the edit history lives in timeline metadata instead + +## Rollback Plan + +All changes are additive (new optional field on TimelineEvent, additional log writes). Rolling back just means deploying the previous version — no data migration needed. + +## Open Questions + +None — the approach is straightforward and low-risk. diff --git a/backend/api/routes/admin.py b/backend/api/routes/admin.py index 75f7e971..2f68743b 100644 --- a/backend/api/routes/admin.py +++ b/backend/api/routes/admin.py @@ -22,6 +22,7 @@ from backend.services.flacfetch_client import get_flacfetch_client, FlacfetchSer from backend.services.storage_service import StorageService from backend.services.audio_search_service import get_audio_search_service, NoResultsError, AudioSearchError from backend.models.job import JobStatus +from backend.services.firestore_service import log_to_job from backend.utils.test_data import is_test_email from karaoke_gen.utils import sanitize_filename @@ -1591,6 +1592,19 @@ async def delete_job_outputs( "gdrive": {"status": "skipped", "reason": "no gdrive_files in state_data"}, } + # Snapshot current outputs before cleanup (preserved in timeline metadata) + deleted_outputs = {} + if state_data.get('youtube_url'): + deleted_outputs['youtube_url'] = state_data['youtube_url'] + if state_data.get('dropbox_link'): + deleted_outputs['dropbox_link'] = state_data['dropbox_link'] + if state_data.get('brand_code'): + deleted_outputs['brand_code'] = state_data['brand_code'] + if state_data.get('gdrive_files'): + deleted_outputs['gdrive_files'] = state_data['gdrive_files'] + + log_to_job(job_id, "admin", "INFO", f"Output deletion initiated by {admin_email}", {"deleted_outputs": deleted_outputs}) + # Clean up YouTube youtube_url = state_data.get('youtube_url') if youtube_url: @@ -1717,11 +1731,17 @@ async def delete_job_outputs( update_payload[f"state_data.{key}"] = DELETE_FIELD cleared_keys.append(key) - # Add timeline event + # Add timeline event with structured metadata timeline_event = { "status": job.status, # Keep current status "timestamp": deletion_timestamp.isoformat(), "message": f"Outputs deleted by admin ({admin_email})", + "metadata": { + "action": "outputs_deleted", + "deleted_by": admin_email, + "deleted_outputs": deleted_outputs, + "cleanup_results": results, + }, } update_payload["timeline"] = ArrayUnion([timeline_event]) @@ -1755,6 +1775,10 @@ async def delete_job_outputs( f"Brand Code: {brand_code_status}. " f"Cleared state_data keys: {cleared_keys}" ) + log_to_job(job_id, "admin", "INFO", f"Outputs deleted by {admin_email}: {overall_status}", { + "cleanup_results": results, + "cleared_keys": cleared_keys, + }) return DeleteOutputsResponse( status=overall_status, diff --git a/backend/api/routes/jobs.py b/backend/api/routes/jobs.py index 7003c0d1..66ae8a4a 100644 --- a/backend/api/routes/jobs.py +++ b/backend/api/routes/jobs.py @@ -40,7 +40,7 @@ from backend.services.metrics import metrics from backend.middleware.tenant import get_tenant_from_request from backend.utils.test_data import is_test_email from backend.exceptions import InsufficientCreditsError -from backend.services.firestore_service import FirestoreService +from backend.services.firestore_service import FirestoreService, log_to_job from backend.models.requests import ChangeVisibilityRequest, ChangeVisibilityResponse from pydantic import BaseModel, Field, validator @@ -272,6 +272,19 @@ async def edit_completed_track( state_data = job.state_data or {} cleanup_results = {} + # Snapshot current outputs before cleanup (preserved in timeline metadata) + previous_outputs = {} + if state_data.get('youtube_url'): + previous_outputs['youtube_url'] = state_data['youtube_url'] + if state_data.get('dropbox_link'): + previous_outputs['dropbox_link'] = state_data['dropbox_link'] + if state_data.get('brand_code'): + previous_outputs['brand_code'] = state_data['brand_code'] + if state_data.get('gdrive_files'): + previous_outputs['gdrive_files'] = state_data['gdrive_files'] + + log_to_job(job_id, "edit", "INFO", f"Edit initiated by {user_email}", {"previous_outputs": previous_outputs}) + # --- Phase 1: Clean up distributed outputs --- # Delete YouTube video @@ -301,6 +314,7 @@ async def edit_completed_track( cleanup_results["youtube"] = {"status": "error", "error": str(e)} else: cleanup_results["youtube"] = {"status": "skipped", "reason": "no youtube_url"} + log_to_job(job_id, "edit", "INFO", f"YouTube cleanup: {cleanup_results['youtube'].get('status')}", cleanup_results["youtube"]) # Delete Dropbox folder brand_code = state_data.get('brand_code') @@ -322,6 +336,7 @@ async def edit_completed_track( cleanup_results["dropbox"] = {"status": "error", "error": str(e)} else: cleanup_results["dropbox"] = {"status": "skipped", "reason": "no brand_code or dropbox_path"} + log_to_job(job_id, "edit", "INFO", f"Dropbox cleanup: {cleanup_results['dropbox'].get('status')}", cleanup_results["dropbox"]) # Delete Google Drive files gdrive_files = state_data.get('gdrive_files') @@ -341,6 +356,7 @@ async def edit_completed_track( cleanup_results["gdrive"] = {"status": "error", "error": str(e)} else: cleanup_results["gdrive"] = {"status": "skipped", "reason": "no gdrive_files"} + log_to_job(job_id, "edit", "INFO", f"GDrive cleanup: {cleanup_results['gdrive'].get('status')}", cleanup_results["gdrive"]) # Delete GCS finals folder try: @@ -427,12 +443,19 @@ async def edit_completed_track( if "screens_progress" in state_data: update_payload[f"state_data.screens_progress"] = DELETE_FIELD - # Add timeline event + # Add timeline event with structured metadata edit_number = (job.edit_count if hasattr(job, 'edit_count') else 0) + 1 timeline_event = { "status": "complete", "timestamp": now.isoformat(), "message": f"Track edit initiated by {user_email} (edit #{edit_number})", + "metadata": { + "action": "edit_initiated", + "initiated_by": user_email, + "edit_number": edit_number, + "previous_outputs": previous_outputs, + "cleanup_results": cleanup_results, + }, } update_payload["timeline"] = ArrayUnion([timeline_event]) @@ -466,6 +489,10 @@ async def edit_completed_track( f"Dropbox={cleanup_results.get('dropbox', {}).get('status')}, " f"GDrive={cleanup_results.get('gdrive', {}).get('status')}" ) + log_to_job(job_id, "edit", "INFO", f"Edit #{edit_number} complete: reopened for review", { + "cleanup_results": cleanup_results, + "metadata_updated": metadata_updated, + }) return EditTrackResponse( status="success", diff --git a/backend/models/job.py b/backend/models/job.py index b51fe5ae..13fd4151 100644 --- a/backend/models/job.py +++ b/backend/models/job.py @@ -179,6 +179,7 @@ class TimelineEvent(BaseModel): timestamp: str progress: Optional[int] = None message: Optional[str] = None + metadata: Optional[Dict[str, Any]] = None class LogEntry(BaseModel): diff --git a/backend/services/firestore_service.py b/backend/services/firestore_service.py index 26a40e10..81a02a84 100644 --- a/backend/services/firestore_service.py +++ b/backend/services/firestore_service.py @@ -75,18 +75,20 @@ class FirestoreService: status: JobStatus, progress: Optional[int] = None, message: Optional[str] = None, + timeline_metadata: Optional[Dict[str, Any]] = None, **additional_fields ) -> None: """Update job status and add timeline event.""" try: doc_ref = self.db.collection(self.collection).document(job_id) - + # Create timeline event timeline_event = TimelineEvent( status=status.value, timestamp=datetime.utcnow().isoformat(), progress=progress, - message=message + message=message, + metadata=timeline_metadata, ) # Prepare updates @@ -710,3 +712,35 @@ def get_firestore_client() -> firestore.Client: _firestore_client = firestore.Client(project=settings.google_cloud_project) return _firestore_client + +_log_service: Optional["FirestoreService"] = None + + +def _get_log_service() -> "FirestoreService": + """Return a cached FirestoreService instance for log_to_job.""" + global _log_service + if _log_service is None: + _log_service = FirestoreService() + return _log_service + + +def log_to_job(job_id: str, worker: str, level: str, message: str, metadata: Optional[Dict[str, Any]] = None) -> None: + """ + Write a log entry to a job's Firestore log subcollection. + + This is a convenience function for API endpoints that need to log + to a job's log history (workers already have JobLogAdapter for this). + + Args: + job_id: Job ID to log to + worker: Worker/source name (e.g., "api", "edit", "admin") + level: Log level (INFO, WARNING, ERROR) + message: Log message + metadata: Optional structured metadata dict + """ + try: + entry = WorkerLogEntry.create(job_id, worker, level, message, metadata) + _get_log_service().append_log_to_subcollection(job_id, entry) + except Exception as e: + logger.debug(f"Failed to write job log for {job_id}: {e}") + diff --git a/backend/services/job_manager.py b/backend/services/job_manager.py index b78ce3e4..e1f435b9 100644 --- a/backend/services/job_manager.py +++ b/backend/services/job_manager.py @@ -413,7 +413,8 @@ class JobManager: progress: Optional[int] = None, message: Optional[str] = None, state_data_updates: Optional[Dict[str, Any]] = None, - raise_on_invalid: bool = True + raise_on_invalid: bool = True, + timeline_metadata: Optional[Dict[str, Any]] = None, ) -> bool: """ Transition job to new state with validation. @@ -426,6 +427,7 @@ class JobManager: state_data_updates: Updates to state_data field raise_on_invalid: If True (default), raise InvalidStateTransitionError on invalid transitions. If False, return False silently. + timeline_metadata: Optional structured metadata to attach to the timeline event Returns: True if transition succeeded, False if failed (only when raise_on_invalid=False) @@ -435,15 +437,15 @@ class JobManager: """ if not self.validate_state_transition(job_id, new_status, raise_on_invalid=raise_on_invalid): return False - + updates = { 'status': new_status, 'updated_at': datetime.utcnow() } - + if progress is not None: updates['progress'] = progress - + # Generate review token when entering AWAITING_REVIEW state # Tokens don't expire - they're job-scoped so low risk, and natural expiry happens when job completes if new_status == JobStatus.AWAITING_REVIEW: @@ -452,14 +454,14 @@ class JobManager: updates['review_token'] = review_token updates['review_token_expires_at'] = None # No expiry - token is job-scoped logger.info(f"Generated review token for job {job_id} (no expiry)") - + # If we have state_data_updates, merge them with existing state_data merged_state_data = None if state_data_updates: job = self.get_job(job_id) if job: merged_state_data = {**job.state_data, **state_data_updates} - + # Update job status (includes timeline event), passing state_data if present if merged_state_data is not None: self.update_job_status( @@ -467,6 +469,7 @@ class JobManager: status=new_status, progress=progress, message=message, + timeline_metadata=timeline_metadata, state_data=merged_state_data ) else: @@ -474,7 +477,8 @@ class JobManager: job_id=job_id, status=new_status, progress=progress, - message=message + message=message, + timeline_metadata=timeline_metadata, ) # Apply review token update separately if generated diff --git a/backend/tests/test_observability.py b/backend/tests/test_observability.py new file mode 100644 index 00000000..4785a4fe --- /dev/null +++ b/backend/tests/test_observability.py @@ -0,0 +1,132 @@ +""" +Tests for job observability improvements. + +Covers: +- TimelineEvent metadata field +- log_to_job helper function +- Timeline metadata in state transitions +""" +import pytest +from unittest.mock import patch, MagicMock +from datetime import datetime, timezone + +from backend.models.job import TimelineEvent + + +class TestTimelineEventMetadata: + """Tests for the metadata field on TimelineEvent.""" + + def test_timeline_event_without_metadata(self): + """Existing events without metadata still work.""" + event = TimelineEvent( + status="complete", + timestamp="2026-03-07T12:00:00", + progress=100, + message="Done!", + ) + data = event.model_dump(mode='json') + assert data["status"] == "complete" + assert data["metadata"] is None + + def test_timeline_event_with_metadata(self): + """Events can include structured metadata.""" + metadata = { + "action": "completed", + "brand_code": "NOMAD-1234", + "youtube_url": "https://youtube.com/watch?v=abc123", + "gdrive_file_ids": {"mp4": "file1", "cdg": "file2"}, + } + event = TimelineEvent( + status="complete", + timestamp="2026-03-07T12:00:00", + progress=100, + message="Karaoke generation complete!", + metadata=metadata, + ) + data = event.model_dump(mode='json') + assert data["metadata"]["brand_code"] == "NOMAD-1234" + assert data["metadata"]["youtube_url"] == "https://youtube.com/watch?v=abc123" + assert data["metadata"]["gdrive_file_ids"]["mp4"] == "file1" + + def test_timeline_event_with_edit_metadata(self): + """Edit-initiated events include previous outputs and cleanup results.""" + metadata = { + "action": "edit_initiated", + "initiated_by": "user@example.com", + "edit_number": 1, + "previous_outputs": { + "youtube_url": "https://youtube.com/watch?v=old123", + "brand_code": "NOMAD-0042", + "dropbox_link": "https://dropbox.com/sh/old", + }, + "cleanup_results": { + "youtube": {"status": "success", "video_id": "old123"}, + "dropbox": {"status": "success", "path": "/Karaoke/NOMAD-0042"}, + "gdrive": {"status": "skipped", "reason": "no gdrive_files"}, + }, + } + event = TimelineEvent( + status="complete", + timestamp="2026-03-07T12:00:00", + message="Track edit initiated by user@example.com (edit #1)", + metadata=metadata, + ) + data = event.model_dump(mode='json') + assert data["metadata"]["previous_outputs"]["youtube_url"] == "https://youtube.com/watch?v=old123" + assert data["metadata"]["cleanup_results"]["youtube"]["status"] == "success" + + def test_timeline_event_metadata_roundtrip(self): + """Metadata survives serialization/deserialization.""" + metadata = {"brand_code": "NOMAD-1234", "nested": {"key": "value"}} + event = TimelineEvent( + status="complete", + timestamp="2026-03-07T12:00:00", + metadata=metadata, + ) + data = event.model_dump(mode='json') + restored = TimelineEvent(**data) + assert restored.metadata == metadata + + +class TestLogToJob: + """Tests for the log_to_job helper function.""" + + @patch('backend.services.firestore_service._get_log_service') + def test_log_to_job_writes_to_subcollection(self, mock_get_service): + """log_to_job creates a WorkerLogEntry and writes to Firestore.""" + mock_service = MagicMock() + mock_get_service.return_value = mock_service + + from backend.services.firestore_service import log_to_job + log_to_job("job123", "edit", "INFO", "Edit started", {"key": "value"}) + + mock_service.append_log_to_subcollection.assert_called_once() + call_args = mock_service.append_log_to_subcollection.call_args + assert call_args[0][0] == "job123" + entry = call_args[0][1] + assert entry.worker == "edit" + assert entry.level == "INFO" + assert entry.message == "Edit started" + assert entry.metadata == {"key": "value"} + + @patch('backend.services.firestore_service._get_log_service') + def test_log_to_job_suppresses_errors(self, mock_get_service): + """log_to_job should never raise — logging failures must be silent.""" + mock_get_service.side_effect = Exception("Firestore down") + + from backend.services.firestore_service import log_to_job + # Should not raise + log_to_job("job123", "edit", "INFO", "Edit started") + + @patch('backend.services.firestore_service._get_log_service') + def test_log_to_job_without_metadata(self, mock_get_service): + """log_to_job works without metadata.""" + mock_service = MagicMock() + mock_get_service.return_value = mock_service + + from backend.services.firestore_service import log_to_job + log_to_job("job123", "admin", "WARNING", "Something happened") + + mock_service.append_log_to_subcollection.assert_called_once() + entry = mock_service.append_log_to_subcollection.call_args[0][1] + assert entry.metadata is None diff --git a/backend/workers/video_worker.py b/backend/workers/video_worker.py index 18d2d928..f6e7d1b7 100644 --- a/backend/workers/video_worker.py +++ b/backend/workers/video_worker.py @@ -364,14 +364,27 @@ async def generate_video_orchestrated(job_id: str) -> bool: ) # Mark job as complete (triggers completion email with youtube_url now available) + duration = time.time() - start_time + completion_metadata = { + "action": "completed", + "brand_code": result.brand_code, + "youtube_url": result.youtube_url, + "youtube_upload_queued": result.youtube_upload_queued, + "dropbox_link": result.dropbox_link, + "gdrive_file_ids": result.gdrive_files, + "edit_count": getattr(job, 'edit_count', 0), + "duration_seconds": round(duration, 1), + } + if result.distribution_warnings: + completion_metadata["distribution_warnings"] = result.distribution_warnings job_manager.transition_to_state( job_id=job_id, new_status=JobStatus.COMPLETE, progress=100, - message="Karaoke generation complete!" + message="Karaoke generation complete!", + timeline_metadata=completion_metadata, ) - duration = time.time() - start_time root_span.set_attribute("duration_seconds", duration) root_span.set_attribute("brand_code", result.brand_code or '') logger.info(f"[job:{job_id}] WORKER_END worker=video orchestrator=true status=success duration={duration:.1f}s") @@ -859,14 +872,24 @@ async def generate_video_legacy(job_id: str) -> bool: }) # Mark job as complete (triggers completion email with youtube_url now available) + duration = time.time() - start_time + completion_metadata = { + "action": "completed", + "brand_code": result.get('brand_code'), + "youtube_url": result.get('youtube_url'), + "dropbox_link": result.get('dropbox_link'), + "gdrive_file_ids": result.get('gdrive_files'), + "edit_count": getattr(job, 'edit_count', 0), + "duration_seconds": round(duration, 1), + } job_manager.transition_to_state( job_id=job_id, new_status=JobStatus.COMPLETE, progress=100, - message="Karaoke generation complete!" + message="Karaoke generation complete!", + timeline_metadata=completion_metadata, ) - duration = time.time() - start_time root_span.set_attribute("duration_seconds", duration) root_span.set_attribute("brand_code", result.get('brand_code', '')) logger.info(f"[job:{job_id}] WORKER_END worker=video status=success duration={duration:.1f}s") diff --git a/backend/workers/video_worker_orchestrator.py b/backend/workers/video_worker_orchestrator.py index 7266353b..361c3f32 100644 --- a/backend/workers/video_worker_orchestrator.py +++ b/backend/workers/video_worker_orchestrator.py @@ -638,7 +638,7 @@ class VideoWorkerOrchestrator: if video_url: self.result.youtube_url = video_url - self.job_log.info(f"Uploaded to YouTube: {video_url}") + self.job_log.info(f"Uploaded to YouTube: {video_url} (video_id={video_id})") # Record upload in pending buffer (bridges ~7min GCP monitoring delay) try: @@ -715,7 +715,7 @@ class VideoWorkerOrchestrator: except Exception as e: self.job_log.warning(f"Failed to create Dropbox sharing link: {e}") - self.job_log.info("Dropbox upload complete") + self.job_log.info(f"Dropbox upload complete: {remote_folder}") except Exception as e: self.job_log.error(f"Dropbox upload failed: {e}") @@ -752,7 +752,7 @@ class VideoWorkerOrchestrator: ) self.result.gdrive_files = uploaded - self.job_log.info(f"Google Drive upload complete: {len(uploaded)} files") + self.job_log.info(f"Google Drive upload complete: {len(uploaded)} files (ids={uploaded})") except Exception as e: self.job_log.error(f"Google Drive upload failed: {e}") diff --git a/docs/LESSONS-LEARNED.md b/docs/LESSONS-LEARNED.md index 09be1b39..cfc664aa 100644 --- a/docs/LESSONS-LEARNED.md +++ b/docs/LESSONS-LEARNED.md @@ -885,3 +885,14 @@ Even with idempotent endpoints (PR #413) and basic retry logic (PR #242), jobs c **Key insight:** When investigating "what did the client send?", Cloud Trace request size can serve as a forensic tool even without request body logging. Compact JSON serialization (`JSON.stringify` with no spaces) is deterministic, so byte-level size matching is reliable. +### Timeline Metadata for Job Traceability (2026-03-07) + +**Problem:** After shipping the "edit completed tracks" feature, tracing what happened to edited jobs was very difficult. Timeline events only had `status`, `timestamp`, `progress`, `message` — no output details. Cleanup operations (YouTube deletion, Dropbox removal) weren't logged to Firestore. Previous outputs were overwritten in `state_data` with no history. API endpoints (edit, admin-delete) logged to Cloud Run stdout but not to the job's Firestore log subcollection. + +**Solution:** +1. **`TimelineEvent.metadata`** — Optional `Dict[str, Any]` field on timeline events. Completion events now capture `brand_code`, `youtube_url`, `dropbox_link`, `gdrive_file_ids`, `duration_seconds`. Edit/delete events capture `previous_outputs` and `cleanup_results`. +2. **`log_to_job()` helper** — Module-level function in `firestore_service.py` that writes to the job's Firestore log subcollection. Wraps errors silently so logging never breaks the caller. Used by API endpoints that previously only logged to stdout. +3. **Plumbed `timeline_metadata`** through `job_manager.transition_to_state()` → `firestore_service.update_job_status()`. + +**Key insight:** The job's Firestore document (timeline + log subcollection) should be the single source of truth for "what happened to this job." Cloud Logging has retention limits and requires knowing which service to search. Timeline metadata makes the job self-documenting — any agent or human can read the timeline and understand the full lifecycle without cross-referencing logs. + diff --git a/docs/README.md b/docs/README.md index e1962e25..9d4debae 100644 --- a/docs/README.md +++ b/docs/README.md @@ -45,6 +45,8 @@ ## Recent Changes +- **Job Observability Improvements** (2026-03-07): Enhanced timeline events with a `metadata` field to capture structured output details (brand codes, YouTube video IDs, Dropbox paths, GDrive file IDs) at key moments — job completion, edit initiation, and admin output deletion. Added `log_to_job()` helper for API endpoints to write to Firestore job log subcollections (previously only workers logged there). Edit and admin-delete endpoints now snapshot previous outputs before cleanup and log each cleanup step. Distribution logging enhanced with video IDs, folder paths, and file IDs. See [plans/improve-job-observability.plan.md](../.claude/plans/improve-job-observability.plan.md). + - **Edit Completed Tracks** (2026-03-06): Users can now edit completed karaoke tracks — correcting lyrics, changing instrumental selection, or updating artist/title metadata — without consuming additional credits. Clicking "Edit" on a completed job opens a confirmation modal (with optional artist/title editing), then cleans up all distributed outputs (YouTube video deleted, Dropbox folder removed, GDrive files deleted, GCS finals purged, brand code recycled), resets the job to `awaiting_review`, and redirects to the existing review screen. After re-review, the standard pipeline re-renders and re-distributes with a fresh brand code. New endpoint: `POST /api/jobs/{job_id}/edit`. New Job model field: `edit_count`. State machine updated: `COMPLETE → AWAITING_REVIEW` transition allowed. See [archive/2026-03-06-edit-completed-tracks-plan.md](archive/2026-03-06-edit-completed-tracks-plan.md) and [API.md](API.md#edit-completed-track). - **Change Visibility** (2026-03-06): Users can now toggle completed jobs between public and private via a "Make Public"/"Make Private" button on the job card. Two distinct flows: Public→Private is fast (~1-2 min, redistributes existing finals), Private→Public is slow (~15-30 min, resets styles to Nomad theme, regenerates screens, re-renders video). Confirmation modal explains the implications. Tenant jobs excluded. See [archive/2026-03-06-change-visibility-plan.md](archive/2026-03-06-change-visibility-plan.md) and [API.md](API.md#change-visibility). diff --git a/pyproject.toml b/pyproject.toml index 6a3f57ec..d3b74a78 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [tool.poetry] name = "karaoke-gen" -version = "0.137.1" +version = "0.137.2" description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens." authors = ["Andrew Beveridge "] license = "MIT"