Files
karaoke-gen/docs/GDRIVE-VALIDATOR.md
Andrew Beveridge ☄️andClaude Opus 5.5 d59b5bf3bd fix(gdrive-validator): don't report gaps for tracks still uploading (#1259)
## Summary
On 2026-11-07 the validator emailed **"SEQUENCE GAPS: CDG missing 1872,
MP4-720p missing 1872"**. That was a false alarm, caused by a race:

- NOMAD-1871 completed at 12:20:23 UTC. Its delayed post-job check ran
at **12:24:47**.
- NOMAD-1872 was in `packaging` from 12:23:39 to 12:25:40. Its 4K MP4
was already in Drive, but the 720p MP4 and the CDG were not.
- The global-max gap logic saw 1872 in MP4 but not in the other two
folders. All three files (`gdrive_files` in Firestore) landed about a
minute later, and the later post-job runs (12:29 and 12:36) found no
issues.

## Change
- `list_folder_contents` can now also collect each file's Drive
`createdTime`.
- New helper `recently_uploaded_sequences()`: a track with any file
created within `IN_FLIGHT_GRACE_MINUTES` (default 30) counts as still
uploading, and `validate_files(..., in_flight=...)` leaves it out of the
gap check.
- These tracks are still covered by their own post-job trigger (the
targeted `brand_code` completeness check, which this change doesn't
touch) and by the daily 9 PM run.
- Docs: `docs/GDRIVE-VALIDATOR.md` (new env var, plus a note on the race
in the gap runbook).

## Testing
- `infrastructure/functions/gdrive_validator/test_main.py`: 40 passed. 3
new tests cover a track still uploading, an old missing track (still
reported), and bad timestamps.
- `backend/tests/test_gdrive_validation_delay.py`: 13 passed.
- Local CodeRabbit review: 2 minor findings, both fixed (env var parse
fallback, offset-less timestamps).

## Deploy
This only affects the Cloud Function, so no backend version bump.
Deploying it takes
`infrastructure/functions/gdrive_validator/deploy.sh`, which needs
`/gcp-admin`.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVUwKapcjF8czVQrPD5

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 22:23:25 -05:00

20 KiB
Raw Permalink Blame History

Google Drive Validator

Automated validation of the Nomad Karaoke public Google Drive share folder. Detects duplicate brand codes, invalid filenames, and sequence gaps.

Same-run partial-publish check (incident-hardening D2): when the post-job trigger names the just-published brand_code (passed orchestrator → Cloud Tasks → internal endpoint → Cloud Function), the validator additionally verifies that specific track is present in all expected folders (MP4, MP4-720p, CDG) — independent of the global-max gap logic, which only notices a missing latest track once a later track raises the max. This surfaces a silent partial publish (e.g. NOMAD-1632's missing 720p variant) same-run (~5 min post-job) as a missing_for_track issue, instead of up to 24h later. The brand_code is optional; absent → full-scan behaviour as before.

Overview

The public share folder (1laRKAyxo0v817SstfM5XkpbWiNKNAMSX) contains all published karaoke tracks organized into subfolders:

Public Share/
  CDG/         # .zip files (MP3 + CDG for KJ players)
  MP4/         # .mp4 files (4K lossless)
  MP4-720p/    # .mp4 files (720p lossy)

Each file follows the naming convention: NOMAD-XXXX - Artist - Title.(mp4|zip)

The validator checks:

  • Duplicate sequence numbers - e.g., two files both named NOMAD-1271
  • Invalid filenames - files that don't match the expected pattern
  • Sequence gaps - missing numbers in the NOMAD-XXXX sequence (with known exclusions)

Architecture

                              ┌────────────────────────┐
                              │  Cloud Scheduler       │
                              │  (9 PM ET daily)       │
                              └──────────┬─────────────┘
                                         │ OIDC auth
                                         ▼
┌──────────────────┐          ┌────────────────────────┐          ┌──────────────┐
│ Backend (post-   │  OIDC    │  gdrive-validator      │  API     │ Google Drive │
│ job completion)  │─────────→│  Cloud Function (Gen2) │─────────→│ API v3       │
└──────────────────┘          └──────────┬─────────────┘          └──────────────┘
                                         │
┌──────────────────┐              ┌──────┴──────┐
│ gcloud functions │              │             │
│ call (manual)    │─────────→    ▼             ▼
└──────────────────┘       ┌───────────┐ ┌────────────┐
                           │ Postmark  │ │ Pushbullet │
                           │ (primary) │ │ (fallback) │
                           └───────────┘ └────────────┘

Trigger Points

Trigger When How
Daily schedule 9:00 PM ET every day Cloud Scheduler → HTTP POST with OIDC token
Post-job 5 minutes after each job uploads to public share Backend orchestrator → Cloud Tasks (5 min delay) → internal endpoint → Cloud Function
Manual On demand python scripts/check_public_share.py via gcloud CLI

Notifications

Notifications are sent via both channels (belt + suspenders):

  1. Postmark email (primary) - Sent to gen@nomadkaraoke.com. Provides a persistent, searchable record.
  2. Pushbullet (fallback) - Push notification for immediate visibility.

Notifications are sent when:

  • Issues are found (always)
  • No issues found and NOTIFY_ON_SUCCESS=true (daily summary)
  • Errors occur during validation

Files

File Purpose
infrastructure/functions/gdrive_validator/main.py Cloud Function source code
infrastructure/functions/gdrive_validator/requirements.txt Python dependencies
infrastructure/functions/gdrive_validator/deploy.sh Manual deployment script
infrastructure/__main__.py Pulumi config (env vars, secrets, IAM)
infrastructure/modules/iam/worker_sas.py Service account definition
backend/services/gdrive_validator_client.py Backend HTTP client (OIDC auth)
backend/services/worker_service.py Cloud Tasks scheduling (5 min delay)
backend/api/routes/internal.py /internal/trigger-gdrive-validation endpoint
backend/workers/video_worker_orchestrator.py Post-job trigger integration
infrastructure/modules/cloud_tasks.py Pulumi config for gdrive-validation-queue
infrastructure/functions/gdrive_validator/test_main.py Unit tests for Cloud Function
scripts/check_public_share.py Local CLI for manual checks

Configuration

Environment Variables (Cloud Function)

Variable Value Description
GDRIVE_FOLDER_ID 1laRKAyxo0v817SstfM5XkpbWiNKNAMSX Root folder of public share
NOTIFY_ON_SUCCESS false (prod) Send an "all clear" email on every clean run; off in prod because the post-job trigger runs after every job
EMAIL_TO gen@nomadkaraoke.com Email recipient
EMAIL_FROM gen@nomadkaraoke.com Email sender
IN_FLIGHT_GRACE_MINUTES 30 (default) Tracks with any file created in Drive this recently count as still uploading and are left out of the gap check

Secrets (Cloud Function)

Secret Source Description
POSTMARK_SERVER_TOKEN postmark-server-token:latest Postmark server token for email
PUSHBULLET_API_KEY pushbullet-api-key:latest Pushbullet API key for push notifications

Secrets (Backend / Cloud Run)

Secret Source Description
GDRIVE_VALIDATOR_URL gdrive-validator-url:latest Cloud Function URL for post-job trigger

IAM

Principal Role Resource Purpose
gdrive-validator@ SA roles/cloudfunctions.invoker gdrive-validator function Self-invoke (Cloud Scheduler)
Cloud Scheduler agent roles/cloudfunctions.invoker gdrive-validator function Daily trigger
karaoke-backend@ SA roles/cloudfunctions.invoker gdrive-validator function Post-job trigger
karaoke-backend@ SA roles/run.invoker gdrive-validator Cloud Run service Gen2 function HTTP access

Known Gaps (Historical Only)

The validator knows about expected sequence gaps from the pre-generator era (before Dec 2024). These are configured identically in both:

  • infrastructure/functions/gdrive_validator/main.py (Cloud Function)
  • scripts/validate_and_sync.py (local script)

Do NOT add new entries to KNOWN_GAPS. All new gaps are bugs — see Sequence Gap Detected for the investigation and fix procedure.

Usage

Manual Check

# Formatted output
python scripts/check_public_share.py

# Raw JSON
python scripts/check_public_share.py --json

Exit codes: 0 = all clear, 1 = issues found, 2 = error.

Checking Cloud Function Logs

gcloud functions logs read gdrive-validator --region=us-central1 --project=nomadkaraoke --limit=50

Redeploying the Cloud Function

The Cloud Function source is not deployed via CI — it's deployed either:

  1. Via Pulumi (infrastructure changes like env vars, secrets, IAM)
  2. Via deploy.sh (code changes to main.py)
cd infrastructure/functions/gdrive_validator
./deploy.sh

Note: deploy.sh deploys the source code. Pulumi manages the infrastructure (env vars, secrets, IAM, scheduler). After running deploy.sh, the Pulumi-managed config is overwritten by the gcloud deploy — run pulumi up afterward if you need to restore Pulumi-managed settings.

Troubleshooting

Email Not Received

  1. Check Cloud Function logs for Postmark errors:
    gcloud functions logs read gdrive-validator --region=us-central1 --limit=20
    
  2. Verify the Postmark server token is set:
    gcloud secrets versions access latest --secret=postmark-server-token --project=nomadkaraoke | head -c 10
    
  3. Check Postmark dashboard (Messages → Outbound) for delivery status
  4. Pushbullet should still work as fallback

Post-Job Trigger Not Firing

The post-job trigger uses a 5-minute Cloud Tasks delay (via gdrive-validation-queue) to allow E2E test cleanup to finish before validation runs.

  1. Check that GDRIVE_VALIDATOR_URL secret is set:
    gcloud secrets versions access latest --secret=gdrive-validator-url --project=nomadkaraoke
    
  2. Check backend logs for "Scheduled delayed GDrive validation" or "GDrive validation trigger failed"
  3. Check Cloud Tasks queue for pending tasks:
    gcloud tasks list --queue=gdrive-validation-queue --location=us-central1 --project=nomadkaraoke
    
  4. Verify the backend SA has invoker permissions:
    gcloud functions get-iam-policy gdrive-validator --region=us-central1 --project=nomadkaraoke
    
  5. The trigger only fires for non-private jobs (jobs with gdrive_folder_id set)
  6. If Cloud Tasks scheduling fails, the orchestrator falls back to immediate validation

Sequence Gap Detected

IMPORTANT: Never add new entries to KNOWN_GAPS. The existing known gaps are historical (pre-generator era, over a year old). Every new gap is a real bug that must be investigated and fixed.

Tracks still uploading are not gaps. A job's packaging step uploads MP4, then MP4-720p, then CDG over ~2 minutes, and another job's post-job trigger can run in that window. On 2026-11-07 NOMAD-1871's check ran at 12:24:47 UTC while NOMAD-1872 was uploading (12:23:39–12:25:40) and emailed "CDG/MP4-720p: missing 1872"; the files were all there a minute later. Since then the validator reads each file's Drive createdTime and leaves out of the gap check any track with a file created in the last IN_FLIGHT_GRACE_MINUTES (30). That track is still checked by its own post-job trigger (which names its brand_code) and by the daily 9 PM run. If you get a gap alert for the newest track, first check whether that job was in packaging at the alert time.

Investigation steps:

  1. Identify the missing brand code(s) from the alert (e.g., "MP4: missing 1295")

  2. Check if a job exists with that brand code:

    # Query Firestore for the brand code
    db.collection('jobs').where(
        filter=FieldFilter('state_data.brand_code', '==', 'NOMAD-XXXX')
    ).get()
    
  3. If no job found, the brand code was allocated but the job was deleted, failed, or re-triggered with a new brand code. Check Cloud Run logs for the allocation:

    gcloud logging read \
      'resource.type="cloud_run_revision" AND resource.labels.service_name="karaoke-backend" AND "Allocated brand code: NOMAD-XXXX"' \
      --project=nomadkaraoke --limit=10 --format="table(timestamp,jsonPayload.message)" --freshness=7d
    
  4. Find the job that consumed it by checking logs around the allocation timestamp:

    # Search for the brand code in all log messages
    gcloud logging read \
      'resource.type="cloud_run_revision" AND resource.labels.service_name="karaoke-backend" AND "NOMAD-XXXX"' \
      --project=nomadkaraoke --limit=50 --format="table(timestamp,jsonPayload.message)" --freshness=7d
    
  5. Determine root cause — common scenarios:

    • Job re-triggered (admin reset): First run allocates brand code N, second run allocates N+M. Brand code N is orphaned.
    • E2E test cleanup missed recycling: Test job consumed a number but cleanup didn't return it.
    • Job failed mid-distribution: Brand code allocated but GDrive upload never happened.
  6. Fix — recycle the brand code so the next public job fills the gap:

    import os
    os.environ['GOOGLE_CLOUD_PROJECT'] = 'nomadkaraoke'
    from google.cloud import firestore
    db = firestore.Client(project='nomadkaraoke')
    
    doc_ref = db.collection('brand_code_counters').document('NOMAD')
    doc = doc_ref.get()
    data = doc.to_dict()
    recycled = data.get('recycled', [])
    recycled.append(XXXX)  # The missing number
    recycled.sort()
    doc_ref.update({'recycled': recycled, 'last_updated': firestore.SERVER_TIMESTAMP})
    

    The BrandCodeService always allocates the smallest recycled number first, so the next public job will fill the gap.

  7. Check for orphan files in GDrive that need manual cleanup (e.g., a CDG zip from a partial first run). Local gcloud auth typically lacks GDrive write scopes, so report orphan files to the user for manual deletion.

  8. Add the incident to History (below) so future agents can see patterns.

Duplicate Brand Code Detected

This is the most critical alert. Steps:

  1. Check which files have the duplicate: the notification lists them
  2. Determine which is correct (check Firestore job records — usually only ONE of the two tracks still has a job with that state_data.brand_code; the other is orphaned)
  3. Remove the orphaned track's outputs (see below)
  4. Re-trigger validation to confirm the fix

Known cause — job deleted while outputs still published (NOMAD-1583 incident, 2026-08-29): before v0.206.0, DELETE /api/jobs/{id} recycled the brand code even when the job's outputs had never been cleaned up, leaving the files live in the public share. The next job to allocate reused the recycled number → duplicate. Since v0.206.0, deleting a job with published outputs runs the full distribution cleanup first (YouTube + Dropbox + GDrive + kjbox mirror) and only recycles the code once Dropbox and GDrive are confirmed clean.

Cleaning up orphaned outputs when the job doc is gone — use the admin endpoint POST /api/admin/orphaned-outputs/cleanup with explicit targets (find file IDs and the YouTube video ID in Cloud Run logs from the job's publish window):

curl -X POST "https://api.nomadkaraoke.com/api/admin/orphaned-outputs/cleanup" \
  -H "Authorization: Bearer $KARAOKE_ADMIN_TOKEN" -H "Content-Type: application/json" \
  -d '{
    "youtube_video_id": "VIDEO_ID",
    "dropbox_folder_path": "/MediaUnsynced/Karaoke/Tracks-Organized/NOMAD-XXXX - Artist - Title",
    "gdrive_file_ids": ["...", "...", "..."],
    "mirror_filename_prefix": "NOMAD-XXXX - Artist"
  }'

The mirror_filename_prefix must include artist text after the brand code — the mirror delete refuses a bare NOMAD-XXXX so it can never touch the OTHER track legitimately holding the recycled code.

Cloud Function vs Local Script

Check Cloud Function Local Script (validate_and_sync.py)
Duplicate sequence numbers Yes Yes
Invalid filenames Yes Yes
Sequence gaps Yes Yes
ZIP contents (MP3+CDG) No (expensive) Yes
Video resolution No (expensive) Yes
Render quality No (expensive) Yes
Bidirectional sync No (read-only) Yes (rclone)

The cloud function does metadata-only checks via the GDrive API (fast, cheap, can run frequently). The local script does deep content validation that requires downloading files (slow, run manually).

History

  • 2025-12-24: Initial cloud function deployed with Pushbullet notifications and daily Cloud Scheduler trigger
  • 2026-02-28: Added SendGrid email (primary), kept Pushbullet (fallback). Added post-job trigger from backend. Added scripts/check_public_share.py for manual invocation. Motivated by NOMAD-1271 duplicate brand code incident where daily-only checks left a duplicate undetected for ~24 hours.
  • 2026-03-02: NOMAD-1276 incident fix. Post-job trigger now uses 5-minute Cloud Tasks delay instead of immediate invocation, preventing false "all clear" when E2E test files are still in GDrive during cleanup. Also fixed cross-folder gap detection (uses global max across all folders) and added 28 unit tests for the Cloud Function.
  • 2026-03-06: NOMAD-1295 gap. Stale GCE encoding cache (#499) caused job f4a552bf to produce bad output on first run (allocated NOMAD-1295, only CDG uploaded). Admin re-triggered the job, which allocated a new brand code (NOMAD-1297) without recycling 1295. Fix: recycled 1295 back to the pool. Updated docs to clarify that new gaps must always be investigated and fixed, never added to KNOWN_GAPS.
  • 2026-08-29: NOMAD-1583 duplicate. Job e97475ff (Mazzy Star – Fade Into You) published at 02:15 UTC with NOMAD-1583, then was deleted at 03:02 via DELETE /api/jobs/{id}?delete_files=true — which recycled the brand code but left all published outputs live (GDrive ×3, Dropbox folder, YouTube video, kjbox mirror ×2). At 04:16 job 326e1aef (Chase & Status – No Problem) reused the recycled 1583 → duplicate in all three share folders. Fixes (v0.206.0): DELETE /api/jobs/{id} now runs full distribution cleanup before deleting when outputs are still published (recycle stays gated on Dropbox+GDrive success, shared _run_distribution_cleanup helper); jobs whose outputs were already deleted no longer re-recycle; new POST /api/admin/orphaned-outputs/cleanup endpoint removes orphaned outputs by explicit IDs when the job doc is gone (used to clean up the Mazzy Star orphans).
  • 2026-09-11: NOMAD-1632 MP4-720p-only gap (Nat King Cole – Portrait of Jennie, job 21150e73). Unlike the recycled-brand-code gaps above, only the 720p variant was missing — MP4 and CDG were present and correct — so recycling the brand code would have been wrong (it would have duplicated the existing MP4/CDG). Root cause: a filename-substring collision with the song title. The encoder produced and uploaded the 720p to GCS correctly (… (Final Karaoke Lossy 720p).mp4), but GCEEncodingBackend.encode() mapped the worker's returned file list to format keys by matching the whole filename — checking "portrait" in filename_lower (to detect the 9:16 portrait video) before "720p". The title "Portrait of Jennie" contains "portrait", so the 720p master matched the portrait branch and was filed as portrait_mp4; mp4_720p was never set. The orchestrator then had final_video_720p=None, and gdrive_service.upload_to_public_share silently skips a file whose path is None → the track published to MP4/ + CDG/ with no MP4-720p/ entry. Deterministic for any song whose artist/title contains a format word ("portrait", "720p", "4k", "lossless", "vocals", …). The daily validator was the only detector (~a day later). Fixes (v0.222.2): (1) a shared, title-collision-proof classify_encoded_output() matches the format parenthetical only (never the whole filename), used by both the encode path and redistribute_video; (2) a completeness guard in GCEEncodingBackend.encode() — if the worker's returned output_files is missing any requested format, it returns success=False (orchestrator raises), so any other partial-encode cause also fails loud and is retried instead of publishing an incomplete release. Remediation of the live gap: the correct 720p already existed in GCS (jobs/21150e73/finals/…(Final Karaoke Lossy 720p).mp4), so it was uploaded to MP4-720p/ + the Divebar mirror directly — no re-render, no YouTube disruption. Do NOT recycle 1632.
  • 2026-08-18: NOMAD-1537 gap. A public→private visibility change on job 25d076e5 (Alma Nocturna – Mejor Cállate) deleted the public GDrive/YouTube/Dropbox outputs and recycled 1537, then the private redistribution crashed (redistribute_video drove an illegal complete → packaging transition), so the change failed after the public files were already gone — a delete-first, no-rollback flow. The daily validator caught the hole; it self-closed when an unrelated public job reused the recycled 1537. Going-private is a legitimate cause of a transient gap (the number is recycled and refilled by the next public track) — the alert is expected and desired, not suppressed. Fixes (v0.196.1): redistribute_video runs in redistribute_mode (no status transition, stays complete); change_to_private is now distribute-then-delete (public outputs removed only after the private archive succeeds; on failure the job stays fully public); failed redistributions recycle any allocated NOMADNP code. Validator intentionally left unchanged.