## Summary On 2026-11-07 the validator emailed **"SEQUENCE GAPS: CDG missing 1872, MP4-720p missing 1872"**. That was a false alarm, caused by a race: - NOMAD-1871 completed at 12:20:23 UTC. Its delayed post-job check ran at **12:24:47**. - NOMAD-1872 was in `packaging` from 12:23:39 to 12:25:40. Its 4K MP4 was already in Drive, but the 720p MP4 and the CDG were not. - The global-max gap logic saw 1872 in MP4 but not in the other two folders. All three files (`gdrive_files` in Firestore) landed about a minute later, and the later post-job runs (12:29 and 12:36) found no issues. ## Change - `list_folder_contents` can now also collect each file's Drive `createdTime`. - New helper `recently_uploaded_sequences()`: a track with any file created within `IN_FLIGHT_GRACE_MINUTES` (default 30) counts as still uploading, and `validate_files(..., in_flight=...)` leaves it out of the gap check. - These tracks are still covered by their own post-job trigger (the targeted `brand_code` completeness check, which this change doesn't touch) and by the daily 9 PM run. - Docs: `docs/GDRIVE-VALIDATOR.md` (new env var, plus a note on the race in the gap runbook). ## Testing - `infrastructure/functions/gdrive_validator/test_main.py`: 40 passed. 3 new tests cover a track still uploading, an old missing track (still reported), and bad timestamps. - `backend/tests/test_gdrive_validation_delay.py`: 13 passed. - Local CodeRabbit review: 2 minor findings, both fixed (env var parse fallback, offset-less timestamps). ## Deploy This only affects the Cloud Function, so no backend version bump. Deploying it takes `infrastructure/functions/gdrive_validator/deploy.sh`, which needs `/gcp-admin`. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVUwKapcjF8czVQrPD5 --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
20 KiB
Google Drive Validator
Automated validation of the Nomad Karaoke public Google Drive share folder. Detects duplicate brand codes, invalid filenames, and sequence gaps.
Same-run partial-publish check (incident-hardening D2): when the post-job trigger
names the just-published brand_code (passed orchestrator → Cloud Tasks → internal
endpoint → Cloud Function), the validator additionally verifies that specific track
is present in all expected folders (MP4, MP4-720p, CDG) — independent of the global-max
gap logic, which only notices a missing latest track once a later track raises the
max. This surfaces a silent partial publish (e.g. NOMAD-1632's missing 720p variant)
same-run (~5 min post-job) as a missing_for_track issue, instead of up to 24h later.
The brand_code is optional; absent → full-scan behaviour as before.
Overview
The public share folder (1laRKAyxo0v817SstfM5XkpbWiNKNAMSX) contains all published karaoke tracks organized into subfolders:
Public Share/
CDG/ # .zip files (MP3 + CDG for KJ players)
MP4/ # .mp4 files (4K lossless)
MP4-720p/ # .mp4 files (720p lossy)
Each file follows the naming convention: NOMAD-XXXX - Artist - Title.(mp4|zip)
The validator checks:
- Duplicate sequence numbers - e.g., two files both named
NOMAD-1271 - Invalid filenames - files that don't match the expected pattern
- Sequence gaps - missing numbers in the NOMAD-XXXX sequence (with known exclusions)
Architecture
┌────────────────────────┐
│ Cloud Scheduler │
│ (9 PM ET daily) │
└──────────┬─────────────┘
│ OIDC auth
▼
┌──────────────────┐ ┌────────────────────────┐ ┌──────────────┐
│ Backend (post- │ OIDC │ gdrive-validator │ API │ Google Drive │
│ job completion) │─────────→│ Cloud Function (Gen2) │─────────→│ API v3 │
└──────────────────┘ └──────────┬─────────────┘ └──────────────┘
│
┌──────────────────┐ ┌──────┴──────┐
│ gcloud functions │ │ │
│ call (manual) │─────────→ ▼ ▼
└──────────────────┘ ┌───────────┐ ┌────────────┐
│ Postmark │ │ Pushbullet │
│ (primary) │ │ (fallback) │
└───────────┘ └────────────┘
Trigger Points
| Trigger | When | How |
|---|---|---|
| Daily schedule | 9:00 PM ET every day | Cloud Scheduler → HTTP POST with OIDC token |
| Post-job | 5 minutes after each job uploads to public share | Backend orchestrator → Cloud Tasks (5 min delay) → internal endpoint → Cloud Function |
| Manual | On demand | python scripts/check_public_share.py via gcloud CLI |
Notifications
Notifications are sent via both channels (belt + suspenders):
- Postmark email (primary) - Sent to
gen@nomadkaraoke.com. Provides a persistent, searchable record. - Pushbullet (fallback) - Push notification for immediate visibility.
Notifications are sent when:
- Issues are found (always)
- No issues found and
NOTIFY_ON_SUCCESS=true(daily summary) - Errors occur during validation
Files
| File | Purpose |
|---|---|
infrastructure/functions/gdrive_validator/main.py |
Cloud Function source code |
infrastructure/functions/gdrive_validator/requirements.txt |
Python dependencies |
infrastructure/functions/gdrive_validator/deploy.sh |
Manual deployment script |
infrastructure/__main__.py |
Pulumi config (env vars, secrets, IAM) |
infrastructure/modules/iam/worker_sas.py |
Service account definition |
backend/services/gdrive_validator_client.py |
Backend HTTP client (OIDC auth) |
backend/services/worker_service.py |
Cloud Tasks scheduling (5 min delay) |
backend/api/routes/internal.py |
/internal/trigger-gdrive-validation endpoint |
backend/workers/video_worker_orchestrator.py |
Post-job trigger integration |
infrastructure/modules/cloud_tasks.py |
Pulumi config for gdrive-validation-queue |
infrastructure/functions/gdrive_validator/test_main.py |
Unit tests for Cloud Function |
scripts/check_public_share.py |
Local CLI for manual checks |
Configuration
Environment Variables (Cloud Function)
| Variable | Value | Description |
|---|---|---|
GDRIVE_FOLDER_ID |
1laRKAyxo0v817SstfM5XkpbWiNKNAMSX |
Root folder of public share |
NOTIFY_ON_SUCCESS |
false (prod) |
Send an "all clear" email on every clean run; off in prod because the post-job trigger runs after every job |
EMAIL_TO |
gen@nomadkaraoke.com |
Email recipient |
EMAIL_FROM |
gen@nomadkaraoke.com |
Email sender |
IN_FLIGHT_GRACE_MINUTES |
30 (default) |
Tracks with any file created in Drive this recently count as still uploading and are left out of the gap check |
Secrets (Cloud Function)
| Secret | Source | Description |
|---|---|---|
POSTMARK_SERVER_TOKEN |
postmark-server-token:latest |
Postmark server token for email |
PUSHBULLET_API_KEY |
pushbullet-api-key:latest |
Pushbullet API key for push notifications |
Secrets (Backend / Cloud Run)
| Secret | Source | Description |
|---|---|---|
GDRIVE_VALIDATOR_URL |
gdrive-validator-url:latest |
Cloud Function URL for post-job trigger |
IAM
| Principal | Role | Resource | Purpose |
|---|---|---|---|
gdrive-validator@ SA |
roles/cloudfunctions.invoker |
gdrive-validator function | Self-invoke (Cloud Scheduler) |
| Cloud Scheduler agent | roles/cloudfunctions.invoker |
gdrive-validator function | Daily trigger |
karaoke-backend@ SA |
roles/cloudfunctions.invoker |
gdrive-validator function | Post-job trigger |
karaoke-backend@ SA |
roles/run.invoker |
gdrive-validator Cloud Run service | Gen2 function HTTP access |
Known Gaps (Historical Only)
The validator knows about expected sequence gaps from the pre-generator era (before Dec 2024). These are configured identically in both:
infrastructure/functions/gdrive_validator/main.py(Cloud Function)scripts/validate_and_sync.py(local script)
Do NOT add new entries to KNOWN_GAPS. All new gaps are bugs — see Sequence Gap Detected for the investigation and fix procedure.
Usage
Manual Check
# Formatted output
python scripts/check_public_share.py
# Raw JSON
python scripts/check_public_share.py --json
Exit codes: 0 = all clear, 1 = issues found, 2 = error.
Checking Cloud Function Logs
gcloud functions logs read gdrive-validator --region=us-central1 --project=nomadkaraoke --limit=50
Redeploying the Cloud Function
The Cloud Function source is not deployed via CI — it's deployed either:
- Via Pulumi (infrastructure changes like env vars, secrets, IAM)
- Via
deploy.sh(code changes tomain.py)
cd infrastructure/functions/gdrive_validator
./deploy.sh
Note: deploy.sh deploys the source code. Pulumi manages the infrastructure (env vars, secrets, IAM, scheduler). After running deploy.sh, the Pulumi-managed config is overwritten by the gcloud deploy — run pulumi up afterward if you need to restore Pulumi-managed settings.
Troubleshooting
Email Not Received
- Check Cloud Function logs for Postmark errors:
gcloud functions logs read gdrive-validator --region=us-central1 --limit=20 - Verify the Postmark server token is set:
gcloud secrets versions access latest --secret=postmark-server-token --project=nomadkaraoke | head -c 10 - Check Postmark dashboard (Messages → Outbound) for delivery status
- Pushbullet should still work as fallback
Post-Job Trigger Not Firing
The post-job trigger uses a 5-minute Cloud Tasks delay (via gdrive-validation-queue) to allow E2E test cleanup to finish before validation runs.
- Check that
GDRIVE_VALIDATOR_URLsecret is set:gcloud secrets versions access latest --secret=gdrive-validator-url --project=nomadkaraoke - Check backend logs for "Scheduled delayed GDrive validation" or "GDrive validation trigger failed"
- Check Cloud Tasks queue for pending tasks:
gcloud tasks list --queue=gdrive-validation-queue --location=us-central1 --project=nomadkaraoke - Verify the backend SA has invoker permissions:
gcloud functions get-iam-policy gdrive-validator --region=us-central1 --project=nomadkaraoke - The trigger only fires for non-private jobs (jobs with
gdrive_folder_idset) - If Cloud Tasks scheduling fails, the orchestrator falls back to immediate validation
Sequence Gap Detected
IMPORTANT: Never add new entries to KNOWN_GAPS. The existing known gaps are historical (pre-generator era, over a year old). Every new gap is a real bug that must be investigated and fixed.
Tracks still uploading are not gaps. A job's packaging step uploads MP4, then
MP4-720p, then CDG over ~2 minutes, and another job's post-job trigger can run in that
window. On 2026-11-07 NOMAD-1871's check ran at 12:24:47 UTC while NOMAD-1872 was
uploading (12:23:39–12:25:40) and emailed "CDG/MP4-720p: missing 1872"; the files were
all there a minute later. Since then the validator reads each file's Drive createdTime
and leaves out of the gap check any track with a file created in the last
IN_FLIGHT_GRACE_MINUTES (30). That track is still checked by its own post-job trigger
(which names its brand_code) and by the daily 9 PM run. If you get a gap alert for the
newest track, first check whether that job was in packaging at the alert time.
Investigation steps:
-
Identify the missing brand code(s) from the alert (e.g., "MP4: missing 1295")
-
Check if a job exists with that brand code:
# Query Firestore for the brand code db.collection('jobs').where( filter=FieldFilter('state_data.brand_code', '==', 'NOMAD-XXXX') ).get() -
If no job found, the brand code was allocated but the job was deleted, failed, or re-triggered with a new brand code. Check Cloud Run logs for the allocation:
gcloud logging read \ 'resource.type="cloud_run_revision" AND resource.labels.service_name="karaoke-backend" AND "Allocated brand code: NOMAD-XXXX"' \ --project=nomadkaraoke --limit=10 --format="table(timestamp,jsonPayload.message)" --freshness=7d -
Find the job that consumed it by checking logs around the allocation timestamp:
# Search for the brand code in all log messages gcloud logging read \ 'resource.type="cloud_run_revision" AND resource.labels.service_name="karaoke-backend" AND "NOMAD-XXXX"' \ --project=nomadkaraoke --limit=50 --format="table(timestamp,jsonPayload.message)" --freshness=7d -
Determine root cause — common scenarios:
- Job re-triggered (admin reset): First run allocates brand code N, second run allocates N+M. Brand code N is orphaned.
- E2E test cleanup missed recycling: Test job consumed a number but cleanup didn't return it.
- Job failed mid-distribution: Brand code allocated but GDrive upload never happened.
-
Fix — recycle the brand code so the next public job fills the gap:
import os os.environ['GOOGLE_CLOUD_PROJECT'] = 'nomadkaraoke' from google.cloud import firestore db = firestore.Client(project='nomadkaraoke') doc_ref = db.collection('brand_code_counters').document('NOMAD') doc = doc_ref.get() data = doc.to_dict() recycled = data.get('recycled', []) recycled.append(XXXX) # The missing number recycled.sort() doc_ref.update({'recycled': recycled, 'last_updated': firestore.SERVER_TIMESTAMP})The
BrandCodeServicealways allocates the smallest recycled number first, so the next public job will fill the gap. -
Check for orphan files in GDrive that need manual cleanup (e.g., a CDG zip from a partial first run). Local
gcloudauth typically lacks GDrive write scopes, so report orphan files to the user for manual deletion. -
Add the incident to History (below) so future agents can see patterns.
Duplicate Brand Code Detected
This is the most critical alert. Steps:
- Check which files have the duplicate: the notification lists them
- Determine which is correct (check Firestore job records — usually only ONE of the
two tracks still has a job with that
state_data.brand_code; the other is orphaned) - Remove the orphaned track's outputs (see below)
- Re-trigger validation to confirm the fix
Known cause — job deleted while outputs still published (NOMAD-1583 incident,
2026-08-29): before v0.206.0, DELETE /api/jobs/{id} recycled the brand code even
when the job's outputs had never been cleaned up, leaving the files live in the
public share. The next job to allocate reused the recycled number → duplicate.
Since v0.206.0, deleting a job with published outputs runs the full distribution
cleanup first (YouTube + Dropbox + GDrive + kjbox mirror) and only recycles the
code once Dropbox and GDrive are confirmed clean.
Cleaning up orphaned outputs when the job doc is gone — use the admin endpoint
POST /api/admin/orphaned-outputs/cleanup with explicit targets (find file IDs and
the YouTube video ID in Cloud Run logs from the job's publish window):
curl -X POST "https://api.nomadkaraoke.com/api/admin/orphaned-outputs/cleanup" \
-H "Authorization: Bearer $KARAOKE_ADMIN_TOKEN" -H "Content-Type: application/json" \
-d '{
"youtube_video_id": "VIDEO_ID",
"dropbox_folder_path": "/MediaUnsynced/Karaoke/Tracks-Organized/NOMAD-XXXX - Artist - Title",
"gdrive_file_ids": ["...", "...", "..."],
"mirror_filename_prefix": "NOMAD-XXXX - Artist"
}'
The mirror_filename_prefix must include artist text after the brand code — the
mirror delete refuses a bare NOMAD-XXXX so it can never touch the OTHER track
legitimately holding the recycled code.
Cloud Function vs Local Script
| Check | Cloud Function | Local Script (validate_and_sync.py) |
|---|---|---|
| Duplicate sequence numbers | Yes | Yes |
| Invalid filenames | Yes | Yes |
| Sequence gaps | Yes | Yes |
| ZIP contents (MP3+CDG) | No (expensive) | Yes |
| Video resolution | No (expensive) | Yes |
| Render quality | No (expensive) | Yes |
| Bidirectional sync | No (read-only) | Yes (rclone) |
The cloud function does metadata-only checks via the GDrive API (fast, cheap, can run frequently). The local script does deep content validation that requires downloading files (slow, run manually).
History
- 2025-12-24: Initial cloud function deployed with Pushbullet notifications and daily Cloud Scheduler trigger
- 2026-02-28: Added SendGrid email (primary), kept Pushbullet (fallback). Added post-job trigger from backend. Added
scripts/check_public_share.pyfor manual invocation. Motivated by NOMAD-1271 duplicate brand code incident where daily-only checks left a duplicate undetected for ~24 hours. - 2026-03-02: NOMAD-1276 incident fix. Post-job trigger now uses 5-minute Cloud Tasks delay instead of immediate invocation, preventing false "all clear" when E2E test files are still in GDrive during cleanup. Also fixed cross-folder gap detection (uses global max across all folders) and added 28 unit tests for the Cloud Function.
- 2026-03-06: NOMAD-1295 gap. Stale GCE encoding cache (#499) caused job
f4a552bfto produce bad output on first run (allocated NOMAD-1295, only CDG uploaded). Admin re-triggered the job, which allocated a new brand code (NOMAD-1297) without recycling 1295. Fix: recycled 1295 back to the pool. Updated docs to clarify that new gaps must always be investigated and fixed, never added to KNOWN_GAPS. - 2026-08-29: NOMAD-1583 duplicate. Job
e97475ff(Mazzy Star – Fade Into You) published at 02:15 UTC with NOMAD-1583, then was deleted at 03:02 viaDELETE /api/jobs/{id}?delete_files=true— which recycled the brand code but left all published outputs live (GDrive ×3, Dropbox folder, YouTube video, kjbox mirror ×2). At 04:16 job326e1aef(Chase & Status – No Problem) reused the recycled 1583 → duplicate in all three share folders. Fixes (v0.206.0):DELETE /api/jobs/{id}now runs full distribution cleanup before deleting when outputs are still published (recycle stays gated on Dropbox+GDrive success, shared_run_distribution_cleanuphelper); jobs whose outputs were already deleted no longer re-recycle; newPOST /api/admin/orphaned-outputs/cleanupendpoint removes orphaned outputs by explicit IDs when the job doc is gone (used to clean up the Mazzy Star orphans). - 2026-09-11: NOMAD-1632 MP4-720p-only gap (Nat King Cole – Portrait of Jennie, job
21150e73). Unlike the recycled-brand-code gaps above, only the 720p variant was missing — MP4 and CDG were present and correct — so recycling the brand code would have been wrong (it would have duplicated the existing MP4/CDG). Root cause: a filename-substring collision with the song title. The encoder produced and uploaded the 720p to GCS correctly (… (Final Karaoke Lossy 720p).mp4), butGCEEncodingBackend.encode()mapped the worker's returned file list to format keys by matching the whole filename — checking"portrait" in filename_lower(to detect the 9:16 portrait video) before"720p". The title "Portrait of Jennie" contains "portrait", so the 720p master matched the portrait branch and was filed asportrait_mp4;mp4_720pwas never set. The orchestrator then hadfinal_video_720p=None, andgdrive_service.upload_to_public_sharesilently skips a file whose path isNone→ the track published to MP4/ + CDG/ with no MP4-720p/ entry. Deterministic for any song whose artist/title contains a format word ("portrait", "720p", "4k", "lossless", "vocals", …). The daily validator was the only detector (~a day later). Fixes (v0.222.2): (1) a shared, title-collision-proofclassify_encoded_output()matches the format parenthetical only (never the whole filename), used by both the encode path andredistribute_video; (2) a completeness guard inGCEEncodingBackend.encode()— if the worker's returnedoutput_filesis missing any requestedformat, it returnssuccess=False(orchestrator raises), so any other partial-encode cause also fails loud and is retried instead of publishing an incomplete release. Remediation of the live gap: the correct 720p already existed in GCS (jobs/21150e73/finals/…(Final Karaoke Lossy 720p).mp4), so it was uploaded to MP4-720p/ + the Divebar mirror directly — no re-render, no YouTube disruption. Do NOT recycle 1632. - 2026-08-18: NOMAD-1537 gap. A public→private visibility change on job
25d076e5(Alma Nocturna – Mejor Cállate) deleted the public GDrive/YouTube/Dropbox outputs and recycled 1537, then the private redistribution crashed (redistribute_videodrove an illegalcomplete → packagingtransition), so the change failed after the public files were already gone — a delete-first, no-rollback flow. The daily validator caught the hole; it self-closed when an unrelated public job reused the recycled 1537. Going-private is a legitimate cause of a transient gap (the number is recycled and refilled by the next public track) — the alert is expected and desired, not suppressed. Fixes (v0.196.1):redistribute_videoruns inredistribute_mode(no status transition, stayscomplete);change_to_privateis now distribute-then-delete (public outputs removed only after the private archive succeeds; on failure the job stays fully public); failed redistributions recycle any allocatedNOMADNPcode. Validator intentionally left unchanged.