fix(gdrive-validator): don't report gaps for tracks still uploading (#1259)

## Summary
On 2026-11-07 the validator emailed **"SEQUENCE GAPS: CDG missing 1872,
MP4-720p missing 1872"**. That was a false alarm, caused by a race:

- NOMAD-1871 completed at 12:20:23 UTC. Its delayed post-job check ran
at **12:24:47**.
- NOMAD-1872 was in `packaging` from 12:23:39 to 12:25:40. Its 4K MP4
was already in Drive, but the 720p MP4 and the CDG were not.
- The global-max gap logic saw 1872 in MP4 but not in the other two
folders. All three files (`gdrive_files` in Firestore) landed about a
minute later, and the later post-job runs (12:29 and 12:36) found no
issues.

## Change
- `list_folder_contents` can now also collect each file's Drive
`createdTime`.
- New helper `recently_uploaded_sequences()`: a track with any file
created within `IN_FLIGHT_GRACE_MINUTES` (default 30) counts as still
uploading, and `validate_files(..., in_flight=...)` leaves it out of the
gap check.
- These tracks are still covered by their own post-job trigger (the
targeted `brand_code` completeness check, which this change doesn't
touch) and by the daily 9 PM run.
- Docs: `docs/GDRIVE-VALIDATOR.md` (new env var, plus a note on the race
in the gap runbook).

## Testing
- `infrastructure/functions/gdrive_validator/test_main.py`: 40 passed. 3
new tests cover a track still uploading, an old missing track (still
reported), and bad timestamps.
- `backend/tests/test_gdrive_validation_delay.py`: 13 passed.
- Local CodeRabbit review: 2 minor findings, both fixed (env var parse
fallback, offset-less timestamps).

## Deploy
This only affects the Cloud Function, so no backend version bump.
Deploying it takes
`infrastructure/functions/gdrive_validator/deploy.sh`, which needs
`/gcp-admin`.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVUwKapcjF8czVQrPD5

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Andrew Beveridge ☄️andClaude Opus 5.5 authored and GitHub committed 2026-11-08 22:23:25 -05:00
1 parent bb01e9e36a
commit d59b5bf3bd
4 files changed
+110 -9

No files matched your search

+12 -1
View File
@@ -96,9 +96,10 @@ Notifications are sent when:
| Variable | Value | Description |
|----------|-------|-------------|
| `GDRIVE_FOLDER_ID` | `1laRKAyxo0v817SstfM5XkpbWiNKNAMSX` | Root folder of public share |
| `NOTIFY_ON_SUCCESS` | `true` | Send daily "all clear" summary |
| `NOTIFY_ON_SUCCESS` | `false` (prod) | Send an "all clear" email on every clean run; off in prod because the post-job trigger runs after every job |
| `EMAIL_TO` | `gen@nomadkaraoke.com` | Email recipient |
| `EMAIL_FROM` | `gen@nomadkaraoke.com` | Email sender |
| `IN_FLIGHT_GRACE_MINUTES` | `30` (default) | Tracks with any file created in Drive this recently count as still uploading and are left out of the gap check |
### Secrets (Cloud Function)
@@ -202,6 +203,16 @@ The post-job trigger uses a 5-minute Cloud Tasks delay (via `gdrive-validation-q
**IMPORTANT: Never add new entries to `KNOWN_GAPS`.** The existing known gaps are historical (pre-generator era, over a year old). Every new gap is a real bug that must be investigated and fixed.
**Tracks still uploading are not gaps.** A job's packaging step uploads MP4, then
MP4-720p, then CDG over ~2 minutes, and another job's post-job trigger can run in that
window. On 2026-11-07 NOMAD-1871's check ran at 12:24:47 UTC while NOMAD-1872 was
uploading (12:23:39–12:25:40) and emailed "CDG/MP4-720p: missing 1872"; the files were
all there a minute later. Since then the validator reads each file's Drive `createdTime`
and leaves out of the gap check any track with a file created in the last
`IN_FLIGHT_GRACE_MINUTES` (30). That track is still checked by its own post-job trigger
(which names its `brand_code`) and by the daily 9 PM run. If you get a gap alert for the
newest track, first check whether that job was in `packaging` at the alert time.
**Investigation steps:**
1. **Identify the missing brand code(s)** from the alert (e.g., "MP4: missing 1295")
@@ -53,7 +53,7 @@ gcloud functions deploy "${FUNCTION_NAME}" \
--min-instances=0 \
--max-instances=1 \
--service-account="gdrive-validator@${PROJECT_ID}.iam.gserviceaccount.com" \
--set-env-vars="GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,EMAIL_TO=gen@nomadkaraoke.com,EMAIL_FROM=gen@nomadkaraoke.com,NOTIFY_ON_SUCCESS=true" \
--set-env-vars="GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,EMAIL_TO=gen@nomadkaraoke.com,EMAIL_FROM=gen@nomadkaraoke.com,NOTIFY_ON_SUCCESS=false,LOG_EXECUTION_ID=true" \
--set-secrets="PUSHBULLET_API_KEY=pushbullet-api-key:latest,POSTMARK_SERVER_TOKEN=postmark-server-token:latest" \
--project="${PROJECT_ID}"
@@ -16,6 +16,7 @@ import re
import json
import logging
from collections import defaultdict
from datetime import datetime, timedelta, timezone
from typing import Union
import functions_framework
import requests
@@ -38,6 +39,21 @@ EMAIL_FROM = os.environ.get("EMAIL_FROM", "gen@nomadkaraoke.com")
# Set to "false" to only receive notifications when issues are detected
NOTIFY_ON_SUCCESS = os.environ.get("NOTIFY_ON_SUCCESS", "true").lower() == "true"
# A track whose newest file landed in Drive this recently is treated as still
# uploading: a job's packaging step uploads MP4, then MP4-720p, then CDG over a
# couple of minutes, and the post-job trigger for an earlier job can fire right
# in the middle (2026-11-07: NOMAD-1871's check flagged NOMAD-1872 mid-upload).
# Gaps for such tracks are held back; the track's own post-job trigger (which
# names its brand_code) and the daily run still check it once the upload ends.
def _grace_minutes() -> int:
try:
return int(os.environ.get("IN_FLIGHT_GRACE_MINUTES", "30"))
except ValueError:
return 30
IN_FLIGHT_GRACE_MINUTES = _grace_minutes()
# Known gaps configuration - matches the local validate_and_sync.py script
# These are HISTORICAL ONLY (pre-generator era, before Dec 2024). Do NOT add new entries.
# New gaps are bugs — see docs/GDRIVE-VALIDATOR.md § "Sequence Gap Detected" for fix procedure.
@@ -93,7 +109,9 @@ def get_drive_service():
return build('drive', 'v3', credentials=credentials)
def list_folder_contents(service, folder_id: str) -> dict[str, list[str]]:
def list_folder_contents(
service, folder_id: str, created_times: dict[str, str] | None = None
) -> dict[str, list[str]]:
"""
List all files in a Google Drive folder and its subfolders.
@@ -102,6 +120,9 @@ def list_folder_contents(service, folder_id: str) -> dict[str, list[str]]:
- CDG/ (contains .zip files)
- MP4/ (contains .mp4 files)
- MP4-720p/ (contains .mp4 files)
If ``created_times`` is given, it is filled with ``{filename: createdTime}``
(RFC 3339) so the caller can tell which tracks are still being uploaded.
"""
results = {}
@@ -134,12 +155,15 @@ def list_folder_contents(service, folder_id: str) -> dict[str, list[str]]:
query = f"'{folder_id_inner}' in parents and trashed=false and mimeType!='application/vnd.google-apps.folder'"
response = service.files().list(
q=query,
fields="nextPageToken, files(name)",
fields="nextPageToken, files(name, createdTime)",
pageSize=1000,
pageToken=page_token
).execute()
files.extend([f['name'] for f in response.get('files', [])])
for f in response.get('files', []):
files.append(f['name'])
if created_times is not None and f.get('createdTime'):
created_times[f['name']] = f['createdTime']
page_token = response.get('nextPageToken')
if not page_token:
@@ -187,10 +211,39 @@ def get_sequence_number(filename: str) -> int | None:
return None
def validate_files(files_by_folder: dict[str, list[str]]) -> dict:
def recently_uploaded_sequences(
created_times: dict[str, str],
now: datetime | None = None,
grace_minutes: int = IN_FLIGHT_GRACE_MINUTES,
) -> set[int]:
"""Sequence numbers that have any file created within the grace window."""
now = now or datetime.now(timezone.utc)
cutoff = now - timedelta(minutes=grace_minutes)
recent = set()
for filename, created in created_times.items():
seq = get_sequence_number(filename)
if seq is None:
continue
try:
ts = datetime.fromisoformat(created.replace("Z", "+00:00"))
except (TypeError, ValueError):
continue
if ts.tzinfo is None:
ts = ts.replace(tzinfo=timezone.utc)
if ts >= cutoff:
recent.add(seq)
return recent
def validate_files(
files_by_folder: dict[str, list[str]], in_flight: set[int] | None = None
) -> dict:
"""
Validate all files and return a dict of issues found.
``in_flight`` sequence numbers (tracks still uploading) are not reported
as gaps.
Returns:
{
'duplicates': {'MP4': {1107: ['file1.mp4', 'file2.mp4']}, ...},
@@ -255,7 +308,7 @@ def validate_files(files_by_folder: dict[str, list[str]]) -> dict:
min_num = min(sequence_numbers)
expected_range = set(range(min_num, global_max + 1))
excluded = EXCLUDED_NUMBERS.get(folder_name, set())
missing = sorted(expected_range - sequence_numbers - excluded)
missing = sorted(expected_range - sequence_numbers - excluded - (in_flight or set()))
if missing:
issues['gaps'][folder_name] = missing
@@ -501,7 +554,8 @@ def validate_gdrive(request):
service = get_drive_service()
# List all files
files_by_folder = list_folder_contents(service, GDRIVE_FOLDER_ID)
created_times: dict[str, str] = {}
files_by_folder = list_folder_contents(service, GDRIVE_FOLDER_ID, created_times)
if not files_by_folder:
logger.warning("No folders found in Google Drive")
@@ -511,7 +565,10 @@ def validate_gdrive(request):
}), 200, {"Content-Type": "application/json"}
# Validate files
issues = validate_files(files_by_folder)
in_flight = recently_uploaded_sequences(created_times)
if in_flight:
logger.info("Holding back gap checks for tracks still uploading: %s", sorted(in_flight))
issues = validate_files(files_by_folder, in_flight=in_flight)
# Targeted same-run completeness for the just-published track.
if brand_code:
@@ -31,6 +31,7 @@ from main import ( # noqa: E402
expand_gaps,
has_issues,
check_track_completeness,
recently_uploaded_sequences,
)
@@ -333,3 +334,35 @@ class TestHasIssues:
def test_has_gaps(self):
assert has_issues({"duplicates": {}, "invalid_filenames": {}, "gaps": {"MP4": [3]}})
class TestInFlightUploads:
"""2026-11-07: NOMAD-1871's post-job check ran while NOMAD-1872 was mid-upload
(MP4 done, 720p + CDG not yet) and emailed a false sequence-gap alert."""
NOW = __import__("datetime").datetime(2026, 11, 7, 12, 24, 47, tzinfo=__import__("datetime").timezone.utc)
def _files(self):
return {
"MP4": [make_mp4(i) for i in range(100, 106)],
"MP4-720p": [make_mp4(i) for i in range(100, 105)],
"CDG": [make_cdg(i) for i in range(100, 105)],
}
def test_track_still_uploading_is_not_a_gap(self):
created = {make_mp4(105): "2026-11-07T12:24:30.000Z", make_mp4(104): "2026-11-06T09:00:00.000Z"}
in_flight = recently_uploaded_sequences(created, now=self.NOW, grace_minutes=30)
assert in_flight == {105}
issues = validate_files(self._files(), in_flight=in_flight)
assert issues["gaps"] == {}
def test_old_track_missing_is_still_a_gap(self):
created = {make_mp4(105): "2026-11-07T10:00:00.000Z"}
in_flight = recently_uploaded_sequences(created, now=self.NOW, grace_minutes=30)
assert in_flight == set()
issues = validate_files(self._files(), in_flight=in_flight)
assert issues["gaps"] == {"MP4-720p": [105], "CDG": [105]}
def test_bad_timestamps_and_non_track_files_ignored(self):
created = {"desktop.ini": "2026-11-07T12:24:00Z", make_mp4(105): "garbage"}
assert recently_uploaded_sequences(created, now=self.NOW) == set()