Files
karaoke-gen/docs
Andrew Beveridge ☄️andClaude Opus 5.5 70b2f4279a feat(alerts): collapse error bursts, suppress link previews, de-noise #alerts (v0.278.9) (#1188)
Andrew's goal: "I'd like to make sure that our alerts channel only has
alerts in it that are actionable and important, or are just signal and
not noise." This follows #1186 (#noise routing).

## Changes
1. **Burst collapse** (`error_monitor/monitor.py`, `discord.py`,
`config.py`). When a run has more than 3 new patterns for one service,
including leftover un-alerted ones, the monitor sends ONE #alerts
message. It has the count, total occurrences, the top 5 by count,
first/last seen, and a `scripts/query-error-patterns.py --service X
--status new` pointer. All of those patterns are marked alerted. The
grouping is deterministic, since LLM grouping is off in prod. The
10-messages-per-run cap still applies, and unsent patterns are picked up
again next run. Evidence: on 2026-10-07/08, one mb-refresh run produced
73 separate alerts.
2. **No link previews.** Every error-monitor post (`_post`) and every
`ops_alerts` post now sets `flags: 4` (SUPPRESS_EMBEDS), through a new
`suppress_embeds` kwarg on `DiscordNotificationService.post_message`.
The YouTube "New upload" post keeps its embed.
3. **Loop watchdog.** `EVENT_LOOP_STALL in progress` is now always
WARNING. The `ended` line keeps the existing ERROR rule. In the last two
weeks there were 24 "in progress" pattern alerts on top of 15 "ended"
ones. I checked `infrastructure/modules/monitoring.py`
`EVENT_LOOP_STALL_LOG_FILTER`: it keys on `"EVENT_LOOP_STALL ended"` +
`severity>=ERROR`, so this needs no Pulumi change.
4. **Frontend chunk-load errors** (`frontend_ingestion.py`,
`firestore_adapter.py`). Reports matching `isChunkLoadError` collapse
into one pattern per build with `alert_min_hourly_count=10`. The monitor
holds that pattern un-alerted until more than 10 arrive within an hour,
which means a broken deploy. Separately, Safari/Firefox stacks now get
the message prepended before hashing (a pattern was named
`@https://gen.nomadkaraoke.com`). Chrome hashes are unchanged. Existing
Safari/Firefox patterns will re-key once, and burst collapse groups
them.
5. **known_issues.** Added a service-scoped ignore for
`github-runner-manager` "no available instance" aborts during scale-up
(seen 09-20, 09-23, 09-27, 10-08). The same message from other services
still alerts.
6. **divebar-mirror schedulers.** Added `attempt_deadline="600s"` on
both `divebar-mirror-daily` and `divebar-mirror-refresh`; the function
timeout is 540s. With the 180s default, Cloud Scheduler retried syncs
that were still running, and the overlapping runs hit OOM (4x in 30
days).

## ⚠️ Pulumi
`infrastructure/modules/divebar_mirror.py` changed, so **this needs
`pulumi up` by Andrew before merge**. Auto-merge is NOT enabled.

## Testing
- New and updated unit tests in
`tests/unit/services/error_monitor/{test_monitor,test_discord,test_frontend_ingestion,test_firestore_adapter,test_known_issues}.py`,
`backend/tests/{test_loop_watchdog,test_discord_service,test_ops_alerts}.py`,
and `tests/unit/test_divebar_mirror_scheduler.py` (an AST guard:
deadline > function timeout).
- `pytest backend/tests --ignore=backend/tests/emulator`: 6421 passed,
36 skipped.
- `tests/unit/services/error_monitor`: all pass.
- Local CodeRabbit review (`--committed`) completed with 0 findings.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_019pmG2XAgtUguEJWLFpkKHN

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 05:13:13 +00:00
..

Karaoke-Gen Documentation

Status: Production-ready. First full E2E flow completed 2025-12-28.

What Works

  • Web App: https://gen.nomadkaraoke.com
    • / - Landing page with pricing
    • /app - Main app (upload audio, review lyrics, download videos)
      • Bulk Mode (Jun 2026): submit up to 100 jobs at once by text or by album (MusicBrainz lookup + KaraokeNerds availability); auto-picks confident lossless audio, parks the rest in a review queue. See docs/API.md § Bulk Mode.
    • /admin - Admin dashboard (user management, job monitoring)
  • Medleys (agent-run, local): one gen job per original song, then scripts/medley/make_medley.py cuts the audio and renders a synced karaoke video from the reviewed lyrics. See MEDLEYS.md.
  • Backend API: https://api.nomadkaraoke.com - FastAPI on Cloud Run
  • CLI Tools: karaoke-gen (local), karaoke-gen-remote (cloud)

Current State

Component Status
Audio upload & separation Working
Lyrics transcription Working
Auto-correction (agentic + heuristic) Disabled (raw transcription goes to human review)
AI auto-correct suggestions (opt-in) Working ("AI Suggest" button in lyrics review — whole-song LLM call, per-suggestion accept/reject)
Combined review (lyrics + instrumental) Working
Auto-approval (fully-auto review) Live (default review_mode="auto"): when the scorer is fully confident (synced-reference lyrics with all gaps AI-covered + no audible backing vocals), the review screens are skipped and the job renders straight through; every other job falls back to human review. Verdict on every job in processing_metadata.auto_approval. Kill switch AUTO_APPROVAL_ENFORCE_ENABLED=false; per-job review_mode="always_review" (admin-editable). See docs/archive/2026-08-25-full-auto-review-design.md
Review replay tool (admin) Working (?replay=1 read-only replay of completed jobs — see docs/REPLAY.md)
Preview video generation Working
Multi-format encoding Working
Token-based auth Working
Magic link auth Working
Payment flow (Stripe) Working
Feedback-for-credits Working (2+ jobs → 1 free credit)
Admin dashboard Working
Abuse prevention & YouTube quota Working (blocklists, IP/fingerprint rate limiting, credits on verification, quota-aware YouTube uploads)
CI/CD self-hosted runner Working (GCP)
E2E happy path test Working (~20-25 min full pipeline)
White-label B2B portals Working (Vocal Star, Singa)
Private (non-published) tracks Working
Custom video styling (private jobs) Working
Song suggestions & fuzzy correction Working (catalog lookup on Step 2 + "Did you mean?" typo correction)
Community version detection Working (via karaokenerds.com)
Review session backup/restore Working (auto-save + restore UI)
Review reminder & auto-expiry Working (24h reminder, 48h cancel + refund)
i18n / Multilingual (frontend) Working (next-intl, en/es/de, 1062 strings, [locale] routing)
Referral system Working (referral links with configurable discounts, 20% cash kickback via Stripe Connect, auto-payout at $20)
Error monitor New (Cloud Run Job hourly — log collection, normalization, LLM dedup/grouping, Discord alerts, auto-resolution, daily digest)
Edge security (Cloudflare) Working (api.nomadkaraoke.com proxied — WAF + rate limit + origin lock enforced; 2026-07-20)

Known Issues

(No known issues)

Pending Work

(No pending work items)

Recent Changes

  • Guitar chords in karaoke videos (2026-10-16, v0.277.0). OutputGenerator.generate_outputs(chords=[ChordChange(...)]) draws each chord above the word it lands on (it lights up when played) in a row reserved above every line, plus a "now / next" chord box in the top-right corner that also covers intros and instrumental breaks. Chords come from scripts/medley/detect_chords.py, an ensemble of ChordMini models over the mix, instrumental and harmonic stems. It's used by the medley script for now ("chords": true); there's no job/UI option yet. Theme keys: chord_color, chord_font_size/chord_font_ratio, chord_max_visible_lines. See MEDLEYS.md § Guitar chords.

  • Customer chord charts + medley mastering fix (2026-10-17, v0.278.5). make_medley.py chart turns a customer's chords-over-lyrics chart (.txt or .docx, capo shapes kept, ChordChange.capo shown in the chord box) into timed chords on the reviewed lyrics, with detected chords filling intros and breaks (scripts/medley/chord_chart.py). The medley master is now one static gain + limiter instead of single-pass loudnorm, which had boosted a near-silent a cappella intro into robotic artifacts. See MEDLEYS.md.

  • Local E2E stack + email-code sign-in (2026-10-15, v0.275.0). Plan: archive/2026-10-15-local-stack-e2e-plan.md.

    • Local stack. scripts/local-stack.sh up --frontend runs the real API on the Firestore + GCS emulators with FAKE_WORKERS=1. In-process fake workers (backend/services/fake_workers/) move jobs through their real states in seconds with small real media; the search, download, Stripe Checkout, email, Secret Manager and encoding-VM seams are faked or skipped. The /api/test/* router (only mounted in that mode; startup refuses it in production) seeds scenarios, tunes the fakes per user, exposes captured email and serves a fake checkout page.
    • New suite. frontend/e2e/local-stack covers sign-up, make, review, instrumental, download, buying credits, the duration-confirm and failure states, and code sign-in through the real UI and API in about 1 minute. It runs in CI as the required local-stack-e2e job. Mobile Maestro flows will use the same stack. See TESTING.md § Local stack.
    • Email-code sign-in for the mobile app (nomad-mobile spec §4.1). POST /api/users/auth/send-code emails a 6-digit code plus the usual magic link. POST /api/users/auth/verify-code returns the same response as magic-link verify, including the welcome credit. Codes are HMAC-hashed, expire after 10 minutes, allow 5 attempts, are single-use, and sends are limited per email and per IP. The code logic is shared with kjbox (backend/services/login_code_store.py). See API.md.
    • Fix: jobs confirmed at "This song is longer than expected" never reached review. confirm-duration moved the job to separating_stage1. That status can only move on to separating_stage2, and the lyrics worker only advances jobs from the prep statuses, so every confirmed job would have stalled. It now resumes at downloading. No production job had hit this path yet; the local-stack suite found it on its first run.
  • Encoding worker heals itself after a bad boot (2026-10-06, v0.266.4). The primary encoding VM booted before its network was up, so bootstrap.sh couldn't download startup.sh and fell back to the image's startup-fallback.sh. That script left out ENCODING_API_KEY and GCE_METADATA_MTLS_MODE=none, so every render and preview failed with CERTIFICATE_VERIFY_FAILED on the metadata server. A Randy Vild render looped in render_pending_capacity for an hour until the service was restarted by hand. Fixes:

    • The worker defaults GCE_METADATA_MTLS_MODE=none itself (gce_encoding/main.py), so every boot path gets it.
    • The worker refuses to start when the API key is missing or it can't get GCP credentials, and exits after an infra/auth job failure once idle. systemd then re-runs the bootstrap (gce_encoding/self_heal.py; only on worker VMs).
    • The Packer bootstrap retries the GCS download for about 2 minutes. The fallback writes the mTLS setting and exits if it can't fetch the API key. This needs the new image.
    • An infra failure on the primary now logs at ERROR, so it alerts.
    • The job card stops showing "(waiting for encoding server availability)" once a retried render reports progress: state_data.render_pending_capacity is cleared.
  • Encoding-service construction moved off the event loop (2026-10-03, v0.264.1). The new loop watchdog's first prod catch was a 9.9s stall in /api/health/detailed. A cold get_encoding_service() imports google.cloud.compute_v1 (~10s). It's now built in the startup warmup thread, is thread-safe, and async callers use asyncio.to_thread. (v0.264.0 was shipped by both #1120 and #1122.)

  • API event-loop freezes fixed + degradation telemetry identifies users (2026-10-03, v0.264.0). The "Reconnecting" pill and "servers unavailable" banner were mostly caused by the single API instance freezing for 20-45s. Plan + evidence: docs/archive/2026-10-03-backend-loop-freezes-telemetry-plan.md.

    • Screens worker: now runs inline in its Cloud Tasks request (so CPU stays allocated under --cpu-throttling), with the heavy steps in worker threads: 4K PIL renders, GCS uploads, review-audio transcode, timing signals and pre-apply.
    • Hourly stale-review cron: runs in a thread.
    • Matplotlib font cache: pre-built in the base image (MPLCONFIGDIR=/opt/mplconfig), and pyplot is imported lazily.
    • Route handlers: 161 async def routes that never awaited (sync Firestore/GCS/email/Gemini on the loop) are now plain def, so they run in FastAPI's threadpool. backend/tests/test_no_blocking_async_routes.py keeps it that way, and also blocks sync routes that call asyncio.create_task.
    • Loop watchdog: new backend/services/loop_watchdog.py logs EVENT_LOOP_STALL with the blocking stack. Stalls ≥5s are recorded to client_events (server_loop_stall). A Cloud Monitoring alert fires on ≥2 stalls of ≥10s in 30 min.
    • Telemetry identity: client_events now records the server-resolved user (plus admin/internal/test flags), device fingerprint, tab id, tenant, and a banner_recovered event with episode duration. Use scripts/client_events_report.py for summaries.
  • Gemini moved off Vertex AI (2026-10-01, v0.262.0): every Gemini call (auto-correct compare leg, match-judge, parse-titles, credit eval, tenant bulk analyze, custom lyrics, lyrics translation, error-monitor LLM, agentic correction, translate.py) now uses the Gemini Developer API with the gemini-api-key secret via backend/services/gemini_client.py, so no AI spend lands on the nomadkaraoke project (Vertex APIs disabled there). Quota/billing/key failures degrade per caller and send one Discord ops alert per 6h. See LESSONS-LEARNED "Gemini: Developer API key, never Vertex".

  • Translated lyrics follow-ups + review waveform spinner (2026-10-01, v0.261.0):

    • The translation row now renders at 70% of the unsung colour's opacity, with outline and shadow faded to match, so it reads as secondary to the sung line (subtitles.py:TRANSLATION_OPACITY). A theme's karaoke.translation_color still wins.
    • Published videos with translations get "(With Translation into X)" in the YouTube title, before "(Karaoke)". Only the artist/title get truncated, never the label. The description also gets a notice. This applies to the live upload and the quota queue, but only when state_data.lyrics_translation.status == "translated" (youtube_description.build_youtube_title).
    • The review preview modal explains that the preview is a quick black-background sync check, and that translated lyrics are added after review.
    • Lyrics review Waveforms toggle: shows a spinner and tooltip while audio separation is still producing the vocal stem (the peaks endpoint returns 202). Previously the rows were just empty (VocalsAudioDataLoaderContext.status).
  • Translated lyrics: Japanese/Chinese translations no longer split mid-word (2026-10-01, v0.260.1): when a long lyric line is split across rows, its translation is shared between them. Unspaced scripts were cut at an arbitrary character, so prod job ca2e881a showed 無|駄 ("waste") across two rows. split_text_proportionally now prefers punctuation, then a break after a Japanese particle or before a kanji/katakana word start, and only cuts mid-word as a last resort; it no longer leaves a piece empty when there are enough characters. Thai can still split inside a word (no word-boundary data).

  • Translated lyrics for language learners (2026-10-01, v0.260.0): an "Add translated lyrics" option at job creation (any job, free for now) with a language picker (33 UI locales) and a live preview of the real video frame (GET /api/themes/translation-preview). After review, the render worker translates the final lyrics line by line with Gemini (backend/services/lyrics_translation.py, GCS-cached so re-renders are free) into lyrics/translations.json; the landscape and portrait videos draw each translation in smaller unsung-colour text beneath its line, 3 lines per screen (ass/config.py:translation_layout, LyricsLine._create_translation_event). Translations are their own ASS events with auto paragraph direction, so mixed scripts work (a Hebrew song keeps its right-to-left highlight with an English translation beneath, and vice versa); long translations wrap/scale to fit; a line split by the resizer shares its translation across the pieces. Not editable; a translation failure renders without it (state_data.lyrics_translation). See archive/2026-10-01-translated-lyrics-plan.md.

  • User-uploaded inputs kept forever (2026-10-08, v0.268.2): uploads/ had a 7-day lifecycle delete and only the audio worker copied the mix out, so bring-your-own-instrumental/tenant jobs (and any job left in review >7 days) lost their mix and instrumental; re-renders 404'd. Every pipeline worker now copies uploads/{id}/** to jobs/{id}/input/** and repoints the job (backend/services/input_persistence.py); POST /api/admin/persist-uploads backfills; uploads/ now expires after 180 days. randy-vild inputs were restored from Dropbox. See LESSONS-LEARNED.md.

  • End screen fully editable + previewed; title card / end screen can be left out (2026-10-08, v0.268.0): Randy Vild reported "large text at the end even though I have in my settings smaller text like in my beginning". The theme editor's sizes/positions only edited intro.*; the end screen keeps its own end.* layout (title/artist regions, artist colour, capitals), and the editor neither showed it nor previewed it. The End screen tab now has its own colours, size/position sliders, show/hide for the song title and artist name, capitals toggles, and Copy title card layout (matches the title card and moves the closing message above the title so nothing overlaps). The preview renders the real end screen (create_end_video). Both tabs have an "Include …" switch (intro.enabled / end.enabled); leaving a screen out flows through the screens worker, /retry, the orchestrator, the GCE encoder, LocalEncodingService and the portrait render (see API.md).

  • Theme edits reach tracks in progress; outdated finished tracks get a re-render prompt (2026-10-06, v0.267.0): Randy Vild edited his theme and the next video still had the old title card. Its job had copied the theme at creation, 30 minutes before the edit, and was waiting for encoding capacity when he saved. A theme save now re-applies the theme to every tenant track that hasn't started rendering. If a track's screens were already made, the render worker regenerates them first, at render time, so nobody is interrupted mid-review. Editing a finished tenant track also picks up the current theme. After saving, the theme editor lists finished tracks that still have the old look and offers a quiet Re-render all N (no per-track emails). Those tracks' cards show an "Older theme" badge. Admins no longer see two identical Re-render buttons on portal tracks (backend/services/theme_change_service.py).

  • Re-render works for tenants that archive to Dropbox (2026-09-30, v0.254.1): randy-vild jobs get a brand code and a Dropbox archive folder (RVILD-0001 - …), so v0.253.0's "never distributed" guard rejected the very jobs the button was for. A re-render now reuses the job's brand code and overwrites the same folder instead of allocating a new code. YouTube/GDrive-published jobs are still rejected.

  • Tenant "Re-render" with the current theme + edited tenant tracks no longer lose their downloads (2026-09-30, v0.253.0): Finished portal tracks have a Re-render button (POST /api/jobs/{id}/rerender). It re-applies the tenant's current theme and rebuilds the screens, lyrics video, encodes and CDG/TXT from the already-reviewed lyrics, with no review step (backend/services/theme_rerender_service.py, same mechanism as the private→public visibility change). The encoding worker caches finished jobs by id in memory, so a re-run submitted while the VM was still up could get the previous render back. Render and encode now submit as {job_id}_g{worker_generation} (encoding_worker_job_id). Separately, outputs_deleted_at (set by Edit) was only cleared when the re-render uploaded to YouTube/Dropbox/GDrive, so edited tenant jobs re-rendered but hid every download (job 2579a1ea). Every successful render now clears it.

  • Hold a word's resize handle at the row edge to keep extending it (2026-09-30, v0.252.0): in the lyrics review Waveforms rows (and the Edit Segment timeline) the view is the segment ±1s and was fixed during a drag, so a word ending 5s+ too early took several drag-release cycles. Holding a resize handle within 24px of the edge it's heading toward (or past it) now keeps growing the word at 1–4 s/s (faster nearer the edge), zooming the view out so the handle stays under the cursor. It stops when the pointer leaves the zone, on release, or at the neighbouring word, including the next/previous line's first/last word (computeNeighbourBoundsBySegment), else the audio bounds. Drags are tracked on window, so leaving the row no longer abandons them, and resizes clamp to the neighbour instead of freezing short of it (TimelineEditor.tsx).

  • Mismatched user instrumentals are lined up, not rejected + tenant bulk upload modal (2026-09-29, v0.251.0): A Randy Vild bulk upload lost 3 of 10 tracks to the 0.5s duration check, and all three were valid edits of the same song: an extra 2.97s count-in on the instrumental, a 6.5s longer outro, and an outro 4.45s short. uploads-complete now conforms the instrumental to the mix instead of cancelling (backend/services/instrumental_conform.py, reusing derived_vocals.find_offset). It measures the offset at 25% and 75% of the song, trims/pads the start, and fades-and-cuts or pads the end. It still rejects files that don't correlate (wrong pairing, ~0.01 vs 0.78–0.90 for real pairs) or whose offset changes mid-song. The single-track tenant form and own-instrumental field now only check the size cap in the browser. Tenant bulk: shows UploadProgressModal for the whole batch (aggregate bytes, speed/ETA, tracks submitted/failed, offline notice); a job the server rejected is dropped so a retry creates a fresh job (previously a retry's 400 on the cancelled job was read as "already processing" and shown as Submitted).

  • Vocal waveforms for own-instrumental jobs (2026-09-26, v0.244.1): jobs with a user-supplied instrumental skip GPU separation, so the lyrics review Waveforms mode had no vocals stem (/vocals-peaks 404). The lyrics worker now derives approximate vocals on CPU, alongside transcription: it aligns the instrumental to the mix by FFT cross-correlation (±2s), fits a gain by least squares, and subtracts (backend/services/derived_vocals.py, numpy + ffmpeg, a few seconds). The result is stored as stems.vocals_derived (mono 22.05 kHz FLAC), which vocals_stem_path uses only as a last resort, so screens pre-computes the peaks as usual. It's waveform-only: never shipped as a deliverable stem, never downloaded by the remote CLI, and never used for the original-vocals guide. Safeguards: alignment correlates a fixed 60s excerpt (memory doesn't scale with track length); inputs over 20 min are skipped; ffmpeg calls time out; the handoff waits at most 3 min; nothing is registered when the instrumental didn't cancel (residual > 85% of the mix), since a full-mix envelope would mislead timing edits. Best-effort: any failure logs and the job proceeds.

  • User-instrumental renders: WAV/M4A/OGG/AAC no longer fail (2026-09-26, v0.243.1): the orchestrator render path sent only instrumental_selection="custom" to the GCE encoder, whose custom lookup accepted just *custom_instrumental*.{flac,mp3}. Tenant MP3s worked, but a .wav user instrumental failed with "No instrumental audio found" (job 3ea34552). The orchestrator now passes existing_instrumental (the encoder downloads it and matches any extension), and the custom lookup (resolve_instrumental in gce_encoding/main.py) accepts any audio extension.

  • Crash reports carry diagnostics + breadcrumbs (2026-09-26, v0.243.1): A Firefox "Uncaught out of memory" froze a tab mid-upload, and the alert was useless. The browser gave no Error object, so the reporter's own stack became the pattern, and that stack pushed URL/UA/build out of Discord's view (extra was never kept). Now lib/diagnostics.ts keeps a 30-entry breadcrumb trail (navigation, clicks, upload phases, errors) and snapshots page age, DOM/canvas/audio counts, live blob URLs and the JS heap (Chromium) into every report. Errors without an Error object are flagged synthetic (stack dropped, location kept) and group by message. One page load can send at most 20 reports. The alert sample now lists message → location → URL/UA/build → Diag → Trail → trimmed stack, and the full extra is logged as client_extra on frontend_crash_reported.

  • Unified browser upload flow + bring-your-own instrumental (2026-09-26, v0.243.0): Two invisible upload failures on the same song. Job 8729f19a: the job was created and charged, then the browser upload was abandoned; the only UI hint was the "Creating..." button text and the card already said "Setting up", so it sat pending forever. Job 2d0c1ae0: a 44 MB instrumental upload on the review screen was a multipart POST through Cloud Run (32 MiB cap) and failed silently. Now every user upload (guided job creation, tenant single job, instrumental review) goes through one shared layer: lib/upload.ts (signed-URL PUT with progress, multi-file aggregate progress, client-side duration pre-check), hooks/useUploadTask.ts (progress state + beforeunload guard; tenant bulk uses the guard) and components/upload/UploadProgressModal.tsx (a modal that can't be dismissed, showing progress, speed and ETA). The guided flow always uses the signed-URL path, which fixed requires_audio_edit being silently dropped for 25MB+ uploads. Private upload jobs can attach the user's own instrumental (existing_instrumental), which skips separation and instrumental review. Instrumental review uploads go direct to GCS (instrumental-upload-url / instrumental-upload-complete), and the review now plays the upload via the returned signed audio_url (the old /audio-stream URL had no route). Backend: state_data.awaiting_upload → "Waiting for upload" label; the hourly stale-review run cancels + refunds jobs still awaiting upload after 6h (24h if some files landed; tenant bulk exempt); a failed browser upload cancels its own job immediately; uploads-complete checks ownership and cancels + refunds on an instrumental duration mismatch. See archive/2026-09-26-unified-upload-flow-plan.md.

  • Recent Jobs sort control (2026-09-23, v0.238.0): The dashboard's Recent Jobs list now has a "Sort by" dropdown (submission date, recently active, completion date, artist, title) plus an ascending/descending toggle, replacing the fixed newest-first order. Client-side sort (sortJobs() in lib/job-status.ts) applied to whatever page the existing limit/status/search filters already fetched — it doesn't refetch or sort across the full job history. "Recently active" uses updated_at, newly added to the Firestore summary-field projection (SUMMARY_FIELD_PATHS). There's no dedicated completion timestamp, so "Completion date" reuses updated_at for jobs in a terminal success status (complete/prep_complete) and pushes everything else to the end of the list.

  • Admin Dashboard Stats 18s → ~1s (2026-08-23): GET /api/admin/stats/overview?exclude_test=true streamed every user and job document in full to filter test data in Python — ~110 MB across 658 jobs (job docs carry state_data/timeline payloads), ~18s per dashboard load. Worse, the sync streaming ran inside an async route, blocking the event loop so every other request on the instance (/me, /config, payments summary) stalled behind it. Fix: Firestore .select() field masks on all three streams (jobs need only user_email/created_at/status; measured 15.0s → 1.1s for the jobs stream) and the whole computation now runs via asyncio.to_thread() so it can't stall other requests. Unit tests assert the field masks stay in place. Same sync-over-async trap as the Feb 2026 audio-search fix — see LESSONS-LEARNED.md.

  • Admin UI/UX Overhaul (2026-08-22): Made the admin pages usable on mobile and fixed the long-standing overflow problems on the job detail page (v0.197.0). Long status timelines (retried/edited jobs) now collapse to the last 6 stages behind a "+N earlier (M failed)" expander; the Source field no longer bleeds across the info grid (CSS grid min-w-0); the Audio Source Details modal got wrap-safe stacked rows plus an "Open in YouTube/Spotify" link built from provider + source ID; Configuration fields are click-to-edit with always-visible pencils. The jobs list search box jumps straight to a job when given a job ID, and the status filter is URL-synced (/admin/jobs?status=failed), which the dashboard's now-clickable stat cards and status-breakdown rows link into. Also fixed a real crash: the "Trigger Now" button in the reset-result dialog referenced a deleted handler. Users list/detail, breadcrumbs, and the admin header are all responsive down to phone widths. 4 new e2e tests.

  • Portrait (9:16) Karaoke Video Output (2026-07-13): When enabled (PORTRAIT_RENDER_ENABLED, default on), every finalised job also gets a portrait 1080×1920 karaoke video for phone viewing / social sharing, distributed to GCS + the Dropbox brand-code folder only (never Drive/YouTube). It is a genuine re-render at portrait resolution with re-wrapped lyrics — not a transform of the finished landscape video (letterbox = tiny lyrics, crop = cuts the ends off every line; proven in the design experiments). The new karaoke_gen/portrait/ module reuses the same SubtitlesGenerator engine + corrected-lyrics data, so word-by-word timing is identical; it adds an orphan-merging wrapper, a branded portrait background (wordmark/header + song/artist + footer), and title/end cards. Rendered on the GCE encoder inside run_encoding (best-effort + PORTRAIT_RENDER_ENABLED flag; a failure never fails the job) and carried to Dropbox via the existing finals/ upload path. ~15–30s + ~10 MB per job. See archive/2026-06-14-portrait-video-design.md and archive/2026-07-13-portrait-video-plan.md.

  • Cloudflare Edge Security for the Backend API (2026-07-20): api.nomadkaraoke.com is now proxied through Cloudflare (WAF + rate limiting + origin lock), after a vuln-scanner hammered the previously directly-exposed Cloud Run origin. WAF blocks exploit/secret-exposure paths (contains operator — regex needs a Business plan); per-IP rate limiting excludes /api/internal/*; a Transform Rule injects a secret X-Edge-Auth header that the backend's EdgeAuthMiddleware (EDGE_AUTH_MODE=enforce) requires, blocking direct-to-*.run.app bypass (health/root exempt). Key finding: the Cloud Run managed cert can't provision behind Cloudflare, but zone SSL mode "full" (not strict) doesn't validate the origin cert, so it works and is renewal-safe — never switch the zone to "full (strict)". Managed as IaC in infrastructure/modules/edge_security.py. Also retuned the "High Error Rate" alert to 5xx-only (scanner 404s no longer page). See ARCHITECTURE.md § Edge Security, LESSONS-LEARNED.md, and archive/2026-07-20-edge-security-cutover-runbook.md.

  • AI Auto-Correct Suggestions (2026-06-11): Opt-in AI correction suggestions in the lyrics review UI — the third attempt at lyrics auto-correction, redesigned around the failures that led to disabling it in PR #321 (Jan 2026). One whole-song LLM call (default Gemini 3.1 Pro via Vertex, AUTO_CORRECT_MODEL env) compares the working transcription against reference lyrics and returns word-id-keyed suggestions; nothing is applied until the reviewer accepts each one (or accept-all) in a dedicated panel with per-row undo. New POST /api/review/{job_id}/auto-correct endpoint + backend/services/auto_correct/; frontend useAutoCorrect hook, AutoCorrectModal (knobs: adlib removal, insertions, confidence filter), AutoCorrectPanel. Accept/reject decisions land in the EditLog (ai_suggestion_* ops) to measure real accept rates. 5-model offline eval over 30 real jobs with human ground truth picked the default (Fable 5 ≈ Gemini 3.1 Pro ≫ GPT-5.2, which made lyrics worse). See archive/2026-06-10-lyrics-auto-correction-reeval-plan.md.

  • Syllable-Aware Custom Lyrics (2026-05-02): Enhanced the Custom Lyrics LLM mode with syllable-count validation and a per-line repair loop. Backend computes per-line target/candidate syllable counts via the existing SyllablesMatchHandler engine and re-prompts Gemini up to 3 times until lines fit budget (Δ ≤ 2 default, configurable via Verbatim/Loose/Balanced/Tight/Strict strictness). New GenerationSettings (allow_reword, allow_omit, fixed_line_count, strictness) exposed in the modal as a collapsible settings panel. New CustomLyricsPreview component shows per-line syllable annotations (target / actual / Δ) with severity badges and stop-reason banners. Variable-line-count mode (when fixed_line_count=false) re-times segments proportionally. Eval harness with disk-backed Gemini cache, per-line + per-fixture + corpus scorers, and 4 fixtures (1 from production, 3 handcrafted public-domain). v1-ships criteria validation (Year 5 ≥75% pass@2, macro corpus ≥70% pass@2) deferred to post-merge. See archive/2026-05-02-custom-lyrics-syllable-aware-design.md and archive/2026-05-02-custom-lyrics-syllable-aware-plan.md.

  • Custom Lyrics LLM Mode (2026-04-28): New AI-powered lyrics-replacement mode in the lyrics review modal. Users (or admins editing client jobs) can paste custom lyrics or instructions — or upload a document — and an LLM returns lyrics that match the original line count exactly, ready to drop into the existing replace-segments flow. Includes backend CustomLyricsService with file extraction, prompt construction, and line-count validation; new POST /api/review/{job_id}/custom-lyrics/generate endpoint; preview UI with regenerate/save controls; manual-sync toast guidance after save. See archive/2026-04-28-llm-custom-lyrics-design.md and archive/2026-04-28-llm-custom-lyrics-plan.md.

  • GCE Encoding Cold-Start Resilience (2026-04-27): Fixed renders failing when the encoding worker VM was cold (TERMINATED). Job 2c577535 hit this on 2026-04-24: the existing 90s connection retry window (sized for systemctl restart during deploys) was too short for a 2-5 min GCE cold boot. Fix: new EncodingWorkerManager.wait_for_worker_ready() two-phase poll (VM status → /health 200) is now awaited when ensure_primary_running() reports started=True (VM was actually TERMINATED). Deploy-restart fast path unchanged. Also: _format_exception() helper so empty aiohttp.ClientConnectorError messages still produce diagnostic logs (the original incident logged "GCE worker connection failed after 8 attempts: " with no detail). 17 new tests including a regression integration test. See archive/2026-04-27-gce-encoding-cold-start-design.md, LESSONS-LEARNED.md, and TROUBLESHOOTING.md.

  • Anti-Abuse: Credits on Verification + Rate Limiting (2026-03-21): Three-layer defense against free credit abuse. (1) Welcome credits now granted on first magic link verification, not account creation — unverified accounts get 0 credits. (2) Per-IP signup rate limiting: max 2 new accounts per IP per 24 hours, silent reject. (3) Device fingerprinting (FingerprintJS open source): browser fingerprint sent with magic link request, combined with IP for rate limiting. Either signal hitting 2/day triggers block. Returning users are never rate limited. Graceful degradation if fingerprinting fails. Frontend verify page shows "You received N free credits" on first login. See archive/2026-03-21-anti-abuse-credits.plan.md.

  • Review Reminder & Auto-Expiry (2026-03-19): Jobs stuck in review (awaiting_review / in_review) now get automatic cleanup. At 24 hours, a gentle reminder email is sent with a review link, help offer, and warning. At 48 hours, the job is auto-cancelled and the user's credit is refunded. Expiry notification email confirms the refund and current balance. Excludes made-for-you and tenant jobs. Powered by hourly Cloud Scheduler cron (stale-review-hourly). New endpoint: POST /api/internal/process-stale-reviews. Also fixed make test hanging on frontend Playwright E2E tests by splitting into test-frontend (Jest unit only) and test-frontend-e2e (Playwright, run manually). See archive/2026-03-19-review-reminder-autoexpiry.plan.md.

  • Step 2 Song Suggestions & Fuzzy Correction (2026-03-08): Replaced the Step 1 autocomplete with a simpler approach: plain text inputs on Step 1, then catalog lookup + community version check fire in parallel with the audio search on Step 2. Results appear as a "Song found in our database" panel (exact match) or correction suggestion panel while the user waits. Added fuzzy "Did you mean?" correction — when audio search returns poor results (Tier 3) and catalog has no exact title match, fetches the artist's discography and fuzzy-matches against the user's title using LCS similarity. Shows an amber banner with one-click correction + expandable alternatives list. Handles garbled artist fields via title-based fallback with artist tiebreaker. Visibility step reworked with action buttons at top for one-click select+continue. See archive/2026-03-08-step2-song-suggestions.plan.md and archive/2026-03-08-fuzzy-did-you-mean.plan.md.

  • Review Session Backup/Restore (2026-03-08): Server-side auto-save of lyrics review sessions replaces fragile localStorage-only mechanism. Sessions auto-save every 60s (if dirty), on preview video generation, and on page unload. Deduplication via SHA-256 hash prevents redundant saves. Correction data stored in GCS (can exceed Firestore's 1MB doc limit); metadata in Firestore subcollection (jobs/{jobId}/review_sessions/{sessionId}). Restore UI shows split-pane session browser with edit diff preview. Cross-job session search allows reusing edits across different jobs (with duration mismatch warning). On load, automatically prompts to restore if saved sessions exist. See archive/2026-03-07-lyrics-review-sessions-plan.md.

  • Job Observability Improvements (2026-03-07): Enhanced timeline events with a metadata field to capture structured output details (brand codes, YouTube video IDs, Dropbox paths, GDrive file IDs) at key moments — job completion, edit initiation, and admin output deletion. Added log_to_job() helper for API endpoints to write to Firestore job log subcollections (previously only workers logged there). Edit and admin-delete endpoints now snapshot previous outputs before cleanup and log each cleanup step. Distribution logging enhanced with video IDs, folder paths, and file IDs. See plans/improve-job-observability.plan.md.

  • Edit Completed Tracks (2026-03-06): Users can now edit completed karaoke tracks — correcting lyrics, changing instrumental selection, or updating artist/title metadata — without consuming additional credits. Clicking "Edit" on a completed job opens a confirmation modal (with optional artist/title editing), then cleans up all distributed outputs (YouTube video deleted, Dropbox folder removed, GDrive files deleted, GCS finals purged, brand code recycled), resets the job to awaiting_review, and redirects to the existing review screen. After re-review, the standard pipeline re-renders and re-distributes with a fresh brand code. New endpoint: POST /api/jobs/{job_id}/edit. New Job model field: edit_count. State machine updated: COMPLETE → AWAITING_REVIEW transition allowed. See archive/2026-03-06-edit-completed-tracks-plan.md and API.md.

  • Change Visibility (2026-03-06): Users can now toggle completed jobs between public and private via a "Make Public"/"Make Private" button on the job card. Two distinct flows: Public→Private is fast (~1-2 min, redistributes existing finals), Private→Public is slow (~15-30 min, resets styles to Nomad theme, regenerates screens, re-renders video). Confirmation modal explains the implications. Tenant jobs excluded. See archive/2026-03-06-change-visibility-plan.md and API.md.

  • Tenant Portal UI Rework (2026-03-04): Replaced the multi-step consumer wizard with a streamlined 4-field form (Artist, Title, Mixed Audio, Instrumental Audio) for B2B tenant portals. Vocal Star's portal now uses the new TenantJobFlow component instead of GuidedJobFlow. All tenant jobs are automatically private with tenant-specific distribution: Dropbox delivery to /MediaUnsynced/Karaoke/Tracks-VocalStar, Google Drive upload to tenant folder, VSTAR brand prefix. Backend adds _apply_tenant_overrides() helper that forces private, applies locked theme, and overlays tenant distribution settings. Private tenant jobs now use their own Dropbox/GDrive paths instead of the NonPublished override. Frontend conditionally renders TenantJobFlow vs GuidedJobFlow based on tenant detection. See archive/2026-03-04-tenant-portal-ui-rework.md.

  • Custom Video Styling for Private Jobs (2026-03-04): Users who select "Private (no YouTube upload)" can now customize their karaoke video styling. New options in the guided flow's customize step: upload a custom karaoke background image (replaces the starfield behind lyrics), upload a custom title card background image, and change artist/title text colors via color pickers. Both backgrounds show accurate canvas-based live previews. Style assets are uploaded to GCS via signed URLs after job creation (non-blocking — job falls back to default theme on upload failure). Backend: 2 new endpoints (style-upload-urls, style-uploads-complete) that validate ownership, cutoff status, and merge custom assets into the theme's style_params.json. See API.md.

  • Tenant Portal Frontend Separation (2026-03-03): B2B tenants now served from a dedicated Cloudflare Pages project (karaoke-gen-tenant) built with NEXT_PUBLIC_TENANT_PORTAL=true to tree-shake consumer content. A Cloudflare edge function injects window.__TENANT_CONFIG__ and CSS variables into HTML at the edge, eliminating the initial API round-trip for tenant config. Magic link emails now use tenant-specific URLs (e.g., vocalstar.nomadkaraoke.com/auth/verify). Admins can preview any tenant's branding via ?preview_tenant=<id>. See ARCHITECTURE.md.

  • YouTube Quota-Aware Upload System (2026-03-03): Replaced the hard-coded YouTube upload limit (10/day) with quota-unit tracking against the YouTube Data API v3's 10,000 units/day quota. Problem: 4 jobs on 2026-03-03 had YouTube uploads silently skipped with no user notification. The hard limit of 10 uploads/day was overly conservative (actual capacity: ~33 uploads/day at ~300 units each). Solution: (1) YouTubeQuotaService tracks quota units consumed per day in Firestore with Pacific Time midnight reset, (2) YouTubeUploadQueueService queues uploads when quota is low instead of silently skipping, (3) Hourly Cloud Scheduler job processes the queue when quota is available, (4) Users get a follow-up email with their YouTube link when deferred uploads complete, (5) Admin dashboard shows quota usage progress bar and queue management. New Firestore collections: youtube_quota (daily quota tracking), youtube_upload_queue (deferred uploads). New config: YOUTUBE_QUOTA_DAILY_LIMIT, YOUTUBE_QUOTA_UPLOAD_COST, YOUTUBE_QUOTA_SAFETY_MARGIN. See archive/2026-03-03-youtube-quota-fix-plan.md and API.md.

  • Singa Whitelabel Tenant (2026-03-03): Added Singa (singa.com) as second B2B tenant. Green/black branding, file-upload-only mode, locked to "singa" theme, restricted to @singa.com emails. Setup script generates all branding assets (4K backgrounds, CDG backgrounds, logo) programmatically with Pillow. Also fixed critical Zustand getter bug where computed store properties (branding, features, defaults) became stale static values after any set() call — Object.assign invokes getters and copies return values, destroying the getter. Added TenantLandingPage component for minimal tenant portal experience (logo + sign-in). Added Cloudflare Pages _headers for proper cache control. See LESSONS-LEARNED.md.

  • Song Lookup Autocomplete & Community Version Detection (2026-03-03): Added song catalog integration to the job creation flow. Two features: (1) Track autocomplete on the title field provides canonical artist/title names from MusicBrainz + Spotify data via the karaoke-decide catalog API, improving text consistency for YouTube releases. Autocomplete activates when the artist field has a value; selecting a suggestion overwrites both artist and title with canonical names. (2) Community version detection checks karaokenerds.com for existing high-quality karaoke versions — if found, a dismissible green banner suggests the user check those versions first before spending credits. New backend: 3 endpoints (/api/catalog/artists, /api/catalog/tracks, /api/catalog/community-check) with per-user rate limiting (20 req/min) and in-memory TTL caching. New services: catalog_proxy_service.py (proxies to karaoke-decide), karaokenerds_service.py (scrapes karaokenerds.com). New frontend components: AutocompleteInput (reusable), CommunityVersionBanner. See API.md.

  • Feedback-for-Credits (2026-03-01): Added a feedback-for-credits mechanism available to all users (replaces the old beta-only feedback flow). Users who complete 2+ karaoke videos can submit product feedback (star ratings + text) to earn 1 free credit (total possible free credits: 2 — 1 welcome + 1 feedback). New endpoints: GET /api/users/feedback/eligibility, POST /api/users/feedback. The /api/users/me response includes feedback_eligible: bool so the frontend can show/hide prompts without an extra API call. Frontend shows a dismissible banner on the dashboard and a "Earn 1 Free Credit" menu item in the user dropdown when eligible. Feedback stored in user_feedback Firestore collection with duplicate prevention via has_submitted_feedback flag on user model. Welcome email updated with credit-focused messaging and feedback teaser. See API.md.

  • Remove Beta Program from Homepage (2026-03-01): Removed beta enrollment from landing page and replaced with prominent "1 Free Credit" messaging. Beta enrollment endpoint preserved for backward compatibility but no longer actively promoted. See PR #430.

  • Fix Event Loop Blocking in Audio Search Endpoints (2026-02-28): Fixed /api/audio-search/{job_id}/select taking 20-36s to respond (should be <1s) and /api/audio-search/search sometimes taking 25-47s. Root cause: Async endpoints called sync methods that create new event loops via loop.run_until_complete(), blocking the FastAPI event loop — response body was queued but never flushed. Fix: RED/OPS and Spotify download paths now call FlacfetchClient async methods directly (matching YouTubeDownloadService pattern), generic downloads use asyncio.to_thread(), search endpoint uses search_async(). Also fixed InsufficientCreditsError occasionally returning 500 instead of 402 by using isinstance() check. Added 14 new tests. See LESSONS-LEARNED.md.

  • GDrive Cleanup Robustness + Brand Code Race Fix (2026-02-27): Fixed duplicate brand codes appearing in the public Google Drive share after the daily E2E test (NOMAD-1271 appearing for both the test job "piri - dog" and real song "Hot Mulligan - Pop Shuvit"). Root causes: (1) if gdrive_files: was falsy for empty dict {} — when GDrive upload silently failed (pre-PR #417), the entire GDrive cleanup block was skipped, (2) Brand code was recycled immediately after Dropbox cleanup succeeded, without confirming GDrive was also clean — a new job could claim the recycled code while old GDrive files still existed. Fixes: Added find_files_by_brand_code() fallback to GoogleDriveService that searches CDG/MP4/MP4-720p subfolders by brand code prefix when tracked file IDs aren't available. cleanup-distribution endpoint now: (a) uses tracked file IDs (fast path) or falls back to brand_code search, (b) only recycles the brand code after both Dropbox AND GDrive are confirmed clean. Added scripts/cleanup_test_job_gdrive.py remediation script. Added 14 new tests (94% coverage on gdrive_service.py).

  • CDG Packaging Fail-Fast Validation (2026-02-27): Fixed silent CDG generation failures where jobs completed with enable_cdg=True but no CDG ZIP was produced. Root cause: Jobs 5b6aba25 and 5161b069 had instrumental_selection: custom but before commit b671b83f, the orchestrator had no custom case — it fell through to "Backing" suffix. The CDG generator then hit a FileNotFoundError looking for the wrong filename, which was silently caught. Fix: (1) Orchestrator now validates CDG/TXT outputs after packaging stage and before encoding — if enable_cdg=True but no CDG ZIP was produced, the pipeline fails fast instead of silently continuing to expensive encoding, (2) Missing LRC file now raises when CDG/TXT is enabled instead of silently skipping, (3) Added scripts/regenerate_cdg.py remediation script for affected jobs. Removed "CDG requires additional style configuration" known issue (the nomad theme has complete CDG styles). Added 7 new tests for validation logic.

  • GCE Encoding Pipeline Resilience (2026-02-26): Made encoding pipeline resilient to Cloud Run deployment interruptions. A deployment killed a poller mid-encoding, leaving job a03ad39c permanently stuck at encoding status. Three fixes: (1) GCE worker /encode endpoint now idempotent — returns cached results for completed jobs, allows retry on failure, returns status for in-progress (previously hard 409 reject), (2) encoding_service.py handles cached/in_progress responses and falls back to status check on 409, (3) Health service detects jobs stuck in encoding >50 min. See LESSONS-LEARNED.md.

  • Private (Non-Published) Tracks (2026-02-24): Added is_private flag on jobs for private/non-published track production. Private jobs upload to Dropbox Tracks-NonPublished folder with NOMADNP brand prefix, skip YouTube and Google Drive entirely, and send completion emails without the YouTube section. Available as a checkbox on all job creation tabs and as an admin toggle (auto-deletes outputs when toggling a completed job to private). See archive/2026-02-24-private-tracks-plan.md.

  • GCE Encoding Worker Version Sorting Fix (2026-02-11): Fixed critical regression from PR #384 where alphabetical sorting caused encoder to use v0.99.9 instead of v0.116.1. Root cause: sorted(wheels)[-1] uses string comparison, not semantic versioning. String comparison puts "0.99.9" > "0.116.1" (because '9' > '1' in second position). Impact: Production encoder was running an extremely old version (v0.99.9 from months ago) instead of the latest (v0.116.1). Fix: Extract version from wheel filename using regex, sort by packaging.version.Version objects instead of strings. Added 10 comprehensive unit tests covering the exact regression case (0.99.9 vs 0.116.1), edge cases (empty list, invalid filenames), and complex version comparisons (major/minor/patch). Production now correctly selects the latest semantic version. See LESSONS-LEARNED.md.

  • GCE Encoding Worker Wheel Fix (2026-02-10): Fixed encoding worker crash caused by attempting to install karaoke_gen-current.whl, which is not a valid PEP 427 wheel filename (missing version and platform tags). Root cause: The ensure_latest_wheel() function downloaded all wheels including karaoke_gen-current.whl, then tried to install it. pip rejected it with "Invalid wheel filename (wrong number of parts)", causing encoding to fail and the service to restart, resulting in "Cannot connect" errors from Cloud Run. Fix: Filter out wheels containing '-current' before selecting which wheel to install. Only properly versioned wheels (e.g., karaoke_gen-0.116.0-py3-none-any.whl) are now considered. Added 5 comprehensive unit tests covering current wheel exclusion, empty results, version sorting, and path-agnostic filtering. Fixes job 749141f8 and prevents future encoding failures. See LESSONS-LEARNED.md.

  • Flacfetch Retry Logic & Audio Search Recovery (2026-02-10): Added comprehensive retry logic for flacfetch download failures to handle transient network issues and service restarts. Jobs now retry downloads up to 9 times over ~6 minutes with exponential backoff (10s → 20s → 40s → 60s capped), significantly reducing failures from temporary network hiccups. Key improvements: (1) Retry logic with selective error handling (retries on connection errors and 5xx, immediate fail on 4xx), (2) State persistence - download parameters (source_name, source_id, target_file, download_url) now saved BEFORE download attempt, enabling retry even if download fails, (3) Enhanced retry endpoint - /api/jobs/{id}/retry now handles audio search jobs by re-downloading using saved params, (4) User feedback - retry button now shows error toast with actual API error message instead of silent failures. Added 5 comprehensive retry tests covering network errors, 4xx/5xx responses, max retries, and exponential backoff timing. Job 749141f8 investigation revealed the root cause: no source info was saved when flacfetch calls failed, making retry impossible. See LESSONS-LEARNED.md for retry implementation patterns.

  • CDG Sync Fix for Countdown-Padded Songs (2026-02-05): Fixed CDG files being 3 seconds out of sync with audio when songs have countdown padding applied. Root cause: Video rendering adds 3s countdown padding to audio AND shifts LRC timestamps by +3s. CDG generation was using these countdown-shifted LRC timestamps but with instrumental audio (which has no countdown padding), causing lyrics to appear 3s late. Fix: CDG generator now receives lrc_has_countdown_padding flag; when True, it strips the countdown segment and shifts timestamps back before generating the CDG. Added 11 unit tests and updated orchestrator/packaging pipeline. See LESSONS-LEARNED.md.

  • Custom Instrumental Upload Fix (2026-02-26): Fixed critical bug where uploaded custom instrumentals were ignored during encoding — GCE encoding worker and orchestrator had no "custom" case, falling back to clean instrumental instead. Upload endpoint now converts all formats to FLAC for pipeline consistency. Three fixes: (1) GCE encoding worker find_file now searches for custom_instrumental*.flac when selection is "custom", (2) orchestrator config constructs correct (Instrumental Custom).flac path, (3) upload endpoint converts mp3/m4a/wav/etc. to FLAC before storing. Added 10 new regression tests for GCE resolution and orchestrator paths. See API.md.

  • Render Video GCE Routing (2026-03-31): Routed the render_video_worker (OutputGenerator/ffmpeg lyrics rendering) to the GCE encoding worker instead of running on Cloud Run. Previously, render_video always ran on Cloud Run and OOM'd after memory was right-sized to 2Gi. Now when USE_GCE_ENCODING=true, Cloud Run constructs GCS paths from job metadata and delegates to the GCE worker's new /render-video endpoint. Style assets (fonts, backgrounds) are cached on the GCE worker's disk between jobs to avoid repeated downloads of the same default theme. Local fallback preserved when GCE is disabled. See spec.

  • Store Processing Metadata Permanently (2026-03-10): Added processing_metadata field to the Job model — a top-level dict where each worker writes immutable provenance data during processing. Stores AudioShake task_id/asset_id (the original ask — these were previously lost after processing), worker timing, separation model names, ffprobe audio properties, correction stats, cache hit/miss data, video encoding details, and auth context at job creation. All metadata writes are non-fatal (wrapped in try/except). Includes update_processing_metadata helper on JobManager using Firestore dot-notation for atomic section updates. 23 tests covering model, helpers, all worker functions, and a SUMMARY_FIELD_PATHS regression guard. See archive/2026-03-10-store-job-processing-metadata.md and LESSONS-LEARNED.md.

  • Custom Instrumental Upload (2026-02-25): Added POST /api/jobs/{job_id}/upload-instrumental endpoint — was missing from backend, causing 404 in production when users tried to upload custom instrumental files during review. Validates uploaded audio duration matches original (±0.5s tolerance) using ffprobe on GCS signed URLs. Fixed frontend bug where "uploaded" selection type was sent to API instead of "custom" (the only value the backend accepts). Added 22 tests covering upload validation, duration checks, and frontend/backend contract. See API.md.

  • Unicode Filename Fix (2026-02-04): Fixed FFmpeg failures (exit code 183) when processing songs with non-ASCII characters in titles. Job 1eab1172 with Russian title "я куплю тебе дом" failed because Python's str.isalnum() returns True for ALL Unicode alphanumeric characters (Cyrillic, Arabic, Chinese, emoji, etc.), not just ASCII. These Unicode characters in temp subtitle file paths (temp_subtitles_zivert___я_куплю_тебе_дом_1770150285852.ass) caused FFmpeg to fail. Fix: Changed filename sanitization from c.isalnum() to c.isascii() and c.isalnum() in three locations: video.py:222 (generate_video), video.py:282 (generate_preview_video), and remote_cli.py:1970 (output folders). Added comprehensive test suite covering Cyrillic, Chinese, Arabic, emoji, and accented characters. See LESSONS-LEARNED.md.

  • Admin MFY Restart & Override Tools (2026-02-04): Added three new admin endpoints for better Made-For-You order management: (1) Regenerate Screens (POST /api/admin/jobs/{job_id}/regenerate-screens) - regenerates title/end screens with current artist/title metadata without full reprocessing; (2) Full Restart (POST /api/admin/jobs/{job_id}/restart) - unlike "Reset to Start", this actually triggers workers to run, with option to preserve audio stems for faster restart; (3) Override Audio Source (POST /api/admin/jobs/{job_id}/override-audio-source) - switch from YouTube URL to audio search mode for higher quality audio. Frontend admin toolbar updated with new buttons (Regen Screens, Full Restart, Audio Search) with confirmation dialogs. Use case: Fix MFY orders where customer provided wrong artist/title or low-quality YouTube URL. See API.md.

  • Made-For-You YouTube URL Download Fix (2026-01-28): Fixed bug where made-for-you orders with YouTube URLs failed with "Failed to download audio file". Root cause: The webhook handler triggered workers directly without downloading the audio first via YouTubeDownloadService. Workers require input_media_gcs_path to be set, which only happens after download completes. Fix: Handler now downloads YouTube audio to GCS before triggering workers, matching the pattern in file_upload.py and audio_search.py. Added 4 new contract tests with call-order tracking and AST import validation to prevent regression. See LESSONS-LEARNED.md.

  • Lyrics Corrections Now Applied to Video (2026-01-26): Fixed critical bug where user's lyrics corrections made during human review were being ignored. Videos were rendering with raw transcription instead of corrected lyrics. Root cause: The GET /api/review/{job_id}/correction-data endpoint was returning the original corrections.json instead of corrections_updated.json (where user edits are saved). The combined review flow would then overwrite the user's edits when completing the review. Fix: The endpoint now checks for corrections_updated.json first, matching the behavior of render_video_worker.py. Added 35 new tests (backend unit + integration + frontend E2E) to prevent regression. See LESSONS-LEARNED.md.

  • Instrumental Review UX Improvements (2026-01-26): Fixed three UX issues in the instrumental review flow: (1) Loading spinner now visible when switching audio tabs (shows on active tab with disabled state on others), (2) Removed large gap under waveform by fixing flexbox layout, (3) Success screen now appears in cloud mode with countdown before redirect (previously only showed in local CLI mode). Also fixed isLocalMode() detection to correctly distinguish cloud mode (hash-based routing like #/jobId/review) from local CLI mode (path-based routing like /local/review) on localhost, allowing E2E tests to run in cloud mode. Added 17 unit tests for local-mode.ts and 12 E2E regression tests for instrumental review UX. Backend fix: Changed audio_urls['backing'] to audio_urls['backing_vocals'] to match frontend expectations.

  • Dashboard Performance & UX Improvements (2026-01-25): Improved recent jobs list loading performance and usability. Changes: (1) Reduced default job limit from 20 to 5 for faster initial load (typical users only need to see recent jobs), (2) Added job limit dropdown (5/10/20/All options) for power users who want more history, (3) Implemented priority sorting - blocking jobs (awaiting user action) now appear first, followed by processing, completed, and failed jobs, (4) Made OutputLinks buttons responsive by hiding text labels on mobile (icons-only) to prevent horizontal overflow. Added 14 comprehensive unit tests for priority sorting logic. Frontend-only changes, no API modifications.

  • Combined Lyrics + Instrumental Review (2026-01-24): Merged the two-step human review process (lyrics review, then instrumental selection) into a single unified flow. Users now review lyrics AND select instrumental track in one session, submitting both with a single click. Benefits: Better UX (one interaction vs two), reduced drop-off, single email notification instead of two. Technical changes: Backing vocals analysis moved to screens_worker (runs before review), instrumental options included in /api/review/{job_id}/correction-data response, POST /api/review/{job_id}/complete now requires instrumental_selection field. The AWAITING_INSTRUMENTAL_SELECTION state is retained for finalise-only jobs but no longer used in the normal flow. See ARCHITECTURE.md and LESSONS-LEARNED.md.

  • Edit Segment Time Input UX Fix (2026-01-24): Fixed cursor jumping issue when typing multi-digit time values in the edit segment modal. Replaced controlled number inputs with debounced TimeInput component that maintains local state while focused and only syncs to parent after 300ms or on blur. This allows users to smoothly type values like "12.34" without the cursor jumping to the end on each keystroke. Added 15 comprehensive unit tests and E2E regression test to prevent future regressions. See LESSONS-LEARNED.md.

  • Cloud Run Jobs for Workers (2026-01-24): Migrated lyrics and audio workers from FastAPI BackgroundTasks to Cloud Run Jobs. This fixes instance termination issues where Cloud Run would shut down containers mid-processing (e.g., job c94cc9d6 was killed after AudioShake completed but before anchor sequence processing finished). Workers now have CLI entry points (python -m backend.workers.lyrics_worker --job-id abc123) and run as standalone batch containers. Cloud Run Jobs run to completion without HTTP lifecycle concerns, with up to 24-hour execution time. Infrastructure changes via Pulumi define lyrics-transcription-job and audio-separation-job. See ARCHITECTURE.md and LESSONS-LEARNED.md.

  • Encoding Worker Immutable Deployment (2026-01-22): Implemented robust deployment pattern for GCE encoding worker to eliminate version mismatch issues. Changes: (1) Fixed wheel path karaoke_gen-current.whl eliminates version sorting bugs, (2) Self-updating startup script downloaded from GCS on every service start (no Packer rebuild needed for logic changes), (3) Version manifest with strict CI verification (deployment fails on mismatch). Root cause: startup.sh was baked into Packer image, requiring image rebuild + VM recreation for fixes. New pattern allows CI to update deployment logic without infrastructure changes. See infrastructure/encoding-worker/README.md and LESSONS-LEARNED.md.

  • Theme Validation & Default Removal (2026-01-22): BREAKING CHANGE in v0.109.0 - Removed all default style fallback logic to ensure cloud jobs always use complete themes. Style getter functions now raise ValueError if themes are incomplete instead of merging with defaults. Added --list-themes and --validate-theme CLI flags. All cloud jobs guaranteed to use complete nomad theme with no silent fallbacks. See archive/2026-01-22-theme-validation-remove-defaults.md.

  • Admin Job Reprocess (2026-01-21, improved 2026-01-24): Added "Reprocess" button to admin job reset toolbar. Allows re-encoding and re-distributing a job with existing settings after deleting outputs. Resets job to instrumental_selected state and automatically triggers video worker. Clears video/encoding/distribution state while preserving instrumental selection and lyrics metadata. Use case: Fix distribution issues, re-upload to YouTube with different settings, or re-run encoding after infrastructure changes. Improvements (2026-01-24): API response now includes worker_triggered and worker_trigger_error fields for transparency. Added new /api/admin/jobs/{job_id}/trigger-worker endpoint for manual worker triggering when auto-trigger fails. Frontend shows detailed result dialog after reset with status change, cleared data, and worker trigger status. Added "Trigger" button in admin toolbar for manual worker triggering. See API.md.

  • Countdown Audio Sync Fix (2026-01-21, completed 2026-01-24): Fixed bug where songs starting within 3 seconds would have audio desync. PR #328 fixed the writer side: render_video_worker now updates lyrics_metadata.has_countdown_padding. PR #338 fixed the reader side: video_worker_orchestrator now reads the countdown state and passes it through to GCE encoding, which pads the instrumental audio to match. This was a "Fix Both Sides of Dual Code Paths" bug - see LESSONS-LEARNED.md. Added comprehensive test suite (contract tests, pipeline integration, orchestrator countdown tests) to prevent regression. See archive/2026-01-21-countdown-audio-sync-testing-strategy.md.

  • Anchor Sequences Fix (2026-01-21): Fixed anchor sequence calculation when auto-correction is disabled. Previously, disabling auto-correction (SKIP_CORRECTION=true) also stopped anchor/gap calculation, leaving reviewers without guidance on uncertain lyrics areas. Now anchor sequences are calculated independently, so reviewers always see highlighted matches against reference lyrics regardless of correction settings.

  • Auto-Correction Disabled (2026-01-20): Disabled all auto-correction during lyrics transcription (both agentic AI via Gemini and heuristic rule-based handlers). Raw transcription now goes directly to human review without automatic fixes. This change was made because auto-correction was creating more work for reviewers by introducing errors that needed manual correction. Anchor sequences are still calculated to help reviewers identify uncertain areas. The correction feature can be re-enabled by setting SKIP_CORRECTION=false environment variable. See backend/config.py for configuration.

  • Worker Coordination Fix (2026-01-20): Fixed bug where Cloud Run could shut down prematurely when one parallel worker completed while another was still running. Added WorkerRegistry to track active workers per job. Workers register at start and unregister in finally block. FastAPI lifespan shutdown handler now waits up to 10 minutes for active workers to complete before allowing container termination. Fixes job failures where lyrics worker was killed mid-processing after audio worker completed. See ARCHITECTURE.md and LESSONS-LEARNED.md.

  • Admin Delete Outputs (2026-01-10, UX improved 2026-01-24): Added admin button to delete all distributed outputs (YouTube, Dropbox, Google Drive) for a job while preserving the job record. Use case: fix quality issues by deleting outputs, resetting to awaiting_review, correcting lyrics, and re-generating. The brand code is freed for reuse when Dropbox folder is deleted. New endpoint: POST /api/admin/jobs/{job_id}/delete-outputs. UI: Delete Outputs button in admin job detail page, "Outputs Deleted" badge when flag is set. UX improvements (2026-01-24): Download buttons (4K, 720p, CDG, TXT, etc.) are now hidden when outputs have been deleted, preventing broken links. Delete confirmation now shows a detailed result dialog with per-service status (YouTube video ID deleted, Dropbox folder path, Google Drive files) instead of a simple toast. See API.md.

  • Rate Limiting & Abuse Prevention (2026-01-09): Added per-user rate limiting (5 jobs/day default), system-wide YouTube upload limits (10/day), and beta enrollment abuse prevention. Features: email normalization (Gmail alias detection), disposable domain blocklist (130+ domains), blocked email/IP lists, IP-based enrollment rate limiting (1 per 24h), and admin UI for managing blocklists and user overrides at /admin/rate-limits. Admins can grant users bypass permissions or custom limits. All limits configurable via environment variables. See API.md.

  • Fast Preview Video Option (2026-01-10): Added toggle in lyrics review UI to render preview videos with black background (~10s) instead of theme background image (~30-60s). Default is now black background for faster iteration during review. Users can enable theme background via checkbox when they want to verify the final look. See API.md.

  • Frontend Consolidation (2026-01-09): Consolidated three separate frontends into a single Next.js application. Previously: (1) Main frontend (Next.js + Tailwind), (2) Lyrics Review (React + Vite + MUI, ~14k lines), and (3) Instrumental Review (vanilla HTML/JS, ~1.7k lines) were deployed separately. Now all UIs are unified in the main frontend at /app/jobs#/{id}/review and /app/jobs#/{id}/instrumental. Benefits: single deployment to Cloudflare Pages, no external redirects during user workflows, unified design system (Radix/Tailwind), simpler authentication via job ownership model. See archive/2026-01-09-frontend-consolidation-plan.md and tasks/.

  • Admin Login Token for Made-For-You Orders (2026-01-10): Added one-click admin login via email. When a made-for-you order is placed, the admin notification email now includes a direct link with an embedded login token (?admin_token=TOKEN). Clicking this link auto-authenticates the admin and opens the job directly - no manual login required. Tokens expire after 24 hours. Also added: BCC to done@nomadkaraoke.com on all job completion emails, fixed timezone display (EST→ET for DST correctness). See API.md.

  • Made-For-You Audio Selection Flow (2026-01-08, fixed 2026-01-09): Refactored made-for-you order processing to pause at audio selection like regular search jobs. New flow: customer places order → admin receives notification email → admin selects audio source in UI → job processes under admin ownership → on completion, ownership transfers to customer → customer receives delivery email. Added model fields (made_for_you, customer_email, customer_notes), professional email templates for order confirmation and admin notification, intermediate email suppression (no lyrics/instrumental reminders to customer during processing), and ownership transfer on job completion. Distribution defaults (Dropbox, YouTube, GDrive, brand prefix) are now properly applied from server settings. Bug fix (2026-01-09): Fixed webhook handler that was not setting made_for_you=True, not using admin ownership, not storing customer_email/customer_notes, and not pausing at audio selection. Added 17 comprehensive unit tests for the webhook handler. See API.md and LESSONS-LEARNED.md.

  • White-Label B2B Portal Infrastructure (2026-01-08): Added multi-tenant support for white-label karaoke portals. Vocal Star is the first tenant at vocalstar.nomadkaraoke.com. Features: subdomain-based tenant detection (frontend and backend), GCS-backed tenant config storage, per-tenant feature flags (audio search, file upload, YouTube, theme selection, etc.), dynamic branding (logo, colors, site title via CSS variables), locked themes per tenant, email domain restrictions for auth, and tenant-scoped jobs/users in Firestore. Backend: TenantMiddleware extracts tenant from X-Tenant-ID header, query param (dev only), or Host subdomain; TenantService loads config from GCS with caching. Frontend: Zustand store for tenant state, TenantProvider for initialization, TenantLogo component. Setup script: scripts/setup-vocalstar-tenant.py. See ARCHITECTURE.md and API.md.

  • Encoding Worker Infrastructure Upgrade (2026-01-08): Upgraded GCE encoding worker from c4-standard-8 (Intel) to c4d-highcpu-32 (AMD EPYC 9B45 Turin). Benchmarking showed 4.92x faster encoding (135s vs 666s total). Key improvements: With Vocals 4K rendering 6x faster (54s vs 324s), 720p downscale 4.6x faster (11s vs 50s). Reduces total encoding time from ~11 min to ~2.3 min per job. See archive/2026-01-08-performance-investigation.md for full benchmark methodology and results.

  • Theme Enforcement & Encoding Resilience (2026-01-08): Added defense-in-depth theme enforcement - jobs without theme_id are now rejected at creation (JobManager) with a safety net at processing time (screens_worker). Fixed made-for-you webhook to apply default theme. Added retry logic with exponential backoff (3 retries, 2s→4s→8s) to GCE encoding service for transient connection failures during worker restarts. See LESSONS-LEARNED.md.

  • Comprehensive Performance Optimization (2026-01-08): Reduced lyrics processing from 16+ minutes to ~5 minutes through multiple optimizations. Key changes: (1) NLTK cmudict preloading at startup saves 100-150s, (2) Langfuse callback handler preloading saves 200s, (3) Model warmup in AgenticCorrector.from_model() prevents parallel initialization race, (4) Parallel anchor sequence search (4 workers) reduces n-gram processing from 38s to ~10s. Added /health/preload-status endpoint for deployment verification. PR #236. See archive/2026-01-08-performance-investigation.md and LESSONS-LEARNED.md.

  • UI Simplification & Brand Accessibility (2026-01-08): Simplified the "Create Karaoke Video" form by hiding advanced options (theme settings now admin-only, "Display As" behind toggle). Improved audio search dialog UX by hiding technical tracker names and showing user-friendly availability badges. Fixed brand pink color contrast for accessibility - swapped default/hover values so white text on pink buttons meets WCAG contrast ratio. See BRAND-STYLE-GUIDE.md v1.2.

  • SpaCy Preloading (2026-01-08): Implemented SpaCy model preloading at container startup to eliminate 60+ second delay during agentic correction. The en_core_web_sm model is now loaded during FastAPI lifespan startup, and PhraseAnalyzer/SyllablesMatchHandler reuse the preloaded model. Added timing logs to verify performance. See archive/2026-01-08-spacy-preload-plan.md.

  • Thread-Safe LangChain Model Initialization (2026-01-08): Fixed race condition in LangChainBridge where parallel threads could all try to initialize the AI model simultaneously, causing 6+ minute delays. Added double-checked locking with threading.Lock(). PR #232. See LESSONS-LEARNED.md.

  • Brand Consistency & Style Guide (2026-01-07): Implemented unified brand theming across all Nomad Karaoke frontends. Primary color changed from blue to brand pink (#ff7acc). Added comprehensive BRAND-STYLE-GUIDE.md for human and LLM reference. Includes: CSS variables for light/dark mode, MUI theme updates for lyrics transcriber, email template branding, and Playwright visual tests. See BRAND-STYLE-GUIDE.md.

  • Audio Search Display Override (2026-01-06): Added optional "Display As" fields to audio search, allowing users to search for audio by one artist/title (e.g., "Jeremy Kushnier") but display a different artist/title on title screens and filenames (e.g., "Footloose (Broadway Cast)"). Useful for Broadway casts, covers, remixes where tracker metadata differs from desired display. See API.md.

  • Infrastructure Modularization (2026-01-06): Refactored infrastructure/__main__.py from 2,602 lines to 339 lines (87% reduction). Split into organized modules: modules/ for core GCP resources, compute/ for VMs and startup scripts, config.py for shared constants. Extracted embedded encoding worker Python app to backend/services/gce_encoding/. See infrastructure/README.md for new structure and infrastructure/docs/PHASE5-PACKER-IMAGE-PLAN.md for planned Packer image optimization.

  • Email Notification System (2026-01-06): Added automated email notifications for job completion and user action reminders. Features: GCS-backed HTML email templates with fallback defaults, Postmark with CC support, auto-completion emails on job finish, idle reminder emails via Cloud Tasks (5-min delay for blocking states), admin UI buttons to copy message or send email manually. Endpoints: GET /api/admin/jobs/{id}/completion-message, POST /api/admin/jobs/{id}/send-completion-email. Feature flag ENABLE_AUTO_EMAILS (default: false). See API.md and ARCHITECTURE.md.

  • GCE Preview Encoding Unified with LocalPreviewEncodingService (2026-01-06): GCE preview encoding now uses the same LocalPreviewEncodingService from the installed wheel, eliminating 100 lines of duplicated FFmpeg logic. Preview videos are now identical across local CLI, Cloud Run, and GCE worker environments. Benefits include: consistent encoding settings (480x270, 24fps, crf 28), proper FFmpeg filter path escaping for special characters, hardware acceleration (NVENC) when available, and custom font support. See LESSONS-LEARNED.md.

  • GCE Encoding Worker Python 3.13 (2026-01-06): Upgraded GCE encoding worker from Debian's Python 3.11 to Python 3.13 built from source. Uses dedicated virtual environment at /opt/encoding-worker/venv, removing the need for --break-system-packages. Aligns the encoding worker with CI/Cloud Run Python version.

  • GCE Encoding Unified with LocalEncodingService (2026-01-06): GCE encoding worker now uses the same LocalEncodingService as the local CLI, eliminating duplicated encoding logic. Output files now have proper names (Artist - Title (Final Karaoke Lossless 4k).mp4 instead of output_4k_lossless.mp4) and include title/end screen concatenation. Wheel deployed to GCS; worker installs at job start for hot updates without VM restart. Removed all fallback logic - single code path for consistent output across CLI, Cloud Run, and GCE. See LESSONS-LEARNED.md.

  • Cloud Distribution Matches Local CLI (2026-01-06): Fixed cloud distribution (Dropbox uploads) to match local CLI output structure. The lyrics/ folder now includes: proper filenames ((Lyrics Corrections).json instead of (Karaoke).json), all reference lyrics files ((Lyrics Genius).txt, (Lyrics Spotify).txt, etc.), (With Vocals).mkv, and previews/ subfolder with preview ASS files. See LESSONS-LEARNED.md.

  • Agentic Correction Performance (2026-01-05): Optimized agentic AI correction from ~5 minutes to ~55 seconds for 20 gaps (~5-6x speedup). Fixed anti-pattern of creating new model instance per gap (caused repeated 2s warm-up overhead). Now creates model once and processes gaps in parallel using ThreadPoolExecutor (default 5 workers). Configure via AGENTIC_MAX_PARALLEL_GAPS env var. See LESSONS-LEARNED.md.

  • Worker Logs Subcollection (2026-01-04): Moved worker_logs from embedded array in job documents to Firestore subcollection (jobs/{job_id}/logs). Fixes job failures when logs exceed 1MB (job 501258e1 had 1.26MB of logs). New logs stored with 30-day TTL via Firestore TTL policy. Feature flag USE_LOG_SUBCOLLECTION=true (default). See LESSONS-LEARNED.md.

  • Admin Job Detail Enhancements (2026-01-09): Enhanced admin job detail page with comprehensive job management features. Added: file downloads with signed GCS URLs grouped by category (stems, lyrics, outputs), inline editing of job fields (artist, title, theme, etc.), job reset actions to re-process from specific workflow checkpoints (pending, audio selection, review, instrumental selection), visual timeline with color-coded stages. New endpoints: GET /api/admin/jobs/{job_id}/files, PATCH /api/admin/jobs/{job_id}, POST /api/admin/jobs/{job_id}/reset. See API.md.

  • GCE Encoding Response Fixes (2026-01-04): Fixed multiple response format mismatches with GCE encoding worker. The worker returns output_files as a list of paths, not a dict with format keys - added conversion logic. Also added defensive type checking for status responses that could be lists. See LESSONS-LEARNED.md.

  • Video Worker Orchestrator (2026-01-04): Major refactor to unify video generation pipeline. Created VideoWorkerOrchestrator that coordinates all stages (packaging, encoding, distribution, notifications) regardless of encoding backend (GCE or local). Fixes issue where GCE encoding path bypassed YouTube upload, Discord notifications, and CDG/TXT packaging. Feature flag USE_NEW_ORCHESTRATOR (default: true) enables rollback. 139 new tests across 6 new service modules. See ARCHITECTURE.md and archive/2026-01-04-video-worker-orchestrator-refactor.md.

  • Full Unicode Font Support (2026-01-03): Fixed rendering of musical symbols (♪) and added comprehensive international font support to Docker base image. Installed Noto fonts covering Latin, CJK (Chinese/Japanese/Korean), Arabic, Hebrew, Thai, and other scripts. Changed default karaoke font from Arial to Noto Sans. See LESSONS-LEARNED.md.

  • Flacfetch Cache API Integration (2026-01-03): Enhanced audio search cache management to also clear flacfetch's GCS cache. When admins clear a job's cache, both Firestore (job state) and flacfetch (GCS search results) are cleared simultaneously. Added "Clear All Cache" button and cache stats display to admin UI. New endpoints: DELETE /api/admin/cache, GET /api/admin/cache/stats. See API.md.

  • Audio Search Cache Management (2026-01-03): Added admin UI at /admin/searches to view and manage cached audio search results. Admins can clear stale caches (e.g., when flacfetch is updated with new providers) and allow users to re-search. Fixes issue where jobs showed only YouTube results when lossless sources should be available. See API.md and LESSONS-LEARNED.md.

  • User Database Separation (2026-01-03): Migrated karaoke-gen users from shared users collection to dedicated gen_users collection. This separates karaoke-gen user data from karaoke-decide user data (both apps share the same GCP project/Firestore instance). Migration script in scripts/migrate_users_to_gen_users.py. See ARCHITECTURE.md for collection details.

  • Admin Dashboard (2026-01-03): Added comprehensive admin dashboard at /admin with: stats overview (users, jobs, credits), user management (search, sort, add credits, toggle role, enable/disable), job browser (filter by status/user, view details, delete), and beta program monitoring. Query-param based routing for static export compatibility. See API.md for backend endpoints.

  • GCE Encoding Worker (2026-01-03): Added optional high-performance encoding on dedicated GCE VM (C4-standard-8 with Intel Granite Rapids 3.9 GHz). Uses USE_GCE_ENCODING=true env var. Falls back to Cloud Run if unavailable. See ARCHITECTURE.md.

  • Runner Disk Space Auto-Cleanup (2026-01-03): Fixed CI failures caused by self-hosted runners filling up with Docker images. Changed cleanup from daily/7-day-old to hourly/threshold-based (70%). Added pre-job disk checks to all self-hosted CI jobs. See LESSONS-LEARNED.md

  • E2E Happy Path Test (2026-01-02): Added comprehensive E2E test that validates the complete karaoke generation journey using ONLY browser UI interactions (no API shortcuts). Test covers all 12 steps from landing page through cleanup. Runs daily via GitHub Actions and takes ~38 minutes. See E2E-HAPPY-PATH-TEST-SPEC.md

  • LyricsTranscriber Cache Persistence (2026-01-02): Added GCS-backed cache for LyricsTranscriber to avoid redundant API calls. AudioShake transcription and lyrics provider responses (Genius, Spotify, LRCLib, Musixmatch) are now cached to gs://karaoke-gen-storage/lyrics-transcriber-cache/. Cache is synced before/after each job, persisting across Cloud Run instances. Significantly reduces API costs and speeds up repeated processing of same songs.

  • Worker Timeout Fixes (2026-01-01): Fixed 3 timeout issues blocking job completion: (1) Lyrics transcription timeout increased to 20 min (PR #153), (2) Cloud Tasks dispatch_deadline added for audio worker - default 10 min was killing Modal API calls (PR #154), (3) Enabled Cloud Run Jobs for video encoding - 1-hour timeout vs 30-min Cloud Run service limit (PR #155). Full E2E pipeline now verified working. See archive/2026-01-01-worker-timeout-fixes.md

  • Agentic Correction Timeout (2025-12-31): Added 3-minute configurable timeout for agentic AI lyrics correction. Prevents stuck jobs when songs have many gaps (74+ gaps could take 30+ minutes). On timeout, skips correction and proceeds to human review with raw transcription. See archive/2025-12-31-agentic-timeout-implementation.md

  • Job Failure Fixes (2025-12-31): Fixed 4 critical issues causing job failures: (1) user_email not set on jobs - users couldn't see their jobs, (2) YouTube URL race condition - lyrics worker started before audio downloaded, (3) Audio search cache not persisting across Cloud Run instances, (4) Jobs stuck in downloading state - added 10min transcription timeout. See archive/2024-12-31-job-failure-investigation.md

  • Vertex AI Auth Fix (2025-12-31): Fixed ChatGoogleGenerativeAI to use Vertex AI backend with service account auth on Cloud Run. PR #145 broke cloud deployments by requiring GOOGLE_API_KEY. Solution: pass project parameter to trigger Vertex AI backend with ADC. See archive/2025-12-31-vertex-ai-auth-fix.md

  • LangChain Provider Migration (2025-12-30): Migrated from langchain-google-vertexai (gRPC) to langchain-google-genai (REST) to fix silent hangs during model initialization. Added initialization and warm-up timeouts using ThreadPoolExecutor for fail-fast behavior. See archive/2025-12-30-langchain-vertexai-to-genai-migration.md

  • LangFuse Prompt Management (2025-12-30): Replaced hardcoded LLM prompts with LangFuse-managed prompts for dynamic iteration without redeploying. Added migration script, LangFuse Vertex AI service account for GCP credits. See archive/2025-12-30-langfuse-prompt-management.md

  • Gemini 3 Flash Agentic Correction Fix (2025-12-30): Fixed agentic correction hanging due to Gemini 3 requiring global location instead of regional endpoints, and multimodal response format handling. Added local test script for faster iteration. See archive/2025-12-30-gemini3-agentic-correction-fix.md

  • Cold Start UX (2025-12-30): Frontend now shows friendly "Warming up the generator..." messages when backend is slow to respond (cold start after deployment or inactivity). Progressive messages appear after 5s and 15s to reassure users during 30s+ cold starts.

  • Cloud Output Structure Fix (2025-12-30): Fixed remote CLI (karaoke-gen-remote) to produce identical output as local CLI: YouTube uploads now work, Dropbox includes stems/ and lyrics/ subfolders, instrumental filenames use actual model names. See archive/2025-12-30-cloud-output-structure-fix.md

  • Auth Session Persistence Fix (2025-12-30): Fixed two bugs causing 401 errors despite valid sessions: timezone-aware datetime comparisons in backend, and stale token caching in frontend. Firestore indexes now tracked in Pulumi. See archive/2025-12-30-auth-session-persistence-fix.md

  • E2E Test Suite Consolidated (2025-12-29): Tests reorganized into production/ (real API) and regression/ (mocked API, CI-safe). See archive/2025-12-29-e2e-test-consolidation-plan.md

Doc Purpose
PRODUCT-VISION.md Product goals, audience, business model
ARCHITECTURE.md System design, data flow, integrations
DEVELOPMENT.md Local setup, testing, deployment
TESTING.md Testing standards, CI requirements, Playwright
API.md Backend endpoint reference
REQUESTS-BOARD.md Public voting board (requests.nomadkaraoke.com) — what it is, UX, how it works; Phase 2 handoff linked
BRAND-STYLE-GUIDE.md Brand colors, typography, UI patterns
LESSONS-LEARNED.md Key insights for future agents
STRIPE-SETUP.md Payment integration setup
MOBILE-UX-BEST-PRACTICES.md Mobile/responsive design guidelines
archive/ Historical documentation

Workflow Summary

Upload Audio → Parallel Processing → Human Review → Video Generation → Download
                   │                      │
        (Cloud Run GPU + AudioShake)  (LyricsTranscriber UI)

For AI Agents

Start with CLAUDE.md in the repo root for essential rules, then reference these docs as needed. Keep this README current when project state changes.