Files
karaoke-gen/docs
Andrew Beveridge ☄️andClaude Opus 4.8 6800c14e57 admin: make made_for_you editable on jobs (stale-review exemption) + KaraokeHunt outreach tooling (#1004)
## Summary
- Add `made_for_you` to `EDITABLE_JOB_FIELDS` (with a boolean guard) so
admin batch flows can exempt jobs from the 48h stale-review auto-expiry
(`stale_review_processor` skips MFY jobs). Needed for the KaraokeHunt
outreach batch: ~90 jobs held at review under a holding account until
coordinated emails go out.
- KaraokeHunt outreach tooling under `scripts/karaokehunt_outreach/`
(phase1b/1c analysis, phase2 executor, batch generator, torrent review)
— standalone scripts, no runtime impact.
- Session handoff docs in `docs/archive/`.

## Testing
- `backend/tests/test_admin_job_update.py`: new
`TestUpdateJobMadeForYou` (editable + non-bool rejected) — 20/20 pass.
- Full backend suite: 4417 passed (unit) + emulator suite green via
`make test-e2e`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Administrators can mark jobs as “made for you,” keeping them available
for review indefinitely instead of automatically expiring after 48
hours.
- Invalid values for this setting are rejected with a clear validation
error.

- **Documentation**
- Added operational documentation for the KaraokeHunt outreach and
migration workflow, including request review, song matching, account
credits, and generation stages.

- **Chores**
  - Updated the application version to 0.225.4.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-09-15 06:12:06 +00:00
..

Karaoke-Gen Documentation

Status: Production-ready. First full E2E flow completed 2025-12-28.

What Works

  • Web App: https://gen.nomadkaraoke.com
    • / - Landing page with pricing
    • /app - Main app (upload audio, review lyrics, download videos)
      • Bulk Mode (Jun 2026): submit up to 100 jobs at once by text or by album (MusicBrainz lookup + KaraokeNerds availability); auto-picks confident lossless audio, parks the rest in a review queue. See docs/API.md § Bulk Mode.
    • /admin - Admin dashboard (user management, job monitoring)
  • Backend API: https://api.nomadkaraoke.com - FastAPI on Cloud Run
  • CLI Tools: karaoke-gen (local), karaoke-gen-remote (cloud)

Current State

Component Status
Audio upload & separation Working
Lyrics transcription Working
Auto-correction (agentic + heuristic) Disabled (raw transcription goes to human review)
AI auto-correct suggestions (opt-in) Working ("AI Suggest" button in lyrics review — whole-song LLM call, per-suggestion accept/reject)
Combined review (lyrics + instrumental) Working
Auto-approval (fully-auto review) Live (default review_mode="auto"): when the scorer is fully confident (synced-reference lyrics with all gaps AI-covered + no audible backing vocals), the review screens are skipped and the job renders straight through; every other job falls back to human review. Verdict on every job in processing_metadata.auto_approval. Kill switch AUTO_APPROVAL_ENFORCE_ENABLED=false; per-job review_mode="always_review" (admin-editable). See docs/archive/2026-08-25-full-auto-review-design.md
Review replay tool (admin) Working (?replay=1 read-only replay of completed jobs — see docs/REPLAY.md)
Preview video generation Working
Multi-format encoding Working
Token-based auth Working
Magic link auth Working
Payment flow (Stripe) Working
Feedback-for-credits Working (2+ jobs → 1 free credit)
Admin dashboard Working
Abuse prevention & YouTube quota Working (blocklists, IP/fingerprint rate limiting, credits on verification, quota-aware YouTube uploads)
CI/CD self-hosted runner Working (GCP)
E2E happy path test Working (~20-25 min full pipeline)
White-label B2B portals Working (Vocal Star, Singa)
Private (non-published) tracks Working
Custom video styling (private jobs) Working
Song suggestions & fuzzy correction Working (catalog lookup on Step 2 + "Did you mean?" typo correction)
Community version detection Working (via karaokenerds.com)
Review session backup/restore Working (auto-save + restore UI)
Review reminder & auto-expiry Working (24h reminder, 48h cancel + refund)
i18n / Multilingual (frontend) Working (next-intl, en/es/de, 1062 strings, [locale] routing)
Referral system Working (referral links with configurable discounts, 20% cash kickback via Stripe Connect, auto-payout at $20)
Error monitor New (Cloud Run Job every 15 min — log collection, normalization, LLM dedup/grouping, Discord alerts, auto-resolution, daily digest)
Edge security (Cloudflare) Working (api.nomadkaraoke.com proxied — WAF + rate limit + origin lock enforced; 2026-07-20)

Known Issues

(No known issues)

Pending Work

(No pending work items)

Recent Changes

  • Admin Dashboard Stats 18s → ~1s (2026-08-23): GET /api/admin/stats/overview?exclude_test=true streamed every user and job document in full to filter test data in Python — ~110 MB across 658 jobs (job docs carry state_data/timeline payloads), ~18s per dashboard load. Worse, the sync streaming ran inside an async route, blocking the event loop so every other request on the instance (/me, /config, payments summary) stalled behind it. Fix: Firestore .select() field masks on all three streams (jobs need only user_email/created_at/status; measured 15.0s → 1.1s for the jobs stream) and the whole computation now runs via asyncio.to_thread() so it can't stall other requests. Unit tests assert the field masks stay in place. Same sync-over-async trap as the Feb 2026 audio-search fix — see LESSONS-LEARNED.md.

  • Admin UI/UX Overhaul (2026-08-22): Made the admin pages usable on mobile and fixed the long-standing overflow problems on the job detail page (v0.197.0). Long status timelines (retried/edited jobs) now collapse to the last 6 stages behind a "+N earlier (M failed)" expander; the Source field no longer bleeds across the info grid (CSS grid min-w-0); the Audio Source Details modal got wrap-safe stacked rows plus an "Open in YouTube/Spotify" link built from provider + source ID; Configuration fields are click-to-edit with always-visible pencils. The jobs list search box jumps straight to a job when given a job ID, and the status filter is URL-synced (/admin/jobs?status=failed), which the dashboard's now-clickable stat cards and status-breakdown rows link into. Also fixed a real crash: the "Trigger Now" button in the reset-result dialog referenced a deleted handler. Users list/detail, breadcrumbs, and the admin header are all responsive down to phone widths. 4 new e2e tests.

  • Portrait (9:16) Karaoke Video Output (2026-07-13): When enabled (PORTRAIT_RENDER_ENABLED, default on), every finalised job also gets a portrait 1080×1920 karaoke video for phone viewing / social sharing, distributed to GCS + the Dropbox brand-code folder only (never Drive/YouTube). It is a genuine re-render at portrait resolution with re-wrapped lyrics — not a transform of the finished landscape video (letterbox = tiny lyrics, crop = cuts the ends off every line; proven in the design experiments). The new karaoke_gen/portrait/ module reuses the same SubtitlesGenerator engine + corrected-lyrics data, so word-by-word timing is identical; it adds an orphan-merging wrapper, a branded portrait background (wordmark/header + song/artist + footer), and title/end cards. Rendered on the GCE encoder inside run_encoding (best-effort + PORTRAIT_RENDER_ENABLED flag; a failure never fails the job) and carried to Dropbox via the existing finals/ upload path. ~15–30s + ~10 MB per job. See archive/2026-06-14-portrait-video-design.md and archive/2026-07-13-portrait-video-plan.md.

  • Cloudflare Edge Security for the Backend API (2026-07-20): api.nomadkaraoke.com is now proxied through Cloudflare (WAF + rate limiting + origin lock), after a vuln-scanner hammered the previously directly-exposed Cloud Run origin. WAF blocks exploit/secret-exposure paths (contains operator — regex needs a Business plan); per-IP rate limiting excludes /api/internal/*; a Transform Rule injects a secret X-Edge-Auth header that the backend's EdgeAuthMiddleware (EDGE_AUTH_MODE=enforce) requires, blocking direct-to-*.run.app bypass (health/root exempt). Key finding: the Cloud Run managed cert can't provision behind Cloudflare, but zone SSL mode "full" (not strict) doesn't validate the origin cert, so it works and is renewal-safe — never switch the zone to "full (strict)". Managed as IaC in infrastructure/modules/edge_security.py. Also retuned the "High Error Rate" alert to 5xx-only (scanner 404s no longer page). See ARCHITECTURE.md § Edge Security, LESSONS-LEARNED.md, and archive/2026-07-20-edge-security-cutover-runbook.md.

  • AI Auto-Correct Suggestions (2026-06-11): Opt-in AI correction suggestions in the lyrics review UI — the third attempt at lyrics auto-correction, redesigned around the failures that led to disabling it in PR #321 (Jan 2026). One whole-song LLM call (default Gemini 3.1 Pro via Vertex, AUTO_CORRECT_MODEL env) compares the working transcription against reference lyrics and returns word-id-keyed suggestions; nothing is applied until the reviewer accepts each one (or accept-all) in a dedicated panel with per-row undo. New POST /api/review/{job_id}/auto-correct endpoint + backend/services/auto_correct/; frontend useAutoCorrect hook, AutoCorrectModal (knobs: adlib removal, insertions, confidence filter), AutoCorrectPanel. Accept/reject decisions land in the EditLog (ai_suggestion_* ops) to measure real accept rates. 5-model offline eval over 30 real jobs with human ground truth picked the default (Fable 5 ≈ Gemini 3.1 Pro ≫ GPT-5.2, which made lyrics worse). See archive/2026-06-10-lyrics-auto-correction-reeval-plan.md.

  • Syllable-Aware Custom Lyrics (2026-05-02): Enhanced the Custom Lyrics LLM mode with syllable-count validation and a per-line repair loop. Backend computes per-line target/candidate syllable counts via the existing SyllablesMatchHandler engine and re-prompts Gemini up to 3 times until lines fit budget (Δ ≤ 2 default, configurable via Verbatim/Loose/Balanced/Tight/Strict strictness). New GenerationSettings (allow_reword, allow_omit, fixed_line_count, strictness) exposed in the modal as a collapsible settings panel. New CustomLyricsPreview component shows per-line syllable annotations (target / actual / Δ) with severity badges and stop-reason banners. Variable-line-count mode (when fixed_line_count=false) re-times segments proportionally. Eval harness with disk-backed Gemini cache, per-line + per-fixture + corpus scorers, and 4 fixtures (1 from production, 3 handcrafted public-domain). v1-ships criteria validation (Year 5 ≥75% pass@2, macro corpus ≥70% pass@2) deferred to post-merge. See archive/2026-05-02-custom-lyrics-syllable-aware-design.md and archive/2026-05-02-custom-lyrics-syllable-aware-plan.md.

  • Custom Lyrics LLM Mode (2026-04-28): New AI-powered lyrics-replacement mode in the lyrics review modal. Users (or admins editing client jobs) can paste custom lyrics or instructions — or upload a document — and an LLM returns lyrics that match the original line count exactly, ready to drop into the existing replace-segments flow. Includes backend CustomLyricsService with file extraction, prompt construction, and line-count validation; new POST /api/review/{job_id}/custom-lyrics/generate endpoint; preview UI with regenerate/save controls; manual-sync toast guidance after save. See archive/2026-04-28-llm-custom-lyrics-design.md and archive/2026-04-28-llm-custom-lyrics-plan.md.

  • GCE Encoding Cold-Start Resilience (2026-04-27): Fixed renders failing when the encoding worker VM was cold (TERMINATED). Job 2c577535 hit this on 2026-04-24: the existing 90s connection retry window (sized for systemctl restart during deploys) was too short for a 2-5 min GCE cold boot. Fix: new EncodingWorkerManager.wait_for_worker_ready() two-phase poll (VM status → /health 200) is now awaited when ensure_primary_running() reports started=True (VM was actually TERMINATED). Deploy-restart fast path unchanged. Also: _format_exception() helper so empty aiohttp.ClientConnectorError messages still produce diagnostic logs (the original incident logged "GCE worker connection failed after 8 attempts: " with no detail). 17 new tests including a regression integration test. See archive/2026-04-27-gce-encoding-cold-start-design.md, LESSONS-LEARNED.md, and TROUBLESHOOTING.md.

  • Anti-Abuse: Credits on Verification + Rate Limiting (2026-03-21): Three-layer defense against free credit abuse. (1) Welcome credits now granted on first magic link verification, not account creation — unverified accounts get 0 credits. (2) Per-IP signup rate limiting: max 2 new accounts per IP per 24 hours, silent reject. (3) Device fingerprinting (FingerprintJS open source): browser fingerprint sent with magic link request, combined with IP for rate limiting. Either signal hitting 2/day triggers block. Returning users are never rate limited. Graceful degradation if fingerprinting fails. Frontend verify page shows "You received N free credits" on first login. See archive/2026-03-21-anti-abuse-credits.plan.md.

  • Review Reminder & Auto-Expiry (2026-03-19): Jobs stuck in review (awaiting_review / in_review) now get automatic cleanup. At 24 hours, a gentle reminder email is sent with a review link, help offer, and warning. At 48 hours, the job is auto-cancelled and the user's credit is refunded. Expiry notification email confirms the refund and current balance. Excludes made-for-you and tenant jobs. Powered by hourly Cloud Scheduler cron (stale-review-hourly). New endpoint: POST /api/internal/process-stale-reviews. Also fixed make test hanging on frontend Playwright E2E tests by splitting into test-frontend (Jest unit only) and test-frontend-e2e (Playwright, run manually). See archive/2026-03-19-review-reminder-autoexpiry.plan.md.

  • Step 2 Song Suggestions & Fuzzy Correction (2026-03-08): Replaced the Step 1 autocomplete with a simpler approach: plain text inputs on Step 1, then catalog lookup + community version check fire in parallel with the audio search on Step 2. Results appear as a "Song found in our database" panel (exact match) or correction suggestion panel while the user waits. Added fuzzy "Did you mean?" correction — when audio search returns poor results (Tier 3) and catalog has no exact title match, fetches the artist's discography and fuzzy-matches against the user's title using LCS similarity. Shows an amber banner with one-click correction + expandable alternatives list. Handles garbled artist fields via title-based fallback with artist tiebreaker. Visibility step reworked with action buttons at top for one-click select+continue. See archive/2026-03-08-step2-song-suggestions.plan.md and archive/2026-03-08-fuzzy-did-you-mean.plan.md.

  • Review Session Backup/Restore (2026-03-08): Server-side auto-save of lyrics review sessions replaces fragile localStorage-only mechanism. Sessions auto-save every 60s (if dirty), on preview video generation, and on page unload. Deduplication via SHA-256 hash prevents redundant saves. Correction data stored in GCS (can exceed Firestore's 1MB doc limit); metadata in Firestore subcollection (jobs/{jobId}/review_sessions/{sessionId}). Restore UI shows split-pane session browser with edit diff preview. Cross-job session search allows reusing edits across different jobs (with duration mismatch warning). On load, automatically prompts to restore if saved sessions exist. See archive/2026-03-07-lyrics-review-sessions-plan.md.

  • Job Observability Improvements (2026-03-07): Enhanced timeline events with a metadata field to capture structured output details (brand codes, YouTube video IDs, Dropbox paths, GDrive file IDs) at key moments — job completion, edit initiation, and admin output deletion. Added log_to_job() helper for API endpoints to write to Firestore job log subcollections (previously only workers logged there). Edit and admin-delete endpoints now snapshot previous outputs before cleanup and log each cleanup step. Distribution logging enhanced with video IDs, folder paths, and file IDs. See plans/improve-job-observability.plan.md.

  • Edit Completed Tracks (2026-03-06): Users can now edit completed karaoke tracks — correcting lyrics, changing instrumental selection, or updating artist/title metadata — without consuming additional credits. Clicking "Edit" on a completed job opens a confirmation modal (with optional artist/title editing), then cleans up all distributed outputs (YouTube video deleted, Dropbox folder removed, GDrive files deleted, GCS finals purged, brand code recycled), resets the job to awaiting_review, and redirects to the existing review screen. After re-review, the standard pipeline re-renders and re-distributes with a fresh brand code. New endpoint: POST /api/jobs/{job_id}/edit. New Job model field: edit_count. State machine updated: COMPLETE → AWAITING_REVIEW transition allowed. See archive/2026-03-06-edit-completed-tracks-plan.md and API.md.

  • Change Visibility (2026-03-06): Users can now toggle completed jobs between public and private via a "Make Public"/"Make Private" button on the job card. Two distinct flows: Public→Private is fast (~1-2 min, redistributes existing finals), Private→Public is slow (~15-30 min, resets styles to Nomad theme, regenerates screens, re-renders video). Confirmation modal explains the implications. Tenant jobs excluded. See archive/2026-03-06-change-visibility-plan.md and API.md.

  • Tenant Portal UI Rework (2026-03-04): Replaced the multi-step consumer wizard with a streamlined 4-field form (Artist, Title, Mixed Audio, Instrumental Audio) for B2B tenant portals. Vocal Star's portal now uses the new TenantJobFlow component instead of GuidedJobFlow. All tenant jobs are automatically private with tenant-specific distribution: Dropbox delivery to /MediaUnsynced/Karaoke/Tracks-VocalStar, Google Drive upload to tenant folder, VSTAR brand prefix. Backend adds _apply_tenant_overrides() helper that forces private, applies locked theme, and overlays tenant distribution settings. Private tenant jobs now use their own Dropbox/GDrive paths instead of the NonPublished override. Frontend conditionally renders TenantJobFlow vs GuidedJobFlow based on tenant detection. See archive/2026-03-04-tenant-portal-ui-rework.md.

  • Custom Video Styling for Private Jobs (2026-03-04): Users who select "Private (no YouTube upload)" can now customize their karaoke video styling. New options in the guided flow's customize step: upload a custom karaoke background image (replaces the starfield behind lyrics), upload a custom title card background image, and change artist/title text colors via color pickers. Both backgrounds show accurate canvas-based live previews. Style assets are uploaded to GCS via signed URLs after job creation (non-blocking — job falls back to default theme on upload failure). Backend: 2 new endpoints (style-upload-urls, style-uploads-complete) that validate ownership, cutoff status, and merge custom assets into the theme's style_params.json. See API.md.

  • Tenant Portal Frontend Separation (2026-03-03): B2B tenants now served from a dedicated Cloudflare Pages project (karaoke-gen-tenant) built with NEXT_PUBLIC_TENANT_PORTAL=true to tree-shake consumer content. A Cloudflare edge function injects window.__TENANT_CONFIG__ and CSS variables into HTML at the edge, eliminating the initial API round-trip for tenant config. Magic link emails now use tenant-specific URLs (e.g., vocalstar.nomadkaraoke.com/auth/verify). Admins can preview any tenant's branding via ?preview_tenant=<id>. See ARCHITECTURE.md.

  • YouTube Quota-Aware Upload System (2026-03-03): Replaced the hard-coded YouTube upload limit (10/day) with quota-unit tracking against the YouTube Data API v3's 10,000 units/day quota. Problem: 4 jobs on 2026-03-03 had YouTube uploads silently skipped with no user notification. The hard limit of 10 uploads/day was overly conservative (actual capacity: ~33 uploads/day at ~300 units each). Solution: (1) YouTubeQuotaService tracks quota units consumed per day in Firestore with Pacific Time midnight reset, (2) YouTubeUploadQueueService queues uploads when quota is low instead of silently skipping, (3) Hourly Cloud Scheduler job processes the queue when quota is available, (4) Users get a follow-up email with their YouTube link when deferred uploads complete, (5) Admin dashboard shows quota usage progress bar and queue management. New Firestore collections: youtube_quota (daily quota tracking), youtube_upload_queue (deferred uploads). New config: YOUTUBE_QUOTA_DAILY_LIMIT, YOUTUBE_QUOTA_UPLOAD_COST, YOUTUBE_QUOTA_SAFETY_MARGIN. See archive/2026-03-03-youtube-quota-fix-plan.md and API.md.

  • Singa Whitelabel Tenant (2026-03-03): Added Singa (singa.com) as second B2B tenant. Green/black branding, file-upload-only mode, locked to "singa" theme, restricted to @singa.com emails. Setup script generates all branding assets (4K backgrounds, CDG backgrounds, logo) programmatically with Pillow. Also fixed critical Zustand getter bug where computed store properties (branding, features, defaults) became stale static values after any set() call — Object.assign invokes getters and copies return values, destroying the getter. Added TenantLandingPage component for minimal tenant portal experience (logo + sign-in). Added Cloudflare Pages _headers for proper cache control. See LESSONS-LEARNED.md.

  • Song Lookup Autocomplete & Community Version Detection (2026-03-03): Added song catalog integration to the job creation flow. Two features: (1) Track autocomplete on the title field provides canonical artist/title names from MusicBrainz + Spotify data via the karaoke-decide catalog API, improving text consistency for YouTube releases. Autocomplete activates when the artist field has a value; selecting a suggestion overwrites both artist and title with canonical names. (2) Community version detection checks karaokenerds.com for existing high-quality karaoke versions — if found, a dismissible green banner suggests the user check those versions first before spending credits. New backend: 3 endpoints (/api/catalog/artists, /api/catalog/tracks, /api/catalog/community-check) with per-user rate limiting (20 req/min) and in-memory TTL caching. New services: catalog_proxy_service.py (proxies to karaoke-decide), karaokenerds_service.py (scrapes karaokenerds.com). New frontend components: AutocompleteInput (reusable), CommunityVersionBanner. See API.md.

  • Feedback-for-Credits (2026-03-01): Added a feedback-for-credits mechanism available to all users (replaces the old beta-only feedback flow). Users who complete 2+ karaoke videos can submit product feedback (star ratings + text) to earn 1 free credit (total possible free credits: 2 — 1 welcome + 1 feedback). New endpoints: GET /api/users/feedback/eligibility, POST /api/users/feedback. The /api/users/me response includes feedback_eligible: bool so the frontend can show/hide prompts without an extra API call. Frontend shows a dismissible banner on the dashboard and a "Earn 1 Free Credit" menu item in the user dropdown when eligible. Feedback stored in user_feedback Firestore collection with duplicate prevention via has_submitted_feedback flag on user model. Welcome email updated with credit-focused messaging and feedback teaser. See API.md.

  • Remove Beta Program from Homepage (2026-03-01): Removed beta enrollment from landing page and replaced with prominent "1 Free Credit" messaging. Beta enrollment endpoint preserved for backward compatibility but no longer actively promoted. See PR #430.

  • Fix Event Loop Blocking in Audio Search Endpoints (2026-02-28): Fixed /api/audio-search/{job_id}/select taking 20-36s to respond (should be <1s) and /api/audio-search/search sometimes taking 25-47s. Root cause: Async endpoints called sync methods that create new event loops via loop.run_until_complete(), blocking the FastAPI event loop — response body was queued but never flushed. Fix: RED/OPS and Spotify download paths now call FlacfetchClient async methods directly (matching YouTubeDownloadService pattern), generic downloads use asyncio.to_thread(), search endpoint uses search_async(). Also fixed InsufficientCreditsError occasionally returning 500 instead of 402 by using isinstance() check. Added 14 new tests. See LESSONS-LEARNED.md.

  • GDrive Cleanup Robustness + Brand Code Race Fix (2026-02-27): Fixed duplicate brand codes appearing in the public Google Drive share after the daily E2E test (NOMAD-1271 appearing for both the test job "piri - dog" and real song "Hot Mulligan - Pop Shuvit"). Root causes: (1) if gdrive_files: was falsy for empty dict {} — when GDrive upload silently failed (pre-PR #417), the entire GDrive cleanup block was skipped, (2) Brand code was recycled immediately after Dropbox cleanup succeeded, without confirming GDrive was also clean — a new job could claim the recycled code while old GDrive files still existed. Fixes: Added find_files_by_brand_code() fallback to GoogleDriveService that searches CDG/MP4/MP4-720p subfolders by brand code prefix when tracked file IDs aren't available. cleanup-distribution endpoint now: (a) uses tracked file IDs (fast path) or falls back to brand_code search, (b) only recycles the brand code after both Dropbox AND GDrive are confirmed clean. Added scripts/cleanup_test_job_gdrive.py remediation script. Added 14 new tests (94% coverage on gdrive_service.py).

  • CDG Packaging Fail-Fast Validation (2026-02-27): Fixed silent CDG generation failures where jobs completed with enable_cdg=True but no CDG ZIP was produced. Root cause: Jobs 5b6aba25 and 5161b069 had instrumental_selection: custom but before commit b671b83f, the orchestrator had no custom case — it fell through to "Backing" suffix. The CDG generator then hit a FileNotFoundError looking for the wrong filename, which was silently caught. Fix: (1) Orchestrator now validates CDG/TXT outputs after packaging stage and before encoding — if enable_cdg=True but no CDG ZIP was produced, the pipeline fails fast instead of silently continuing to expensive encoding, (2) Missing LRC file now raises when CDG/TXT is enabled instead of silently skipping, (3) Added scripts/regenerate_cdg.py remediation script for affected jobs. Removed "CDG requires additional style configuration" known issue (the nomad theme has complete CDG styles). Added 7 new tests for validation logic.

  • GCE Encoding Pipeline Resilience (2026-02-26): Made encoding pipeline resilient to Cloud Run deployment interruptions. A deployment killed a poller mid-encoding, leaving job a03ad39c permanently stuck at encoding status. Three fixes: (1) GCE worker /encode endpoint now idempotent — returns cached results for completed jobs, allows retry on failure, returns status for in-progress (previously hard 409 reject), (2) encoding_service.py handles cached/in_progress responses and falls back to status check on 409, (3) Health service detects jobs stuck in encoding >50 min. See LESSONS-LEARNED.md.

  • Private (Non-Published) Tracks (2026-02-24): Added is_private flag on jobs for private/non-published track production. Private jobs upload to Dropbox Tracks-NonPublished folder with NOMADNP brand prefix, skip YouTube and Google Drive entirely, and send completion emails without the YouTube section. Available as a checkbox on all job creation tabs and as an admin toggle (auto-deletes outputs when toggling a completed job to private). See archive/2026-02-24-private-tracks-plan.md.

  • GCE Encoding Worker Version Sorting Fix (2026-02-11): Fixed critical regression from PR #384 where alphabetical sorting caused encoder to use v0.99.9 instead of v0.116.1. Root cause: sorted(wheels)[-1] uses string comparison, not semantic versioning. String comparison puts "0.99.9" > "0.116.1" (because '9' > '1' in second position). Impact: Production encoder was running an extremely old version (v0.99.9 from months ago) instead of the latest (v0.116.1). Fix: Extract version from wheel filename using regex, sort by packaging.version.Version objects instead of strings. Added 10 comprehensive unit tests covering the exact regression case (0.99.9 vs 0.116.1), edge cases (empty list, invalid filenames), and complex version comparisons (major/minor/patch). Production now correctly selects the latest semantic version. See LESSONS-LEARNED.md.

  • GCE Encoding Worker Wheel Fix (2026-02-10): Fixed encoding worker crash caused by attempting to install karaoke_gen-current.whl, which is not a valid PEP 427 wheel filename (missing version and platform tags). Root cause: The ensure_latest_wheel() function downloaded all wheels including karaoke_gen-current.whl, then tried to install it. pip rejected it with "Invalid wheel filename (wrong number of parts)", causing encoding to fail and the service to restart, resulting in "Cannot connect" errors from Cloud Run. Fix: Filter out wheels containing '-current' before selecting which wheel to install. Only properly versioned wheels (e.g., karaoke_gen-0.116.0-py3-none-any.whl) are now considered. Added 5 comprehensive unit tests covering current wheel exclusion, empty results, version sorting, and path-agnostic filtering. Fixes job 749141f8 and prevents future encoding failures. See LESSONS-LEARNED.md.

  • Flacfetch Retry Logic & Audio Search Recovery (2026-02-10): Added comprehensive retry logic for flacfetch download failures to handle transient network issues and service restarts. Jobs now retry downloads up to 9 times over ~6 minutes with exponential backoff (10s → 20s → 40s → 60s capped), significantly reducing failures from temporary network hiccups. Key improvements: (1) Retry logic with selective error handling (retries on connection errors and 5xx, immediate fail on 4xx), (2) State persistence - download parameters (source_name, source_id, target_file, download_url) now saved BEFORE download attempt, enabling retry even if download fails, (3) Enhanced retry endpoint - /api/jobs/{id}/retry now handles audio search jobs by re-downloading using saved params, (4) User feedback - retry button now shows error toast with actual API error message instead of silent failures. Added 5 comprehensive retry tests covering network errors, 4xx/5xx responses, max retries, and exponential backoff timing. Job 749141f8 investigation revealed the root cause: no source info was saved when flacfetch calls failed, making retry impossible. See LESSONS-LEARNED.md for retry implementation patterns.

  • CDG Sync Fix for Countdown-Padded Songs (2026-02-05): Fixed CDG files being 3 seconds out of sync with audio when songs have countdown padding applied. Root cause: Video rendering adds 3s countdown padding to audio AND shifts LRC timestamps by +3s. CDG generation was using these countdown-shifted LRC timestamps but with instrumental audio (which has no countdown padding), causing lyrics to appear 3s late. Fix: CDG generator now receives lrc_has_countdown_padding flag; when True, it strips the countdown segment and shifts timestamps back before generating the CDG. Added 11 unit tests and updated orchestrator/packaging pipeline. See LESSONS-LEARNED.md.

  • Custom Instrumental Upload Fix (2026-02-26): Fixed critical bug where uploaded custom instrumentals were ignored during encoding — GCE encoding worker and orchestrator had no "custom" case, falling back to clean instrumental instead. Upload endpoint now converts all formats to FLAC for pipeline consistency. Three fixes: (1) GCE encoding worker find_file now searches for custom_instrumental*.flac when selection is "custom", (2) orchestrator config constructs correct (Instrumental Custom).flac path, (3) upload endpoint converts mp3/m4a/wav/etc. to FLAC before storing. Added 10 new regression tests for GCE resolution and orchestrator paths. See API.md.

  • Render Video GCE Routing (2026-03-31): Routed the render_video_worker (OutputGenerator/ffmpeg lyrics rendering) to the GCE encoding worker instead of running on Cloud Run. Previously, render_video always ran on Cloud Run and OOM'd after memory was right-sized to 2Gi. Now when USE_GCE_ENCODING=true, Cloud Run constructs GCS paths from job metadata and delegates to the GCE worker's new /render-video endpoint. Style assets (fonts, backgrounds) are cached on the GCE worker's disk between jobs to avoid repeated downloads of the same default theme. Local fallback preserved when GCE is disabled. See spec.

  • Store Processing Metadata Permanently (2026-03-10): Added processing_metadata field to the Job model — a top-level dict where each worker writes immutable provenance data during processing. Stores AudioShake task_id/asset_id (the original ask — these were previously lost after processing), worker timing, separation model names, ffprobe audio properties, correction stats, cache hit/miss data, video encoding details, and auth context at job creation. All metadata writes are non-fatal (wrapped in try/except). Includes update_processing_metadata helper on JobManager using Firestore dot-notation for atomic section updates. 23 tests covering model, helpers, all worker functions, and a SUMMARY_FIELD_PATHS regression guard. See archive/2026-03-10-store-job-processing-metadata.md and LESSONS-LEARNED.md.

  • Custom Instrumental Upload (2026-02-25): Added POST /api/jobs/{job_id}/upload-instrumental endpoint — was missing from backend, causing 404 in production when users tried to upload custom instrumental files during review. Validates uploaded audio duration matches original (±0.5s tolerance) using ffprobe on GCS signed URLs. Fixed frontend bug where "uploaded" selection type was sent to API instead of "custom" (the only value the backend accepts). Added 22 tests covering upload validation, duration checks, and frontend/backend contract. See API.md.

  • Unicode Filename Fix (2026-02-04): Fixed FFmpeg failures (exit code 183) when processing songs with non-ASCII characters in titles. Job 1eab1172 with Russian title "я куплю тебе дом" failed because Python's str.isalnum() returns True for ALL Unicode alphanumeric characters (Cyrillic, Arabic, Chinese, emoji, etc.), not just ASCII. These Unicode characters in temp subtitle file paths (temp_subtitles_zivert___я_куплю_тебе_дом_1770150285852.ass) caused FFmpeg to fail. Fix: Changed filename sanitization from c.isalnum() to c.isascii() and c.isalnum() in three locations: video.py:222 (generate_video), video.py:282 (generate_preview_video), and remote_cli.py:1970 (output folders). Added comprehensive test suite covering Cyrillic, Chinese, Arabic, emoji, and accented characters. See LESSONS-LEARNED.md.

  • Admin MFY Restart & Override Tools (2026-02-04): Added three new admin endpoints for better Made-For-You order management: (1) Regenerate Screens (POST /api/admin/jobs/{job_id}/regenerate-screens) - regenerates title/end screens with current artist/title metadata without full reprocessing; (2) Full Restart (POST /api/admin/jobs/{job_id}/restart) - unlike "Reset to Start", this actually triggers workers to run, with option to preserve audio stems for faster restart; (3) Override Audio Source (POST /api/admin/jobs/{job_id}/override-audio-source) - switch from YouTube URL to audio search mode for higher quality audio. Frontend admin toolbar updated with new buttons (Regen Screens, Full Restart, Audio Search) with confirmation dialogs. Use case: Fix MFY orders where customer provided wrong artist/title or low-quality YouTube URL. See API.md.

  • Made-For-You YouTube URL Download Fix (2026-01-28): Fixed bug where made-for-you orders with YouTube URLs failed with "Failed to download audio file". Root cause: The webhook handler triggered workers directly without downloading the audio first via YouTubeDownloadService. Workers require input_media_gcs_path to be set, which only happens after download completes. Fix: Handler now downloads YouTube audio to GCS before triggering workers, matching the pattern in file_upload.py and audio_search.py. Added 4 new contract tests with call-order tracking and AST import validation to prevent regression. See LESSONS-LEARNED.md.

  • Lyrics Corrections Now Applied to Video (2026-01-26): Fixed critical bug where user's lyrics corrections made during human review were being ignored. Videos were rendering with raw transcription instead of corrected lyrics. Root cause: The GET /api/review/{job_id}/correction-data endpoint was returning the original corrections.json instead of corrections_updated.json (where user edits are saved). The combined review flow would then overwrite the user's edits when completing the review. Fix: The endpoint now checks for corrections_updated.json first, matching the behavior of render_video_worker.py. Added 35 new tests (backend unit + integration + frontend E2E) to prevent regression. See LESSONS-LEARNED.md.

  • Instrumental Review UX Improvements (2026-01-26): Fixed three UX issues in the instrumental review flow: (1) Loading spinner now visible when switching audio tabs (shows on active tab with disabled state on others), (2) Removed large gap under waveform by fixing flexbox layout, (3) Success screen now appears in cloud mode with countdown before redirect (previously only showed in local CLI mode). Also fixed isLocalMode() detection to correctly distinguish cloud mode (hash-based routing like #/jobId/review) from local CLI mode (path-based routing like /local/review) on localhost, allowing E2E tests to run in cloud mode. Added 17 unit tests for local-mode.ts and 12 E2E regression tests for instrumental review UX. Backend fix: Changed audio_urls['backing'] to audio_urls['backing_vocals'] to match frontend expectations.

  • Dashboard Performance & UX Improvements (2026-01-25): Improved recent jobs list loading performance and usability. Changes: (1) Reduced default job limit from 20 to 5 for faster initial load (typical users only need to see recent jobs), (2) Added job limit dropdown (5/10/20/All options) for power users who want more history, (3) Implemented priority sorting - blocking jobs (awaiting user action) now appear first, followed by processing, completed, and failed jobs, (4) Made OutputLinks buttons responsive by hiding text labels on mobile (icons-only) to prevent horizontal overflow. Added 14 comprehensive unit tests for priority sorting logic. Frontend-only changes, no API modifications.

  • Combined Lyrics + Instrumental Review (2026-01-24): Merged the two-step human review process (lyrics review, then instrumental selection) into a single unified flow. Users now review lyrics AND select instrumental track in one session, submitting both with a single click. Benefits: Better UX (one interaction vs two), reduced drop-off, single email notification instead of two. Technical changes: Backing vocals analysis moved to screens_worker (runs before review), instrumental options included in /api/review/{job_id}/correction-data response, POST /api/review/{job_id}/complete now requires instrumental_selection field. The AWAITING_INSTRUMENTAL_SELECTION state is retained for finalise-only jobs but no longer used in the normal flow. See ARCHITECTURE.md and LESSONS-LEARNED.md.

  • Edit Segment Time Input UX Fix (2026-01-24): Fixed cursor jumping issue when typing multi-digit time values in the edit segment modal. Replaced controlled number inputs with debounced TimeInput component that maintains local state while focused and only syncs to parent after 300ms or on blur. This allows users to smoothly type values like "12.34" without the cursor jumping to the end on each keystroke. Added 15 comprehensive unit tests and E2E regression test to prevent future regressions. See LESSONS-LEARNED.md.

  • Cloud Run Jobs for Workers (2026-01-24): Migrated lyrics and audio workers from FastAPI BackgroundTasks to Cloud Run Jobs. This fixes instance termination issues where Cloud Run would shut down containers mid-processing (e.g., job c94cc9d6 was killed after AudioShake completed but before anchor sequence processing finished). Workers now have CLI entry points (python -m backend.workers.lyrics_worker --job-id abc123) and run as standalone batch containers. Cloud Run Jobs run to completion without HTTP lifecycle concerns, with up to 24-hour execution time. Infrastructure changes via Pulumi define lyrics-transcription-job and audio-separation-job. See ARCHITECTURE.md and LESSONS-LEARNED.md.

  • Encoding Worker Immutable Deployment (2026-01-22): Implemented robust deployment pattern for GCE encoding worker to eliminate version mismatch issues. Changes: (1) Fixed wheel path karaoke_gen-current.whl eliminates version sorting bugs, (2) Self-updating startup script downloaded from GCS on every service start (no Packer rebuild needed for logic changes), (3) Version manifest with strict CI verification (deployment fails on mismatch). Root cause: startup.sh was baked into Packer image, requiring image rebuild + VM recreation for fixes. New pattern allows CI to update deployment logic without infrastructure changes. See infrastructure/encoding-worker/README.md and LESSONS-LEARNED.md.

  • Theme Validation & Default Removal (2026-01-22): BREAKING CHANGE in v0.109.0 - Removed all default style fallback logic to ensure cloud jobs always use complete themes. Style getter functions now raise ValueError if themes are incomplete instead of merging with defaults. Added --list-themes and --validate-theme CLI flags. All cloud jobs guaranteed to use complete nomad theme with no silent fallbacks. See archive/2026-01-22-theme-validation-remove-defaults.md.

  • Admin Job Reprocess (2026-01-21, improved 2026-01-24): Added "Reprocess" button to admin job reset toolbar. Allows re-encoding and re-distributing a job with existing settings after deleting outputs. Resets job to instrumental_selected state and automatically triggers video worker. Clears video/encoding/distribution state while preserving instrumental selection and lyrics metadata. Use case: Fix distribution issues, re-upload to YouTube with different settings, or re-run encoding after infrastructure changes. Improvements (2026-01-24): API response now includes worker_triggered and worker_trigger_error fields for transparency. Added new /api/admin/jobs/{job_id}/trigger-worker endpoint for manual worker triggering when auto-trigger fails. Frontend shows detailed result dialog after reset with status change, cleared data, and worker trigger status. Added "Trigger" button in admin toolbar for manual worker triggering. See API.md.

  • Countdown Audio Sync Fix (2026-01-21, completed 2026-01-24): Fixed bug where songs starting within 3 seconds would have audio desync. PR #328 fixed the writer side: render_video_worker now updates lyrics_metadata.has_countdown_padding. PR #338 fixed the reader side: video_worker_orchestrator now reads the countdown state and passes it through to GCE encoding, which pads the instrumental audio to match. This was a "Fix Both Sides of Dual Code Paths" bug - see LESSONS-LEARNED.md. Added comprehensive test suite (contract tests, pipeline integration, orchestrator countdown tests) to prevent regression. See archive/2026-01-21-countdown-audio-sync-testing-strategy.md.

  • Anchor Sequences Fix (2026-01-21): Fixed anchor sequence calculation when auto-correction is disabled. Previously, disabling auto-correction (SKIP_CORRECTION=true) also stopped anchor/gap calculation, leaving reviewers without guidance on uncertain lyrics areas. Now anchor sequences are calculated independently, so reviewers always see highlighted matches against reference lyrics regardless of correction settings.

  • Auto-Correction Disabled (2026-01-20): Disabled all auto-correction during lyrics transcription (both agentic AI via Gemini and heuristic rule-based handlers). Raw transcription now goes directly to human review without automatic fixes. This change was made because auto-correction was creating more work for reviewers by introducing errors that needed manual correction. Anchor sequences are still calculated to help reviewers identify uncertain areas. The correction feature can be re-enabled by setting SKIP_CORRECTION=false environment variable. See backend/config.py for configuration.

  • Worker Coordination Fix (2026-01-20): Fixed bug where Cloud Run could shut down prematurely when one parallel worker completed while another was still running. Added WorkerRegistry to track active workers per job. Workers register at start and unregister in finally block. FastAPI lifespan shutdown handler now waits up to 10 minutes for active workers to complete before allowing container termination. Fixes job failures where lyrics worker was killed mid-processing after audio worker completed. See ARCHITECTURE.md and LESSONS-LEARNED.md.

  • Admin Delete Outputs (2026-01-10, UX improved 2026-01-24): Added admin button to delete all distributed outputs (YouTube, Dropbox, Google Drive) for a job while preserving the job record. Use case: fix quality issues by deleting outputs, resetting to awaiting_review, correcting lyrics, and re-generating. The brand code is freed for reuse when Dropbox folder is deleted. New endpoint: POST /api/admin/jobs/{job_id}/delete-outputs. UI: Delete Outputs button in admin job detail page, "Outputs Deleted" badge when flag is set. UX improvements (2026-01-24): Download buttons (4K, 720p, CDG, TXT, etc.) are now hidden when outputs have been deleted, preventing broken links. Delete confirmation now shows a detailed result dialog with per-service status (YouTube video ID deleted, Dropbox folder path, Google Drive files) instead of a simple toast. See API.md.

  • Rate Limiting & Abuse Prevention (2026-01-09): Added per-user rate limiting (5 jobs/day default), system-wide YouTube upload limits (10/day), and beta enrollment abuse prevention. Features: email normalization (Gmail alias detection), disposable domain blocklist (130+ domains), blocked email/IP lists, IP-based enrollment rate limiting (1 per 24h), and admin UI for managing blocklists and user overrides at /admin/rate-limits. Admins can grant users bypass permissions or custom limits. All limits configurable via environment variables. See API.md.

  • Fast Preview Video Option (2026-01-10): Added toggle in lyrics review UI to render preview videos with black background (~10s) instead of theme background image (~30-60s). Default is now black background for faster iteration during review. Users can enable theme background via checkbox when they want to verify the final look. See API.md.

  • Frontend Consolidation (2026-01-09): Consolidated three separate frontends into a single Next.js application. Previously: (1) Main frontend (Next.js + Tailwind), (2) Lyrics Review (React + Vite + MUI, ~14k lines), and (3) Instrumental Review (vanilla HTML/JS, ~1.7k lines) were deployed separately. Now all UIs are unified in the main frontend at /app/jobs#/{id}/review and /app/jobs#/{id}/instrumental. Benefits: single deployment to Cloudflare Pages, no external redirects during user workflows, unified design system (Radix/Tailwind), simpler authentication via job ownership model. See archive/2026-01-09-frontend-consolidation-plan.md and tasks/.

  • Admin Login Token for Made-For-You Orders (2026-01-10): Added one-click admin login via email. When a made-for-you order is placed, the admin notification email now includes a direct link with an embedded login token (?admin_token=TOKEN). Clicking this link auto-authenticates the admin and opens the job directly - no manual login required. Tokens expire after 24 hours. Also added: BCC to done@nomadkaraoke.com on all job completion emails, fixed timezone display (EST→ET for DST correctness). See API.md.

  • Made-For-You Audio Selection Flow (2026-01-08, fixed 2026-01-09): Refactored made-for-you order processing to pause at audio selection like regular search jobs. New flow: customer places order → admin receives notification email → admin selects audio source in UI → job processes under admin ownership → on completion, ownership transfers to customer → customer receives delivery email. Added model fields (made_for_you, customer_email, customer_notes), professional email templates for order confirmation and admin notification, intermediate email suppression (no lyrics/instrumental reminders to customer during processing), and ownership transfer on job completion. Distribution defaults (Dropbox, YouTube, GDrive, brand prefix) are now properly applied from server settings. Bug fix (2026-01-09): Fixed webhook handler that was not setting made_for_you=True, not using admin ownership, not storing customer_email/customer_notes, and not pausing at audio selection. Added 17 comprehensive unit tests for the webhook handler. See API.md and LESSONS-LEARNED.md.

  • White-Label B2B Portal Infrastructure (2026-01-08): Added multi-tenant support for white-label karaoke portals. Vocal Star is the first tenant at vocalstar.nomadkaraoke.com. Features: subdomain-based tenant detection (frontend and backend), GCS-backed tenant config storage, per-tenant feature flags (audio search, file upload, YouTube, theme selection, etc.), dynamic branding (logo, colors, site title via CSS variables), locked themes per tenant, email domain restrictions for auth, and tenant-scoped jobs/users in Firestore. Backend: TenantMiddleware extracts tenant from X-Tenant-ID header, query param (dev only), or Host subdomain; TenantService loads config from GCS with caching. Frontend: Zustand store for tenant state, TenantProvider for initialization, TenantLogo component. Setup script: scripts/setup-vocalstar-tenant.py. See ARCHITECTURE.md and API.md.

  • Encoding Worker Infrastructure Upgrade (2026-01-08): Upgraded GCE encoding worker from c4-standard-8 (Intel) to c4d-highcpu-32 (AMD EPYC 9B45 Turin). Benchmarking showed 4.92x faster encoding (135s vs 666s total). Key improvements: With Vocals 4K rendering 6x faster (54s vs 324s), 720p downscale 4.6x faster (11s vs 50s). Reduces total encoding time from ~11 min to ~2.3 min per job. See archive/2026-01-08-performance-investigation.md for full benchmark methodology and results.

  • Theme Enforcement & Encoding Resilience (2026-01-08): Added defense-in-depth theme enforcement - jobs without theme_id are now rejected at creation (JobManager) with a safety net at processing time (screens_worker). Fixed made-for-you webhook to apply default theme. Added retry logic with exponential backoff (3 retries, 2s→4s→8s) to GCE encoding service for transient connection failures during worker restarts. See LESSONS-LEARNED.md.

  • Comprehensive Performance Optimization (2026-01-08): Reduced lyrics processing from 16+ minutes to ~5 minutes through multiple optimizations. Key changes: (1) NLTK cmudict preloading at startup saves 100-150s, (2) Langfuse callback handler preloading saves 200s, (3) Model warmup in AgenticCorrector.from_model() prevents parallel initialization race, (4) Parallel anchor sequence search (4 workers) reduces n-gram processing from 38s to ~10s. Added /health/preload-status endpoint for deployment verification. PR #236. See archive/2026-01-08-performance-investigation.md and LESSONS-LEARNED.md.

  • UI Simplification & Brand Accessibility (2026-01-08): Simplified the "Create Karaoke Video" form by hiding advanced options (theme settings now admin-only, "Display As" behind toggle). Improved audio search dialog UX by hiding technical tracker names and showing user-friendly availability badges. Fixed brand pink color contrast for accessibility - swapped default/hover values so white text on pink buttons meets WCAG contrast ratio. See BRAND-STYLE-GUIDE.md v1.2.

  • SpaCy Preloading (2026-01-08): Implemented SpaCy model preloading at container startup to eliminate 60+ second delay during agentic correction. The en_core_web_sm model is now loaded during FastAPI lifespan startup, and PhraseAnalyzer/SyllablesMatchHandler reuse the preloaded model. Added timing logs to verify performance. See archive/2026-01-08-spacy-preload-plan.md.

  • Thread-Safe LangChain Model Initialization (2026-01-08): Fixed race condition in LangChainBridge where parallel threads could all try to initialize the AI model simultaneously, causing 6+ minute delays. Added double-checked locking with threading.Lock(). PR #232. See LESSONS-LEARNED.md.

  • Brand Consistency & Style Guide (2026-01-07): Implemented unified brand theming across all Nomad Karaoke frontends. Primary color changed from blue to brand pink (#ff7acc). Added comprehensive BRAND-STYLE-GUIDE.md for human and LLM reference. Includes: CSS variables for light/dark mode, MUI theme updates for lyrics transcriber, email template branding, and Playwright visual tests. See BRAND-STYLE-GUIDE.md.

  • Audio Search Display Override (2026-01-06): Added optional "Display As" fields to audio search, allowing users to search for audio by one artist/title (e.g., "Jeremy Kushnier") but display a different artist/title on title screens and filenames (e.g., "Footloose (Broadway Cast)"). Useful for Broadway casts, covers, remixes where tracker metadata differs from desired display. See API.md.

  • Infrastructure Modularization (2026-01-06): Refactored infrastructure/__main__.py from 2,602 lines to 339 lines (87% reduction). Split into organized modules: modules/ for core GCP resources, compute/ for VMs and startup scripts, config.py for shared constants. Extracted embedded encoding worker Python app to backend/services/gce_encoding/. See infrastructure/README.md for new structure and infrastructure/docs/PHASE5-PACKER-IMAGE-PLAN.md for planned Packer image optimization.

  • Email Notification System (2026-01-06): Added automated email notifications for job completion and user action reminders. Features: GCS-backed HTML email templates with fallback defaults, Postmark with CC support, auto-completion emails on job finish, idle reminder emails via Cloud Tasks (5-min delay for blocking states), admin UI buttons to copy message or send email manually. Endpoints: GET /api/admin/jobs/{id}/completion-message, POST /api/admin/jobs/{id}/send-completion-email. Feature flag ENABLE_AUTO_EMAILS (default: false). See API.md and ARCHITECTURE.md.

  • GCE Preview Encoding Unified with LocalPreviewEncodingService (2026-01-06): GCE preview encoding now uses the same LocalPreviewEncodingService from the installed wheel, eliminating 100 lines of duplicated FFmpeg logic. Preview videos are now identical across local CLI, Cloud Run, and GCE worker environments. Benefits include: consistent encoding settings (480x270, 24fps, crf 28), proper FFmpeg filter path escaping for special characters, hardware acceleration (NVENC) when available, and custom font support. See LESSONS-LEARNED.md.

  • GCE Encoding Worker Python 3.13 (2026-01-06): Upgraded GCE encoding worker from Debian's Python 3.11 to Python 3.13 built from source. Uses dedicated virtual environment at /opt/encoding-worker/venv, removing the need for --break-system-packages. Aligns the encoding worker with CI/Cloud Run Python version.

  • GCE Encoding Unified with LocalEncodingService (2026-01-06): GCE encoding worker now uses the same LocalEncodingService as the local CLI, eliminating duplicated encoding logic. Output files now have proper names (Artist - Title (Final Karaoke Lossless 4k).mp4 instead of output_4k_lossless.mp4) and include title/end screen concatenation. Wheel deployed to GCS; worker installs at job start for hot updates without VM restart. Removed all fallback logic - single code path for consistent output across CLI, Cloud Run, and GCE. See LESSONS-LEARNED.md.

  • Cloud Distribution Matches Local CLI (2026-01-06): Fixed cloud distribution (Dropbox uploads) to match local CLI output structure. The lyrics/ folder now includes: proper filenames ((Lyrics Corrections).json instead of (Karaoke).json), all reference lyrics files ((Lyrics Genius).txt, (Lyrics Spotify).txt, etc.), (With Vocals).mkv, and previews/ subfolder with preview ASS files. See LESSONS-LEARNED.md.

  • Agentic Correction Performance (2026-01-05): Optimized agentic AI correction from ~5 minutes to ~55 seconds for 20 gaps (~5-6x speedup). Fixed anti-pattern of creating new model instance per gap (caused repeated 2s warm-up overhead). Now creates model once and processes gaps in parallel using ThreadPoolExecutor (default 5 workers). Configure via AGENTIC_MAX_PARALLEL_GAPS env var. See LESSONS-LEARNED.md.

  • Worker Logs Subcollection (2026-01-04): Moved worker_logs from embedded array in job documents to Firestore subcollection (jobs/{job_id}/logs). Fixes job failures when logs exceed 1MB (job 501258e1 had 1.26MB of logs). New logs stored with 30-day TTL via Firestore TTL policy. Feature flag USE_LOG_SUBCOLLECTION=true (default). See LESSONS-LEARNED.md.

  • Admin Job Detail Enhancements (2026-01-09): Enhanced admin job detail page with comprehensive job management features. Added: file downloads with signed GCS URLs grouped by category (stems, lyrics, outputs), inline editing of job fields (artist, title, theme, etc.), job reset actions to re-process from specific workflow checkpoints (pending, audio selection, review, instrumental selection), visual timeline with color-coded stages. New endpoints: GET /api/admin/jobs/{job_id}/files, PATCH /api/admin/jobs/{job_id}, POST /api/admin/jobs/{job_id}/reset. See API.md.

  • GCE Encoding Response Fixes (2026-01-04): Fixed multiple response format mismatches with GCE encoding worker. The worker returns output_files as a list of paths, not a dict with format keys - added conversion logic. Also added defensive type checking for status responses that could be lists. See LESSONS-LEARNED.md.

  • Video Worker Orchestrator (2026-01-04): Major refactor to unify video generation pipeline. Created VideoWorkerOrchestrator that coordinates all stages (packaging, encoding, distribution, notifications) regardless of encoding backend (GCE or local). Fixes issue where GCE encoding path bypassed YouTube upload, Discord notifications, and CDG/TXT packaging. Feature flag USE_NEW_ORCHESTRATOR (default: true) enables rollback. 139 new tests across 6 new service modules. See ARCHITECTURE.md and archive/2026-01-04-video-worker-orchestrator-refactor.md.

  • Full Unicode Font Support (2026-01-03): Fixed rendering of musical symbols (♪) and added comprehensive international font support to Docker base image. Installed Noto fonts covering Latin, CJK (Chinese/Japanese/Korean), Arabic, Hebrew, Thai, and other scripts. Changed default karaoke font from Arial to Noto Sans. See LESSONS-LEARNED.md.

  • Flacfetch Cache API Integration (2026-01-03): Enhanced audio search cache management to also clear flacfetch's GCS cache. When admins clear a job's cache, both Firestore (job state) and flacfetch (GCS search results) are cleared simultaneously. Added "Clear All Cache" button and cache stats display to admin UI. New endpoints: DELETE /api/admin/cache, GET /api/admin/cache/stats. See API.md.

  • Audio Search Cache Management (2026-01-03): Added admin UI at /admin/searches to view and manage cached audio search results. Admins can clear stale caches (e.g., when flacfetch is updated with new providers) and allow users to re-search. Fixes issue where jobs showed only YouTube results when lossless sources should be available. See API.md and LESSONS-LEARNED.md.

  • User Database Separation (2026-01-03): Migrated karaoke-gen users from shared users collection to dedicated gen_users collection. This separates karaoke-gen user data from karaoke-decide user data (both apps share the same GCP project/Firestore instance). Migration script in scripts/migrate_users_to_gen_users.py. See ARCHITECTURE.md for collection details.

  • Admin Dashboard (2026-01-03): Added comprehensive admin dashboard at /admin with: stats overview (users, jobs, credits), user management (search, sort, add credits, toggle role, enable/disable), job browser (filter by status/user, view details, delete), and beta program monitoring. Query-param based routing for static export compatibility. See API.md for backend endpoints.

  • GCE Encoding Worker (2026-01-03): Added optional high-performance encoding on dedicated GCE VM (C4-standard-8 with Intel Granite Rapids 3.9 GHz). Uses USE_GCE_ENCODING=true env var. Falls back to Cloud Run if unavailable. See ARCHITECTURE.md.

  • Runner Disk Space Auto-Cleanup (2026-01-03): Fixed CI failures caused by self-hosted runners filling up with Docker images. Changed cleanup from daily/7-day-old to hourly/threshold-based (70%). Added pre-job disk checks to all self-hosted CI jobs. See LESSONS-LEARNED.md

  • E2E Happy Path Test (2026-01-02): Added comprehensive E2E test that validates the complete karaoke generation journey using ONLY browser UI interactions (no API shortcuts). Test covers all 12 steps from landing page through cleanup. Runs daily via GitHub Actions and takes ~38 minutes. See E2E-HAPPY-PATH-TEST-SPEC.md

  • LyricsTranscriber Cache Persistence (2026-01-02): Added GCS-backed cache for LyricsTranscriber to avoid redundant API calls. AudioShake transcription and lyrics provider responses (Genius, Spotify, LRCLib, Musixmatch) are now cached to gs://karaoke-gen-storage/lyrics-transcriber-cache/. Cache is synced before/after each job, persisting across Cloud Run instances. Significantly reduces API costs and speeds up repeated processing of same songs.

  • Worker Timeout Fixes (2026-01-01): Fixed 3 timeout issues blocking job completion: (1) Lyrics transcription timeout increased to 20 min (PR #153), (2) Cloud Tasks dispatch_deadline added for audio worker - default 10 min was killing Modal API calls (PR #154), (3) Enabled Cloud Run Jobs for video encoding - 1-hour timeout vs 30-min Cloud Run service limit (PR #155). Full E2E pipeline now verified working. See archive/2026-01-01-worker-timeout-fixes.md

  • Agentic Correction Timeout (2025-12-31): Added 3-minute configurable timeout for agentic AI lyrics correction. Prevents stuck jobs when songs have many gaps (74+ gaps could take 30+ minutes). On timeout, skips correction and proceeds to human review with raw transcription. See archive/2025-12-31-agentic-timeout-implementation.md

  • Job Failure Fixes (2025-12-31): Fixed 4 critical issues causing job failures: (1) user_email not set on jobs - users couldn't see their jobs, (2) YouTube URL race condition - lyrics worker started before audio downloaded, (3) Audio search cache not persisting across Cloud Run instances, (4) Jobs stuck in downloading state - added 10min transcription timeout. See archive/2024-12-31-job-failure-investigation.md

  • Vertex AI Auth Fix (2025-12-31): Fixed ChatGoogleGenerativeAI to use Vertex AI backend with service account auth on Cloud Run. PR #145 broke cloud deployments by requiring GOOGLE_API_KEY. Solution: pass project parameter to trigger Vertex AI backend with ADC. See archive/2025-12-31-vertex-ai-auth-fix.md

  • LangChain Provider Migration (2025-12-30): Migrated from langchain-google-vertexai (gRPC) to langchain-google-genai (REST) to fix silent hangs during model initialization. Added initialization and warm-up timeouts using ThreadPoolExecutor for fail-fast behavior. See archive/2025-12-30-langchain-vertexai-to-genai-migration.md

  • LangFuse Prompt Management (2025-12-30): Replaced hardcoded LLM prompts with LangFuse-managed prompts for dynamic iteration without redeploying. Added migration script, LangFuse Vertex AI service account for GCP credits. See archive/2025-12-30-langfuse-prompt-management.md

  • Gemini 3 Flash Agentic Correction Fix (2025-12-30): Fixed agentic correction hanging due to Gemini 3 requiring global location instead of regional endpoints, and multimodal response format handling. Added local test script for faster iteration. See archive/2025-12-30-gemini3-agentic-correction-fix.md

  • Cold Start UX (2025-12-30): Frontend now shows friendly "Warming up the generator..." messages when backend is slow to respond (cold start after deployment or inactivity). Progressive messages appear after 5s and 15s to reassure users during 30s+ cold starts.

  • Cloud Output Structure Fix (2025-12-30): Fixed remote CLI (karaoke-gen-remote) to produce identical output as local CLI: YouTube uploads now work, Dropbox includes stems/ and lyrics/ subfolders, instrumental filenames use actual model names. See archive/2025-12-30-cloud-output-structure-fix.md

  • Auth Session Persistence Fix (2025-12-30): Fixed two bugs causing 401 errors despite valid sessions: timezone-aware datetime comparisons in backend, and stale token caching in frontend. Firestore indexes now tracked in Pulumi. See archive/2025-12-30-auth-session-persistence-fix.md

  • E2E Test Suite Consolidated (2025-12-29): Tests reorganized into production/ (real API) and regression/ (mocked API, CI-safe). See archive/2025-12-29-e2e-test-consolidation-plan.md

Doc Purpose
PRODUCT-VISION.md Product goals, audience, business model
ARCHITECTURE.md System design, data flow, integrations
DEVELOPMENT.md Local setup, testing, deployment
TESTING.md Testing standards, CI requirements, Playwright
API.md Backend endpoint reference
REQUESTS-BOARD.md Public voting board (requests.nomadkaraoke.com) — what it is, UX, how it works; Phase 2 handoff linked
BRAND-STYLE-GUIDE.md Brand colors, typography, UI patterns
LESSONS-LEARNED.md Key insights for future agents
STRIPE-SETUP.md Payment integration setup
MOBILE-UX-BEST-PRACTICES.md Mobile/responsive design guidelines
archive/ Historical documentation

Workflow Summary

Upload Audio → Parallel Processing → Human Review → Video Generation → Download
                   │                      │
        (Cloud Run GPU + AudioShake)  (LyricsTranscriber UI)

For AI Agents

Start with CLAUDE.md in the repo root for essential rules, then reference these docs as needed. Keep this README current when project state changes.