## Summary
Both production error patterns alerted on 2026-06-04 traced to a
**single oversized input** — job `9a12cf0f`, a 32-minute mashup (422
segments, 3,742 words). Two fixes, plus a follow-up design doc for the
real solution (duration-based credit pricing).
## Changes
- **Firestore doc-size (`backend/api/routes/jobs.py`):**
`submit_corrections` now persists only a lightweight summary
(`state_data.corrected_lyrics = {corrected_segment_count: N}`) instead
of the multi-MB segment arrays
(`corrected_segments`/`original_segments`/`resized_segments`, ~600KB
each) that exceeded Firestore's 1MB limit. The prior `reference_lyrics`
strip missed the real bulk. Full corrections still go to GCS
(`corrections_updated.json`), which the render worker reads. The
`complete_review` 0-segment guard reads the summary and is
backward-compatible with the legacy full-dict shape.
- **Transcription timeout (`backend/workers/lyrics_worker.py`,
`backend/config.py`):** new `compute_transcription_timeout()` scales the
outer safety-net timeout by ffprobe-measured audio duration —
`clamp(floor 1200, duration × 1.0, cap 1700)` — instead of a fixed 1200s
that flapped long tracks to `failed`.
- **Infra (`infrastructure/modules/cloud_run.py`):**
lyrics-transcription-job task timeout `1800s → 3000s` + env
`TRANSCRIPTION_TIMEOUT_CAP_SECONDS=2700` so long mashups get real
headroom. **Requires `pulumi up`.**
- **Docs:** prompt/design doc for the follow-up duration-based credit
pricing + warn/block feature
(`docs/archive/2026-06-04-long-duration-input-handling-prompt.md`); two
LESSONS-LEARNED entries.
- **Tests:** new coverage for the summary contract, guard
backward-compat, and timeout boundaries; updated obsolete
`reference_lyrics`-stripping tests; modernized 4 flaky
`asyncio.get_event_loop()` tests.
## Testing
- [x] Backend suite: **3184 passed, 36 skipped** (`pytest
backend/tests`, excluding emulator/integration)
- [x] Targeted new tests green (doc-size summary, guard, timeout
scaling)
## Review
- [x] CodeRabbit CLI review completed locally (0 findings)
- [x] Issues addressed
> Note: these are targeted fixes for the symptoms. The structural fix is
the duration-pricing feature in the linked design doc.
@coderabbitai ignore
---
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Karaoke-Gen Documentation
Status: Production-ready. First full E2E flow completed 2025-12-28.
What Works
- Web App: https://gen.nomadkaraoke.com
/- Landing page with pricing/app- Main app (upload audio, review lyrics, download videos)/admin- Admin dashboard (user management, job monitoring)
- Backend API: https://api.nomadkaraoke.com - FastAPI on Cloud Run
- CLI Tools:
karaoke-gen(local),karaoke-gen-remote(cloud)
Current State
| Component | Status |
|---|---|
| Audio upload & separation | Working |
| Lyrics transcription | Working |
| Auto-correction (agentic + heuristic) | Disabled (raw transcription goes to human review) |
| Combined review (lyrics + instrumental) | Working |
| Preview video generation | Working |
| Multi-format encoding | Working |
| Token-based auth | Working |
| Magic link auth | Working |
| Payment flow (Stripe) | Working |
| Feedback-for-credits | Working (2+ jobs → 1 free credit) |
| Admin dashboard | Working |
| Abuse prevention & YouTube quota | Working (blocklists, IP/fingerprint rate limiting, credits on verification, quota-aware YouTube uploads) |
| CI/CD self-hosted runner | Working (GCP) |
| E2E happy path test | Working (~20-25 min full pipeline) |
| White-label B2B portals | Working (Vocal Star, Singa) |
| Private (non-published) tracks | Working |
| Custom video styling (private jobs) | Working |
| Song suggestions & fuzzy correction | Working (catalog lookup on Step 2 + "Did you mean?" typo correction) |
| Community version detection | Working (via karaokenerds.com) |
| Review session backup/restore | Working (auto-save + restore UI) |
| Review reminder & auto-expiry | Working (24h reminder, 48h cancel + refund) |
| i18n / Multilingual (frontend) | Working (next-intl, en/es/de, 1062 strings, [locale] routing) |
| Referral system | Working (referral links with configurable discounts, 20% cash kickback via Stripe Connect, auto-payout at $20) |
| Error monitor | New (Cloud Run Job every 15 min — log collection, normalization, LLM dedup/grouping, Discord alerts, auto-resolution, daily digest) |
Known Issues
(No known issues)
Pending Work
(No pending work items)
Recent Changes
-
Syllable-Aware Custom Lyrics (2026-05-02): Enhanced the Custom Lyrics LLM mode with syllable-count validation and a per-line repair loop. Backend computes per-line target/candidate syllable counts via the existing
SyllablesMatchHandlerengine and re-prompts Gemini up to 3 times until lines fit budget (Δ ≤ 2 default, configurable via Verbatim/Loose/Balanced/Tight/Strict strictness). NewGenerationSettings(allow_reword,allow_omit,fixed_line_count,strictness) exposed in the modal as a collapsible settings panel. NewCustomLyricsPreviewcomponent shows per-line syllable annotations (target / actual / Δ) with severity badges and stop-reason banners. Variable-line-count mode (whenfixed_line_count=false) re-times segments proportionally. Eval harness with disk-backed Gemini cache, per-line + per-fixture + corpus scorers, and 4 fixtures (1 from production, 3 handcrafted public-domain). v1-ships criteria validation (Year 5 ≥75% pass@2, macro corpus ≥70% pass@2) deferred to post-merge. See archive/2026-05-02-custom-lyrics-syllable-aware-design.md and archive/2026-05-02-custom-lyrics-syllable-aware-plan.md. -
Custom Lyrics LLM Mode (2026-04-28): New AI-powered lyrics-replacement mode in the lyrics review modal. Users (or admins editing client jobs) can paste custom lyrics or instructions — or upload a document — and an LLM returns lyrics that match the original line count exactly, ready to drop into the existing replace-segments flow. Includes backend
CustomLyricsServicewith file extraction, prompt construction, and line-count validation; newPOST /api/review/{job_id}/custom-lyrics/generateendpoint; preview UI with regenerate/save controls; manual-sync toast guidance after save. See archive/2026-04-28-llm-custom-lyrics-design.md and archive/2026-04-28-llm-custom-lyrics-plan.md. -
GCE Encoding Cold-Start Resilience (2026-04-27): Fixed renders failing when the encoding worker VM was cold (TERMINATED). Job 2c577535 hit this on 2026-04-24: the existing 90s connection retry window (sized for
systemctl restartduring deploys) was too short for a 2-5 min GCE cold boot. Fix: newEncodingWorkerManager.wait_for_worker_ready()two-phase poll (VM status →/health200) is now awaited whenensure_primary_running()reportsstarted=True(VM was actually TERMINATED). Deploy-restart fast path unchanged. Also:_format_exception()helper so emptyaiohttp.ClientConnectorErrormessages still produce diagnostic logs (the original incident logged "GCE worker connection failed after 8 attempts: " with no detail). 17 new tests including a regression integration test. See archive/2026-04-27-gce-encoding-cold-start-design.md, LESSONS-LEARNED.md, and TROUBLESHOOTING.md. -
Anti-Abuse: Credits on Verification + Rate Limiting (2026-03-21): Three-layer defense against free credit abuse. (1) Welcome credits now granted on first magic link verification, not account creation — unverified accounts get 0 credits. (2) Per-IP signup rate limiting: max 2 new accounts per IP per 24 hours, silent reject. (3) Device fingerprinting (FingerprintJS open source): browser fingerprint sent with magic link request, combined with IP for rate limiting. Either signal hitting 2/day triggers block. Returning users are never rate limited. Graceful degradation if fingerprinting fails. Frontend verify page shows "You received N free credits" on first login. See archive/2026-03-21-anti-abuse-credits.plan.md.
-
Review Reminder & Auto-Expiry (2026-03-19): Jobs stuck in review (
awaiting_review/in_review) now get automatic cleanup. At 24 hours, a gentle reminder email is sent with a review link, help offer, and warning. At 48 hours, the job is auto-cancelled and the user's credit is refunded. Expiry notification email confirms the refund and current balance. Excludes made-for-you and tenant jobs. Powered by hourly Cloud Scheduler cron (stale-review-hourly). New endpoint:POST /api/internal/process-stale-reviews. Also fixedmake testhanging on frontend Playwright E2E tests by splitting intotest-frontend(Jest unit only) andtest-frontend-e2e(Playwright, run manually). See archive/2026-03-19-review-reminder-autoexpiry.plan.md. -
Step 2 Song Suggestions & Fuzzy Correction (2026-03-08): Replaced the Step 1 autocomplete with a simpler approach: plain text inputs on Step 1, then catalog lookup + community version check fire in parallel with the audio search on Step 2. Results appear as a "Song found in our database" panel (exact match) or correction suggestion panel while the user waits. Added fuzzy "Did you mean?" correction — when audio search returns poor results (Tier 3) and catalog has no exact title match, fetches the artist's discography and fuzzy-matches against the user's title using LCS similarity. Shows an amber banner with one-click correction + expandable alternatives list. Handles garbled artist fields via title-based fallback with artist tiebreaker. Visibility step reworked with action buttons at top for one-click select+continue. See archive/2026-03-08-step2-song-suggestions.plan.md and archive/2026-03-08-fuzzy-did-you-mean.plan.md.
-
Review Session Backup/Restore (2026-03-08): Server-side auto-save of lyrics review sessions replaces fragile localStorage-only mechanism. Sessions auto-save every 60s (if dirty), on preview video generation, and on page unload. Deduplication via SHA-256 hash prevents redundant saves. Correction data stored in GCS (can exceed Firestore's 1MB doc limit); metadata in Firestore subcollection (
jobs/{jobId}/review_sessions/{sessionId}). Restore UI shows split-pane session browser with edit diff preview. Cross-job session search allows reusing edits across different jobs (with duration mismatch warning). On load, automatically prompts to restore if saved sessions exist. See archive/2026-03-07-lyrics-review-sessions-plan.md. -
Job Observability Improvements (2026-03-07): Enhanced timeline events with a
metadatafield to capture structured output details (brand codes, YouTube video IDs, Dropbox paths, GDrive file IDs) at key moments — job completion, edit initiation, and admin output deletion. Addedlog_to_job()helper for API endpoints to write to Firestore job log subcollections (previously only workers logged there). Edit and admin-delete endpoints now snapshot previous outputs before cleanup and log each cleanup step. Distribution logging enhanced with video IDs, folder paths, and file IDs. See plans/improve-job-observability.plan.md. -
Edit Completed Tracks (2026-03-06): Users can now edit completed karaoke tracks — correcting lyrics, changing instrumental selection, or updating artist/title metadata — without consuming additional credits. Clicking "Edit" on a completed job opens a confirmation modal (with optional artist/title editing), then cleans up all distributed outputs (YouTube video deleted, Dropbox folder removed, GDrive files deleted, GCS finals purged, brand code recycled), resets the job to
awaiting_review, and redirects to the existing review screen. After re-review, the standard pipeline re-renders and re-distributes with a fresh brand code. New endpoint:POST /api/jobs/{job_id}/edit. New Job model field:edit_count. State machine updated:COMPLETE → AWAITING_REVIEWtransition allowed. See archive/2026-03-06-edit-completed-tracks-plan.md and API.md. -
Change Visibility (2026-03-06): Users can now toggle completed jobs between public and private via a "Make Public"/"Make Private" button on the job card. Two distinct flows: Public→Private is fast (~1-2 min, redistributes existing finals), Private→Public is slow (~15-30 min, resets styles to Nomad theme, regenerates screens, re-renders video). Confirmation modal explains the implications. Tenant jobs excluded. See archive/2026-03-06-change-visibility-plan.md and API.md.
-
Tenant Portal UI Rework (2026-03-04): Replaced the multi-step consumer wizard with a streamlined 4-field form (Artist, Title, Mixed Audio, Instrumental Audio) for B2B tenant portals. Vocal Star's portal now uses the new
TenantJobFlowcomponent instead ofGuidedJobFlow. All tenant jobs are automatically private with tenant-specific distribution: Dropbox delivery to/MediaUnsynced/Karaoke/Tracks-VocalStar, Google Drive upload to tenant folder, VSTAR brand prefix. Backend adds_apply_tenant_overrides()helper that forces private, applies locked theme, and overlays tenant distribution settings. Private tenant jobs now use their own Dropbox/GDrive paths instead of the NonPublished override. Frontend conditionally rendersTenantJobFlowvsGuidedJobFlowbased on tenant detection. See archive/2026-03-04-tenant-portal-ui-rework.md. -
Custom Video Styling for Private Jobs (2026-03-04): Users who select "Private (no YouTube upload)" can now customize their karaoke video styling. New options in the guided flow's customize step: upload a custom karaoke background image (replaces the starfield behind lyrics), upload a custom title card background image, and change artist/title text colors via color pickers. Both backgrounds show accurate canvas-based live previews. Style assets are uploaded to GCS via signed URLs after job creation (non-blocking — job falls back to default theme on upload failure). Backend: 2 new endpoints (
style-upload-urls,style-uploads-complete) that validate ownership, cutoff status, and merge custom assets into the theme'sstyle_params.json. See API.md. -
Tenant Portal Frontend Separation (2026-03-03): B2B tenants now served from a dedicated Cloudflare Pages project (
karaoke-gen-tenant) built withNEXT_PUBLIC_TENANT_PORTAL=trueto tree-shake consumer content. A Cloudflare edge function injectswindow.__TENANT_CONFIG__and CSS variables into HTML at the edge, eliminating the initial API round-trip for tenant config. Magic link emails now use tenant-specific URLs (e.g.,vocalstar.nomadkaraoke.com/auth/verify). Admins can preview any tenant's branding via?preview_tenant=<id>. See ARCHITECTURE.md. -
YouTube Quota-Aware Upload System (2026-03-03): Replaced the hard-coded YouTube upload limit (10/day) with quota-unit tracking against the YouTube Data API v3's 10,000 units/day quota. Problem: 4 jobs on 2026-03-03 had YouTube uploads silently skipped with no user notification. The hard limit of 10 uploads/day was overly conservative (actual capacity: ~33 uploads/day at ~300 units each). Solution: (1)
YouTubeQuotaServicetracks quota units consumed per day in Firestore with Pacific Time midnight reset, (2)YouTubeUploadQueueServicequeues uploads when quota is low instead of silently skipping, (3) Hourly Cloud Scheduler job processes the queue when quota is available, (4) Users get a follow-up email with their YouTube link when deferred uploads complete, (5) Admin dashboard shows quota usage progress bar and queue management. New Firestore collections:youtube_quota(daily quota tracking),youtube_upload_queue(deferred uploads). New config:YOUTUBE_QUOTA_DAILY_LIMIT,YOUTUBE_QUOTA_UPLOAD_COST,YOUTUBE_QUOTA_SAFETY_MARGIN. See archive/2026-03-03-youtube-quota-fix-plan.md and API.md. -
Singa Whitelabel Tenant (2026-03-03): Added Singa (singa.com) as second B2B tenant. Green/black branding, file-upload-only mode, locked to "singa" theme, restricted to @singa.com emails. Setup script generates all branding assets (4K backgrounds, CDG backgrounds, logo) programmatically with Pillow. Also fixed critical Zustand getter bug where computed store properties (branding, features, defaults) became stale static values after any
set()call —Object.assigninvokes getters and copies return values, destroying the getter. AddedTenantLandingPagecomponent for minimal tenant portal experience (logo + sign-in). Added Cloudflare Pages_headersfor proper cache control. See LESSONS-LEARNED.md. -
Song Lookup Autocomplete & Community Version Detection (2026-03-03): Added song catalog integration to the job creation flow. Two features: (1) Track autocomplete on the title field provides canonical artist/title names from MusicBrainz + Spotify data via the karaoke-decide catalog API, improving text consistency for YouTube releases. Autocomplete activates when the artist field has a value; selecting a suggestion overwrites both artist and title with canonical names. (2) Community version detection checks karaokenerds.com for existing high-quality karaoke versions — if found, a dismissible green banner suggests the user check those versions first before spending credits. New backend: 3 endpoints (
/api/catalog/artists,/api/catalog/tracks,/api/catalog/community-check) with per-user rate limiting (20 req/min) and in-memory TTL caching. New services:catalog_proxy_service.py(proxies to karaoke-decide),karaokenerds_service.py(scrapes karaokenerds.com). New frontend components:AutocompleteInput(reusable),CommunityVersionBanner. See API.md. -
Feedback-for-Credits (2026-03-01): Added a feedback-for-credits mechanism available to all users (replaces the old beta-only feedback flow). Users who complete 2+ karaoke videos can submit product feedback (star ratings + text) to earn 1 free credit (total possible free credits: 2 — 1 welcome + 1 feedback). New endpoints:
GET /api/users/feedback/eligibility,POST /api/users/feedback. The/api/users/meresponse includesfeedback_eligible: boolso the frontend can show/hide prompts without an extra API call. Frontend shows a dismissible banner on the dashboard and a "Earn 1 Free Credit" menu item in the user dropdown when eligible. Feedback stored inuser_feedbackFirestore collection with duplicate prevention viahas_submitted_feedbackflag on user model. Welcome email updated with credit-focused messaging and feedback teaser. See API.md. -
Remove Beta Program from Homepage (2026-03-01): Removed beta enrollment from landing page and replaced with prominent "1 Free Credit" messaging. Beta enrollment endpoint preserved for backward compatibility but no longer actively promoted. See PR #430.
-
Fix Event Loop Blocking in Audio Search Endpoints (2026-02-28): Fixed
/api/audio-search/{job_id}/selecttaking 20-36s to respond (should be <1s) and/api/audio-search/searchsometimes taking 25-47s. Root cause: Async endpoints called sync methods that create new event loops vialoop.run_until_complete(), blocking the FastAPI event loop — response body was queued but never flushed. Fix: RED/OPS and Spotify download paths now callFlacfetchClientasync methods directly (matchingYouTubeDownloadServicepattern), generic downloads useasyncio.to_thread(), search endpoint usessearch_async(). Also fixedInsufficientCreditsErroroccasionally returning 500 instead of 402 by usingisinstance()check. Added 14 new tests. See LESSONS-LEARNED.md. -
GDrive Cleanup Robustness + Brand Code Race Fix (2026-02-27): Fixed duplicate brand codes appearing in the public Google Drive share after the daily E2E test (NOMAD-1271 appearing for both the test job "piri - dog" and real song "Hot Mulligan - Pop Shuvit"). Root causes: (1)
if gdrive_files:was falsy for empty dict{}— when GDrive upload silently failed (pre-PR #417), the entire GDrive cleanup block was skipped, (2) Brand code was recycled immediately after Dropbox cleanup succeeded, without confirming GDrive was also clean — a new job could claim the recycled code while old GDrive files still existed. Fixes: Addedfind_files_by_brand_code()fallback toGoogleDriveServicethat searches CDG/MP4/MP4-720p subfolders by brand code prefix when tracked file IDs aren't available.cleanup-distributionendpoint now: (a) uses tracked file IDs (fast path) or falls back to brand_code search, (b) only recycles the brand code after both Dropbox AND GDrive are confirmed clean. Addedscripts/cleanup_test_job_gdrive.pyremediation script. Added 14 new tests (94% coverage on gdrive_service.py). -
CDG Packaging Fail-Fast Validation (2026-02-27): Fixed silent CDG generation failures where jobs completed with
enable_cdg=Truebut no CDG ZIP was produced. Root cause: Jobs 5b6aba25 and 5161b069 hadinstrumental_selection: custombut before commitb671b83f, the orchestrator had nocustomcase — it fell through to"Backing"suffix. The CDG generator then hit aFileNotFoundErrorlooking for the wrong filename, which was silently caught. Fix: (1) Orchestrator now validates CDG/TXT outputs after packaging stage and before encoding — ifenable_cdg=Truebut no CDG ZIP was produced, the pipeline fails fast instead of silently continuing to expensive encoding, (2) Missing LRC file now raises when CDG/TXT is enabled instead of silently skipping, (3) Addedscripts/regenerate_cdg.pyremediation script for affected jobs. Removed "CDG requires additional style configuration" known issue (the nomad theme has complete CDG styles). Added 7 new tests for validation logic. -
GCE Encoding Pipeline Resilience (2026-02-26): Made encoding pipeline resilient to Cloud Run deployment interruptions. A deployment killed a poller mid-encoding, leaving job a03ad39c permanently stuck at
encodingstatus. Three fixes: (1) GCE worker/encodeendpoint now idempotent — returns cached results for completed jobs, allows retry on failure, returns status for in-progress (previously hard 409 reject), (2)encoding_service.pyhandles cached/in_progress responses and falls back to status check on 409, (3) Health service detects jobs stuck inencoding>50 min. See LESSONS-LEARNED.md. -
Private (Non-Published) Tracks (2026-02-24): Added
is_privateflag on jobs for private/non-published track production. Private jobs upload to DropboxTracks-NonPublishedfolder withNOMADNPbrand prefix, skip YouTube and Google Drive entirely, and send completion emails without the YouTube section. Available as a checkbox on all job creation tabs and as an admin toggle (auto-deletes outputs when toggling a completed job to private). See archive/2026-02-24-private-tracks-plan.md. -
GCE Encoding Worker Version Sorting Fix (2026-02-11): Fixed critical regression from PR #384 where alphabetical sorting caused encoder to use v0.99.9 instead of v0.116.1. Root cause:
sorted(wheels)[-1]uses string comparison, not semantic versioning. String comparison puts "0.99.9" > "0.116.1" (because '9' > '1' in second position). Impact: Production encoder was running an extremely old version (v0.99.9 from months ago) instead of the latest (v0.116.1). Fix: Extract version from wheel filename using regex, sort bypackaging.version.Versionobjects instead of strings. Added 10 comprehensive unit tests covering the exact regression case (0.99.9 vs 0.116.1), edge cases (empty list, invalid filenames), and complex version comparisons (major/minor/patch). Production now correctly selects the latest semantic version. See LESSONS-LEARNED.md. -
GCE Encoding Worker Wheel Fix (2026-02-10): Fixed encoding worker crash caused by attempting to install
karaoke_gen-current.whl, which is not a valid PEP 427 wheel filename (missing version and platform tags). Root cause: Theensure_latest_wheel()function downloaded all wheels includingkaraoke_gen-current.whl, then tried to install it. pip rejected it with "Invalid wheel filename (wrong number of parts)", causing encoding to fail and the service to restart, resulting in "Cannot connect" errors from Cloud Run. Fix: Filter out wheels containing '-current' before selecting which wheel to install. Only properly versioned wheels (e.g.,karaoke_gen-0.116.0-py3-none-any.whl) are now considered. Added 5 comprehensive unit tests covering current wheel exclusion, empty results, version sorting, and path-agnostic filtering. Fixes job 749141f8 and prevents future encoding failures. See LESSONS-LEARNED.md. -
Flacfetch Retry Logic & Audio Search Recovery (2026-02-10): Added comprehensive retry logic for flacfetch download failures to handle transient network issues and service restarts. Jobs now retry downloads up to 9 times over ~6 minutes with exponential backoff (10s → 20s → 40s → 60s capped), significantly reducing failures from temporary network hiccups. Key improvements: (1) Retry logic with selective error handling (retries on connection errors and 5xx, immediate fail on 4xx), (2) State persistence - download parameters (source_name, source_id, target_file, download_url) now saved BEFORE download attempt, enabling retry even if download fails, (3) Enhanced retry endpoint -
/api/jobs/{id}/retrynow handles audio search jobs by re-downloading using saved params, (4) User feedback - retry button now shows error toast with actual API error message instead of silent failures. Added 5 comprehensive retry tests covering network errors, 4xx/5xx responses, max retries, and exponential backoff timing. Job 749141f8 investigation revealed the root cause: no source info was saved when flacfetch calls failed, making retry impossible. See LESSONS-LEARNED.md for retry implementation patterns. -
CDG Sync Fix for Countdown-Padded Songs (2026-02-05): Fixed CDG files being 3 seconds out of sync with audio when songs have countdown padding applied. Root cause: Video rendering adds 3s countdown padding to audio AND shifts LRC timestamps by +3s. CDG generation was using these countdown-shifted LRC timestamps but with instrumental audio (which has no countdown padding), causing lyrics to appear 3s late. Fix: CDG generator now receives
lrc_has_countdown_paddingflag; when True, it strips the countdown segment and shifts timestamps back before generating the CDG. Added 11 unit tests and updated orchestrator/packaging pipeline. See LESSONS-LEARNED.md. -
Custom Instrumental Upload Fix (2026-02-26): Fixed critical bug where uploaded custom instrumentals were ignored during encoding — GCE encoding worker and orchestrator had no
"custom"case, falling back to clean instrumental instead. Upload endpoint now converts all formats to FLAC for pipeline consistency. Three fixes: (1) GCE encoding workerfind_filenow searches forcustom_instrumental*.flacwhen selection is"custom", (2) orchestrator config constructs correct(Instrumental Custom).flacpath, (3) upload endpoint converts mp3/m4a/wav/etc. to FLAC before storing. Added 10 new regression tests for GCE resolution and orchestrator paths. See API.md. -
Render Video GCE Routing (2026-03-31): Routed the
render_video_worker(OutputGenerator/ffmpeg lyrics rendering) to the GCE encoding worker instead of running on Cloud Run. Previously, render_video always ran on Cloud Run and OOM'd after memory was right-sized to 2Gi. Now whenUSE_GCE_ENCODING=true, Cloud Run constructs GCS paths from job metadata and delegates to the GCE worker's new/render-videoendpoint. Style assets (fonts, backgrounds) are cached on the GCE worker's disk between jobs to avoid repeated downloads of the same default theme. Local fallback preserved when GCE is disabled. See spec. -
Store Processing Metadata Permanently (2026-03-10): Added
processing_metadatafield to the Job model — a top-level dict where each worker writes immutable provenance data during processing. Stores AudioShake task_id/asset_id (the original ask — these were previously lost after processing), worker timing, separation model names, ffprobe audio properties, correction stats, cache hit/miss data, video encoding details, and auth context at job creation. All metadata writes are non-fatal (wrapped in try/except). Includesupdate_processing_metadatahelper on JobManager using Firestore dot-notation for atomic section updates. 23 tests covering model, helpers, all worker functions, and a SUMMARY_FIELD_PATHS regression guard. See archive/2026-03-10-store-job-processing-metadata.md and LESSONS-LEARNED.md. -
Custom Instrumental Upload (2026-02-25): Added
POST /api/jobs/{job_id}/upload-instrumentalendpoint — was missing from backend, causing 404 in production when users tried to upload custom instrumental files during review. Validates uploaded audio duration matches original (±0.5s tolerance) using ffprobe on GCS signed URLs. Fixed frontend bug where "uploaded" selection type was sent to API instead of "custom" (the only value the backend accepts). Added 22 tests covering upload validation, duration checks, and frontend/backend contract. See API.md. -
Unicode Filename Fix (2026-02-04): Fixed FFmpeg failures (exit code 183) when processing songs with non-ASCII characters in titles. Job
1eab1172with Russian title "я куплю тебе дом" failed because Python'sstr.isalnum()returns True for ALL Unicode alphanumeric characters (Cyrillic, Arabic, Chinese, emoji, etc.), not just ASCII. These Unicode characters in temp subtitle file paths (temp_subtitles_zivert___я_куплю_тебе_дом_1770150285852.ass) caused FFmpeg to fail. Fix: Changed filename sanitization fromc.isalnum()toc.isascii() and c.isalnum()in three locations:video.py:222(generate_video),video.py:282(generate_preview_video), andremote_cli.py:1970(output folders). Added comprehensive test suite covering Cyrillic, Chinese, Arabic, emoji, and accented characters. See LESSONS-LEARNED.md. -
Admin MFY Restart & Override Tools (2026-02-04): Added three new admin endpoints for better Made-For-You order management: (1) Regenerate Screens (
POST /api/admin/jobs/{job_id}/regenerate-screens) - regenerates title/end screens with current artist/title metadata without full reprocessing; (2) Full Restart (POST /api/admin/jobs/{job_id}/restart) - unlike "Reset to Start", this actually triggers workers to run, with option to preserve audio stems for faster restart; (3) Override Audio Source (POST /api/admin/jobs/{job_id}/override-audio-source) - switch from YouTube URL to audio search mode for higher quality audio. Frontend admin toolbar updated with new buttons (Regen Screens, Full Restart, Audio Search) with confirmation dialogs. Use case: Fix MFY orders where customer provided wrong artist/title or low-quality YouTube URL. See API.md. -
Made-For-You YouTube URL Download Fix (2026-01-28): Fixed bug where made-for-you orders with YouTube URLs failed with "Failed to download audio file". Root cause: The webhook handler triggered workers directly without downloading the audio first via
YouTubeDownloadService. Workers requireinput_media_gcs_pathto be set, which only happens after download completes. Fix: Handler now downloads YouTube audio to GCS before triggering workers, matching the pattern infile_upload.pyandaudio_search.py. Added 4 new contract tests with call-order tracking and AST import validation to prevent regression. See LESSONS-LEARNED.md. -
Lyrics Corrections Now Applied to Video (2026-01-26): Fixed critical bug where user's lyrics corrections made during human review were being ignored. Videos were rendering with raw transcription instead of corrected lyrics. Root cause: The
GET /api/review/{job_id}/correction-dataendpoint was returning the originalcorrections.jsoninstead ofcorrections_updated.json(where user edits are saved). The combined review flow would then overwrite the user's edits when completing the review. Fix: The endpoint now checks forcorrections_updated.jsonfirst, matching the behavior ofrender_video_worker.py. Added 35 new tests (backend unit + integration + frontend E2E) to prevent regression. See LESSONS-LEARNED.md. -
Instrumental Review UX Improvements (2026-01-26): Fixed three UX issues in the instrumental review flow: (1) Loading spinner now visible when switching audio tabs (shows on active tab with disabled state on others), (2) Removed large gap under waveform by fixing flexbox layout, (3) Success screen now appears in cloud mode with countdown before redirect (previously only showed in local CLI mode). Also fixed
isLocalMode()detection to correctly distinguish cloud mode (hash-based routing like#/jobId/review) from local CLI mode (path-based routing like/local/review) on localhost, allowing E2E tests to run in cloud mode. Added 17 unit tests forlocal-mode.tsand 12 E2E regression tests for instrumental review UX. Backend fix: Changedaudio_urls['backing']toaudio_urls['backing_vocals']to match frontend expectations. -
Dashboard Performance & UX Improvements (2026-01-25): Improved recent jobs list loading performance and usability. Changes: (1) Reduced default job limit from 20 to 5 for faster initial load (typical users only need to see recent jobs), (2) Added job limit dropdown (5/10/20/All options) for power users who want more history, (3) Implemented priority sorting - blocking jobs (awaiting user action) now appear first, followed by processing, completed, and failed jobs, (4) Made OutputLinks buttons responsive by hiding text labels on mobile (icons-only) to prevent horizontal overflow. Added 14 comprehensive unit tests for priority sorting logic. Frontend-only changes, no API modifications.
-
Combined Lyrics + Instrumental Review (2026-01-24): Merged the two-step human review process (lyrics review, then instrumental selection) into a single unified flow. Users now review lyrics AND select instrumental track in one session, submitting both with a single click. Benefits: Better UX (one interaction vs two), reduced drop-off, single email notification instead of two. Technical changes: Backing vocals analysis moved to screens_worker (runs before review), instrumental options included in
/api/review/{job_id}/correction-dataresponse,POST /api/review/{job_id}/completenow requiresinstrumental_selectionfield. TheAWAITING_INSTRUMENTAL_SELECTIONstate is retained for finalise-only jobs but no longer used in the normal flow. See ARCHITECTURE.md and LESSONS-LEARNED.md. -
Edit Segment Time Input UX Fix (2026-01-24): Fixed cursor jumping issue when typing multi-digit time values in the edit segment modal. Replaced controlled number inputs with debounced
TimeInputcomponent that maintains local state while focused and only syncs to parent after 300ms or on blur. This allows users to smoothly type values like "12.34" without the cursor jumping to the end on each keystroke. Added 15 comprehensive unit tests and E2E regression test to prevent future regressions. See LESSONS-LEARNED.md. -
Cloud Run Jobs for Workers (2026-01-24): Migrated lyrics and audio workers from FastAPI BackgroundTasks to Cloud Run Jobs. This fixes instance termination issues where Cloud Run would shut down containers mid-processing (e.g., job c94cc9d6 was killed after AudioShake completed but before anchor sequence processing finished). Workers now have CLI entry points (
python -m backend.workers.lyrics_worker --job-id abc123) and run as standalone batch containers. Cloud Run Jobs run to completion without HTTP lifecycle concerns, with up to 24-hour execution time. Infrastructure changes via Pulumi definelyrics-transcription-jobandaudio-separation-job. See ARCHITECTURE.md and LESSONS-LEARNED.md. -
Encoding Worker Immutable Deployment (2026-01-22): Implemented robust deployment pattern for GCE encoding worker to eliminate version mismatch issues. Changes: (1) Fixed wheel path
karaoke_gen-current.whleliminates version sorting bugs, (2) Self-updating startup script downloaded from GCS on every service start (no Packer rebuild needed for logic changes), (3) Version manifest with strict CI verification (deployment fails on mismatch). Root cause: startup.sh was baked into Packer image, requiring image rebuild + VM recreation for fixes. New pattern allows CI to update deployment logic without infrastructure changes. See infrastructure/encoding-worker/README.md and LESSONS-LEARNED.md. -
Theme Validation & Default Removal (2026-01-22): BREAKING CHANGE in v0.109.0 - Removed all default style fallback logic to ensure cloud jobs always use complete themes. Style getter functions now raise
ValueErrorif themes are incomplete instead of merging with defaults. Added--list-themesand--validate-themeCLI flags. All cloud jobs guaranteed to use complete nomad theme with no silent fallbacks. See archive/2026-01-22-theme-validation-remove-defaults.md. -
Admin Job Reprocess (2026-01-21, improved 2026-01-24): Added "Reprocess" button to admin job reset toolbar. Allows re-encoding and re-distributing a job with existing settings after deleting outputs. Resets job to
instrumental_selectedstate and automatically triggers video worker. Clears video/encoding/distribution state while preserving instrumental selection and lyrics metadata. Use case: Fix distribution issues, re-upload to YouTube with different settings, or re-run encoding after infrastructure changes. Improvements (2026-01-24): API response now includesworker_triggeredandworker_trigger_errorfields for transparency. Added new/api/admin/jobs/{job_id}/trigger-workerendpoint for manual worker triggering when auto-trigger fails. Frontend shows detailed result dialog after reset with status change, cleared data, and worker trigger status. Added "Trigger" button in admin toolbar for manual worker triggering. See API.md. -
Countdown Audio Sync Fix (2026-01-21, completed 2026-01-24): Fixed bug where songs starting within 3 seconds would have audio desync. PR #328 fixed the writer side:
render_video_workernow updateslyrics_metadata.has_countdown_padding. PR #338 fixed the reader side:video_worker_orchestratornow reads the countdown state and passes it through to GCE encoding, which pads the instrumental audio to match. This was a "Fix Both Sides of Dual Code Paths" bug - see LESSONS-LEARNED.md. Added comprehensive test suite (contract tests, pipeline integration, orchestrator countdown tests) to prevent regression. See archive/2026-01-21-countdown-audio-sync-testing-strategy.md. -
Anchor Sequences Fix (2026-01-21): Fixed anchor sequence calculation when auto-correction is disabled. Previously, disabling auto-correction (
SKIP_CORRECTION=true) also stopped anchor/gap calculation, leaving reviewers without guidance on uncertain lyrics areas. Now anchor sequences are calculated independently, so reviewers always see highlighted matches against reference lyrics regardless of correction settings. -
Auto-Correction Disabled (2026-01-20): Disabled all auto-correction during lyrics transcription (both agentic AI via Gemini and heuristic rule-based handlers). Raw transcription now goes directly to human review without automatic fixes. This change was made because auto-correction was creating more work for reviewers by introducing errors that needed manual correction. Anchor sequences are still calculated to help reviewers identify uncertain areas. The correction feature can be re-enabled by setting
SKIP_CORRECTION=falseenvironment variable. Seebackend/config.pyfor configuration. -
Worker Coordination Fix (2026-01-20): Fixed bug where Cloud Run could shut down prematurely when one parallel worker completed while another was still running. Added
WorkerRegistryto track active workers per job. Workers register at start and unregister in finally block. FastAPI lifespan shutdown handler now waits up to 10 minutes for active workers to complete before allowing container termination. Fixes job failures where lyrics worker was killed mid-processing after audio worker completed. See ARCHITECTURE.md and LESSONS-LEARNED.md. -
Admin Delete Outputs (2026-01-10, UX improved 2026-01-24): Added admin button to delete all distributed outputs (YouTube, Dropbox, Google Drive) for a job while preserving the job record. Use case: fix quality issues by deleting outputs, resetting to
awaiting_review, correcting lyrics, and re-generating. The brand code is freed for reuse when Dropbox folder is deleted. New endpoint:POST /api/admin/jobs/{job_id}/delete-outputs. UI: Delete Outputs button in admin job detail page, "Outputs Deleted" badge when flag is set. UX improvements (2026-01-24): Download buttons (4K, 720p, CDG, TXT, etc.) are now hidden when outputs have been deleted, preventing broken links. Delete confirmation now shows a detailed result dialog with per-service status (YouTube video ID deleted, Dropbox folder path, Google Drive files) instead of a simple toast. See API.md. -
Rate Limiting & Abuse Prevention (2026-01-09): Added per-user rate limiting (5 jobs/day default), system-wide YouTube upload limits (10/day), and beta enrollment abuse prevention. Features: email normalization (Gmail alias detection), disposable domain blocklist (130+ domains), blocked email/IP lists, IP-based enrollment rate limiting (1 per 24h), and admin UI for managing blocklists and user overrides at
/admin/rate-limits. Admins can grant users bypass permissions or custom limits. All limits configurable via environment variables. See API.md. -
Fast Preview Video Option (2026-01-10): Added toggle in lyrics review UI to render preview videos with black background (~10s) instead of theme background image (~30-60s). Default is now black background for faster iteration during review. Users can enable theme background via checkbox when they want to verify the final look. See API.md.
-
Frontend Consolidation (2026-01-09): Consolidated three separate frontends into a single Next.js application. Previously: (1) Main frontend (Next.js + Tailwind), (2) Lyrics Review (React + Vite + MUI, ~14k lines), and (3) Instrumental Review (vanilla HTML/JS, ~1.7k lines) were deployed separately. Now all UIs are unified in the main frontend at
/app/jobs#/{id}/reviewand/app/jobs#/{id}/instrumental. Benefits: single deployment to Cloudflare Pages, no external redirects during user workflows, unified design system (Radix/Tailwind), simpler authentication via job ownership model. See archive/2026-01-09-frontend-consolidation-plan.md and tasks/. -
Admin Login Token for Made-For-You Orders (2026-01-10): Added one-click admin login via email. When a made-for-you order is placed, the admin notification email now includes a direct link with an embedded login token (
?admin_token=TOKEN). Clicking this link auto-authenticates the admin and opens the job directly - no manual login required. Tokens expire after 24 hours. Also added: BCC todone@nomadkaraoke.comon all job completion emails, fixed timezone display (EST→ET for DST correctness). See API.md. -
Made-For-You Audio Selection Flow (2026-01-08, fixed 2026-01-09): Refactored made-for-you order processing to pause at audio selection like regular search jobs. New flow: customer places order → admin receives notification email → admin selects audio source in UI → job processes under admin ownership → on completion, ownership transfers to customer → customer receives delivery email. Added model fields (
made_for_you,customer_email,customer_notes), professional email templates for order confirmation and admin notification, intermediate email suppression (no lyrics/instrumental reminders to customer during processing), and ownership transfer on job completion. Distribution defaults (Dropbox, YouTube, GDrive, brand prefix) are now properly applied from server settings. Bug fix (2026-01-09): Fixed webhook handler that was not settingmade_for_you=True, not using admin ownership, not storingcustomer_email/customer_notes, and not pausing at audio selection. Added 17 comprehensive unit tests for the webhook handler. See API.md and LESSONS-LEARNED.md. -
White-Label B2B Portal Infrastructure (2026-01-08): Added multi-tenant support for white-label karaoke portals. Vocal Star is the first tenant at
vocalstar.nomadkaraoke.com. Features: subdomain-based tenant detection (frontend and backend), GCS-backed tenant config storage, per-tenant feature flags (audio search, file upload, YouTube, theme selection, etc.), dynamic branding (logo, colors, site title via CSS variables), locked themes per tenant, email domain restrictions for auth, and tenant-scoped jobs/users in Firestore. Backend:TenantMiddlewareextracts tenant from X-Tenant-ID header, query param (dev only), or Host subdomain;TenantServiceloads config from GCS with caching. Frontend: Zustand store for tenant state,TenantProviderfor initialization,TenantLogocomponent. Setup script:scripts/setup-vocalstar-tenant.py. See ARCHITECTURE.md and API.md. -
Encoding Worker Infrastructure Upgrade (2026-01-08): Upgraded GCE encoding worker from c4-standard-8 (Intel) to c4d-highcpu-32 (AMD EPYC 9B45 Turin). Benchmarking showed 4.92x faster encoding (135s vs 666s total). Key improvements: With Vocals 4K rendering 6x faster (54s vs 324s), 720p downscale 4.6x faster (11s vs 50s). Reduces total encoding time from ~11 min to ~2.3 min per job. See archive/2026-01-08-performance-investigation.md for full benchmark methodology and results.
-
Theme Enforcement & Encoding Resilience (2026-01-08): Added defense-in-depth theme enforcement - jobs without theme_id are now rejected at creation (JobManager) with a safety net at processing time (screens_worker). Fixed made-for-you webhook to apply default theme. Added retry logic with exponential backoff (3 retries, 2s→4s→8s) to GCE encoding service for transient connection failures during worker restarts. See LESSONS-LEARNED.md.
-
Comprehensive Performance Optimization (2026-01-08): Reduced lyrics processing from 16+ minutes to ~5 minutes through multiple optimizations. Key changes: (1) NLTK cmudict preloading at startup saves 100-150s, (2) Langfuse callback handler preloading saves 200s, (3) Model warmup in
AgenticCorrector.from_model()prevents parallel initialization race, (4) Parallel anchor sequence search (4 workers) reduces n-gram processing from 38s to ~10s. Added/health/preload-statusendpoint for deployment verification. PR #236. See archive/2026-01-08-performance-investigation.md and LESSONS-LEARNED.md. -
UI Simplification & Brand Accessibility (2026-01-08): Simplified the "Create Karaoke Video" form by hiding advanced options (theme settings now admin-only, "Display As" behind toggle). Improved audio search dialog UX by hiding technical tracker names and showing user-friendly availability badges. Fixed brand pink color contrast for accessibility - swapped default/hover values so white text on pink buttons meets WCAG contrast ratio. See BRAND-STYLE-GUIDE.md v1.2.
-
SpaCy Preloading (2026-01-08): Implemented SpaCy model preloading at container startup to eliminate 60+ second delay during agentic correction. The
en_core_web_smmodel is now loaded during FastAPI lifespan startup, andPhraseAnalyzer/SyllablesMatchHandlerreuse the preloaded model. Added timing logs to verify performance. See archive/2026-01-08-spacy-preload-plan.md. -
Thread-Safe LangChain Model Initialization (2026-01-08): Fixed race condition in
LangChainBridgewhere parallel threads could all try to initialize the AI model simultaneously, causing 6+ minute delays. Added double-checked locking withthreading.Lock(). PR #232. See LESSONS-LEARNED.md. -
Brand Consistency & Style Guide (2026-01-07): Implemented unified brand theming across all Nomad Karaoke frontends. Primary color changed from blue to brand pink (#ff7acc). Added comprehensive BRAND-STYLE-GUIDE.md for human and LLM reference. Includes: CSS variables for light/dark mode, MUI theme updates for lyrics transcriber, email template branding, and Playwright visual tests. See BRAND-STYLE-GUIDE.md.
-
Audio Search Display Override (2026-01-06): Added optional "Display As" fields to audio search, allowing users to search for audio by one artist/title (e.g., "Jeremy Kushnier") but display a different artist/title on title screens and filenames (e.g., "Footloose (Broadway Cast)"). Useful for Broadway casts, covers, remixes where tracker metadata differs from desired display. See API.md.
-
Infrastructure Modularization (2026-01-06): Refactored
infrastructure/__main__.pyfrom 2,602 lines to 339 lines (87% reduction). Split into organized modules:modules/for core GCP resources,compute/for VMs and startup scripts,config.pyfor shared constants. Extracted embedded encoding worker Python app tobackend/services/gce_encoding/. Seeinfrastructure/README.mdfor new structure andinfrastructure/docs/PHASE5-PACKER-IMAGE-PLAN.mdfor planned Packer image optimization. -
Email Notification System (2026-01-06): Added automated email notifications for job completion and user action reminders. Features: GCS-backed HTML email templates with fallback defaults, Postmark with CC support, auto-completion emails on job finish, idle reminder emails via Cloud Tasks (5-min delay for blocking states), admin UI buttons to copy message or send email manually. Endpoints:
GET /api/admin/jobs/{id}/completion-message,POST /api/admin/jobs/{id}/send-completion-email. Feature flagENABLE_AUTO_EMAILS(default: false). See API.md and ARCHITECTURE.md. -
GCE Preview Encoding Unified with LocalPreviewEncodingService (2026-01-06): GCE preview encoding now uses the same
LocalPreviewEncodingServicefrom the installed wheel, eliminating 100 lines of duplicated FFmpeg logic. Preview videos are now identical across local CLI, Cloud Run, and GCE worker environments. Benefits include: consistent encoding settings (480x270, 24fps, crf 28), proper FFmpeg filter path escaping for special characters, hardware acceleration (NVENC) when available, and custom font support. See LESSONS-LEARNED.md. -
GCE Encoding Worker Python 3.13 (2026-01-06): Upgraded GCE encoding worker from Debian's Python 3.11 to Python 3.13 built from source. Uses dedicated virtual environment at
/opt/encoding-worker/venv, removing the need for--break-system-packages. Aligns the encoding worker with CI/Cloud Run Python version. -
GCE Encoding Unified with LocalEncodingService (2026-01-06): GCE encoding worker now uses the same
LocalEncodingServiceas the local CLI, eliminating duplicated encoding logic. Output files now have proper names (Artist - Title (Final Karaoke Lossless 4k).mp4instead ofoutput_4k_lossless.mp4) and include title/end screen concatenation. Wheel deployed to GCS; worker installs at job start for hot updates without VM restart. Removed all fallback logic - single code path for consistent output across CLI, Cloud Run, and GCE. See LESSONS-LEARNED.md. -
Cloud Distribution Matches Local CLI (2026-01-06): Fixed cloud distribution (Dropbox uploads) to match local CLI output structure. The
lyrics/folder now includes: proper filenames ((Lyrics Corrections).jsoninstead of(Karaoke).json), all reference lyrics files ((Lyrics Genius).txt,(Lyrics Spotify).txt, etc.),(With Vocals).mkv, andpreviews/subfolder with preview ASS files. See LESSONS-LEARNED.md. -
Agentic Correction Performance (2026-01-05): Optimized agentic AI correction from ~5 minutes to ~55 seconds for 20 gaps (~5-6x speedup). Fixed anti-pattern of creating new model instance per gap (caused repeated 2s warm-up overhead). Now creates model once and processes gaps in parallel using ThreadPoolExecutor (default 5 workers). Configure via
AGENTIC_MAX_PARALLEL_GAPSenv var. See LESSONS-LEARNED.md. -
Worker Logs Subcollection (2026-01-04): Moved
worker_logsfrom embedded array in job documents to Firestore subcollection (jobs/{job_id}/logs). Fixes job failures when logs exceed 1MB (job 501258e1 had 1.26MB of logs). New logs stored with 30-day TTL via Firestore TTL policy. Feature flagUSE_LOG_SUBCOLLECTION=true(default). See LESSONS-LEARNED.md. -
Admin Job Detail Enhancements (2026-01-09): Enhanced admin job detail page with comprehensive job management features. Added: file downloads with signed GCS URLs grouped by category (stems, lyrics, outputs), inline editing of job fields (artist, title, theme, etc.), job reset actions to re-process from specific workflow checkpoints (pending, audio selection, review, instrumental selection), visual timeline with color-coded stages. New endpoints:
GET /api/admin/jobs/{job_id}/files,PATCH /api/admin/jobs/{job_id},POST /api/admin/jobs/{job_id}/reset. See API.md. -
GCE Encoding Response Fixes (2026-01-04): Fixed multiple response format mismatches with GCE encoding worker. The worker returns
output_filesas a list of paths, not a dict with format keys - added conversion logic. Also added defensive type checking for status responses that could be lists. See LESSONS-LEARNED.md. -
Video Worker Orchestrator (2026-01-04): Major refactor to unify video generation pipeline. Created VideoWorkerOrchestrator that coordinates all stages (packaging, encoding, distribution, notifications) regardless of encoding backend (GCE or local). Fixes issue where GCE encoding path bypassed YouTube upload, Discord notifications, and CDG/TXT packaging. Feature flag
USE_NEW_ORCHESTRATOR(default: true) enables rollback. 139 new tests across 6 new service modules. See ARCHITECTURE.md and archive/2026-01-04-video-worker-orchestrator-refactor.md. -
Full Unicode Font Support (2026-01-03): Fixed rendering of musical symbols (♪) and added comprehensive international font support to Docker base image. Installed Noto fonts covering Latin, CJK (Chinese/Japanese/Korean), Arabic, Hebrew, Thai, and other scripts. Changed default karaoke font from Arial to Noto Sans. See LESSONS-LEARNED.md.
-
Flacfetch Cache API Integration (2026-01-03): Enhanced audio search cache management to also clear flacfetch's GCS cache. When admins clear a job's cache, both Firestore (job state) and flacfetch (GCS search results) are cleared simultaneously. Added "Clear All Cache" button and cache stats display to admin UI. New endpoints:
DELETE /api/admin/cache,GET /api/admin/cache/stats. See API.md. -
Audio Search Cache Management (2026-01-03): Added admin UI at
/admin/searchesto view and manage cached audio search results. Admins can clear stale caches (e.g., when flacfetch is updated with new providers) and allow users to re-search. Fixes issue where jobs showed only YouTube results when lossless sources should be available. See API.md and LESSONS-LEARNED.md. -
User Database Separation (2026-01-03): Migrated karaoke-gen users from shared
userscollection to dedicatedgen_userscollection. This separates karaoke-gen user data from karaoke-decide user data (both apps share the same GCP project/Firestore instance). Migration script inscripts/migrate_users_to_gen_users.py. See ARCHITECTURE.md for collection details. -
Admin Dashboard (2026-01-03): Added comprehensive admin dashboard at
/adminwith: stats overview (users, jobs, credits), user management (search, sort, add credits, toggle role, enable/disable), job browser (filter by status/user, view details, delete), and beta program monitoring. Query-param based routing for static export compatibility. See API.md for backend endpoints. -
GCE Encoding Worker (2026-01-03): Added optional high-performance encoding on dedicated GCE VM (C4-standard-8 with Intel Granite Rapids 3.9 GHz). Uses
USE_GCE_ENCODING=trueenv var. Falls back to Cloud Run if unavailable. See ARCHITECTURE.md. -
Runner Disk Space Auto-Cleanup (2026-01-03): Fixed CI failures caused by self-hosted runners filling up with Docker images. Changed cleanup from daily/7-day-old to hourly/threshold-based (70%). Added pre-job disk checks to all self-hosted CI jobs. See LESSONS-LEARNED.md
-
E2E Happy Path Test (2026-01-02): Added comprehensive E2E test that validates the complete karaoke generation journey using ONLY browser UI interactions (no API shortcuts). Test covers all 12 steps from landing page through cleanup. Runs daily via GitHub Actions and takes ~38 minutes. See E2E-HAPPY-PATH-TEST-SPEC.md
-
LyricsTranscriber Cache Persistence (2026-01-02): Added GCS-backed cache for LyricsTranscriber to avoid redundant API calls. AudioShake transcription and lyrics provider responses (Genius, Spotify, LRCLib, Musixmatch) are now cached to
gs://karaoke-gen-storage/lyrics-transcriber-cache/. Cache is synced before/after each job, persisting across Cloud Run instances. Significantly reduces API costs and speeds up repeated processing of same songs. -
Worker Timeout Fixes (2026-01-01): Fixed 3 timeout issues blocking job completion: (1) Lyrics transcription timeout increased to 20 min (PR #153), (2) Cloud Tasks dispatch_deadline added for audio worker - default 10 min was killing Modal API calls (PR #154), (3) Enabled Cloud Run Jobs for video encoding - 1-hour timeout vs 30-min Cloud Run service limit (PR #155). Full E2E pipeline now verified working. See archive/2026-01-01-worker-timeout-fixes.md
-
Agentic Correction Timeout (2025-12-31): Added 3-minute configurable timeout for agentic AI lyrics correction. Prevents stuck jobs when songs have many gaps (74+ gaps could take 30+ minutes). On timeout, skips correction and proceeds to human review with raw transcription. See archive/2025-12-31-agentic-timeout-implementation.md
-
Job Failure Fixes (2025-12-31): Fixed 4 critical issues causing job failures: (1) user_email not set on jobs - users couldn't see their jobs, (2) YouTube URL race condition - lyrics worker started before audio downloaded, (3) Audio search cache not persisting across Cloud Run instances, (4) Jobs stuck in downloading state - added 10min transcription timeout. See archive/2024-12-31-job-failure-investigation.md
-
Vertex AI Auth Fix (2025-12-31): Fixed
ChatGoogleGenerativeAIto use Vertex AI backend with service account auth on Cloud Run. PR #145 broke cloud deployments by requiringGOOGLE_API_KEY. Solution: passprojectparameter to trigger Vertex AI backend with ADC. See archive/2025-12-31-vertex-ai-auth-fix.md -
LangChain Provider Migration (2025-12-30): Migrated from
langchain-google-vertexai(gRPC) tolangchain-google-genai(REST) to fix silent hangs during model initialization. Added initialization and warm-up timeouts using ThreadPoolExecutor for fail-fast behavior. See archive/2025-12-30-langchain-vertexai-to-genai-migration.md -
LangFuse Prompt Management (2025-12-30): Replaced hardcoded LLM prompts with LangFuse-managed prompts for dynamic iteration without redeploying. Added migration script, LangFuse Vertex AI service account for GCP credits. See archive/2025-12-30-langfuse-prompt-management.md
-
Gemini 3 Flash Agentic Correction Fix (2025-12-30): Fixed agentic correction hanging due to Gemini 3 requiring
globallocation instead of regional endpoints, and multimodal response format handling. Added local test script for faster iteration. See archive/2025-12-30-gemini3-agentic-correction-fix.md -
Cold Start UX (2025-12-30): Frontend now shows friendly "Warming up the generator..." messages when backend is slow to respond (cold start after deployment or inactivity). Progressive messages appear after 5s and 15s to reassure users during 30s+ cold starts.
-
Cloud Output Structure Fix (2025-12-30): Fixed remote CLI (
karaoke-gen-remote) to produce identical output as local CLI: YouTube uploads now work, Dropbox includes stems/ and lyrics/ subfolders, instrumental filenames use actual model names. See archive/2025-12-30-cloud-output-structure-fix.md -
Auth Session Persistence Fix (2025-12-30): Fixed two bugs causing 401 errors despite valid sessions: timezone-aware datetime comparisons in backend, and stale token caching in frontend. Firestore indexes now tracked in Pulumi. See archive/2025-12-30-auth-session-persistence-fix.md
-
E2E Test Suite Consolidated (2025-12-29): Tests reorganized into
production/(real API) andregression/(mocked API, CI-safe). See archive/2025-12-29-e2e-test-consolidation-plan.md
Quick Links
| Doc | Purpose |
|---|---|
| PRODUCT-VISION.md | Product goals, audience, business model |
| ARCHITECTURE.md | System design, data flow, integrations |
| DEVELOPMENT.md | Local setup, testing, deployment |
| TESTING.md | Testing standards, CI requirements, Playwright |
| API.md | Backend endpoint reference |
| BRAND-STYLE-GUIDE.md | Brand colors, typography, UI patterns |
| LESSONS-LEARNED.md | Key insights for future agents |
| STRIPE-SETUP.md | Payment integration setup |
| MOBILE-UX-BEST-PRACTICES.md | Mobile/responsive design guidelines |
| archive/ | Historical documentation |
Workflow Summary
Upload Audio → Parallel Processing → Human Review → Video Generation → Download
│ │
(Cloud Run GPU + AudioShake) (LyricsTranscriber UI)
For AI Agents
Start with CLAUDE.md in the repo root for essential rules, then reference these docs as needed. Keep this README current when project state changes.