## Why Claude sessions couldn't read secrets such as the backend admin API token. ADC is `claude-readonly@`, which had `secretmanager.viewer` (metadata only). The gcloud CLI's `admin@nomadkaraoke.com` login lapses and needs an interactive re-login, so every documented `gcloud secrets versions access …` command failed in sessions. For example, retrying jobs ae995697/e2f60e2d on 2026-11-19 had to go through a browser session. Andrew asked for `claude-readonly@` to be able to read all secrets. ## Change - Project-level `roles/secretmanager.secretAccessor` for `claude-readonly@`. - The token commands in CLAUDE.md, TESTING.md, TROUBLESHOOTING.md and STORAGE-RETENTION.md now pass `--account=claude-readonly@nomadkaraoke.iam.gserviceaccount.com`, so they work without a fresh gcloud login. ## Trade-off The SA's key is now secret-equivalent: whoever holds it can read every secret and act as backend admin through the API. The key stays on Andrew's Mac only. ## Applied `pulumi up --target <claude-readonly-secretmanager-accessor>`: **+1 created, 290 unchanged**, 2026-11-19. Verified: `claude-readonly@` reads `admin-tokens` (and the API accepts the token) and other secrets. The full `pulumi preview` also shows unrelated drift, not touched here: runner_manager function source (last applied from the `karaoke-gen-runner-scan-retry` worktree) and its scheduler retry config. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVVVVLQSWvtZqBGxRFb --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
11 KiB
Karaoke-Gen AI Assistant Guidelines
Project Overview
Karaoke video generation platform - CLI tool and web service that creates professional karaoke videos with synchronized lyrics.
- Production: https://gen.nomadkaraoke.com (frontend), https://api.nomadkaraoke.com (backend)
- CLI:
karaoke-gen(local),karaoke-gen-remote(cloud) - Repo: https://github.com/nomadkaraoke/karaoke-gen
Quick Reference
| What | Where |
|---|---|
| Product vision & goals | docs/PRODUCT-VISION.md |
| Current status | docs/README.md |
| Architecture | docs/ARCHITECTURE.md |
| Dev setup & testing | docs/DEVELOPMENT.md |
| Testing & code quality | docs/TESTING.md |
| API reference | docs/API.md |
| Past learnings | docs/LESSONS-LEARNED.md |
| Operational runbooks | docs/TROUBLESHOOTING.md |
| GDrive validator & gaps | docs/GDRIVE-VALIDATOR.md |
| Product communication | docs/PRODUCT-COMMUNICATION-GUIDE.md |
| Brand style guide | docs/BRAND-STYLE-GUIDE.md |
| Making medleys (multi-song karaoke) | docs/MEDLEYS.md + scripts/medley/make_medley.py |
Tech Stack
- Backend: FastAPI on Cloud Run, Firestore, GCS, Secret Manager
- Frontend: Next.js on Cloudflare Pages
- Processing: Cloud Run GPU (L4 audio separation), AudioShake/Whisper (transcription)
- Infra: Pulumi IaC, GitHub Actions CI/CD
Essential Rules
Git Workflow
- Never commit directly to
main- use/new-worktree <description>to start - Follow global workflow - see
~/.claude/CLAUDE.mdfor command sequence - Create PR with summary, changes, testing info
- Merge only after CI passes
Testing (Required)
Before planning any implementation work, read docs/TESTING.md and ensure your plan includes a thorough testing strategy that follows the guidance there. This includes:
- Which test types are needed (unit, integration, E2E)
- Where tests should be placed
- What mocking approach to use
- Coverage expectations
- Production E2E tests for user-facing features - see below
Critical: If your plan includes "Manual Testing" steps, convert them to automated Playwright E2E tests. Production E2E tests (in frontend/e2e/production/) are valuable even if only run once after deployment - they codify expected behavior and verify the deployed code works. See docs/TESTING.md for details.
Before committing, run all tests:
make test 2>&1 | tail -n 500
This single command:
- Installs dependencies automatically (backend + frontend)
- Runs all backend tests (unit, integration, emulator)
- Runs all frontend tests (Jest unit + Playwright E2E)
- Takes ~5-10 minutes
All tests must pass. Don't dismiss failures as "pre-existing" - investigate and fix them.
Testing in Production (for Agents)
When asked to "test it yourself in prod" or verify a fix in production:
# Get admin token. claude-readonly@ can read all secrets (Pulumi:
# infrastructure/modules/iam/claude_readonly_sa.py), so no gcloud login is needed.
export KARAOKE_ADMIN_TOKEN=$(gcloud secrets versions access latest --secret=admin-tokens --project=nomadkaraoke --account=claude-readonly@nomadkaraoke.iam.gserviceaccount.com | cut -d',' -f1)
# Option 1: Use the debug template directly
cd frontend && node e2e/helpers/debug-prod-template.mjs
# Option 2: Copy template for customization (gitignored)
cp frontend/e2e/helpers/debug-prod-template.mjs frontend/test-my-issue.local.mjs
# Edit the script, then run it
node test-my-issue.local.mjs
# Option 3: Run existing production E2E tests
KARAOKE_ADMIN_TOKEN=$KARAOKE_ADMIN_TOKEN npx playwright test e2e/production/admin-dashboard.spec.ts
Files matching test-*.local.* and debug-*.local.* are gitignored - you can hardcode tokens in them safely.
See docs/TESTING.md § "Ad-Hoc Production Debugging" for full details.
Version Bumping
- Bump
tool.poetry.versioninpyproject.tomlfor code changes - Skip for docs-only changes
Infrastructure
- All GCP changes via Pulumi PRs - but CI does not run Pulumi (see the note in
.github/workflows/ci.yml): a mergedinfrastructure/change is not live until someone runspulumi uplocally. Claude sessions have read-only ADC, so they need Andrew's/gcp-admingrant to apply. Ask for it early, and vet with a targetedpulumi preview --difffirst infrastructure/is the shared platform IaC home: the GCP project (infrastructure/) and the Nomad Karaoke Cloudflare account (infrastructure/cloudflare/, own Pulumi project + stack, see its README). Never hand-edit Cloudflare config that stack manages; app repos keep only app-specific stacks- Run
pulumi uplocally BEFORE merging infra PRs - CI only validates; it never applies. If you merge first, main and the live state drift until someone applies - Never modify GCP resources directly via console or
gcloudCLI gcloudCLI for reading/debugging only (e.g., checking logs, SSH to VMs)- Stop and notify user on auth issues
- GCS ops on macOS: never use
gsutil -m— its fork-based multiprocessing segfaults on macOS (crash-report popups + the parent process hangs forever). Usegcloud storageinstead (gcloud storage cp/rm/rsync …) — no fork, and faster. Plaingsutil(no-m) is safe but slower.~/.botopinsparallel_process_count=1as a safety net; don't remove it.
Internationalization (i18n)
- All user-facing strings live in
frontend/messages/{locale}.json(33 locales), NOT in components - Backend strings live in
backend/translations/{locale}.json(currently 3 locales: en, es, de) - Components use
useTranslations('namespace')from next-intl - Pages are under
frontend/app/[locale]/— locale-aware routing (customer-facing:/,/app/*,/order/*,/auth/*,/payment/*,/r/*) - Non-locale routes at those same paths (e.g.
frontend/app/app/jobs/[[...slug]]/page.tsx) are<LocaleRedirect />shims that redirect into[locale]— they are NOT English-only surfaces - The
/app/jobs/#/{id}/review,/instrumental, and/audio-editreview UIs are fully multilingual (components undercomponents/lyrics-review/,components/instrumental-review/,components/audio-editor/useuseTranslationsheavily) - Only
/admin/*is English-only — it has no[locale]/admincounterpart and inheritsDefaultIntlProviderfrom the root layout - Internal links use
Linkfrom@/i18n/routing(locale-aware) - After adding/changing English strings:
python frontend/scripts/translate.py --messages-dir frontend/messages --target all - Translation uses Gemini 3.8 Flash via the Gemini Developer API (key:
GEMINI_API_KEY, else Secret Managergemini-api-keyvia your gcloud login —frontend/scripts/gemini_client.py) with GCS cache for efficiency. Vertex AI is disabled in the GCP project — never usevertexai=True; backend code usesbackend/services/gemini_client.get_genai_client(). - CI validates all locale files have matching keys — PR will fail if translations are missing
- Pre-commit hook auto-translates when en.json changes (enable:
git config core.hooksPath .githooks) - Don't hardcode user-facing strings — add to
messages/en.jsonand uset('key')
API Authentication (for Agents)
When you need to call backend APIs programmatically:
Internal Worker Endpoints
Use the X-Admin-Token header with the token from Secret Manager:
# Trigger video worker
curl -X POST "https://api.nomadkaraoke.com/api/internal/workers/video" \
-H "X-Admin-Token: $(gcloud secrets versions access latest --secret=admin-tokens --project=nomadkaraoke --account=claude-readonly@nomadkaraoke.iam.gserviceaccount.com)" \
-H "Content-Type: application/json" \
-d '{"job_id": "YOUR_JOB_ID"}'
# Other internal endpoints follow the same pattern
Firestore Direct Access
# Set the correct project
export GOOGLE_CLOUD_PROJECT=nomadkaraoke
# Use Python with google-cloud-firestore
python3 << 'EOF'
import os
os.environ['GOOGLE_CLOUD_PROJECT'] = 'nomadkaraoke'
from google.cloud import firestore
db = firestore.Client(project='nomadkaraoke')
# Query/update documents...
EOF
GCE Encoding Worker
Since 2026-11 the only GCE encoding VM is the stopped on-demand fallback
encoding-worker-fallback-c2df (c2d-highcpu-16, us-central1-f); the c4d Spot pair
(encoding-worker-a/-b) was retired and deleted, and primary_vm/secondary_vm in
config/encoding-worker are empty. The serving worker is active_override_vm (normally the
AWS box below). Idle VMs stop after 5 min. CI's "blue-green" deploy now only runs when worker
code changed (infrastructure/encoding-worker/worker_code_paths.txt, diffed against
validated_version): it boots c2df, runs a test encode, records validated_version and stops
it again, leaving the AWS override serving. Force it via Actions → CI → Run workflow →
"Force a blue-green redeploy of the GCE encoding worker".
# Check health (includes wheel_version) on the GCE fallback.
gcloud compute ssh encoding-worker-fallback-c2df --zone=us-central1-f --project=nomadkaraoke \
--command="curl -s http://localhost:8080/health"
# Restart the service (pick up new worker-side code after a deploy)
gcloud compute ssh encoding-worker-fallback-c2df --zone=us-central1-f --project=nomadkaraoke \
--command="sudo systemctl restart encoding-worker"
For job stuck at encoding status, see docs/TROUBLESHOOTING.md.
AWS encoding worker (aws-encoding-worker-1, EC2 c7a.4xlarge on-demand, us-east-2) is a pool
candidate too (provider: aws in encoding-worker-fallback-vms): last resort by default,
preferred when Firestore config/encoding-worker.speed_rank_overrides = {"c7a": 5}. Not
blue-green deployed, idle-stops itself, shell via SSM, logs are on the box (journald). See
infrastructure/aws/README.md.
What Doesn't Work
gcloud auth print-identity-token- Wrong token type for internal endpoints- Cloud Tasks without proper service account - Gets 401 from Cloud Run
Documentation Maintenance
Structure
docs/
├── README.md # Current status + navigation (UPDATE THIS)
├── ARCHITECTURE.md # System design
├── DEVELOPMENT.md # Dev setup, testing, deployment
├── API.md # Backend API reference
├── LESSONS-LEARNED.md # Key insights for future agents
└── archive/ # Historical docs (YYYY-MM-DD-topic.md)
Rules
- Before merging PRs: Run
/docs-reviewto check if docs need updating - Periodically: Run
/docs-maintainto verify docs are organized - For significant work: Create
docs/archive/YYYY-MM-DD-topic.md - Add learnings: Update
docs/LESSONS-LEARNED.mdwith insights - Keep status current: Update
docs/README.mdwhen project state changes
What Goes Where
- README.md: Current status, known issues, quick links
- ARCHITECTURE.md: System design, data flow, tech decisions
- DEVELOPMENT.md: Setup, testing, deployment, CI/CD
- API.md: Endpoints, request/response formats
- LESSONS-LEARNED.md: What worked, what didn't, gotchas
- archive/: Completed work, session summaries, old plans
Related Projects
flacfetch
Location: /Users/andrew/Projects/flacfetch
Workflow: Make changes on main, bump version, push, wait for CI, then poetry update flacfetch in karaoke-gen.
Pre-PR Checklist
Follow global workflow (~/.claude/CLAUDE.md):
- Tests pass:
/test(ormake test/npm run test:all) - Version bumped (if code changed)
- Docs checked:
/docs-review - Code reviewed:
/review(CodeRabbit CLI) - PR created:
/pr(adds @coderabbitai ignore)