Compare commits

...
Author SHA1 Message Date
Andrew Beveridge ☄️ 1bc6dbe1a2 Merge pull request #100 from nomadkaraoke/fix/upload-endpoint-theme-support
Fix: Add theme support to /api/jobs/upload endpoint
2025-12-28 13:27:23 -05:00
Andrew Beveridge ☄️ 71b33a6b93 Merge pull request #101 from nomadkaraoke/feature/multiple-runners
feat: scale to 5 self-hosted GitHub runners for parallel CI jobs
2025-12-28 13:26:04 -05:00
Andrew BeveridgeandClaude Opus 4.5 c3e1edfcb4 feat: scale to 5 self-hosted GitHub runners for parallel CI jobs
- Replace single github-runner VM with 5 instances (github-runner-1 through 5)
- Enables parallel job execution across multiple PRs
- Each runner: e2-standard-4, 200GB SSD, same tooling
- Update README with new runner count and troubleshooting commands

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-28 13:11:49 -05:00
Andrew BeveridgeandClaude Opus 4.5 44c591a629 Fix: Add theme support to /api/jobs/upload endpoint
The /api/jobs/upload endpoint was missing theme_id and color_overrides
parameters, causing themes selected in the frontend to not be passed
through when creating jobs via file upload. This resulted in preview
videos having black backgrounds instead of themed backgrounds.

Changes:
- Add theme_id and color_overrides Form parameters to upload endpoint
- Change enable_cdg/enable_txt to Optional[bool] for smart defaults
- Add _resolve_cdg_txt_defaults() to enable CDG/TXT when theme is set
- Add _prepare_theme_for_job() call when theme_id is set (no custom style files)
- Pass theme_id and color_overrides to JobCreate
- Use theme's YouTube description template if no custom one provided

Root cause: The frontend was sending theme_id in the form data, but the
backend endpoint was not accepting this parameter, so it was silently
ignored. PR #72 fixed this for audio_search.py but this endpoint was missed.

Fixes issue where job c9516773 (file upload) had black background while
job 47475700 (song search) correctly showed themed background.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-28 12:32:18 -05:00
Andrew Beveridge ☄️ 2203595e51 Merge pull request #99 from nomadkaraoke/feature/coderabbit-review-command
feat: add /coderabbit-review slash command
2025-12-28 12:17:40 -05:00
Andrew BeveridgeandClaude Opus 4.5 635c4e7ee6 feat: add /coderabbit-review slash command
Adds a Claude Code slash command that reviews and addresses CodeRabbit
comments on the current PR:
- Fetches both review threads and nitpick comments
- Analyzes each comment and decides if action is needed
- Makes appropriate code fixes
- Commits/pushes changes and resolves threads

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-28 12:16:28 -05:00
Andrew Beveridge ☄️ 1c5711cfa5 Merge pull request #98 from nomadkaraoke/docs/runner-setup
docs: add self-hosted runner documentation
2025-12-28 12:06:31 -05:00
Andrew Beveridge ☄️ 08d7db9943 Merge pull request #97 from nomadkaraoke/fix/runner-startup-script
fix: make runner startup script reproducible
2025-12-28 12:06:24 -05:00
Andrew BeveridgeandClaude Opus 4.5 d2c8c24eee fix: make runner startup script reproducible
Fixes issues discovered during initial runner setup:

- Use `gpg --batch` for all GPG commands to avoid tty errors
- Install Python 3.13 via pyenv (not broken PPA/source build)
- Set up Python 3.13 in tool cache for setup-python action
- Use Temurin directly for Java 21 (skip broken Debian repos)

These changes ensure the runner VM can be recreated from scratch
and work correctly with GitHub Actions setup-python.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-28 12:03:10 -05:00
6 changed files with 425 additions and 82 deletions

No files matched your search

+109
View File
@@ -0,0 +1,109 @@
---
description: Review and address CodeRabbit comments on current PR
allowed-tools: Read, Glob, Grep, Edit, Write, Bash, mcp__github__pull_request_read, mcp__github__get_me, mcp__github__list_pull_requests, mcp__github__add_issue_comment
---
# CodeRabbit Comment Review
You are reviewing the current PR to address all CodeRabbit automated review comments.
## Your Task
### 1. Identify the Current PR
First, determine the current branch and find the associated PR:
```bash
git branch --show-current
```
Then use GitHub tools to find the PR for this branch in the `nomadkaraoke/karaoke-gen` repository.
### 2. Fetch CodeRabbit Comments
Get all review comments and regular comments on the PR:
**Review Comments (Potential Issues)**
- Use `pull_request_read` with `method: "get_review_comments"` to get all review comment threads
- Look for comments from the `coderabbitai` bot user
- These are typically marked as "Potential issue" and MUST be resolved before merge
**PR Comments (Including Nitpicks)**
- Use `pull_request_read` with `method: "get_comments"` to get regular PR comments
- CodeRabbit posts a summary comment that includes "Nitpick comments" section
- These should also be reviewed and considered for action
### 3. Analyze Each Comment
For each CodeRabbit comment:
1. Read the comment carefully to understand the concern
2. Look at the referenced code location
3. Decide if a code change is warranted:
- **Make changes** for: valid bugs, security issues, clear improvements, style violations
- **Skip changes** for: subjective preferences, false positives, already addressed, not applicable
4. If skipping, note the reason
### 4. Make Code Changes
For each comment requiring action:
1. Navigate to the file and line mentioned
2. Understand the surrounding context
3. Make the minimal change needed to address the concern
4. Verify the change doesn't break anything obvious
### 5. Commit and Push
After making all changes:
```bash
# Stage all changes
git add -A
# Commit with descriptive message
git commit -m "fix: address CodeRabbit review comments
- [list each fix briefly]
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>"
# Push to the PR branch
git push
```
### 6. Resolve Threads
After pushing:
- Reply to each CodeRabbit review thread indicating what was done
- If a comment was addressed, reply with what was changed
- If a comment was intentionally skipped, explain why
- This helps mark threads as resolved so the PR can be merged
## Output Format
Provide a summary report:
```
## CodeRabbit Review Summary
### Comments Addressed
- [ ] File:line - Brief description of issue → What was changed
### Comments Skipped (with reason)
- [ ] File:line - Issue → Reason for skipping
### Actions Taken
- Committed: [commit hash]
- Pushed to: [branch name]
- Threads replied to: [count]
### Status
Ready for merge: Yes/No (if No, explain why)
```
## Guidelines
- Focus on code quality improvements, not just making CodeRabbit happy
- Don't blindly accept every suggestion - use judgment
- Group related fixes into logical commits if needed
- Keep changes minimal and focused on the specific issues raised
- If a suggestion would require significant refactoring, note it for future work instead
+74 -14
View File
@@ -341,9 +341,12 @@ async def upload_and_create_job(
style_cdg_instrumental_background: Optional[UploadFile] = File(None, description="CDG instrumental background"),
style_cdg_title_background: Optional[UploadFile] = File(None, description="CDG title screen background"),
style_cdg_outro_background: Optional[UploadFile] = File(None, description="CDG outro screen background"),
# Processing options (CDG/TXT require style config, disabled by default)
enable_cdg: bool = Form(False, description="Generate CDG+MP3 package (requires style config)"),
enable_txt: bool = Form(False, description="Generate TXT+MP3 package (requires style config)"),
# Theme configuration (use pre-made theme from GCS)
theme_id: Optional[str] = Form(None, description="Theme ID to use (e.g., 'nomad', 'default'). If set, CDG/TXT are enabled by default."),
color_overrides: Optional[str] = Form(None, description="JSON-encoded color overrides: artist_color, title_color, sung_lyrics_color, unsung_lyrics_color (hex #RRGGBB)"),
# Processing options (CDG/TXT require style config or theme, disabled by default unless theme is set)
enable_cdg: Optional[bool] = Form(None, description="Generate CDG+MP3 package. Default: True if theme_id set, False otherwise"),
enable_txt: Optional[bool] = Form(None, description="Generate TXT+MP3 package. Default: True if theme_id set, False otherwise"),
# Finalisation options
brand_prefix: Optional[str] = Form(None, description="Brand code prefix (e.g., NOMAD)"),
enable_youtube_upload: bool = Form(False, description="Upload to YouTube"),
@@ -477,7 +480,35 @@ async def upload_and_create_job(
# Extract request metadata for tracking and filtering
request_metadata = extract_request_metadata(request, created_from="upload")
# Parse color_overrides from JSON if provided
parsed_color_overrides: Dict[str, str] = {}
if color_overrides:
try:
parsed_color_overrides = json.loads(color_overrides)
except json.JSONDecodeError as e:
raise HTTPException(
status_code=400,
detail=f"Invalid color_overrides JSON: {e}"
)
# Resolve CDG/TXT defaults based on theme
resolved_cdg, resolved_txt = _resolve_cdg_txt_defaults(
theme_id, enable_cdg, enable_txt
)
# Check if any custom style files are being uploaded (overrides theme)
has_custom_style_files = any([
style_params,
style_intro_background,
style_karaoke_background,
style_end_background,
style_font,
style_cdg_instrumental_background,
style_cdg_title_background,
style_cdg_outro_background,
])
# Parse comma-separated model lists into arrays
parsed_backing_vocals_models = None
if backing_vocals_models:
@@ -492,8 +523,10 @@ async def upload_and_create_job(
artist=artist,
title=title,
filename=file.filename,
enable_cdg=enable_cdg,
enable_txt=enable_txt,
theme_id=theme_id,
color_overrides=parsed_color_overrides,
enable_cdg=resolved_cdg,
enable_txt=resolved_txt,
brand_prefix=brand_prefix,
enable_youtube_upload=enable_youtube_upload,
youtube_description=youtube_description,
@@ -521,9 +554,27 @@ async def upload_and_create_job(
# Record job creation metric
metrics.record_job_created(job_id, source="upload")
logger.info(f"Created job {job_id} for {artist} - {title}")
# If theme is set and no custom style files are being uploaded, prepare theme style now
# This copies the theme's style_params.json to the job folder so LyricsTranscriber
# can access the style configuration for preview videos
theme_style_params_path = None
theme_style_assets = {}
theme_youtube_desc = None
if theme_id and not has_custom_style_files:
try:
theme_style_params_path, theme_style_assets, theme_youtube_desc = _prepare_theme_for_job(
job_id, theme_id, parsed_color_overrides or None
)
logger.info(f"Applied theme '{theme_id}' to job {job_id}")
except HTTPException:
raise # Re-raise validation errors (e.g., theme not found)
except Exception as e:
logger.warning(f"Failed to prepare theme '{theme_id}' for job {job_id}: {e}")
# Continue without theme - job can still be processed with defaults
# Upload main audio file to GCS
audio_gcs_path = f"uploads/{job_id}/audio/{file.filename}"
logger.info(f"Uploading audio to GCS: {audio_gcs_path}")
@@ -592,21 +643,30 @@ async def upload_and_create_job(
update_data = {
'input_media_gcs_path': audio_gcs_path,
'filename': file.filename,
'enable_cdg': enable_cdg,
'enable_txt': enable_txt,
'enable_cdg': resolved_cdg,
'enable_txt': resolved_txt,
}
# Handle style assets - either from custom uploads or from theme
if style_assets:
# Custom style files uploaded
update_data['style_assets'] = style_assets
if 'style_params' in style_assets:
update_data['style_params_gcs_path'] = style_assets['style_params']
elif theme_style_assets:
# Theme style assets (no custom uploads)
update_data['style_assets'] = theme_style_assets
if theme_style_params_path:
update_data['style_params_gcs_path'] = theme_style_params_path
if brand_prefix:
update_data['brand_prefix'] = brand_prefix
if effective_discord_webhook_url:
update_data['discord_webhook_url'] = effective_discord_webhook_url
if youtube_description:
update_data['youtube_description_template'] = youtube_description
# Use theme YouTube description if no custom one provided
effective_youtube_description = youtube_description or theme_youtube_desc
if effective_youtube_description:
update_data['youtube_description_template'] = effective_youtube_description
# Native API distribution (use effective values which include defaults)
if effective_dropbox_path:
+128
View File
@@ -1606,5 +1606,133 @@ class TestIsUrlFunction:
assert is_url("") is False
class TestUploadEndpointThemeSupport:
"""Test that /jobs/upload endpoint supports theme configuration.
CRITICAL: These tests verify that the /api/jobs/upload endpoint correctly handles
theme_id and color_overrides parameters, ensuring preview videos have themed
backgrounds instead of black backgrounds.
This addresses a bug where:
- The frontend sends theme_id and color_overrides when uploading files
- But the backend /api/jobs/upload endpoint was ignoring these parameters
- Result: Jobs created via file upload had black backgrounds instead of themed ones
"""
def test_upload_endpoint_accepts_theme_id_parameter(self):
"""Verify the upload endpoint has theme_id as a form parameter.
CRITICAL: The frontend sends theme_id when uploading files with a theme.
If this parameter is missing from the endpoint, the theme is silently ignored
and preview videos will have black backgrounds instead of themed ones.
"""
from backend.api.routes import file_upload as file_upload_module
with open(file_upload_module.__file__, 'r') as f:
source_code = f.read()
has_theme_id_param = 'theme_id: Optional[str] = Form(' in source_code
assert has_theme_id_param, (
"file_upload.py /jobs/upload endpoint does not have theme_id as a Form parameter. "
"The frontend sends theme_id when uploading files with a theme, but the backend "
"ignores it. Add: theme_id: Optional[str] = Form(None, description='Theme ID...')"
)
def test_upload_endpoint_accepts_color_overrides_parameter(self):
"""Verify the upload endpoint has color_overrides as a form parameter."""
from backend.api.routes import file_upload as file_upload_module
with open(file_upload_module.__file__, 'r') as f:
source_code = f.read()
has_color_overrides_param = 'color_overrides: Optional[str] = Form(' in source_code
assert has_color_overrides_param, (
"file_upload.py /jobs/upload endpoint does not have color_overrides as a Form parameter. "
"The frontend sends color_overrides when customizing theme colors."
)
def test_upload_endpoint_calls_prepare_theme_for_job(self):
"""Verify the upload endpoint calls _prepare_theme_for_job when theme_id is set.
CRITICAL: When a job is created via the upload endpoint with a theme_id
(and no custom style files), the code must call _prepare_theme_for_job() to set:
1. style_params_gcs_path (pointing to the copied style_params.json)
2. style_assets (populated with asset mappings)
Without this, LyricsTranscriber won't have access to the theme's styles
and preview videos will have black backgrounds instead of themed ones.
"""
from backend.api.routes import file_upload as file_upload_module
with open(file_upload_module.__file__, 'r') as f:
source_code = f.read()
# The function should be called in upload_and_create_job
has_theme_prep_call = '_prepare_theme_for_job(' in source_code
assert has_theme_prep_call, (
"file_upload.py does not call _prepare_theme_for_job(). "
"When theme_id is provided to /jobs/upload without custom style files, "
"the endpoint MUST call _prepare_theme_for_job() to copy the theme's "
"style_params.json to the job folder."
)
def test_upload_endpoint_uses_resolve_cdg_txt_defaults(self):
"""Verify the upload endpoint uses _resolve_cdg_txt_defaults for theme-based defaults."""
from backend.api.routes import file_upload as file_upload_module
with open(file_upload_module.__file__, 'r') as f:
source_code = f.read()
has_resolve_call = '_resolve_cdg_txt_defaults(' in source_code
assert has_resolve_call, (
"file_upload.py does not call _resolve_cdg_txt_defaults(). "
"When theme_id is set, enable_cdg and enable_txt should default to True."
)
def test_upload_endpoint_has_optional_cdg_txt_params(self):
"""Verify enable_cdg and enable_txt are Optional[bool] to support theme defaults.
CRITICAL: If enable_cdg/enable_txt are bool instead of Optional[bool],
they will default to False and override the theme-based defaults.
"""
from backend.api.routes import file_upload as file_upload_module
with open(file_upload_module.__file__, 'r') as f:
source_code = f.read()
has_optional_cdg = 'enable_cdg: Optional[bool] = Form(' in source_code
has_optional_txt = 'enable_txt: Optional[bool] = Form(' in source_code
assert has_optional_cdg, (
"file_upload.py has enable_cdg as bool instead of Optional[bool]. "
"This prevents theme-based defaults from working."
)
assert has_optional_txt, (
"file_upload.py has enable_txt as bool instead of Optional[bool]. "
"This prevents theme-based defaults from working."
)
def test_job_create_includes_theme_id_and_color_overrides(self):
"""Verify JobCreate is called with theme_id and color_overrides."""
from backend.api.routes import file_upload as file_upload_module
with open(file_upload_module.__file__, 'r') as f:
source_code = f.read()
has_theme_id_in_job_create = 'theme_id=theme_id,' in source_code
has_color_overrides_in_job_create = 'color_overrides=parsed_color_overrides' in source_code
assert has_theme_id_in_job_create, (
"file_upload.py does not pass theme_id to JobCreate."
)
assert has_color_overrides_in_job_create, (
"file_upload.py does not pass color_overrides to JobCreate."
)
if __name__ == "__main__":
pytest.main([__file__, "-v"])
+24 -13
View File
@@ -254,15 +254,16 @@ The function uses environment variables:
Known gaps are configured in `main.py` and should match your local `validate_and_sync.py`.
## Self-Hosted GitHub Actions Runner
## Self-Hosted GitHub Actions Runners
A dedicated GCP VM runs self-hosted GitHub Actions runners to solve disk space issues
on GitHub-hosted runners (14GB vs 200GB).
Multiple GCP VMs run self-hosted GitHub Actions runners to:
- Solve disk space issues on GitHub-hosted runners (14GB vs 200GB)
- Enable parallel job execution across multiple runners
### Specs
- **Instance**: e2-standard-4 (4 vCPU, 16GB RAM)
- **Disk**: 200GB SSD
- **Instances**: 5x e2-standard-4 (4 vCPU, 16GB RAM each)
- **Disk**: 200GB SSD per runner
- **Pre-installed**: Docker, Python 3.13, Node.js 20, Java 21, Poetry, FFmpeg, gcloud CLI
- **Labels**: `self-hosted`, `linux`, `x64`, `gcp`, `large-disk`
@@ -280,15 +281,15 @@ on GitHub-hosted runners (14GB vs 200GB).
gcloud secrets versions add github-runner-pat --data-file=-
```
3. Run `pulumi up` to create the runner VM
3. Run `pulumi up` to create the runner VMs
4. Verify the runner registered:
4. Verify the runners registered:
- Go to https://github.com/nomadkaraoke/karaoke-gen/settings/actions/runners
- You should see `gcp-runner-github-runner` with status "Idle"
- You should see `gcp-runner-github-runner-1` through `gcp-runner-github-runner-5`
### Usage in Workflows
Jobs that need more disk space use the self-hosted runner:
Jobs that need more disk space use the self-hosted runners:
```yaml
jobs:
@@ -299,27 +300,37 @@ jobs:
# ... rest of job
```
GitHub automatically distributes jobs across available runners.
### Troubleshooting
**Runner not appearing in GitHub:**
```bash
# Check startup logs
gcloud compute ssh github-runner --zone=us-central1-a -- \
# Check startup logs (replace N with runner number 1-3)
gcloud compute ssh github-runner-N --zone=us-central1-a -- \
"sudo cat /var/log/github-runner-startup.log"
```
**Re-register runner:**
```bash
# Restart the VM to re-run startup script
gcloud compute instances reset github-runner --zone=us-central1-a
gcloud compute instances reset github-runner-N --zone=us-central1-a
```
**Check runner service status:**
```bash
gcloud compute ssh github-runner --zone=us-central1-a -- \
gcloud compute ssh github-runner-N --zone=us-central1-a -- \
"sudo systemctl status actions.runner.*"
```
**List all runners:**
```bash
for i in 1 2 3 4 5; do
echo "=== github-runner-$i ==="
gcloud compute instances describe github-runner-$i --zone=us-central1-a --format='get(status)'
done
```
## Next Steps
1. Run `pulumi up` to create infrastructure
+89 -54
View File
@@ -1255,7 +1255,7 @@ apt-get install -y \\
# ==================== Docker ====================
echo "Installing Docker..."
curl -fsSL https://download.docker.com/linux/debian/gpg | gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
curl -fsSL https://download.docker.com/linux/debian/gpg | gpg --batch --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian $(lsb_release -cs) stable" > /etc/apt/sources.list.d/docker.list
apt-get update
apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
@@ -1264,23 +1264,38 @@ apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin do
systemctl enable docker
systemctl start docker
# ==================== Python 3.13 ====================
echo "Installing Python 3.13..."
add-apt-repository -y ppa:deadsnakes/ppa || true
apt-get update
apt-get install -y python3.13 python3.13-venv python3.13-dev python3-pip || {
# Fallback: build from source if PPA not available
echo "PPA not available, installing Python 3.13 from source..."
apt-get install -y build-essential zlib1g-dev libncurses5-dev libgdbm-dev libnss3-dev libreadline-dev libffi-dev libsqlite3-dev wget libbz2-dev
cd /tmp
wget https://www.python.org/ftp/python/3.13.0/Python-3.13.0.tgz
tar -xf Python-3.13.0.tgz
cd Python-3.13.0
./configure --enable-optimizations
make -j $(nproc)
make altinstall
ln -sf /usr/local/bin/python3.13 /usr/bin/python3.13
}
# ==================== Python 3.13 via pyenv ====================
# setup-python action on self-hosted runners needs Python in the tool cache
echo "Installing Python 3.13 via pyenv..."
# Install pyenv dependencies
apt-get install -y make build-essential libssl-dev zlib1g-dev \\
libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm \\
libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev
# Install pyenv system-wide
export PYENV_ROOT=/opt/pyenv
curl https://pyenv.run | bash
# Add to system profile for future sessions
cat > /etc/profile.d/pyenv.sh << 'PYENV_PROFILE'
export PYENV_ROOT=/opt/pyenv
export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"
PYENV_PROFILE
# Source pyenv for current session
export PATH="/opt/pyenv/bin:$PATH"
eval "$(/opt/pyenv/bin/pyenv init -)"
# Install Python 3.13
/opt/pyenv/bin/pyenv install 3.13.0
/opt/pyenv/bin/pyenv global 3.13.0
# Create symlinks for system-wide access
ln -sf /opt/pyenv/shims/python3.13 /usr/local/bin/python3.13
ln -sf /opt/pyenv/shims/python3 /usr/local/bin/python3
ln -sf /opt/pyenv/shims/pip3 /usr/local/bin/pip3
# ==================== Node.js 20 ====================
echo "Installing Node.js 20..."
@@ -1288,14 +1303,12 @@ curl -fsSL https://deb.nodesource.com/setup_20.x | bash -
apt-get install -y nodejs
# ==================== Java 21 (for Firestore emulator) ====================
echo "Installing Java 21..."
apt-get install -y openjdk-21-jdk || {
# Fallback: use Temurin
curl -fsSL https://packages.adoptium.net/artifactory/api/gpg/key/public | gpg --dearmor -o /usr/share/keyrings/adoptium.gpg
echo "deb [signed-by=/usr/share/keyrings/adoptium.gpg] https://packages.adoptium.net/artifactory/deb $(lsb_release -cs) main" > /etc/apt/sources.list.d/adoptium.list
apt-get update
apt-get install -y temurin-21-jdk
}
echo "Installing Java 21 via Temurin..."
# Use Temurin (Adoptium) - more reliable than Debian repos
curl -fsSL https://packages.adoptium.net/artifactory/api/gpg/key/public | gpg --batch --dearmor -o /usr/share/keyrings/adoptium.gpg
echo "deb [signed-by=/usr/share/keyrings/adoptium.gpg] https://packages.adoptium.net/artifactory/deb $(lsb_release -cs) main" > /etc/apt/sources.list.d/adoptium.list
apt-get update
apt-get install -y temurin-21-jdk
# ==================== FFmpeg ====================
echo "Installing FFmpeg..."
@@ -1308,7 +1321,7 @@ ln -sf /root/.local/bin/poetry /usr/local/bin/poetry
# ==================== Google Cloud SDK ====================
echo "Installing Google Cloud SDK..."
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | gpg --dearmor -o /usr/share/keyrings/cloud.google.gpg
curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | gpg --batch --dearmor -o /usr/share/keyrings/cloud.google.gpg
echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" > /etc/apt/sources.list.d/google-cloud-sdk.list
apt-get update
apt-get install -y google-cloud-cli google-cloud-cli-firestore-emulator
@@ -1375,6 +1388,23 @@ echo "Installing runner service..."
echo "GitHub Actions runner setup complete at $(date)"
echo "Runner registered with labels: self-hosted,linux,x64,gcp,large-disk"
# ==================== Setup Python in tool cache ====================
# setup-python action looks for Python in RUNNER_TOOL_CACHE/_tool/Python
echo "Setting up Python 3.13 in tool cache for setup-python action..."
TOOL_CACHE="/home/runner/actions-runner/_work/_tool"
mkdir -p $TOOL_CACHE/Python/3.13.0/x64
# Copy pyenv Python to tool cache
cp -r /opt/pyenv/versions/3.13.0/* $TOOL_CACHE/Python/3.13.0/x64/
# Create marker file that setup-python looks for
touch $TOOL_CACHE/Python/3.13.0/x64.complete
# Fix permissions
chown -R runner:runner $TOOL_CACHE
echo "Python 3.13 added to tool cache"
# ==================== Docker cleanup cron ====================
# Prevent disk from filling up with old Docker images
echo "Setting up Docker cleanup cron..."
@@ -1388,35 +1418,40 @@ chmod +x /etc/cron.daily/docker-cleanup
echo "Setup complete!"
"""
# GitHub runner VM instance
github_runner_vm = gcp.compute.Instance(
"github-runner",
name="github-runner",
machine_type="e2-standard-4", # 4 vCPU, 16GB RAM - good for Docker builds
zone="us-central1-a",
boot_disk=gcp.compute.InstanceBootDiskArgs(
initialize_params=gcp.compute.InstanceBootDiskInitializeParamsArgs(
image="debian-cloud/debian-12",
size=200, # GB - plenty for Docker builds and caches
type="pd-ssd", # SSD for faster I/O
# GitHub runner VM instances (multiple runners for parallel job execution)
NUM_RUNNERS = 5
github_runner_vms = []
for i in range(1, NUM_RUNNERS + 1):
vm = gcp.compute.Instance(
f"github-runner-{i}",
name=f"github-runner-{i}",
machine_type="e2-standard-4", # 4 vCPU, 16GB RAM - good for Docker builds
zone="us-central1-a",
boot_disk=gcp.compute.InstanceBootDiskArgs(
initialize_params=gcp.compute.InstanceBootDiskInitializeParamsArgs(
image="debian-cloud/debian-12",
size=200, # GB - plenty for Docker builds and caches
type="pd-ssd", # SSD for faster I/O
),
),
),
network_interfaces=[gcp.compute.InstanceNetworkInterfaceArgs(
network="default",
access_configs=[gcp.compute.InstanceNetworkInterfaceAccessConfigArgs()], # Ephemeral IP is fine
)],
service_account=gcp.compute.InstanceServiceAccountArgs(
email=github_runner_sa.email,
scopes=["cloud-platform"],
),
metadata_startup_script=GITHUB_RUNNER_STARTUP_SCRIPT,
tags=["github-runner"],
allow_stopping_for_update=True,
# Ensure the secret exists before creating the VM
opts=pulumi.ResourceOptions(depends_on=[github_runner_pat_secret]),
)
network_interfaces=[gcp.compute.InstanceNetworkInterfaceArgs(
network="default",
access_configs=[gcp.compute.InstanceNetworkInterfaceAccessConfigArgs()], # Ephemeral IP is fine
)],
service_account=gcp.compute.InstanceServiceAccountArgs(
email=github_runner_sa.email,
scopes=["cloud-platform"],
),
metadata_startup_script=GITHUB_RUNNER_STARTUP_SCRIPT,
tags=["github-runner"],
allow_stopping_for_update=True,
# Ensure the secret exists before creating the VM
opts=pulumi.ResourceOptions(depends_on=[github_runner_pat_secret]),
)
github_runner_vms.append(vm)
# Export GitHub runner resources
pulumi.export("github_runner_vm_name", github_runner_vm.name)
pulumi.export("github_runner_vm_names", [vm.name for vm in github_runner_vms])
pulumi.export("github_runner_service_account", github_runner_sa.email)
+1 -1
View File
@@ -1,6 +1,6 @@
[tool.poetry]
name = "karaoke-gen"
version = "0.76.25"
version = "0.76.26"
description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens."
authors = ["Andrew Beveridge <andrew@beveridge.uk>"]
license = "MIT"