mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary Follow-up to #1077. `video-encoding-job` runs both the post-review render (`render_video_worker`) and the final video (`video_worker`). In us-central1 each run waited **2–5 min** for Cloud Run Jobs to start it, so every customer job paid that twice. This moves it to us-east4 alongside the download, lyrics and bulk-search jobs. ## Cost (measured, not estimated) - Over the last 30 days the job read **156 GB** from Google services, mostly the us-central1 bucket, across 494 runs (Cloud Monitoring `container/network/received_bytes_count`, kind=google). That's about 0.32 GB per run. - Pricing from the Billing Catalog: GCS inter-region within North America is **free for the first 100 GB/month, then $0.02/GB**. Cloud Run writes to Google APIs are **$0**. - Internet egress (224 GB/month, Dropbox etc.) and Cloud Run CPU cost the same in either region. - Existing cross-region reads are about 24 GB/month (GPU job and lyrics), which the billing export shows at about $0. - **Net: about $1.60/month**, or about $3/month worst case. Time saved: roughly 4–8 min per customer job. ## Changes - **backend**: `trigger_video_worker` (via `_trigger_cloud_run_job`) and `trigger_render_video_worker` now use `cpu_jobs_region`. `GCP_REGION` stays us-central1; encoding VM and zone logic are unchanged. - **infra (Pulumi, already applied)**: new us-east4 `video-encoding-job`, with the legacy us-central1 copy kept for rollback and its deletion protection turned off. A preview after applying showed 290 unchanged. - **ci**: the video job image update now targets us-east4, using the us-east4 CPU image copy. - **Also updated**: rollback.sh, tests (video and render routing), TROUBLESHOOTING/ARCHITECTURE/LESSONS-LEARNED. ## Testing - `make test` passes: backend tests plus 1443 frontend tests. - Prod smoke test of the new us-east4 job: it imports `video_worker` and `render_video_worker` and calls the encoding VM's `/health` on its public IP (port 8080 is open to 0.0.0.0/0 with API-key auth). **Started in 25 s, exit 0.** 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Updates** - Video encoding and render jobs now run in the `us-east4` region, alongside other CPU jobs. - Video encoding continues to access storage and encoding resources in `us-central1`; cross-region transfer may incur costs. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
258 lines
11 KiB
Bash
Executable File
258 lines
11 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
#
|
|
# rollback.sh — fast, codified rollback for the karaoke-gen production backend.
|
|
#
|
|
# WHY THIS EXISTS (incident hardening item R2):
|
|
# The deploy pipeline auto-deploys on merge to main and has NO automatic
|
|
# rollback. During an outage we should not be composing `gcloud` from memory.
|
|
# This script is the single, documented, parameterized rollback path for:
|
|
# - the `karaoke-backend` Cloud Run SERVICE, and
|
|
# - the three Cloud Run JOBS that share the backend image:
|
|
# * video-encoding-job (CPU, us-east4 — CPU image copy)
|
|
# * lyrics-transcription-job (CPU, us-east4 — CPU image copy)
|
|
# * audio-separation-job (GPU, us-east4 — uses the GPU image)
|
|
#
|
|
# TWO DIFFERENT MECHANISMS (intentional — see docs/TROUBLESHOOTING.md "Fast rollback"):
|
|
# * SERVICE → shift 100% traffic to a previous, already-built revision
|
|
# (`gcloud run services update-traffic --to-revisions=<rev>=100`).
|
|
# This is instant and needs no rebuild — the safest fast path.
|
|
# * JOBS → Cloud Run Jobs cannot traffic-split, so we re-pin them to a
|
|
# previous image tag (`gcloud run jobs update --image ...:<tag>`).
|
|
# The next job invocation then runs the pinned (old) image.
|
|
#
|
|
# USAGE:
|
|
# scripts/rollback.sh --list
|
|
# Show recent service revisions + recent backend image tags so you can
|
|
# pick a known-good target. Run this first during an incident.
|
|
#
|
|
# scripts/rollback.sh --to-revision <REVISION> [--service-only] [--dry-run]
|
|
# Roll the SERVICE back to an existing revision (fast, no rebuild).
|
|
#
|
|
# scripts/rollback.sh --image-version <vX.Y.Z|SHA> [--jobs-only] [--dry-run]
|
|
# Re-pin the three JOBS to a previous image tag.
|
|
#
|
|
# scripts/rollback.sh --to-revision <REV> --image-version <vX.Y.Z> [--dry-run]
|
|
# Roll the service AND re-pin the jobs in one shot (typical full rollback).
|
|
#
|
|
# FLAGS:
|
|
# --to-revision <REV> Target service revision (e.g. karaoke-backend-00123-abc).
|
|
# --image-version <TAG> Target image tag for the jobs (e.g. v0.222.2 or a git SHA).
|
|
# --service-only Only roll the service (ignore jobs).
|
|
# --jobs-only Only re-pin the jobs (ignore the service).
|
|
# --dry-run Print the gcloud commands WITHOUT executing them.
|
|
# --list List recent revisions + image tags, then exit.
|
|
# -h | --help Show this help.
|
|
#
|
|
# SAFETY:
|
|
# * Nothing runs until you pass an explicit target (--to-revision / --image-version).
|
|
# * --dry-run prints every gcloud command verbatim so you can eyeball it first.
|
|
# * The script echoes exactly what it will do and (when interactive) asks for
|
|
# confirmation before mutating anything.
|
|
#
|
|
set -euo pipefail
|
|
|
|
# --------------------------------------------------------------------------
|
|
# Constants — mirror .github/workflows/ci.yml deploy-backend (see ~ci.yml:1809,
|
|
# :1911, :1939). Keep these in sync if the CI registry/region/names change.
|
|
# --------------------------------------------------------------------------
|
|
PROJECT_ID="nomadkaraoke"
|
|
SERVICE_NAME="karaoke-backend"
|
|
SERVICE_REGION="us-central1"
|
|
|
|
# CPU (service + CPU jobs) image repository.
|
|
CPU_IMAGE_REPO="us-central1-docker.pkg.dev/${PROJECT_ID}/karaoke-repo/karaoke-backend"
|
|
# GPU (audio separation) image repository, co-located with the L4 GPU in us-east4.
|
|
GPU_IMAGE_REPO="us-east4-docker.pkg.dev/${PROJECT_ID}/karaoke-backend-gpu/karaoke-backend"
|
|
# CPU image copy for the CPU jobs that run in us-east4 (CPU_JOBS_REGION).
|
|
CPU_EAST_IMAGE_REPO="us-east4-docker.pkg.dev/${PROJECT_ID}/karaoke-backend-gpu/karaoke-backend-cpu"
|
|
|
|
# job_name:region:repo — audio-separation-job uses the GPU repo/region.
|
|
# audio-download-job / bulk-search-job are NOT re-pinned: they track :latest and
|
|
# CI never re-pins them, so a pin here would stick past the next deploy.
|
|
JOBS=(
|
|
"video-encoding-job:us-east4:${CPU_EAST_IMAGE_REPO}"
|
|
"lyrics-transcription-job:us-east4:${CPU_EAST_IMAGE_REPO}"
|
|
"audio-separation-job:us-east4:${GPU_IMAGE_REPO}"
|
|
)
|
|
|
|
# --------------------------------------------------------------------------
|
|
# Arg parsing
|
|
# --------------------------------------------------------------------------
|
|
TO_REVISION=""
|
|
IMAGE_VERSION=""
|
|
SERVICE_ONLY=false
|
|
JOBS_ONLY=false
|
|
DRY_RUN=false
|
|
DO_LIST=false
|
|
|
|
usage() {
|
|
# Print the leading comment block (everything between the shebang and `set -e`)
|
|
# as help text so this stays the single source of truth.
|
|
sed -n '3,60p' "$0" | sed 's/^# \{0,1\}//'
|
|
}
|
|
|
|
while [[ $# -gt 0 ]]; do
|
|
case "$1" in
|
|
--to-revision) TO_REVISION="${2:?--to-revision requires a revision name}"; shift 2 ;;
|
|
--image-version) IMAGE_VERSION="${2:?--image-version requires a tag}"; shift 2 ;;
|
|
--service-only) SERVICE_ONLY=true; shift ;;
|
|
--jobs-only) JOBS_ONLY=true; shift ;;
|
|
--dry-run) DRY_RUN=true; shift ;;
|
|
--list) DO_LIST=true; shift ;;
|
|
-h|--help) usage; exit 0 ;;
|
|
*) echo "ERROR: unknown argument: $1" >&2; echo "Run '$0 --help' for usage." >&2; exit 2 ;;
|
|
esac
|
|
done
|
|
|
|
# run_cmd: echo the command, then run it unless --dry-run is set.
|
|
run_cmd() {
|
|
echo "+ $*"
|
|
if [[ "$DRY_RUN" == "true" ]]; then
|
|
echo " (dry-run — not executed)"
|
|
return 0
|
|
fi
|
|
"$@"
|
|
}
|
|
|
|
confirm() {
|
|
# Skip confirmation for dry-runs and for non-interactive shells (e.g. CI).
|
|
if [[ "$DRY_RUN" == "true" ]] || [[ ! -t 0 ]]; then
|
|
return 0
|
|
fi
|
|
read -r -p "Proceed with the above rollback? [y/N] " reply
|
|
case "$reply" in
|
|
y|Y|yes|YES) return 0 ;;
|
|
*) echo "Aborted."; exit 1 ;;
|
|
esac
|
|
}
|
|
|
|
# --------------------------------------------------------------------------
|
|
# --list: show the operator what targets are available.
|
|
# --------------------------------------------------------------------------
|
|
if [[ "$DO_LIST" == "true" ]]; then
|
|
echo "=== Recent '${SERVICE_NAME}' service revisions (newest first) ==="
|
|
gcloud run revisions list \
|
|
--service="${SERVICE_NAME}" \
|
|
--region="${SERVICE_REGION}" \
|
|
--project="${PROJECT_ID}" \
|
|
--format='table(metadata.name, status.conditions[0].lastTransitionTime.date("%Y-%m-%d %H:%M"), spec.containers[0].image.basename())' \
|
|
--limit=15 || echo "(could not list revisions — check gcloud auth/project)"
|
|
|
|
echo
|
|
echo "=== Current traffic split ==="
|
|
gcloud run services describe "${SERVICE_NAME}" \
|
|
--region="${SERVICE_REGION}" \
|
|
--project="${PROJECT_ID}" \
|
|
--format='table(status.traffic[].revisionName, status.traffic[].percent, status.traffic[].tag)' \
|
|
|| echo "(could not describe service)"
|
|
|
|
echo
|
|
echo "=== Recent CPU backend image tags (for --image-version) ==="
|
|
gcloud artifacts docker tags list "${CPU_IMAGE_REPO}" \
|
|
--project="${PROJECT_ID}" \
|
|
--format='value(tag)' --limit=20 2>/dev/null \
|
|
| grep -E '^v[0-9]' | sort -V | tail -15 || echo "(could not list image tags)"
|
|
exit 0
|
|
fi
|
|
|
|
# --------------------------------------------------------------------------
|
|
# Validate inputs.
|
|
# --------------------------------------------------------------------------
|
|
if [[ -z "$TO_REVISION" && -z "$IMAGE_VERSION" ]]; then
|
|
echo "ERROR: you must pass an explicit target." >&2
|
|
echo " - Roll the SERVICE: --to-revision <REVISION>" >&2
|
|
echo " - Re-pin the JOBS: --image-version <vX.Y.Z|SHA>" >&2
|
|
echo " - Not sure what's available? $0 --list" >&2
|
|
echo "Run '$0 --help' for full usage." >&2
|
|
exit 2
|
|
fi
|
|
|
|
if [[ "$SERVICE_ONLY" == "true" && "$JOBS_ONLY" == "true" ]]; then
|
|
echo "ERROR: --service-only and --jobs-only are mutually exclusive." >&2
|
|
exit 2
|
|
fi
|
|
|
|
# Decide what we're doing.
|
|
ROLL_SERVICE=false
|
|
ROLL_JOBS=false
|
|
[[ "$JOBS_ONLY" != "true" && -n "$TO_REVISION" ]] && ROLL_SERVICE=true
|
|
[[ "$SERVICE_ONLY" != "true" && -n "$IMAGE_VERSION" ]] && ROLL_JOBS=true
|
|
|
|
if [[ "$ROLL_SERVICE" != "true" && "$ROLL_JOBS" != "true" ]]; then
|
|
echo "ERROR: the given flags don't select anything to roll back." >&2
|
|
echo " (e.g. --service-only needs --to-revision; --jobs-only needs --image-version)" >&2
|
|
exit 2
|
|
fi
|
|
|
|
# --------------------------------------------------------------------------
|
|
# Show the plan before doing anything.
|
|
# --------------------------------------------------------------------------
|
|
echo "=============================================================="
|
|
echo " karaoke-gen production ROLLBACK"
|
|
echo " project: ${PROJECT_ID}"
|
|
echo " dry-run: ${DRY_RUN}"
|
|
if [[ "$ROLL_SERVICE" == "true" ]]; then
|
|
echo " SERVICE: ${SERVICE_NAME} (${SERVICE_REGION})"
|
|
echo " → shift 100% traffic to revision '${TO_REVISION}'"
|
|
fi
|
|
if [[ "$ROLL_JOBS" == "true" ]]; then
|
|
echo " JOBS: re-pin to image tag '${IMAGE_VERSION}'"
|
|
for entry in "${JOBS[@]}"; do
|
|
IFS=':' read -r job_name job_region job_repo <<< "$entry"
|
|
echo " → ${job_name} (${job_region}) = ${job_repo}:${IMAGE_VERSION}"
|
|
done
|
|
fi
|
|
echo "=============================================================="
|
|
confirm
|
|
|
|
# --------------------------------------------------------------------------
|
|
# Roll the SERVICE (traffic shift — instant, no rebuild).
|
|
# --------------------------------------------------------------------------
|
|
if [[ "$ROLL_SERVICE" == "true" ]]; then
|
|
echo
|
|
echo ">>> Rolling service '${SERVICE_NAME}' traffic to '${TO_REVISION}'..."
|
|
run_cmd gcloud run services update-traffic "${SERVICE_NAME}" \
|
|
--region="${SERVICE_REGION}" \
|
|
--project="${PROJECT_ID}" \
|
|
--to-revisions="${TO_REVISION}=100"
|
|
fi
|
|
|
|
# --------------------------------------------------------------------------
|
|
# Re-pin the JOBS (image update — next invocation runs the pinned image).
|
|
# --------------------------------------------------------------------------
|
|
if [[ "$ROLL_JOBS" == "true" ]]; then
|
|
echo
|
|
echo ">>> Re-pinning Cloud Run Jobs to '${IMAGE_VERSION}'..."
|
|
for entry in "${JOBS[@]}"; do
|
|
IFS=':' read -r job_name job_region job_repo <<< "$entry"
|
|
echo "--- ${job_name} (${job_region}) ---"
|
|
# The us-east4 CPU image copy only has tags from v0.247.1 onward. Skip a job
|
|
# ONLY when its target tag genuinely doesn't exist, so the remaining jobs
|
|
# still get re-pinned; any other lookup error (auth, network) aborts loudly
|
|
# rather than letting the rollback look complete with a job on the bad image.
|
|
if ! describe_err=$(gcloud artifacts docker images describe "${job_repo}:${IMAGE_VERSION}" \
|
|
--project="${PROJECT_ID}" 2>&1 >/dev/null); then
|
|
if grep -qiE "not ?found|NOT_FOUND|manifest unknown" <<< "$describe_err"; then
|
|
echo "⚠️ ${job_repo}:${IMAGE_VERSION} not found — SKIPPING ${job_name} (left on its current image)" >&2
|
|
continue
|
|
fi
|
|
echo "❌ Could not inspect ${job_repo}:${IMAGE_VERSION}: ${describe_err}" >&2
|
|
exit 1
|
|
fi
|
|
run_cmd gcloud run jobs update "${job_name}" \
|
|
--image="${job_repo}:${IMAGE_VERSION}" \
|
|
--region="${job_region}" \
|
|
--project="${PROJECT_ID}" \
|
|
--quiet
|
|
done
|
|
fi
|
|
|
|
echo
|
|
if [[ "$DRY_RUN" == "true" ]]; then
|
|
echo "✅ Dry-run complete — no changes were made."
|
|
else
|
|
echo "✅ Rollback complete."
|
|
echo " Verify the service: curl -s https://api.nomadkaraoke.com/api/health/detailed | jq .version"
|
|
echo " Jobs pick up the pinned image on their NEXT invocation (in-flight runs are unaffected)."
|
|
fi
|