mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary Approved GCP cost cuts for the encoding pipeline (target: whole project < $300/mo; encoding VMs were ~$300/mo for ~5-20 jobs/day). Details + before/after table: `docs/archive/2026-09-26-encoding-cost-cuts.md`. | | Before | After | |---|---|---| | Primary pair a/b | c4d-highcpu-32 on-demand | **c4d-highcpu-16 Spot** (STOP on preemption) | | Fallbacks | 8 VMs / 6 families | **1 on-demand** `encoding-worker-fallback-c2df` (c2d-highcpu-16, us-central1-f) | | Idle shutdown | 15 min / check every 5 | **5 min / check every 2** | | CI worker blue-green | every push to main | only when `infrastructure/encoding-worker/worker_code_paths.txt` paths changed since the serving version (+ `workflow_dispatch` `force_encoding_worker_deploy`) | | Nomad Dropbox upload | whole folder | minus lossless 4K **MP4** (tenant folders unchanged) | **Infra already applied locally (targeted Pulumi):** 7 fallback VMs + disks + 7 static IPs deleted; a/b + c2df resized in place; idle function env + scheduler updated. The Spot switch was done in place with `gcloud compute instances set-scheduling` on the stopped VMs + `pulumi refresh` (pulumi-gcp treats a `scheduling` diff as *replace*, and recreating a c4d VM risks a stockout). `encoding-worker-fallback-vms` secret → v6 (c2df only). CI's pulumi run should be a no-op for these resources. ### Scope change (per Andrew) **Task 5 dropped:** review previews stay on the encoding VM (`USE_GCE_PREVIEW_ENCODING=true` unchanged) and the review-page warmup is kept, since it deliberately warms the VM for the preview and the final render. To make sure the new 5-min idle timeout doesn't undermine that, the review page now also heartbeats **every 2 min** while the tab is visible and was used in the last 15 min, and re-warms on tab refocus (`useEncodingWorkerKeepAlive`). Before this, the heartbeat only fired on lyric edits. The idle checker keeps the routed VM alive while `last_activity_at` < 5 min, and both warmup and heartbeat write it. Tests cover both sides. ### Code changes - **Spot preemption mid-encode**: when a poll fails, `wait_for_completion` checks the pinned worker VM. If it is STOPPING/TERMINATED it raises `EncodingJobLostError`, so `run_with_lost_job_resubmit` resubmits the encode, and that resubmit restarts the Spot VM or falls back to c2df. Before, the failure showed up after ~7 min as a "lost contact" error that was not resubmitted. - **Stale fallback lists**: a candidate VM that no longer exists (`NotFound`) is skipped instead of aborting the whole failover. This protects the window before running instances pick up the new secret. - `SPEED_RANK` is now keyed by machine family, so the ranking is independent of VM size. - **Idle-stop vs. in-flight encode** (review follow-up): the idle function's `/health active_jobs` only counts RUNNING jobs, and `last_activity_at` was only written when a VM was started. With a 5-min window, a long encode's VM could be stopped in the seconds between the job finishing and the next poll. The poll loop now refreshes `last_activity_at` at most once a minute. A VM seen in STAGING/PROVISIONING after it accepted a job also counts as lost. CI `Read Firestore config` now writes empty values instead of "None". - **Worker deploy gate**: `backend/tests/test_encoding_worker_code_paths.py` derives the worker's transitive `backend.*` import set and fails if `worker_code_paths.txt` doesn't cover it. Every deploy still uploads the wheel and `startup.sh`, so a cold-booted VM always runs the latest code. - **Dropbox**: nothing automated reads lossless files from Dropbox. The lossless **MKV** is still uploaded because the completion email and the live Fiverr auto-reply bot both tell customers it is in the Dropbox folder, and the VocalStar tenant expects lossless files. To drop the MKV as well, update those two texts first, then add the MKV suffix to `DROPBOX_SKIP_OUTPUT_SUFFIXES`. ## Testing - New/updated unit tests: preemption → lost job → resubmit, NotFound candidate skip, family ranking, idle 5-min/heartbeat window, fleet-shape invariants, worker-path coverage, Dropbox exclusion, keep-alive hook (Jest, fake timers). - `pytest backend/tests --ignore=backend/tests/emulator`: 4802 passed. The only failures are 6 tests that need the spaCy `en_core_web_sm` model, which isn't installed in my local venv; unrelated to this PR. - Jest lyrics-review + lib/lyrics-review: 423 passed. Infra tests (`test_encoding_worker_config.py`, idle function): all pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
1189 lines
52 KiB
Python
1189 lines
52 KiB
Python
"""
|
|
Tests for the blue-green encoding worker manager service.
|
|
|
|
Tests cover:
|
|
- Task 1: Firestore config document schema, get_config, update_activity, swap_roles
|
|
- Task 2: VM lifecycle (get_vm_status, start_vm, stop_vm, ensure_primary_running)
|
|
"""
|
|
|
|
import pytest
|
|
import aiohttp
|
|
from datetime import datetime, UTC
|
|
from unittest.mock import MagicMock, patch, call, AsyncMock
|
|
|
|
from backend.services.encoding_worker_manager import (
|
|
EncodingWorkerCandidate,
|
|
EncodingWorkerConfig,
|
|
EncodingWorkerManager,
|
|
CONFIG_COLLECTION,
|
|
CONFIG_DOCUMENT,
|
|
ENCODING_WORKER_PORT,
|
|
)
|
|
from backend.services.encoding_errors import (
|
|
EncodingWorkerCapacityError,
|
|
EncodingWorkerStartError,
|
|
)
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Fixtures
|
|
# ---------------------------------------------------------------------------
|
|
|
|
def _make_firestore_data(**overrides):
|
|
"""Create a realistic Firestore document dict for encoding-worker config."""
|
|
data = {
|
|
"primary_vm": "encoding-worker-blue",
|
|
"primary_ip": "10.128.0.50",
|
|
"primary_version": "1.0.0",
|
|
"primary_deployed_at": "2026-03-20T10:00:00Z",
|
|
"secondary_vm": "encoding-worker-green",
|
|
"secondary_ip": "10.128.0.51",
|
|
"secondary_version": "0.9.0",
|
|
"secondary_deployed_at": "2026-03-19T08:00:00Z",
|
|
"last_swap_at": "2026-03-20T10:00:00Z",
|
|
"last_activity_at": "2026-03-24T12:00:00Z",
|
|
"deploy_in_progress": False,
|
|
"deploy_in_progress_since": None,
|
|
}
|
|
data.update(overrides)
|
|
return data
|
|
|
|
|
|
@pytest.fixture
|
|
def mock_db():
|
|
"""Return a mock Firestore client with a pre-configured config document."""
|
|
db = MagicMock()
|
|
snapshot = MagicMock()
|
|
snapshot.exists = True
|
|
snapshot.to_dict.return_value = _make_firestore_data()
|
|
|
|
doc_ref = MagicMock()
|
|
doc_ref.get.return_value = snapshot
|
|
|
|
collection_ref = MagicMock()
|
|
collection_ref.document.return_value = doc_ref
|
|
|
|
db.collection.return_value = collection_ref
|
|
return db
|
|
|
|
|
|
@pytest.fixture
|
|
def mock_compute():
|
|
"""Return a mock Compute Engine InstancesClient."""
|
|
return MagicMock()
|
|
|
|
|
|
@pytest.fixture
|
|
def manager(mock_db, mock_compute):
|
|
return EncodingWorkerManager(
|
|
db=mock_db,
|
|
compute_client=mock_compute,
|
|
project_id="nomadkaraoke",
|
|
zone="us-central1-c",
|
|
)
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Task 1: Config dataclass
|
|
# ---------------------------------------------------------------------------
|
|
|
|
class TestEncodingWorkerConfig:
|
|
"""Tests for the EncodingWorkerConfig dataclass."""
|
|
|
|
def test_primary_url(self):
|
|
config = EncodingWorkerConfig(
|
|
primary_vm="blue", primary_ip="10.0.0.1", primary_version="1.0",
|
|
primary_deployed_at=None,
|
|
secondary_vm="green", secondary_ip="10.0.0.2",
|
|
secondary_version=None, secondary_deployed_at=None,
|
|
last_swap_at=None, last_activity_at=None,
|
|
deploy_in_progress=False, deploy_in_progress_since=None,
|
|
)
|
|
assert config.primary_url == "http://10.0.0.1:8080"
|
|
|
|
def test_secondary_url(self):
|
|
config = EncodingWorkerConfig(
|
|
primary_vm="blue", primary_ip="10.0.0.1", primary_version="1.0",
|
|
primary_deployed_at=None,
|
|
secondary_vm="green", secondary_ip="10.0.0.2",
|
|
secondary_version=None, secondary_deployed_at=None,
|
|
last_swap_at=None, last_activity_at=None,
|
|
deploy_in_progress=False, deploy_in_progress_since=None,
|
|
)
|
|
assert config.secondary_url == "http://10.0.0.2:8080"
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Task 1: get_config / update_activity / swap_roles
|
|
# ---------------------------------------------------------------------------
|
|
|
|
class TestEncodingWorkerManager:
|
|
"""Tests for Firestore config operations."""
|
|
|
|
def test_get_config_returns_dataclass(self, manager, mock_db):
|
|
config = manager.get_config()
|
|
assert isinstance(config, EncodingWorkerConfig)
|
|
assert config.primary_vm == "encoding-worker-blue"
|
|
assert config.primary_ip == "10.128.0.50"
|
|
assert config.primary_version == "1.0.0"
|
|
assert config.secondary_vm == "encoding-worker-green"
|
|
assert config.deploy_in_progress is False
|
|
|
|
def test_get_config_document_not_found(self, manager, mock_db):
|
|
snapshot = MagicMock()
|
|
snapshot.exists = False
|
|
mock_db.collection.return_value.document.return_value.get.return_value = snapshot
|
|
|
|
with pytest.raises(ValueError, match="not found"):
|
|
manager.get_config()
|
|
|
|
def test_get_config_reads_correct_path(self, manager, mock_db):
|
|
manager.get_config()
|
|
mock_db.collection.assert_called_with(CONFIG_COLLECTION)
|
|
mock_db.collection.return_value.document.assert_called_with(CONFIG_DOCUMENT)
|
|
|
|
def test_update_activity_sets_timestamp(self, manager, mock_db):
|
|
with patch("backend.services.encoding_worker_manager.datetime") as mock_dt:
|
|
mock_dt.now.return_value = datetime(2026, 3, 24, 15, 0, 0, tzinfo=UTC)
|
|
mock_dt.side_effect = lambda *a, **kw: datetime(*a, **kw)
|
|
manager.update_activity()
|
|
|
|
doc_ref = mock_db.collection.return_value.document.return_value
|
|
doc_ref.update.assert_called_once_with({
|
|
"last_activity_at": "2026-03-24T15:00:00+00:00",
|
|
})
|
|
|
|
def test_swap_roles_uses_transaction(self, manager, mock_db):
|
|
"""swap_roles should use a Firestore transaction for atomicity."""
|
|
# Set up the transaction mock
|
|
mock_transaction = MagicMock()
|
|
mock_db.transaction.return_value = mock_transaction
|
|
|
|
# The transactional function is called with the transaction
|
|
manager.swap_roles(
|
|
new_primary_vm="encoding-worker-green",
|
|
new_primary_ip="10.128.0.51",
|
|
new_primary_version="2.0.0",
|
|
old_primary_vm="encoding-worker-blue",
|
|
old_primary_ip="10.128.0.50",
|
|
old_primary_version="1.0.0",
|
|
)
|
|
|
|
# Verify transaction was created and the doc was updated within it
|
|
mock_db.transaction.assert_called_once()
|
|
|
|
def test_swap_roles_writes_correct_data(self, manager, mock_db):
|
|
"""swap_roles should write the swapped primary/secondary fields."""
|
|
mock_transaction = MagicMock()
|
|
mock_db.transaction.return_value = mock_transaction
|
|
|
|
with patch("backend.services.encoding_worker_manager.datetime") as mock_dt:
|
|
mock_dt.now.return_value = datetime(2026, 3, 24, 16, 0, 0, tzinfo=UTC)
|
|
mock_dt.side_effect = lambda *a, **kw: datetime(*a, **kw)
|
|
|
|
manager.swap_roles(
|
|
new_primary_vm="encoding-worker-green",
|
|
new_primary_ip="10.128.0.51",
|
|
new_primary_version="2.0.0",
|
|
old_primary_vm="encoding-worker-blue",
|
|
old_primary_ip="10.128.0.50",
|
|
old_primary_version="1.0.0",
|
|
)
|
|
|
|
# The transaction.update call should contain the swapped values
|
|
doc_ref = mock_db.collection.return_value.document.return_value
|
|
update_call = mock_transaction.update.call_args
|
|
assert update_call is not None
|
|
data = update_call[0][1] # second positional arg is the data dict
|
|
assert data["primary_vm"] == "encoding-worker-green"
|
|
assert data["primary_ip"] == "10.128.0.51"
|
|
assert data["primary_version"] == "2.0.0"
|
|
assert data["secondary_vm"] == "encoding-worker-blue"
|
|
assert data["secondary_ip"] == "10.128.0.50"
|
|
assert data["secondary_version"] == "1.0.0"
|
|
assert data["last_swap_at"] == "2026-03-24T16:00:00+00:00"
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Task 2: VM lifecycle
|
|
# ---------------------------------------------------------------------------
|
|
|
|
class TestVMLifecycle:
|
|
"""Tests for Compute Engine VM operations."""
|
|
|
|
def test_get_vm_status_returns_status(self, manager, mock_compute):
|
|
mock_instance = MagicMock()
|
|
mock_instance.status = "RUNNING"
|
|
mock_compute.get.return_value = mock_instance
|
|
|
|
status = manager.get_vm_status("encoding-worker-blue")
|
|
assert status == "RUNNING"
|
|
mock_compute.get.assert_called_once_with(
|
|
project="nomadkaraoke",
|
|
zone="us-central1-c",
|
|
instance="encoding-worker-blue",
|
|
)
|
|
|
|
def test_get_vm_status_terminated(self, manager, mock_compute):
|
|
mock_instance = MagicMock()
|
|
mock_instance.status = "TERMINATED"
|
|
mock_compute.get.return_value = mock_instance
|
|
|
|
assert manager.get_vm_status("encoding-worker-blue") == "TERMINATED"
|
|
|
|
def test_start_vm_calls_compute_start(self, manager, mock_compute):
|
|
mock_instance = MagicMock()
|
|
mock_instance.status = "TERMINATED"
|
|
mock_compute.get.return_value = mock_instance
|
|
|
|
# Successful start: operation has no error_code/error_message and
|
|
# .result() returns without raising.
|
|
op = MagicMock()
|
|
op.error_code = ""
|
|
op.error_message = ""
|
|
op.result.return_value = None
|
|
mock_compute.start.return_value = op
|
|
|
|
manager.start_vm("encoding-worker-blue")
|
|
mock_compute.start.assert_called_once_with(
|
|
project="nomadkaraoke",
|
|
zone="us-central1-c",
|
|
instance="encoding-worker-blue",
|
|
)
|
|
op.result.assert_called_once()
|
|
|
|
def test_start_vm_skips_if_already_running(self, manager, mock_compute):
|
|
mock_instance = MagicMock()
|
|
mock_instance.status = "RUNNING"
|
|
mock_compute.get.return_value = mock_instance
|
|
|
|
manager.start_vm("encoding-worker-blue")
|
|
mock_compute.start.assert_not_called()
|
|
|
|
def test_is_transient_start_error_classification(self):
|
|
from backend.services.encoding_worker_manager import _is_transient_start_error
|
|
from backend.services.encoding_errors import EncodingWorkerStartError
|
|
assert _is_transient_start_error(
|
|
EncodingWorkerStartError("VM x start failed: 503 — SERVICE UNAVAILABLE", code="503")
|
|
)
|
|
assert not _is_transient_start_error(
|
|
EncodingWorkerStartError("VM x start failed: PERMISSION_DENIED", code="PERMISSION_DENIED")
|
|
)
|
|
|
|
def test_stop_vm_calls_compute_stop(self, manager, mock_compute):
|
|
mock_instance = MagicMock()
|
|
mock_instance.status = "RUNNING"
|
|
mock_compute.get.return_value = mock_instance
|
|
|
|
manager.stop_vm("encoding-worker-blue")
|
|
mock_compute.stop.assert_called_once_with(
|
|
project="nomadkaraoke",
|
|
zone="us-central1-c",
|
|
instance="encoding-worker-blue",
|
|
)
|
|
|
|
def test_stop_vm_skips_if_already_terminated(self, manager, mock_compute):
|
|
mock_instance = MagicMock()
|
|
mock_instance.status = "TERMINATED"
|
|
mock_compute.get.return_value = mock_instance
|
|
|
|
manager.stop_vm("encoding-worker-blue")
|
|
mock_compute.stop.assert_not_called()
|
|
|
|
def test_ensure_primary_running_starts_stopped_vm(self, manager, mock_db, mock_compute):
|
|
"""When primary is stopped, ensure_primary_running should start it."""
|
|
mock_instance = MagicMock()
|
|
mock_instance.status = "TERMINATED"
|
|
mock_compute.get.return_value = mock_instance
|
|
|
|
op = MagicMock()
|
|
op.error_code = ""
|
|
op.error_message = ""
|
|
op.result.return_value = None
|
|
mock_compute.start.return_value = op
|
|
|
|
with patch("backend.services.encoding_worker_manager.datetime") as mock_dt:
|
|
mock_dt.now.return_value = datetime(2026, 3, 24, 15, 0, 0, tzinfo=UTC)
|
|
mock_dt.side_effect = lambda *a, **kw: datetime(*a, **kw)
|
|
result = manager.ensure_primary_running()
|
|
|
|
assert result["started"] is True
|
|
assert result["vm_name"] == "encoding-worker-blue"
|
|
assert result["primary_url"] == "http://10.128.0.50:8080"
|
|
mock_compute.start.assert_called_once()
|
|
op.result.assert_called_once()
|
|
|
|
def test_ensure_primary_running_already_running(self, manager, mock_db, mock_compute):
|
|
"""When primary is already running, should not start and still update activity."""
|
|
mock_instance = MagicMock()
|
|
mock_instance.status = "RUNNING"
|
|
mock_compute.get.return_value = mock_instance
|
|
|
|
with patch("backend.services.encoding_worker_manager.datetime") as mock_dt:
|
|
mock_dt.now.return_value = datetime(2026, 3, 24, 15, 0, 0, tzinfo=UTC)
|
|
mock_dt.side_effect = lambda *a, **kw: datetime(*a, **kw)
|
|
result = manager.ensure_primary_running()
|
|
|
|
assert result["started"] is False
|
|
assert result["vm_name"] == "encoding-worker-blue"
|
|
assert result["primary_url"] == "http://10.128.0.50:8080"
|
|
mock_compute.start.assert_not_called()
|
|
|
|
def test_ensure_primary_running_updates_activity(self, manager, mock_db, mock_compute):
|
|
"""ensure_primary_running should always update last_activity_at."""
|
|
mock_instance = MagicMock()
|
|
mock_instance.status = "RUNNING"
|
|
mock_compute.get.return_value = mock_instance
|
|
|
|
with patch("backend.services.encoding_worker_manager.datetime") as mock_dt:
|
|
mock_dt.now.return_value = datetime(2026, 3, 24, 15, 0, 0, tzinfo=UTC)
|
|
mock_dt.side_effect = lambda *a, **kw: datetime(*a, **kw)
|
|
manager.ensure_primary_running()
|
|
|
|
doc_ref = mock_db.collection.return_value.document.return_value
|
|
doc_ref.update.assert_called_once()
|
|
|
|
def test_start_vm_skips_if_staging(self, manager, mock_compute):
|
|
"""start_vm should also skip if VM is in STAGING state (already starting)."""
|
|
mock_instance = MagicMock()
|
|
mock_instance.status = "STAGING"
|
|
mock_compute.get.return_value = mock_instance
|
|
|
|
manager.start_vm("encoding-worker-blue")
|
|
mock_compute.start.assert_not_called()
|
|
|
|
def test_stop_vm_skips_if_stopping(self, manager, mock_compute):
|
|
"""stop_vm should skip if VM is already in STOPPING state."""
|
|
mock_instance = MagicMock()
|
|
mock_instance.status = "STOPPING"
|
|
mock_compute.get.return_value = mock_instance
|
|
|
|
manager.stop_vm("encoding-worker-blue")
|
|
mock_compute.stop.assert_not_called()
|
|
|
|
# ------------------------------------------------------------------
|
|
# Operation-result inspection (capacity / start error classification)
|
|
# ------------------------------------------------------------------
|
|
|
|
def test_start_vm_raises_capacity_error_when_zone_exhausted(self, manager, mock_compute):
|
|
"""start_vm must surface ZONE_RESOURCE_POOL_EXHAUSTED as a typed exception.
|
|
|
|
Without this, the GCE error stays hidden behind a fire-and-forget call
|
|
and the readiness gate eventually times out with no useful message.
|
|
"""
|
|
mock_instance = MagicMock()
|
|
mock_instance.status = "TERMINATED"
|
|
mock_compute.get.return_value = mock_instance
|
|
|
|
op = MagicMock()
|
|
op.error_code = "ZONE_RESOURCE_POOL_EXHAUSTED"
|
|
op.error_message = "The zone does not have enough resources available"
|
|
op.result.return_value = None
|
|
mock_compute.start.return_value = op
|
|
|
|
with pytest.raises(EncodingWorkerCapacityError) as exc_info:
|
|
manager.start_vm("encoding-worker-blue")
|
|
|
|
assert exc_info.value.code == "ZONE_RESOURCE_POOL_EXHAUSTED"
|
|
assert exc_info.value.vm_name == "encoding-worker-blue"
|
|
assert exc_info.value.zone == "us-central1-c"
|
|
assert "ZONE_RESOURCE_POOL_EXHAUSTED" in str(exc_info.value)
|
|
|
|
def test_start_vm_raises_generic_start_error_for_other_codes(self, manager, mock_compute):
|
|
"""start_vm raises EncodingWorkerStartError for non-capacity GCE errors."""
|
|
mock_instance = MagicMock()
|
|
mock_instance.status = "TERMINATED"
|
|
mock_compute.get.return_value = mock_instance
|
|
|
|
op = MagicMock()
|
|
op.error_code = "RESOURCE_NOT_READY"
|
|
op.error_message = "VM is in transition"
|
|
op.result.return_value = None
|
|
mock_compute.start.return_value = op
|
|
|
|
with pytest.raises(EncodingWorkerStartError) as exc_info:
|
|
manager.start_vm("encoding-worker-blue")
|
|
|
|
assert not isinstance(exc_info.value, EncodingWorkerCapacityError)
|
|
assert exc_info.value.code == "RESOURCE_NOT_READY"
|
|
|
|
def test_start_vm_raises_when_operation_result_throws(self, manager, mock_compute):
|
|
"""If operation.result() raises (e.g. timeout), surface as a start error."""
|
|
mock_instance = MagicMock()
|
|
mock_instance.status = "TERMINATED"
|
|
mock_compute.get.return_value = mock_instance
|
|
|
|
op = MagicMock()
|
|
op.error_code = ""
|
|
op.error_message = ""
|
|
op.result.side_effect = TimeoutError("operation timed out")
|
|
mock_compute.start.return_value = op
|
|
|
|
with pytest.raises(EncodingWorkerStartError) as exc_info:
|
|
manager.start_vm("encoding-worker-blue")
|
|
|
|
assert "timed out" in str(exc_info.value).lower() or "operation" in str(exc_info.value).lower()
|
|
|
|
def test_ensure_primary_running_raises_capacity_error(
|
|
self, manager, mock_db, mock_compute
|
|
):
|
|
"""ensure_primary_running propagates capacity errors so callers can react."""
|
|
mock_instance = MagicMock()
|
|
mock_instance.status = "TERMINATED"
|
|
mock_compute.get.return_value = mock_instance
|
|
|
|
op = MagicMock()
|
|
op.error_code = "ZONE_RESOURCE_POOL_EXHAUSTED"
|
|
op.error_message = "out of capacity"
|
|
op.result.return_value = None
|
|
mock_compute.start.return_value = op
|
|
|
|
with patch("backend.services.encoding_worker_manager.datetime") as mock_dt:
|
|
mock_dt.now.return_value = datetime(2026, 5, 5, 9, 0, 0, tzinfo=UTC)
|
|
mock_dt.side_effect = lambda *a, **kw: datetime(*a, **kw)
|
|
with pytest.raises(EncodingWorkerCapacityError):
|
|
manager.ensure_primary_running()
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Phase 3: multi-zone failover (ensure_any_running)
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def _ok_op():
|
|
"""Build a successful compute Operation mock."""
|
|
op = MagicMock()
|
|
op.error_code = ""
|
|
op.error_message = ""
|
|
op.result.return_value = None
|
|
return op
|
|
|
|
|
|
def _capacity_op():
|
|
"""Build a ZONE_RESOURCE_POOL_EXHAUSTED Operation mock."""
|
|
op = MagicMock()
|
|
op.error_code = "ZONE_RESOURCE_POOL_EXHAUSTED"
|
|
op.error_message = "out of capacity"
|
|
op.result.return_value = None
|
|
return op
|
|
|
|
|
|
class TestEnsureAnyRunning:
|
|
"""Tests for ensure_any_running() — capacity-fallback iteration."""
|
|
|
|
def test_starts_first_candidate_when_available(self, manager, mock_db, mock_compute):
|
|
"""When the primary candidate has capacity, it's used and no fallback engaged."""
|
|
mock_instance = MagicMock()
|
|
mock_instance.status = "TERMINATED"
|
|
mock_compute.get.return_value = mock_instance
|
|
mock_compute.start.return_value = _ok_op()
|
|
|
|
candidates = [
|
|
EncodingWorkerCandidate(vm_name="encoding-worker-blue", zone="us-central1-c", ip="10.0.0.1"),
|
|
EncodingWorkerCandidate(vm_name="encoding-worker-fb-a", zone="us-central1-a", ip="10.0.0.2"),
|
|
]
|
|
|
|
with patch("backend.services.encoding_worker_manager.datetime") as mock_dt:
|
|
mock_dt.now.return_value = datetime(2026, 5, 5, 9, 0, 0, tzinfo=UTC)
|
|
mock_dt.side_effect = lambda *a, **kw: datetime(*a, **kw)
|
|
result = manager.ensure_any_running(candidates)
|
|
|
|
assert result["fell_back"] is False
|
|
assert result["vm_name"] == "encoding-worker-blue"
|
|
assert result["zone"] == "us-central1-c"
|
|
# Only the first candidate's start was attempted.
|
|
mock_compute.start.assert_called_once()
|
|
|
|
def test_skips_deleted_candidate_and_starts_next(self, manager, mock_db, mock_compute):
|
|
"""A candidate VM that no longer exists (removed from the fleet while an
|
|
instance still holds a stale fallback list) is skipped, not fatal."""
|
|
from google.api_core import exceptions as gcp_exceptions
|
|
|
|
stopped = MagicMock()
|
|
stopped.status = "TERMINATED"
|
|
|
|
def _get(project, zone, instance):
|
|
if instance == "encoding-worker-fallback-gone":
|
|
raise gcp_exceptions.NotFound("instance not found")
|
|
return stopped
|
|
|
|
mock_compute.get.side_effect = _get
|
|
# Primary: Spot capacity exhausted. Deleted fallback: NotFound. Live fallback: ok.
|
|
mock_compute.start.side_effect = [_capacity_op(), _ok_op()]
|
|
|
|
candidates = [
|
|
EncodingWorkerCandidate(vm_name="encoding-worker-blue", zone="us-central1-c",
|
|
ip="10.0.0.1", is_primary=True),
|
|
EncodingWorkerCandidate(vm_name="encoding-worker-fallback-gone", zone="us-central1-a",
|
|
ip="10.0.0.2"),
|
|
EncodingWorkerCandidate(vm_name="encoding-worker-fallback-c2df", zone="us-central1-f",
|
|
ip="10.0.0.3"),
|
|
]
|
|
|
|
with patch("backend.services.encoding_worker_manager.datetime") as mock_dt:
|
|
mock_dt.now.return_value = datetime(2026, 9, 26, 9, 0, 0, tzinfo=UTC)
|
|
mock_dt.side_effect = lambda *a, **kw: datetime(*a, **kw)
|
|
result = manager.ensure_any_running(candidates)
|
|
|
|
assert result["vm_name"] == "encoding-worker-fallback-c2df"
|
|
assert result["fell_back"] is True
|
|
# The deleted VM was never asked to start.
|
|
started = [c.kwargs["instance"] for c in mock_compute.start.call_args_list]
|
|
assert started == ["encoding-worker-blue", "encoding-worker-fallback-c2df"]
|
|
|
|
def test_all_candidates_missing_raises_start_error(self, manager, mock_compute):
|
|
"""If every candidate is gone, surface a typed start error (not a bare NotFound)."""
|
|
from google.api_core import exceptions as gcp_exceptions
|
|
|
|
mock_compute.get.side_effect = gcp_exceptions.NotFound("gone")
|
|
candidates = [
|
|
EncodingWorkerCandidate(vm_name="vm-x", zone="us-central1-a", ip="10.0.0.9"),
|
|
]
|
|
with pytest.raises(EncodingWorkerStartError) as exc_info:
|
|
manager.ensure_any_running(candidates)
|
|
assert exc_info.value.code == "NOT_FOUND"
|
|
mock_compute.start.assert_not_called()
|
|
|
|
def test_falls_through_to_next_zone_on_capacity_error(self, manager, mock_db, mock_compute):
|
|
"""Capacity error on candidate 1 triggers a try of candidate 2."""
|
|
mock_instance = MagicMock()
|
|
mock_instance.status = "TERMINATED"
|
|
mock_compute.get.return_value = mock_instance
|
|
# First start: capacity error. Second start: succeeds.
|
|
mock_compute.start.side_effect = [_capacity_op(), _ok_op()]
|
|
|
|
candidates = [
|
|
EncodingWorkerCandidate(vm_name="encoding-worker-blue", zone="us-central1-c", ip="10.0.0.1"),
|
|
EncodingWorkerCandidate(vm_name="encoding-worker-fb-a", zone="us-central1-a", ip="10.0.0.2"),
|
|
]
|
|
|
|
with patch("backend.services.encoding_worker_manager.datetime") as mock_dt:
|
|
mock_dt.now.return_value = datetime(2026, 5, 5, 9, 0, 0, tzinfo=UTC)
|
|
mock_dt.side_effect = lambda *a, **kw: datetime(*a, **kw)
|
|
result = manager.ensure_any_running(candidates)
|
|
|
|
assert result["fell_back"] is True
|
|
assert result["vm_name"] == "encoding-worker-fb-a"
|
|
assert result["zone"] == "us-central1-a"
|
|
|
|
# Both starts were attempted, in order.
|
|
assert mock_compute.start.call_count == 2
|
|
|
|
# Active override was persisted in Firestore.
|
|
doc_ref = mock_db.collection.return_value.document.return_value
|
|
# Multiple update calls (for activity + override) — find the override one
|
|
override_calls = [
|
|
c for c in doc_ref.update.call_args_list
|
|
if "active_override_vm" in c.args[0]
|
|
]
|
|
assert override_calls, "Expected Firestore update with active_override_vm"
|
|
override_data = override_calls[0].args[0]
|
|
assert override_data["active_override_vm"] == "encoding-worker-fb-a"
|
|
assert override_data["active_override_zone"] == "us-central1-a"
|
|
assert override_data["active_override_ip"] == "10.0.0.2"
|
|
|
|
def test_raises_when_all_candidates_exhausted(self, manager, mock_db, mock_compute):
|
|
"""If every candidate hits capacity error, the last one is re-raised."""
|
|
from backend.services.encoding_errors import EncodingWorkerCapacityError
|
|
|
|
mock_instance = MagicMock()
|
|
mock_instance.status = "TERMINATED"
|
|
mock_compute.get.return_value = mock_instance
|
|
mock_compute.start.side_effect = [_capacity_op(), _capacity_op(), _capacity_op()]
|
|
|
|
candidates = [
|
|
EncodingWorkerCandidate(vm_name="vm-c", zone="us-central1-c", ip="1.1.1.1"),
|
|
EncodingWorkerCandidate(vm_name="vm-a", zone="us-central1-a", ip="1.1.1.2"),
|
|
EncodingWorkerCandidate(vm_name="vm-f", zone="us-central1-f", ip="1.1.1.3"),
|
|
]
|
|
|
|
with pytest.raises(EncodingWorkerCapacityError):
|
|
manager.ensure_any_running(candidates)
|
|
|
|
# All three were attempted before giving up.
|
|
assert mock_compute.start.call_count == 3
|
|
|
|
def test_falls_through_on_generic_start_error_too(self, manager, mock_db, mock_compute):
|
|
"""Non-capacity start errors (e.g. 503 SERVICE_UNAVAILABLE) also trigger fallback.
|
|
|
|
Observed in prod 2026-05-06: GCE returned 503 from instances.start
|
|
for the primary VM. The original ensure_any_running only fell through
|
|
on EncodingWorkerCapacityError, so the 503 short-circuited multi-zone
|
|
failover. Broaden to catch the parent class.
|
|
"""
|
|
from backend.services.encoding_errors import EncodingWorkerStartError
|
|
|
|
mock_instance = MagicMock()
|
|
mock_instance.status = "TERMINATED"
|
|
mock_compute.get.return_value = mock_instance
|
|
|
|
# First start: 503 SERVICE_UNAVAILABLE (generic start error). Second: succeeds.
|
|
op_503 = MagicMock()
|
|
op_503.error_code = "503"
|
|
op_503.error_message = "SERVICE UNAVAILABLE"
|
|
op_503.result.return_value = None
|
|
mock_compute.start.side_effect = [op_503, _ok_op()]
|
|
|
|
candidates = [
|
|
EncodingWorkerCandidate(vm_name="primary", zone="us-central1-c", ip="10.0.0.1"),
|
|
EncodingWorkerCandidate(vm_name="fb-a", zone="us-central1-a", ip="10.0.0.2"),
|
|
]
|
|
|
|
with patch("backend.services.encoding_worker_manager.datetime") as mock_dt:
|
|
mock_dt.now.return_value = datetime(2026, 5, 6, 17, 0, 0, tzinfo=UTC)
|
|
mock_dt.side_effect = lambda *a, **kw: datetime(*a, **kw)
|
|
result = manager.ensure_any_running(candidates)
|
|
|
|
assert result["fell_back"] is True
|
|
assert result["vm_name"] == "fb-a"
|
|
assert mock_compute.start.call_count == 2
|
|
|
|
def test_raises_last_start_error_when_no_capacity_errors(self, manager, mock_db, mock_compute):
|
|
"""If every candidate fails with generic start errors (no capacity), surface the last one."""
|
|
from backend.services.encoding_errors import (
|
|
EncodingWorkerCapacityError, EncodingWorkerStartError,
|
|
)
|
|
|
|
mock_instance = MagicMock()
|
|
mock_instance.status = "TERMINATED"
|
|
mock_compute.get.return_value = mock_instance
|
|
|
|
def _op_503():
|
|
op = MagicMock()
|
|
op.error_code = "503"
|
|
op.error_message = "SERVICE UNAVAILABLE"
|
|
op.result.return_value = None
|
|
return op
|
|
|
|
# All-transient: the whole candidate set is retried a few times before
|
|
# giving up, so provide enough operations for every pass.
|
|
mock_compute.start.side_effect = [_op_503() for _ in range(20)]
|
|
|
|
candidates = [
|
|
EncodingWorkerCandidate(vm_name="primary", zone="us-central1-c", ip="10.0.0.1"),
|
|
EncodingWorkerCandidate(vm_name="fb-a", zone="us-central1-a", ip="10.0.0.2"),
|
|
]
|
|
|
|
with patch("backend.services.encoding_worker_manager.time.sleep") as sleep:
|
|
with pytest.raises(EncodingWorkerStartError) as exc_info:
|
|
manager.ensure_any_running(candidates)
|
|
# Should NOT be a capacity error — there were none in this scenario.
|
|
assert not isinstance(exc_info.value, EncodingWorkerCapacityError)
|
|
# Retried the whole 2-candidate set across multiple passes before failing.
|
|
assert mock_compute.start.call_count > 2
|
|
assert sleep.called
|
|
|
|
def test_retries_whole_set_on_all_transient_then_succeeds(self, manager, mock_db, mock_compute):
|
|
"""A brief all-zones 503 blip: first pass fails everywhere, retry succeeds."""
|
|
mock_instance = MagicMock()
|
|
mock_instance.status = "TERMINATED"
|
|
mock_compute.get.return_value = mock_instance
|
|
|
|
def _op_503():
|
|
op = MagicMock(); op.error_code = "503"; op.error_message = "SERVICE UNAVAILABLE"
|
|
op.result.return_value = None
|
|
return op
|
|
|
|
# Pass 1: both candidates 503. Pass 2: primary succeeds.
|
|
mock_compute.start.side_effect = [_op_503(), _op_503(), _ok_op()]
|
|
candidates = [
|
|
EncodingWorkerCandidate(vm_name="primary", zone="us-central1-c", ip="10.0.0.1"),
|
|
EncodingWorkerCandidate(vm_name="fb-a", zone="us-central1-a", ip="10.0.0.2"),
|
|
]
|
|
with patch("backend.services.encoding_worker_manager.time.sleep") as sleep, \
|
|
patch("backend.services.encoding_worker_manager.datetime") as mock_dt:
|
|
mock_dt.now.return_value = datetime(2026, 5, 6, 17, 0, 0, tzinfo=UTC)
|
|
mock_dt.side_effect = lambda *a, **kw: datetime(*a, **kw)
|
|
result = manager.ensure_any_running(candidates)
|
|
assert result["vm_name"] == "primary"
|
|
assert result["fell_back"] is False
|
|
assert mock_compute.start.call_count == 3
|
|
sleep.assert_called_once()
|
|
|
|
def test_does_not_retry_whole_set_on_mixed_nontransient(self, manager, mock_db, mock_compute):
|
|
"""If any candidate fails non-transiently (PERMISSION_DENIED), do NOT retry
|
|
the whole set even though the last failure was a transient 503."""
|
|
from backend.services.encoding_errors import EncodingWorkerStartError
|
|
mock_instance = MagicMock()
|
|
mock_instance.status = "TERMINATED"
|
|
mock_compute.get.return_value = mock_instance
|
|
|
|
op_perm = MagicMock()
|
|
op_perm.error_code = "PERMISSION_DENIED"; op_perm.error_message = "denied"
|
|
op_perm.result.return_value = None
|
|
op_503 = MagicMock()
|
|
op_503.error_code = "503"; op_503.error_message = "SERVICE UNAVAILABLE"
|
|
op_503.result.return_value = None
|
|
# primary: PERMISSION_DENIED (non-transient); fallback: 503 (transient/last)
|
|
mock_compute.start.side_effect = [op_perm, op_503]
|
|
candidates = [
|
|
EncodingWorkerCandidate(vm_name="primary", zone="us-central1-c", ip="10.0.0.1"),
|
|
EncodingWorkerCandidate(vm_name="fb-a", zone="us-central1-a", ip="10.0.0.2"),
|
|
]
|
|
with patch("backend.services.encoding_worker_manager.time.sleep") as sleep:
|
|
with pytest.raises(EncodingWorkerStartError) as exc:
|
|
manager.ensure_any_running(candidates)
|
|
assert exc.value.code == "PERMISSION_DENIED" # non-transient surfaced
|
|
assert mock_compute.start.call_count == 2 # one pass only
|
|
sleep.assert_not_called()
|
|
|
|
def test_does_not_retry_whole_set_on_capacity(self, manager, mock_db, mock_compute):
|
|
"""Capacity exhaustion is not spun on — it raises immediately for the park path."""
|
|
from backend.services.encoding_errors import EncodingWorkerCapacityError
|
|
mock_instance = MagicMock()
|
|
mock_instance.status = "TERMINATED"
|
|
mock_compute.get.return_value = mock_instance
|
|
mock_compute.start.side_effect = [_capacity_op(), _capacity_op()]
|
|
candidates = [
|
|
EncodingWorkerCandidate(vm_name="primary", zone="us-central1-c", ip="10.0.0.1"),
|
|
EncodingWorkerCandidate(vm_name="fb-a", zone="us-central1-a", ip="10.0.0.2"),
|
|
]
|
|
with patch("backend.services.encoding_worker_manager.time.sleep") as sleep:
|
|
with pytest.raises(EncodingWorkerCapacityError):
|
|
manager.ensure_any_running(candidates)
|
|
assert mock_compute.start.call_count == 2 # one pass only, no whole-set retry
|
|
sleep.assert_not_called()
|
|
|
|
def test_prefers_capacity_error_when_mixed_failures(self, manager, mock_db, mock_compute):
|
|
"""When some candidates hit capacity and others hit generic errors, prefer raising
|
|
the capacity error — it's more actionable for the user-facing message."""
|
|
from backend.services.encoding_errors import EncodingWorkerCapacityError
|
|
|
|
mock_instance = MagicMock()
|
|
mock_instance.status = "TERMINATED"
|
|
mock_compute.get.return_value = mock_instance
|
|
|
|
op_503 = MagicMock()
|
|
op_503.error_code = "503"
|
|
op_503.error_message = "SERVICE UNAVAILABLE"
|
|
op_503.result.return_value = None
|
|
# First: capacity. Second: 503.
|
|
mock_compute.start.side_effect = [_capacity_op(), op_503]
|
|
|
|
candidates = [
|
|
EncodingWorkerCandidate(vm_name="primary", zone="us-central1-c", ip="10.0.0.1"),
|
|
EncodingWorkerCandidate(vm_name="fb-a", zone="us-central1-a", ip="10.0.0.2"),
|
|
]
|
|
|
|
with pytest.raises(EncodingWorkerCapacityError):
|
|
manager.ensure_any_running(candidates)
|
|
|
|
def test_clears_override_when_primary_succeeds(self, manager, mock_db, mock_compute):
|
|
"""When the primary starts successfully, any stale active_override is cleared."""
|
|
mock_instance = MagicMock()
|
|
mock_instance.status = "TERMINATED"
|
|
mock_compute.get.return_value = mock_instance
|
|
mock_compute.start.return_value = _ok_op()
|
|
|
|
# Pretend Firestore has an override set from a prior fallback.
|
|
snapshot = MagicMock()
|
|
snapshot.exists = True
|
|
snapshot.to_dict.return_value = {
|
|
"active_override_vm": "encoding-worker-fb-a",
|
|
"active_override_ip": "10.0.0.2",
|
|
"active_override_zone": "us-central1-a",
|
|
}
|
|
mock_db.collection.return_value.document.return_value.get.return_value = snapshot
|
|
|
|
candidates = [
|
|
EncodingWorkerCandidate(vm_name="encoding-worker-blue", zone="us-central1-c", ip="10.0.0.1"),
|
|
]
|
|
|
|
with patch("backend.services.encoding_worker_manager.datetime") as mock_dt:
|
|
mock_dt.now.return_value = datetime(2026, 5, 5, 9, 0, 0, tzinfo=UTC)
|
|
mock_dt.side_effect = lambda *a, **kw: datetime(*a, **kw)
|
|
result = manager.ensure_any_running(candidates)
|
|
|
|
assert result["fell_back"] is False
|
|
|
|
# Verify the clear-override write happened.
|
|
doc_ref = mock_db.collection.return_value.document.return_value
|
|
clear_calls = [
|
|
c for c in doc_ref.update.call_args_list
|
|
if c.args[0].get("active_override_vm") is None and "active_override_vm" in c.args[0]
|
|
]
|
|
assert clear_calls, "Expected an update call clearing active_override_vm to None"
|
|
|
|
def test_empty_candidates_raises_value_error(self, manager):
|
|
with pytest.raises(ValueError):
|
|
manager.ensure_any_running([])
|
|
|
|
|
|
class TestActiveUrl:
|
|
"""Tests for EncodingWorkerConfig.active_url override behavior."""
|
|
|
|
def test_active_url_returns_primary_when_no_override(self):
|
|
config = EncodingWorkerConfig(
|
|
primary_vm="vm-a", primary_ip="10.0.0.1", primary_version="1.0.0",
|
|
primary_deployed_at=None,
|
|
secondary_vm="vm-b", secondary_ip="10.0.0.2", secondary_version=None,
|
|
secondary_deployed_at=None,
|
|
last_swap_at=None, last_activity_at=None,
|
|
deploy_in_progress=False, deploy_in_progress_since=None,
|
|
)
|
|
assert config.active_url == f"http://10.0.0.1:{ENCODING_WORKER_PORT}"
|
|
|
|
def test_active_url_returns_override_when_set(self):
|
|
config = EncodingWorkerConfig(
|
|
primary_vm="vm-a", primary_ip="10.0.0.1", primary_version="1.0.0",
|
|
primary_deployed_at=None,
|
|
secondary_vm="vm-b", secondary_ip="10.0.0.2", secondary_version=None,
|
|
secondary_deployed_at=None,
|
|
last_swap_at=None, last_activity_at=None,
|
|
deploy_in_progress=False, deploy_in_progress_since=None,
|
|
active_override_vm="vm-fb-a",
|
|
active_override_ip="34.5.6.7",
|
|
active_override_zone="us-central1-a",
|
|
active_override_set_at="2026-05-05T09:00:00+00:00",
|
|
)
|
|
assert config.active_url == f"http://34.5.6.7:{ENCODING_WORKER_PORT}"
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Task 3: wait_for_worker_ready (cold-start readiness gate)
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestWaitForWorkerReady:
|
|
"""Tests for the wait_for_worker_ready() readiness gate."""
|
|
|
|
# ------------------------------------------------------------------
|
|
# Helpers
|
|
# ------------------------------------------------------------------
|
|
|
|
@staticmethod
|
|
def _mk_health_session(get_callable):
|
|
"""Build a mock ClientSession whose .get(url, ...) returns an async-CM Resp."""
|
|
mock_session = MagicMock()
|
|
mock_session.get = get_callable
|
|
mock_session.__aenter__ = AsyncMock(return_value=mock_session)
|
|
mock_session.__aexit__ = AsyncMock(return_value=None)
|
|
return mock_session
|
|
|
|
@staticmethod
|
|
def _mk_resp_cm(status):
|
|
"""Return a callable that, when invoked with a URL, returns an async-CM Resp."""
|
|
class _Resp:
|
|
def __init__(self, s):
|
|
self.status = s
|
|
async def __aenter__(self): return self
|
|
async def __aexit__(self, *a): return None
|
|
def _get(url, **kwargs):
|
|
return _Resp(status)
|
|
return _get
|
|
|
|
# ------------------------------------------------------------------
|
|
# Tests
|
|
# ------------------------------------------------------------------
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_returns_immediately_when_vm_running_and_healthy(self, manager, mock_compute):
|
|
"""Happy path: VM already RUNNING, /health returns 200 on first poll."""
|
|
mock_instance = MagicMock()
|
|
mock_instance.status = "RUNNING"
|
|
mock_compute.get.return_value = mock_instance
|
|
|
|
mock_session = self._mk_health_session(self._mk_resp_cm(200))
|
|
|
|
with patch("backend.services.encoding_worker_manager.aiohttp.ClientSession", return_value=mock_session):
|
|
await manager.wait_for_worker_ready(
|
|
"encoding-worker-blue",
|
|
"http://10.0.0.1:8080/health",
|
|
vm_timeout=2.0,
|
|
health_timeout=2.0,
|
|
poll_interval=0.05,
|
|
)
|
|
|
|
# Single VM status check, single health check
|
|
assert mock_compute.get.call_count == 1
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_polls_vm_until_running(self, manager, mock_compute):
|
|
"""Polls VM status through STAGING → RUNNING."""
|
|
statuses = ["STAGING", "STAGING", "RUNNING"]
|
|
instances = [MagicMock(status=s) for s in statuses]
|
|
mock_compute.get.side_effect = instances
|
|
|
|
mock_session = self._mk_health_session(self._mk_resp_cm(200))
|
|
|
|
with patch("backend.services.encoding_worker_manager.aiohttp.ClientSession", return_value=mock_session):
|
|
await manager.wait_for_worker_ready(
|
|
"encoding-worker-blue",
|
|
"http://10.0.0.1:8080/health",
|
|
vm_timeout=2.0,
|
|
health_timeout=2.0,
|
|
poll_interval=0.05,
|
|
)
|
|
|
|
assert mock_compute.get.call_count == 3
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_times_out_when_vm_never_runs(self, manager, mock_compute):
|
|
"""Raises TimeoutError if VM stays in STAGING past vm_timeout."""
|
|
mock_instance = MagicMock()
|
|
mock_instance.status = "STAGING"
|
|
mock_compute.get.return_value = mock_instance
|
|
|
|
with pytest.raises(TimeoutError, match="did not reach RUNNING"):
|
|
await manager.wait_for_worker_ready(
|
|
"encoding-worker-blue",
|
|
"http://10.0.0.1:8080/health",
|
|
vm_timeout=0.2,
|
|
health_timeout=2.0,
|
|
poll_interval=0.05,
|
|
)
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_times_out_when_health_never_200(self, manager, mock_compute):
|
|
"""Raises TimeoutError if /health never returns 200 past health_timeout."""
|
|
mock_instance = MagicMock()
|
|
mock_instance.status = "RUNNING"
|
|
mock_compute.get.return_value = mock_instance
|
|
|
|
mock_session = self._mk_health_session(self._mk_resp_cm(503))
|
|
|
|
with patch("backend.services.encoding_worker_manager.aiohttp.ClientSession", return_value=mock_session):
|
|
with pytest.raises(TimeoutError, match="did not become healthy"):
|
|
await manager.wait_for_worker_ready(
|
|
"encoding-worker-blue",
|
|
"http://10.0.0.1:8080/health",
|
|
vm_timeout=2.0,
|
|
health_timeout=0.2,
|
|
poll_interval=0.05,
|
|
)
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_tolerates_health_connection_errors(self, manager, mock_compute):
|
|
"""Connection errors during /health polling are treated as 'not ready', keep polling."""
|
|
mock_instance = MagicMock()
|
|
mock_instance.status = "RUNNING"
|
|
mock_compute.get.return_value = mock_instance
|
|
|
|
call_count = {"n": 0}
|
|
|
|
class _OkResp:
|
|
status = 200
|
|
async def __aenter__(self): return self
|
|
async def __aexit__(self, *a): return None
|
|
|
|
def flaky_get(url, **kwargs):
|
|
call_count["n"] += 1
|
|
if call_count["n"] < 3:
|
|
raise aiohttp.ClientConnectorError(MagicMock(), OSError("connection refused"))
|
|
return _OkResp()
|
|
|
|
mock_session = self._mk_health_session(flaky_get)
|
|
|
|
with patch("backend.services.encoding_worker_manager.aiohttp.ClientSession", return_value=mock_session):
|
|
await manager.wait_for_worker_ready(
|
|
"encoding-worker-blue",
|
|
"http://10.0.0.1:8080/health",
|
|
vm_timeout=2.0,
|
|
health_timeout=2.0,
|
|
poll_interval=0.05,
|
|
)
|
|
|
|
assert call_count["n"] == 3
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# capacity_state (availability awareness) + is_primary override routing
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestCapacityStateFeedback:
|
|
"""ensure_any_running records/clears per-(type,zone) stockout state."""
|
|
|
|
def test_records_stockout_on_capacity_error(self, manager, mock_db, mock_compute):
|
|
mock_instance = MagicMock()
|
|
mock_instance.status = "TERMINATED"
|
|
mock_compute.get.return_value = mock_instance
|
|
# Primary c4d hits capacity; fallback n2 succeeds.
|
|
mock_compute.start.side_effect = [_capacity_op(), _ok_op()]
|
|
|
|
candidates = [
|
|
EncodingWorkerCandidate(vm_name="encoding-worker-a", zone="us-central1-c",
|
|
ip="10.0.0.1", machine_type="c4d-highcpu-16", is_primary=True),
|
|
EncodingWorkerCandidate(vm_name="encoding-worker-fallback-n2f", zone="us-central1-f",
|
|
ip="10.0.0.2", machine_type="n2-highcpu-16"),
|
|
]
|
|
|
|
with patch("backend.services.encoding_worker_manager.datetime") as mock_dt:
|
|
mock_dt.now.return_value = datetime(2026, 5, 5, 9, 0, 0, tzinfo=UTC)
|
|
mock_dt.side_effect = lambda *a, **kw: datetime(*a, **kw)
|
|
manager.ensure_any_running(candidates)
|
|
|
|
doc_ref = mock_db.collection.return_value.document.return_value
|
|
# A merge-set recorded the c4d stockout under its type@zone key.
|
|
set_calls = [c for c in doc_ref.set.call_args_list
|
|
if "capacity_state" in (c.args[0] if c.args else {})]
|
|
recorded = {}
|
|
for c in set_calls:
|
|
recorded.update(c.args[0]["capacity_state"])
|
|
assert recorded.get("c4d-highcpu-16@us-central1-c") == "2026-05-05T09:00:00+00:00"
|
|
|
|
def test_clears_stockout_for_type_on_clean_start(self, manager, mock_db, mock_compute):
|
|
mock_instance = MagicMock()
|
|
mock_instance.status = "TERMINATED"
|
|
mock_compute.get.return_value = mock_instance
|
|
mock_compute.start.return_value = _ok_op()
|
|
|
|
candidates = [
|
|
EncodingWorkerCandidate(vm_name="encoding-worker-a", zone="us-central1-c",
|
|
ip="10.0.0.1", machine_type="c4d-highcpu-16", is_primary=True),
|
|
]
|
|
|
|
with patch("backend.services.encoding_worker_manager.datetime") as mock_dt:
|
|
mock_dt.now.return_value = datetime(2026, 5, 5, 9, 0, 0, tzinfo=UTC)
|
|
mock_dt.side_effect = lambda *a, **kw: datetime(*a, **kw)
|
|
manager.ensure_any_running(candidates)
|
|
|
|
doc_ref = mock_db.collection.return_value.document.return_value
|
|
cleared = {}
|
|
for c in doc_ref.set.call_args_list:
|
|
payload = c.args[0] if c.args else {}
|
|
if "capacity_state" in payload:
|
|
cleared.update(payload["capacity_state"])
|
|
assert cleared.get("c4d-highcpu-16@us-central1-c") is None
|
|
|
|
def test_stockout_write_failure_does_not_break_start(self, manager, mock_db, mock_compute):
|
|
"""A Firestore error while recording a stockout must not abort failover."""
|
|
mock_instance = MagicMock()
|
|
mock_instance.status = "TERMINATED"
|
|
mock_compute.get.return_value = mock_instance
|
|
mock_compute.start.side_effect = [_capacity_op(), _ok_op()]
|
|
|
|
doc_ref = mock_db.collection.return_value.document.return_value
|
|
doc_ref.set.side_effect = RuntimeError("firestore down")
|
|
|
|
candidates = [
|
|
EncodingWorkerCandidate(vm_name="encoding-worker-a", zone="us-central1-c",
|
|
ip="10.0.0.1", machine_type="c4d-highcpu-16", is_primary=True),
|
|
EncodingWorkerCandidate(vm_name="encoding-worker-fallback-n2f", zone="us-central1-f",
|
|
ip="10.0.0.2", machine_type="n2-highcpu-16"),
|
|
]
|
|
|
|
with patch("backend.services.encoding_worker_manager.datetime") as mock_dt:
|
|
mock_dt.now.return_value = datetime(2026, 5, 5, 9, 0, 0, tzinfo=UTC)
|
|
mock_dt.side_effect = lambda *a, **kw: datetime(*a, **kw)
|
|
result = manager.ensure_any_running(candidates)
|
|
|
|
assert result["vm_name"] == "encoding-worker-fallback-n2f"
|
|
assert result["fell_back"] is True
|
|
|
|
def test_is_primary_flag_routes_override_regardless_of_position(self, manager, mock_db, mock_compute):
|
|
"""A fallback ranked FIRST (is_primary=False) must still set active_override.
|
|
|
|
When the preference logic demotes a stocked-out primary, the fallback can be
|
|
candidate index 0 — the override must key off is_primary, not position.
|
|
"""
|
|
mock_instance = MagicMock()
|
|
mock_instance.status = "TERMINATED"
|
|
mock_compute.get.return_value = mock_instance
|
|
mock_compute.start.return_value = _ok_op()
|
|
|
|
# Fallback is first (primary demoted), primary flagged but ranked second.
|
|
candidates = [
|
|
EncodingWorkerCandidate(vm_name="encoding-worker-fallback-c2df", zone="us-central1-f",
|
|
ip="10.0.0.9", machine_type="c2d-highcpu-16", is_primary=False),
|
|
EncodingWorkerCandidate(vm_name="encoding-worker-a", zone="us-central1-c",
|
|
ip="10.0.0.1", machine_type="c4d-highcpu-16", is_primary=True),
|
|
]
|
|
|
|
with patch("backend.services.encoding_worker_manager.datetime") as mock_dt:
|
|
mock_dt.now.return_value = datetime(2026, 5, 5, 9, 0, 0, tzinfo=UTC)
|
|
mock_dt.side_effect = lambda *a, **kw: datetime(*a, **kw)
|
|
result = manager.ensure_any_running(candidates)
|
|
|
|
assert result["vm_name"] == "encoding-worker-fallback-c2df"
|
|
assert result["fell_back"] is True
|
|
doc_ref = mock_db.collection.return_value.document.return_value
|
|
override_calls = [c for c in doc_ref.update.call_args_list
|
|
if "active_override_vm" in c.args[0]]
|
|
assert override_calls, "Fallback at index 0 must still set the override"
|
|
assert override_calls[0].args[0]["active_override_vm"] == "encoding-worker-fallback-c2df"
|
|
|
|
def test_legacy_candidates_without_flag_use_position(self, manager, mock_db, mock_compute):
|
|
"""No is_primary flag anywhere → first candidate is treated as primary (legacy)."""
|
|
mock_instance = MagicMock()
|
|
mock_instance.status = "TERMINATED"
|
|
mock_compute.get.return_value = mock_instance
|
|
mock_compute.start.return_value = _ok_op()
|
|
|
|
candidates = [
|
|
EncodingWorkerCandidate(vm_name="encoding-worker-blue", zone="us-central1-c", ip="10.0.0.1"),
|
|
EncodingWorkerCandidate(vm_name="encoding-worker-fb-a", zone="us-central1-a", ip="10.0.0.2"),
|
|
]
|
|
|
|
with patch("backend.services.encoding_worker_manager.datetime") as mock_dt:
|
|
mock_dt.now.return_value = datetime(2026, 5, 5, 9, 0, 0, tzinfo=UTC)
|
|
mock_dt.side_effect = lambda *a, **kw: datetime(*a, **kw)
|
|
result = manager.ensure_any_running(candidates)
|
|
|
|
assert result["fell_back"] is False
|
|
assert result["vm_name"] == "encoding-worker-blue"
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# demote_active_worker: skip a broken fallback VM after a runtime infra failure
|
|
# ---------------------------------------------------------------------------
|
|
|
|
class TestDemoteActiveWorker:
|
|
"""A worker VM that accepted a job but then hit a VM-level infra/auth failure
|
|
(e.g. could not reach the metadata server — job 6452888e) must be demoted so
|
|
the retry lands on a healthy worker."""
|
|
|
|
def test_demotes_fallback_and_clears_override(self, manager):
|
|
"""When a fallback is serving, record its family cooldown then clear the
|
|
override so selection re-resolves from the primary."""
|
|
doc_ref = manager._doc_ref()
|
|
doc_ref.get.return_value.exists = True
|
|
doc_ref.get.return_value.to_dict.return_value = _make_firestore_data(
|
|
active_override_vm="encoding-worker-fallback-c4a",
|
|
active_override_ip="10.128.0.99",
|
|
active_override_zone="us-central1-c",
|
|
active_override_set_at="2026-08-31T12:00:00Z",
|
|
)
|
|
|
|
manager.demote_active_worker(reason="job 6452888e: metadata server unavailable")
|
|
|
|
# Recorded a cooldown for the fallback's (machine_type@zone).
|
|
assert doc_ref.set.called
|
|
set_args, set_kwargs = doc_ref.set.call_args
|
|
assert "capacity_state" in set_args[0]
|
|
assert "c4-highcpu-16@us-central1-c" in set_args[0]["capacity_state"]
|
|
assert set_kwargs.get("merge") is True
|
|
|
|
# Cleared the active_override so the retry starts from the primary.
|
|
override_clears = [
|
|
c for c in doc_ref.update.call_args_list
|
|
if c.args and "active_override_vm" in c.args[0]
|
|
]
|
|
assert override_clears, "expected active_override to be cleared"
|
|
assert override_clears[-1].args[0]["active_override_vm"] is None
|
|
|
|
def test_no_op_when_primary_is_serving(self, manager):
|
|
"""When no fallback override is set (primary pair serving), demote is a no-op:
|
|
we don't family-demote the fast c4d primary on a rare blip."""
|
|
doc_ref = manager._doc_ref()
|
|
doc_ref.get.return_value.exists = True
|
|
doc_ref.get.return_value.to_dict.return_value = _make_firestore_data() # no override
|
|
|
|
manager.demote_active_worker(reason="job zzz: metadata server unavailable")
|
|
|
|
assert not doc_ref.set.called
|
|
override_clears = [
|
|
c for c in doc_ref.update.call_args_list
|
|
if c.args and "active_override_vm" in c.args[0]
|
|
]
|
|
assert not override_clears
|
|
|
|
def test_never_raises_on_config_read_failure(self, manager):
|
|
"""Demotion is best-effort telemetry — a Firestore hiccup must not break the
|
|
failure path it runs on."""
|
|
doc_ref = manager._doc_ref()
|
|
doc_ref.get.side_effect = RuntimeError("firestore unavailable")
|
|
|
|
# Should swallow the error rather than propagate.
|
|
manager.demote_active_worker(reason="job qqq")
|