Files
karaoke-gen/backend/tests/test_encoding_worker_manager.py
Andrew Beveridge ☄️andClaude Opus 5.5 87a36bd2a5 feat(encoding): GCP cost cuts — Spot 16-vCPU workers, 3-VM fleet, 5-min idle, gated worker deploys (v0.245.0) (#1061)
## Summary
Approved GCP cost cuts for the encoding pipeline (target: whole project
< $300/mo; encoding VMs were ~$300/mo for ~5-20 jobs/day). Details +
before/after table: `docs/archive/2026-09-26-encoding-cost-cuts.md`.

| | Before | After |
|---|---|---|
| Primary pair a/b | c4d-highcpu-32 on-demand | **c4d-highcpu-16 Spot**
(STOP on preemption) |
| Fallbacks | 8 VMs / 6 families | **1 on-demand**
`encoding-worker-fallback-c2df` (c2d-highcpu-16, us-central1-f) |
| Idle shutdown | 15 min / check every 5 | **5 min / check every 2** |
| CI worker blue-green | every push to main | only when
`infrastructure/encoding-worker/worker_code_paths.txt` paths changed
since the serving version (+ `workflow_dispatch`
`force_encoding_worker_deploy`) |
| Nomad Dropbox upload | whole folder | minus lossless 4K **MP4**
(tenant folders unchanged) |

**Infra already applied locally (targeted Pulumi):** 7 fallback VMs +
disks + 7 static IPs deleted; a/b + c2df resized in place; idle function
env + scheduler updated. The Spot switch was done in place with `gcloud
compute instances set-scheduling` on the stopped VMs + `pulumi refresh`
(pulumi-gcp treats a `scheduling` diff as *replace*, and recreating a
c4d VM risks a stockout). `encoding-worker-fallback-vms` secret → v6
(c2df only). CI's pulumi run should be a no-op for these resources.

### Scope change (per Andrew)
**Task 5 dropped:** review previews stay on the encoding VM
(`USE_GCE_PREVIEW_ENCODING=true` unchanged) and the review-page warmup
is kept, since it deliberately warms the VM for the preview and the
final render. To make sure the new 5-min idle timeout doesn't undermine
that, the review page now also heartbeats **every 2 min** while the tab
is visible and was used in the last 15 min, and re-warms on tab refocus
(`useEncodingWorkerKeepAlive`). Before this, the heartbeat only fired on
lyric edits. The idle checker keeps the routed VM alive while
`last_activity_at` < 5 min, and both warmup and heartbeat write it.
Tests cover both sides.

### Code changes
- **Spot preemption mid-encode**: when a poll fails,
`wait_for_completion` checks the pinned worker VM. If it is
STOPPING/TERMINATED it raises `EncodingJobLostError`, so
`run_with_lost_job_resubmit` resubmits the encode, and that resubmit
restarts the Spot VM or falls back to c2df. Before, the failure showed
up after ~7 min as a "lost contact" error that was not resubmitted.
- **Stale fallback lists**: a candidate VM that no longer exists
(`NotFound`) is skipped instead of aborting the whole failover. This
protects the window before running instances pick up the new secret.
- `SPEED_RANK` is now keyed by machine family, so the ranking is
independent of VM size.
- **Idle-stop vs. in-flight encode** (review follow-up): the idle
function's `/health active_jobs` only counts RUNNING jobs, and
`last_activity_at` was only written when a VM was started. With a 5-min
window, a long encode's VM could be stopped in the seconds between the
job finishing and the next poll. The poll loop now refreshes
`last_activity_at` at most once a minute. A VM seen in
STAGING/PROVISIONING after it accepted a job also counts as lost. CI
`Read Firestore config` now writes empty values instead of "None".
- **Worker deploy gate**:
`backend/tests/test_encoding_worker_code_paths.py` derives the worker's
transitive `backend.*` import set and fails if `worker_code_paths.txt`
doesn't cover it. Every deploy still uploads the wheel and `startup.sh`,
so a cold-booted VM always runs the latest code.
- **Dropbox**: nothing automated reads lossless files from Dropbox. The
lossless **MKV** is still uploaded because the completion email and the
live Fiverr auto-reply bot both tell customers it is in the Dropbox
folder, and the VocalStar tenant expects lossless files. To drop the MKV
as well, update those two texts first, then add the MKV suffix to
`DROPBOX_SKIP_OUTPUT_SUFFIXES`.

## Testing
- New/updated unit tests: preemption → lost job → resubmit, NotFound
candidate skip, family ranking, idle 5-min/heartbeat window, fleet-shape
invariants, worker-path coverage, Dropbox exclusion, keep-alive hook
(Jest, fake timers).
- `pytest backend/tests --ignore=backend/tests/emulator`: 4802 passed.
The only failures are 6 tests that need the spaCy `en_core_web_sm`
model, which isn't installed in my local venv; unrelated to this PR.
- Jest lyrics-review + lib/lyrics-review: 423 passed. Infra tests
(`test_encoding_worker_config.py`, idle function): all pass.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-26 23:20:54 -04:00

1189 lines
52 KiB
Python

"""
Tests for the blue-green encoding worker manager service.
Tests cover:
- Task 1: Firestore config document schema, get_config, update_activity, swap_roles
- Task 2: VM lifecycle (get_vm_status, start_vm, stop_vm, ensure_primary_running)
"""
import pytest
import aiohttp
from datetime import datetime, UTC
from unittest.mock import MagicMock, patch, call, AsyncMock
from backend.services.encoding_worker_manager import (
EncodingWorkerCandidate,
EncodingWorkerConfig,
EncodingWorkerManager,
CONFIG_COLLECTION,
CONFIG_DOCUMENT,
ENCODING_WORKER_PORT,
)
from backend.services.encoding_errors import (
EncodingWorkerCapacityError,
EncodingWorkerStartError,
)
# ---------------------------------------------------------------------------
# Fixtures
# ---------------------------------------------------------------------------
def _make_firestore_data(**overrides):
"""Create a realistic Firestore document dict for encoding-worker config."""
data = {
"primary_vm": "encoding-worker-blue",
"primary_ip": "10.128.0.50",
"primary_version": "1.0.0",
"primary_deployed_at": "2026-03-20T10:00:00Z",
"secondary_vm": "encoding-worker-green",
"secondary_ip": "10.128.0.51",
"secondary_version": "0.9.0",
"secondary_deployed_at": "2026-03-19T08:00:00Z",
"last_swap_at": "2026-03-20T10:00:00Z",
"last_activity_at": "2026-03-24T12:00:00Z",
"deploy_in_progress": False,
"deploy_in_progress_since": None,
}
data.update(overrides)
return data
@pytest.fixture
def mock_db():
"""Return a mock Firestore client with a pre-configured config document."""
db = MagicMock()
snapshot = MagicMock()
snapshot.exists = True
snapshot.to_dict.return_value = _make_firestore_data()
doc_ref = MagicMock()
doc_ref.get.return_value = snapshot
collection_ref = MagicMock()
collection_ref.document.return_value = doc_ref
db.collection.return_value = collection_ref
return db
@pytest.fixture
def mock_compute():
"""Return a mock Compute Engine InstancesClient."""
return MagicMock()
@pytest.fixture
def manager(mock_db, mock_compute):
return EncodingWorkerManager(
db=mock_db,
compute_client=mock_compute,
project_id="nomadkaraoke",
zone="us-central1-c",
)
# ---------------------------------------------------------------------------
# Task 1: Config dataclass
# ---------------------------------------------------------------------------
class TestEncodingWorkerConfig:
"""Tests for the EncodingWorkerConfig dataclass."""
def test_primary_url(self):
config = EncodingWorkerConfig(
primary_vm="blue", primary_ip="10.0.0.1", primary_version="1.0",
primary_deployed_at=None,
secondary_vm="green", secondary_ip="10.0.0.2",
secondary_version=None, secondary_deployed_at=None,
last_swap_at=None, last_activity_at=None,
deploy_in_progress=False, deploy_in_progress_since=None,
)
assert config.primary_url == "http://10.0.0.1:8080"
def test_secondary_url(self):
config = EncodingWorkerConfig(
primary_vm="blue", primary_ip="10.0.0.1", primary_version="1.0",
primary_deployed_at=None,
secondary_vm="green", secondary_ip="10.0.0.2",
secondary_version=None, secondary_deployed_at=None,
last_swap_at=None, last_activity_at=None,
deploy_in_progress=False, deploy_in_progress_since=None,
)
assert config.secondary_url == "http://10.0.0.2:8080"
# ---------------------------------------------------------------------------
# Task 1: get_config / update_activity / swap_roles
# ---------------------------------------------------------------------------
class TestEncodingWorkerManager:
"""Tests for Firestore config operations."""
def test_get_config_returns_dataclass(self, manager, mock_db):
config = manager.get_config()
assert isinstance(config, EncodingWorkerConfig)
assert config.primary_vm == "encoding-worker-blue"
assert config.primary_ip == "10.128.0.50"
assert config.primary_version == "1.0.0"
assert config.secondary_vm == "encoding-worker-green"
assert config.deploy_in_progress is False
def test_get_config_document_not_found(self, manager, mock_db):
snapshot = MagicMock()
snapshot.exists = False
mock_db.collection.return_value.document.return_value.get.return_value = snapshot
with pytest.raises(ValueError, match="not found"):
manager.get_config()
def test_get_config_reads_correct_path(self, manager, mock_db):
manager.get_config()
mock_db.collection.assert_called_with(CONFIG_COLLECTION)
mock_db.collection.return_value.document.assert_called_with(CONFIG_DOCUMENT)
def test_update_activity_sets_timestamp(self, manager, mock_db):
with patch("backend.services.encoding_worker_manager.datetime") as mock_dt:
mock_dt.now.return_value = datetime(2026, 3, 24, 15, 0, 0, tzinfo=UTC)
mock_dt.side_effect = lambda *a, **kw: datetime(*a, **kw)
manager.update_activity()
doc_ref = mock_db.collection.return_value.document.return_value
doc_ref.update.assert_called_once_with({
"last_activity_at": "2026-03-24T15:00:00+00:00",
})
def test_swap_roles_uses_transaction(self, manager, mock_db):
"""swap_roles should use a Firestore transaction for atomicity."""
# Set up the transaction mock
mock_transaction = MagicMock()
mock_db.transaction.return_value = mock_transaction
# The transactional function is called with the transaction
manager.swap_roles(
new_primary_vm="encoding-worker-green",
new_primary_ip="10.128.0.51",
new_primary_version="2.0.0",
old_primary_vm="encoding-worker-blue",
old_primary_ip="10.128.0.50",
old_primary_version="1.0.0",
)
# Verify transaction was created and the doc was updated within it
mock_db.transaction.assert_called_once()
def test_swap_roles_writes_correct_data(self, manager, mock_db):
"""swap_roles should write the swapped primary/secondary fields."""
mock_transaction = MagicMock()
mock_db.transaction.return_value = mock_transaction
with patch("backend.services.encoding_worker_manager.datetime") as mock_dt:
mock_dt.now.return_value = datetime(2026, 3, 24, 16, 0, 0, tzinfo=UTC)
mock_dt.side_effect = lambda *a, **kw: datetime(*a, **kw)
manager.swap_roles(
new_primary_vm="encoding-worker-green",
new_primary_ip="10.128.0.51",
new_primary_version="2.0.0",
old_primary_vm="encoding-worker-blue",
old_primary_ip="10.128.0.50",
old_primary_version="1.0.0",
)
# The transaction.update call should contain the swapped values
doc_ref = mock_db.collection.return_value.document.return_value
update_call = mock_transaction.update.call_args
assert update_call is not None
data = update_call[0][1] # second positional arg is the data dict
assert data["primary_vm"] == "encoding-worker-green"
assert data["primary_ip"] == "10.128.0.51"
assert data["primary_version"] == "2.0.0"
assert data["secondary_vm"] == "encoding-worker-blue"
assert data["secondary_ip"] == "10.128.0.50"
assert data["secondary_version"] == "1.0.0"
assert data["last_swap_at"] == "2026-03-24T16:00:00+00:00"
# ---------------------------------------------------------------------------
# Task 2: VM lifecycle
# ---------------------------------------------------------------------------
class TestVMLifecycle:
"""Tests for Compute Engine VM operations."""
def test_get_vm_status_returns_status(self, manager, mock_compute):
mock_instance = MagicMock()
mock_instance.status = "RUNNING"
mock_compute.get.return_value = mock_instance
status = manager.get_vm_status("encoding-worker-blue")
assert status == "RUNNING"
mock_compute.get.assert_called_once_with(
project="nomadkaraoke",
zone="us-central1-c",
instance="encoding-worker-blue",
)
def test_get_vm_status_terminated(self, manager, mock_compute):
mock_instance = MagicMock()
mock_instance.status = "TERMINATED"
mock_compute.get.return_value = mock_instance
assert manager.get_vm_status("encoding-worker-blue") == "TERMINATED"
def test_start_vm_calls_compute_start(self, manager, mock_compute):
mock_instance = MagicMock()
mock_instance.status = "TERMINATED"
mock_compute.get.return_value = mock_instance
# Successful start: operation has no error_code/error_message and
# .result() returns without raising.
op = MagicMock()
op.error_code = ""
op.error_message = ""
op.result.return_value = None
mock_compute.start.return_value = op
manager.start_vm("encoding-worker-blue")
mock_compute.start.assert_called_once_with(
project="nomadkaraoke",
zone="us-central1-c",
instance="encoding-worker-blue",
)
op.result.assert_called_once()
def test_start_vm_skips_if_already_running(self, manager, mock_compute):
mock_instance = MagicMock()
mock_instance.status = "RUNNING"
mock_compute.get.return_value = mock_instance
manager.start_vm("encoding-worker-blue")
mock_compute.start.assert_not_called()
def test_is_transient_start_error_classification(self):
from backend.services.encoding_worker_manager import _is_transient_start_error
from backend.services.encoding_errors import EncodingWorkerStartError
assert _is_transient_start_error(
EncodingWorkerStartError("VM x start failed: 503 — SERVICE UNAVAILABLE", code="503")
)
assert not _is_transient_start_error(
EncodingWorkerStartError("VM x start failed: PERMISSION_DENIED", code="PERMISSION_DENIED")
)
def test_stop_vm_calls_compute_stop(self, manager, mock_compute):
mock_instance = MagicMock()
mock_instance.status = "RUNNING"
mock_compute.get.return_value = mock_instance
manager.stop_vm("encoding-worker-blue")
mock_compute.stop.assert_called_once_with(
project="nomadkaraoke",
zone="us-central1-c",
instance="encoding-worker-blue",
)
def test_stop_vm_skips_if_already_terminated(self, manager, mock_compute):
mock_instance = MagicMock()
mock_instance.status = "TERMINATED"
mock_compute.get.return_value = mock_instance
manager.stop_vm("encoding-worker-blue")
mock_compute.stop.assert_not_called()
def test_ensure_primary_running_starts_stopped_vm(self, manager, mock_db, mock_compute):
"""When primary is stopped, ensure_primary_running should start it."""
mock_instance = MagicMock()
mock_instance.status = "TERMINATED"
mock_compute.get.return_value = mock_instance
op = MagicMock()
op.error_code = ""
op.error_message = ""
op.result.return_value = None
mock_compute.start.return_value = op
with patch("backend.services.encoding_worker_manager.datetime") as mock_dt:
mock_dt.now.return_value = datetime(2026, 3, 24, 15, 0, 0, tzinfo=UTC)
mock_dt.side_effect = lambda *a, **kw: datetime(*a, **kw)
result = manager.ensure_primary_running()
assert result["started"] is True
assert result["vm_name"] == "encoding-worker-blue"
assert result["primary_url"] == "http://10.128.0.50:8080"
mock_compute.start.assert_called_once()
op.result.assert_called_once()
def test_ensure_primary_running_already_running(self, manager, mock_db, mock_compute):
"""When primary is already running, should not start and still update activity."""
mock_instance = MagicMock()
mock_instance.status = "RUNNING"
mock_compute.get.return_value = mock_instance
with patch("backend.services.encoding_worker_manager.datetime") as mock_dt:
mock_dt.now.return_value = datetime(2026, 3, 24, 15, 0, 0, tzinfo=UTC)
mock_dt.side_effect = lambda *a, **kw: datetime(*a, **kw)
result = manager.ensure_primary_running()
assert result["started"] is False
assert result["vm_name"] == "encoding-worker-blue"
assert result["primary_url"] == "http://10.128.0.50:8080"
mock_compute.start.assert_not_called()
def test_ensure_primary_running_updates_activity(self, manager, mock_db, mock_compute):
"""ensure_primary_running should always update last_activity_at."""
mock_instance = MagicMock()
mock_instance.status = "RUNNING"
mock_compute.get.return_value = mock_instance
with patch("backend.services.encoding_worker_manager.datetime") as mock_dt:
mock_dt.now.return_value = datetime(2026, 3, 24, 15, 0, 0, tzinfo=UTC)
mock_dt.side_effect = lambda *a, **kw: datetime(*a, **kw)
manager.ensure_primary_running()
doc_ref = mock_db.collection.return_value.document.return_value
doc_ref.update.assert_called_once()
def test_start_vm_skips_if_staging(self, manager, mock_compute):
"""start_vm should also skip if VM is in STAGING state (already starting)."""
mock_instance = MagicMock()
mock_instance.status = "STAGING"
mock_compute.get.return_value = mock_instance
manager.start_vm("encoding-worker-blue")
mock_compute.start.assert_not_called()
def test_stop_vm_skips_if_stopping(self, manager, mock_compute):
"""stop_vm should skip if VM is already in STOPPING state."""
mock_instance = MagicMock()
mock_instance.status = "STOPPING"
mock_compute.get.return_value = mock_instance
manager.stop_vm("encoding-worker-blue")
mock_compute.stop.assert_not_called()
# ------------------------------------------------------------------
# Operation-result inspection (capacity / start error classification)
# ------------------------------------------------------------------
def test_start_vm_raises_capacity_error_when_zone_exhausted(self, manager, mock_compute):
"""start_vm must surface ZONE_RESOURCE_POOL_EXHAUSTED as a typed exception.
Without this, the GCE error stays hidden behind a fire-and-forget call
and the readiness gate eventually times out with no useful message.
"""
mock_instance = MagicMock()
mock_instance.status = "TERMINATED"
mock_compute.get.return_value = mock_instance
op = MagicMock()
op.error_code = "ZONE_RESOURCE_POOL_EXHAUSTED"
op.error_message = "The zone does not have enough resources available"
op.result.return_value = None
mock_compute.start.return_value = op
with pytest.raises(EncodingWorkerCapacityError) as exc_info:
manager.start_vm("encoding-worker-blue")
assert exc_info.value.code == "ZONE_RESOURCE_POOL_EXHAUSTED"
assert exc_info.value.vm_name == "encoding-worker-blue"
assert exc_info.value.zone == "us-central1-c"
assert "ZONE_RESOURCE_POOL_EXHAUSTED" in str(exc_info.value)
def test_start_vm_raises_generic_start_error_for_other_codes(self, manager, mock_compute):
"""start_vm raises EncodingWorkerStartError for non-capacity GCE errors."""
mock_instance = MagicMock()
mock_instance.status = "TERMINATED"
mock_compute.get.return_value = mock_instance
op = MagicMock()
op.error_code = "RESOURCE_NOT_READY"
op.error_message = "VM is in transition"
op.result.return_value = None
mock_compute.start.return_value = op
with pytest.raises(EncodingWorkerStartError) as exc_info:
manager.start_vm("encoding-worker-blue")
assert not isinstance(exc_info.value, EncodingWorkerCapacityError)
assert exc_info.value.code == "RESOURCE_NOT_READY"
def test_start_vm_raises_when_operation_result_throws(self, manager, mock_compute):
"""If operation.result() raises (e.g. timeout), surface as a start error."""
mock_instance = MagicMock()
mock_instance.status = "TERMINATED"
mock_compute.get.return_value = mock_instance
op = MagicMock()
op.error_code = ""
op.error_message = ""
op.result.side_effect = TimeoutError("operation timed out")
mock_compute.start.return_value = op
with pytest.raises(EncodingWorkerStartError) as exc_info:
manager.start_vm("encoding-worker-blue")
assert "timed out" in str(exc_info.value).lower() or "operation" in str(exc_info.value).lower()
def test_ensure_primary_running_raises_capacity_error(
self, manager, mock_db, mock_compute
):
"""ensure_primary_running propagates capacity errors so callers can react."""
mock_instance = MagicMock()
mock_instance.status = "TERMINATED"
mock_compute.get.return_value = mock_instance
op = MagicMock()
op.error_code = "ZONE_RESOURCE_POOL_EXHAUSTED"
op.error_message = "out of capacity"
op.result.return_value = None
mock_compute.start.return_value = op
with patch("backend.services.encoding_worker_manager.datetime") as mock_dt:
mock_dt.now.return_value = datetime(2026, 5, 5, 9, 0, 0, tzinfo=UTC)
mock_dt.side_effect = lambda *a, **kw: datetime(*a, **kw)
with pytest.raises(EncodingWorkerCapacityError):
manager.ensure_primary_running()
# ---------------------------------------------------------------------------
# Phase 3: multi-zone failover (ensure_any_running)
# ---------------------------------------------------------------------------
def _ok_op():
"""Build a successful compute Operation mock."""
op = MagicMock()
op.error_code = ""
op.error_message = ""
op.result.return_value = None
return op
def _capacity_op():
"""Build a ZONE_RESOURCE_POOL_EXHAUSTED Operation mock."""
op = MagicMock()
op.error_code = "ZONE_RESOURCE_POOL_EXHAUSTED"
op.error_message = "out of capacity"
op.result.return_value = None
return op
class TestEnsureAnyRunning:
"""Tests for ensure_any_running() — capacity-fallback iteration."""
def test_starts_first_candidate_when_available(self, manager, mock_db, mock_compute):
"""When the primary candidate has capacity, it's used and no fallback engaged."""
mock_instance = MagicMock()
mock_instance.status = "TERMINATED"
mock_compute.get.return_value = mock_instance
mock_compute.start.return_value = _ok_op()
candidates = [
EncodingWorkerCandidate(vm_name="encoding-worker-blue", zone="us-central1-c", ip="10.0.0.1"),
EncodingWorkerCandidate(vm_name="encoding-worker-fb-a", zone="us-central1-a", ip="10.0.0.2"),
]
with patch("backend.services.encoding_worker_manager.datetime") as mock_dt:
mock_dt.now.return_value = datetime(2026, 5, 5, 9, 0, 0, tzinfo=UTC)
mock_dt.side_effect = lambda *a, **kw: datetime(*a, **kw)
result = manager.ensure_any_running(candidates)
assert result["fell_back"] is False
assert result["vm_name"] == "encoding-worker-blue"
assert result["zone"] == "us-central1-c"
# Only the first candidate's start was attempted.
mock_compute.start.assert_called_once()
def test_skips_deleted_candidate_and_starts_next(self, manager, mock_db, mock_compute):
"""A candidate VM that no longer exists (removed from the fleet while an
instance still holds a stale fallback list) is skipped, not fatal."""
from google.api_core import exceptions as gcp_exceptions
stopped = MagicMock()
stopped.status = "TERMINATED"
def _get(project, zone, instance):
if instance == "encoding-worker-fallback-gone":
raise gcp_exceptions.NotFound("instance not found")
return stopped
mock_compute.get.side_effect = _get
# Primary: Spot capacity exhausted. Deleted fallback: NotFound. Live fallback: ok.
mock_compute.start.side_effect = [_capacity_op(), _ok_op()]
candidates = [
EncodingWorkerCandidate(vm_name="encoding-worker-blue", zone="us-central1-c",
ip="10.0.0.1", is_primary=True),
EncodingWorkerCandidate(vm_name="encoding-worker-fallback-gone", zone="us-central1-a",
ip="10.0.0.2"),
EncodingWorkerCandidate(vm_name="encoding-worker-fallback-c2df", zone="us-central1-f",
ip="10.0.0.3"),
]
with patch("backend.services.encoding_worker_manager.datetime") as mock_dt:
mock_dt.now.return_value = datetime(2026, 9, 26, 9, 0, 0, tzinfo=UTC)
mock_dt.side_effect = lambda *a, **kw: datetime(*a, **kw)
result = manager.ensure_any_running(candidates)
assert result["vm_name"] == "encoding-worker-fallback-c2df"
assert result["fell_back"] is True
# The deleted VM was never asked to start.
started = [c.kwargs["instance"] for c in mock_compute.start.call_args_list]
assert started == ["encoding-worker-blue", "encoding-worker-fallback-c2df"]
def test_all_candidates_missing_raises_start_error(self, manager, mock_compute):
"""If every candidate is gone, surface a typed start error (not a bare NotFound)."""
from google.api_core import exceptions as gcp_exceptions
mock_compute.get.side_effect = gcp_exceptions.NotFound("gone")
candidates = [
EncodingWorkerCandidate(vm_name="vm-x", zone="us-central1-a", ip="10.0.0.9"),
]
with pytest.raises(EncodingWorkerStartError) as exc_info:
manager.ensure_any_running(candidates)
assert exc_info.value.code == "NOT_FOUND"
mock_compute.start.assert_not_called()
def test_falls_through_to_next_zone_on_capacity_error(self, manager, mock_db, mock_compute):
"""Capacity error on candidate 1 triggers a try of candidate 2."""
mock_instance = MagicMock()
mock_instance.status = "TERMINATED"
mock_compute.get.return_value = mock_instance
# First start: capacity error. Second start: succeeds.
mock_compute.start.side_effect = [_capacity_op(), _ok_op()]
candidates = [
EncodingWorkerCandidate(vm_name="encoding-worker-blue", zone="us-central1-c", ip="10.0.0.1"),
EncodingWorkerCandidate(vm_name="encoding-worker-fb-a", zone="us-central1-a", ip="10.0.0.2"),
]
with patch("backend.services.encoding_worker_manager.datetime") as mock_dt:
mock_dt.now.return_value = datetime(2026, 5, 5, 9, 0, 0, tzinfo=UTC)
mock_dt.side_effect = lambda *a, **kw: datetime(*a, **kw)
result = manager.ensure_any_running(candidates)
assert result["fell_back"] is True
assert result["vm_name"] == "encoding-worker-fb-a"
assert result["zone"] == "us-central1-a"
# Both starts were attempted, in order.
assert mock_compute.start.call_count == 2
# Active override was persisted in Firestore.
doc_ref = mock_db.collection.return_value.document.return_value
# Multiple update calls (for activity + override) — find the override one
override_calls = [
c for c in doc_ref.update.call_args_list
if "active_override_vm" in c.args[0]
]
assert override_calls, "Expected Firestore update with active_override_vm"
override_data = override_calls[0].args[0]
assert override_data["active_override_vm"] == "encoding-worker-fb-a"
assert override_data["active_override_zone"] == "us-central1-a"
assert override_data["active_override_ip"] == "10.0.0.2"
def test_raises_when_all_candidates_exhausted(self, manager, mock_db, mock_compute):
"""If every candidate hits capacity error, the last one is re-raised."""
from backend.services.encoding_errors import EncodingWorkerCapacityError
mock_instance = MagicMock()
mock_instance.status = "TERMINATED"
mock_compute.get.return_value = mock_instance
mock_compute.start.side_effect = [_capacity_op(), _capacity_op(), _capacity_op()]
candidates = [
EncodingWorkerCandidate(vm_name="vm-c", zone="us-central1-c", ip="1.1.1.1"),
EncodingWorkerCandidate(vm_name="vm-a", zone="us-central1-a", ip="1.1.1.2"),
EncodingWorkerCandidate(vm_name="vm-f", zone="us-central1-f", ip="1.1.1.3"),
]
with pytest.raises(EncodingWorkerCapacityError):
manager.ensure_any_running(candidates)
# All three were attempted before giving up.
assert mock_compute.start.call_count == 3
def test_falls_through_on_generic_start_error_too(self, manager, mock_db, mock_compute):
"""Non-capacity start errors (e.g. 503 SERVICE_UNAVAILABLE) also trigger fallback.
Observed in prod 2026-05-06: GCE returned 503 from instances.start
for the primary VM. The original ensure_any_running only fell through
on EncodingWorkerCapacityError, so the 503 short-circuited multi-zone
failover. Broaden to catch the parent class.
"""
from backend.services.encoding_errors import EncodingWorkerStartError
mock_instance = MagicMock()
mock_instance.status = "TERMINATED"
mock_compute.get.return_value = mock_instance
# First start: 503 SERVICE_UNAVAILABLE (generic start error). Second: succeeds.
op_503 = MagicMock()
op_503.error_code = "503"
op_503.error_message = "SERVICE UNAVAILABLE"
op_503.result.return_value = None
mock_compute.start.side_effect = [op_503, _ok_op()]
candidates = [
EncodingWorkerCandidate(vm_name="primary", zone="us-central1-c", ip="10.0.0.1"),
EncodingWorkerCandidate(vm_name="fb-a", zone="us-central1-a", ip="10.0.0.2"),
]
with patch("backend.services.encoding_worker_manager.datetime") as mock_dt:
mock_dt.now.return_value = datetime(2026, 5, 6, 17, 0, 0, tzinfo=UTC)
mock_dt.side_effect = lambda *a, **kw: datetime(*a, **kw)
result = manager.ensure_any_running(candidates)
assert result["fell_back"] is True
assert result["vm_name"] == "fb-a"
assert mock_compute.start.call_count == 2
def test_raises_last_start_error_when_no_capacity_errors(self, manager, mock_db, mock_compute):
"""If every candidate fails with generic start errors (no capacity), surface the last one."""
from backend.services.encoding_errors import (
EncodingWorkerCapacityError, EncodingWorkerStartError,
)
mock_instance = MagicMock()
mock_instance.status = "TERMINATED"
mock_compute.get.return_value = mock_instance
def _op_503():
op = MagicMock()
op.error_code = "503"
op.error_message = "SERVICE UNAVAILABLE"
op.result.return_value = None
return op
# All-transient: the whole candidate set is retried a few times before
# giving up, so provide enough operations for every pass.
mock_compute.start.side_effect = [_op_503() for _ in range(20)]
candidates = [
EncodingWorkerCandidate(vm_name="primary", zone="us-central1-c", ip="10.0.0.1"),
EncodingWorkerCandidate(vm_name="fb-a", zone="us-central1-a", ip="10.0.0.2"),
]
with patch("backend.services.encoding_worker_manager.time.sleep") as sleep:
with pytest.raises(EncodingWorkerStartError) as exc_info:
manager.ensure_any_running(candidates)
# Should NOT be a capacity error — there were none in this scenario.
assert not isinstance(exc_info.value, EncodingWorkerCapacityError)
# Retried the whole 2-candidate set across multiple passes before failing.
assert mock_compute.start.call_count > 2
assert sleep.called
def test_retries_whole_set_on_all_transient_then_succeeds(self, manager, mock_db, mock_compute):
"""A brief all-zones 503 blip: first pass fails everywhere, retry succeeds."""
mock_instance = MagicMock()
mock_instance.status = "TERMINATED"
mock_compute.get.return_value = mock_instance
def _op_503():
op = MagicMock(); op.error_code = "503"; op.error_message = "SERVICE UNAVAILABLE"
op.result.return_value = None
return op
# Pass 1: both candidates 503. Pass 2: primary succeeds.
mock_compute.start.side_effect = [_op_503(), _op_503(), _ok_op()]
candidates = [
EncodingWorkerCandidate(vm_name="primary", zone="us-central1-c", ip="10.0.0.1"),
EncodingWorkerCandidate(vm_name="fb-a", zone="us-central1-a", ip="10.0.0.2"),
]
with patch("backend.services.encoding_worker_manager.time.sleep") as sleep, \
patch("backend.services.encoding_worker_manager.datetime") as mock_dt:
mock_dt.now.return_value = datetime(2026, 5, 6, 17, 0, 0, tzinfo=UTC)
mock_dt.side_effect = lambda *a, **kw: datetime(*a, **kw)
result = manager.ensure_any_running(candidates)
assert result["vm_name"] == "primary"
assert result["fell_back"] is False
assert mock_compute.start.call_count == 3
sleep.assert_called_once()
def test_does_not_retry_whole_set_on_mixed_nontransient(self, manager, mock_db, mock_compute):
"""If any candidate fails non-transiently (PERMISSION_DENIED), do NOT retry
the whole set even though the last failure was a transient 503."""
from backend.services.encoding_errors import EncodingWorkerStartError
mock_instance = MagicMock()
mock_instance.status = "TERMINATED"
mock_compute.get.return_value = mock_instance
op_perm = MagicMock()
op_perm.error_code = "PERMISSION_DENIED"; op_perm.error_message = "denied"
op_perm.result.return_value = None
op_503 = MagicMock()
op_503.error_code = "503"; op_503.error_message = "SERVICE UNAVAILABLE"
op_503.result.return_value = None
# primary: PERMISSION_DENIED (non-transient); fallback: 503 (transient/last)
mock_compute.start.side_effect = [op_perm, op_503]
candidates = [
EncodingWorkerCandidate(vm_name="primary", zone="us-central1-c", ip="10.0.0.1"),
EncodingWorkerCandidate(vm_name="fb-a", zone="us-central1-a", ip="10.0.0.2"),
]
with patch("backend.services.encoding_worker_manager.time.sleep") as sleep:
with pytest.raises(EncodingWorkerStartError) as exc:
manager.ensure_any_running(candidates)
assert exc.value.code == "PERMISSION_DENIED" # non-transient surfaced
assert mock_compute.start.call_count == 2 # one pass only
sleep.assert_not_called()
def test_does_not_retry_whole_set_on_capacity(self, manager, mock_db, mock_compute):
"""Capacity exhaustion is not spun on — it raises immediately for the park path."""
from backend.services.encoding_errors import EncodingWorkerCapacityError
mock_instance = MagicMock()
mock_instance.status = "TERMINATED"
mock_compute.get.return_value = mock_instance
mock_compute.start.side_effect = [_capacity_op(), _capacity_op()]
candidates = [
EncodingWorkerCandidate(vm_name="primary", zone="us-central1-c", ip="10.0.0.1"),
EncodingWorkerCandidate(vm_name="fb-a", zone="us-central1-a", ip="10.0.0.2"),
]
with patch("backend.services.encoding_worker_manager.time.sleep") as sleep:
with pytest.raises(EncodingWorkerCapacityError):
manager.ensure_any_running(candidates)
assert mock_compute.start.call_count == 2 # one pass only, no whole-set retry
sleep.assert_not_called()
def test_prefers_capacity_error_when_mixed_failures(self, manager, mock_db, mock_compute):
"""When some candidates hit capacity and others hit generic errors, prefer raising
the capacity error — it's more actionable for the user-facing message."""
from backend.services.encoding_errors import EncodingWorkerCapacityError
mock_instance = MagicMock()
mock_instance.status = "TERMINATED"
mock_compute.get.return_value = mock_instance
op_503 = MagicMock()
op_503.error_code = "503"
op_503.error_message = "SERVICE UNAVAILABLE"
op_503.result.return_value = None
# First: capacity. Second: 503.
mock_compute.start.side_effect = [_capacity_op(), op_503]
candidates = [
EncodingWorkerCandidate(vm_name="primary", zone="us-central1-c", ip="10.0.0.1"),
EncodingWorkerCandidate(vm_name="fb-a", zone="us-central1-a", ip="10.0.0.2"),
]
with pytest.raises(EncodingWorkerCapacityError):
manager.ensure_any_running(candidates)
def test_clears_override_when_primary_succeeds(self, manager, mock_db, mock_compute):
"""When the primary starts successfully, any stale active_override is cleared."""
mock_instance = MagicMock()
mock_instance.status = "TERMINATED"
mock_compute.get.return_value = mock_instance
mock_compute.start.return_value = _ok_op()
# Pretend Firestore has an override set from a prior fallback.
snapshot = MagicMock()
snapshot.exists = True
snapshot.to_dict.return_value = {
"active_override_vm": "encoding-worker-fb-a",
"active_override_ip": "10.0.0.2",
"active_override_zone": "us-central1-a",
}
mock_db.collection.return_value.document.return_value.get.return_value = snapshot
candidates = [
EncodingWorkerCandidate(vm_name="encoding-worker-blue", zone="us-central1-c", ip="10.0.0.1"),
]
with patch("backend.services.encoding_worker_manager.datetime") as mock_dt:
mock_dt.now.return_value = datetime(2026, 5, 5, 9, 0, 0, tzinfo=UTC)
mock_dt.side_effect = lambda *a, **kw: datetime(*a, **kw)
result = manager.ensure_any_running(candidates)
assert result["fell_back"] is False
# Verify the clear-override write happened.
doc_ref = mock_db.collection.return_value.document.return_value
clear_calls = [
c for c in doc_ref.update.call_args_list
if c.args[0].get("active_override_vm") is None and "active_override_vm" in c.args[0]
]
assert clear_calls, "Expected an update call clearing active_override_vm to None"
def test_empty_candidates_raises_value_error(self, manager):
with pytest.raises(ValueError):
manager.ensure_any_running([])
class TestActiveUrl:
"""Tests for EncodingWorkerConfig.active_url override behavior."""
def test_active_url_returns_primary_when_no_override(self):
config = EncodingWorkerConfig(
primary_vm="vm-a", primary_ip="10.0.0.1", primary_version="1.0.0",
primary_deployed_at=None,
secondary_vm="vm-b", secondary_ip="10.0.0.2", secondary_version=None,
secondary_deployed_at=None,
last_swap_at=None, last_activity_at=None,
deploy_in_progress=False, deploy_in_progress_since=None,
)
assert config.active_url == f"http://10.0.0.1:{ENCODING_WORKER_PORT}"
def test_active_url_returns_override_when_set(self):
config = EncodingWorkerConfig(
primary_vm="vm-a", primary_ip="10.0.0.1", primary_version="1.0.0",
primary_deployed_at=None,
secondary_vm="vm-b", secondary_ip="10.0.0.2", secondary_version=None,
secondary_deployed_at=None,
last_swap_at=None, last_activity_at=None,
deploy_in_progress=False, deploy_in_progress_since=None,
active_override_vm="vm-fb-a",
active_override_ip="34.5.6.7",
active_override_zone="us-central1-a",
active_override_set_at="2026-05-05T09:00:00+00:00",
)
assert config.active_url == f"http://34.5.6.7:{ENCODING_WORKER_PORT}"
# ---------------------------------------------------------------------------
# Task 3: wait_for_worker_ready (cold-start readiness gate)
# ---------------------------------------------------------------------------
class TestWaitForWorkerReady:
"""Tests for the wait_for_worker_ready() readiness gate."""
# ------------------------------------------------------------------
# Helpers
# ------------------------------------------------------------------
@staticmethod
def _mk_health_session(get_callable):
"""Build a mock ClientSession whose .get(url, ...) returns an async-CM Resp."""
mock_session = MagicMock()
mock_session.get = get_callable
mock_session.__aenter__ = AsyncMock(return_value=mock_session)
mock_session.__aexit__ = AsyncMock(return_value=None)
return mock_session
@staticmethod
def _mk_resp_cm(status):
"""Return a callable that, when invoked with a URL, returns an async-CM Resp."""
class _Resp:
def __init__(self, s):
self.status = s
async def __aenter__(self): return self
async def __aexit__(self, *a): return None
def _get(url, **kwargs):
return _Resp(status)
return _get
# ------------------------------------------------------------------
# Tests
# ------------------------------------------------------------------
@pytest.mark.asyncio
async def test_returns_immediately_when_vm_running_and_healthy(self, manager, mock_compute):
"""Happy path: VM already RUNNING, /health returns 200 on first poll."""
mock_instance = MagicMock()
mock_instance.status = "RUNNING"
mock_compute.get.return_value = mock_instance
mock_session = self._mk_health_session(self._mk_resp_cm(200))
with patch("backend.services.encoding_worker_manager.aiohttp.ClientSession", return_value=mock_session):
await manager.wait_for_worker_ready(
"encoding-worker-blue",
"http://10.0.0.1:8080/health",
vm_timeout=2.0,
health_timeout=2.0,
poll_interval=0.05,
)
# Single VM status check, single health check
assert mock_compute.get.call_count == 1
@pytest.mark.asyncio
async def test_polls_vm_until_running(self, manager, mock_compute):
"""Polls VM status through STAGING → RUNNING."""
statuses = ["STAGING", "STAGING", "RUNNING"]
instances = [MagicMock(status=s) for s in statuses]
mock_compute.get.side_effect = instances
mock_session = self._mk_health_session(self._mk_resp_cm(200))
with patch("backend.services.encoding_worker_manager.aiohttp.ClientSession", return_value=mock_session):
await manager.wait_for_worker_ready(
"encoding-worker-blue",
"http://10.0.0.1:8080/health",
vm_timeout=2.0,
health_timeout=2.0,
poll_interval=0.05,
)
assert mock_compute.get.call_count == 3
@pytest.mark.asyncio
async def test_times_out_when_vm_never_runs(self, manager, mock_compute):
"""Raises TimeoutError if VM stays in STAGING past vm_timeout."""
mock_instance = MagicMock()
mock_instance.status = "STAGING"
mock_compute.get.return_value = mock_instance
with pytest.raises(TimeoutError, match="did not reach RUNNING"):
await manager.wait_for_worker_ready(
"encoding-worker-blue",
"http://10.0.0.1:8080/health",
vm_timeout=0.2,
health_timeout=2.0,
poll_interval=0.05,
)
@pytest.mark.asyncio
async def test_times_out_when_health_never_200(self, manager, mock_compute):
"""Raises TimeoutError if /health never returns 200 past health_timeout."""
mock_instance = MagicMock()
mock_instance.status = "RUNNING"
mock_compute.get.return_value = mock_instance
mock_session = self._mk_health_session(self._mk_resp_cm(503))
with patch("backend.services.encoding_worker_manager.aiohttp.ClientSession", return_value=mock_session):
with pytest.raises(TimeoutError, match="did not become healthy"):
await manager.wait_for_worker_ready(
"encoding-worker-blue",
"http://10.0.0.1:8080/health",
vm_timeout=2.0,
health_timeout=0.2,
poll_interval=0.05,
)
@pytest.mark.asyncio
async def test_tolerates_health_connection_errors(self, manager, mock_compute):
"""Connection errors during /health polling are treated as 'not ready', keep polling."""
mock_instance = MagicMock()
mock_instance.status = "RUNNING"
mock_compute.get.return_value = mock_instance
call_count = {"n": 0}
class _OkResp:
status = 200
async def __aenter__(self): return self
async def __aexit__(self, *a): return None
def flaky_get(url, **kwargs):
call_count["n"] += 1
if call_count["n"] < 3:
raise aiohttp.ClientConnectorError(MagicMock(), OSError("connection refused"))
return _OkResp()
mock_session = self._mk_health_session(flaky_get)
with patch("backend.services.encoding_worker_manager.aiohttp.ClientSession", return_value=mock_session):
await manager.wait_for_worker_ready(
"encoding-worker-blue",
"http://10.0.0.1:8080/health",
vm_timeout=2.0,
health_timeout=2.0,
poll_interval=0.05,
)
assert call_count["n"] == 3
# ---------------------------------------------------------------------------
# capacity_state (availability awareness) + is_primary override routing
# ---------------------------------------------------------------------------
class TestCapacityStateFeedback:
"""ensure_any_running records/clears per-(type,zone) stockout state."""
def test_records_stockout_on_capacity_error(self, manager, mock_db, mock_compute):
mock_instance = MagicMock()
mock_instance.status = "TERMINATED"
mock_compute.get.return_value = mock_instance
# Primary c4d hits capacity; fallback n2 succeeds.
mock_compute.start.side_effect = [_capacity_op(), _ok_op()]
candidates = [
EncodingWorkerCandidate(vm_name="encoding-worker-a", zone="us-central1-c",
ip="10.0.0.1", machine_type="c4d-highcpu-16", is_primary=True),
EncodingWorkerCandidate(vm_name="encoding-worker-fallback-n2f", zone="us-central1-f",
ip="10.0.0.2", machine_type="n2-highcpu-16"),
]
with patch("backend.services.encoding_worker_manager.datetime") as mock_dt:
mock_dt.now.return_value = datetime(2026, 5, 5, 9, 0, 0, tzinfo=UTC)
mock_dt.side_effect = lambda *a, **kw: datetime(*a, **kw)
manager.ensure_any_running(candidates)
doc_ref = mock_db.collection.return_value.document.return_value
# A merge-set recorded the c4d stockout under its type@zone key.
set_calls = [c for c in doc_ref.set.call_args_list
if "capacity_state" in (c.args[0] if c.args else {})]
recorded = {}
for c in set_calls:
recorded.update(c.args[0]["capacity_state"])
assert recorded.get("c4d-highcpu-16@us-central1-c") == "2026-05-05T09:00:00+00:00"
def test_clears_stockout_for_type_on_clean_start(self, manager, mock_db, mock_compute):
mock_instance = MagicMock()
mock_instance.status = "TERMINATED"
mock_compute.get.return_value = mock_instance
mock_compute.start.return_value = _ok_op()
candidates = [
EncodingWorkerCandidate(vm_name="encoding-worker-a", zone="us-central1-c",
ip="10.0.0.1", machine_type="c4d-highcpu-16", is_primary=True),
]
with patch("backend.services.encoding_worker_manager.datetime") as mock_dt:
mock_dt.now.return_value = datetime(2026, 5, 5, 9, 0, 0, tzinfo=UTC)
mock_dt.side_effect = lambda *a, **kw: datetime(*a, **kw)
manager.ensure_any_running(candidates)
doc_ref = mock_db.collection.return_value.document.return_value
cleared = {}
for c in doc_ref.set.call_args_list:
payload = c.args[0] if c.args else {}
if "capacity_state" in payload:
cleared.update(payload["capacity_state"])
assert cleared.get("c4d-highcpu-16@us-central1-c") is None
def test_stockout_write_failure_does_not_break_start(self, manager, mock_db, mock_compute):
"""A Firestore error while recording a stockout must not abort failover."""
mock_instance = MagicMock()
mock_instance.status = "TERMINATED"
mock_compute.get.return_value = mock_instance
mock_compute.start.side_effect = [_capacity_op(), _ok_op()]
doc_ref = mock_db.collection.return_value.document.return_value
doc_ref.set.side_effect = RuntimeError("firestore down")
candidates = [
EncodingWorkerCandidate(vm_name="encoding-worker-a", zone="us-central1-c",
ip="10.0.0.1", machine_type="c4d-highcpu-16", is_primary=True),
EncodingWorkerCandidate(vm_name="encoding-worker-fallback-n2f", zone="us-central1-f",
ip="10.0.0.2", machine_type="n2-highcpu-16"),
]
with patch("backend.services.encoding_worker_manager.datetime") as mock_dt:
mock_dt.now.return_value = datetime(2026, 5, 5, 9, 0, 0, tzinfo=UTC)
mock_dt.side_effect = lambda *a, **kw: datetime(*a, **kw)
result = manager.ensure_any_running(candidates)
assert result["vm_name"] == "encoding-worker-fallback-n2f"
assert result["fell_back"] is True
def test_is_primary_flag_routes_override_regardless_of_position(self, manager, mock_db, mock_compute):
"""A fallback ranked FIRST (is_primary=False) must still set active_override.
When the preference logic demotes a stocked-out primary, the fallback can be
candidate index 0 — the override must key off is_primary, not position.
"""
mock_instance = MagicMock()
mock_instance.status = "TERMINATED"
mock_compute.get.return_value = mock_instance
mock_compute.start.return_value = _ok_op()
# Fallback is first (primary demoted), primary flagged but ranked second.
candidates = [
EncodingWorkerCandidate(vm_name="encoding-worker-fallback-c2df", zone="us-central1-f",
ip="10.0.0.9", machine_type="c2d-highcpu-16", is_primary=False),
EncodingWorkerCandidate(vm_name="encoding-worker-a", zone="us-central1-c",
ip="10.0.0.1", machine_type="c4d-highcpu-16", is_primary=True),
]
with patch("backend.services.encoding_worker_manager.datetime") as mock_dt:
mock_dt.now.return_value = datetime(2026, 5, 5, 9, 0, 0, tzinfo=UTC)
mock_dt.side_effect = lambda *a, **kw: datetime(*a, **kw)
result = manager.ensure_any_running(candidates)
assert result["vm_name"] == "encoding-worker-fallback-c2df"
assert result["fell_back"] is True
doc_ref = mock_db.collection.return_value.document.return_value
override_calls = [c for c in doc_ref.update.call_args_list
if "active_override_vm" in c.args[0]]
assert override_calls, "Fallback at index 0 must still set the override"
assert override_calls[0].args[0]["active_override_vm"] == "encoding-worker-fallback-c2df"
def test_legacy_candidates_without_flag_use_position(self, manager, mock_db, mock_compute):
"""No is_primary flag anywhere → first candidate is treated as primary (legacy)."""
mock_instance = MagicMock()
mock_instance.status = "TERMINATED"
mock_compute.get.return_value = mock_instance
mock_compute.start.return_value = _ok_op()
candidates = [
EncodingWorkerCandidate(vm_name="encoding-worker-blue", zone="us-central1-c", ip="10.0.0.1"),
EncodingWorkerCandidate(vm_name="encoding-worker-fb-a", zone="us-central1-a", ip="10.0.0.2"),
]
with patch("backend.services.encoding_worker_manager.datetime") as mock_dt:
mock_dt.now.return_value = datetime(2026, 5, 5, 9, 0, 0, tzinfo=UTC)
mock_dt.side_effect = lambda *a, **kw: datetime(*a, **kw)
result = manager.ensure_any_running(candidates)
assert result["fell_back"] is False
assert result["vm_name"] == "encoding-worker-blue"
# ---------------------------------------------------------------------------
# demote_active_worker: skip a broken fallback VM after a runtime infra failure
# ---------------------------------------------------------------------------
class TestDemoteActiveWorker:
"""A worker VM that accepted a job but then hit a VM-level infra/auth failure
(e.g. could not reach the metadata server — job 6452888e) must be demoted so
the retry lands on a healthy worker."""
def test_demotes_fallback_and_clears_override(self, manager):
"""When a fallback is serving, record its family cooldown then clear the
override so selection re-resolves from the primary."""
doc_ref = manager._doc_ref()
doc_ref.get.return_value.exists = True
doc_ref.get.return_value.to_dict.return_value = _make_firestore_data(
active_override_vm="encoding-worker-fallback-c4a",
active_override_ip="10.128.0.99",
active_override_zone="us-central1-c",
active_override_set_at="2026-08-31T12:00:00Z",
)
manager.demote_active_worker(reason="job 6452888e: metadata server unavailable")
# Recorded a cooldown for the fallback's (machine_type@zone).
assert doc_ref.set.called
set_args, set_kwargs = doc_ref.set.call_args
assert "capacity_state" in set_args[0]
assert "c4-highcpu-16@us-central1-c" in set_args[0]["capacity_state"]
assert set_kwargs.get("merge") is True
# Cleared the active_override so the retry starts from the primary.
override_clears = [
c for c in doc_ref.update.call_args_list
if c.args and "active_override_vm" in c.args[0]
]
assert override_clears, "expected active_override to be cleared"
assert override_clears[-1].args[0]["active_override_vm"] is None
def test_no_op_when_primary_is_serving(self, manager):
"""When no fallback override is set (primary pair serving), demote is a no-op:
we don't family-demote the fast c4d primary on a rare blip."""
doc_ref = manager._doc_ref()
doc_ref.get.return_value.exists = True
doc_ref.get.return_value.to_dict.return_value = _make_firestore_data() # no override
manager.demote_active_worker(reason="job zzz: metadata server unavailable")
assert not doc_ref.set.called
override_clears = [
c for c in doc_ref.update.call_args_list
if c.args and "active_override_vm" in c.args[0]
]
assert not override_clears
def test_never_raises_on_config_read_failure(self, manager):
"""Demotion is best-effort telemetry — a Firestore hiccup must not break the
failure path it runs on."""
doc_ref = manager._doc_ref()
doc_ref.get.side_effect = RuntimeError("firestore unavailable")
# Should swallow the error rather than propagate.
manager.demote_active_worker(reason="job qqq")