Files
karaoke-gen/infrastructure/modules/source_archive.py
T
Andrew Beveridge ☄️andClaude Opus 5.5 0449d0c107 fix(infra): build Cloud Function source archives without __pycache__ (v0.297.4) (#1272)
## Summary
A full `pulumi preview` from main showed runner-manager drift ("replace
source, update function"), last applied from the
`karaoke-gen-runner-scan-retry` worktree. I checked it:

- #1262 (that worktree's branch) is fully on main. The squash merge is
byte-identical to the branch tip. The live scheduler job (`retryCount
2`, `maxRetryDuration 60s`, `attemptDeadline 300s`) also matches the
code exactly.
- The drift comes from the **deployed zip containing
`__pycache__/*.pyc`** (including a pytest pyc) from the worktree's test
run. `pulumi.FileArchive` hashes every file, so every clean checkout saw
a phantom diff. Separately, the 7-day lifecycle rule had already deleted
the bucket object.

## Changes
- New `infrastructure/modules/source_archive.clean_source_archive()`
builds an `AssetArchive` of source files only. It skips `__pycache__`,
`*.pyc`, `.pytest_cache`, other tool caches and `.DS_Store`.
- Used for runner-manager, divebar-lookup and divebar-mirror (the three
`FileArchive` users).
- Unit tests, added to the "Infra - Cloud Function tests" CI job.
- LESSONS-LEARNED entry.

## Applied (before merge, per repo convention)
- I ran a targeted preview with `__pycache__` present in the worktree.
Runner-manager showed source replace + function update (the one-time
correction). Divebar-lookup and divebar-mirror showed **no change**.
- Targeted `pulumi up` ran as claude-admin (Andrew's grant): 2 changes,
289 unchanged.
- The new revision is `github-runner-manager-00031-sam`, and the
deployed zip now holds only the 4 source files.
- The re-preview shows 291 unchanged.

## Testing
- `pytest infrastructure/test_source_archive.py`: 4 passed
- `pytest infrastructure/functions/runner_manager`: 75 passed
- Local CodeRabbit review: no findings

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVSfwoV9WyncjqMPHrq

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 15:16:34 -05:00

45 lines
1.7 KiB
Python

"""Deterministic Cloud Function source archives.
``pulumi.FileArchive(dir)`` zips and hashes *every* file in the directory. If
someone runs the function's tests first, ``__pycache__/*.pyc`` and
``.pytest_cache`` get packed into the deployed zip. The archive hash then
depends on which checkout last ran ``pulumi up``, so a clean checkout of main
shows phantom drift: the source object gets replaced and the function updated
with identical code. This happened to runner-manager on 2026-11-09.
``clean_source_archive`` packs only the real source files, so the hash depends
only on the code.
"""
from pathlib import Path
import pulumi
EXCLUDED_DIRS = frozenset({"__pycache__", ".pytest_cache", ".mypy_cache", ".ruff_cache", "venv", ".venv"})
EXCLUDED_SUFFIXES = (".pyc", ".pyo")
EXCLUDED_NAMES = frozenset({".DS_Store"})
def source_files(source_dir: Path) -> dict[str, Path]:
"""Map archive-relative path -> file, skipping caches and OS junk."""
files: dict[str, Path] = {}
for path in sorted(source_dir.rglob("*")):
if not path.is_file():
continue
rel = path.relative_to(source_dir)
if any(part in EXCLUDED_DIRS for part in rel.parts[:-1]):
continue
if path.name in EXCLUDED_NAMES or path.suffix in EXCLUDED_SUFFIXES:
continue
files[rel.as_posix()] = path
if not files:
raise ValueError(f"No source files found in {source_dir}")
return files
def clean_source_archive(source_dir: Path) -> pulumi.AssetArchive:
"""An archive of ``source_dir``'s source files only (see module docstring)."""
return pulumi.AssetArchive(
{rel: pulumi.FileAsset(str(path)) for rel, path in source_files(source_dir).items()}
)