fix(divebar): folder-brand-aware index parser + symmetric xref normalization (#857)

## Summary
- Rewrites the Divebar Drive→BigQuery **filename parser** to be
folder-brand-aware, fixing mis-split artists and brand-polluted titles
that prevented mirror files from surfacing in kjbox.
- Fixes a **normalization asymmetry** in the KN↔Divebar xref join (KN
side was only `LOWER(TRIM())` vs the db's fully-normalized columns) and
removes a buggy `brand_match` branch that produced wrong-song links.
- Switches both divebar Cloud Function modules to `pulumi.FileArchive`
so `pulumi up` actually redeploys changed code.

## Changes
- `divebar_mirror/filename_parser.py` — folder-brand-aware rewrite:
strip the brand token wherever it sits (disc-id prefix/suffix,
`(CODE)`/`CODE -` prefix, `[Brand]`/`(Brand)`/` - Brand` suffix, generic
karaoke tags), each strip gated on a folder-brand match so real song
text (`(Live)`, `(G)I-DLE`, bands like `UB40`/`311`/`ACDC`) survives.
Curated folder→KN-code map fills `brand_code`. En-dash separator +
leading-dash cleanup.
- `divebar_lookup/main.py` — new `_norm_sql(col)` routes **both** join
sides through one identical BigQuery expression replicating
`normalize_for_search`; `brand_match` branch removed (it matched
brand+artist with no title → Cartesian wrong-song links; 107,485/107,485
pairs were title mismatches in live data).
- `modules/divebar_mirror.py` + `modules/divebar_lookup.py` —
`FileArchive` + `BucketObject` with `generation` pinning (matches
`runner_manager`), so code changes redeploy.
- New `tests/unit/test_divebar_filename_parser.py` (48 cases) + 3
`_norm_sql` guard tests; version bump 0.188.3→0.188.4.

## Impact (validated read-only over all 50,046 live rows)
- null `brand_code`: 18,437 (37%) → **1,619 (3%)**
- Divebar files cross-referenced to a KN song: production **23,318** →
**33,449** on current data (+10,131 from the symmetry fix alone),
lifting toward ~38k after the parser re-index.

## Testing
- [x] 64 divebar unit tests pass locally (parser 48, index_builder 5,
divebar_lookup 11)
- [x] Offline validation harness over all 50k live rows (read-only
BigQuery); `_norm_sql` confirmed to reproduce `normalize_for_search`
exactly
- [ ] CI full suite

## Deploy / re-index (post-merge)
`pulumi up` redeploys both functions; then trigger scheduler
`divebar-mirror-daily` (re-parse → MERGE, preserves `gcs_path`) **then**
`divebar-xref-rebuild-daily`. Order matters. (`kn_data_sync` still uses
a static source object — future-edit redeploy trap, out of scope here.)

## Review
- [x] CodeRabbit CLI review completed locally (1 finding → fixed:
brand_match removal; cycle 2 clean)

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Andrew Beveridge ☄️andClaude Opus 4.8 authored and GitHub committed 2026-06-28 14:16:35 -04:00
1 parent ef65adedc7
commit eb711b32a8
9 files changed
+928 -122

No files matched your search

@@ -0,0 +1,255 @@
# Divebar Index Filename-Parsing Fix — Plan
_Worktree: `karaoke-gen-divebar-index-parsing` · branch `feat/sess-20260628-0511-divebar-index-parsing`_
_Picks up: HANDOFF-divebar-mirror-parsing.md (kjbox side shipped v0.41.1; this is the upstream index fix)_
## Problem (refined by live data, not just the handoff examples)
The Divebar Drive→BigQuery indexer (`infrastructure/functions/divebar_mirror/filename_parser.py`,
Cloud Function `divebar-mirror`) mis-parses community-brand filenames. kjbox cross-references
mirror files on `(normalized artist, normalized title)` + brand, so bad parses → mirror files
never surface in the rotation selector even though the file is in GCS.
### Measured state (BigQuery `karaoke_decide.divebar_catalog`, 50,046 rows)
| Signal | Count | Meaning |
|---|---|---|
| null `artist` | 164 (0.3%) | rare |
| null `title` | 0 | — |
| **null `brand_code`** | **18,437 (37%)** | biggest visible gap |
| `artist` is 2–6 all-caps/digits | 1,908 | brand-code-as-artist mis-split (e.g. `CKK`, `EBK`) |
| `artist` starts with `(` | 3,051 | `(SDK) Artist` prefix (Sandell) |
| `title` contains ` - ` | 5,947 | unstripped ` - Brand` suffix / disc-id-in-title |
| `title` ends in `[..]`/`(..)` | 2,583 | `[DJ Sauly Karaoke]`, `[Karaoke]`, etc. |
Current xref coverage: 90,719 matches, 17,261 KN songs, **23,318 / 50,046 divebar files matched (47%)**.
> Reframe vs handoff: `artist`/`title` are rarely *null*, but **mis-splits + polluted titles**
> are common and silently defeat the EXACT xref (`LOWER(TRIM(kn.Title)) = db.title_normalized`).
> Every polluted title misses the join. That + missing `brand_code` is the real lever.
### Root cause: parser has only 3 rigid patterns and never uses the folder brand to clean
`parse_filename(name, brand_folder)`:
1. `^CODE-### - Artist - Title` (disc-id prefix) — works.
2. `... (BrandTag)$` — strips ANY trailing paren (so it also eats `(KARAOKE)`, `(Live)`,
`(Banda Remix)` → clobbers `brand_name`), and never sets `brand_code`.
3. `Artist - Title` fallback split on ` - `.
It ignores `[...]` brackets, ` - Brand` dash-suffixes, disc-id *suffixes*, `(CODE) ` prefixes,
and `CODE - ` (no-number) prefixes. The folder name (reliable brand) is passed in but only used as
a fallback `brand_name` — **never to strip the brand token before splitting artist/title**.
`_KARAOKE_SUFFIXES` is applied only to the normalized fields, never the display `title`.
Note: `build_rows` writes `brand` = folder (always, reliable) and only consumes `brand_code` from
the parser (not `brand_name`) — so Pattern-2's `brand_name` clobbering is harmless. The fields that
matter from the parser are **artist, title, brand_code, disc_id**.
## Real-world conventions (gold fixtures, observed live)
| Convention | Example filename | Brand (folder) | Correct artist / title |
|---|---|---|---|
| disc-id prefix (OK today) | `FATBIRD-163 - Incubus - I Miss You.zip` | Fatbird | Incubus / I Miss You |
| code prefix, no number | `CKK - Buoys, The - Timothy.zip` | Cereal Killer Karaoke | Buoys, The / Timothy |
| `(CODE)` paren prefix | `(SDK) (G)I-DLE - Queencard.cdg` | Sandell Karaoke | (G)I-DLE / Queencard |
| `(Brand)` paren suffix (OK today) | `10 Years - Fix Me (BellySings).mp4` | BellySings | 10 Years / Fix Me |
| `[Brand]` bracket suffix | `3rd Bass - Green Eggs And Swine [DJ Sauly Karaoke].zip` | DJ Sauly Collection | 3rd Bass / Green Eggs And Swine |
| ` - Brand` dash suffix | `100 Gecs - Last Train To Awesometown - Rock Solid Karaoke.zip` | Rock Solid Karaoke | 100 Gecs / Last Train To Awesometown |
| ` - CODE` dash suffix | `Akon - Right Now (Na Na Na) - EBK.avi` | ForteKaraoke Discord | Akon / Right Now (Na Na Na) |
| disc-id *suffix* | `Against Me - Wagon Wheel - SOKC-0306 - (KARAOKE).zip` | Steve-O Karaoke Cult | Against Me / Wagon Wheel |
| disc-id suffix no dash | `Angelcorpse - Wolflust - ATG0014.zip` | ATG Karaoke | Angelcorpse / Wolflust |
| `[Karaoke]` generic tag | `Bill Murray - Star Wars [Karaoke].mp4` | Mobile Karaoke Unit | Bill Murray / Star Wars |
| mixed-case brand prefix | `Lopenash-002 - Dexter's Laboratory - Breathe in The Sunshine.mp4` | Lopenash | Dexter's Laboratory / Breathe in The Sunshine |
| clean (only brand_code missing) | `A Tribe Called Quest - Lyrics To Go.avi` | WIZZYOKE | A Tribe Called Quest / Lyrics To Go |
**Guard cases (must NOT over-strip):** `(G)I-DLE`, `R.E.M.`, artist `311`; titles `(Live)`,
`(Banda Remix)`, `(Single Version 2 WBV)`, `(99X Live)`, `[Acoustic]` must survive.
## Design — folder-brand-aware parser
Core principle (per handoff): **the top-level folder reliably names the brand; use it to recognise
and strip the brand token wherever it sits, then split the remainder into artist/title.**
`parse_filename(name, brand_folder)` rewrite, in order:
1. **Strip generic karaoke tags** from the display name: trailing `(karaoke|instrumental|backing
track|no vocals|kj version|with vocals|demo)` in either `()` or `[]` (reuse `_KARAOKE_SUFFIXES`,
now applied to display too).
2. **Build brand matchers from `brand_folder`**: normalized full name, and name with trailing
descriptors removed (`karaoke`, `collection`, `cdg files`, `discord`, `videos`, `mp4`,
`zipped mp3+g`, etc.); plus an acronym of significant words (Cereal Killer Karaoke→CKK,
Steve-O Karaoke Cult→SOKC, ATG Karaoke→ATG).
3. **Strip the brand token from whichever position it occupies — each gated so we only remove a
*known* brand, never song text:**
- disc-id **prefix** `^CODE[-\s]?### - ` → `brand_code`, `disc_id`. (existing)
- disc-id **suffix** ` - CODE-### ` / ` CODE#### ` at end, gated on `CODE` ≈ folder acronym/known → `brand_code`,`disc_id`.
- `[Brand]`/`(Brand)` **bracket/paren suffix** where content ≈ folder brand → strip; `brand_code` from folder map.
- ` - Brand` **dash suffix** where the last ` - ` segment ≈ folder brand → strip.
- `(CODE) ` **paren prefix** (≤6 chars, alnum) with content after → strip; `brand_code`=CODE.
- `CODE - ` / `Folder-### - ` **prefix** where token ≈ folder acronym/name → strip; `brand_code`.
4. **Split remainder** on ` - ` → artist = parts[0], title = join(parts[1:]); strip any leftover
generic karaoke tag from title.
5. **brand_code resolution order:** captured-during-strip → small curated **folder→code map** →
null. `brand` always = folder (unchanged in `build_rows`).
### Brand registry decision (diverges from handoff)
Handoff suggested importing kjbox `version_priority.py` `COMMUNITY_BRANDS`. **Rejected** because:
(a) cross-repo import isn't viable inside a GCP Cloud Function, (b) that registry is **incomplete**
vs the 63 real Divebar folder brands (no Cereal Killer/DJ Sauly/Rock Solid/Sandell), and (c) it
**conflicts** — registry maps `SDK`→SNDL but Sandell's folder uses `(SDK)`. Instead: derive
`brand_code` from the in-filename token / folder, and keep a **small curated `folder→KN-code` map**
seeded from the actual KN community codes that intersect Divebar folders (FBK, BELLY, DJS, C11K,
PLAY, REEKIES, HALJAM, NOMAD, FAKEY, CC, …). kjbox's read-time `canonical_brand_for_match` already
folds codes/names, so the index doesn't need canonical output — just resolvable.
## Refined design from full 63-brand data (read-only export)
Exported all 50,046 rows + per-brand samples + KN community codes to scratchpad. Key refinement:
**artist is wrong-but-rarely-null; the measurable gaps are polluted titles + null brand_code.**
Brands needing a fix fall into these convention buckets (✅ already-correct buckets only need the
brand_code map):
| Bucket | Brands (examples) | Fix |
|---|---|---|
| disc-id PREFIX `CODE-### - A - T` | Nomad, Imperfekt, Play, Gnome, Fakey, C11K, Reekies, Fatbird, … | works today — **preserve** |
| disc-id PREFIX space-sep `CODE-#### Artist - T` | Potos (`RABZ-0001 The Seatbelts - …`) | allow space (not just ` - `) after disc-id |
| disc-id PREFIX mixed-case/digit-lead `Tok-### - …` | Lopenash (`Lopenash-002 -`), 7uLpA (`7uLpA-0003 -`) | match when prefix token ≈ folder name |
| leading track-no after disc-id `CODE-### - NN - A - T` | Mix Tape (`MIX001 - 02 - …`) | drop leading pure-numeric segment |
| disc-id SUFFIX `A - T - CODE-####` | It Might Be (IMBK), Steve-O (SOKC), ATG (ATG####) | **strip trailing disc-id (unconditional), set brand_code/disc_id** |
| `(CODE)` paren PREFIX | Sandell (`(SDK) A - T`) | strip leading `(short-alnum) ` |
| `CODE - ` PREFIX no number | Cereal Killer (`CKK - A - T`), Nerdy (`TNSK -`), Crossfire (`Crossfire - A - T`) | strip when token ≈ folder acronym/name |
| `[Brand]` bracket SUFFIX | DJ Sauly (`… [DJ Sauly Karaoke]`) | strip bracket suffix gated on folder-name match |
| `(Brand)` paren SUFFIX | BellySings (`… (BellySings)`), Popeoke (`(popeoke)`/`[popeoke]`) | already strips paren; add bracket + folder-gate |
| ` - Brand` dash SUFFIX | Rock Solid (`… - Rock Solid Karaoke`), Forte (`… - EBK`) | strip last ` - ` segment gated on folder-name match (Forte EBK via per-folder token) |
| generic karaoke tag `[Karaoke]`/`(Karaoke HD)` | Mobile, Funbox, Matt Joy, Steve-O | strip generic tag from **display** title (currently only normalized), tolerate trailing words |
| clean `Artist - Title`, only brand_code missing | Funbox, WIZZYOKE, Brilliant Trash, Monster Tracks, KaddaOK, April's Choice | **curated folder→code map only** |
### Parser algorithm (folder-brand-aware), ordered
1. strip extension; `format = detect_format`.
2. strip generic karaoke tags from display name (both `()`/`[]`, token optionally followed by words).
3. brand-token strip, first match wins, each sets `brand_code`/`disc_id` and removes the token:
(a) disc-id PREFIX `^CODE[-\s]?NUM` followed by ` - ` **or** whitespace; CODE all-caps **or** ≈ folder token (covers Lopenash/7uLpA/Potos);
(b) disc-id SUFFIX `[-\s]\s*CODE-?NUM$` — **unconditional** (trailing `ABCD-0123` is a strong signal);
(c) `[Brand]`/`(Brand)` SUFFIX gated on `norm(content) ≈ norm(folder)`;
(d) ` - Brand` dash SUFFIX gated on `norm(last segment) ≈ norm(folder)` (or per-folder token);
(e) `(CODE)` paren PREFIX (≤6 alnum) with content after;
(f) `CODE - ` PREFIX (no number) gated on `CODE ≈ folder acronym/name`.
4. split remainder on ` - ` → artist=parts[0], title=join(rest); drop a leading pure-numeric segment;
1 part → title only.
5. strip any leftover generic karaoke tag from title.
6. `brand_code` = captured → **curated folder→code map** → null. `brand` always = folder (in `build_rows`).
**Guards (must survive):** `(G)I-DLE`, `R.E.M.`, artist `311`, titles `(Live)`, `(Banda Remix)`,
`(Single Version 2 WBV)`, `(99X Live)`, `[Acoustic]`. Folder-gating + "disc-id has digits" +
"prefix code ≈ folder" are what prevent eating these.
### Curated folder→KN-code map (minimal, high-confidence only)
Aligned to actual KN community `Brand` codes (so the secondary brand_match xref can also fire);
brands with no confident KN code are left null (artist/title fix already surfaces them via the
primary 0.95 exact xref):
```
Funbox Karaoke→FBK · Sandell Karaoke→SDK · BellySings Karaoke→BELLY · DJ Sauly Collection→DJS
PlayKaraoke→PLAY · Reekies Karaoke→REEKIES · Cereal Killer Karaoke→CKK · WIZZYOKE→WIZZY
Brilliant Trash Karaoke→BTRASH · Lopenash→LOPE · Monster Tracks→MTRAX · Popeoke→POPE
KaddaOK→KADDA · Potos Power Plant Productions→PPPP · April's Choice Karaoke→APRIL
The Nerdy Singer Karaoke→TNS · Mobile Karaoke Unit→MKU · ATG Karaoke - Zipped MP3+G→ATG
It Might Be Karaoke→IMBK
```
(Map *overrides* a prefix-derived code where they differ, e.g. Play `PLK`→`PLAY`, Reekies
`REEK`→`REEKIES`; `disc_id` keeps the literal prefix.) Skipped (no confident code): Rock Solid,
Steve-O (uses SOKC, not a KN code — set from disc-id suffix anyway), ForteKaraoke (EBK≠KN), Gnome,
7uLpA, Crossfire, Monster… handled where a code exists.
## Work breakdown
**Phase 1 — Parser rewrite (TDD, primary deliverable).** New `tests/unit/test_divebar_filename_parser.py`
with all gold fixtures + guard cases (add function dir to `sys.path`; parser deps are stdlib only).
Rewrite `filename_parser.py` to the design above. Keep `normalize_for_search`, `detect_format`,
`should_index_file` behaviour. Keep existing `test_divebar_index_builder.py` green.
**Phase 2 — brand_code coverage.** Curated `folder→code` map + disc-id-suffix / leading-token
capture. Target: cut null `brand_code` well below 37% on the offline harness.
**Phase 3 — Deploy enablement (infra papercut).** The module points the function at a *static*
source object (`divebar-mirror-source.zip`) with no content hash, so overwriting it won't make
`pulumi up` redeploy. Switch `modules/divebar_mirror.py` to `pulumi.FileArchive` + `BucketObject`
(the proven pattern in `modules/runner_manager.py`) so new code deploys on `pulumi up`. (Optional
but needed for the fix to ship cleanly; Andrew runs the apply.)
**Phase 4 — Offline validation harness + prod verify.**
- Export distinct `(filename, brand)` from BQ (read-only) → run NEW parser locally over all 50k →
report before/after: % null brand_code, % artist-looks-like-code, % title-has-dash, % title-ends-bracket,
plus a spot-check sample. **Proves improvement with zero prod writes** (fits read-only ADC).
- After Andrew deploys + triggers `divebar-mirror-daily` then `divebar-xref-rebuild-daily`
(MERGE preserves `gcs_path`; no Drive→GCS re-sync needed): re-measure xref `db_files` matched
(baseline 23,318/50,046) and the live `incubus admiration` repro.
## Constraints / notes
- **My ADC is read-only** (workspace memory `feedback_claude_readonly_adc`): I can read BQ and run
the offline harness, but **cannot** `pulumi up`, upload source, or run scheduler jobs. Andrew
deploys + re-indexes; I validate offline first and via BQ reads after.
- Re-index re-parses ALL files (full Drive listing → MERGE by `file_id`), so existing rows are
corrected in place; `gcs_path` is preserved.
- Version bump `pyproject.toml`; this is infra/function code (no i18n, no frontend).
- Tests: `make test` (or targeted `pytest tests/unit/test_divebar_filename_parser.py`).
## Results (offline harness over all 50,046 live rows)
| metric | before | after | delta |
|---|---|---|---|
| null brand_code | 18,437 (37%) | 1,619 (3%) | **−16,818** |
| title contains ` - ` | 5,947 | 2,523 | −3,424 (residual mostly legit multi-part titles) |
| artist looks like a code | 1,908 | 1,400 | −508 (residual are real bands: ACDC, KISS, ABBA, U2…) |
| null artist | 164 | 179 | +15 (medleys / single-field — acceptable) |
| rows with changed artist/title | — | 13,348 (26.7%) | — |
Per-brand `brand_code` null% collapsed to ~0% for every broken brand (Sandell, BellySings,
DJ Sauly, Cereal Killer, Steve-O, WIZZYOKE, Lopenash, ForteKaraoke, ATG, Mobile, 7uLpA, …);
Rock Solid stays null by design (no confident KN code).
**Xref lift (replaying the live `_rebuild_xref` EXACT join):** Divebar files that match a KN
song **20,802 → 23,394 (+2,592, +12.5%)** — i.e. that many more mirror files now surface in kjbox.
### BUNDLED: the xref SQL normalization asymmetry (fixed in this PR)
`_rebuild_xref` joined `LOWER(TRIM(kn.Artist)) = db.artist_normalized`, but `db.artist_normalized`
is produced by `normalize_for_search` (strips diacritics, a leading "the ", and all punctuation)
while the KN side got only `LOWER(TRIM(...))` — so most rows could never match. **Fix:** a new
`_norm_sql(col)` helper routes **both** sides through one identical BigQuery expression
replicating `normalize_for_search` (NFD + drop `\p{Mn}`, lower/trim, strip trailing karaoke tag,
strip leading "the ", drop non-`\p{L}\p{N}_\s`, collapse whitespace). Applied to the exact join,
the brand_match join, and the NOT EXISTS de-dup.
**Validated read-only against live BigQuery** (the real `_norm_sql`): the SQL normalization
reproduces the Python `normalize_for_search` baseline exactly (32,496 on current data), and the
**full new union already matches 33,449 distinct Divebar files on current (old-parse) data vs
production's 23,318 — +10,131 from symmetry alone**, before any re-index. After the parser
re-index the exact portion lifts toward ~37,910, compounding further. Residual misses after the parser fix are tiny and documented: `(WTF Karaone)` typo
(~6), `SOKC-O104` letter-O typo (~21), `GGZ007-02,2` comma in disc-id (~16), `RVZ-033.d` (~5).
## Status: IMPLEMENTED
- `filename_parser.py` rewritten folder-brand-aware; `tests/unit/test_divebar_filename_parser.py`
added (48 cases, green); existing `test_divebar_index_builder.py` still green.
- `divebar_lookup/main.py` `_rebuild_xref` rewritten to normalize both sides via `_norm_sql`;
3 guard tests added to `test_divebar_lookup` (`test_main.py`). All 64 divebar tests green.
- `modules/divebar_mirror.py` switched to `pulumi.FileArchive` + `BucketObject` with
generation-pinning so `pulumi up` actually redeploys new function code; `deploy.sh` note updated.
- Version bumped 0.188.3 → 0.188.4.
- **Deploy (Andrew, read-only ADC blocks me):** `cd infrastructure && pulumi up` (redeploys both
`divebar-mirror` and `divebar-lookup`), then trigger scheduler jobs `divebar-mirror-daily`
(re-parse → MERGE, preserves `gcs_path`) then `divebar-xref-rebuild-daily`. Then re-measure xref
`db_files` and the live `incubus admiration` repro. (Order matters: re-index before xref rebuild.)
## Open questions for Andrew
1. **Include Phase 3** (FileArchive deploy fix) in this PR, or keep PR parser-only and handle deploy
mechanics separately?
2. **brand_code map scope** — minimal (only folders that clearly map to a KN community code) vs
broader best-effort? Minimal is safer (no mismaps); recommend minimal.
3. Confirm **you'll run the `pulumi up` + scheduler re-index** after merge (I can't with read-only ADC).
+42 -37
View File
@@ -177,16 +177,51 @@ def _lookup_kn_ids(kn_ids: list[int]) -> dict[int, list[dict]]:
return result
def _norm_sql(col: str) -> str:
"""A BigQuery expression replicating divebar_mirror's normalize_for_search.
Both the KaraokeNerds side and the Divebar side of the xref join are passed
through this so the comparison is symmetric. Previously the join compared the
KN side (only LOWER+TRIM) against the Divebar side's fully-normalized columns
(diacritics / leading "the " / punctuation stripped), so most rows could never
match. Steps mirror normalize_for_search: strip diacritics (NFD + drop combining
marks), lower/trim, strip a trailing karaoke tag, strip a leading "the ", drop
non-word chars (keeping unicode letters/digits/underscore), collapse whitespace.
"""
return (
"TRIM(REGEXP_REPLACE("
"REGEXP_REPLACE("
"REGEXP_REPLACE("
"REGEXP_REPLACE("
"LOWER(TRIM(REGEXP_REPLACE(NORMALIZE(COALESCE(" + col + ", ''), NFD), r'\\p{Mn}', ''))),"
r" r'\s*[\(\[]\s*(?:[^)\]]*karaoke[^)\]]*|(?:instrumental|backing track|no vocals?|kj version|with vocals?|demo)[^)\]]*)[\)\]]\s*$', ''),"
r" r'^the ', ''),"
r" r'[^\p{L}\p{N}_\s]', ''),"
r" r'\s+', ' '))"
)
def _rebuild_xref() -> dict:
"""Rebuild the KN ↔ Divebar cross-reference index using exact + fuzzy matching."""
"""Rebuild the KN ↔ Divebar cross-reference index by exact normalized match.
Both sides are normalized through the identical `_norm_sql` expression, so a
KN song links to a Divebar file only when their artist AND title agree after
normalization. (The previous "brand_match" branch matched on brand_code +
artist only — with no title constraint it was a Cartesian product that linked
every KN song by an artist/brand to every Divebar file by the same
artist/brand, i.e. wrong-song links. Constraining it by title would make it a
strict subset of the exact match, so it was removed.)
"""
client = bigquery.Client(project=GCP_PROJECT_ID)
start = time.time()
# Step 1: Exact match on normalized artist + title
kn_artist, kn_title = _norm_sql("kn.Artist"), _norm_sql("kn.Title")
db_artist, db_title = _norm_sql("db.artist"), _norm_sql("db.title")
exact_sql = f"""
CREATE OR REPLACE TABLE `{GCP_PROJECT_ID}.{DATASET}.kn_divebar_xref` AS
-- Exact normalized match (high confidence)
-- Exact match on symmetrically-normalized artist + title (high confidence)
SELECT DISTINCT
kn.Id AS kn_id,
db.file_id AS divebar_file_id,
@@ -195,40 +230,10 @@ def _rebuild_xref() -> dict:
CURRENT_TIMESTAMP() AS matched_at
FROM `{GCP_PROJECT_ID}.{DATASET}.karaokenerds_raw` kn
JOIN `{GCP_PROJECT_ID}.{DATASET}.divebar_catalog` db
ON LOWER(TRIM(kn.Artist)) = db.artist_normalized
AND LOWER(TRIM(kn.Title)) = db.title_normalized
WHERE db.artist_normalized IS NOT NULL
AND db.artist_normalized != ''
AND db.title_normalized IS NOT NULL
AND db.title_normalized != ''
UNION ALL
-- Community brand match: KN community track brand matches Divebar folder brand code
SELECT DISTINCT
kn.Id AS kn_id,
db.file_id AS divebar_file_id,
'brand_match' AS match_type,
0.90 AS confidence,
CURRENT_TIMESTAMP() AS matched_at
FROM `{GCP_PROJECT_ID}.{DATASET}.karaokenerds_community` c
JOIN `{GCP_PROJECT_ID}.{DATASET}.karaokenerds_raw` kn
ON LOWER(TRIM(c.Artist)) = LOWER(TRIM(kn.Artist))
AND LOWER(TRIM(c.Title)) = LOWER(TRIM(kn.Title))
JOIN `{GCP_PROJECT_ID}.{DATASET}.divebar_catalog` db
ON LOWER(c.Brand) = LOWER(COALESCE(db.brand_code, ''))
AND LOWER(TRIM(c.Artist)) = db.artist_normalized
WHERE c.Brand IS NOT NULL
AND c.Brand != ''
AND db.artist_normalized IS NOT NULL
AND db.artist_normalized != ''
-- Exclude exact matches (already captured above)
AND NOT EXISTS (
SELECT 1 FROM `{GCP_PROJECT_ID}.{DATASET}.divebar_catalog` db2
WHERE LOWER(TRIM(kn.Artist)) = db2.artist_normalized
AND LOWER(TRIM(kn.Title)) = db2.title_normalized
AND db2.file_id = db.file_id
)
ON {kn_artist} = {db_artist}
AND {kn_title} = {db_title}
WHERE {db_artist} != ''
AND {db_title} != ''
"""
logger.info("Rebuilding cross-reference index...")
@@ -108,6 +108,33 @@ class TestRefreshTriggers:
assert result["failed"][0]["job"] == main.REFRESH_SCHEDULER_JOBS[0]
# ---------------------------------------------------------------------------
# _norm_sql — symmetric normalization for the xref join
# ---------------------------------------------------------------------------
class TestNormSql:
def test_embeds_column_and_is_deterministic(self):
a = main._norm_sql("kn.Artist")
assert "kn.Artist" in a
assert main._norm_sql("kn.Artist") == a # pure / deterministic
def test_replicates_normalize_for_search_steps(self):
expr = main._norm_sql("db.title")
# diacritics (NFD + drop combining marks), lower, leading "the", and the
# unicode-aware punctuation strip must all be present.
assert "NORMALIZE(COALESCE(db.title" in expr
assert "NFD" in expr and r"\p{Mn}" in expr
assert "LOWER(" in expr
assert r"'^the '" in expr
assert r"\p{L}" in expr and r"\p{N}" in expr
def test_both_sides_use_same_expression(self):
# The whole point of the fix: KN and Divebar sides normalize identically.
kn = main._norm_sql("X")
db = main._norm_sql("X")
assert kn == db
# ---------------------------------------------------------------------------
# divebar_lookup dispatch — refresh action
# ---------------------------------------------------------------------------
@@ -1,17 +1,14 @@
#!/bin/bash
# Deploy the Divebar Mirror Cloud Function
#
# This script:
# 1. Creates a ZIP of the function source
# 2. Uploads it to GCS
# 3. Deploys the function via gcloud
# NOTE: As of 2026-06, the function source is managed by Pulumi
# (modules/divebar_mirror.py zips this directory via pulumi.FileArchive and pins
# the function to the object generation). The normal way to ship a code change is:
#
# Prerequisites:
# - gcloud CLI authenticated
# - Pulumi infrastructure already deployed (creates the bucket)
# cd infrastructure && pulumi up
#
# Usage:
# ./deploy.sh
# This script remains for ad-hoc manual uploads/testing only; a manual upload
# will be reset to the repo contents on the next `pulumi up`.
set -e
@@ -1,67 +1,148 @@
"""
Parse karaoke filenames into structured metadata.
Handles the various naming conventions found across Divebar karaoke brands:
Pattern 1: "BRAND-001 - Artist - Title.ext" (disc ID prefix)
Pattern 2: "Artist - Title - (Brand Tag).ext" (brand suffix)
Pattern 3: "Artist - Title.ext" (brand from folder name)
Community karaoke brands use many naming conventions. Rather than a handful of
rigid patterns, this parser is *folder-brand-aware*: the top-level Drive folder
reliably names the brand, so we use it to recognise and strip the brand token
wherever it sits (prefix, suffix, bracket, disc-id) and then split the remainder
into artist / title. Brand stripping is gated on the known folder brand so we
never eat real song text like "(Live)", "(Banda Remix)" or "(G)I-DLE".
Returned dict keys: artist, title, disc_id, brand_code, brand_name, format.
"""
import os
import re
import unicodedata
# Common karaoke suffixes to strip from titles
_KARAOKE_SUFFIXES = re.compile(
r"\s*[\(\[](karaoke|instrumental|backing track|no vocals?|kj version|"
r"karaoke version|with vocals?|demo)[\)\]]\s*$",
# Generic karaoke/source tags to strip from the end of a title, in () or [].
# Matches any bracket whose content mentions "karaoke" ("(WTF Karaoke)",
# "[DJ Sauly Karaoke]", "(Karaoke HD)") or starts with another generic tag
# ("(instrumental)", "(backing track)").
_KARAOKE_TAG = re.compile(
r"\s*[\(\[]\s*(?:[^)\]]*karaoke[^)\]]*|"
r"(?:instrumental|backing track|no vocals?|kj version|with vocals?|demo)[^)\]]*)"
r"[\)\]]\s*$",
re.IGNORECASE,
)
# Back-compat alias (normalize_for_search historically referenced this name).
_KARAOKE_SUFFIXES = _KARAOKE_TAG
# Brand code pattern at start of filename: "ABC-001" or "ABC001" or "ABC 001"
_DISC_ID_PATTERN = re.compile(
r"^([A-Z][A-Z0-9]{1,10})[\s\-]?(\d{1,5}(?:-\d{1,3})?)\s*-\s*"
)
# disc-id PREFIX token followed by a " - " or plain-space separator. The token
# (e.g. "FATBIRD-163", "IK00-01", "KFNO101", "RABZ-0001", "Lopenash-002") must
# contain both a letter and a digit; acceptance is further gated in code (code
# all-uppercase, or the code matches the folder brand) so alphanumeric artists
# like "2Pac", "311" or "M83" are never eaten.
_DISC_ID_PREFIX = re.compile(r"^([A-Za-z0-9][A-Za-z0-9.\-]*?)(?:\s*-\s+|\s+)(?=\S)")
# disc-id SUFFIX at end: " - SOKC-0306", " ATG0072", " IMBK-0076". Captures the
# whole token (preserving any internal dash) so disc_id stays literal.
_DISC_ID_SUFFIX = re.compile(r"[\s\-]+([A-Z]{2,6}-?\d{2,5}(?:-\d{1,3})?)\s*$")
# Leading "(CODE) " brand prefix (Sandell "(SDK) ...").
_PAREN_CODE_PREFIX = re.compile(r"^\(\s*([A-Za-z0-9]{1,6})\s*\)\s+")
# A leading pure track-number segment, e.g. "02" or "08 (and a half)".
_LEADING_TRACK_NO = re.compile(r"^\d{1,3}(?:\s*\(.*?\))?$")
# Brand tag in parentheses at end of filename (before extension)
_BRAND_SUFFIX_PATTERN = re.compile(r"\s*-?\s*\(([^)]+)\)\s*$")
# Curated folder -> brand code map. Aligned to real KaraokeNerds community codes
# where confident (so the secondary brand_match xref can also fire). Overrides a
# prefix/suffix-derived code (disc_id keeps the literal). Folders with no
# confident code are omitted (artist/title fix already surfaces them via the
# primary exact xref). Keys are lowercased exact folder names.
FOLDER_CODE = {
"funbox karaoke": "FBK",
"sandell karaoke": "SDK",
"bellysings karaoke": "BELLY",
"dj sauly collection": "DJS",
"playkaraoke": "PLAY",
"reekies karaoke": "REEKIES",
"cereal killer karaoke": "CKK",
"wizzyoke": "WIZZY",
"brilliant trash karaoke": "BTRASH",
"lopenash": "LOPE",
"monster tracks": "MTRAX",
"popeoke": "POPE",
"kaddaok": "KADDA",
"potos power plant productions": "PPPP",
"april's choice karaoke": "APRIL",
"the nerdy singer karaoke": "TNS",
"mobile karaoke unit": "MKU",
"atg karaoke - zipped mp3+g": "ATG",
"it might be karaoke": "IMBK",
"mix tape karaoke": "MIXTAPE",
"fortekaraoke discord": "EBK",
"steve-o karaoke cult": "SOKC",
}
# Descriptor words dropped when fuzzy-matching a brand token against a folder name.
_BRAND_NOISE = {
"karaoke", "collection", "cdg", "files", "file", "discord", "videos", "video",
"mp4", "mp3", "zipped", "presents", "studio", "made", "the", "of", "and",
}
# Video and audio extensions we care about
KARAOKE_EXTENSIONS = {
".mp4", ".mkv", ".avi", ".webm", ".mov", # Video
".mp3", ".cdg", # CDG+MP3 pairs
".zip", # Zipped CDG+MP3
}
# Files to skip during indexing
IGNORED_FILES = {
".DS_Store", ".keep", ".gitkeep", "desktop.ini", "Thumbs.db",
"kj-nomad.index.json",
}
def _strip_diacritics(text: str) -> str:
text = unicodedata.normalize("NFD", text)
return "".join(c for c in text if unicodedata.category(c) != "Mn")
def normalize_for_search(text: str) -> str:
"""Normalize text for fuzzy matching: lowercase, strip diacritics, remove special chars."""
"""Normalize text for fuzzy matching: lowercase, strip diacritics/special chars."""
if not text:
return ""
# NFD decomposition then strip combining characters (diacritics)
text = unicodedata.normalize("NFD", text)
text = "".join(c for c in text if unicodedata.category(c) != "Mn")
text = _strip_diacritics(text)
text = text.lower().strip()
# Strip common karaoke suffixes
text = _KARAOKE_SUFFIXES.sub("", text)
# Remove "the " prefix for matching
text = _KARAOKE_TAG.sub("", text)
if text.startswith("the "):
text = text[4:]
# Normalize whitespace and strip non-alphanumeric (keep spaces)
text = re.sub(r"[^\w\s]", "", text)
text = re.sub(r"\s+", " ", text).strip()
return text
def _brand_key(folder: str) -> str:
"""Significant-word signature of a folder/brand name for fuzzy comparison."""
base = re.sub(r"[^\w\s]", " ", _strip_diacritics(folder or "").lower())
words = [w for w in base.split() if w and w not in _BRAND_NOISE]
return "".join(words)
def _brand_acronym(folder: str) -> str:
# Initials of ALL words (brands often acronym the full name, e.g.
# "The Nerdy Singer Karaoke" -> TNSK, "Steve-O Karaoke Cult" -> SOKC).
words = re.findall(r"[A-Za-z0-9]+", _strip_diacritics(folder or "").lower())
return "".join(w[0] for w in words).upper()
def _folder_code(folder: str) -> str | None:
return FOLDER_CODE.get((folder or "").strip().lower())
def _matches_folder_brand(token: str, folder: str) -> bool:
"""True if a stripped token looks like this folder's brand (not song text)."""
if not token or not folder:
return False
tk = _brand_key(token)
fk = _brand_key(folder)
if tk and fk and (tk == fk or (len(tk) >= 4 and (tk in fk or fk in tk))):
return True
up = token.strip().strip("()[]").upper()
if up and (up == _brand_acronym(folder) or up == (_folder_code(folder) or "\0")):
return True
return False
def detect_format(filename: str) -> str:
"""Detect the karaoke format from a filename."""
"""Detect karaoke file format from extension."""
ext = os.path.splitext(filename)[1].lower()
if ext == ".zip":
return "zip"
@@ -70,32 +151,97 @@ def detect_format(filename: str) -> str:
if ext == ".mp3":
return "mp3"
if ext in (".mp4", ".mkv", ".avi", ".webm", ".mov"):
return ext[1:] # Strip the dot
return ext[1:]
return ext[1:] if ext else "unknown"
def should_index_file(filename: str) -> bool:
"""Check if a file should be included in the index."""
"""Whether a file should be included in the index."""
if filename in IGNORED_FILES or filename.startswith("."):
return False
ext = os.path.splitext(filename)[1].lower()
return ext in KARAOKE_EXTENSIONS
def _strip_trailing_noise(text: str) -> str:
"""Remove a trailing generic karaoke tag and dangling separators."""
prev = None
while prev != text:
prev = text
text = _KARAOKE_TAG.sub("", text).strip()
text = re.sub(r"[\s\-]+$", "", text).strip()
return text
def _disc_code(token: str) -> str:
"""Derive the brand code (stem) from a disc-id token.
"IK00-01" -> "IK00", "FATBIRD-163" -> "FATBIRD", "C11K-0000A" -> "C11K",
"KFNO101" -> "KFNO", "Lopenash-002" -> "Lopenash".
"""
m = re.match(r"^(.*?)-\d[\w]*$", token)
if m:
return m.group(1)
m = re.match(r"^([A-Za-z]+)\d+$", token)
if m:
return m.group(1)
return token
def _try_prefix(name: str, folder: str) -> tuple[str, str | None, str | None, bool]:
"""Strip a leading brand token.
Returns (remainder, brand_code, disc_id, disc_prefix_stripped).
"""
# 1. disc-id prefix (token contains a letter and a digit; accepted only when
# the code is all-uppercase or matches the folder brand).
m = _DISC_ID_PREFIX.match(name)
if m:
token = m.group(1)
if re.search(r"[A-Za-z]", token) and re.search(r"\d", token):
code = _disc_code(token)
num_part = token[len(code):]
# "strong" disc-id: code separated from number by a hyphen, or the
# number has >=3 digits. Prevents alphanumeric artists like "UB40",
# "D12", "AC3" (uppercase but not disc-shaped) from being eaten.
strong = "-" in num_part or len(re.findall(r"\d", num_part)) >= 3
if _matches_folder_brand(code, folder) or (
code.isupper() and len(code) >= 2 and strong
):
return name[m.end():].strip(), code, token, True
# 2. "(CODE) " paren prefix
m = _PAREN_CODE_PREFIX.match(name)
if m:
return name[m.end():].strip(), m.group(1).upper(), None, False
# 3. "CODE - " / "Brand - " prefix (no number), gated on folder match
m = re.match(r"^(.{1,30}?)\s+-\s+(?=\S)", name)
if m:
token = m.group(1).strip()
if _matches_folder_brand(token, folder):
code = token.upper() if re.fullmatch(r"[A-Za-z0-9]{1,8}", token) else None
return name[m.end():].strip(), code, None, False
return name, None, None, False
def _strip_brand_suffix(core: str, folder: str) -> str:
"""Strip a trailing [Brand]/(Brand)/ - Brand token when it matches the folder."""
# bracket / paren suffix
m = re.search(r"\s*[\(\[]([^)\]]+)[\)\]]\s*$", core)
if m and _matches_folder_brand(m.group(1), folder):
return core[: m.start()].strip()
# " - Brand" dash suffix (only if there is something before it to keep)
if " - " in core:
head, _, last = core.rpartition(" - ")
if head and _matches_folder_brand(last, folder):
return head.strip()
return core
def parse_filename(filename: str, brand_folder: str = "") -> dict:
"""
Parse a karaoke filename into structured metadata.
Args:
filename: The file name (not path), e.g. "NOMAD-0001 - Artist - Title.mp4"
brand_folder: The parent brand folder name, used as fallback brand
Returns:
dict with keys: artist, title, disc_id, brand_code, brand_name, format
"""
# Strip extension
name, ext = os.path.splitext(filename)
file_format = detect_format(filename)
"""Parse a karaoke filename into structured metadata (see module docstring)."""
name, _ext = os.path.splitext(filename)
# Normalise spaced en/em dashes to the standard " - " artist/title separator.
name = name.replace(" – ", " - ").replace(" — ", " - ").strip()
result = {
"artist": None,
@@ -103,45 +249,53 @@ def parse_filename(filename: str, brand_folder: str = "") -> dict:
"disc_id": None,
"brand_code": None,
"brand_name": brand_folder or None,
"format": file_format,
"format": detect_format(filename),
}
# Try Pattern 1: "BRAND-001 - Artist - Title"
disc_match = _DISC_ID_PATTERN.match(name)
if disc_match:
result["brand_code"] = disc_match.group(1)
result["disc_id"] = f"{disc_match.group(1)}-{disc_match.group(2)}"
remainder = name[disc_match.end():]
parts = [p.strip() for p in remainder.split(" - ")]
if len(parts) >= 2:
result["artist"] = parts[0]
result["title"] = " - ".join(parts[1:])
elif len(parts) == 1 and parts[0]:
result["title"] = parts[0]
# Strip brand suffix from title if present
if result["title"]:
result["title"] = _BRAND_SUFFIX_PATTERN.sub("", result["title"])
return result
# 1. leading brand token (disc-id / paren-code / folder-name prefix)
core, brand_code, disc_id, disc_prefix_stripped = _try_prefix(name, brand_folder)
if disc_prefix_stripped:
# Some brands write "CODE-0002 -Artist" (dash hugging the artist); drop the
# leftover leading separator so the artist isn't "-Artist".
core = core.lstrip(" -–").strip()
# Try Pattern 2: "Artist - Title - (Brand Tag)"
brand_suffix_match = _BRAND_SUFFIX_PATTERN.search(name)
if brand_suffix_match:
result["brand_name"] = brand_suffix_match.group(1)
name_without_brand = name[:brand_suffix_match.start()]
parts = [p.strip() for p in name_without_brand.split(" - ")]
if len(parts) >= 2:
result["artist"] = parts[0]
result["title"] = " - ".join(parts[1:])
elif len(parts) == 1 and parts[0]:
result["title"] = parts[0]
return result
# 2. trailing generic karaoke tag (so a disc-id suffix is reachable)
core = _strip_trailing_noise(core)
# Pattern 3: "Artist - Title" (brand from folder name)
parts = [p.strip() for p in name.split(" - ")]
# 3. disc-id SUFFIX (strong signal; unconditional)
if disc_id is None:
m = _DISC_ID_SUFFIX.search(core)
if m:
token = m.group(1)
core = core[: m.start()].strip()
core = _strip_trailing_noise(core)
brand_code = brand_code or _disc_code(token)
disc_id = token
# 4. trailing brand token in brackets/parens/dash, gated on folder
core = _strip_brand_suffix(core, brand_folder)
core = _strip_trailing_noise(core)
# 5. split remainder into artist / title. Drop a leading track-number segment
# only when it followed a disc-id prefix (e.g. "MIX001 - 02 - Artist - T"),
# so numeric artists like "311" are never mistaken for a track number.
parts = [p.strip() for p in core.split(" - ") if p.strip()]
if (
disc_prefix_stripped
and len(parts) >= 3
and _LEADING_TRACK_NO.match(parts[0])
):
parts = parts[1:]
if len(parts) >= 2:
result["artist"] = parts[0]
result["title"] = " - ".join(parts[1:])
elif len(parts) == 1 and parts[0]:
elif len(parts) == 1:
result["title"] = parts[0]
if result["title"]:
result["title"] = _KARAOKE_TAG.sub("", result["title"]).strip()
# 6. brand_code: curated folder map overrides; else captured code
result["disc_id"] = disc_id
result["brand_code"] = _folder_code(brand_folder) or brand_code
return result
+15 -1
View File
@@ -8,6 +8,8 @@ Creates:
- Cloud Scheduler to rebuild xref daily (after mirror + KN sync complete)
"""
from pathlib import Path
import pulumi
import pulumi_gcp as gcp
from pulumi_gcp import (
@@ -72,6 +74,17 @@ def create_divebar_lookup_resources(all_secrets: dict) -> dict:
)
resources["source_bucket"] = source_bucket
# Content-hashed source archive so `pulumi up` redeploys the function on code
# changes (a static object name never triggered a redeploy).
source_dir = Path(__file__).parent.parent / "functions" / "divebar_lookup"
source_archive = storage.BucketObject(
"divebar-lookup-source",
bucket=source_bucket.name,
name="divebar-lookup-source.zip",
source=pulumi.FileArchive(str(source_dir)),
)
resources["source_archive"] = source_archive
# ==================== BigQuery: Cross-Reference Table ====================
xref_table = bigquery.Table(
@@ -103,7 +116,8 @@ def create_divebar_lookup_resources(all_secrets: dict) -> dict:
source=cloudfunctionsv2.FunctionBuildConfigSourceArgs(
storage_source=cloudfunctionsv2.FunctionBuildConfigSourceStorageSourceArgs(
bucket=source_bucket.name,
object="divebar-lookup-source.zip",
object=source_archive.name,
generation=source_archive.generation,
),
),
),
+19 -1
View File
@@ -10,6 +10,8 @@ Creates:
- BigQuery table for the Divebar catalog index
"""
from pathlib import Path
import pulumi
import pulumi_gcp as gcp
from pulumi_gcp import (
@@ -99,6 +101,18 @@ def create_divebar_mirror_resources(all_secrets: dict) -> dict:
)
resources["source_bucket"] = source_bucket
# Zip the function source and upload it as a content-hashed object so that
# `pulumi up` redeploys the function whenever the code changes (the previous
# static object name never triggered a redeploy on code edits).
source_dir = Path(__file__).parent.parent / "functions" / "divebar_mirror"
source_archive = storage.BucketObject(
"divebar-mirror-source",
bucket=source_bucket.name,
name="divebar-mirror-source.zip",
source=pulumi.FileArchive(str(source_dir)),
)
resources["source_archive"] = source_archive
# ==================== BigQuery Table ====================
# The table is in the karaoke_decide dataset (managed by karaoke-decide Pulumi).
@@ -143,7 +157,11 @@ def create_divebar_mirror_resources(all_secrets: dict) -> dict:
source=cloudfunctionsv2.FunctionBuildConfigSourceArgs(
storage_source=cloudfunctionsv2.FunctionBuildConfigSourceStorageSourceArgs(
bucket=source_bucket.name,
object="divebar-mirror-source.zip",
object=source_archive.name,
# Pin to the object generation so replacing the source bundle
# (a code change) registers as a diff on the Function and
# forces a redeploy, instead of silently serving the old copy.
generation=source_archive.generation,
),
),
),
+1 -1
View File
@@ -1,6 +1,6 @@
[tool.poetry]
name = "karaoke-gen"
version = "0.188.3"
version = "0.188.4"
description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens."
authors = ["Andrew Beveridge <andrew@beveridge.uk>"]
license = "MIT"
+336
View File
@@ -0,0 +1,336 @@
"""Unit tests for the Divebar filename parser.
The parser lives in the Cloud Function package (not on the default path), so we
add its directory to sys.path and import it directly. It has stdlib-only deps.
Fixtures are real filenames observed in the live `divebar_catalog` index, grouped
by the naming convention each community brand uses. The goal of the rewrite is
folder-brand-aware parsing: use the (reliable) top-level folder brand to strip the
brand token wherever it sits, then split the remainder into artist/title.
"""
import pathlib
import sys
import pytest
FUNC_DIR = (
pathlib.Path(__file__).resolve().parents[2]
/ "infrastructure"
/ "functions"
/ "divebar_mirror"
)
sys.path.insert(0, str(FUNC_DIR))
import filename_parser as fp # noqa: E402
def parse(filename, folder):
return fp.parse_filename(filename, brand_folder=folder)
# --------------------------------------------------------------------------
# disc-id PREFIX (works today — must be preserved)
# --------------------------------------------------------------------------
@pytest.mark.parametrize(
"filename,folder,artist,title,brand_code,disc_id",
[
("FATBIRD-163 - Incubus - I Miss You.zip", "Fatbird Karaoke",
"Incubus", "I Miss You", "FATBIRD", "FATBIRD-163"),
("NOMAD-0003 - Cher - Apples Don't Fall Far From The Tree.mp4", "Nomad Karaoke",
"Cher", "Apples Don't Fall Far From The Tree", "NOMAD", "NOMAD-0003"),
("IK00-01 - Costello, Elvis - Goon Squad.zip", "Imperfekt Karaoke",
"Costello, Elvis", "Goon Squad", "IK00", "IK00-01"),
("GGZ001-02 - Partman Parthorse - Magik (ggnzla REMIX).cdg", "GGZ-ggnzla",
"Partman Parthorse", "Magik (ggnzla REMIX)", "GGZ001", "GGZ001-02"),
("KFNO101 - The Kills - List of Demands (Reparations).mp4", "Karaoke For No One",
"The Kills", "List of Demands (Reparations)", "KFNO", "KFNO101"),
],
)
def test_disc_id_prefix(filename, folder, artist, title, brand_code, disc_id):
r = parse(filename, folder)
assert r["artist"] == artist
assert r["title"] == title
assert r["brand_code"] == brand_code
assert r["disc_id"] == disc_id
# --------------------------------------------------------------------------
# disc-id PREFIX, space-separated / mixed-case / digit-leading
# --------------------------------------------------------------------------
@pytest.mark.parametrize(
"filename,folder,artist,title,disc_id",
[
# Potos: "CODE-#### Artist - Title" (space, not " - ", after disc-id)
("RABZ-0001 The Seatbelts & Steve Conte - Call Me, Call Me.zip",
"Potos Power Plant Productions",
"The Seatbelts & Steve Conte", "Call Me, Call Me", "RABZ-0001"),
# Lopenash: mixed-case folder-name prefix
("Lopenash-002 - Dexter's Laboratory - Breathe in The Sunshine.mp4", "Lopenash",
"Dexter's Laboratory", "Breathe in The Sunshine", "Lopenash-002"),
# 7uLpA: digit-leading mixed-case token
("7uLpA-0004 - The Beauty of Gemina - Wonders.mp4", "7uLpA_Made",
"The Beauty of Gemina", "Wonders", "7uLpA-0004"),
],
)
def test_disc_id_prefix_variants(filename, folder, artist, title, disc_id):
r = parse(filename, folder)
assert r["artist"] == artist
assert r["title"] == title
assert r["disc_id"] == disc_id
def test_disc_id_prefix_single_field():
# Lopenash medley with no artist/title split -> title only, artist None
r = parse("Lopenash-001 - Super Energy Apocalypse.zip", "Lopenash")
assert r["artist"] is None
assert r["title"] == "Super Energy Apocalypse"
assert r["disc_id"] == "Lopenash-001"
def test_disc_id_prefix_drops_leading_track_number():
# Mix Tape: "CODE-### - NN - Artist - Title" -> drop the NN track number
r = parse("MIX001 - 02 - Psychostick - Mega Man.mp4", "Mix Tape Karaoke")
assert r["artist"] == "Psychostick"
assert r["title"] == "Mega Man"
# --------------------------------------------------------------------------
# disc-id SUFFIX (unconditional strip)
# --------------------------------------------------------------------------
@pytest.mark.parametrize(
"filename,folder,artist,title,brand_code,disc_id",
[
("3Blue1Brown - How They Fool Ya - IMBK-0076.mp4", "It Might Be Karaoke",
"3Blue1Brown", "How They Fool Ya", "IMBK", "IMBK-0076"),
("54-40 - Nice to Luv you - SOKC-0036 (KARAOKE).zip", "Steve-O Karaoke Cult",
"54-40", "Nice to Luv you", "SOKC", "SOKC-0036"),
("Against Me - Wagon Wheel - SOKC-0306 - (KARAOKE).zip", "Steve-O Karaoke Cult",
"Against Me", "Wagon Wheel", "SOKC", "SOKC-0306"),
# disc-id suffix with no dash before the code
("Age Of Electric - Remote Control SOKC-0042 (KARAOKE).zip", "Steve-O Karaoke Cult",
"Age Of Electric", "Remote Control", "SOKC", "SOKC-0042"),
# ATG: code with no dash inside (ATG0014)
("Angelcorpse - Wolflust - ATG0014.zip", "ATG Karaoke - Zipped MP3+G",
"Angelcorpse", "Wolflust", "ATG", "ATG0014"),
("1349 - Singer Of Strange Songs - ATG0072.zip", "ATG Karaoke - Zipped MP3+G",
"1349", "Singer Of Strange Songs", "ATG", "ATG0072"),
],
)
def test_disc_id_suffix(filename, folder, artist, title, brand_code, disc_id):
r = parse(filename, folder)
assert r["artist"] == artist
assert r["title"] == title
assert r["brand_code"] == brand_code
assert r["disc_id"] == disc_id
# --------------------------------------------------------------------------
# bracket / paren brand SUFFIX (gated on folder match)
# --------------------------------------------------------------------------
@pytest.mark.parametrize(
"filename,folder,artist,title",
[
("3rd Bass - Green Eggs And Swine [DJ Sauly Karaoke].zip", "DJ Sauly Collection",
"3rd Bass", "Green Eggs And Swine"),
# keep the legit (Banda Remix), strip only the [DJ Sauly Karaoke] tag
("50 Cent - In Da Club (Banda Remix) [DJ Sauly Karaoke].zip", "DJ Sauly Collection",
"50 Cent", "In Da Club (Banda Remix)"),
("10 Years - Fix Me (BellySings).mp4", "BellySings Karaoke",
"10 Years", "Fix Me"),
("Bad Religion - Fuck You (popeoke).zip", "Popeoke",
"Bad Religion", "Fuck You"),
("B.E.R. - The Night Begins to Shine [popeoke].zip", "Popeoke",
"B.E.R.", "The Night Begins to Shine"),
],
)
def test_brand_suffix_bracket_or_paren(filename, folder, artist, title):
r = parse(filename, folder)
assert r["artist"] == artist
assert r["title"] == title
# --------------------------------------------------------------------------
# dash brand SUFFIX (gated on folder match / per-folder token)
# --------------------------------------------------------------------------
@pytest.mark.parametrize(
"filename,folder,artist,title",
[
("100 Gecs - Last Train To Awesometown - Rock Solid Karaoke.zip",
"Rock Solid Karaoke CDG files", "100 Gecs", "Last Train To Awesometown"),
("311 - Amber - Rock Solid Karaoke.zip",
"Rock Solid Karaoke CDG files", "311", "Amber"),
("Akon - Right Now (Na Na Na) - EBK.avi", "ForteKaraoke Discord",
"Akon", "Right Now (Na Na Na)"),
],
)
def test_brand_suffix_dash(filename, folder, artist, title):
r = parse(filename, folder)
assert r["artist"] == artist
assert r["title"] == title
# --------------------------------------------------------------------------
# code / folder-name PREFIX (no number)
# --------------------------------------------------------------------------
@pytest.mark.parametrize(
"filename,folder,artist,title,brand_code",
[
("(SDK) (G)I-DLE - Queencard.cdg", "Sandell Karaoke",
"(G)I-DLE", "Queencard", "SDK"),
("(SDK) 21 Pilots - House Of Gold.cdg", "Sandell Karaoke",
"21 Pilots", "House Of Gold", "SDK"),
("CKK - Buoys, The - Timothy.zip", "Cereal Killer Karaoke",
"Buoys, The", "Timothy", "CKK"),
("CKK - A Day To Remember - If It Means A Lot To You.zip", "Cereal Killer Karaoke",
"A Day To Remember", "If It Means A Lot To You", "CKK"),
("TNSK - Davy Jones - Girl.zip", "The Nerdy Singer Karaoke",
"Davy Jones", "Girl", "TNS"),
("Crossfire - Eurythmics - Missionary Man.mp4", "Crossfire Karaoke",
"Eurythmics", "Missionary Man", None),
],
)
def test_code_or_name_prefix(filename, folder, artist, title, brand_code):
r = parse(filename, folder)
assert r["artist"] == artist
assert r["title"] == title
assert r["brand_code"] == brand_code
# --------------------------------------------------------------------------
# generic karaoke tag strip (display title, both () and [])
# --------------------------------------------------------------------------
@pytest.mark.parametrize(
"filename,folder,artist,title",
[
("Bill Murray - Star Wars [Karaoke].mp4", "Mobile Karaoke Unit",
"Bill Murray", "Star Wars"),
("3 Doors Down - Down Poison (Karaoke HD).zip", "Funbox Karaoke",
"3 Doors Down", "Down Poison"),
("Islands - Pumpkin (Karaoke Version).mp4", "Matt Joy Karaoke",
"Islands", "Pumpkin"),
],
)
def test_generic_karaoke_tag_stripped_from_title(filename, folder, artist, title):
r = parse(filename, folder)
assert r["artist"] == artist
assert r["title"] == title
# --------------------------------------------------------------------------
# clean "Artist - Title" — only the curated brand_code map applies
# --------------------------------------------------------------------------
@pytest.mark.parametrize(
"filename,folder,artist,title,brand_code",
[
("A Tribe Called Quest - Lyrics To Go.avi", "WIZZYOKE",
"A Tribe Called Quest", "Lyrics To Go", "WIZZY"),
("100 gecs - Hollywood Baby.cdg", "Funbox Karaoke",
"100 gecs", "Hollywood Baby", "FBK"),
("Alice in Chains - Angry Chair.mp4", "Brilliant Trash Karaoke",
"Alice in Chains", "Angry Chair", "BTRASH"),
("16 Horsepower - American Wheeze.cdg", "Monster Tracks",
"16 Horsepower", "American Wheeze", "MTRAX"),
],
)
def test_clean_artist_title_with_curated_code(filename, folder, artist, title, brand_code):
r = parse(filename, folder)
assert r["artist"] == artist
assert r["title"] == title
assert r["brand_code"] == brand_code
# --------------------------------------------------------------------------
# GUARDS — must NOT over-strip real song text
# --------------------------------------------------------------------------
def test_guard_keeps_non_brand_bracket_in_title():
# [Nuxx] and (RVZN Edit) are NOT the folder brand -> keep them
r = parse("RVZ-016 - UNDERWORLD - Born Slippy [Nuxx] (RVZN Edit).mp4", "R3V1Z10N Karaoke")
assert r["artist"] == "UNDERWORLD"
assert r["title"] == "Born Slippy [Nuxx] (RVZN Edit)"
assert r["brand_code"] == "RVZ"
def test_guard_keeps_paren_artist():
r = parse("(SDK) (G)I-DLE - Queencard.cdg", "Sandell Karaoke")
assert r["artist"] == "(G)I-DLE" # only the (SDK) brand prefix removed
def test_guard_keeps_live_qualifier_in_title():
# A trailing (Live) that is not the brand must survive (folder doesn't match)
r = parse("311 - All Re-Mixed Up (live) - Rock Solid Karaoke.zip",
"Rock Solid Karaoke CDG files")
assert r["artist"] == "311"
assert r["title"] == "All Re-Mixed Up (live)"
def test_guard_numeric_artist_preserved():
r = parse("311 - Amber - Rock Solid Karaoke.zip", "Rock Solid Karaoke CDG files")
assert r["artist"] == "311"
def test_guard_alphanumeric_artist_not_eaten_as_disc_id():
# "UB40", "D12" are uppercase+digits but NOT disc-shaped (no hyphen, <3 digit
# number) -> must stay the artist, not be stripped as a disc-id prefix.
r = parse("UB40 - Kingston Town (BellySings).mp4", "BellySings Karaoke")
assert r["artist"] == "UB40"
assert r["title"] == "Kingston Town"
r2 = parse("D12 - My Band.mp4", "Funbox Karaoke")
assert r2["artist"] == "D12"
assert r2["title"] == "My Band"
def test_en_dash_separator():
r = parse("Iron Maiden – The Writing On The Wall.mp4", "ATG Karaoke - Zipped MP3+G")
assert r["artist"] == "Iron Maiden"
assert r["title"] == "The Writing On The Wall"
def test_strips_other_brand_karaoke_tag_in_title():
# Funbox re-hosts WTF Karaoke files tagged "- (WTF Karaoke)"; the tag mentions
# karaoke so it is stripped even though it isn't the folder brand.
r = parse("Alice in Chains - Heaven Beside You - (WTF Karaoke).zip", "Funbox Karaoke")
assert r["artist"] == "Alice in Chains"
assert r["title"] == "Heaven Beside You"
def test_leading_dash_hugging_artist_cleaned():
r = parse("7uLpA-0013 -Zynic - After Effects.mp4", "7uLpA_Made")
assert r["artist"] == "Zynic"
assert r["title"] == "After Effects"
# --------------------------------------------------------------------------
# normalized fields flow from parsed artist/title (xref keys)
# --------------------------------------------------------------------------
def test_normalized_fields_use_cleaned_title():
r = parse("3rd Bass - Green Eggs And Swine [DJ Sauly Karaoke].zip", "DJ Sauly Collection")
assert fp.normalize_for_search(r["artist"]) == "3rd bass"
assert fp.normalize_for_search(r["title"]) == "green eggs and swine"
# --------------------------------------------------------------------------
# existing helpers still behave
# --------------------------------------------------------------------------
def test_should_index_file():
assert fp.should_index_file("x.mp4") is True
assert fp.should_index_file("x.zip") is True
assert fp.should_index_file(".DS_Store") is False
assert fp.should_index_file("notes.txt") is False
def test_detect_format():
assert fp.detect_format("x.zip") == "zip"
assert fp.detect_format("x.MP4") == "mp4"
assert fp.detect_format("x.cdg") == "cdg"