diff --git a/docs/archive/2026-06-28-divebar-index-parsing-plan.md b/docs/archive/2026-06-28-divebar-index-parsing-plan.md new file mode 100644 index 00000000..7fa19692 --- /dev/null +++ b/docs/archive/2026-06-28-divebar-index-parsing-plan.md @@ -0,0 +1,255 @@ +# Divebar Index Filename-Parsing Fix — Plan + +_Worktree: `karaoke-gen-divebar-index-parsing` · branch `feat/sess-20260628-0511-divebar-index-parsing`_ +_Picks up: HANDOFF-divebar-mirror-parsing.md (kjbox side shipped v0.41.1; this is the upstream index fix)_ + +## Problem (refined by live data, not just the handoff examples) + +The Divebar Drive→BigQuery indexer (`infrastructure/functions/divebar_mirror/filename_parser.py`, +Cloud Function `divebar-mirror`) mis-parses community-brand filenames. kjbox cross-references +mirror files on `(normalized artist, normalized title)` + brand, so bad parses → mirror files +never surface in the rotation selector even though the file is in GCS. + +### Measured state (BigQuery `karaoke_decide.divebar_catalog`, 50,046 rows) + +| Signal | Count | Meaning | +|---|---|---| +| null `artist` | 164 (0.3%) | rare | +| null `title` | 0 | — | +| **null `brand_code`** | **18,437 (37%)** | biggest visible gap | +| `artist` is 2–6 all-caps/digits | 1,908 | brand-code-as-artist mis-split (e.g. `CKK`, `EBK`) | +| `artist` starts with `(` | 3,051 | `(SDK) Artist` prefix (Sandell) | +| `title` contains ` - ` | 5,947 | unstripped ` - Brand` suffix / disc-id-in-title | +| `title` ends in `[..]`/`(..)` | 2,583 | `[DJ Sauly Karaoke]`, `[Karaoke]`, etc. | + +Current xref coverage: 90,719 matches, 17,261 KN songs, **23,318 / 50,046 divebar files matched (47%)**. + +> Reframe vs handoff: `artist`/`title` are rarely *null*, but **mis-splits + polluted titles** +> are common and silently defeat the EXACT xref (`LOWER(TRIM(kn.Title)) = db.title_normalized`). +> Every polluted title misses the join. That + missing `brand_code` is the real lever. + +### Root cause: parser has only 3 rigid patterns and never uses the folder brand to clean + +`parse_filename(name, brand_folder)`: +1. `^CODE-### - Artist - Title` (disc-id prefix) — works. +2. `... (BrandTag)$` — strips ANY trailing paren (so it also eats `(KARAOKE)`, `(Live)`, + `(Banda Remix)` → clobbers `brand_name`), and never sets `brand_code`. +3. `Artist - Title` fallback split on ` - `. + +It ignores `[...]` brackets, ` - Brand` dash-suffixes, disc-id *suffixes*, `(CODE) ` prefixes, +and `CODE - ` (no-number) prefixes. The folder name (reliable brand) is passed in but only used as +a fallback `brand_name` — **never to strip the brand token before splitting artist/title**. +`_KARAOKE_SUFFIXES` is applied only to the normalized fields, never the display `title`. + +Note: `build_rows` writes `brand` = folder (always, reliable) and only consumes `brand_code` from +the parser (not `brand_name`) — so Pattern-2's `brand_name` clobbering is harmless. The fields that +matter from the parser are **artist, title, brand_code, disc_id**. + +## Real-world conventions (gold fixtures, observed live) + +| Convention | Example filename | Brand (folder) | Correct artist / title | +|---|---|---|---| +| disc-id prefix (OK today) | `FATBIRD-163 - Incubus - I Miss You.zip` | Fatbird | Incubus / I Miss You | +| code prefix, no number | `CKK - Buoys, The - Timothy.zip` | Cereal Killer Karaoke | Buoys, The / Timothy | +| `(CODE)` paren prefix | `(SDK) (G)I-DLE - Queencard.cdg` | Sandell Karaoke | (G)I-DLE / Queencard | +| `(Brand)` paren suffix (OK today) | `10 Years - Fix Me (BellySings).mp4` | BellySings | 10 Years / Fix Me | +| `[Brand]` bracket suffix | `3rd Bass - Green Eggs And Swine [DJ Sauly Karaoke].zip` | DJ Sauly Collection | 3rd Bass / Green Eggs And Swine | +| ` - Brand` dash suffix | `100 Gecs - Last Train To Awesometown - Rock Solid Karaoke.zip` | Rock Solid Karaoke | 100 Gecs / Last Train To Awesometown | +| ` - CODE` dash suffix | `Akon - Right Now (Na Na Na) - EBK.avi` | ForteKaraoke Discord | Akon / Right Now (Na Na Na) | +| disc-id *suffix* | `Against Me - Wagon Wheel - SOKC-0306 - (KARAOKE).zip` | Steve-O Karaoke Cult | Against Me / Wagon Wheel | +| disc-id suffix no dash | `Angelcorpse - Wolflust - ATG0014.zip` | ATG Karaoke | Angelcorpse / Wolflust | +| `[Karaoke]` generic tag | `Bill Murray - Star Wars [Karaoke].mp4` | Mobile Karaoke Unit | Bill Murray / Star Wars | +| mixed-case brand prefix | `Lopenash-002 - Dexter's Laboratory - Breathe in The Sunshine.mp4` | Lopenash | Dexter's Laboratory / Breathe in The Sunshine | +| clean (only brand_code missing) | `A Tribe Called Quest - Lyrics To Go.avi` | WIZZYOKE | A Tribe Called Quest / Lyrics To Go | + +**Guard cases (must NOT over-strip):** `(G)I-DLE`, `R.E.M.`, artist `311`; titles `(Live)`, +`(Banda Remix)`, `(Single Version 2 WBV)`, `(99X Live)`, `[Acoustic]` must survive. + +## Design — folder-brand-aware parser + +Core principle (per handoff): **the top-level folder reliably names the brand; use it to recognise +and strip the brand token wherever it sits, then split the remainder into artist/title.** + +`parse_filename(name, brand_folder)` rewrite, in order: + +1. **Strip generic karaoke tags** from the display name: trailing `(karaoke|instrumental|backing + track|no vocals|kj version|with vocals|demo)` in either `()` or `[]` (reuse `_KARAOKE_SUFFIXES`, + now applied to display too). +2. **Build brand matchers from `brand_folder`**: normalized full name, and name with trailing + descriptors removed (`karaoke`, `collection`, `cdg files`, `discord`, `videos`, `mp4`, + `zipped mp3+g`, etc.); plus an acronym of significant words (Cereal Killer Karaoke→CKK, + Steve-O Karaoke Cult→SOKC, ATG Karaoke→ATG). +3. **Strip the brand token from whichever position it occupies — each gated so we only remove a + *known* brand, never song text:** + - disc-id **prefix** `^CODE[-\s]?### - ` → `brand_code`, `disc_id`. (existing) + - disc-id **suffix** ` - CODE-### ` / ` CODE#### ` at end, gated on `CODE` ≈ folder acronym/known → `brand_code`,`disc_id`. + - `[Brand]`/`(Brand)` **bracket/paren suffix** where content ≈ folder brand → strip; `brand_code` from folder map. + - ` - Brand` **dash suffix** where the last ` - ` segment ≈ folder brand → strip. + - `(CODE) ` **paren prefix** (≤6 chars, alnum) with content after → strip; `brand_code`=CODE. + - `CODE - ` / `Folder-### - ` **prefix** where token ≈ folder acronym/name → strip; `brand_code`. +4. **Split remainder** on ` - ` → artist = parts[0], title = join(parts[1:]); strip any leftover + generic karaoke tag from title. +5. **brand_code resolution order:** captured-during-strip → small curated **folder→code map** → + null. `brand` always = folder (unchanged in `build_rows`). + +### Brand registry decision (diverges from handoff) + +Handoff suggested importing kjbox `version_priority.py` `COMMUNITY_BRANDS`. **Rejected** because: +(a) cross-repo import isn't viable inside a GCP Cloud Function, (b) that registry is **incomplete** +vs the 63 real Divebar folder brands (no Cereal Killer/DJ Sauly/Rock Solid/Sandell), and (c) it +**conflicts** — registry maps `SDK`→SNDL but Sandell's folder uses `(SDK)`. Instead: derive +`brand_code` from the in-filename token / folder, and keep a **small curated `folder→KN-code` map** +seeded from the actual KN community codes that intersect Divebar folders (FBK, BELLY, DJS, C11K, +PLAY, REEKIES, HALJAM, NOMAD, FAKEY, CC, …). kjbox's read-time `canonical_brand_for_match` already +folds codes/names, so the index doesn't need canonical output — just resolvable. + +## Refined design from full 63-brand data (read-only export) + +Exported all 50,046 rows + per-brand samples + KN community codes to scratchpad. Key refinement: +**artist is wrong-but-rarely-null; the measurable gaps are polluted titles + null brand_code.** +Brands needing a fix fall into these convention buckets (✅ already-correct buckets only need the +brand_code map): + +| Bucket | Brands (examples) | Fix | +|---|---|---| +| disc-id PREFIX `CODE-### - A - T` | Nomad, Imperfekt, Play, Gnome, Fakey, C11K, Reekies, Fatbird, … | works today — **preserve** | +| disc-id PREFIX space-sep `CODE-#### Artist - T` | Potos (`RABZ-0001 The Seatbelts - …`) | allow space (not just ` - `) after disc-id | +| disc-id PREFIX mixed-case/digit-lead `Tok-### - …` | Lopenash (`Lopenash-002 -`), 7uLpA (`7uLpA-0003 -`) | match when prefix token ≈ folder name | +| leading track-no after disc-id `CODE-### - NN - A - T` | Mix Tape (`MIX001 - 02 - …`) | drop leading pure-numeric segment | +| disc-id SUFFIX `A - T - CODE-####` | It Might Be (IMBK), Steve-O (SOKC), ATG (ATG####) | **strip trailing disc-id (unconditional), set brand_code/disc_id** | +| `(CODE)` paren PREFIX | Sandell (`(SDK) A - T`) | strip leading `(short-alnum) ` | +| `CODE - ` PREFIX no number | Cereal Killer (`CKK - A - T`), Nerdy (`TNSK -`), Crossfire (`Crossfire - A - T`) | strip when token ≈ folder acronym/name | +| `[Brand]` bracket SUFFIX | DJ Sauly (`… [DJ Sauly Karaoke]`) | strip bracket suffix gated on folder-name match | +| `(Brand)` paren SUFFIX | BellySings (`… (BellySings)`), Popeoke (`(popeoke)`/`[popeoke]`) | already strips paren; add bracket + folder-gate | +| ` - Brand` dash SUFFIX | Rock Solid (`… - Rock Solid Karaoke`), Forte (`… - EBK`) | strip last ` - ` segment gated on folder-name match (Forte EBK via per-folder token) | +| generic karaoke tag `[Karaoke]`/`(Karaoke HD)` | Mobile, Funbox, Matt Joy, Steve-O | strip generic tag from **display** title (currently only normalized), tolerate trailing words | +| clean `Artist - Title`, only brand_code missing | Funbox, WIZZYOKE, Brilliant Trash, Monster Tracks, KaddaOK, April's Choice | **curated folder→code map only** | + +### Parser algorithm (folder-brand-aware), ordered + +1. strip extension; `format = detect_format`. +2. strip generic karaoke tags from display name (both `()`/`[]`, token optionally followed by words). +3. brand-token strip, first match wins, each sets `brand_code`/`disc_id` and removes the token: + (a) disc-id PREFIX `^CODE[-\s]?NUM` followed by ` - ` **or** whitespace; CODE all-caps **or** ≈ folder token (covers Lopenash/7uLpA/Potos); + (b) disc-id SUFFIX `[-\s]\s*CODE-?NUM$` — **unconditional** (trailing `ABCD-0123` is a strong signal); + (c) `[Brand]`/`(Brand)` SUFFIX gated on `norm(content) ≈ norm(folder)`; + (d) ` - Brand` dash SUFFIX gated on `norm(last segment) ≈ norm(folder)` (or per-folder token); + (e) `(CODE)` paren PREFIX (≤6 alnum) with content after; + (f) `CODE - ` PREFIX (no number) gated on `CODE ≈ folder acronym/name`. +4. split remainder on ` - ` → artist=parts[0], title=join(rest); drop a leading pure-numeric segment; + 1 part → title only. +5. strip any leftover generic karaoke tag from title. +6. `brand_code` = captured → **curated folder→code map** → null. `brand` always = folder (in `build_rows`). + +**Guards (must survive):** `(G)I-DLE`, `R.E.M.`, artist `311`, titles `(Live)`, `(Banda Remix)`, +`(Single Version 2 WBV)`, `(99X Live)`, `[Acoustic]`. Folder-gating + "disc-id has digits" + +"prefix code ≈ folder" are what prevent eating these. + +### Curated folder→KN-code map (minimal, high-confidence only) + +Aligned to actual KN community `Brand` codes (so the secondary brand_match xref can also fire); +brands with no confident KN code are left null (artist/title fix already surfaces them via the +primary 0.95 exact xref): + +``` +Funbox Karaoke→FBK · Sandell Karaoke→SDK · BellySings Karaoke→BELLY · DJ Sauly Collection→DJS +PlayKaraoke→PLAY · Reekies Karaoke→REEKIES · Cereal Killer Karaoke→CKK · WIZZYOKE→WIZZY +Brilliant Trash Karaoke→BTRASH · Lopenash→LOPE · Monster Tracks→MTRAX · Popeoke→POPE +KaddaOK→KADDA · Potos Power Plant Productions→PPPP · April's Choice Karaoke→APRIL +The Nerdy Singer Karaoke→TNS · Mobile Karaoke Unit→MKU · ATG Karaoke - Zipped MP3+G→ATG +It Might Be Karaoke→IMBK +``` +(Map *overrides* a prefix-derived code where they differ, e.g. Play `PLK`→`PLAY`, Reekies +`REEK`→`REEKIES`; `disc_id` keeps the literal prefix.) Skipped (no confident code): Rock Solid, +Steve-O (uses SOKC, not a KN code — set from disc-id suffix anyway), ForteKaraoke (EBK≠KN), Gnome, +7uLpA, Crossfire, Monster… handled where a code exists. + +## Work breakdown + +**Phase 1 — Parser rewrite (TDD, primary deliverable).** New `tests/unit/test_divebar_filename_parser.py` +with all gold fixtures + guard cases (add function dir to `sys.path`; parser deps are stdlib only). +Rewrite `filename_parser.py` to the design above. Keep `normalize_for_search`, `detect_format`, +`should_index_file` behaviour. Keep existing `test_divebar_index_builder.py` green. + +**Phase 2 — brand_code coverage.** Curated `folder→code` map + disc-id-suffix / leading-token +capture. Target: cut null `brand_code` well below 37% on the offline harness. + +**Phase 3 — Deploy enablement (infra papercut).** The module points the function at a *static* +source object (`divebar-mirror-source.zip`) with no content hash, so overwriting it won't make +`pulumi up` redeploy. Switch `modules/divebar_mirror.py` to `pulumi.FileArchive` + `BucketObject` +(the proven pattern in `modules/runner_manager.py`) so new code deploys on `pulumi up`. (Optional +but needed for the fix to ship cleanly; Andrew runs the apply.) + +**Phase 4 — Offline validation harness + prod verify.** +- Export distinct `(filename, brand)` from BQ (read-only) → run NEW parser locally over all 50k → + report before/after: % null brand_code, % artist-looks-like-code, % title-has-dash, % title-ends-bracket, + plus a spot-check sample. **Proves improvement with zero prod writes** (fits read-only ADC). +- After Andrew deploys + triggers `divebar-mirror-daily` then `divebar-xref-rebuild-daily` + (MERGE preserves `gcs_path`; no Drive→GCS re-sync needed): re-measure xref `db_files` matched + (baseline 23,318/50,046) and the live `incubus admiration` repro. + +## Constraints / notes + +- **My ADC is read-only** (workspace memory `feedback_claude_readonly_adc`): I can read BQ and run + the offline harness, but **cannot** `pulumi up`, upload source, or run scheduler jobs. Andrew + deploys + re-indexes; I validate offline first and via BQ reads after. +- Re-index re-parses ALL files (full Drive listing → MERGE by `file_id`), so existing rows are + corrected in place; `gcs_path` is preserved. +- Version bump `pyproject.toml`; this is infra/function code (no i18n, no frontend). +- Tests: `make test` (or targeted `pytest tests/unit/test_divebar_filename_parser.py`). + +## Results (offline harness over all 50,046 live rows) + +| metric | before | after | delta | +|---|---|---|---| +| null brand_code | 18,437 (37%) | 1,619 (3%) | **−16,818** | +| title contains ` - ` | 5,947 | 2,523 | −3,424 (residual mostly legit multi-part titles) | +| artist looks like a code | 1,908 | 1,400 | −508 (residual are real bands: ACDC, KISS, ABBA, U2…) | +| null artist | 164 | 179 | +15 (medleys / single-field — acceptable) | +| rows with changed artist/title | — | 13,348 (26.7%) | — | + +Per-brand `brand_code` null% collapsed to ~0% for every broken brand (Sandell, BellySings, +DJ Sauly, Cereal Killer, Steve-O, WIZZYOKE, Lopenash, ForteKaraoke, ATG, Mobile, 7uLpA, …); +Rock Solid stays null by design (no confident KN code). + +**Xref lift (replaying the live `_rebuild_xref` EXACT join):** Divebar files that match a KN +song **20,802 → 23,394 (+2,592, +12.5%)** — i.e. that many more mirror files now surface in kjbox. + +### BUNDLED: the xref SQL normalization asymmetry (fixed in this PR) + +`_rebuild_xref` joined `LOWER(TRIM(kn.Artist)) = db.artist_normalized`, but `db.artist_normalized` +is produced by `normalize_for_search` (strips diacritics, a leading "the ", and all punctuation) +while the KN side got only `LOWER(TRIM(...))` — so most rows could never match. **Fix:** a new +`_norm_sql(col)` helper routes **both** sides through one identical BigQuery expression +replicating `normalize_for_search` (NFD + drop `\p{Mn}`, lower/trim, strip trailing karaoke tag, +strip leading "the ", drop non-`\p{L}\p{N}_\s`, collapse whitespace). Applied to the exact join, +the brand_match join, and the NOT EXISTS de-dup. + +**Validated read-only against live BigQuery** (the real `_norm_sql`): the SQL normalization +reproduces the Python `normalize_for_search` baseline exactly (32,496 on current data), and the +**full new union already matches 33,449 distinct Divebar files on current (old-parse) data vs +production's 23,318 — +10,131 from symmetry alone**, before any re-index. After the parser +re-index the exact portion lifts toward ~37,910, compounding further. Residual misses after the parser fix are tiny and documented: `(WTF Karaone)` typo +(~6), `SOKC-O104` letter-O typo (~21), `GGZ007-02,2` comma in disc-id (~16), `RVZ-033.d` (~5). + +## Status: IMPLEMENTED + +- `filename_parser.py` rewritten folder-brand-aware; `tests/unit/test_divebar_filename_parser.py` + added (48 cases, green); existing `test_divebar_index_builder.py` still green. +- `divebar_lookup/main.py` `_rebuild_xref` rewritten to normalize both sides via `_norm_sql`; + 3 guard tests added to `test_divebar_lookup` (`test_main.py`). All 64 divebar tests green. +- `modules/divebar_mirror.py` switched to `pulumi.FileArchive` + `BucketObject` with + generation-pinning so `pulumi up` actually redeploys new function code; `deploy.sh` note updated. +- Version bumped 0.188.3 → 0.188.4. +- **Deploy (Andrew, read-only ADC blocks me):** `cd infrastructure && pulumi up` (redeploys both + `divebar-mirror` and `divebar-lookup`), then trigger scheduler jobs `divebar-mirror-daily` + (re-parse → MERGE, preserves `gcs_path`) then `divebar-xref-rebuild-daily`. Then re-measure xref + `db_files` and the live `incubus admiration` repro. (Order matters: re-index before xref rebuild.) + +## Open questions for Andrew + +1. **Include Phase 3** (FileArchive deploy fix) in this PR, or keep PR parser-only and handle deploy + mechanics separately? +2. **brand_code map scope** — minimal (only folders that clearly map to a KN community code) vs + broader best-effort? Minimal is safer (no mismaps); recommend minimal. +3. Confirm **you'll run the `pulumi up` + scheduler re-index** after merge (I can't with read-only ADC). diff --git a/infrastructure/functions/divebar_lookup/main.py b/infrastructure/functions/divebar_lookup/main.py index 714b6ac8..ffc53d86 100644 --- a/infrastructure/functions/divebar_lookup/main.py +++ b/infrastructure/functions/divebar_lookup/main.py @@ -177,16 +177,51 @@ def _lookup_kn_ids(kn_ids: list[int]) -> dict[int, list[dict]]: return result +def _norm_sql(col: str) -> str: + """A BigQuery expression replicating divebar_mirror's normalize_for_search. + + Both the KaraokeNerds side and the Divebar side of the xref join are passed + through this so the comparison is symmetric. Previously the join compared the + KN side (only LOWER+TRIM) against the Divebar side's fully-normalized columns + (diacritics / leading "the " / punctuation stripped), so most rows could never + match. Steps mirror normalize_for_search: strip diacritics (NFD + drop combining + marks), lower/trim, strip a trailing karaoke tag, strip a leading "the ", drop + non-word chars (keeping unicode letters/digits/underscore), collapse whitespace. + """ + return ( + "TRIM(REGEXP_REPLACE(" + "REGEXP_REPLACE(" + "REGEXP_REPLACE(" + "REGEXP_REPLACE(" + "LOWER(TRIM(REGEXP_REPLACE(NORMALIZE(COALESCE(" + col + ", ''), NFD), r'\\p{Mn}', '')))," + r" r'\s*[\(\[]\s*(?:[^)\]]*karaoke[^)\]]*|(?:instrumental|backing track|no vocals?|kj version|with vocals?|demo)[^)\]]*)[\)\]]\s*$', '')," + r" r'^the ', '')," + r" r'[^\p{L}\p{N}_\s]', '')," + r" r'\s+', ' '))" + ) + + def _rebuild_xref() -> dict: - """Rebuild the KN ↔ Divebar cross-reference index using exact + fuzzy matching.""" + """Rebuild the KN ↔ Divebar cross-reference index by exact normalized match. + + Both sides are normalized through the identical `_norm_sql` expression, so a + KN song links to a Divebar file only when their artist AND title agree after + normalization. (The previous "brand_match" branch matched on brand_code + + artist only — with no title constraint it was a Cartesian product that linked + every KN song by an artist/brand to every Divebar file by the same + artist/brand, i.e. wrong-song links. Constraining it by title would make it a + strict subset of the exact match, so it was removed.) + """ client = bigquery.Client(project=GCP_PROJECT_ID) start = time.time() - # Step 1: Exact match on normalized artist + title + kn_artist, kn_title = _norm_sql("kn.Artist"), _norm_sql("kn.Title") + db_artist, db_title = _norm_sql("db.artist"), _norm_sql("db.title") + exact_sql = f""" CREATE OR REPLACE TABLE `{GCP_PROJECT_ID}.{DATASET}.kn_divebar_xref` AS - -- Exact normalized match (high confidence) + -- Exact match on symmetrically-normalized artist + title (high confidence) SELECT DISTINCT kn.Id AS kn_id, db.file_id AS divebar_file_id, @@ -195,40 +230,10 @@ def _rebuild_xref() -> dict: CURRENT_TIMESTAMP() AS matched_at FROM `{GCP_PROJECT_ID}.{DATASET}.karaokenerds_raw` kn JOIN `{GCP_PROJECT_ID}.{DATASET}.divebar_catalog` db - ON LOWER(TRIM(kn.Artist)) = db.artist_normalized - AND LOWER(TRIM(kn.Title)) = db.title_normalized - WHERE db.artist_normalized IS NOT NULL - AND db.artist_normalized != '' - AND db.title_normalized IS NOT NULL - AND db.title_normalized != '' - - UNION ALL - - -- Community brand match: KN community track brand matches Divebar folder brand code - SELECT DISTINCT - kn.Id AS kn_id, - db.file_id AS divebar_file_id, - 'brand_match' AS match_type, - 0.90 AS confidence, - CURRENT_TIMESTAMP() AS matched_at - FROM `{GCP_PROJECT_ID}.{DATASET}.karaokenerds_community` c - JOIN `{GCP_PROJECT_ID}.{DATASET}.karaokenerds_raw` kn - ON LOWER(TRIM(c.Artist)) = LOWER(TRIM(kn.Artist)) - AND LOWER(TRIM(c.Title)) = LOWER(TRIM(kn.Title)) - JOIN `{GCP_PROJECT_ID}.{DATASET}.divebar_catalog` db - ON LOWER(c.Brand) = LOWER(COALESCE(db.brand_code, '')) - AND LOWER(TRIM(c.Artist)) = db.artist_normalized - WHERE c.Brand IS NOT NULL - AND c.Brand != '' - AND db.artist_normalized IS NOT NULL - AND db.artist_normalized != '' - -- Exclude exact matches (already captured above) - AND NOT EXISTS ( - SELECT 1 FROM `{GCP_PROJECT_ID}.{DATASET}.divebar_catalog` db2 - WHERE LOWER(TRIM(kn.Artist)) = db2.artist_normalized - AND LOWER(TRIM(kn.Title)) = db2.title_normalized - AND db2.file_id = db.file_id - ) + ON {kn_artist} = {db_artist} + AND {kn_title} = {db_title} + WHERE {db_artist} != '' + AND {db_title} != '' """ logger.info("Rebuilding cross-reference index...") diff --git a/infrastructure/functions/divebar_lookup/test_main.py b/infrastructure/functions/divebar_lookup/test_main.py index 176d7a19..94b1278f 100644 --- a/infrastructure/functions/divebar_lookup/test_main.py +++ b/infrastructure/functions/divebar_lookup/test_main.py @@ -108,6 +108,33 @@ class TestRefreshTriggers: assert result["failed"][0]["job"] == main.REFRESH_SCHEDULER_JOBS[0] +# --------------------------------------------------------------------------- +# _norm_sql — symmetric normalization for the xref join +# --------------------------------------------------------------------------- + +class TestNormSql: + def test_embeds_column_and_is_deterministic(self): + a = main._norm_sql("kn.Artist") + assert "kn.Artist" in a + assert main._norm_sql("kn.Artist") == a # pure / deterministic + + def test_replicates_normalize_for_search_steps(self): + expr = main._norm_sql("db.title") + # diacritics (NFD + drop combining marks), lower, leading "the", and the + # unicode-aware punctuation strip must all be present. + assert "NORMALIZE(COALESCE(db.title" in expr + assert "NFD" in expr and r"\p{Mn}" in expr + assert "LOWER(" in expr + assert r"'^the '" in expr + assert r"\p{L}" in expr and r"\p{N}" in expr + + def test_both_sides_use_same_expression(self): + # The whole point of the fix: KN and Divebar sides normalize identically. + kn = main._norm_sql("X") + db = main._norm_sql("X") + assert kn == db + + # --------------------------------------------------------------------------- # divebar_lookup dispatch — refresh action # --------------------------------------------------------------------------- diff --git a/infrastructure/functions/divebar_mirror/deploy.sh b/infrastructure/functions/divebar_mirror/deploy.sh index 934b3c43..f2747933 100644 --- a/infrastructure/functions/divebar_mirror/deploy.sh +++ b/infrastructure/functions/divebar_mirror/deploy.sh @@ -1,17 +1,14 @@ #!/bin/bash # Deploy the Divebar Mirror Cloud Function # -# This script: -# 1. Creates a ZIP of the function source -# 2. Uploads it to GCS -# 3. Deploys the function via gcloud +# NOTE: As of 2026-06, the function source is managed by Pulumi +# (modules/divebar_mirror.py zips this directory via pulumi.FileArchive and pins +# the function to the object generation). The normal way to ship a code change is: # -# Prerequisites: -# - gcloud CLI authenticated -# - Pulumi infrastructure already deployed (creates the bucket) +# cd infrastructure && pulumi up # -# Usage: -# ./deploy.sh +# This script remains for ad-hoc manual uploads/testing only; a manual upload +# will be reset to the repo contents on the next `pulumi up`. set -e diff --git a/infrastructure/functions/divebar_mirror/filename_parser.py b/infrastructure/functions/divebar_mirror/filename_parser.py index d9d2fdb7..8b8524b8 100644 --- a/infrastructure/functions/divebar_mirror/filename_parser.py +++ b/infrastructure/functions/divebar_mirror/filename_parser.py @@ -1,67 +1,148 @@ """ Parse karaoke filenames into structured metadata. -Handles the various naming conventions found across Divebar karaoke brands: - Pattern 1: "BRAND-001 - Artist - Title.ext" (disc ID prefix) - Pattern 2: "Artist - Title - (Brand Tag).ext" (brand suffix) - Pattern 3: "Artist - Title.ext" (brand from folder name) +Community karaoke brands use many naming conventions. Rather than a handful of +rigid patterns, this parser is *folder-brand-aware*: the top-level Drive folder +reliably names the brand, so we use it to recognise and strip the brand token +wherever it sits (prefix, suffix, bracket, disc-id) and then split the remainder +into artist / title. Brand stripping is gated on the known folder brand so we +never eat real song text like "(Live)", "(Banda Remix)" or "(G)I-DLE". + +Returned dict keys: artist, title, disc_id, brand_code, brand_name, format. """ import os import re import unicodedata - -# Common karaoke suffixes to strip from titles -_KARAOKE_SUFFIXES = re.compile( - r"\s*[\(\[](karaoke|instrumental|backing track|no vocals?|kj version|" - r"karaoke version|with vocals?|demo)[\)\]]\s*$", +# Generic karaoke/source tags to strip from the end of a title, in () or []. +# Matches any bracket whose content mentions "karaoke" ("(WTF Karaoke)", +# "[DJ Sauly Karaoke]", "(Karaoke HD)") or starts with another generic tag +# ("(instrumental)", "(backing track)"). +_KARAOKE_TAG = re.compile( + r"\s*[\(\[]\s*(?:[^)\]]*karaoke[^)\]]*|" + r"(?:instrumental|backing track|no vocals?|kj version|with vocals?|demo)[^)\]]*)" + r"[\)\]]\s*$", re.IGNORECASE, ) +# Back-compat alias (normalize_for_search historically referenced this name). +_KARAOKE_SUFFIXES = _KARAOKE_TAG -# Brand code pattern at start of filename: "ABC-001" or "ABC001" or "ABC 001" -_DISC_ID_PATTERN = re.compile( - r"^([A-Z][A-Z0-9]{1,10})[\s\-]?(\d{1,5}(?:-\d{1,3})?)\s*-\s*" -) +# disc-id PREFIX token followed by a " - " or plain-space separator. The token +# (e.g. "FATBIRD-163", "IK00-01", "KFNO101", "RABZ-0001", "Lopenash-002") must +# contain both a letter and a digit; acceptance is further gated in code (code +# all-uppercase, or the code matches the folder brand) so alphanumeric artists +# like "2Pac", "311" or "M83" are never eaten. +_DISC_ID_PREFIX = re.compile(r"^([A-Za-z0-9][A-Za-z0-9.\-]*?)(?:\s*-\s+|\s+)(?=\S)") +# disc-id SUFFIX at end: " - SOKC-0306", " ATG0072", " IMBK-0076". Captures the +# whole token (preserving any internal dash) so disc_id stays literal. +_DISC_ID_SUFFIX = re.compile(r"[\s\-]+([A-Z]{2,6}-?\d{2,5}(?:-\d{1,3})?)\s*$") +# Leading "(CODE) " brand prefix (Sandell "(SDK) ..."). +_PAREN_CODE_PREFIX = re.compile(r"^\(\s*([A-Za-z0-9]{1,6})\s*\)\s+") +# A leading pure track-number segment, e.g. "02" or "08 (and a half)". +_LEADING_TRACK_NO = re.compile(r"^\d{1,3}(?:\s*\(.*?\))?$") -# Brand tag in parentheses at end of filename (before extension) -_BRAND_SUFFIX_PATTERN = re.compile(r"\s*-?\s*\(([^)]+)\)\s*$") +# Curated folder -> brand code map. Aligned to real KaraokeNerds community codes +# where confident (so the secondary brand_match xref can also fire). Overrides a +# prefix/suffix-derived code (disc_id keeps the literal). Folders with no +# confident code are omitted (artist/title fix already surfaces them via the +# primary exact xref). Keys are lowercased exact folder names. +FOLDER_CODE = { + "funbox karaoke": "FBK", + "sandell karaoke": "SDK", + "bellysings karaoke": "BELLY", + "dj sauly collection": "DJS", + "playkaraoke": "PLAY", + "reekies karaoke": "REEKIES", + "cereal killer karaoke": "CKK", + "wizzyoke": "WIZZY", + "brilliant trash karaoke": "BTRASH", + "lopenash": "LOPE", + "monster tracks": "MTRAX", + "popeoke": "POPE", + "kaddaok": "KADDA", + "potos power plant productions": "PPPP", + "april's choice karaoke": "APRIL", + "the nerdy singer karaoke": "TNS", + "mobile karaoke unit": "MKU", + "atg karaoke - zipped mp3+g": "ATG", + "it might be karaoke": "IMBK", + "mix tape karaoke": "MIXTAPE", + "fortekaraoke discord": "EBK", + "steve-o karaoke cult": "SOKC", +} + +# Descriptor words dropped when fuzzy-matching a brand token against a folder name. +_BRAND_NOISE = { + "karaoke", "collection", "cdg", "files", "file", "discord", "videos", "video", + "mp4", "mp3", "zipped", "presents", "studio", "made", "the", "of", "and", +} -# Video and audio extensions we care about KARAOKE_EXTENSIONS = { ".mp4", ".mkv", ".avi", ".webm", ".mov", # Video ".mp3", ".cdg", # CDG+MP3 pairs ".zip", # Zipped CDG+MP3 } -# Files to skip during indexing IGNORED_FILES = { ".DS_Store", ".keep", ".gitkeep", "desktop.ini", "Thumbs.db", "kj-nomad.index.json", } +def _strip_diacritics(text: str) -> str: + text = unicodedata.normalize("NFD", text) + return "".join(c for c in text if unicodedata.category(c) != "Mn") + + def normalize_for_search(text: str) -> str: - """Normalize text for fuzzy matching: lowercase, strip diacritics, remove special chars.""" + """Normalize text for fuzzy matching: lowercase, strip diacritics/special chars.""" if not text: return "" - # NFD decomposition then strip combining characters (diacritics) - text = unicodedata.normalize("NFD", text) - text = "".join(c for c in text if unicodedata.category(c) != "Mn") + text = _strip_diacritics(text) text = text.lower().strip() - # Strip common karaoke suffixes - text = _KARAOKE_SUFFIXES.sub("", text) - # Remove "the " prefix for matching + text = _KARAOKE_TAG.sub("", text) if text.startswith("the "): text = text[4:] - # Normalize whitespace and strip non-alphanumeric (keep spaces) text = re.sub(r"[^\w\s]", "", text) text = re.sub(r"\s+", " ", text).strip() return text +def _brand_key(folder: str) -> str: + """Significant-word signature of a folder/brand name for fuzzy comparison.""" + base = re.sub(r"[^\w\s]", " ", _strip_diacritics(folder or "").lower()) + words = [w for w in base.split() if w and w not in _BRAND_NOISE] + return "".join(words) + + +def _brand_acronym(folder: str) -> str: + # Initials of ALL words (brands often acronym the full name, e.g. + # "The Nerdy Singer Karaoke" -> TNSK, "Steve-O Karaoke Cult" -> SOKC). + words = re.findall(r"[A-Za-z0-9]+", _strip_diacritics(folder or "").lower()) + return "".join(w[0] for w in words).upper() + + +def _folder_code(folder: str) -> str | None: + return FOLDER_CODE.get((folder or "").strip().lower()) + + +def _matches_folder_brand(token: str, folder: str) -> bool: + """True if a stripped token looks like this folder's brand (not song text).""" + if not token or not folder: + return False + tk = _brand_key(token) + fk = _brand_key(folder) + if tk and fk and (tk == fk or (len(tk) >= 4 and (tk in fk or fk in tk))): + return True + up = token.strip().strip("()[]").upper() + if up and (up == _brand_acronym(folder) or up == (_folder_code(folder) or "\0")): + return True + return False + + def detect_format(filename: str) -> str: - """Detect the karaoke format from a filename.""" + """Detect karaoke file format from extension.""" ext = os.path.splitext(filename)[1].lower() if ext == ".zip": return "zip" @@ -70,32 +151,97 @@ def detect_format(filename: str) -> str: if ext == ".mp3": return "mp3" if ext in (".mp4", ".mkv", ".avi", ".webm", ".mov"): - return ext[1:] # Strip the dot + return ext[1:] return ext[1:] if ext else "unknown" def should_index_file(filename: str) -> bool: - """Check if a file should be included in the index.""" + """Whether a file should be included in the index.""" if filename in IGNORED_FILES or filename.startswith("."): return False ext = os.path.splitext(filename)[1].lower() return ext in KARAOKE_EXTENSIONS +def _strip_trailing_noise(text: str) -> str: + """Remove a trailing generic karaoke tag and dangling separators.""" + prev = None + while prev != text: + prev = text + text = _KARAOKE_TAG.sub("", text).strip() + text = re.sub(r"[\s\-]+$", "", text).strip() + return text + + +def _disc_code(token: str) -> str: + """Derive the brand code (stem) from a disc-id token. + + "IK00-01" -> "IK00", "FATBIRD-163" -> "FATBIRD", "C11K-0000A" -> "C11K", + "KFNO101" -> "KFNO", "Lopenash-002" -> "Lopenash". + """ + m = re.match(r"^(.*?)-\d[\w]*$", token) + if m: + return m.group(1) + m = re.match(r"^([A-Za-z]+)\d+$", token) + if m: + return m.group(1) + return token + + +def _try_prefix(name: str, folder: str) -> tuple[str, str | None, str | None, bool]: + """Strip a leading brand token. + + Returns (remainder, brand_code, disc_id, disc_prefix_stripped). + """ + # 1. disc-id prefix (token contains a letter and a digit; accepted only when + # the code is all-uppercase or matches the folder brand). + m = _DISC_ID_PREFIX.match(name) + if m: + token = m.group(1) + if re.search(r"[A-Za-z]", token) and re.search(r"\d", token): + code = _disc_code(token) + num_part = token[len(code):] + # "strong" disc-id: code separated from number by a hyphen, or the + # number has >=3 digits. Prevents alphanumeric artists like "UB40", + # "D12", "AC3" (uppercase but not disc-shaped) from being eaten. + strong = "-" in num_part or len(re.findall(r"\d", num_part)) >= 3 + if _matches_folder_brand(code, folder) or ( + code.isupper() and len(code) >= 2 and strong + ): + return name[m.end():].strip(), code, token, True + # 2. "(CODE) " paren prefix + m = _PAREN_CODE_PREFIX.match(name) + if m: + return name[m.end():].strip(), m.group(1).upper(), None, False + # 3. "CODE - " / "Brand - " prefix (no number), gated on folder match + m = re.match(r"^(.{1,30}?)\s+-\s+(?=\S)", name) + if m: + token = m.group(1).strip() + if _matches_folder_brand(token, folder): + code = token.upper() if re.fullmatch(r"[A-Za-z0-9]{1,8}", token) else None + return name[m.end():].strip(), code, None, False + return name, None, None, False + + +def _strip_brand_suffix(core: str, folder: str) -> str: + """Strip a trailing [Brand]/(Brand)/ - Brand token when it matches the folder.""" + # bracket / paren suffix + m = re.search(r"\s*[\(\[]([^)\]]+)[\)\]]\s*$", core) + if m and _matches_folder_brand(m.group(1), folder): + return core[: m.start()].strip() + # " - Brand" dash suffix (only if there is something before it to keep) + if " - " in core: + head, _, last = core.rpartition(" - ") + if head and _matches_folder_brand(last, folder): + return head.strip() + return core + + def parse_filename(filename: str, brand_folder: str = "") -> dict: - """ - Parse a karaoke filename into structured metadata. - - Args: - filename: The file name (not path), e.g. "NOMAD-0001 - Artist - Title.mp4" - brand_folder: The parent brand folder name, used as fallback brand - - Returns: - dict with keys: artist, title, disc_id, brand_code, brand_name, format - """ - # Strip extension - name, ext = os.path.splitext(filename) - file_format = detect_format(filename) + """Parse a karaoke filename into structured metadata (see module docstring).""" + name, _ext = os.path.splitext(filename) + # Normalise spaced en/em dashes to the standard " - " artist/title separator. + name = name.replace(" – ", " - ").replace(" — ", " - ").strip() result = { "artist": None, @@ -103,45 +249,53 @@ def parse_filename(filename: str, brand_folder: str = "") -> dict: "disc_id": None, "brand_code": None, "brand_name": brand_folder or None, - "format": file_format, + "format": detect_format(filename), } - # Try Pattern 1: "BRAND-001 - Artist - Title" - disc_match = _DISC_ID_PATTERN.match(name) - if disc_match: - result["brand_code"] = disc_match.group(1) - result["disc_id"] = f"{disc_match.group(1)}-{disc_match.group(2)}" - remainder = name[disc_match.end():] - parts = [p.strip() for p in remainder.split(" - ")] - if len(parts) >= 2: - result["artist"] = parts[0] - result["title"] = " - ".join(parts[1:]) - elif len(parts) == 1 and parts[0]: - result["title"] = parts[0] - # Strip brand suffix from title if present - if result["title"]: - result["title"] = _BRAND_SUFFIX_PATTERN.sub("", result["title"]) - return result + # 1. leading brand token (disc-id / paren-code / folder-name prefix) + core, brand_code, disc_id, disc_prefix_stripped = _try_prefix(name, brand_folder) + if disc_prefix_stripped: + # Some brands write "CODE-0002 -Artist" (dash hugging the artist); drop the + # leftover leading separator so the artist isn't "-Artist". + core = core.lstrip(" -–").strip() - # Try Pattern 2: "Artist - Title - (Brand Tag)" - brand_suffix_match = _BRAND_SUFFIX_PATTERN.search(name) - if brand_suffix_match: - result["brand_name"] = brand_suffix_match.group(1) - name_without_brand = name[:brand_suffix_match.start()] - parts = [p.strip() for p in name_without_brand.split(" - ")] - if len(parts) >= 2: - result["artist"] = parts[0] - result["title"] = " - ".join(parts[1:]) - elif len(parts) == 1 and parts[0]: - result["title"] = parts[0] - return result + # 2. trailing generic karaoke tag (so a disc-id suffix is reachable) + core = _strip_trailing_noise(core) - # Pattern 3: "Artist - Title" (brand from folder name) - parts = [p.strip() for p in name.split(" - ")] + # 3. disc-id SUFFIX (strong signal; unconditional) + if disc_id is None: + m = _DISC_ID_SUFFIX.search(core) + if m: + token = m.group(1) + core = core[: m.start()].strip() + core = _strip_trailing_noise(core) + brand_code = brand_code or _disc_code(token) + disc_id = token + + # 4. trailing brand token in brackets/parens/dash, gated on folder + core = _strip_brand_suffix(core, brand_folder) + core = _strip_trailing_noise(core) + + # 5. split remainder into artist / title. Drop a leading track-number segment + # only when it followed a disc-id prefix (e.g. "MIX001 - 02 - Artist - T"), + # so numeric artists like "311" are never mistaken for a track number. + parts = [p.strip() for p in core.split(" - ") if p.strip()] + if ( + disc_prefix_stripped + and len(parts) >= 3 + and _LEADING_TRACK_NO.match(parts[0]) + ): + parts = parts[1:] if len(parts) >= 2: result["artist"] = parts[0] result["title"] = " - ".join(parts[1:]) - elif len(parts) == 1 and parts[0]: + elif len(parts) == 1: result["title"] = parts[0] + if result["title"]: + result["title"] = _KARAOKE_TAG.sub("", result["title"]).strip() + + # 6. brand_code: curated folder map overrides; else captured code + result["disc_id"] = disc_id + result["brand_code"] = _folder_code(brand_folder) or brand_code return result diff --git a/infrastructure/modules/divebar_lookup.py b/infrastructure/modules/divebar_lookup.py index a3449320..11090fea 100644 --- a/infrastructure/modules/divebar_lookup.py +++ b/infrastructure/modules/divebar_lookup.py @@ -8,6 +8,8 @@ Creates: - Cloud Scheduler to rebuild xref daily (after mirror + KN sync complete) """ +from pathlib import Path + import pulumi import pulumi_gcp as gcp from pulumi_gcp import ( @@ -72,6 +74,17 @@ def create_divebar_lookup_resources(all_secrets: dict) -> dict: ) resources["source_bucket"] = source_bucket + # Content-hashed source archive so `pulumi up` redeploys the function on code + # changes (a static object name never triggered a redeploy). + source_dir = Path(__file__).parent.parent / "functions" / "divebar_lookup" + source_archive = storage.BucketObject( + "divebar-lookup-source", + bucket=source_bucket.name, + name="divebar-lookup-source.zip", + source=pulumi.FileArchive(str(source_dir)), + ) + resources["source_archive"] = source_archive + # ==================== BigQuery: Cross-Reference Table ==================== xref_table = bigquery.Table( @@ -103,7 +116,8 @@ def create_divebar_lookup_resources(all_secrets: dict) -> dict: source=cloudfunctionsv2.FunctionBuildConfigSourceArgs( storage_source=cloudfunctionsv2.FunctionBuildConfigSourceStorageSourceArgs( bucket=source_bucket.name, - object="divebar-lookup-source.zip", + object=source_archive.name, + generation=source_archive.generation, ), ), ), diff --git a/infrastructure/modules/divebar_mirror.py b/infrastructure/modules/divebar_mirror.py index 6da94f3f..a2905bec 100644 --- a/infrastructure/modules/divebar_mirror.py +++ b/infrastructure/modules/divebar_mirror.py @@ -10,6 +10,8 @@ Creates: - BigQuery table for the Divebar catalog index """ +from pathlib import Path + import pulumi import pulumi_gcp as gcp from pulumi_gcp import ( @@ -99,6 +101,18 @@ def create_divebar_mirror_resources(all_secrets: dict) -> dict: ) resources["source_bucket"] = source_bucket + # Zip the function source and upload it as a content-hashed object so that + # `pulumi up` redeploys the function whenever the code changes (the previous + # static object name never triggered a redeploy on code edits). + source_dir = Path(__file__).parent.parent / "functions" / "divebar_mirror" + source_archive = storage.BucketObject( + "divebar-mirror-source", + bucket=source_bucket.name, + name="divebar-mirror-source.zip", + source=pulumi.FileArchive(str(source_dir)), + ) + resources["source_archive"] = source_archive + # ==================== BigQuery Table ==================== # The table is in the karaoke_decide dataset (managed by karaoke-decide Pulumi). @@ -143,7 +157,11 @@ def create_divebar_mirror_resources(all_secrets: dict) -> dict: source=cloudfunctionsv2.FunctionBuildConfigSourceArgs( storage_source=cloudfunctionsv2.FunctionBuildConfigSourceStorageSourceArgs( bucket=source_bucket.name, - object="divebar-mirror-source.zip", + object=source_archive.name, + # Pin to the object generation so replacing the source bundle + # (a code change) registers as a diff on the Function and + # forces a redeploy, instead of silently serving the old copy. + generation=source_archive.generation, ), ), ), diff --git a/pyproject.toml b/pyproject.toml index 8baf0e66..24c2449d 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [tool.poetry] name = "karaoke-gen" -version = "0.188.3" +version = "0.188.4" description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens." authors = ["Andrew Beveridge "] license = "MIT" diff --git a/tests/unit/test_divebar_filename_parser.py b/tests/unit/test_divebar_filename_parser.py new file mode 100644 index 00000000..e638deea --- /dev/null +++ b/tests/unit/test_divebar_filename_parser.py @@ -0,0 +1,336 @@ +"""Unit tests for the Divebar filename parser. + +The parser lives in the Cloud Function package (not on the default path), so we +add its directory to sys.path and import it directly. It has stdlib-only deps. + +Fixtures are real filenames observed in the live `divebar_catalog` index, grouped +by the naming convention each community brand uses. The goal of the rewrite is +folder-brand-aware parsing: use the (reliable) top-level folder brand to strip the +brand token wherever it sits, then split the remainder into artist/title. +""" + +import pathlib +import sys + +import pytest + +FUNC_DIR = ( + pathlib.Path(__file__).resolve().parents[2] + / "infrastructure" + / "functions" + / "divebar_mirror" +) +sys.path.insert(0, str(FUNC_DIR)) + +import filename_parser as fp # noqa: E402 + + +def parse(filename, folder): + return fp.parse_filename(filename, brand_folder=folder) + + +# -------------------------------------------------------------------------- +# disc-id PREFIX (works today — must be preserved) +# -------------------------------------------------------------------------- + +@pytest.mark.parametrize( + "filename,folder,artist,title,brand_code,disc_id", + [ + ("FATBIRD-163 - Incubus - I Miss You.zip", "Fatbird Karaoke", + "Incubus", "I Miss You", "FATBIRD", "FATBIRD-163"), + ("NOMAD-0003 - Cher - Apples Don't Fall Far From The Tree.mp4", "Nomad Karaoke", + "Cher", "Apples Don't Fall Far From The Tree", "NOMAD", "NOMAD-0003"), + ("IK00-01 - Costello, Elvis - Goon Squad.zip", "Imperfekt Karaoke", + "Costello, Elvis", "Goon Squad", "IK00", "IK00-01"), + ("GGZ001-02 - Partman Parthorse - Magik (ggnzla REMIX).cdg", "GGZ-ggnzla", + "Partman Parthorse", "Magik (ggnzla REMIX)", "GGZ001", "GGZ001-02"), + ("KFNO101 - The Kills - List of Demands (Reparations).mp4", "Karaoke For No One", + "The Kills", "List of Demands (Reparations)", "KFNO", "KFNO101"), + ], +) +def test_disc_id_prefix(filename, folder, artist, title, brand_code, disc_id): + r = parse(filename, folder) + assert r["artist"] == artist + assert r["title"] == title + assert r["brand_code"] == brand_code + assert r["disc_id"] == disc_id + + +# -------------------------------------------------------------------------- +# disc-id PREFIX, space-separated / mixed-case / digit-leading +# -------------------------------------------------------------------------- + +@pytest.mark.parametrize( + "filename,folder,artist,title,disc_id", + [ + # Potos: "CODE-#### Artist - Title" (space, not " - ", after disc-id) + ("RABZ-0001 The Seatbelts & Steve Conte - Call Me, Call Me.zip", + "Potos Power Plant Productions", + "The Seatbelts & Steve Conte", "Call Me, Call Me", "RABZ-0001"), + # Lopenash: mixed-case folder-name prefix + ("Lopenash-002 - Dexter's Laboratory - Breathe in The Sunshine.mp4", "Lopenash", + "Dexter's Laboratory", "Breathe in The Sunshine", "Lopenash-002"), + # 7uLpA: digit-leading mixed-case token + ("7uLpA-0004 - The Beauty of Gemina - Wonders.mp4", "7uLpA_Made", + "The Beauty of Gemina", "Wonders", "7uLpA-0004"), + ], +) +def test_disc_id_prefix_variants(filename, folder, artist, title, disc_id): + r = parse(filename, folder) + assert r["artist"] == artist + assert r["title"] == title + assert r["disc_id"] == disc_id + + +def test_disc_id_prefix_single_field(): + # Lopenash medley with no artist/title split -> title only, artist None + r = parse("Lopenash-001 - Super Energy Apocalypse.zip", "Lopenash") + assert r["artist"] is None + assert r["title"] == "Super Energy Apocalypse" + assert r["disc_id"] == "Lopenash-001" + + +def test_disc_id_prefix_drops_leading_track_number(): + # Mix Tape: "CODE-### - NN - Artist - Title" -> drop the NN track number + r = parse("MIX001 - 02 - Psychostick - Mega Man.mp4", "Mix Tape Karaoke") + assert r["artist"] == "Psychostick" + assert r["title"] == "Mega Man" + + +# -------------------------------------------------------------------------- +# disc-id SUFFIX (unconditional strip) +# -------------------------------------------------------------------------- + +@pytest.mark.parametrize( + "filename,folder,artist,title,brand_code,disc_id", + [ + ("3Blue1Brown - How They Fool Ya - IMBK-0076.mp4", "It Might Be Karaoke", + "3Blue1Brown", "How They Fool Ya", "IMBK", "IMBK-0076"), + ("54-40 - Nice to Luv you - SOKC-0036 (KARAOKE).zip", "Steve-O Karaoke Cult", + "54-40", "Nice to Luv you", "SOKC", "SOKC-0036"), + ("Against Me - Wagon Wheel - SOKC-0306 - (KARAOKE).zip", "Steve-O Karaoke Cult", + "Against Me", "Wagon Wheel", "SOKC", "SOKC-0306"), + # disc-id suffix with no dash before the code + ("Age Of Electric - Remote Control SOKC-0042 (KARAOKE).zip", "Steve-O Karaoke Cult", + "Age Of Electric", "Remote Control", "SOKC", "SOKC-0042"), + # ATG: code with no dash inside (ATG0014) + ("Angelcorpse - Wolflust - ATG0014.zip", "ATG Karaoke - Zipped MP3+G", + "Angelcorpse", "Wolflust", "ATG", "ATG0014"), + ("1349 - Singer Of Strange Songs - ATG0072.zip", "ATG Karaoke - Zipped MP3+G", + "1349", "Singer Of Strange Songs", "ATG", "ATG0072"), + ], +) +def test_disc_id_suffix(filename, folder, artist, title, brand_code, disc_id): + r = parse(filename, folder) + assert r["artist"] == artist + assert r["title"] == title + assert r["brand_code"] == brand_code + assert r["disc_id"] == disc_id + + +# -------------------------------------------------------------------------- +# bracket / paren brand SUFFIX (gated on folder match) +# -------------------------------------------------------------------------- + +@pytest.mark.parametrize( + "filename,folder,artist,title", + [ + ("3rd Bass - Green Eggs And Swine [DJ Sauly Karaoke].zip", "DJ Sauly Collection", + "3rd Bass", "Green Eggs And Swine"), + # keep the legit (Banda Remix), strip only the [DJ Sauly Karaoke] tag + ("50 Cent - In Da Club (Banda Remix) [DJ Sauly Karaoke].zip", "DJ Sauly Collection", + "50 Cent", "In Da Club (Banda Remix)"), + ("10 Years - Fix Me (BellySings).mp4", "BellySings Karaoke", + "10 Years", "Fix Me"), + ("Bad Religion - Fuck You (popeoke).zip", "Popeoke", + "Bad Religion", "Fuck You"), + ("B.E.R. - The Night Begins to Shine [popeoke].zip", "Popeoke", + "B.E.R.", "The Night Begins to Shine"), + ], +) +def test_brand_suffix_bracket_or_paren(filename, folder, artist, title): + r = parse(filename, folder) + assert r["artist"] == artist + assert r["title"] == title + + +# -------------------------------------------------------------------------- +# dash brand SUFFIX (gated on folder match / per-folder token) +# -------------------------------------------------------------------------- + +@pytest.mark.parametrize( + "filename,folder,artist,title", + [ + ("100 Gecs - Last Train To Awesometown - Rock Solid Karaoke.zip", + "Rock Solid Karaoke CDG files", "100 Gecs", "Last Train To Awesometown"), + ("311 - Amber - Rock Solid Karaoke.zip", + "Rock Solid Karaoke CDG files", "311", "Amber"), + ("Akon - Right Now (Na Na Na) - EBK.avi", "ForteKaraoke Discord", + "Akon", "Right Now (Na Na Na)"), + ], +) +def test_brand_suffix_dash(filename, folder, artist, title): + r = parse(filename, folder) + assert r["artist"] == artist + assert r["title"] == title + + +# -------------------------------------------------------------------------- +# code / folder-name PREFIX (no number) +# -------------------------------------------------------------------------- + +@pytest.mark.parametrize( + "filename,folder,artist,title,brand_code", + [ + ("(SDK) (G)I-DLE - Queencard.cdg", "Sandell Karaoke", + "(G)I-DLE", "Queencard", "SDK"), + ("(SDK) 21 Pilots - House Of Gold.cdg", "Sandell Karaoke", + "21 Pilots", "House Of Gold", "SDK"), + ("CKK - Buoys, The - Timothy.zip", "Cereal Killer Karaoke", + "Buoys, The", "Timothy", "CKK"), + ("CKK - A Day To Remember - If It Means A Lot To You.zip", "Cereal Killer Karaoke", + "A Day To Remember", "If It Means A Lot To You", "CKK"), + ("TNSK - Davy Jones - Girl.zip", "The Nerdy Singer Karaoke", + "Davy Jones", "Girl", "TNS"), + ("Crossfire - Eurythmics - Missionary Man.mp4", "Crossfire Karaoke", + "Eurythmics", "Missionary Man", None), + ], +) +def test_code_or_name_prefix(filename, folder, artist, title, brand_code): + r = parse(filename, folder) + assert r["artist"] == artist + assert r["title"] == title + assert r["brand_code"] == brand_code + + +# -------------------------------------------------------------------------- +# generic karaoke tag strip (display title, both () and []) +# -------------------------------------------------------------------------- + +@pytest.mark.parametrize( + "filename,folder,artist,title", + [ + ("Bill Murray - Star Wars [Karaoke].mp4", "Mobile Karaoke Unit", + "Bill Murray", "Star Wars"), + ("3 Doors Down - Down Poison (Karaoke HD).zip", "Funbox Karaoke", + "3 Doors Down", "Down Poison"), + ("Islands - Pumpkin (Karaoke Version).mp4", "Matt Joy Karaoke", + "Islands", "Pumpkin"), + ], +) +def test_generic_karaoke_tag_stripped_from_title(filename, folder, artist, title): + r = parse(filename, folder) + assert r["artist"] == artist + assert r["title"] == title + + +# -------------------------------------------------------------------------- +# clean "Artist - Title" — only the curated brand_code map applies +# -------------------------------------------------------------------------- + +@pytest.mark.parametrize( + "filename,folder,artist,title,brand_code", + [ + ("A Tribe Called Quest - Lyrics To Go.avi", "WIZZYOKE", + "A Tribe Called Quest", "Lyrics To Go", "WIZZY"), + ("100 gecs - Hollywood Baby.cdg", "Funbox Karaoke", + "100 gecs", "Hollywood Baby", "FBK"), + ("Alice in Chains - Angry Chair.mp4", "Brilliant Trash Karaoke", + "Alice in Chains", "Angry Chair", "BTRASH"), + ("16 Horsepower - American Wheeze.cdg", "Monster Tracks", + "16 Horsepower", "American Wheeze", "MTRAX"), + ], +) +def test_clean_artist_title_with_curated_code(filename, folder, artist, title, brand_code): + r = parse(filename, folder) + assert r["artist"] == artist + assert r["title"] == title + assert r["brand_code"] == brand_code + + +# -------------------------------------------------------------------------- +# GUARDS — must NOT over-strip real song text +# -------------------------------------------------------------------------- + +def test_guard_keeps_non_brand_bracket_in_title(): + # [Nuxx] and (RVZN Edit) are NOT the folder brand -> keep them + r = parse("RVZ-016 - UNDERWORLD - Born Slippy [Nuxx] (RVZN Edit).mp4", "R3V1Z10N Karaoke") + assert r["artist"] == "UNDERWORLD" + assert r["title"] == "Born Slippy [Nuxx] (RVZN Edit)" + assert r["brand_code"] == "RVZ" + + +def test_guard_keeps_paren_artist(): + r = parse("(SDK) (G)I-DLE - Queencard.cdg", "Sandell Karaoke") + assert r["artist"] == "(G)I-DLE" # only the (SDK) brand prefix removed + + +def test_guard_keeps_live_qualifier_in_title(): + # A trailing (Live) that is not the brand must survive (folder doesn't match) + r = parse("311 - All Re-Mixed Up (live) - Rock Solid Karaoke.zip", + "Rock Solid Karaoke CDG files") + assert r["artist"] == "311" + assert r["title"] == "All Re-Mixed Up (live)" + + +def test_guard_numeric_artist_preserved(): + r = parse("311 - Amber - Rock Solid Karaoke.zip", "Rock Solid Karaoke CDG files") + assert r["artist"] == "311" + + +def test_guard_alphanumeric_artist_not_eaten_as_disc_id(): + # "UB40", "D12" are uppercase+digits but NOT disc-shaped (no hyphen, <3 digit + # number) -> must stay the artist, not be stripped as a disc-id prefix. + r = parse("UB40 - Kingston Town (BellySings).mp4", "BellySings Karaoke") + assert r["artist"] == "UB40" + assert r["title"] == "Kingston Town" + r2 = parse("D12 - My Band.mp4", "Funbox Karaoke") + assert r2["artist"] == "D12" + assert r2["title"] == "My Band" + + +def test_en_dash_separator(): + r = parse("Iron Maiden – The Writing On The Wall.mp4", "ATG Karaoke - Zipped MP3+G") + assert r["artist"] == "Iron Maiden" + assert r["title"] == "The Writing On The Wall" + + +def test_strips_other_brand_karaoke_tag_in_title(): + # Funbox re-hosts WTF Karaoke files tagged "- (WTF Karaoke)"; the tag mentions + # karaoke so it is stripped even though it isn't the folder brand. + r = parse("Alice in Chains - Heaven Beside You - (WTF Karaoke).zip", "Funbox Karaoke") + assert r["artist"] == "Alice in Chains" + assert r["title"] == "Heaven Beside You" + + +def test_leading_dash_hugging_artist_cleaned(): + r = parse("7uLpA-0013 -Zynic - After Effects.mp4", "7uLpA_Made") + assert r["artist"] == "Zynic" + assert r["title"] == "After Effects" + + +# -------------------------------------------------------------------------- +# normalized fields flow from parsed artist/title (xref keys) +# -------------------------------------------------------------------------- + +def test_normalized_fields_use_cleaned_title(): + r = parse("3rd Bass - Green Eggs And Swine [DJ Sauly Karaoke].zip", "DJ Sauly Collection") + assert fp.normalize_for_search(r["artist"]) == "3rd bass" + assert fp.normalize_for_search(r["title"]) == "green eggs and swine" + + +# -------------------------------------------------------------------------- +# existing helpers still behave +# -------------------------------------------------------------------------- + +def test_should_index_file(): + assert fp.should_index_file("x.mp4") is True + assert fp.should_index_file("x.zip") is True + assert fp.should_index_file(".DS_Store") is False + assert fp.should_index_file("notes.txt") is False + + +def test_detect_format(): + assert fp.detect_format("x.zip") == "zip" + assert fp.detect_format("x.MP4") == "mp4" + assert fp.detect_format("x.cdg") == "cdg"