mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
fix(divebar): folder-brand-aware index parser + symmetric xref normalization (#857)
## Summary - Rewrites the Divebar Drive→BigQuery **filename parser** to be folder-brand-aware, fixing mis-split artists and brand-polluted titles that prevented mirror files from surfacing in kjbox. - Fixes a **normalization asymmetry** in the KN↔Divebar xref join (KN side was only `LOWER(TRIM())` vs the db's fully-normalized columns) and removes a buggy `brand_match` branch that produced wrong-song links. - Switches both divebar Cloud Function modules to `pulumi.FileArchive` so `pulumi up` actually redeploys changed code. ## Changes - `divebar_mirror/filename_parser.py` — folder-brand-aware rewrite: strip the brand token wherever it sits (disc-id prefix/suffix, `(CODE)`/`CODE -` prefix, `[Brand]`/`(Brand)`/` - Brand` suffix, generic karaoke tags), each strip gated on a folder-brand match so real song text (`(Live)`, `(G)I-DLE`, bands like `UB40`/`311`/`ACDC`) survives. Curated folder→KN-code map fills `brand_code`. En-dash separator + leading-dash cleanup. - `divebar_lookup/main.py` — new `_norm_sql(col)` routes **both** join sides through one identical BigQuery expression replicating `normalize_for_search`; `brand_match` branch removed (it matched brand+artist with no title → Cartesian wrong-song links; 107,485/107,485 pairs were title mismatches in live data). - `modules/divebar_mirror.py` + `modules/divebar_lookup.py` — `FileArchive` + `BucketObject` with `generation` pinning (matches `runner_manager`), so code changes redeploy. - New `tests/unit/test_divebar_filename_parser.py` (48 cases) + 3 `_norm_sql` guard tests; version bump 0.188.3→0.188.4. ## Impact (validated read-only over all 50,046 live rows) - null `brand_code`: 18,437 (37%) → **1,619 (3%)** - Divebar files cross-referenced to a KN song: production **23,318** → **33,449** on current data (+10,131 from the symmetry fix alone), lifting toward ~38k after the parser re-index. ## Testing - [x] 64 divebar unit tests pass locally (parser 48, index_builder 5, divebar_lookup 11) - [x] Offline validation harness over all 50k live rows (read-only BigQuery); `_norm_sql` confirmed to reproduce `normalize_for_search` exactly - [ ] CI full suite ## Deploy / re-index (post-merge) `pulumi up` redeploys both functions; then trigger scheduler `divebar-mirror-daily` (re-parse → MERGE, preserves `gcs_path`) **then** `divebar-xref-rebuild-daily`. Order matters. (`kn_data_sync` still uses a static source object — future-edit redeploy trap, out of scope here.) ## Review - [x] CodeRabbit CLI review completed locally (1 finding → fixed: brand_match removal; cycle 2 clean) @coderabbitai ignore --- 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
ef65adedc7
commit
eb711b32a8
9 files changed
+928
-122
No files matched your search
@@ -0,0 +1,255 @@
|
||||
# Divebar Index Filename-Parsing Fix — Plan
|
||||
|
||||
_Worktree: `karaoke-gen-divebar-index-parsing` · branch `feat/sess-20260628-0511-divebar-index-parsing`_
|
||||
_Picks up: HANDOFF-divebar-mirror-parsing.md (kjbox side shipped v0.41.1; this is the upstream index fix)_
|
||||
|
||||
## Problem (refined by live data, not just the handoff examples)
|
||||
|
||||
The Divebar Drive→BigQuery indexer (`infrastructure/functions/divebar_mirror/filename_parser.py`,
|
||||
Cloud Function `divebar-mirror`) mis-parses community-brand filenames. kjbox cross-references
|
||||
mirror files on `(normalized artist, normalized title)` + brand, so bad parses → mirror files
|
||||
never surface in the rotation selector even though the file is in GCS.
|
||||
|
||||
### Measured state (BigQuery `karaoke_decide.divebar_catalog`, 50,046 rows)
|
||||
|
||||
| Signal | Count | Meaning |
|
||||
|---|---|---|
|
||||
| null `artist` | 164 (0.3%) | rare |
|
||||
| null `title` | 0 | — |
|
||||
| **null `brand_code`** | **18,437 (37%)** | biggest visible gap |
|
||||
| `artist` is 2–6 all-caps/digits | 1,908 | brand-code-as-artist mis-split (e.g. `CKK`, `EBK`) |
|
||||
| `artist` starts with `(` | 3,051 | `(SDK) Artist` prefix (Sandell) |
|
||||
| `title` contains ` - ` | 5,947 | unstripped ` - Brand` suffix / disc-id-in-title |
|
||||
| `title` ends in `[..]`/`(..)` | 2,583 | `[DJ Sauly Karaoke]`, `[Karaoke]`, etc. |
|
||||
|
||||
Current xref coverage: 90,719 matches, 17,261 KN songs, **23,318 / 50,046 divebar files matched (47%)**.
|
||||
|
||||
> Reframe vs handoff: `artist`/`title` are rarely *null*, but **mis-splits + polluted titles**
|
||||
> are common and silently defeat the EXACT xref (`LOWER(TRIM(kn.Title)) = db.title_normalized`).
|
||||
> Every polluted title misses the join. That + missing `brand_code` is the real lever.
|
||||
|
||||
### Root cause: parser has only 3 rigid patterns and never uses the folder brand to clean
|
||||
|
||||
`parse_filename(name, brand_folder)`:
|
||||
1. `^CODE-### - Artist - Title` (disc-id prefix) — works.
|
||||
2. `... (BrandTag)$` — strips ANY trailing paren (so it also eats `(KARAOKE)`, `(Live)`,
|
||||
`(Banda Remix)` → clobbers `brand_name`), and never sets `brand_code`.
|
||||
3. `Artist - Title` fallback split on ` - `.
|
||||
|
||||
It ignores `[...]` brackets, ` - Brand` dash-suffixes, disc-id *suffixes*, `(CODE) ` prefixes,
|
||||
and `CODE - ` (no-number) prefixes. The folder name (reliable brand) is passed in but only used as
|
||||
a fallback `brand_name` — **never to strip the brand token before splitting artist/title**.
|
||||
`_KARAOKE_SUFFIXES` is applied only to the normalized fields, never the display `title`.
|
||||
|
||||
Note: `build_rows` writes `brand` = folder (always, reliable) and only consumes `brand_code` from
|
||||
the parser (not `brand_name`) — so Pattern-2's `brand_name` clobbering is harmless. The fields that
|
||||
matter from the parser are **artist, title, brand_code, disc_id**.
|
||||
|
||||
## Real-world conventions (gold fixtures, observed live)
|
||||
|
||||
| Convention | Example filename | Brand (folder) | Correct artist / title |
|
||||
|---|---|---|---|
|
||||
| disc-id prefix (OK today) | `FATBIRD-163 - Incubus - I Miss You.zip` | Fatbird | Incubus / I Miss You |
|
||||
| code prefix, no number | `CKK - Buoys, The - Timothy.zip` | Cereal Killer Karaoke | Buoys, The / Timothy |
|
||||
| `(CODE)` paren prefix | `(SDK) (G)I-DLE - Queencard.cdg` | Sandell Karaoke | (G)I-DLE / Queencard |
|
||||
| `(Brand)` paren suffix (OK today) | `10 Years - Fix Me (BellySings).mp4` | BellySings | 10 Years / Fix Me |
|
||||
| `[Brand]` bracket suffix | `3rd Bass - Green Eggs And Swine [DJ Sauly Karaoke].zip` | DJ Sauly Collection | 3rd Bass / Green Eggs And Swine |
|
||||
| ` - Brand` dash suffix | `100 Gecs - Last Train To Awesometown - Rock Solid Karaoke.zip` | Rock Solid Karaoke | 100 Gecs / Last Train To Awesometown |
|
||||
| ` - CODE` dash suffix | `Akon - Right Now (Na Na Na) - EBK.avi` | ForteKaraoke Discord | Akon / Right Now (Na Na Na) |
|
||||
| disc-id *suffix* | `Against Me - Wagon Wheel - SOKC-0306 - (KARAOKE).zip` | Steve-O Karaoke Cult | Against Me / Wagon Wheel |
|
||||
| disc-id suffix no dash | `Angelcorpse - Wolflust - ATG0014.zip` | ATG Karaoke | Angelcorpse / Wolflust |
|
||||
| `[Karaoke]` generic tag | `Bill Murray - Star Wars [Karaoke].mp4` | Mobile Karaoke Unit | Bill Murray / Star Wars |
|
||||
| mixed-case brand prefix | `Lopenash-002 - Dexter's Laboratory - Breathe in The Sunshine.mp4` | Lopenash | Dexter's Laboratory / Breathe in The Sunshine |
|
||||
| clean (only brand_code missing) | `A Tribe Called Quest - Lyrics To Go.avi` | WIZZYOKE | A Tribe Called Quest / Lyrics To Go |
|
||||
|
||||
**Guard cases (must NOT over-strip):** `(G)I-DLE`, `R.E.M.`, artist `311`; titles `(Live)`,
|
||||
`(Banda Remix)`, `(Single Version 2 WBV)`, `(99X Live)`, `[Acoustic]` must survive.
|
||||
|
||||
## Design — folder-brand-aware parser
|
||||
|
||||
Core principle (per handoff): **the top-level folder reliably names the brand; use it to recognise
|
||||
and strip the brand token wherever it sits, then split the remainder into artist/title.**
|
||||
|
||||
`parse_filename(name, brand_folder)` rewrite, in order:
|
||||
|
||||
1. **Strip generic karaoke tags** from the display name: trailing `(karaoke|instrumental|backing
|
||||
track|no vocals|kj version|with vocals|demo)` in either `()` or `[]` (reuse `_KARAOKE_SUFFIXES`,
|
||||
now applied to display too).
|
||||
2. **Build brand matchers from `brand_folder`**: normalized full name, and name with trailing
|
||||
descriptors removed (`karaoke`, `collection`, `cdg files`, `discord`, `videos`, `mp4`,
|
||||
`zipped mp3+g`, etc.); plus an acronym of significant words (Cereal Killer Karaoke→CKK,
|
||||
Steve-O Karaoke Cult→SOKC, ATG Karaoke→ATG).
|
||||
3. **Strip the brand token from whichever position it occupies — each gated so we only remove a
|
||||
*known* brand, never song text:**
|
||||
- disc-id **prefix** `^CODE[-\s]?### - ` → `brand_code`, `disc_id`. (existing)
|
||||
- disc-id **suffix** ` - CODE-### ` / ` CODE#### ` at end, gated on `CODE` ≈ folder acronym/known → `brand_code`,`disc_id`.
|
||||
- `[Brand]`/`(Brand)` **bracket/paren suffix** where content ≈ folder brand → strip; `brand_code` from folder map.
|
||||
- ` - Brand` **dash suffix** where the last ` - ` segment ≈ folder brand → strip.
|
||||
- `(CODE) ` **paren prefix** (≤6 chars, alnum) with content after → strip; `brand_code`=CODE.
|
||||
- `CODE - ` / `Folder-### - ` **prefix** where token ≈ folder acronym/name → strip; `brand_code`.
|
||||
4. **Split remainder** on ` - ` → artist = parts[0], title = join(parts[1:]); strip any leftover
|
||||
generic karaoke tag from title.
|
||||
5. **brand_code resolution order:** captured-during-strip → small curated **folder→code map** →
|
||||
null. `brand` always = folder (unchanged in `build_rows`).
|
||||
|
||||
### Brand registry decision (diverges from handoff)
|
||||
|
||||
Handoff suggested importing kjbox `version_priority.py` `COMMUNITY_BRANDS`. **Rejected** because:
|
||||
(a) cross-repo import isn't viable inside a GCP Cloud Function, (b) that registry is **incomplete**
|
||||
vs the 63 real Divebar folder brands (no Cereal Killer/DJ Sauly/Rock Solid/Sandell), and (c) it
|
||||
**conflicts** — registry maps `SDK`→SNDL but Sandell's folder uses `(SDK)`. Instead: derive
|
||||
`brand_code` from the in-filename token / folder, and keep a **small curated `folder→KN-code` map**
|
||||
seeded from the actual KN community codes that intersect Divebar folders (FBK, BELLY, DJS, C11K,
|
||||
PLAY, REEKIES, HALJAM, NOMAD, FAKEY, CC, …). kjbox's read-time `canonical_brand_for_match` already
|
||||
folds codes/names, so the index doesn't need canonical output — just resolvable.
|
||||
|
||||
## Refined design from full 63-brand data (read-only export)
|
||||
|
||||
Exported all 50,046 rows + per-brand samples + KN community codes to scratchpad. Key refinement:
|
||||
**artist is wrong-but-rarely-null; the measurable gaps are polluted titles + null brand_code.**
|
||||
Brands needing a fix fall into these convention buckets (✅ already-correct buckets only need the
|
||||
brand_code map):
|
||||
|
||||
| Bucket | Brands (examples) | Fix |
|
||||
|---|---|---|
|
||||
| disc-id PREFIX `CODE-### - A - T` | Nomad, Imperfekt, Play, Gnome, Fakey, C11K, Reekies, Fatbird, … | works today — **preserve** |
|
||||
| disc-id PREFIX space-sep `CODE-#### Artist - T` | Potos (`RABZ-0001 The Seatbelts - …`) | allow space (not just ` - `) after disc-id |
|
||||
| disc-id PREFIX mixed-case/digit-lead `Tok-### - …` | Lopenash (`Lopenash-002 -`), 7uLpA (`7uLpA-0003 -`) | match when prefix token ≈ folder name |
|
||||
| leading track-no after disc-id `CODE-### - NN - A - T` | Mix Tape (`MIX001 - 02 - …`) | drop leading pure-numeric segment |
|
||||
| disc-id SUFFIX `A - T - CODE-####` | It Might Be (IMBK), Steve-O (SOKC), ATG (ATG####) | **strip trailing disc-id (unconditional), set brand_code/disc_id** |
|
||||
| `(CODE)` paren PREFIX | Sandell (`(SDK) A - T`) | strip leading `(short-alnum) ` |
|
||||
| `CODE - ` PREFIX no number | Cereal Killer (`CKK - A - T`), Nerdy (`TNSK -`), Crossfire (`Crossfire - A - T`) | strip when token ≈ folder acronym/name |
|
||||
| `[Brand]` bracket SUFFIX | DJ Sauly (`… [DJ Sauly Karaoke]`) | strip bracket suffix gated on folder-name match |
|
||||
| `(Brand)` paren SUFFIX | BellySings (`… (BellySings)`), Popeoke (`(popeoke)`/`[popeoke]`) | already strips paren; add bracket + folder-gate |
|
||||
| ` - Brand` dash SUFFIX | Rock Solid (`… - Rock Solid Karaoke`), Forte (`… - EBK`) | strip last ` - ` segment gated on folder-name match (Forte EBK via per-folder token) |
|
||||
| generic karaoke tag `[Karaoke]`/`(Karaoke HD)` | Mobile, Funbox, Matt Joy, Steve-O | strip generic tag from **display** title (currently only normalized), tolerate trailing words |
|
||||
| clean `Artist - Title`, only brand_code missing | Funbox, WIZZYOKE, Brilliant Trash, Monster Tracks, KaddaOK, April's Choice | **curated folder→code map only** |
|
||||
|
||||
### Parser algorithm (folder-brand-aware), ordered
|
||||
|
||||
1. strip extension; `format = detect_format`.
|
||||
2. strip generic karaoke tags from display name (both `()`/`[]`, token optionally followed by words).
|
||||
3. brand-token strip, first match wins, each sets `brand_code`/`disc_id` and removes the token:
|
||||
(a) disc-id PREFIX `^CODE[-\s]?NUM` followed by ` - ` **or** whitespace; CODE all-caps **or** ≈ folder token (covers Lopenash/7uLpA/Potos);
|
||||
(b) disc-id SUFFIX `[-\s]\s*CODE-?NUM$` — **unconditional** (trailing `ABCD-0123` is a strong signal);
|
||||
(c) `[Brand]`/`(Brand)` SUFFIX gated on `norm(content) ≈ norm(folder)`;
|
||||
(d) ` - Brand` dash SUFFIX gated on `norm(last segment) ≈ norm(folder)` (or per-folder token);
|
||||
(e) `(CODE)` paren PREFIX (≤6 alnum) with content after;
|
||||
(f) `CODE - ` PREFIX (no number) gated on `CODE ≈ folder acronym/name`.
|
||||
4. split remainder on ` - ` → artist=parts[0], title=join(rest); drop a leading pure-numeric segment;
|
||||
1 part → title only.
|
||||
5. strip any leftover generic karaoke tag from title.
|
||||
6. `brand_code` = captured → **curated folder→code map** → null. `brand` always = folder (in `build_rows`).
|
||||
|
||||
**Guards (must survive):** `(G)I-DLE`, `R.E.M.`, artist `311`, titles `(Live)`, `(Banda Remix)`,
|
||||
`(Single Version 2 WBV)`, `(99X Live)`, `[Acoustic]`. Folder-gating + "disc-id has digits" +
|
||||
"prefix code ≈ folder" are what prevent eating these.
|
||||
|
||||
### Curated folder→KN-code map (minimal, high-confidence only)
|
||||
|
||||
Aligned to actual KN community `Brand` codes (so the secondary brand_match xref can also fire);
|
||||
brands with no confident KN code are left null (artist/title fix already surfaces them via the
|
||||
primary 0.95 exact xref):
|
||||
|
||||
```
|
||||
Funbox Karaoke→FBK · Sandell Karaoke→SDK · BellySings Karaoke→BELLY · DJ Sauly Collection→DJS
|
||||
PlayKaraoke→PLAY · Reekies Karaoke→REEKIES · Cereal Killer Karaoke→CKK · WIZZYOKE→WIZZY
|
||||
Brilliant Trash Karaoke→BTRASH · Lopenash→LOPE · Monster Tracks→MTRAX · Popeoke→POPE
|
||||
KaddaOK→KADDA · Potos Power Plant Productions→PPPP · April's Choice Karaoke→APRIL
|
||||
The Nerdy Singer Karaoke→TNS · Mobile Karaoke Unit→MKU · ATG Karaoke - Zipped MP3+G→ATG
|
||||
It Might Be Karaoke→IMBK
|
||||
```
|
||||
(Map *overrides* a prefix-derived code where they differ, e.g. Play `PLK`→`PLAY`, Reekies
|
||||
`REEK`→`REEKIES`; `disc_id` keeps the literal prefix.) Skipped (no confident code): Rock Solid,
|
||||
Steve-O (uses SOKC, not a KN code — set from disc-id suffix anyway), ForteKaraoke (EBK≠KN), Gnome,
|
||||
7uLpA, Crossfire, Monster… handled where a code exists.
|
||||
|
||||
## Work breakdown
|
||||
|
||||
**Phase 1 — Parser rewrite (TDD, primary deliverable).** New `tests/unit/test_divebar_filename_parser.py`
|
||||
with all gold fixtures + guard cases (add function dir to `sys.path`; parser deps are stdlib only).
|
||||
Rewrite `filename_parser.py` to the design above. Keep `normalize_for_search`, `detect_format`,
|
||||
`should_index_file` behaviour. Keep existing `test_divebar_index_builder.py` green.
|
||||
|
||||
**Phase 2 — brand_code coverage.** Curated `folder→code` map + disc-id-suffix / leading-token
|
||||
capture. Target: cut null `brand_code` well below 37% on the offline harness.
|
||||
|
||||
**Phase 3 — Deploy enablement (infra papercut).** The module points the function at a *static*
|
||||
source object (`divebar-mirror-source.zip`) with no content hash, so overwriting it won't make
|
||||
`pulumi up` redeploy. Switch `modules/divebar_mirror.py` to `pulumi.FileArchive` + `BucketObject`
|
||||
(the proven pattern in `modules/runner_manager.py`) so new code deploys on `pulumi up`. (Optional
|
||||
but needed for the fix to ship cleanly; Andrew runs the apply.)
|
||||
|
||||
**Phase 4 — Offline validation harness + prod verify.**
|
||||
- Export distinct `(filename, brand)` from BQ (read-only) → run NEW parser locally over all 50k →
|
||||
report before/after: % null brand_code, % artist-looks-like-code, % title-has-dash, % title-ends-bracket,
|
||||
plus a spot-check sample. **Proves improvement with zero prod writes** (fits read-only ADC).
|
||||
- After Andrew deploys + triggers `divebar-mirror-daily` then `divebar-xref-rebuild-daily`
|
||||
(MERGE preserves `gcs_path`; no Drive→GCS re-sync needed): re-measure xref `db_files` matched
|
||||
(baseline 23,318/50,046) and the live `incubus admiration` repro.
|
||||
|
||||
## Constraints / notes
|
||||
|
||||
- **My ADC is read-only** (workspace memory `feedback_claude_readonly_adc`): I can read BQ and run
|
||||
the offline harness, but **cannot** `pulumi up`, upload source, or run scheduler jobs. Andrew
|
||||
deploys + re-indexes; I validate offline first and via BQ reads after.
|
||||
- Re-index re-parses ALL files (full Drive listing → MERGE by `file_id`), so existing rows are
|
||||
corrected in place; `gcs_path` is preserved.
|
||||
- Version bump `pyproject.toml`; this is infra/function code (no i18n, no frontend).
|
||||
- Tests: `make test` (or targeted `pytest tests/unit/test_divebar_filename_parser.py`).
|
||||
|
||||
## Results (offline harness over all 50,046 live rows)
|
||||
|
||||
| metric | before | after | delta |
|
||||
|---|---|---|---|
|
||||
| null brand_code | 18,437 (37%) | 1,619 (3%) | **−16,818** |
|
||||
| title contains ` - ` | 5,947 | 2,523 | −3,424 (residual mostly legit multi-part titles) |
|
||||
| artist looks like a code | 1,908 | 1,400 | −508 (residual are real bands: ACDC, KISS, ABBA, U2…) |
|
||||
| null artist | 164 | 179 | +15 (medleys / single-field — acceptable) |
|
||||
| rows with changed artist/title | — | 13,348 (26.7%) | — |
|
||||
|
||||
Per-brand `brand_code` null% collapsed to ~0% for every broken brand (Sandell, BellySings,
|
||||
DJ Sauly, Cereal Killer, Steve-O, WIZZYOKE, Lopenash, ForteKaraoke, ATG, Mobile, 7uLpA, …);
|
||||
Rock Solid stays null by design (no confident KN code).
|
||||
|
||||
**Xref lift (replaying the live `_rebuild_xref` EXACT join):** Divebar files that match a KN
|
||||
song **20,802 → 23,394 (+2,592, +12.5%)** — i.e. that many more mirror files now surface in kjbox.
|
||||
|
||||
### BUNDLED: the xref SQL normalization asymmetry (fixed in this PR)
|
||||
|
||||
`_rebuild_xref` joined `LOWER(TRIM(kn.Artist)) = db.artist_normalized`, but `db.artist_normalized`
|
||||
is produced by `normalize_for_search` (strips diacritics, a leading "the ", and all punctuation)
|
||||
while the KN side got only `LOWER(TRIM(...))` — so most rows could never match. **Fix:** a new
|
||||
`_norm_sql(col)` helper routes **both** sides through one identical BigQuery expression
|
||||
replicating `normalize_for_search` (NFD + drop `\p{Mn}`, lower/trim, strip trailing karaoke tag,
|
||||
strip leading "the ", drop non-`\p{L}\p{N}_\s`, collapse whitespace). Applied to the exact join,
|
||||
the brand_match join, and the NOT EXISTS de-dup.
|
||||
|
||||
**Validated read-only against live BigQuery** (the real `_norm_sql`): the SQL normalization
|
||||
reproduces the Python `normalize_for_search` baseline exactly (32,496 on current data), and the
|
||||
**full new union already matches 33,449 distinct Divebar files on current (old-parse) data vs
|
||||
production's 23,318 — +10,131 from symmetry alone**, before any re-index. After the parser
|
||||
re-index the exact portion lifts toward ~37,910, compounding further. Residual misses after the parser fix are tiny and documented: `(WTF Karaone)` typo
|
||||
(~6), `SOKC-O104` letter-O typo (~21), `GGZ007-02,2` comma in disc-id (~16), `RVZ-033.d` (~5).
|
||||
|
||||
## Status: IMPLEMENTED
|
||||
|
||||
- `filename_parser.py` rewritten folder-brand-aware; `tests/unit/test_divebar_filename_parser.py`
|
||||
added (48 cases, green); existing `test_divebar_index_builder.py` still green.
|
||||
- `divebar_lookup/main.py` `_rebuild_xref` rewritten to normalize both sides via `_norm_sql`;
|
||||
3 guard tests added to `test_divebar_lookup` (`test_main.py`). All 64 divebar tests green.
|
||||
- `modules/divebar_mirror.py` switched to `pulumi.FileArchive` + `BucketObject` with
|
||||
generation-pinning so `pulumi up` actually redeploys new function code; `deploy.sh` note updated.
|
||||
- Version bumped 0.188.3 → 0.188.4.
|
||||
- **Deploy (Andrew, read-only ADC blocks me):** `cd infrastructure && pulumi up` (redeploys both
|
||||
`divebar-mirror` and `divebar-lookup`), then trigger scheduler jobs `divebar-mirror-daily`
|
||||
(re-parse → MERGE, preserves `gcs_path`) then `divebar-xref-rebuild-daily`. Then re-measure xref
|
||||
`db_files` and the live `incubus admiration` repro. (Order matters: re-index before xref rebuild.)
|
||||
|
||||
## Open questions for Andrew
|
||||
|
||||
1. **Include Phase 3** (FileArchive deploy fix) in this PR, or keep PR parser-only and handle deploy
|
||||
mechanics separately?
|
||||
2. **brand_code map scope** — minimal (only folders that clearly map to a KN community code) vs
|
||||
broader best-effort? Minimal is safer (no mismaps); recommend minimal.
|
||||
3. Confirm **you'll run the `pulumi up` + scheduler re-index** after merge (I can't with read-only ADC).
|
||||
@@ -177,16 +177,51 @@ def _lookup_kn_ids(kn_ids: list[int]) -> dict[int, list[dict]]:
|
||||
return result
|
||||
|
||||
|
||||
def _norm_sql(col: str) -> str:
|
||||
"""A BigQuery expression replicating divebar_mirror's normalize_for_search.
|
||||
|
||||
Both the KaraokeNerds side and the Divebar side of the xref join are passed
|
||||
through this so the comparison is symmetric. Previously the join compared the
|
||||
KN side (only LOWER+TRIM) against the Divebar side's fully-normalized columns
|
||||
(diacritics / leading "the " / punctuation stripped), so most rows could never
|
||||
match. Steps mirror normalize_for_search: strip diacritics (NFD + drop combining
|
||||
marks), lower/trim, strip a trailing karaoke tag, strip a leading "the ", drop
|
||||
non-word chars (keeping unicode letters/digits/underscore), collapse whitespace.
|
||||
"""
|
||||
return (
|
||||
"TRIM(REGEXP_REPLACE("
|
||||
"REGEXP_REPLACE("
|
||||
"REGEXP_REPLACE("
|
||||
"REGEXP_REPLACE("
|
||||
"LOWER(TRIM(REGEXP_REPLACE(NORMALIZE(COALESCE(" + col + ", ''), NFD), r'\\p{Mn}', ''))),"
|
||||
r" r'\s*[\(\[]\s*(?:[^)\]]*karaoke[^)\]]*|(?:instrumental|backing track|no vocals?|kj version|with vocals?|demo)[^)\]]*)[\)\]]\s*$', ''),"
|
||||
r" r'^the ', ''),"
|
||||
r" r'[^\p{L}\p{N}_\s]', ''),"
|
||||
r" r'\s+', ' '))"
|
||||
)
|
||||
|
||||
|
||||
def _rebuild_xref() -> dict:
|
||||
"""Rebuild the KN ↔ Divebar cross-reference index using exact + fuzzy matching."""
|
||||
"""Rebuild the KN ↔ Divebar cross-reference index by exact normalized match.
|
||||
|
||||
Both sides are normalized through the identical `_norm_sql` expression, so a
|
||||
KN song links to a Divebar file only when their artist AND title agree after
|
||||
normalization. (The previous "brand_match" branch matched on brand_code +
|
||||
artist only — with no title constraint it was a Cartesian product that linked
|
||||
every KN song by an artist/brand to every Divebar file by the same
|
||||
artist/brand, i.e. wrong-song links. Constraining it by title would make it a
|
||||
strict subset of the exact match, so it was removed.)
|
||||
"""
|
||||
client = bigquery.Client(project=GCP_PROJECT_ID)
|
||||
start = time.time()
|
||||
|
||||
# Step 1: Exact match on normalized artist + title
|
||||
kn_artist, kn_title = _norm_sql("kn.Artist"), _norm_sql("kn.Title")
|
||||
db_artist, db_title = _norm_sql("db.artist"), _norm_sql("db.title")
|
||||
|
||||
exact_sql = f"""
|
||||
CREATE OR REPLACE TABLE `{GCP_PROJECT_ID}.{DATASET}.kn_divebar_xref` AS
|
||||
|
||||
-- Exact normalized match (high confidence)
|
||||
-- Exact match on symmetrically-normalized artist + title (high confidence)
|
||||
SELECT DISTINCT
|
||||
kn.Id AS kn_id,
|
||||
db.file_id AS divebar_file_id,
|
||||
@@ -195,40 +230,10 @@ def _rebuild_xref() -> dict:
|
||||
CURRENT_TIMESTAMP() AS matched_at
|
||||
FROM `{GCP_PROJECT_ID}.{DATASET}.karaokenerds_raw` kn
|
||||
JOIN `{GCP_PROJECT_ID}.{DATASET}.divebar_catalog` db
|
||||
ON LOWER(TRIM(kn.Artist)) = db.artist_normalized
|
||||
AND LOWER(TRIM(kn.Title)) = db.title_normalized
|
||||
WHERE db.artist_normalized IS NOT NULL
|
||||
AND db.artist_normalized != ''
|
||||
AND db.title_normalized IS NOT NULL
|
||||
AND db.title_normalized != ''
|
||||
|
||||
UNION ALL
|
||||
|
||||
-- Community brand match: KN community track brand matches Divebar folder brand code
|
||||
SELECT DISTINCT
|
||||
kn.Id AS kn_id,
|
||||
db.file_id AS divebar_file_id,
|
||||
'brand_match' AS match_type,
|
||||
0.90 AS confidence,
|
||||
CURRENT_TIMESTAMP() AS matched_at
|
||||
FROM `{GCP_PROJECT_ID}.{DATASET}.karaokenerds_community` c
|
||||
JOIN `{GCP_PROJECT_ID}.{DATASET}.karaokenerds_raw` kn
|
||||
ON LOWER(TRIM(c.Artist)) = LOWER(TRIM(kn.Artist))
|
||||
AND LOWER(TRIM(c.Title)) = LOWER(TRIM(kn.Title))
|
||||
JOIN `{GCP_PROJECT_ID}.{DATASET}.divebar_catalog` db
|
||||
ON LOWER(c.Brand) = LOWER(COALESCE(db.brand_code, ''))
|
||||
AND LOWER(TRIM(c.Artist)) = db.artist_normalized
|
||||
WHERE c.Brand IS NOT NULL
|
||||
AND c.Brand != ''
|
||||
AND db.artist_normalized IS NOT NULL
|
||||
AND db.artist_normalized != ''
|
||||
-- Exclude exact matches (already captured above)
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM `{GCP_PROJECT_ID}.{DATASET}.divebar_catalog` db2
|
||||
WHERE LOWER(TRIM(kn.Artist)) = db2.artist_normalized
|
||||
AND LOWER(TRIM(kn.Title)) = db2.title_normalized
|
||||
AND db2.file_id = db.file_id
|
||||
)
|
||||
ON {kn_artist} = {db_artist}
|
||||
AND {kn_title} = {db_title}
|
||||
WHERE {db_artist} != ''
|
||||
AND {db_title} != ''
|
||||
"""
|
||||
|
||||
logger.info("Rebuilding cross-reference index...")
|
||||
|
||||
@@ -108,6 +108,33 @@ class TestRefreshTriggers:
|
||||
assert result["failed"][0]["job"] == main.REFRESH_SCHEDULER_JOBS[0]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# _norm_sql — symmetric normalization for the xref join
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestNormSql:
|
||||
def test_embeds_column_and_is_deterministic(self):
|
||||
a = main._norm_sql("kn.Artist")
|
||||
assert "kn.Artist" in a
|
||||
assert main._norm_sql("kn.Artist") == a # pure / deterministic
|
||||
|
||||
def test_replicates_normalize_for_search_steps(self):
|
||||
expr = main._norm_sql("db.title")
|
||||
# diacritics (NFD + drop combining marks), lower, leading "the", and the
|
||||
# unicode-aware punctuation strip must all be present.
|
||||
assert "NORMALIZE(COALESCE(db.title" in expr
|
||||
assert "NFD" in expr and r"\p{Mn}" in expr
|
||||
assert "LOWER(" in expr
|
||||
assert r"'^the '" in expr
|
||||
assert r"\p{L}" in expr and r"\p{N}" in expr
|
||||
|
||||
def test_both_sides_use_same_expression(self):
|
||||
# The whole point of the fix: KN and Divebar sides normalize identically.
|
||||
kn = main._norm_sql("X")
|
||||
db = main._norm_sql("X")
|
||||
assert kn == db
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# divebar_lookup dispatch — refresh action
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@@ -1,17 +1,14 @@
|
||||
#!/bin/bash
|
||||
# Deploy the Divebar Mirror Cloud Function
|
||||
#
|
||||
# This script:
|
||||
# 1. Creates a ZIP of the function source
|
||||
# 2. Uploads it to GCS
|
||||
# 3. Deploys the function via gcloud
|
||||
# NOTE: As of 2026-06, the function source is managed by Pulumi
|
||||
# (modules/divebar_mirror.py zips this directory via pulumi.FileArchive and pins
|
||||
# the function to the object generation). The normal way to ship a code change is:
|
||||
#
|
||||
# Prerequisites:
|
||||
# - gcloud CLI authenticated
|
||||
# - Pulumi infrastructure already deployed (creates the bucket)
|
||||
# cd infrastructure && pulumi up
|
||||
#
|
||||
# Usage:
|
||||
# ./deploy.sh
|
||||
# This script remains for ad-hoc manual uploads/testing only; a manual upload
|
||||
# will be reset to the repo contents on the next `pulumi up`.
|
||||
|
||||
set -e
|
||||
|
||||
|
||||
@@ -1,67 +1,148 @@
|
||||
"""
|
||||
Parse karaoke filenames into structured metadata.
|
||||
|
||||
Handles the various naming conventions found across Divebar karaoke brands:
|
||||
Pattern 1: "BRAND-001 - Artist - Title.ext" (disc ID prefix)
|
||||
Pattern 2: "Artist - Title - (Brand Tag).ext" (brand suffix)
|
||||
Pattern 3: "Artist - Title.ext" (brand from folder name)
|
||||
Community karaoke brands use many naming conventions. Rather than a handful of
|
||||
rigid patterns, this parser is *folder-brand-aware*: the top-level Drive folder
|
||||
reliably names the brand, so we use it to recognise and strip the brand token
|
||||
wherever it sits (prefix, suffix, bracket, disc-id) and then split the remainder
|
||||
into artist / title. Brand stripping is gated on the known folder brand so we
|
||||
never eat real song text like "(Live)", "(Banda Remix)" or "(G)I-DLE".
|
||||
|
||||
Returned dict keys: artist, title, disc_id, brand_code, brand_name, format.
|
||||
"""
|
||||
|
||||
import os
|
||||
import re
|
||||
import unicodedata
|
||||
|
||||
|
||||
# Common karaoke suffixes to strip from titles
|
||||
_KARAOKE_SUFFIXES = re.compile(
|
||||
r"\s*[\(\[](karaoke|instrumental|backing track|no vocals?|kj version|"
|
||||
r"karaoke version|with vocals?|demo)[\)\]]\s*$",
|
||||
# Generic karaoke/source tags to strip from the end of a title, in () or [].
|
||||
# Matches any bracket whose content mentions "karaoke" ("(WTF Karaoke)",
|
||||
# "[DJ Sauly Karaoke]", "(Karaoke HD)") or starts with another generic tag
|
||||
# ("(instrumental)", "(backing track)").
|
||||
_KARAOKE_TAG = re.compile(
|
||||
r"\s*[\(\[]\s*(?:[^)\]]*karaoke[^)\]]*|"
|
||||
r"(?:instrumental|backing track|no vocals?|kj version|with vocals?|demo)[^)\]]*)"
|
||||
r"[\)\]]\s*$",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
# Back-compat alias (normalize_for_search historically referenced this name).
|
||||
_KARAOKE_SUFFIXES = _KARAOKE_TAG
|
||||
|
||||
# Brand code pattern at start of filename: "ABC-001" or "ABC001" or "ABC 001"
|
||||
_DISC_ID_PATTERN = re.compile(
|
||||
r"^([A-Z][A-Z0-9]{1,10})[\s\-]?(\d{1,5}(?:-\d{1,3})?)\s*-\s*"
|
||||
)
|
||||
# disc-id PREFIX token followed by a " - " or plain-space separator. The token
|
||||
# (e.g. "FATBIRD-163", "IK00-01", "KFNO101", "RABZ-0001", "Lopenash-002") must
|
||||
# contain both a letter and a digit; acceptance is further gated in code (code
|
||||
# all-uppercase, or the code matches the folder brand) so alphanumeric artists
|
||||
# like "2Pac", "311" or "M83" are never eaten.
|
||||
_DISC_ID_PREFIX = re.compile(r"^([A-Za-z0-9][A-Za-z0-9.\-]*?)(?:\s*-\s+|\s+)(?=\S)")
|
||||
# disc-id SUFFIX at end: " - SOKC-0306", " ATG0072", " IMBK-0076". Captures the
|
||||
# whole token (preserving any internal dash) so disc_id stays literal.
|
||||
_DISC_ID_SUFFIX = re.compile(r"[\s\-]+([A-Z]{2,6}-?\d{2,5}(?:-\d{1,3})?)\s*$")
|
||||
# Leading "(CODE) " brand prefix (Sandell "(SDK) ...").
|
||||
_PAREN_CODE_PREFIX = re.compile(r"^\(\s*([A-Za-z0-9]{1,6})\s*\)\s+")
|
||||
# A leading pure track-number segment, e.g. "02" or "08 (and a half)".
|
||||
_LEADING_TRACK_NO = re.compile(r"^\d{1,3}(?:\s*\(.*?\))?$")
|
||||
|
||||
# Brand tag in parentheses at end of filename (before extension)
|
||||
_BRAND_SUFFIX_PATTERN = re.compile(r"\s*-?\s*\(([^)]+)\)\s*$")
|
||||
# Curated folder -> brand code map. Aligned to real KaraokeNerds community codes
|
||||
# where confident (so the secondary brand_match xref can also fire). Overrides a
|
||||
# prefix/suffix-derived code (disc_id keeps the literal). Folders with no
|
||||
# confident code are omitted (artist/title fix already surfaces them via the
|
||||
# primary exact xref). Keys are lowercased exact folder names.
|
||||
FOLDER_CODE = {
|
||||
"funbox karaoke": "FBK",
|
||||
"sandell karaoke": "SDK",
|
||||
"bellysings karaoke": "BELLY",
|
||||
"dj sauly collection": "DJS",
|
||||
"playkaraoke": "PLAY",
|
||||
"reekies karaoke": "REEKIES",
|
||||
"cereal killer karaoke": "CKK",
|
||||
"wizzyoke": "WIZZY",
|
||||
"brilliant trash karaoke": "BTRASH",
|
||||
"lopenash": "LOPE",
|
||||
"monster tracks": "MTRAX",
|
||||
"popeoke": "POPE",
|
||||
"kaddaok": "KADDA",
|
||||
"potos power plant productions": "PPPP",
|
||||
"april's choice karaoke": "APRIL",
|
||||
"the nerdy singer karaoke": "TNS",
|
||||
"mobile karaoke unit": "MKU",
|
||||
"atg karaoke - zipped mp3+g": "ATG",
|
||||
"it might be karaoke": "IMBK",
|
||||
"mix tape karaoke": "MIXTAPE",
|
||||
"fortekaraoke discord": "EBK",
|
||||
"steve-o karaoke cult": "SOKC",
|
||||
}
|
||||
|
||||
# Descriptor words dropped when fuzzy-matching a brand token against a folder name.
|
||||
_BRAND_NOISE = {
|
||||
"karaoke", "collection", "cdg", "files", "file", "discord", "videos", "video",
|
||||
"mp4", "mp3", "zipped", "presents", "studio", "made", "the", "of", "and",
|
||||
}
|
||||
|
||||
# Video and audio extensions we care about
|
||||
KARAOKE_EXTENSIONS = {
|
||||
".mp4", ".mkv", ".avi", ".webm", ".mov", # Video
|
||||
".mp3", ".cdg", # CDG+MP3 pairs
|
||||
".zip", # Zipped CDG+MP3
|
||||
}
|
||||
|
||||
# Files to skip during indexing
|
||||
IGNORED_FILES = {
|
||||
".DS_Store", ".keep", ".gitkeep", "desktop.ini", "Thumbs.db",
|
||||
"kj-nomad.index.json",
|
||||
}
|
||||
|
||||
|
||||
def _strip_diacritics(text: str) -> str:
|
||||
text = unicodedata.normalize("NFD", text)
|
||||
return "".join(c for c in text if unicodedata.category(c) != "Mn")
|
||||
|
||||
|
||||
def normalize_for_search(text: str) -> str:
|
||||
"""Normalize text for fuzzy matching: lowercase, strip diacritics, remove special chars."""
|
||||
"""Normalize text for fuzzy matching: lowercase, strip diacritics/special chars."""
|
||||
if not text:
|
||||
return ""
|
||||
# NFD decomposition then strip combining characters (diacritics)
|
||||
text = unicodedata.normalize("NFD", text)
|
||||
text = "".join(c for c in text if unicodedata.category(c) != "Mn")
|
||||
text = _strip_diacritics(text)
|
||||
text = text.lower().strip()
|
||||
# Strip common karaoke suffixes
|
||||
text = _KARAOKE_SUFFIXES.sub("", text)
|
||||
# Remove "the " prefix for matching
|
||||
text = _KARAOKE_TAG.sub("", text)
|
||||
if text.startswith("the "):
|
||||
text = text[4:]
|
||||
# Normalize whitespace and strip non-alphanumeric (keep spaces)
|
||||
text = re.sub(r"[^\w\s]", "", text)
|
||||
text = re.sub(r"\s+", " ", text).strip()
|
||||
return text
|
||||
|
||||
|
||||
def _brand_key(folder: str) -> str:
|
||||
"""Significant-word signature of a folder/brand name for fuzzy comparison."""
|
||||
base = re.sub(r"[^\w\s]", " ", _strip_diacritics(folder or "").lower())
|
||||
words = [w for w in base.split() if w and w not in _BRAND_NOISE]
|
||||
return "".join(words)
|
||||
|
||||
|
||||
def _brand_acronym(folder: str) -> str:
|
||||
# Initials of ALL words (brands often acronym the full name, e.g.
|
||||
# "The Nerdy Singer Karaoke" -> TNSK, "Steve-O Karaoke Cult" -> SOKC).
|
||||
words = re.findall(r"[A-Za-z0-9]+", _strip_diacritics(folder or "").lower())
|
||||
return "".join(w[0] for w in words).upper()
|
||||
|
||||
|
||||
def _folder_code(folder: str) -> str | None:
|
||||
return FOLDER_CODE.get((folder or "").strip().lower())
|
||||
|
||||
|
||||
def _matches_folder_brand(token: str, folder: str) -> bool:
|
||||
"""True if a stripped token looks like this folder's brand (not song text)."""
|
||||
if not token or not folder:
|
||||
return False
|
||||
tk = _brand_key(token)
|
||||
fk = _brand_key(folder)
|
||||
if tk and fk and (tk == fk or (len(tk) >= 4 and (tk in fk or fk in tk))):
|
||||
return True
|
||||
up = token.strip().strip("()[]").upper()
|
||||
if up and (up == _brand_acronym(folder) or up == (_folder_code(folder) or "\0")):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def detect_format(filename: str) -> str:
|
||||
"""Detect the karaoke format from a filename."""
|
||||
"""Detect karaoke file format from extension."""
|
||||
ext = os.path.splitext(filename)[1].lower()
|
||||
if ext == ".zip":
|
||||
return "zip"
|
||||
@@ -70,32 +151,97 @@ def detect_format(filename: str) -> str:
|
||||
if ext == ".mp3":
|
||||
return "mp3"
|
||||
if ext in (".mp4", ".mkv", ".avi", ".webm", ".mov"):
|
||||
return ext[1:] # Strip the dot
|
||||
return ext[1:]
|
||||
return ext[1:] if ext else "unknown"
|
||||
|
||||
|
||||
def should_index_file(filename: str) -> bool:
|
||||
"""Check if a file should be included in the index."""
|
||||
"""Whether a file should be included in the index."""
|
||||
if filename in IGNORED_FILES or filename.startswith("."):
|
||||
return False
|
||||
ext = os.path.splitext(filename)[1].lower()
|
||||
return ext in KARAOKE_EXTENSIONS
|
||||
|
||||
|
||||
def _strip_trailing_noise(text: str) -> str:
|
||||
"""Remove a trailing generic karaoke tag and dangling separators."""
|
||||
prev = None
|
||||
while prev != text:
|
||||
prev = text
|
||||
text = _KARAOKE_TAG.sub("", text).strip()
|
||||
text = re.sub(r"[\s\-]+$", "", text).strip()
|
||||
return text
|
||||
|
||||
|
||||
def _disc_code(token: str) -> str:
|
||||
"""Derive the brand code (stem) from a disc-id token.
|
||||
|
||||
"IK00-01" -> "IK00", "FATBIRD-163" -> "FATBIRD", "C11K-0000A" -> "C11K",
|
||||
"KFNO101" -> "KFNO", "Lopenash-002" -> "Lopenash".
|
||||
"""
|
||||
m = re.match(r"^(.*?)-\d[\w]*$", token)
|
||||
if m:
|
||||
return m.group(1)
|
||||
m = re.match(r"^([A-Za-z]+)\d+$", token)
|
||||
if m:
|
||||
return m.group(1)
|
||||
return token
|
||||
|
||||
|
||||
def _try_prefix(name: str, folder: str) -> tuple[str, str | None, str | None, bool]:
|
||||
"""Strip a leading brand token.
|
||||
|
||||
Returns (remainder, brand_code, disc_id, disc_prefix_stripped).
|
||||
"""
|
||||
# 1. disc-id prefix (token contains a letter and a digit; accepted only when
|
||||
# the code is all-uppercase or matches the folder brand).
|
||||
m = _DISC_ID_PREFIX.match(name)
|
||||
if m:
|
||||
token = m.group(1)
|
||||
if re.search(r"[A-Za-z]", token) and re.search(r"\d", token):
|
||||
code = _disc_code(token)
|
||||
num_part = token[len(code):]
|
||||
# "strong" disc-id: code separated from number by a hyphen, or the
|
||||
# number has >=3 digits. Prevents alphanumeric artists like "UB40",
|
||||
# "D12", "AC3" (uppercase but not disc-shaped) from being eaten.
|
||||
strong = "-" in num_part or len(re.findall(r"\d", num_part)) >= 3
|
||||
if _matches_folder_brand(code, folder) or (
|
||||
code.isupper() and len(code) >= 2 and strong
|
||||
):
|
||||
return name[m.end():].strip(), code, token, True
|
||||
# 2. "(CODE) " paren prefix
|
||||
m = _PAREN_CODE_PREFIX.match(name)
|
||||
if m:
|
||||
return name[m.end():].strip(), m.group(1).upper(), None, False
|
||||
# 3. "CODE - " / "Brand - " prefix (no number), gated on folder match
|
||||
m = re.match(r"^(.{1,30}?)\s+-\s+(?=\S)", name)
|
||||
if m:
|
||||
token = m.group(1).strip()
|
||||
if _matches_folder_brand(token, folder):
|
||||
code = token.upper() if re.fullmatch(r"[A-Za-z0-9]{1,8}", token) else None
|
||||
return name[m.end():].strip(), code, None, False
|
||||
return name, None, None, False
|
||||
|
||||
|
||||
def _strip_brand_suffix(core: str, folder: str) -> str:
|
||||
"""Strip a trailing [Brand]/(Brand)/ - Brand token when it matches the folder."""
|
||||
# bracket / paren suffix
|
||||
m = re.search(r"\s*[\(\[]([^)\]]+)[\)\]]\s*$", core)
|
||||
if m and _matches_folder_brand(m.group(1), folder):
|
||||
return core[: m.start()].strip()
|
||||
# " - Brand" dash suffix (only if there is something before it to keep)
|
||||
if " - " in core:
|
||||
head, _, last = core.rpartition(" - ")
|
||||
if head and _matches_folder_brand(last, folder):
|
||||
return head.strip()
|
||||
return core
|
||||
|
||||
|
||||
def parse_filename(filename: str, brand_folder: str = "") -> dict:
|
||||
"""
|
||||
Parse a karaoke filename into structured metadata.
|
||||
|
||||
Args:
|
||||
filename: The file name (not path), e.g. "NOMAD-0001 - Artist - Title.mp4"
|
||||
brand_folder: The parent brand folder name, used as fallback brand
|
||||
|
||||
Returns:
|
||||
dict with keys: artist, title, disc_id, brand_code, brand_name, format
|
||||
"""
|
||||
# Strip extension
|
||||
name, ext = os.path.splitext(filename)
|
||||
file_format = detect_format(filename)
|
||||
"""Parse a karaoke filename into structured metadata (see module docstring)."""
|
||||
name, _ext = os.path.splitext(filename)
|
||||
# Normalise spaced en/em dashes to the standard " - " artist/title separator.
|
||||
name = name.replace(" – ", " - ").replace(" — ", " - ").strip()
|
||||
|
||||
result = {
|
||||
"artist": None,
|
||||
@@ -103,45 +249,53 @@ def parse_filename(filename: str, brand_folder: str = "") -> dict:
|
||||
"disc_id": None,
|
||||
"brand_code": None,
|
||||
"brand_name": brand_folder or None,
|
||||
"format": file_format,
|
||||
"format": detect_format(filename),
|
||||
}
|
||||
|
||||
# Try Pattern 1: "BRAND-001 - Artist - Title"
|
||||
disc_match = _DISC_ID_PATTERN.match(name)
|
||||
if disc_match:
|
||||
result["brand_code"] = disc_match.group(1)
|
||||
result["disc_id"] = f"{disc_match.group(1)}-{disc_match.group(2)}"
|
||||
remainder = name[disc_match.end():]
|
||||
parts = [p.strip() for p in remainder.split(" - ")]
|
||||
if len(parts) >= 2:
|
||||
result["artist"] = parts[0]
|
||||
result["title"] = " - ".join(parts[1:])
|
||||
elif len(parts) == 1 and parts[0]:
|
||||
result["title"] = parts[0]
|
||||
# Strip brand suffix from title if present
|
||||
if result["title"]:
|
||||
result["title"] = _BRAND_SUFFIX_PATTERN.sub("", result["title"])
|
||||
return result
|
||||
# 1. leading brand token (disc-id / paren-code / folder-name prefix)
|
||||
core, brand_code, disc_id, disc_prefix_stripped = _try_prefix(name, brand_folder)
|
||||
if disc_prefix_stripped:
|
||||
# Some brands write "CODE-0002 -Artist" (dash hugging the artist); drop the
|
||||
# leftover leading separator so the artist isn't "-Artist".
|
||||
core = core.lstrip(" -–").strip()
|
||||
|
||||
# Try Pattern 2: "Artist - Title - (Brand Tag)"
|
||||
brand_suffix_match = _BRAND_SUFFIX_PATTERN.search(name)
|
||||
if brand_suffix_match:
|
||||
result["brand_name"] = brand_suffix_match.group(1)
|
||||
name_without_brand = name[:brand_suffix_match.start()]
|
||||
parts = [p.strip() for p in name_without_brand.split(" - ")]
|
||||
if len(parts) >= 2:
|
||||
result["artist"] = parts[0]
|
||||
result["title"] = " - ".join(parts[1:])
|
||||
elif len(parts) == 1 and parts[0]:
|
||||
result["title"] = parts[0]
|
||||
return result
|
||||
# 2. trailing generic karaoke tag (so a disc-id suffix is reachable)
|
||||
core = _strip_trailing_noise(core)
|
||||
|
||||
# Pattern 3: "Artist - Title" (brand from folder name)
|
||||
parts = [p.strip() for p in name.split(" - ")]
|
||||
# 3. disc-id SUFFIX (strong signal; unconditional)
|
||||
if disc_id is None:
|
||||
m = _DISC_ID_SUFFIX.search(core)
|
||||
if m:
|
||||
token = m.group(1)
|
||||
core = core[: m.start()].strip()
|
||||
core = _strip_trailing_noise(core)
|
||||
brand_code = brand_code or _disc_code(token)
|
||||
disc_id = token
|
||||
|
||||
# 4. trailing brand token in brackets/parens/dash, gated on folder
|
||||
core = _strip_brand_suffix(core, brand_folder)
|
||||
core = _strip_trailing_noise(core)
|
||||
|
||||
# 5. split remainder into artist / title. Drop a leading track-number segment
|
||||
# only when it followed a disc-id prefix (e.g. "MIX001 - 02 - Artist - T"),
|
||||
# so numeric artists like "311" are never mistaken for a track number.
|
||||
parts = [p.strip() for p in core.split(" - ") if p.strip()]
|
||||
if (
|
||||
disc_prefix_stripped
|
||||
and len(parts) >= 3
|
||||
and _LEADING_TRACK_NO.match(parts[0])
|
||||
):
|
||||
parts = parts[1:]
|
||||
if len(parts) >= 2:
|
||||
result["artist"] = parts[0]
|
||||
result["title"] = " - ".join(parts[1:])
|
||||
elif len(parts) == 1 and parts[0]:
|
||||
elif len(parts) == 1:
|
||||
result["title"] = parts[0]
|
||||
|
||||
if result["title"]:
|
||||
result["title"] = _KARAOKE_TAG.sub("", result["title"]).strip()
|
||||
|
||||
# 6. brand_code: curated folder map overrides; else captured code
|
||||
result["disc_id"] = disc_id
|
||||
result["brand_code"] = _folder_code(brand_folder) or brand_code
|
||||
return result
|
||||
@@ -8,6 +8,8 @@ Creates:
|
||||
- Cloud Scheduler to rebuild xref daily (after mirror + KN sync complete)
|
||||
"""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pulumi
|
||||
import pulumi_gcp as gcp
|
||||
from pulumi_gcp import (
|
||||
@@ -72,6 +74,17 @@ def create_divebar_lookup_resources(all_secrets: dict) -> dict:
|
||||
)
|
||||
resources["source_bucket"] = source_bucket
|
||||
|
||||
# Content-hashed source archive so `pulumi up` redeploys the function on code
|
||||
# changes (a static object name never triggered a redeploy).
|
||||
source_dir = Path(__file__).parent.parent / "functions" / "divebar_lookup"
|
||||
source_archive = storage.BucketObject(
|
||||
"divebar-lookup-source",
|
||||
bucket=source_bucket.name,
|
||||
name="divebar-lookup-source.zip",
|
||||
source=pulumi.FileArchive(str(source_dir)),
|
||||
)
|
||||
resources["source_archive"] = source_archive
|
||||
|
||||
# ==================== BigQuery: Cross-Reference Table ====================
|
||||
|
||||
xref_table = bigquery.Table(
|
||||
@@ -103,7 +116,8 @@ def create_divebar_lookup_resources(all_secrets: dict) -> dict:
|
||||
source=cloudfunctionsv2.FunctionBuildConfigSourceArgs(
|
||||
storage_source=cloudfunctionsv2.FunctionBuildConfigSourceStorageSourceArgs(
|
||||
bucket=source_bucket.name,
|
||||
object="divebar-lookup-source.zip",
|
||||
object=source_archive.name,
|
||||
generation=source_archive.generation,
|
||||
),
|
||||
),
|
||||
),
|
||||
|
||||
@@ -10,6 +10,8 @@ Creates:
|
||||
- BigQuery table for the Divebar catalog index
|
||||
"""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pulumi
|
||||
import pulumi_gcp as gcp
|
||||
from pulumi_gcp import (
|
||||
@@ -99,6 +101,18 @@ def create_divebar_mirror_resources(all_secrets: dict) -> dict:
|
||||
)
|
||||
resources["source_bucket"] = source_bucket
|
||||
|
||||
# Zip the function source and upload it as a content-hashed object so that
|
||||
# `pulumi up` redeploys the function whenever the code changes (the previous
|
||||
# static object name never triggered a redeploy on code edits).
|
||||
source_dir = Path(__file__).parent.parent / "functions" / "divebar_mirror"
|
||||
source_archive = storage.BucketObject(
|
||||
"divebar-mirror-source",
|
||||
bucket=source_bucket.name,
|
||||
name="divebar-mirror-source.zip",
|
||||
source=pulumi.FileArchive(str(source_dir)),
|
||||
)
|
||||
resources["source_archive"] = source_archive
|
||||
|
||||
# ==================== BigQuery Table ====================
|
||||
|
||||
# The table is in the karaoke_decide dataset (managed by karaoke-decide Pulumi).
|
||||
@@ -143,7 +157,11 @@ def create_divebar_mirror_resources(all_secrets: dict) -> dict:
|
||||
source=cloudfunctionsv2.FunctionBuildConfigSourceArgs(
|
||||
storage_source=cloudfunctionsv2.FunctionBuildConfigSourceStorageSourceArgs(
|
||||
bucket=source_bucket.name,
|
||||
object="divebar-mirror-source.zip",
|
||||
object=source_archive.name,
|
||||
# Pin to the object generation so replacing the source bundle
|
||||
# (a code change) registers as a diff on the Function and
|
||||
# forces a redeploy, instead of silently serving the old copy.
|
||||
generation=source_archive.generation,
|
||||
),
|
||||
),
|
||||
),
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
[tool.poetry]
|
||||
name = "karaoke-gen"
|
||||
version = "0.188.3"
|
||||
version = "0.188.4"
|
||||
description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens."
|
||||
authors = ["Andrew Beveridge <andrew@beveridge.uk>"]
|
||||
license = "MIT"
|
||||
|
||||
@@ -0,0 +1,336 @@
|
||||
"""Unit tests for the Divebar filename parser.
|
||||
|
||||
The parser lives in the Cloud Function package (not on the default path), so we
|
||||
add its directory to sys.path and import it directly. It has stdlib-only deps.
|
||||
|
||||
Fixtures are real filenames observed in the live `divebar_catalog` index, grouped
|
||||
by the naming convention each community brand uses. The goal of the rewrite is
|
||||
folder-brand-aware parsing: use the (reliable) top-level folder brand to strip the
|
||||
brand token wherever it sits, then split the remainder into artist/title.
|
||||
"""
|
||||
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
import pytest
|
||||
|
||||
FUNC_DIR = (
|
||||
pathlib.Path(__file__).resolve().parents[2]
|
||||
/ "infrastructure"
|
||||
/ "functions"
|
||||
/ "divebar_mirror"
|
||||
)
|
||||
sys.path.insert(0, str(FUNC_DIR))
|
||||
|
||||
import filename_parser as fp # noqa: E402
|
||||
|
||||
|
||||
def parse(filename, folder):
|
||||
return fp.parse_filename(filename, brand_folder=folder)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# disc-id PREFIX (works today — must be preserved)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"filename,folder,artist,title,brand_code,disc_id",
|
||||
[
|
||||
("FATBIRD-163 - Incubus - I Miss You.zip", "Fatbird Karaoke",
|
||||
"Incubus", "I Miss You", "FATBIRD", "FATBIRD-163"),
|
||||
("NOMAD-0003 - Cher - Apples Don't Fall Far From The Tree.mp4", "Nomad Karaoke",
|
||||
"Cher", "Apples Don't Fall Far From The Tree", "NOMAD", "NOMAD-0003"),
|
||||
("IK00-01 - Costello, Elvis - Goon Squad.zip", "Imperfekt Karaoke",
|
||||
"Costello, Elvis", "Goon Squad", "IK00", "IK00-01"),
|
||||
("GGZ001-02 - Partman Parthorse - Magik (ggnzla REMIX).cdg", "GGZ-ggnzla",
|
||||
"Partman Parthorse", "Magik (ggnzla REMIX)", "GGZ001", "GGZ001-02"),
|
||||
("KFNO101 - The Kills - List of Demands (Reparations).mp4", "Karaoke For No One",
|
||||
"The Kills", "List of Demands (Reparations)", "KFNO", "KFNO101"),
|
||||
],
|
||||
)
|
||||
def test_disc_id_prefix(filename, folder, artist, title, brand_code, disc_id):
|
||||
r = parse(filename, folder)
|
||||
assert r["artist"] == artist
|
||||
assert r["title"] == title
|
||||
assert r["brand_code"] == brand_code
|
||||
assert r["disc_id"] == disc_id
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# disc-id PREFIX, space-separated / mixed-case / digit-leading
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"filename,folder,artist,title,disc_id",
|
||||
[
|
||||
# Potos: "CODE-#### Artist - Title" (space, not " - ", after disc-id)
|
||||
("RABZ-0001 The Seatbelts & Steve Conte - Call Me, Call Me.zip",
|
||||
"Potos Power Plant Productions",
|
||||
"The Seatbelts & Steve Conte", "Call Me, Call Me", "RABZ-0001"),
|
||||
# Lopenash: mixed-case folder-name prefix
|
||||
("Lopenash-002 - Dexter's Laboratory - Breathe in The Sunshine.mp4", "Lopenash",
|
||||
"Dexter's Laboratory", "Breathe in The Sunshine", "Lopenash-002"),
|
||||
# 7uLpA: digit-leading mixed-case token
|
||||
("7uLpA-0004 - The Beauty of Gemina - Wonders.mp4", "7uLpA_Made",
|
||||
"The Beauty of Gemina", "Wonders", "7uLpA-0004"),
|
||||
],
|
||||
)
|
||||
def test_disc_id_prefix_variants(filename, folder, artist, title, disc_id):
|
||||
r = parse(filename, folder)
|
||||
assert r["artist"] == artist
|
||||
assert r["title"] == title
|
||||
assert r["disc_id"] == disc_id
|
||||
|
||||
|
||||
def test_disc_id_prefix_single_field():
|
||||
# Lopenash medley with no artist/title split -> title only, artist None
|
||||
r = parse("Lopenash-001 - Super Energy Apocalypse.zip", "Lopenash")
|
||||
assert r["artist"] is None
|
||||
assert r["title"] == "Super Energy Apocalypse"
|
||||
assert r["disc_id"] == "Lopenash-001"
|
||||
|
||||
|
||||
def test_disc_id_prefix_drops_leading_track_number():
|
||||
# Mix Tape: "CODE-### - NN - Artist - Title" -> drop the NN track number
|
||||
r = parse("MIX001 - 02 - Psychostick - Mega Man.mp4", "Mix Tape Karaoke")
|
||||
assert r["artist"] == "Psychostick"
|
||||
assert r["title"] == "Mega Man"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# disc-id SUFFIX (unconditional strip)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"filename,folder,artist,title,brand_code,disc_id",
|
||||
[
|
||||
("3Blue1Brown - How They Fool Ya - IMBK-0076.mp4", "It Might Be Karaoke",
|
||||
"3Blue1Brown", "How They Fool Ya", "IMBK", "IMBK-0076"),
|
||||
("54-40 - Nice to Luv you - SOKC-0036 (KARAOKE).zip", "Steve-O Karaoke Cult",
|
||||
"54-40", "Nice to Luv you", "SOKC", "SOKC-0036"),
|
||||
("Against Me - Wagon Wheel - SOKC-0306 - (KARAOKE).zip", "Steve-O Karaoke Cult",
|
||||
"Against Me", "Wagon Wheel", "SOKC", "SOKC-0306"),
|
||||
# disc-id suffix with no dash before the code
|
||||
("Age Of Electric - Remote Control SOKC-0042 (KARAOKE).zip", "Steve-O Karaoke Cult",
|
||||
"Age Of Electric", "Remote Control", "SOKC", "SOKC-0042"),
|
||||
# ATG: code with no dash inside (ATG0014)
|
||||
("Angelcorpse - Wolflust - ATG0014.zip", "ATG Karaoke - Zipped MP3+G",
|
||||
"Angelcorpse", "Wolflust", "ATG", "ATG0014"),
|
||||
("1349 - Singer Of Strange Songs - ATG0072.zip", "ATG Karaoke - Zipped MP3+G",
|
||||
"1349", "Singer Of Strange Songs", "ATG", "ATG0072"),
|
||||
],
|
||||
)
|
||||
def test_disc_id_suffix(filename, folder, artist, title, brand_code, disc_id):
|
||||
r = parse(filename, folder)
|
||||
assert r["artist"] == artist
|
||||
assert r["title"] == title
|
||||
assert r["brand_code"] == brand_code
|
||||
assert r["disc_id"] == disc_id
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# bracket / paren brand SUFFIX (gated on folder match)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"filename,folder,artist,title",
|
||||
[
|
||||
("3rd Bass - Green Eggs And Swine [DJ Sauly Karaoke].zip", "DJ Sauly Collection",
|
||||
"3rd Bass", "Green Eggs And Swine"),
|
||||
# keep the legit (Banda Remix), strip only the [DJ Sauly Karaoke] tag
|
||||
("50 Cent - In Da Club (Banda Remix) [DJ Sauly Karaoke].zip", "DJ Sauly Collection",
|
||||
"50 Cent", "In Da Club (Banda Remix)"),
|
||||
("10 Years - Fix Me (BellySings).mp4", "BellySings Karaoke",
|
||||
"10 Years", "Fix Me"),
|
||||
("Bad Religion - Fuck You (popeoke).zip", "Popeoke",
|
||||
"Bad Religion", "Fuck You"),
|
||||
("B.E.R. - The Night Begins to Shine [popeoke].zip", "Popeoke",
|
||||
"B.E.R.", "The Night Begins to Shine"),
|
||||
],
|
||||
)
|
||||
def test_brand_suffix_bracket_or_paren(filename, folder, artist, title):
|
||||
r = parse(filename, folder)
|
||||
assert r["artist"] == artist
|
||||
assert r["title"] == title
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# dash brand SUFFIX (gated on folder match / per-folder token)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"filename,folder,artist,title",
|
||||
[
|
||||
("100 Gecs - Last Train To Awesometown - Rock Solid Karaoke.zip",
|
||||
"Rock Solid Karaoke CDG files", "100 Gecs", "Last Train To Awesometown"),
|
||||
("311 - Amber - Rock Solid Karaoke.zip",
|
||||
"Rock Solid Karaoke CDG files", "311", "Amber"),
|
||||
("Akon - Right Now (Na Na Na) - EBK.avi", "ForteKaraoke Discord",
|
||||
"Akon", "Right Now (Na Na Na)"),
|
||||
],
|
||||
)
|
||||
def test_brand_suffix_dash(filename, folder, artist, title):
|
||||
r = parse(filename, folder)
|
||||
assert r["artist"] == artist
|
||||
assert r["title"] == title
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# code / folder-name PREFIX (no number)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"filename,folder,artist,title,brand_code",
|
||||
[
|
||||
("(SDK) (G)I-DLE - Queencard.cdg", "Sandell Karaoke",
|
||||
"(G)I-DLE", "Queencard", "SDK"),
|
||||
("(SDK) 21 Pilots - House Of Gold.cdg", "Sandell Karaoke",
|
||||
"21 Pilots", "House Of Gold", "SDK"),
|
||||
("CKK - Buoys, The - Timothy.zip", "Cereal Killer Karaoke",
|
||||
"Buoys, The", "Timothy", "CKK"),
|
||||
("CKK - A Day To Remember - If It Means A Lot To You.zip", "Cereal Killer Karaoke",
|
||||
"A Day To Remember", "If It Means A Lot To You", "CKK"),
|
||||
("TNSK - Davy Jones - Girl.zip", "The Nerdy Singer Karaoke",
|
||||
"Davy Jones", "Girl", "TNS"),
|
||||
("Crossfire - Eurythmics - Missionary Man.mp4", "Crossfire Karaoke",
|
||||
"Eurythmics", "Missionary Man", None),
|
||||
],
|
||||
)
|
||||
def test_code_or_name_prefix(filename, folder, artist, title, brand_code):
|
||||
r = parse(filename, folder)
|
||||
assert r["artist"] == artist
|
||||
assert r["title"] == title
|
||||
assert r["brand_code"] == brand_code
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# generic karaoke tag strip (display title, both () and [])
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"filename,folder,artist,title",
|
||||
[
|
||||
("Bill Murray - Star Wars [Karaoke].mp4", "Mobile Karaoke Unit",
|
||||
"Bill Murray", "Star Wars"),
|
||||
("3 Doors Down - Down Poison (Karaoke HD).zip", "Funbox Karaoke",
|
||||
"3 Doors Down", "Down Poison"),
|
||||
("Islands - Pumpkin (Karaoke Version).mp4", "Matt Joy Karaoke",
|
||||
"Islands", "Pumpkin"),
|
||||
],
|
||||
)
|
||||
def test_generic_karaoke_tag_stripped_from_title(filename, folder, artist, title):
|
||||
r = parse(filename, folder)
|
||||
assert r["artist"] == artist
|
||||
assert r["title"] == title
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# clean "Artist - Title" — only the curated brand_code map applies
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"filename,folder,artist,title,brand_code",
|
||||
[
|
||||
("A Tribe Called Quest - Lyrics To Go.avi", "WIZZYOKE",
|
||||
"A Tribe Called Quest", "Lyrics To Go", "WIZZY"),
|
||||
("100 gecs - Hollywood Baby.cdg", "Funbox Karaoke",
|
||||
"100 gecs", "Hollywood Baby", "FBK"),
|
||||
("Alice in Chains - Angry Chair.mp4", "Brilliant Trash Karaoke",
|
||||
"Alice in Chains", "Angry Chair", "BTRASH"),
|
||||
("16 Horsepower - American Wheeze.cdg", "Monster Tracks",
|
||||
"16 Horsepower", "American Wheeze", "MTRAX"),
|
||||
],
|
||||
)
|
||||
def test_clean_artist_title_with_curated_code(filename, folder, artist, title, brand_code):
|
||||
r = parse(filename, folder)
|
||||
assert r["artist"] == artist
|
||||
assert r["title"] == title
|
||||
assert r["brand_code"] == brand_code
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# GUARDS — must NOT over-strip real song text
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
def test_guard_keeps_non_brand_bracket_in_title():
|
||||
# [Nuxx] and (RVZN Edit) are NOT the folder brand -> keep them
|
||||
r = parse("RVZ-016 - UNDERWORLD - Born Slippy [Nuxx] (RVZN Edit).mp4", "R3V1Z10N Karaoke")
|
||||
assert r["artist"] == "UNDERWORLD"
|
||||
assert r["title"] == "Born Slippy [Nuxx] (RVZN Edit)"
|
||||
assert r["brand_code"] == "RVZ"
|
||||
|
||||
|
||||
def test_guard_keeps_paren_artist():
|
||||
r = parse("(SDK) (G)I-DLE - Queencard.cdg", "Sandell Karaoke")
|
||||
assert r["artist"] == "(G)I-DLE" # only the (SDK) brand prefix removed
|
||||
|
||||
|
||||
def test_guard_keeps_live_qualifier_in_title():
|
||||
# A trailing (Live) that is not the brand must survive (folder doesn't match)
|
||||
r = parse("311 - All Re-Mixed Up (live) - Rock Solid Karaoke.zip",
|
||||
"Rock Solid Karaoke CDG files")
|
||||
assert r["artist"] == "311"
|
||||
assert r["title"] == "All Re-Mixed Up (live)"
|
||||
|
||||
|
||||
def test_guard_numeric_artist_preserved():
|
||||
r = parse("311 - Amber - Rock Solid Karaoke.zip", "Rock Solid Karaoke CDG files")
|
||||
assert r["artist"] == "311"
|
||||
|
||||
|
||||
def test_guard_alphanumeric_artist_not_eaten_as_disc_id():
|
||||
# "UB40", "D12" are uppercase+digits but NOT disc-shaped (no hyphen, <3 digit
|
||||
# number) -> must stay the artist, not be stripped as a disc-id prefix.
|
||||
r = parse("UB40 - Kingston Town (BellySings).mp4", "BellySings Karaoke")
|
||||
assert r["artist"] == "UB40"
|
||||
assert r["title"] == "Kingston Town"
|
||||
r2 = parse("D12 - My Band.mp4", "Funbox Karaoke")
|
||||
assert r2["artist"] == "D12"
|
||||
assert r2["title"] == "My Band"
|
||||
|
||||
|
||||
def test_en_dash_separator():
|
||||
r = parse("Iron Maiden – The Writing On The Wall.mp4", "ATG Karaoke - Zipped MP3+G")
|
||||
assert r["artist"] == "Iron Maiden"
|
||||
assert r["title"] == "The Writing On The Wall"
|
||||
|
||||
|
||||
def test_strips_other_brand_karaoke_tag_in_title():
|
||||
# Funbox re-hosts WTF Karaoke files tagged "- (WTF Karaoke)"; the tag mentions
|
||||
# karaoke so it is stripped even though it isn't the folder brand.
|
||||
r = parse("Alice in Chains - Heaven Beside You - (WTF Karaoke).zip", "Funbox Karaoke")
|
||||
assert r["artist"] == "Alice in Chains"
|
||||
assert r["title"] == "Heaven Beside You"
|
||||
|
||||
|
||||
def test_leading_dash_hugging_artist_cleaned():
|
||||
r = parse("7uLpA-0013 -Zynic - After Effects.mp4", "7uLpA_Made")
|
||||
assert r["artist"] == "Zynic"
|
||||
assert r["title"] == "After Effects"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# normalized fields flow from parsed artist/title (xref keys)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
def test_normalized_fields_use_cleaned_title():
|
||||
r = parse("3rd Bass - Green Eggs And Swine [DJ Sauly Karaoke].zip", "DJ Sauly Collection")
|
||||
assert fp.normalize_for_search(r["artist"]) == "3rd bass"
|
||||
assert fp.normalize_for_search(r["title"]) == "green eggs and swine"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# existing helpers still behave
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
def test_should_index_file():
|
||||
assert fp.should_index_file("x.mp4") is True
|
||||
assert fp.should_index_file("x.zip") is True
|
||||
assert fp.should_index_file(".DS_Store") is False
|
||||
assert fp.should_index_file("notes.txt") is False
|
||||
|
||||
|
||||
def test_detect_format():
|
||||
assert fp.detect_format("x.zip") == "zip"
|
||||
assert fp.detect_format("x.MP4") == "mp4"
|
||||
assert fp.detect_format("x.cdg") == "cdg"
|
||||
Reference in new issue
Block a user