Files
pyannote-audio/tutorials/multi_objective_optimization.ipynb

253 lines
10 KiB
Plaintext

{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Multi-objective optimization of Community-1\n",
"\n",
"Speaker diarization purity and coverage usually conflict: splitting speakers into more clusters tends to improve purity, while merging clusters tends to improve coverage. This tutorial uses `pyannote-audio optimize` to tune `pyannote/speaker-diarization-community-1` for both metrics and retrieves the Pareto front—the operating points for which neither metric can improve without degrading the other. It then exports configurations for the best purity, best coverage, and best harmonic mean of both.\n",
"\n",
"Install `pyannote.audio` with its `cli` extra in the notebook's Python environment. You need a Hugging Face token with access to Community-1 (log in beforehand or set `HF_TOKEN`) and a [`pyannote.database`](https://github.com/pyannote/pyannote-database) protocol with reference annotations."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from copy import deepcopy\n",
"from pathlib import Path\n",
"import subprocess\n",
"import sys\n",
"\n",
"import matplotlib.pyplot as plt\n",
"import yaml\n",
"from huggingface_hub import snapshot_download\n",
"from pyannote.audio import Pipeline\n",
"from pyannote.audio.core.pipeline import expand_subfolders"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Prepare Community-1 and the development protocol\n",
"\n",
"Download Community-1 into a writable local directory, including its model subfolders. The CLI takes a local pipeline configuration and writes the optimization journal and results beside it. If you already have a complete local copy, set `pipeline_config` to its `config.yaml` and skip the download."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model_directory = Path(\"community1\").resolve()\n",
"snapshot_download(\n",
" \"pyannote/speaker-diarization-community-1\",\n",
" local_dir=model_directory,\n",
")\n",
"pipeline_config = model_directory / \"config.yaml\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Replace the database configuration and protocol names below. Optimize on the development set; reserve the final test set for evaluation."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"database_registry = Path(\"/path/to/database.yml\")\n",
"protocol_name = \"MyDatabase.SpeakerDiarization.MyProtocol\"\n",
"subset = \"development\"\n",
"metrics = [\"DiarizationPurity\", \"DiarizationCoverage\"]\n",
"max_iterations = 100"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Optimize both metrics with the CLI\n",
"\n",
"Repeat `--metric` to define a multi-objective study. The command below runs `pyannote-audio optimize` using the notebook's Python environment. The CLI handles metric evaluation and device selection and keeps updating a `*.pareto.yaml` manifest containing every non-dominated trial, its named metric values, and its pipeline parameters.\n",
"\n",
"`--average-case` uses purity and coverage accumulated over the whole development set. Omit it to optimize the lower 90% confidence bound of each metric instead. The journal lets an interrupted run resume and allows multiple optimization processes to share the study. Re-run with the same settings to continue the study; use a separate pipeline directory when changing evaluation settings.\n",
"\n",
"Other objectives can also be combined using repeated `--metric` options. For example, `--metric DiarizationSpeakerCountError --average-case` minimizes the mean absolute difference between predicted and reference speaker counts, weighting files equally and counting only speakers in annotated evaluation regions. This objective is measured in speakers, not percent."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"command = [\n",
" sys.executable, \"-m\", \"pyannote.audio\", \"optimize\",\n",
" str(pipeline_config), protocol_name,\n",
" \"--registry\", str(database_registry),\n",
" \"--subset\", subset,\n",
" \"--average-case\",\n",
" \"--max-iterations\", str(max_iterations),\n",
"]\n",
"for metric in metrics:\n",
" command.extend([\"--metric\", metric])\n",
"\n",
"subprocess.run(command, check=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Read and plot the Pareto front\n",
"\n",
"The manifest filename includes the protocol, subset, and metrics in command-line order. Each entry contains the trial number, a dictionary of named metric values, and the corresponding nested Community-1 parameters."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"study_name = f\"{protocol_name}.{subset}.{'+'.join(metrics)}\"\n",
"manifest_path = pipeline_config.with_suffix(f\".{study_name}.pareto.yaml\")\n",
"manifest = yaml.safe_load(manifest_path.read_text())\n",
"pareto_front = sorted(\n",
" manifest[\"optimization\"][\"status\"][\"pareto_front\"],\n",
" key=lambda point: point[\"values\"][\"DiarizationPurity\"],\n",
")\n",
"if not pareto_front:\n",
" raise ValueError(\"No Pareto trials are available. Run optimization first.\")\n",
"pareto_front"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"purity = [point[\"values\"][\"DiarizationPurity\"] for point in pareto_front]\n",
"coverage = [point[\"values\"][\"DiarizationCoverage\"] for point in pareto_front]\n",
"\n",
"fig, ax = plt.subplots()\n",
"ax.plot(purity, coverage, \"o-\")\n",
"ax.set(xlabel=\"Diarization purity\", ylabel=\"Diarization coverage\")\n",
"ax.grid(True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Select the best purity, coverage, and harmonic mean\n",
"\n",
"A multi-objective study has no single best configuration. Select three operating points from the Pareto front: maximum purity, maximum coverage, and maximum harmonic mean, `2 * purity * coverage / (purity + coverage)`. Define the harmonic mean as zero when both metrics are zero. These selections can refer to the same trial.\n",
"\n",
"For tied purity or coverage, prefer the higher other metric. For tied harmonic means, prefer higher purity, then coverage. Finally, prefer the earlier trial. With `--average-case`, these are development-set scores; otherwise, the selection uses the optimized confidence bounds."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def harmonic_mean(point):\n",
" purity = point[\"values\"][\"DiarizationPurity\"]\n",
" coverage = point[\"values\"][\"DiarizationCoverage\"]\n",
" return 2 * purity * coverage / (purity + coverage) if purity + coverage else 0.0\n",
"\n",
"\n",
"def rank(point, criterion):\n",
" purity = point[\"values\"][\"DiarizationPurity\"]\n",
" coverage = point[\"values\"][\"DiarizationCoverage\"]\n",
" score = {\n",
" \"purity\": purity,\n",
" \"coverage\": coverage,\n",
" \"harmonic_mean\": harmonic_mean(point),\n",
" }[criterion]\n",
" return score, purity, coverage, -point[\"trial\"]\n",
"\n",
"\n",
"selected_points = {\n",
" criterion: max(pareto_front, key=lambda point: rank(point, criterion))\n",
" for criterion in (\"purity\", \"coverage\", \"harmonic_mean\")\n",
"}\n",
"for criterion, point in selected_points.items():\n",
" print(\n",
" f\"Best {criterion}: trial {point['trial']}, \"\n",
" f\"purity={point['values']['DiarizationPurity']:.3f}, \"\n",
" f\"coverage={point['values']['DiarizationCoverage']:.3f}, \"\n",
" f\"harmonic mean={harmonic_mean(point):.3f}\"\n",
" )"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Return and save the three pipeline configurations\n",
"\n",
"Each configuration includes the pipeline definition, the selected parameters, and optimization provenance. Resolve model subfolders against the original model directory so the configurations can be loaded from dictionaries or YAML files elsewhere on this machine. The model files must remain available at that location.\n",
"\n",
"`best_configs` returns the three configuration dictionaries. The corresponding `config.best_purity.yaml`, `config.best_coverage.yaml`, and `config.best_harmonic_mean.yaml` files are saved beside the original configuration. Load one with, for example, `Pipeline.from_pretrained(best_config_paths[\"harmonic_mean\"])`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"best_configs = {}\n",
"best_config_paths = {}\n",
"for criterion, point in selected_points.items():\n",
" config = deepcopy(manifest)\n",
" config[\"params\"] = deepcopy(point[\"params\"])\n",
" config[\"optimization\"] = {\n",
" \"protocol\": protocol_name,\n",
" \"subset\": subset,\n",
" \"metrics\": metrics.copy(),\n",
" \"selection\": criterion,\n",
" \"trial\": point[\"trial\"],\n",
" \"values\": deepcopy(point[\"values\"]),\n",
" \"harmonic_mean\": harmonic_mean(point),\n",
" \"source\": str(manifest_path.resolve()),\n",
" }\n",
" expand_subfolders(config, str(manifest_path.parent.resolve()))\n",
" best_configs[criterion] = config\n",
" config_path = pipeline_config.with_suffix(f\".best_{criterion}.yaml\")\n",
" config_path.write_text(yaml.safe_dump(config, sort_keys=False))\n",
" best_config_paths[criterion] = config_path\n",
"\n",
"best_configs"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python",
"version": "3.10"
}
},
"nbformat": 4,
"nbformat_minor": 5
}