Files
pyannote-audio/tutorials/multi_objective_optimization.ipynb
T

10 KiB

Multi-objective optimization of Community-1

Speaker diarization purity and coverage usually conflict: splitting speakers into more clusters tends to improve purity, while merging clusters tends to improve coverage. This tutorial uses pyannote-audio optimize to tune pyannote/speaker-diarization-community-1 for both metrics and retrieves the Pareto front—the operating points for which neither metric can improve without degrading the other. It then exports configurations for the best purity, best coverage, and best harmonic mean of both.

Install pyannote.audio with its cli extra in the notebook's Python environment. You need a Hugging Face token with access to Community-1 (log in beforehand or set HF_TOKEN) and a pyannote.database protocol with reference annotations.

In [ ]:
from copy import deepcopy
from pathlib import Path
import subprocess
import sys

import matplotlib.pyplot as plt
import yaml
from huggingface_hub import snapshot_download
from pyannote.audio import Pipeline
from pyannote.audio.core.pipeline import expand_subfolders

Prepare Community-1 and the development protocol

Download Community-1 into a writable local directory, including its model subfolders. The CLI takes a local pipeline configuration and writes the optimization journal and results beside it. If you already have a complete local copy, set pipeline_config to its config.yaml and skip the download.

In [ ]:
model_directory = Path("community1").resolve()
snapshot_download(
    "pyannote/speaker-diarization-community-1",
    local_dir=model_directory,
)
pipeline_config = model_directory / "config.yaml"

Replace the database configuration and protocol names below. Optimize on the development set; reserve the final test set for evaluation.

In [ ]:
database_registry = Path("/path/to/database.yml")
protocol_name = "MyDatabase.SpeakerDiarization.MyProtocol"
subset = "development"
metrics = ["DiarizationPurity", "DiarizationCoverage"]
max_iterations = 100

Optimize both metrics with the CLI

Repeat --metric to define a multi-objective study. The command below runs pyannote-audio optimize using the notebook's Python environment. The CLI handles metric evaluation and device selection and keeps updating a *.pareto.yaml manifest containing every non-dominated trial, its named metric values, and its pipeline parameters.

--average-case uses purity and coverage accumulated over the whole development set. Omit it to optimize the lower 90% confidence bound of each metric instead. The journal lets an interrupted run resume and allows multiple optimization processes to share the study. Re-run with the same settings to continue the study; use a separate pipeline directory when changing evaluation settings.

Other objectives can also be combined using repeated --metric options. For example, --metric DiarizationSpeakerCountError --average-case minimizes the mean absolute difference between predicted and reference speaker counts, weighting files equally and counting only speakers in annotated evaluation regions. This objective is measured in speakers, not percent.

In [ ]:
command = [
    sys.executable, "-m", "pyannote.audio", "optimize",
    str(pipeline_config), protocol_name,
    "--registry", str(database_registry),
    "--subset", subset,
    "--average-case",
    "--max-iterations", str(max_iterations),
]
for metric in metrics:
    command.extend(["--metric", metric])

subprocess.run(command, check=True)

Read and plot the Pareto front

The manifest filename includes the protocol, subset, and metrics in command-line order. Each entry contains the trial number, a dictionary of named metric values, and the corresponding nested Community-1 parameters.

In [ ]:
study_name = f"{protocol_name}.{subset}.{'+'.join(metrics)}"
manifest_path = pipeline_config.with_suffix(f".{study_name}.pareto.yaml")
manifest = yaml.safe_load(manifest_path.read_text())
pareto_front = sorted(
    manifest["optimization"]["status"]["pareto_front"],
    key=lambda point: point["values"]["DiarizationPurity"],
)
if not pareto_front:
    raise ValueError("No Pareto trials are available. Run optimization first.")
pareto_front
In [ ]:
purity = [point["values"]["DiarizationPurity"] for point in pareto_front]
coverage = [point["values"]["DiarizationCoverage"] for point in pareto_front]

fig, ax = plt.subplots()
ax.plot(purity, coverage, "o-")
ax.set(xlabel="Diarization purity", ylabel="Diarization coverage")
ax.grid(True)

Select the best purity, coverage, and harmonic mean

A multi-objective study has no single best configuration. Select three operating points from the Pareto front: maximum purity, maximum coverage, and maximum harmonic mean, 2 * purity * coverage / (purity + coverage). Define the harmonic mean as zero when both metrics are zero. These selections can refer to the same trial.

For tied purity or coverage, prefer the higher other metric. For tied harmonic means, prefer higher purity, then coverage. Finally, prefer the earlier trial. With --average-case, these are development-set scores; otherwise, the selection uses the optimized confidence bounds.

In [ ]:
def harmonic_mean(point):
    purity = point["values"]["DiarizationPurity"]
    coverage = point["values"]["DiarizationCoverage"]
    return 2 * purity * coverage / (purity + coverage) if purity + coverage else 0.0


def rank(point, criterion):
    purity = point["values"]["DiarizationPurity"]
    coverage = point["values"]["DiarizationCoverage"]
    score = {
        "purity": purity,
        "coverage": coverage,
        "harmonic_mean": harmonic_mean(point),
    }[criterion]
    return score, purity, coverage, -point["trial"]


selected_points = {
    criterion: max(pareto_front, key=lambda point: rank(point, criterion))
    for criterion in ("purity", "coverage", "harmonic_mean")
}
for criterion, point in selected_points.items():
    print(
        f"Best {criterion}: trial {point['trial']}, "
        f"purity={point['values']['DiarizationPurity']:.3f}, "
        f"coverage={point['values']['DiarizationCoverage']:.3f}, "
        f"harmonic mean={harmonic_mean(point):.3f}"
    )

Return and save the three pipeline configurations

Each configuration includes the pipeline definition, the selected parameters, and optimization provenance. Resolve model subfolders against the original model directory so the configurations can be loaded from dictionaries or YAML files elsewhere on this machine. The model files must remain available at that location.

best_configs returns the three configuration dictionaries. The corresponding config.best_purity.yaml, config.best_coverage.yaml, and config.best_harmonic_mean.yaml files are saved beside the original configuration. Load one with, for example, Pipeline.from_pretrained(best_config_paths["harmonic_mean"]).

In [ ]:
best_configs = {}
best_config_paths = {}
for criterion, point in selected_points.items():
    config = deepcopy(manifest)
    config["params"] = deepcopy(point["params"])
    config["optimization"] = {
        "protocol": protocol_name,
        "subset": subset,
        "metrics": metrics.copy(),
        "selection": criterion,
        "trial": point["trial"],
        "values": deepcopy(point["values"]),
        "harmonic_mean": harmonic_mean(point),
        "source": str(manifest_path.resolve()),
    }
    expand_subfolders(config, str(manifest_path.parent.resolve()))
    best_configs[criterion] = config
    config_path = pipeline_config.with_suffix(f".best_{criterion}.yaml")
    config_path.write_text(yaml.safe_dump(config, sort_keys=False))
    best_config_paths[criterion] = config_path

best_configs