latent.scores.calibration¶

Calibration scoring: Expected Calibration Error + reliability diagram data.

Classes¶

CalibrationBin¶

CalibrationBin(bin_lower: float, bin_upper: float, mean_confidence: float, mean_accuracy: float, count: int)

A single bin in a reliability diagram.

Functions¶

calibration_report¶

calibration_report(confidences: np.ndarray, correct: np.ndarray, n_bins: int = 10, n_resamples: int = 10000, confidence_level: float = 0.95, seed: int | None = None) -> dict

Compute Expected Calibration Error (ECE) and reliability diagram data.

Args: confidences: Model confidence scores (0-1). correct: Binary correctness labels (0 or 1). n_bins: Number of equal-width bins for the reliability diagram. n_resamples: Bootstrap resamples for ECE CI. confidence_level: CI confidence level. seed: Random seed.

Returns: Dict with keys: - "ece": MetricResult with ECE and bootstrap CI - "bins": list[CalibrationBin] for reliability diagram