latent.stats.ppi

Prediction-Powered Inference (FR-2.2 + FR-2.3).

Implements bias-corrected estimates combining calibration data with judge predictions, following the PPI framework.

Reference: Angelopoulos et al. (2023), via ppi_py.ppi_mean_ci.

Functions

ppi_mean

ppi_mean(judge_scores: np.ndarray, calibration_judge: np.ndarray, calibration_human: np.ndarray, confidence_level: float = 0.95, seed: int | None = None) -> MetricResult

Bias-corrected mean estimate using Prediction-Powered Inference.

Reference: Angelopoulos et al. (2023), "Prediction-Powered Inference", via ppi_py.ppi_mean_ci.

PPI formula: theta_ppi = mean(Y_unlabeled) + mean(Y_labeled - Yhat_labeled)

Where: Y_unlabeled = judge scores on the unlabeled data (judge_scores) Y_labeled = human labels on the calibration set (calibration_human) Yhat_labeled = judge scores on the calibration set (calibration_judge)

The correction term mean(Y_labeled - Yhat_labeled) adjusts for systematic judge bias estimated from the calibration set.

Args: judge_scores: 1D array of judge scores on unlabeled data. calibration_judge: 1D array of judge scores on calibration set. calibration_human: 1D array of human labels on calibration set. confidence_level: Confidence level for CI (default 0.95). seed: Random seed for reproducibility.

Returns: MetricResult with calibration_status="calibrated".

stratified_ppi

stratified_ppi(judge_scores: np.ndarray, calibration_judge: np.ndarray, calibration_human: np.ndarray, strata: np.ndarray, calibration_strata: np.ndarray, confidence_level: float = 0.95, seed: int | None = None) -> MetricResult

Stratified Prediction-Powered Inference.

Apply PPI within each stratum and combine via weighted average. Produces tighter CIs than unstratified PPI when strata are informative.

Args: judge_scores: 1D array of judge scores on unlabeled data. calibration_judge: 1D array of judge scores on calibration set. calibration_human: 1D array of human labels on calibration set. strata: 1D array of stratum assignments for unlabeled data. calibration_strata: 1D array of stratum assignments for calibration data. confidence_level: Confidence level for CI (default 0.95). seed: Random seed for reproducibility.

Returns: MetricResult with calibration_status="calibrated_stratified".

uncalibrated_estimate

uncalibrated_estimate(scores: np.ndarray, confidence_level: float = 0.95, seed: int | None = None) -> MetricResult

Fallback estimate when no calibration data is available.

Computes a simple bootstrap CI on the raw scores with calibration_status="uncalibrated" to indicate no bias correction.

Args: scores: 1D array of judge scores. confidence_level: Confidence level for CI (default 0.95). seed: Random seed for reproducibility.

Returns: MetricResult with calibration_status="uncalibrated".