latent.flows.knowledge_coverage_flow

knowledge_coverage_flow — Two-pass knowledge base sufficiency evaluation.

Functions

knowledge_coverage_flow

knowledge_coverage_flow(eval_data: pd.DataFrame, retriever: Any, expected_column: str = 'expected', question_column: str = 'question', id_column: str | None = None, k: int = 20, model: str = 'claude-sonnet-4-5-20250929', batch_size: int = 5, n_resamples: int = 10000, confidence_level: float = 0.95, seed: int | None = None) -> dict[str, Any]

Two-pass knowledge coverage evaluation.

Pass 1: Extract atomic claims from expected answers. Pass 2: Ground each claim against retrieved chunks. Produces a coverage_score per row (supported_claims / total_claims).

Args: eval_data: DataFrame with question and expected answer columns. retriever: Object with .search(query, k) returning chunks. expected_column: Column with expected answers. question_column: Column with questions (used for retrieval). id_column: Optional ID column. k: Number of chunks to retrieve per question. model: LLM model for claim extraction and grounding. batch_size: Number of items per LLM batch call. n_resamples: Bootstrap resamples. confidence_level: CI confidence level. seed: Random seed.

Returns: Dict with: - coverage_results: list[dict] with per-row coverage_score, claims, grounded_claims. - report: StatisticalReport with coverage_score metric and CI. - markdown: str rendered report. - relevant_chunk_ids: dict[row_id, list[chunk_id]] for supported claims.