latent.rag.metrics

Standalone retrieval evaluation metrics for RAG pipelines.

All functions operate on lists of string IDs and return a float in [0, 1]. Edge cases (empty inputs, no matches) return 0.0.

Classes

ContentMRRMetric

ContentMRRMetric(relevant_content_key: str = 'relevant_content', overlap_threshold: float = 0.5)

MRR using text overlap instead of chunk ID matching.

ContentRecallMetric

ContentRecallMetric(k: int = 5, relevant_content_key: str = 'relevant_content', overlap_threshold: float = 0.5)

Recall@K using text overlap instead of chunk ID matching.

Use when ground truth chunk IDs don't match retrieved chunk IDs (e.g. different chunking configurations).

ContextPrecisionMetric

ContextPrecisionMetric(relevant_ids_key: str = 'relevant_doc_ids')

Wraps :func:context_precision as a :class:RAGMetric.

JudgeMetric

JudgeMetric(judge: Any, score_field: str = 'score', context_separator: str = '\n\n---\n\n')

Delegates scoring to an LLM judge that implements an evaluate method.

MRRMetric

MRRMetric(relevant_ids_key: str = 'relevant_doc_ids')

Wraps :func:mrr as a :class:RAGMetric.

NDCGMetric

NDCGMetric(k: int = 5, relevant_ids_key: str = 'relevant_doc_ids')

Wraps :func:ndcg as a :class:RAGMetric.

PrecisionAtKMetric

PrecisionAtKMetric(k: int = 5, relevant_ids_key: str = 'relevant_doc_ids')

Wraps :func:precision_at_k as a :class:RAGMetric.

RAGMetric

RAGMetric()

Protocol for RAG evaluation metrics.

RecallAtKMetric

RecallAtKMetric(k: int = 5, relevant_ids_key: str = 'relevant_doc_ids')

Wraps :func:recall_at_k as a :class:RAGMetric.

Functions

context_precision

context_precision(retrieved_ids: list[str], relevant_ids: set[str]) -> float

Context Precision — weighted precision by rank, averaged over relevant docs.

For each relevant document at position i, computes precision@(i+1), i.e. the fraction of the top-(i+1) results that are relevant. The score is the mean of these per-hit precision values divided by the number of relevant documents (whether found or not).

Args: retrieved_ids: Ordered list of retrieved document IDs. relevant_ids: Set of ground-truth relevant document IDs.

Returns: Context precision in [0, 1], or 0.0 when relevant_ids is empty or no relevant document appears in retrieved_ids.

mrr

mrr(retrieved_ids: list[str], relevant_ids: set[str]) -> float

Mean Reciprocal Rank — reciprocal of the first relevant result's position.

Args: retrieved_ids: Ordered list of retrieved document IDs. relevant_ids: Set of ground-truth relevant document IDs.

Returns: MRR in [0, 1], or 0.0 when no relevant document is found.

ndcg

ndcg(retrieved_ids: list[str], relevant_ids: set[str], k: int) -> float

Normalized Discounted Cumulative Gain.

Uses binary relevance: gain is 1 for a relevant document, 0 otherwise. DCG = sum of 1 / log2(i + 2) for each relevant document at position i. IDCG is the ideal DCG achieved when all relevant documents appear first.

Args: retrieved_ids: Ordered list of retrieved document IDs. relevant_ids: Set of ground-truth relevant document IDs. k: Number of top results to consider.

Returns: NDCG@k in [0, 1], or 0.0 when relevant_ids is empty.

precision_at_k

precision_at_k(retrieved_ids: list[str], relevant_ids: set[str], k: int) -> float

Fraction of top-k retrieved results that are relevant.

Args: retrieved_ids: Ordered list of retrieved document IDs. relevant_ids: Set of ground-truth relevant document IDs. k: Number of top results to consider.

Returns: Precision@k in [0, 1], or 0.0 when the retrieved list is empty.

recall_at_k

recall_at_k(retrieved_ids: list[str], relevant_ids: set[str], k: int) -> float

Fraction of relevant documents found in the top-k retrieved results.

Args: retrieved_ids: Ordered list of retrieved document IDs. relevant_ids: Set of ground-truth relevant document IDs. k: Number of top results to consider.

Returns: Recall@k in [0, 1], or 0.0 when relevant_ids is empty.

Methods

ContentMRRMetric.score

score(query: str, retrieved: list[RetrievedChunk], reference: str | None = None, eval_row: dict | None = None) -> float

ContentRecallMetric.score

score(query: str, retrieved: list[RetrievedChunk], reference: str | None = None, eval_row: dict | None = None) -> float

ContextPrecisionMetric.score

score(query: str, retrieved: list[RetrievedChunk], reference: str | None = None, eval_row: dict | None = None) -> float

JudgeMetric.score

score(query: str, retrieved: list[RetrievedChunk], reference: str | None = None, eval_row: dict | None = None) -> float

MRRMetric.score

score(query: str, retrieved: list[RetrievedChunk], reference: str | None = None, eval_row: dict | None = None) -> float

NDCGMetric.score

score(query: str, retrieved: list[RetrievedChunk], reference: str | None = None, eval_row: dict | None = None) -> float

PrecisionAtKMetric.score

score(query: str, retrieved: list[RetrievedChunk], reference: str | None = None, eval_row: dict | None = None) -> float

RAGMetric.score

score(query: str, retrieved: list[RetrievedChunk], reference: str | None = None, eval_row: dict | None = None) -> float

RecallAtKMetric.score

score(query: str, retrieved: list[RetrievedChunk], reference: str | None = None, eval_row: dict | None = None) -> float