Judges & Scoring

LLM-as-a-judge turns a language model into a structured grader. A Judge takes a data row, prompts a model with a rubric, and returns a typed, validated score object — not free text — so the result drops straight into statistics, quality gates, and reports.

from typing import Annotated
from latent.agents import Judge
from latent.agents.scores import ScoredModel, OrdinalScore

class Quality(ScoredModel):
    quality: Annotated[int, OrdinalScore(scale=(1, 2, 3, 4, 5), pass_threshold=3)]

judge = Judge("quality", model="gpt-4o", output_type=Quality)
score = await judge.evaluate({"question": "...", "answer": "..."})
print(score.quality, score.quality_rationale)   # -> 4  "polite and on-topic..."

Why a Judge instead of a raw agent?

A Judge[T] is a ReActAgent specialized for evaluation. Compared to prompting a model yourself, it gives you:

• Structured output, validated — the model is forced to return your Pydantic output_type, so you read score.quality: int, never a string you have to parse.
• Auto-generated rubric — annotate fields with score types and the scoring prompt is generated for you (override with prompt_template when you need to).
• Rationales for free — ScoredModel injects a {field}_rationale for every score, so every number ships with the model's reasoning.
• Batch + statistics built in — judges are await-able and .map()-able, and judge_flow runs one (or many) across a dataset with bootstrap confidence intervals, quality gates, and MLflow logging.
• Composable scoring — is_eligible skips rows a judge shouldn't grade, row_mapper reshapes a row before scoring, and column_prefix keeps multiple judges from colliding on shared field names.

Classifier[T] is the same machinery under a semantic name, used when the output model has a prediction field — pair it with classification_flow for accuracy / F1 / precision / recall.

---

Judge and Classifier

Judge[T] scores a data row against a Pydantic output schema. Classifier[T] is a semantic alias with identical behavior, used when the output model has a prediction field.

Construction

from latent.agents import Judge, Classifier

judge = Judge(
    name="qa_judge",
    model="gpt-4o",
    output_type=MyScoreModel,       # Pydantic model (required)
    prompt_template=None,            # auto-generated from score annotations if omitted
    system_prompt=None,              # defaults to "You are an expert evaluator..."
    few_shot=None,                   # Sequence[Message] | None — worked examples replayed before every row (held as a tuple)
    temperature=0.0,
    max_tokens=4096,
    is_eligible=None,                # Callable[[dict], bool] | None — skip rows that don't match a predicate
    row_mapper=None,                 # Callable[[dict], dict] | None — transform a row before scoring
    column_prefix=None,              # str | None — prefix output columns (avoids collisions when scoring with multiple judges)
)

classifier = Classifier(
    name="intent_classifier",
    model="gpt-4o",
    output_type=IntentPrediction,
    prompt_template="Classify this message: {text}\nLabels: {labels}",
)

evaluate(row)

Formats the prompt_template with the row dict, calls the LLM with structured output, and parses the response:

score = await judge.evaluate({"conversation": "...", "question": "..."})
# score is an instance of output_type (e.g. MyScoreModel)

await Judge.__call__(row) delegates to evaluate(), making judges compatible with await task.map(...). Both evaluate() and __call__ are coroutines — there is no synchronous scoring API, so a sync call site adopts a Judge by moving to await (or wrapping in asyncio.run), not by swapping one call for another.

Few-shot examples

Some judgements ride on worked examples more than on the rubric — a fabrication check in a non-English language, say, where the examples pin what counts as "made up". Pass them as few_shot, and they are replayed on every row, after the system message and before the scored row:

from latent.protocol import Message

judge = Judge(
    name="fabrication",
    model="gpt-4o",
    output_type=Fabrication,
    prompt_template="Question: {question}\nAnswer: {answer}",
    few_shot=[
        Message(role="user", content="Question: ...\nAnswer: ..."),
        Message(role="assistant", content='{"fabricated": 1, "fabricated_rationale": "..."}'),
        Message(role="user", content="Question: ...\nAnswer: ..."),
        Message(role="assistant", content='{"fabricated": 0, "fabricated_rationale": "..."}'),
    ],
)

Use alternating user / assistant turns. A system message in few_shot is rejected at construction: it would land first in the message list, which ReActAgent reads as a caller-supplied system message, silently replacing the judge's own system_prompt.

---

ScoredModel

A BaseModel subclass that auto-injects rationale fields. For every field annotated with BinaryScore, OrdinalScore, or ContinuousScore, a corresponding {field}_rationale: str field is created so the LLM can explain its reasoning.

from typing import Annotated
from latent.agents import ScoredModel, OrdinalScore, BinaryScore, ContinuousScore

class QAScores(ScoredModel):
    faithfulness: Annotated[int, OrdinalScore(
        scale=(1, 2, 3, 4, 5),
        pass_threshold=3,
        labels={1: "Hallucinated", 3: "Partial", 5: "Faithful"},
    )]
    relevance: Annotated[int, OrdinalScore(scale=(1, 2, 3, 4, 5))]
    factual: Annotated[int, BinaryScore(description="All claims are factually correct")]
    similarity: Annotated[float, ContinuousScore(min_value=0.0, max_value=1.0)]

This generates four additional fields automatically: faithfulness_rationale, relevance_rationale, factual_rationale, similarity_rationale -- all str with default "".

Score annotations

Annotation	Type hint	Parameters	Use case
BinaryScore	int	description	Pass/fail (0 or 1)
OrdinalScore	int	scale, labels, pass_threshold, description	Likert scales (e.g. 1--5)
ContinuousScore	float	min_value, max_value, description	Float ranges (e.g. 0.0--1.0)

Prompt generation

When prompt_template is omitted, Judge auto-generates a scoring rubric from the annotations using build_scoring_prompt():

An unannotated output_type raises on this path

build_scoring_prompt() returns "" for a model with no BinaryScore / OrdinalScore / ContinuousScore annotation, which would grade the scores against nothing while the run stayed indistinguishable from a real one at report time. Judge therefore raises ValueError at construction rather than warning. Annotate the fields, pass prompt_template= (unaffected), or use Classifier, whose override accepts a plain label with no rubric.

from latent.agents import build_scoring_prompt

prompt = build_scoring_prompt(QAScores)
print(prompt)

Score on the following dimensions:

- faithfulness (1-5): faithfulness
  [1=Hallucinated, 3=Partial, 5=Faithful]
- relevance (1-5): relevance
- factual (0 or 1): All claims are factually correct
- similarity (0.0-1.0): similarity

For each score, provide a brief rationale explaining your reasoning:
- faithfulness_rationale: explain your faithfulness score
- relevance_rationale: explain your relevance score
- factual_rationale: explain your factual score
- similarity_rationale: explain your similarity score

Full example

from typing import Annotated
from latent.agents import Judge, ScoredModel, OrdinalScore

class ResponseQuality(ScoredModel):
    quality: Annotated[int, OrdinalScore(
        scale=(1, 2, 3, 4, 5),
        pass_threshold=3,
        description="Overall response quality",
    )]

judge = Judge("quality_judge", model="gpt-4o", output_type=ResponseQuality)
score = await judge.evaluate({"conversation": "User: Hi\nAssistant: Hello!"})

print(score.quality)              # 4
print(score.quality_rationale)    # "The response is polite and appropriate..."

---

Pre-built judges

The latent.agents.judges module provides ready-to-use judges for common evaluation patterns. Each is a factory function that returns a configured Judge instance.

Judge	Input columns	Score type	What it measures
FaithfulnessJudge	context, response	ContinuousScore(0-1)	Proportion of claims supported by context
HallucinationJudge	context, response	ContinuousScore(0-1)	Hallucination rate
RelevanceJudge	query, response	OrdinalScore(1-5)	Response relevance to query
CompletenessJudge	query, response	OrdinalScore(1-5)	Completeness of the answer
ConcisenessJudge	response	OrdinalScore(1-5)	Brevity without losing information
ConsistencyJudge	response_a, response_b	ContinuousScore(0-1)	Consistency between two responses
StyleJudge	response	OrdinalScore(1-5)	Writing style and tone
InstructionFollowingJudge	instruction, response	OrdinalScore(1-5)	How well instructions were followed
RecoveryJudge	context, response	OrdinalScore(1-5)	Error recovery quality
GuardrailsJudge	response	BinaryScore	Safety/policy compliance

from latent.agents.judges import FaithfulnessJudge, RelevanceJudge

faith_judge = FaithfulnessJudge(model="gpt-4o")
score = await faith_judge.evaluate({
    "context": "The Eiffel Tower is 330 meters tall.",
    "response": "The Eiffel Tower is approximately 330 meters tall.",
})
print(f"Faithfulness: {score.faithfulness:.2f}")

---

Scoring a whole dataset

A single await judge.evaluate(row) scores one row. To score a dataset — with statistics, quality gates, and a rendered report — hand the judge to judge_flow (or classification_flow for a Classifier):

from latent.flows import judge_flow

result = await judge_flow(eval_data=df, judges=judge, gates={"quality": 3.5})
print(result["markdown"])      # statistical report with bootstrap CIs
assert result["all_passed"]    # gate check

judge_flow accepts a single judge or a list of judges (multi-judge scoring). See Eval Flows for the full pipeline and Statistical Analysis for what the numbers mean.

---

See Also

• Eval Flows — run judges across a dataset (judge_flow, classification_flow)
• Statistical Analysis — the statistics behind the scores
• ReAct Agent — the agent class Judge/Classifier extend
• Building Agents — the feature map for the whole stack