Skip to content

Guardrail Benchmark

A reproducible benchmark that scores LLM safety classifiers against the OWASP LLM Top 10 (2025) plus content-harm and out-of-scope categories — in English and Hebrew. It ships as a latent flow (@flow("guardrails")) in evals/guardrails/, so the headline number is something you can re-run, not a marketing claim.

latent run guardrails        # load cases → classify per provider → tiered report

What it measures

624 provenance-tagged cases (258 EN + 366 HE) labelled SAFE or VIOLATION across OWASP categories LLM01 (prompt injection), LLM02, LLM06, LLM07, LLM09, LLM10, plus HARM (content hazards) and NONE (benign). Each provider classifies every case; accuracy is reported with Wilson 95% confidence intervals.

Three reporting tiers

The suite separates honest signal from inflated signal:

Tier What it is Counts toward headline?
primary Human-authored + public benchmarks (AdvBench, HarmBench, JailbreakBench, Lakera Mosscap/Gandalf, deepset) and their Hebrew translations ✅ the headline
synthetic LLM-generated attacks — frontier evaluators share semantic priors with the generators (+8–12 pt inflation), so reported separately ❌
refusal Out-of-OWASP adult content — SAFE under the policy by design, reported separately rather than scored as failures ❌

Latest results — primary tier

From run 20260615T151417Z (455 primary cases). Δ = EN − HE (the Hebrew gap); $/1k is cost per 1,000 classifications.

Provider EN HE Δ EN 95% CI HE 95% CI $/1k
gemini-3.1-flash-lite 93.1% 94.0% −0.8pt [88%, 96%] [91%, 96%] $0.06
gpt-5-mini 90.2% 92.9% −2.7pt [85%, 94%] [89%, 95%] $0.75
claude-haiku-4-5 86.2% 91.1% −4.9pt [80%, 91%] [87%, 94%] $1.21
qwen3.6-35b-a3b 86.2% 90.0% −3.8pt [80%, 91%] [86%, 93%] $0.39
gpt-oss-safeguard-20b 82.8% 85.5% −2.7pt [76%, 88%] [81%, 89%] $0.37
llama-guard-4-12b 59.2% 69.0% −9.8pt [52%, 66%] [63%, 74%] $0.06

Key finding

A general small/fast model with a good policy prompt — Gemini 3.1 Flash Lite (~93% EN / ~94% HE) — beats purpose-built safety models on OWASP-style threats in both languages, at a fraction of the cost. Purpose-built classifiers are brittle by comparison: a prior run saw gpt-oss-safeguard-20b collapse to ~53% on Hebrew, and Llama Guard 4's fixed hazard taxonomy is simply out-of-scope for prompt-injection categories.

This is why latent's LLMGuardrailScanner defaults to a small general model + policy prompt rather than a dedicated safety model.

Methodology

  • Dataset construction (scripts/): seed from public datasets → GPT-5.5 EN→HE translation with review → frontier adversarial generation (4 models) → native-Hebrew authoring → hard human-authored seeds. Every case carries a source tag for provenance.
  • Bilingual by design — Hebrew is a first-class language, not an afterthought; the EN−HE gap (Δ) is reported per provider because most safety models degrade on non-English input.
  • Multi-turn — cases may be single prompts or full turns conversations (e.g. a benign opener followed by a system-prompt-extraction turn).
  • Providers registered by env-var name — no secrets in the tree; models are added in providers.yaml / parameters.yaml.
  • Append-only results log — every latent run guardrails appends a timestamped entry to REPORT.md; full per-tier / per-category detail lives in the gitignored results/.

See Also