Benchmarks & Model Cards
Each guardrail's API page carries a Benchmarks section pairing its capability metadata with measured or recycled benchmark numbers, and a License section — so you can pick between guardrails on evidence, not descriptions.
The numbers live in an import-free registry (any_guardrail.benchmark_registry), exported to schemas/guardrail_benchmarks.json and validated against schemas/guardrail_benchmarks.schema.json. The model-heavy harness that produces them lives in the unshipped benchmarks/ package and never runs in CI.
Comparability is machine-enforced
Every score carries a ComparisonCohort. Two scores are comparable only if their cohort is equal — same dataset revision, label mapping, metric, threshold policy, and harness. The renderer groups by cohort and never aligns scores across cohorts, so ToxicChat 1123 vs 0124, Optimal-F1 vs F1@0.5, or an AUC in an F1 column can't silently become one ranked column. A missing score is None and renders as —, never 0; every number carries provenance.
from any_guardrail import BenchmarkResult, BenchmarkSource, ComparisonCohort
result = BenchmarkResult(
guardrail="deepset",
category="prompt_injection",
value=0.91,
source=BenchmarkSource(kind="published", url="https://huggingface.co/deepset/deberta-v3-base-injection"),
cohort=ComparisonCohort(
dataset="deepset-prompt-injections",
dataset_revision="test",
label_mapping="injection=positive",
metric="f1",
threshold_policy="f1@0.5",
harness="published:model-card",
),
contamination=True, # deepset trained on this dataset
)
assert result.value == 0.91
assert result.source.kind == "published"Adding numbers
Harvest published numbers (tag published:<url>) or run the harness on pinned hardware (tag measured:<harness-version>), append BenchmarkResult(...) entries to src/any_guardrail/_benchmark_data.py, then regenerate:
See benchmarks/README.md for the methodology, per-dataset license/access table, and the one-time legal check on NC datasets.
Last updated