> For the complete documentation index, see [llms.txt](https://docs.mozilla.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.mozilla.ai/any-guardrail/api-reference/benchmarks.md).

# Benchmarks & Model Cards

Each guardrail's API page carries a **Benchmarks** section pairing its capability metadata with *measured* or *recycled* benchmark numbers, and a **License** section — so you can pick between guardrails on evidence, not descriptions.

The numbers live in an import-free registry (`any_guardrail.benchmark_registry`), exported to [`schemas/guardrail_benchmarks.json`](https://github.com/mozilla-ai/any-guardrail/blob/main/schemas/guardrail_benchmarks.json) and validated against [`schemas/guardrail_benchmarks.schema.json`](https://github.com/mozilla-ai/any-guardrail/blob/main/schemas/guardrail_benchmarks.schema.json). The model-heavy harness that produces them lives in the unshipped [`benchmarks/`](https://github.com/mozilla-ai/any-guardrail/tree/main/benchmarks) package and never runs in CI.

## Comparability is machine-enforced

Every score carries a `ComparisonCohort`. Two scores are comparable **only if** their cohort is equal — same dataset revision, label mapping, metric, threshold policy, and harness. The renderer groups by cohort and never aligns scores across cohorts, so ToxicChat 1123 vs 0124, Optimal-F1 vs F1\@0.5, or an AUC in an F1 column can't silently become one ranked column. A missing score is `None` and renders as `—`, never `0`; every number carries provenance.

```python
from any_guardrail import BenchmarkResult, BenchmarkSource, ComparisonCohort

result = BenchmarkResult(
    guardrail="deepset",
    category="prompt_injection",
    value=0.91,
    source=BenchmarkSource(kind="published", url="https://huggingface.co/deepset/deberta-v3-base-injection"),
    cohort=ComparisonCohort(
        dataset="deepset-prompt-injections",
        dataset_revision="test",
        label_mapping="injection=positive",
        metric="f1",
        threshold_policy="f1@0.5",
        harness="published:model-card",
    ),
    contamination=True,  # deepset trained on this dataset
)
assert result.value == 0.91
assert result.source.kind == "published"
```

## Adding numbers

Harvest published numbers (tag `published:<url>`) or run the harness on pinned hardware (tag `measured:<harness-version>`), append `BenchmarkResult(...)` entries to `src/any_guardrail/_benchmark_data.py`, then regenerate:

```
python scripts/generate_benchmarks_json.py
python scripts/generate_benchmark_schema.py
```

See [`benchmarks/README.md`](https://github.com/mozilla-ai/any-guardrail/blob/main/benchmarks/README.md) for the methodology, per-dataset license/access table, and the one-time legal check on NC datasets.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.mozilla.ai/any-guardrail/api-reference/benchmarks.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.