> For the complete documentation index, see [llms.txt](https://docs.mozilla.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.mozilla.ai/any-guardrail/api-reference/index/general-judge/prometheus.md).

# Prometheus

Open rubric-based LLM judge grading a response on a user-defined 1-5 rubric.

Prometheus is an open-source decoder LLM specialized in evaluating other models' outputs. This guardrail drives it in **absolute grading** mode: each call wraps the instruction and response in Prometheus's evaluation prompt together with the caller's `rubric` (and an optional `reference_answer` that would score 5), and the model replies with written feedback followed by `[RESULT] <n>` where `n` is an integer 1-5. It runs through `provider.generate_chat`, so it can be served from either a `HuggingFaceProvider` or a `LlamafileProvider`.

Verdict mapping onto `GuardrailOutput`:

* `valid` is `rubric_score >= pass_threshold` (or `<=` when `higher_is_better=False`).
* `score` (canonical risk: higher = riskier) is the 1-5 rubric score normalized onto \[0, 1] via `normalize_rubric_to_risk` — inverted when higher rubric values mean better, so a high-quality response yields a low risk.
* `explanation` is the model's feedback (the text before the `[RESULT]` marker).
* `extra["rubric_score"]` is the raw integer 1-5.
* When no score can be parsed, the output fails closed: `valid=False` with `extra={"parse_failure": True}`. The parser takes the **last** `[RESULT]` marker, because feedback often quotes other rubric levels inline.

Inputs are strings: `input_text` is the instruction and `output_text` is the response being graded (when `output_text` is omitted, `input_text` is graded as the response). `input_text` also accepts a `list[str]` to judge a batch in one real batched `generate_chat` call when the provider is a `HuggingFaceProvider` (`output_text` may then be a matching-length list, a single value broadcast to every item, or omitted); other providers fall back to one call per item.

For more information, see:

* [prometheus-7b-v2.0 model card](https://huggingface.co/prometheus-eval/prometheus-7b-v2.0) (default)
* [prometheus-8x7b-v2.0 model card](https://huggingface.co/prometheus-eval/prometheus-8x7b-v2.0)
* [prometheus-7b-v1.0 model card](https://huggingface.co/prometheus-eval/prometheus-7b-v1.0)
* [prometheus-13b-v1.0 model card](https://huggingface.co/prometheus-eval/prometheus-13b-v1.0)
* [Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models (arXiv:2405.01535)](https://arxiv.org/abs/2405.01535)
* [prometheus-eval/prometheus-eval on GitHub](https://github.com/prometheus-eval/prometheus-eval)

## Supported Models

* `prometheus-eval/prometheus-7b-v2.0`
* `prometheus-eval/prometheus-8x7b-v2.0`
* `prometheus-eval/prometheus-7b-v1.0`
* `prometheus-eval/prometheus-13b-v1.0`

## Constructor

| Parameter          | Type                                                 | Required | Default | Description                                                                                                                                                                                                                                              |
| ------------------ | ---------------------------------------------------- | -------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `rubric`           | `str`                                                | Yes      | —       | The score rubric applied to every `validate` call — the evaluation criteria plus `Score 1:` … `Score 5:` descriptions, e.g. `"Is the answer factually correct? Score 1: entirely wrong. ... Score 5: fully correct."`.                                   |
| `pass_threshold`   | `int`                                                | Yes      | —       | The score (1-5) at or above which the response passes (or at or below when `higher_is_better=False`), e.g. `4`.                                                                                                                                          |
| `reference_answer` | \`str                                                | None\`   | No      | `None`                                                                                                                                                                                                                                                   |
| `higher_is_better` | `bool`                                               | No       | `True`  | Whether higher rubric scores mean better responses. Set `False` for rubrics where a higher number is worse (e.g. a severity scale). Defaults to `True`.                                                                                                  |
| `model_id`         | \`str                                                | None\`   | No      | `None`                                                                                                                                                                                                                                                   |
| `provider`         | `Optional[Provider[dict[str, Any], dict[str, Any]]]` | No       | `None`  | Optional pre-configured provider. When `None`, a `HuggingFaceProvider` is built targeting `AutoModelForCausalLM` / `AutoTokenizer` (transformers is imported lazily here). Pass a `LlamafileProvider` to run a GGUF build without the huggingface extra. |
| `prompt`           | \`PromptTemplate                                     | None\`   | No      | `None`                                                                                                                                                                                                                                                   |
| `prompt_version`   | \`str                                                | None\`   | No      | `None`                                                                                                                                                                                                                                                   |

Initialize the Prometheus guardrail.

## validate

Judge `output_text` (the response) given `input_text` (the instruction).

**Parameters**

| Parameter     | Type  | Required     | Default | Description |
| ------------- | ----- | ------------ | ------- | ----------- |
| `input_text`  | \`str | list\[str]\` | Yes     | —           |
| `output_text` | \`str | list\[str]   | None\`  | No          |

**Returns:** `GuardrailOutput | list[GuardrailOutput]`

## Benchmarks

### General Judge

| Dataset (rev)              | Metric           | Threshold    | Value    | Harness                | Source                          | Contam. |
| -------------------------- | ---------------- | ------------ | -------- | ---------------------- | ------------------------------- | ------- |
| judgebench (unspecified)   | choice\_accuracy | native-valid | 0.557895 | router\_judge\@c06603a | measured:router\_judge\@c06603a |         |
| llmbar (unspecified)       | choice\_accuracy | native-valid | 0.515789 | router\_judge\@c06603a | measured:router\_judge\@c06603a |         |
| rewardbench2 (unspecified) | choice\_accuracy | native-valid | 0.319298 | router\_judge\@c06603a | measured:router\_judge\@c06603a |         |

### Tool Use

| Dataset (rev)             | Metric   | Threshold    | Value    | Harness                | Source                          | Contam. |
| ------------------------- | -------- | ------------ | -------- | ---------------------- | ------------------------------- | ------- |
| bfcl (unspecified)        | accuracy | native-valid | 0.698246 | router\_judge\@c06603a | measured:router\_judge\@c06603a |         |
| hammerbench (unspecified) | accuracy | native-valid | 0.54386  | router\_judge\@c06603a | measured:router\_judge\@c06603a |         |

## License

* **Vendor:** KAIST / prometheus-eval
* **Default license:** `apache-2.0` (of the default model/service)

| Model variant                          | License      |
| -------------------------------------- | ------------ |
| `prometheus-eval/prometheus-13b-v1.0`  | `llama-2`    |
| `prometheus-eval/prometheus-7b-v1.0`   | `llama-2`    |
| `prometheus-eval/prometheus-7b-v2.0`   | `apache-2.0` |
| `prometheus-eval/prometheus-8x7b-v2.0` | `apache-2.0` |


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.mozilla.ai/any-guardrail/api-reference/index/general-judge/prometheus.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
