> For the complete documentation index, see [llms.txt](https://docs.mozilla.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.mozilla.ai/any-guardrail/api-reference/index/general-judge/compass-judger.md).

# CompassJudger

Generalist LLM judge that scores a response against user-defined criteria and rubric on a 1-10 scale.

Decoder-LLM judge from the OpenCompass evaluation ecosystem. CompassJudger has no canonical pointwise output format, so this guardrail wraps the inputs in a fixed pointwise prompt instructing the model to give a brief justification and then emit its verdict as `Rating: [[X]]` with an integer from 1 to 10. The verdict is the *last* bracketed rating in the generation, so numbers the model quotes while justifying are not mistaken for the final rating.

Inputs are strings: `input_text` is the instruction and `output_text` is the response being judged. When `output_text` is omitted, `input_text` itself is placed in the response slot and judged directly. `input_text` also accepts a `list[str]` to judge a batch in one real batched `generate_chat` call when the provider is a `HuggingFaceProvider` (`output_text` may then be a matching-length list, a single value broadcast to every item, or omitted); other providers fall back to one call per item.

Verdict mapping onto `GuardrailOutput`:

* `valid` is `True` when the rating passes `pass_threshold` (`rating >= pass_threshold` when `higher_is_better`, `<=` otherwise).
* `score` is the rating normalized onto the canonical risk axis in \[0, 1] (higher = riskier), so a high rating under `higher_is_better=True` yields a low risk score.
* `explanation` is the model's full generated justification.
* `extra["rubric_score"]` is the raw 1-10 integer rating.
* Fails closed (`valid=False` with `extra={"parse_failure": True}`) when no in-range rating parses.

For more information, see:

* [CompassJudger-2-7B-Instruct](https://huggingface.co/opencompass/CompassJudger-2-7B-Instruct) (default).
* [CompassJudger-2-32B-Instruct](https://huggingface.co/opencompass/CompassJudger-2-32B-Instruct).
* [CompassJudger-1-1.5B-Instruct](https://huggingface.co/opencompass/CompassJudger-1-1.5B-Instruct).
* [CompassJudger-1-7B-Instruct](https://huggingface.co/opencompass/CompassJudger-1-7B-Instruct).
* [CompassJudger-1-14B-Instruct](https://huggingface.co/opencompass/CompassJudger-1-14B-Instruct).
* [CompassJudger-1-32B-Instruct](https://huggingface.co/opencompass/CompassJudger-1-32B-Instruct).
* [CompassJudger GitHub repository](https://github.com/open-compass/CompassJudger).
* [CompassJudger-2 paper (arXiv:2507.09104)](https://arxiv.org/abs/2507.09104).
* [CompassJudger-1 paper (arXiv:2410.16256)](https://arxiv.org/abs/2410.16256).

## Supported Models

* `opencompass/CompassJudger-2-7B-Instruct`
* `opencompass/CompassJudger-2-32B-Instruct`
* `opencompass/CompassJudger-1-1.5B-Instruct`
* `opencompass/CompassJudger-1-7B-Instruct`
* `opencompass/CompassJudger-1-14B-Instruct`
* `opencompass/CompassJudger-1-32B-Instruct`

## Constructor

| Parameter          | Type                                                 | Required | Default | Description                                                                                                                                                                                                                                                                                                            |
| ------------------ | ---------------------------------------------------- | -------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `criteria`         | `str`                                                | Yes      | —       | A description of what is being judged, e.g. `"Helpfulness of the response to the user's question"`.                                                                                                                                                                                                                    |
| `rubric`           | `str`                                                | Yes      | —       | The scoring-rubric guidance describing what low vs. high ratings mean, e.g. `"1-3: unhelpful ... 8-10: fully answers the question"`.                                                                                                                                                                                   |
| `pass_threshold`   | `int`                                                | Yes      | —       | The 1-10 rating at which the response passes. With `higher_is_better=True`, ratings at or above it yield `valid=True`; with `higher_is_better=False`, ratings at or below it pass.                                                                                                                                     |
| `higher_is_better` | `bool`                                               | No       | `True`  | Whether higher ratings mean better text. Defaults to `True`.                                                                                                                                                                                                                                                           |
| `model_id`         | \`str                                                | None\`   | No      | `None`                                                                                                                                                                                                                                                                                                                 |
| `provider`         | `Optional[Provider[dict[str, Any], dict[str, Any]]]` | No       | `None`  | Optional pre-configured provider (e.g. a `LlamafileProvider` or a customized `HuggingFaceProvider`). Defaults to a `HuggingFaceProvider` loading a causal LM. HuggingFace-backed loads force the SDPA attention kernel because CompassJudger-2's config requests flash\_attention\_2, which is unavailable on CPU/MPS. |
| `prompt`           | \`PromptTemplate                                     | None\`   | No      | `None`                                                                                                                                                                                                                                                                                                                 |
| `prompt_version`   | \`str                                                | None\`   | No      | `None`                                                                                                                                                                                                                                                                                                                 |

Initialize the CompassJudger guardrail.

## validate

Judge `output_text` (the response) given `input_text` (the instruction).

**Parameters**

| Parameter     | Type  | Required     | Default | Description |
| ------------- | ----- | ------------ | ------- | ----------- |
| `input_text`  | \`str | list\[str]\` | Yes     | —           |
| `output_text` | \`str | list\[str]   | None\`  | No          |

**Returns:** `GuardrailOutput | list[GuardrailOutput]`

## Benchmarks

### General Judge

| Dataset (rev)              | Metric           | Threshold    | Value    | Harness                | Source                          | Contam. |
| -------------------------- | ---------------- | ------------ | -------- | ---------------------- | ------------------------------- | ------- |
| judgebench (unspecified)   | choice\_accuracy | native-valid | 0.631579 | router\_judge\@c06603a | measured:router\_judge\@c06603a |         |
| llmbar (unspecified)       | choice\_accuracy | native-valid | 0.740351 | router\_judge\@c06603a | measured:router\_judge\@c06603a |         |
| rewardbench2 (unspecified) | choice\_accuracy | native-valid | 0.484211 | router\_judge\@c06603a | measured:router\_judge\@c06603a |         |

### Tool Use

| Dataset (rev)             | Metric   | Threshold    | Value    | Harness                | Source                          | Contam. |
| ------------------------- | -------- | ------------ | -------- | ---------------------- | ------------------------------- | ------- |
| bfcl (unspecified)        | accuracy | native-valid | 0.912281 | router\_judge\@c06603a | measured:router\_judge\@c06603a |         |
| hammerbench (unspecified) | accuracy | native-valid | 0.621053 | router\_judge\@c06603a | measured:router\_judge\@c06603a |         |

## License

* **Vendor:** OpenCompass
* **Default license:** `apache-2.0` (of the default model/service)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.mozilla.ai/any-guardrail/api-reference/index/general-judge/compass-judger.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
