> For the complete documentation index, see [llms.txt](https://docs.mozilla.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.mozilla.ai/any-guardrail/api-reference/index/general-judge/flowjudge.md).

# Flow Judge

Local LLM judge scoring text against user-defined criteria, metrics, and rubrics.

Flow Judge wraps Flow AI's `flow-judge` library and its 3.8B evaluator LLM (`flowaicom/Flow-Judge-v0.1`, fine-tuned from Phi-3.5-mini). Unlike most guardrails it bypasses the `any-guardrail` provider layer and drives `flow_judge` directly, so the `flowjudge` extra must be installed (a top-of-module `ImportError` is re-raised from `__init__` with a `pip install` hint otherwise).

Each guardrail is bound to a single `flow_judge` `Metric` (a criteria string plus a Likert `rubric`). There are two ways to specify it:

* **Convenience fields**: supply `name` / `criteria` / `rubric` / `required_inputs` / `required_output` and a `Metric` is built for you.
* **Prebuilt metric**: pass `metric=` — a `flow_judge` `Metric` / `CustomMetric` or one of the library's presets (e.g. `RESPONSE_FAITHFULNESS_3POINT`). The convenience fields are then ignored.

Verdict mapping onto `GuardrailOutput`:

* `valid` is `rubric_score >= pass_threshold` (or `<=` when `higher_is_better=False`).
* `score` (canonical risk: higher = riskier) is the Likert score normalized onto \[0, 1] via `normalize_rubric_to_risk`, using the rubric's integer keys as the scale bounds — inverted when higher rubric values mean better.
* `explanation` is the judge's feedback; `extra["rubric_score"]` is the raw integer.
* When the backend returns no score, the output fails closed: `valid=False` with `extra={"parse_failure": True}`.

Inputs follow the `flow_judge` `EvalInput` shape rather than a plain string: `validate(inputs, output)` takes `inputs` — a list of single-key dicts, one per `required_inputs` name (e.g. `[{"query": "..."}, {"context": "..."}]`) — and `output`, a single-key dict for the `required_output` name (e.g. `{"response": "..."}`). The keys must match the metric's declared inputs/output.

For more information, see:

* [Flow-Judge-v0.1 model card](https://huggingface.co/flowaicom/Flow-Judge-v0.1)
* [flowaicom/flow-judge on GitHub](https://github.com/flowaicom/flow-judge)
* [Flow Judge overview (Flow AI)](https://flow-ai.com/judge)

Raises: ImportError: When the `flowjudge` extra is not installed. ValueError: When neither `metric` nor the full convenience field set is provided.

## Constructor

| Parameter           | Type              | Required | Default | Description                                                                                                                                                          |
| ------------------- | ----------------- | -------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name`              | \`str             | None\`   | No      | `None`                                                                                                                                                               |
| `criteria`          | \`str             | None\`   | No      | `None`                                                                                                                                                               |
| `rubric`            | \`dict\[int, str] | None\`   | No      | `None`                                                                                                                                                               |
| `required_inputs`   | \`list\[str]      | None\`   | No      | `None`                                                                                                                                                               |
| `required_output`   | \`str             | None\`   | No      | `None`                                                                                                                                                               |
| `pass_threshold`    | `int`             | No       | `3`     | The rubric score at which the response counts as passing. `valid` is `rubric_score >= pass_threshold` (or `<=` when `higher_is_better` is `False`). Defaults to `3`. |
| `higher_is_better`  | `bool`            | No       | `True`  | Whether higher rubric scores mean better responses. Set `False` for rubrics where a higher number is worse. Defaults to `True`.                                      |
| `metric`            | \`Any             | None\`   | No      | `None`                                                                                                                                                               |
| `model`             | \`Any             | None\`   | No      | `None`                                                                                                                                                               |
| `generation_params` | \`dict\[str, Any] | None\`   | No      | `None`                                                                                                                                                               |

Initialize the Flow Judge guardrail.

Provide either a prebuilt `metric=` or the full set of convenience fields (`name` / `criteria` / `rubric` / `required_inputs` / `required_output`); the metric is built at construction time and the `flow_judge` backend is loaded.

## validate

Classifies the desired input and output according to the associated metric provided to the judge.

**Parameters**

| Parameter | Type                   | Required | Default | Description                                                                                                                                                                                                    |
| --------- | ---------------------- | -------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `inputs`  | `list[dict[str, str]]` | Yes      | —       | A list of single-key dictionaries, one per `required_inputs` name, each mapping that input name to its value, e.g. `[{"query": "What is the capital of France?"}, {"context": "France's capital is Paris."}]`. |
| `output`  | `dict[str, str]`       | Yes      | —       | A single-key dictionary mapping the `required_output` name to the text being graded, e.g. `{"response": "The capital of France is Paris."}`.                                                                   |

**Returns:** `GuardrailOutput`

## Benchmarks

### General Judge

| Dataset (rev)              | Metric           | Threshold    | Value    | Harness                | Source                          | Contam. |
| -------------------------- | ---------------- | ------------ | -------- | ---------------------- | ------------------------------- | ------- |
| judgebench (unspecified)   | choice\_accuracy | native-valid | 0.585965 | router\_judge\@c06603a | measured:router\_judge\@c06603a |         |
| llmbar (unspecified)       | choice\_accuracy | native-valid | 0.624561 | router\_judge\@c06603a | measured:router\_judge\@c06603a |         |
| rewardbench2 (unspecified) | choice\_accuracy | native-valid | 0.329825 | router\_judge\@c06603a | measured:router\_judge\@c06603a |         |

### Tool Use

| Dataset (rev)             | Metric   | Threshold    | Value    | Harness                | Source                          | Contam. |
| ------------------------- | -------- | ------------ | -------- | ---------------------- | ------------------------------- | ------- |
| bfcl (unspecified)        | accuracy | native-valid | 0.712281 | router\_judge\@c06603a | measured:router\_judge\@c06603a |         |
| hammerbench (unspecified) | accuracy | native-valid | 0.691228 | router\_judge\@c06603a | measured:router\_judge\@c06603a |         |

## License

* **Vendor:** Flow AI
* **Default license:** `apache-2.0` (of the default model/service)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.mozilla.ai/any-guardrail/api-reference/index/general-judge/flowjudge.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
