> For the complete documentation index, see [llms.txt](https://docs.mozilla.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.mozilla.ai/any-guardrail/api-reference/index/prompt-injection/harm-guard.md).

# HarmAug-Guard

Binary safety and jailbreak classifier, scoring a prompt or prompt-response pair.

HarmAug-Guard is a 435M DeBERTa-v3-large classifier distilled from a much larger (7B+) teacher safety model using the HarmAug data-augmentation method, which jailbreaks an LLM to synthesize harmful instructions for training. It classifies whether an LLM interaction is safe or unsafe and flags jailbreak attempts, reaching an F1 comparable to 7B+ safety models at a fraction of the compute.

Two input shapes are supported through `validate`:

* a single prompt string — judges the prompt on its own; or
* a prompt plus a response (`output_text`) — tokenized as a text pair so the response is judged in the context of the prompt.

Verdict mapping onto `GuardrailOutput`:

* `score` (canonical risk: higher = riskier) is the `unsafe` probability (`0.0` = safe, `1.0` = unsafe), taken from `softmax(logits)[1]` — or from the column the provider labels `unsafe` when label names are available.
* `valid` is `True` when that unsafe probability is below `threshold` (default 0.5, from the paper).
* `categories` carries a `safe` and an `unsafe` entry with their probabilities and `triggered` flags.

For more information, see:

* [HarmAug-Guard model card](https://huggingface.co/hbseong/HarmAug-Guard).
* [HarmAug: Effective Data Augmentation for Knowledge Distillation of Safety Guard Models (arXiv:2410.01524)](https://arxiv.org/abs/2410.01524).

## Supported Models

* `hbseong/HarmAug-Guard`

## Constructor

| Parameter   | Type                                                 | Required | Default | Description                                                                                                                                                                                                              |
| ----------- | ---------------------------------------------------- | -------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `model_id`  | \`str                                                | None\`   | No      | `None`                                                                                                                                                                                                                   |
| `threshold` | `float`                                              | No       | `0.5`   | Unsafe-probability cutoff at or above which the input is flagged unsafe (`valid=False`). Defaults to 0.5, the value used in the HarmAug paper; lower it to catch borderline content, raise it to reduce false positives. |
| `provider`  | `Optional[Provider[dict[str, Any], dict[str, Any]]]` | No       | `None`  | Optional pre-configured provider. If `None`, a default `HuggingFaceProvider` is built and the model is loaded eagerly.                                                                                                   |

Initialize the HarmGuard guardrail.

## validate

Validate whether the input (and optionally the response) is safe.

**Parameters**

| Parameter     | Type  | Required | Default | Description                                                                                                                              |
| ------------- | ----- | -------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| `input_text`  | `str` | Yes      | —       | The prompt / user text to evaluate, e.g. `"How do I pick a lock?"`. A single string; list/batch input is not supported by this override. |
| `output_text` | \`str | None\`   | No      | `None`                                                                                                                                   |

**Returns:** `GuardrailOutput`

## Benchmarks

### Content Safety

| Dataset (rev)                    | Metric | Threshold    | Value    | Harness                 | Source                           | Contam. |
| -------------------------------- | ------ | ------------ | -------- | ----------------------- | -------------------------------- | ------- |
| openai\_moderation (unspecified) | f1     | native-valid | 0.815331 | guardrail-bench+ag0.7.4 | measured:guardrail-bench+ag0.7.4 |         |
| xstest (unspecified)             | fpr    | native-valid | 0.036    | guardrail-bench+ag0.7.4 | measured:guardrail-bench+ag0.7.4 |         |
| wildguardmix (unspecified)       | f1     | native-valid | 0.958333 | guardrail-bench+ag0.7.4 | measured:guardrail-bench+ag0.7.4 |         |
| aegis (unspecified)              | f1     | native-valid | 0.816456 | guardrail-bench+ag0.7.4 | measured:guardrail-bench+ag0.7.4 |         |
| jbb (unspecified)                | f1     | native-valid | 0.871111 | guardrail-bench+ag0.7.4 | measured:guardrail-bench+ag0.7.4 |         |
| orbench (unspecified)            | fpr    | native-valid | 0.287719 | guardrail-bench+ag0.7.4 | measured:guardrail-bench+ag0.7.4 |         |

### Prompt Injection

| Dataset (rev)              | Metric | Threshold    | Value     | Harness                 | Source                           | Contam. |
| -------------------------- | ------ | ------------ | --------- | ----------------------- | -------------------------------- | ------- |
| deepset\_pi (unspecified)  | f1     | native-valid | 0.38961   | guardrail-bench+ag0.7.4 | measured:guardrail-bench+ag0.7.4 |         |
| notinject (unspecified)    | fpr    | native-valid | 0.0175439 | guardrail-bench+ag0.7.4 | measured:guardrail-bench+ag0.7.4 |         |
| gandalf (unspecified)      | recall | native-valid | 0.3125    | guardrail-bench+ag0.7.4 | measured:guardrail-bench+ag0.7.4 |         |
| bipia\_email (unspecified) | f1     | native-valid | 0.628668  | bir\@fd86c16            | measured:bir\@fd86c16            |         |
| bipia\_table (unspecified) | f1     | native-valid | 0.170683  | bir\@fd86c16            | measured:bir\@fd86c16            |         |

## License

* **Vendor:** hbseong
* **Default license:** `apache-2.0` (of the default model/service)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.mozilla.ai/any-guardrail/api-reference/index/prompt-injection/harm-guard.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.