> For the complete documentation index, see [llms.txt](https://docs.mozilla.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.mozilla.ai/any-guardrail/api-reference/index/content-safety/duo-guard.md). # DuoGuard Multilingual multi-label safety classifier scoring text across 12 harm categories including jailbreak prompts. DuoGuard is a compact (0.5B-1.5B) classifier built on Qwen 2.5 and Llama 3.2 backbones, trained with a two-player reinforcement-learning framework in which a generator and the guardrail co-evolve to synthesize multilingual safety data. It emits an independent probability (sigmoid) for each of the 12 categories in `DUOGUARD_CATEGORIES` — the MLCommons-style hazard taxonomy (violent crimes, non-violent crimes, sex-related crimes, child sexual exploitation, specialized advice, privacy, intellectual property, indiscriminate weapons, hate, suicide and self-harm, sexual content) plus a jailbreak-prompt category. The models are fine-tuned primarily for English, French, German, and Spanish, with broader coverage inherited from the Qwen 2.5 / Llama 3.2 base models. Verdict mapping onto `GuardrailOutput`: * `categories` carries all 12 categories, each with its sigmoid probability and a `triggered` flag (probability strictly above `threshold`). * `valid` is `True` only when no category is triggered. * `score` (canonical risk: higher = riskier) is the maximum category probability. Expected inputs: a single text string, or a `list[str]` for batched classification (the inherited `ThreeStageGuardrail.validate` handles list input). It screens a single body of text; it does not take a separate prompt / response pair. For more information, see: * [DuoGuard model collection](https://huggingface.co/collections/DuoGuard/duoguard-models-67a29ad8bd579a404e504d21). * [DuoGuard-0.5B model card](https://huggingface.co/DuoGuard/DuoGuard-0.5B) (default). * [DuoGuard-1B-Llama-3.2-transfer model card](https://huggingface.co/DuoGuard/DuoGuard-1B-Llama-3.2-transfer). * [DuoGuard-1.5B-transfer model card](https://huggingface.co/DuoGuard/DuoGuard-1.5B-transfer). * [DuoGuard: A Two-Player RL-Driven Framework for Multilingual LLM Guardrails (arXiv:2502.05163)](https://arxiv.org/abs/2502.05163). ## Supported Models * `DuoGuard/DuoGuard-0.5B` * `DuoGuard/DuoGuard-1B-Llama-3.2-transfer` * `DuoGuard/DuoGuard-1.5B-transfer` ## Constructor | Parameter | Type | Required | Default | Description | | ----------- | ---------------------------------------------------- | -------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `model_id` | \`str | None\` | No | `None` | | `threshold` | `float` | No | `0.5` | Per-category probability strictly above which that category is flagged (and the text becomes invalid). Defaults to 0.5, from the model card. | | `provider` | `Optional[Provider[dict[str, Any], dict[str, Any]]]` | No | `None` | Optional pre-configured provider. If `None`, a default `HuggingFaceProvider` is built with `multi_label=True` and the matching base tokenizer (see `MODELS_TO_TOKENIZER`), then the model is loaded eagerly. A pad token is set from the EOS token because the Qwen-family tokenizers ship without one. | Initialize the DuoGuard guardrail. ## validate Default validation pipeline: preprocess -> inference -> postprocess. **Parameters** | Parameter | Type | Required | Default | Description | | ------------ | ----- | ------------ | ------- | ----------- | | `input_text` | \`str | list\[str]\` | Yes | — | **Returns:** `GuardrailOutput | list[GuardrailOutput]` ## Benchmarks ### Content Safety | Dataset (rev) | Metric | Threshold | Value | Harness | Source | Contam. | | -------------------------------- | ------ | ------------ | -------- | ----------------------- | -------------------------------- | ------- | | openai\_moderation (unspecified) | f1 | native-valid | 0.789668 | guardrail-bench+ag0.7.4 | measured:guardrail-bench+ag0.7.4 | | | xstest (unspecified) | fpr | native-valid | 0.028 | guardrail-bench+ag0.7.4 | measured:guardrail-bench+ag0.7.4 | | | wildguardmix (unspecified) | f1 | native-valid | 0.875 | guardrail-bench+ag0.7.4 | measured:guardrail-bench+ag0.7.4 | | | aegis (unspecified) | f1 | native-valid | 0.810997 | guardrail-bench+ag0.7.4 | measured:guardrail-bench+ag0.7.4 | | | jbb (unspecified) | f1 | native-valid | 0.793814 | guardrail-bench+ag0.7.4 | measured:guardrail-bench+ag0.7.4 | | | orbench (unspecified) | fpr | native-valid | 0.242105 | guardrail-bench+ag0.7.4 | measured:guardrail-bench+ag0.7.4 | | ### Prompt Injection | Dataset (rev) | Metric | Threshold | Value | Harness | Source | Contam. | | ------------------------- | ------ | ------------ | --------- | ----------------------- | -------------------------------- | ------- | | deepset\_pi (unspecified) | f1 | native-valid | 0.475 | guardrail-bench+ag0.7.4 | measured:guardrail-bench+ag0.7.4 | | | notinject (unspecified) | fpr | native-valid | 0.0105263 | guardrail-bench+ag0.7.4 | measured:guardrail-bench+ag0.7.4 | | | gandalf (unspecified) | recall | native-valid | 0.473214 | guardrail-bench+ag0.7.4 | measured:guardrail-bench+ag0.7.4 | | ### Toxicity | Dataset (rev) | Metric | Threshold | Value | Harness | Source | Contam. | | ---------------------------- | ------ | ------------ | -------- | ----------------------- | -------------------------------- | ------- | | real\_toxicity (unspecified) | f1 | native-valid | 0.308571 | guardrail-bench+ag0.7.4 | measured:guardrail-bench+ag0.7.4 | | ## License * **Vendor:** DuoGuard * **Default license:** `apache-2.0` (of the default model/service) | Model variant | License | | ----------------------------------------- | ------------ | | `DuoGuard/DuoGuard-0.5B` | `apache-2.0` | | `DuoGuard/DuoGuard-1.5B-transfer` | `apache-2.0` | | `DuoGuard/DuoGuard-1B-Llama-3.2-transfer` | `llama-3.2` | --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.mozilla.ai/any-guardrail/api-reference/index/content-safety/duo-guard.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.