# Batch

{% hint style="warning" %}
The Batch API is experimental and may change in future releases. Provider support is limited - check the [providers page](/providers.md) for availability.
{% endhint %}

The Batch API lets you submit multiple requests as a single job for asynchronous processing, typically at lower cost than real-time requests.

### How It Works

1. Prepare a JSONL file where each line is a batch request object.
2. Call `create_batch()` with the file path and target endpoint.
3. any-llm uploads the file to the provider and creates the batch job.
4. Poll with `retrieve_batch()` to check status.
5. When complete, download results from the provider.

#### Input File Format

The input file must be a JSONL file where each line follows this structure:

```json
{"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-4.1-mini", "messages": [{"role": "user", "content": "Hello"}]}}
{"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-4.1-mini", "messages": [{"role": "user", "content": "World"}]}}
```

### `any_llm.create_batch()`

Create a batch job by uploading a local JSONL file.

```
def create_batch(
    provider: str | LLMProvider,
    input_file_path: str,
    endpoint: str,
    *,
    completion_window: str = "24h",
    metadata: dict[str, str] | None = None,
    api_key: str | None = None,
    api_base: str | None = None,
    client_args: dict[str, Any] | None = None,
    **kwargs: Any,
) -> Batch
```

#### `any_llm.acreate_batch()`

Async variant with the same parameters.

### Parameters (create)

| Parameter           | Type                     | Default    | Description                                                                |
| ------------------- | ------------------------ | ---------- | -------------------------------------------------------------------------- |
| `provider`          | `str \| LLMProvider`     | *required* | Provider name to use for the request (e.g., 'openai', 'mistral')           |
| `input_file_path`   | `str`                    | *required* | Path to a local file containing batch requests in JSONL format.            |
| `endpoint`          | `str`                    | *required* | The endpoint to be used for all requests (e.g., '/v1/chat/completions')    |
| `completion_window` | `str`                    | "24h"      | The time frame within which the batch should be processed (default: '24h') |
| `metadata`          | `dict[str, str] \| None` | None       | Optional custom metadata for the batch                                     |
| `api_key`           | `str \| None`            | None       | API key for the provider                                                   |
| `api_base`          | `str \| None`            | None       | Base URL for the provider API                                              |
| `client_args`       | `dict[str, Any] \| None` | None       | Additional provider-specific arguments for client instantiation            |
| `**kwargs`          | `Any`                    | *required* | Additional provider-specific arguments                                     |

**Returns:** A [`Batch`](/api-reference/completion-1/batch.md) object.

### `any_llm.retrieve_batch()`

Retrieve the current status and details of a batch job.

```
def retrieve_batch(
    provider: str | LLMProvider,
    batch_id: str,
    *,
    api_key: str | None = None,
    api_base: str | None = None,
    client_args: dict[str, Any] | None = None,
    **kwargs: Any,
) -> Batch
```

#### `any_llm.aretrieve_batch()`

Async variant with the same parameters.

### Parameters (retrieve)

| Parameter     | Type                     | Default    | Description                                                      |
| ------------- | ------------------------ | ---------- | ---------------------------------------------------------------- |
| `provider`    | `str \| LLMProvider`     | *required* | Provider name to use for the request (e.g., 'openai', 'mistral') |
| `batch_id`    | `str`                    | *required* | The ID of the batch to retrieve                                  |
| `api_key`     | `str \| None`            | None       | API key for the provider                                         |
| `api_base`    | `str \| None`            | None       | Base URL for the provider API                                    |
| `client_args` | `dict[str, Any] \| None` | None       | Additional provider-specific arguments for client instantiation  |
| `**kwargs`    | `Any`                    | *required* | Additional provider-specific arguments                           |

**Returns:** A [`Batch`](/api-reference/completion-1/batch.md) object.

### `any_llm.cancel_batch()`

Cancel an in-progress batch job.

```
def cancel_batch(
    provider: str | LLMProvider,
    batch_id: str,
    *,
    api_key: str | None = None,
    api_base: str | None = None,
    client_args: dict[str, Any] | None = None,
    **kwargs: Any,
) -> Batch
```

#### `any_llm.acancel_batch()`

Async variant with the same parameters.

**Returns:** The cancelled [`Batch`](/api-reference/completion-1/batch.md) object.

### `any_llm.list_batches()`

List batch jobs for a provider.

```
def list_batches(
    provider: str | LLMProvider,
    *,
    after: str | None = None,
    limit: int | None = None,
    api_key: str | None = None,
    api_base: str | None = None,
    client_args: dict[str, Any] | None = None,
    **kwargs: Any,
) -> Sequence[Batch]
```

#### `any_llm.alist_batches()`

Async variant with the same parameters.

### Parameters (list)

| Parameter     | Type                     | Default    | Description                                                      |
| ------------- | ------------------------ | ---------- | ---------------------------------------------------------------- |
| `provider`    | `str \| LLMProvider`     | *required* | Provider name to use for the request (e.g., 'openai', 'mistral') |
| `after`       | `str \| None`            | None       | A cursor for pagination. Returns batches after this batch ID.    |
| `limit`       | `int \| None`            | None       | Maximum number of batches to return (default: 20)                |
| `api_key`     | `str \| None`            | None       | API key for the provider                                         |
| `api_base`    | `str \| None`            | None       | Base URL for the provider API                                    |
| `client_args` | `dict[str, Any] \| None` | None       | Additional provider-specific arguments for client instantiation  |
| `**kwargs`    | `Any`                    | *required* | Additional provider-specific arguments                           |

**Returns:** A `Sequence` of [`Batch`](/api-reference/completion-1/batch.md) objects.

### Usage

```python
from any_llm import create_batch, retrieve_batch, list_batches

# Create a batch job
batch = create_batch(
    provider="openai",
    input_file_path="requests.jsonl",
    endpoint="/v1/chat/completions",
)
print(f"Batch created: {batch.id}, status: {batch.status}")

# Check status
batch = retrieve_batch("openai", batch.id)
print(f"Status: {batch.status}")

# List all batches
batches = list_batches("openai")
for b in batches:
    print(f"{b.id}: {b.status}")
```


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.mozilla.ai/api-reference/batch.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.