Quickstart

Install any-llm and make your first API call in 5 minutes

Requirements

• Python 3.11 or newer
• API keys for your chosen LLM provider

Installation

pip install any-llm-sdk[all]  # Install with all provider support

Installing Specific Providers

If you want to install a specific provider from our supported providers:

pip install any-llm-sdk[mistral]  # For Mistral provider
pip install any-llm-sdk[ollama]   # For Ollama provider
# install multiple providers
pip install any-llm-sdk[mistral,ollama]

Library Integration

If you're building a library, install just the base package (pip install any-llm-sdk) and let your users install provider dependencies.

API Keys: Set your provider's API key as an environment variable (e.g., export MISTRAL_API_KEY="your-key") or pass it directly using the api_key parameter.

APIs

Using the AnyLLM Class

For applications making multiple requests with the same provider, use the AnyLLM class to avoid repeated provider instantiation:

import os

from any_llm import AnyLLM

# Make sure you have the appropriate API key set
api_key = os.environ.get('MISTRAL_API_KEY')
if not api_key:
    raise ValueError("Please set MISTRAL_API_KEY environment variable")

llm = AnyLLM.create("mistral")

response = llm.completion(
    model="mistral-small-latest",
    messages=[{"role": "user", "content": "Hello!"}]
)
print(response.choices[0].message.content)

metadata = llm.get_provider_metadata()
print(f"Supports streaming: {metadata.streaming}")
print(f"Supports tools: {metadata.completion}")

API Call

import os

from any_llm import completion

# Make sure you have the appropriate API key set
api_key = os.environ.get('MISTRAL_API_KEY')
if not api_key:
    raise ValueError("Please set MISTRAL_API_KEY environment variable")

# Recommended: separate provider and model parameters
response = completion(
    model="mistral-small-latest",
    provider="mistral",
    messages=[{"role": "user", "content": "Hello!"}]
)
print(response.choices[0].message.content)

When to Choose Which Approach

Use Direct API Functions (completion, acompletion) when:

• Making simple, one-off requests
• Prototyping or writing quick scripts
• You want the simplest possible interface

Use Provider Class (AnyLLM.create) when:

• Building applications that make multiple requests with the same provider
• You want to avoid repeated provider instantiation overhead

Finding model names: Check the providers page for provider IDs, or use the list_models API to see available models for your provider.

Custom OpenAI-compatible Endpoints

If your gateway or server speaks the OpenAI API but is not one of the supported providers, point any-llm at it directly with AnyLLM.create_openai_compatible. The provider reports the name you give it rather than reporting itself as openai, and is used exactly like any other provider instance:

from any_llm import AnyLLM

llm = AnyLLM.create_openai_compatible(
    name="mygateway",
    api_base="https://mygateway.example/v1",
    api_key="your-key",  # optional for keyless local servers
)

response = llm.completion(
    model="some-model",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(response.choices[0].message.content)

Capability flags follow the OpenAI-compatible defaults. A capability the endpoint does not implement fails either locally with NotImplementedError (for flag-gated capabilities such as batch) or with the endpoint's own error for calls that any-llm forwards. Use this whenever you need an OpenAI-compatible endpoint that any-llm does not ship a dedicated provider for.

Streaming

For the providers that support streaming, you can enable it by passing stream=True:

output = ""
for chunk in completion(
    model="mistral-small-latest",
    provider="mistral",
    messages=[{"role": "user", "content": "Hello!"}],
    stream=True
):
    chunk_content = chunk.choices[0].delta.content or ""
    print(chunk_content)
    output += chunk_content

Reasoning

For providers that support reasoning, you can request thinking traces alongside the response using reasoning_effort:

from any_llm import completion

response = completion(
    model="claude-sonnet-4-5-20250929",
    provider="anthropic",
    messages=[{"role": "user", "content": "How many r's are in strawberry?"}],
    reasoning_effort="high",
)

# Access the model's thinking trace
if response.choices[0].message.reasoning:
    print(response.choices[0].message.reasoning.content)

# The final answer
print(response.choices[0].message.content)

Reasoning also works with streaming — each chunk may include chunk.choices[0].delta.reasoning.

Responses

For the providers that support it, responses and aresponses expose the OpenAI-style Responses API:

from any_llm import AnyLLM

llm = AnyLLM.create("openai")
response = llm.responses(
    model="gpt-5-nano",
    input_data="Explain why the sky is blue.",
)
print(response.output_text)

Coverage varies by provider. Check the Responses column in providers, and the notes below for parameters a provider rejects.

Gemini

Gemini maps text-only responses() calls to Google's Interactions API. The existing completion() method continues to use generateContent.

from any_llm import AnyLLM

llm = AnyLLM.create("gemini")
response = llm.responses(
    model="gemini-3.8-flash",
    input_data="Explain why the sky is blue.",
)
print(response.output_text)

Pass stream=True to responses() for a synchronous iterator of Responses events, or await aresponses() and consume the returned async iterator.

Gemini text Interactions accept string input, instructions, maximum output tokens, per-request timeouts, and streaming. Omitting store uses Google's default storage behavior. Passing store=True or store=False raises UnsupportedParameterError. Other unsupported Responses parameters include media, tools, reasoning controls, structured output, chaining, metadata, and background execution.

Embeddings

embedding and aembedding allow you to create vector embeddings from text using the same unified interface across providers.

Not all providers support embeddings - check the providers documentation to see which ones do.

from any_llm import embedding

result = embedding(
    model="text-embedding-3-small",
    provider="openai",
    inputs="Hello, world!" # can be either string or list of strings
)

# Access the embedding vector
embedding_vector = result.data[0].embedding
print(f"Embedding vector length: {len(embedding_vector)}")
print(f"Tokens used: {result.usage.total_tokens}")

Moderation

moderation and amoderation run a content-safety classifier against input (or output) text and return a normalized, OpenAI-compatible result.

Not all providers support moderation; calling an unsupported provider raises NotImplementedError. Today, OpenAI and Mistral implement the API.

from any_llm import moderation

result = moderation(
    model="omni-moderation-latest",
    provider="openai",
    input="I want to hurt someone",
)

print(result.results[0].flagged)       # True
print(result.results[0].categories)    # {"violence": True, ...}

Pass include_raw=True to populate ModerationResult.provider_raw with the untouched provider response (useful for debugging or provider-specific fields).

Tools

any-llm supports tool calling for providers that support it. You can pass a list of tools where each tool is either:

1. Python callable - Functions with proper docstrings and type annotations
2. OpenAI Format tool dict - Already in OpenAI tool format

from any_llm import completion

def get_weather(location: str, unit: str = "F") -> str:
    """Get weather information for a location.

    Args:
        location: The city or location to get weather for
        unit: Temperature unit, either 'C' or 'F'

    Returns:
        Current weather description
    """
    return f"Weather in {location} is sunny and 75{unit}!"

response = completion(
    model="mistral-small-latest",
    provider="mistral",
    messages=[{"role": "user", "content": "What's the weather in Pittsburgh PA?"}],
    tools=[get_weather]
)

any-llm automatically converts your Python functions to OpenAI tools format. Functions must have:

• A docstring describing what the function does
• Type annotations for all parameters
• A return type annotation

Retries

Provider SDKs differ in how often they retry a failed request on their own: the OpenAI and Anthropic SDKs retry twice by default (honouring Retry-After), while the Google GenAI SDK used by gemini and vertexai does not retry unless configured to. Set max_retries to give every provider the same retry count, for example 0 when your own code or gateway already handles retries and failover:

from any_llm import AnyLLM, completion

llm = AnyLLM.create("gemini", max_retries=0)

response = completion(
    model="gpt-4o-mini",
    provider="openai",
    messages=[{"role": "user", "content": "Hello!"}],
    client_args={"max_retries": 0},
)

max_retries counts retries after the first request, so 0 means one request in total. Omitting it, or passing None, keeps each SDK's own default. Each provider maps the value to its SDK's native setting:

Providers	SDK setting
OpenAI-compatible providers (openai, azureopenai, deepseek, openrouter, meta, registry gateways, AnyLLM.create_openai_compatible(), ...), anthropic, azureanthropic, vertexaianthropic, groq, cerebras, together, cohere	max_retries on the SDK client
gemini, vertexai	http_options.retry_options.attempts, set to max_retries + 1 (vertexai partner and Mistral models pass max_retries to their OpenAI SDK client)
voyage	max_retries on the SDK client, set to max_retries + 1 because the SDK counts the first request

The other providers (azure, bedrock, huggingface, lmstudio, mistral, ollama, otari, sagemaker, watsonx, xai) raise UnsupportedParameterError for max_retries, because their SDK has no single retry count to set. Configure retries for them through the SDK's own options in client_args, for example retry_config for Mistral or a botocore Config for Bedrock. For gemini and vertexai, retry_options that you pass in http_options take precedence over max_retries.

Exception Handling

The any-llm package provides a unified exception hierarchy that works consistently across all LLM providers.

Enabling Unified Exceptions

Opt-in Feature: Unified exception handling is currently opt-in. Set the ANY_LLM_UNIFIED_EXCEPTIONS environment variable to enable it:

export ANY_LLM_UNIFIED_EXCEPTIONS=1

When enabled, provider-specific exceptions are automatically converted to any-llm exception types. When disabled (default), the original provider exceptions are raised with a deprecation warning.

To configure a single client without changing the environment for other callers, pass unified_exceptions when creating the provider:

from any_llm import AnyLLM

llm = AnyLLM.create("openai", unified_exceptions=True)

An explicit True or False takes precedence over the environment variable for that instance, including errors raised while iterating a streaming response or downloading a file. Omitting the argument, or passing None, preserves the environment-controlled behavior. This option is also accepted by provider constructors and AnyLLM.create_openai_compatible().

Basic Usage

from any_llm import completion
from any_llm.exceptions import (
    AnyLLMError,
    AuthenticationError,
    InvalidRequestError,
    ModelNotFoundError,
    ProviderError,
    RateLimitError,
)

try:
    response = completion(
        model="gpt-4",
        provider="openai",
        messages=[{"role": "user", "content": "Hello!"}]
    )
except ModelNotFoundError as e:
    print(f"Model not found: {e.message}")
except RateLimitError as e:
    print(f"Rate limited: {e.message}")
except AuthenticationError as e:
    print(f"Auth failed: {e.message}")
except InvalidRequestError as e:
    print(f"Invalid request: {e.message}")
except ProviderError as e:
    print(f"Provider error: {e.message}")
except AnyLLMError as e:
    print(f"Error: {e.message}")

Accessing Original Exceptions

All unified exceptions preserve the original provider exception for debugging:

from any_llm.exceptions import RateLimitError

messages = [{"role": "user", "content": "Hello!"}]

try:
    response = completion(model="gpt-4", provider="openai", messages=messages)
except RateLimitError as e:
    print(f"Provider: {e.provider_name}")
    print(f"Original exception: {type(e.original_exception)}")

Structured Error Metadata

Unified exceptions also carry the structured HTTP fields the provider reported, so you can classify a failure without unwrapping original_exception and coupling to a specific SDK's attribute layout:

Attribute	Meaning
status_code	HTTP status the provider returned
code	Provider-specific error code from the response body
param	Request field the provider flagged as the cause
error_type	Provider-specific error category, such as "invalid_request_error"

from any_llm.exceptions import InvalidRequestError

try:
    response = completion(model="gpt-4", provider="openai", messages=messages)
except InvalidRequestError as e:
    if e.status_code == 400 and e.param == "reasoning_effort":
        print("Retry with reasoning_effort set to 'none'")

These fields are populated best-effort from the shapes provider SDKs expose: an attribute on the exception, or the parsed response body. Coverage varies by provider, so treat every field as optional. Anything any-llm cannot recover is None, including status_code for a non-HTTP failure such as a timeout or connection error.

status_code also drives which exception type you get. See Exceptions for the full status-to-type mapping.