Quickstart

Install any-llm and make your first API call in 5 minutes

Requirements

• Python 3.11 or newer
• API keys for your chosen LLM provider

Installation

pip install any-llm-sdk[all]  # Install with all provider support

Installing Specific Providers

If you want to install a specific provider from our supported providers:

pip install any-llm-sdk[mistral]  # For Mistral provider
pip install any-llm-sdk[ollama]   # For Ollama provider
# install multiple providers
pip install any-llm-sdk[mistral,ollama]

Library Integration

If you're building a library, install just the base package (pip install any-llm-sdk) and let your users install provider dependencies.

API Keys: Set your provider's API key as an environment variable (e.g., export MISTRAL_API_KEY="your-key") or pass it directly using the api_key parameter.

APIs

Using the AnyLLM Class

For applications making multiple requests with the same provider, use the AnyLLM class to avoid repeated provider instantiation:

import os

from any_llm import AnyLLM

# Make sure you have the appropriate API key set
api_key = os.environ.get('MISTRAL_API_KEY')
if not api_key:
    raise ValueError("Please set MISTRAL_API_KEY environment variable")

llm = AnyLLM.create("mistral")

response = llm.completion(
    model="mistral-small-latest",
    messages=[{"role": "user", "content": "Hello!"}]
)
print(response.choices[0].message.content)

metadata = llm.get_provider_metadata()
print(f"Supports streaming: {metadata.streaming}")
print(f"Supports tools: {metadata.completion}")

API Call

import os

from any_llm import completion

# Make sure you have the appropriate API key set
api_key = os.environ.get('MISTRAL_API_KEY')
if not api_key:
    raise ValueError("Please set MISTRAL_API_KEY environment variable")

# Recommended: separate provider and model parameters
response = completion(
    model="mistral-small-latest",
    provider="mistral",
    messages=[{"role": "user", "content": "Hello!"}]
)
print(response.choices[0].message.content)

When to Choose Which Approach

Use Direct API Functions (completion, acompletion) when:

• Making simple, one-off requests
• Prototyping or writing quick scripts
• You want the simplest possible interface

Use Provider Class (AnyLLM.create) when:

• Building applications that make multiple requests with the same provider
• You want to avoid repeated provider instantiation overhead

Finding model names: Check the providers page for provider IDs, or use the list_models API to see available models for your provider.

Custom OpenAI-compatible Endpoints

If your gateway or server speaks the OpenAI API but is not one of the supported providers, point any-llm at it directly with AnyLLM.create_openai_compatible. The provider reports the name you give it rather than reporting itself as openai, and is used exactly like any other provider instance:

from any_llm import AnyLLM

llm = AnyLLM.create_openai_compatible(
    name="mygateway",
    api_base="https://mygateway.example/v1",
    api_key="your-key",  # optional for keyless local servers
)

response = llm.completion(
    model="some-model",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(response.choices[0].message.content)

Capability flags follow the OpenAI-compatible defaults. A capability the endpoint does not implement fails either locally with NotImplementedError (for flag-gated capabilities such as batch) or with the endpoint's own error for calls that any-llm forwards. Use this whenever you need an OpenAI-compatible endpoint that any-llm does not ship a dedicated provider for.

Streaming

For the providers that support streaming, you can enable it by passing stream=True:

output = ""
for chunk in completion(
    model="mistral-small-latest",
    provider="mistral",
    messages=[{"role": "user", "content": "Hello!"}],
    stream=True
):
    chunk_content = chunk.choices[0].delta.content or ""
    print(chunk_content)
    output += chunk_content

Reasoning

For providers that support reasoning, you can request thinking traces alongside the response using reasoning_effort:

from any_llm import completion

response = completion(
    model="claude-sonnet-4-5-20250929",
    provider="anthropic",
    messages=[{"role": "user", "content": "How many r's are in strawberry?"}],
    reasoning_effort="high",
)

# Access the model's thinking trace
if response.choices[0].message.reasoning:
    print(response.choices[0].message.reasoning.content)

# The final answer
print(response.choices[0].message.content)

Reasoning also works with streaming — each chunk may include chunk.choices[0].delta.reasoning.

Embeddings

embedding and aembedding allow you to create vector embeddings from text using the same unified interface across providers.

Not all providers support embeddings - check the providers documentation to see which ones do.

from any_llm import embedding

result = embedding(
    model="text-embedding-3-small",
    provider="openai",
    inputs="Hello, world!" # can be either string or list of strings
)

# Access the embedding vector
embedding_vector = result.data[0].embedding
print(f"Embedding vector length: {len(embedding_vector)}")
print(f"Tokens used: {result.usage.total_tokens}")

Moderation

moderation and amoderation run a content-safety classifier against input (or output) text and return a normalized, OpenAI-compatible result.

Not all providers support moderation; calling an unsupported provider raises NotImplementedError. Today, OpenAI and Mistral implement the API.

from any_llm import moderation

result = moderation(
    model="omni-moderation-latest",
    provider="openai",
    input="I want to hurt someone",
)

print(result.results[0].flagged)       # True
print(result.results[0].categories)    # {"violence": True, ...}

Pass include_raw=True to populate ModerationResult.provider_raw with the untouched provider response (useful for debugging or provider-specific fields).

Tools

any-llm supports tool calling for providers that support it. You can pass a list of tools where each tool is either:

1. Python callable - Functions with proper docstrings and type annotations
2. OpenAI Format tool dict - Already in OpenAI tool format

from any_llm import completion

def get_weather(location: str, unit: str = "F") -> str:
    """Get weather information for a location.

    Args:
        location: The city or location to get weather for
        unit: Temperature unit, either 'C' or 'F'

    Returns:
        Current weather description
    """
    return f"Weather in {location} is sunny and 75{unit}!"

response = completion(
    model="mistral-small-latest",
    provider="mistral",
    messages=[{"role": "user", "content": "What's the weather in Pittsburgh PA?"}],
    tools=[get_weather]
)

any-llm automatically converts your Python functions to OpenAI tools format. Functions must have:

• A docstring describing what the function does
• Type annotations for all parameters
• A return type annotation

Exception Handling

The any-llm package provides a unified exception hierarchy that works consistently across all LLM providers.

Enabling Unified Exceptions

Opt-in Feature: Unified exception handling is currently opt-in. Set the ANY_LLM_UNIFIED_EXCEPTIONS environment variable to enable it:

export ANY_LLM_UNIFIED_EXCEPTIONS=1

When enabled, provider-specific exceptions are automatically converted to any-llm exception types. When disabled (default), the original provider exceptions are raised with a deprecation warning.

Basic Usage

from any_llm import completion
from any_llm.exceptions import (
    AnyLLMError,
    AuthenticationError,
    InvalidRequestError,
    ModelNotFoundError,
    ProviderError,
    RateLimitError,
)

try:
    response = completion(
        model="gpt-4",
        provider="openai",
        messages=[{"role": "user", "content": "Hello!"}]
    )
except ModelNotFoundError as e:
    print(f"Model not found: {e.message}")
except RateLimitError as e:
    print(f"Rate limited: {e.message}")
except AuthenticationError as e:
    print(f"Auth failed: {e.message}")
except InvalidRequestError as e:
    print(f"Invalid request: {e.message}")
except ProviderError as e:
    print(f"Provider error: {e.message}")
except AnyLLMError as e:
    print(f"Error: {e.message}")

Accessing Original Exceptions

All unified exceptions preserve the original provider exception for debugging:

from any_llm.exceptions import RateLimitError

messages = [{"role": "user", "content": "Hello!"}]

try:
    response = completion(model="gpt-4", provider="openai", messages=messages)
except RateLimitError as e:
    print(f"Provider: {e.provider_name}")
    print(f"Original exception: {type(e.original_exception)}")

Structured Error Metadata

Unified exceptions also carry the structured HTTP fields the provider reported, so you can classify a failure without unwrapping original_exception and coupling to a specific SDK's attribute layout:

Attribute	Meaning
status_code	HTTP status the provider returned
code	Provider-specific error code from the response body
param	Request field the provider flagged as the cause
error_type	Provider-specific error category, such as "invalid_request_error"

from any_llm.exceptions import InvalidRequestError

try:
    response = completion(model="gpt-4", provider="openai", messages=messages)
except InvalidRequestError as e:
    if e.status_code == 400 and e.param == "reasoning_effort":
        print("Retry with reasoning_effort set to 'none'")

These fields are populated best-effort from the shapes provider SDKs expose: an attribute on the exception, or the parsed response body. Coverage varies by provider, so treat every field as optional. Anything any-llm cannot recover is None, including status_code for a non-HTTP failure such as a timeout or connection error.

status_code also drives which exception type you get. See Exceptions for the full status-to-type mapping.