> For the complete documentation index, see [llms.txt](https://docs.mozilla.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.mozilla.ai/caching.md).

# Prompt Caching

Cache a Gemini or Vertex AI prompt prefix by marking it with Anthropic-style cache\_control

Gemini and Vertex AI turn Anthropic-style `cache_control` breakpoints into [context caches](https://ai.google.dev/gemini-api/docs/caching). Mark the end of the prefix you want cached and any-llm creates the cache, reuses it on later calls, and reports the tokens it wrote and read. There is no cache API to call.

```python
from any_llm import AnyLLM

provider = AnyLLM.create("gemini")
manual = "Widgets are blue and round. " * 200  # the long, stable prefix: at least 1,024 tokens, see below

response = provider.completion(
    model="gemini-2.5-flash",
    messages=[
        {
            "role": "system",
            "content": [{"type": "text", "text": manual, "cache_control": {"type": "ephemeral"}}],
        },
        {"role": "user", "content": "What does chapter 3 say about retries?"},
    ],
)
print(response.usage.prompt_tokens_details)
```

## Where a breakpoint can go

* **Chat Completions**: a `cache_control` dict on a message or on any of its content parts, the convention LiteLLM and OpenRouter use for OpenAI-format requests.
* **Messages** (`provider.messages(...)`): `cache_control` on system blocks, on user, assistant, and `tool_result` blocks, and the top-level `cache_control` parameter, which marks the last message as Anthropic's automatic caching does.

The value is `{"type": "ephemeral"}`, optionally with `"ttl": "5m"` or `"ttl": "1h"`. The Messages bridge does not forward these markers to other providers.

## What gets cached

Gemini caches a single prefix, so **the last breakpoint wins**. A request that uses a cache cannot also send a system instruction or tools, so the cache holds the system instruction, the tools and tool choice, and the conversation up to and including the marked message. The request then sends only the messages after it. Gemini rejects a request with no contents, so when the breakpoint is on the last message that message stays outside the cache.

The cache lives for 5 minutes, or 1 hour if any breakpoint in the prefix asks for `"1h"`. Caches are reused within the process across provider instances, keyed by the credentials, endpoint, model, and cached content, and are recreated shortly before they expire.

Gemini refuses caches below a minimum size of 1,024 to 4,096 tokens, depending on the model. A refused prefix is sent uncached, and any-llm does not try to create it again until the TTL passes. Any other failure to create a cache also falls back to an uncached request. Passing your own `cached_content` turns the automatic path off for that call.

## Usage reporting

On the call that creates the cache, the cached prefix is reported as `usage.prompt_tokens_details.cache_write_tokens` (with `cached_tokens=0`). Later calls that reuse it report `cached_tokens`. Both are subsets of `prompt_tokens`. Through `messages()` they appear as `cache_creation_input_tokens` and `cache_read_input_tokens`.

## Implicit caching

Gemini 2.5 and later models also cache repeated prefixes implicitly, with no markers and no storage cost. Those hits are reported as `cached_tokens` too. Breakpoints are worth adding when you want a guaranteed discount on a large, stable prefix.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.mozilla.ai/caching.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
