Prompt Caching¶
ReActAgent supports provider-stable prompt caching: you declare intent — "cache the system prefix, 1h TTL" — and the framework emits the correct wire markers for the active provider, or none, and never crashes on a provider that doesn't support them.
Prompt caching cuts cost (cache reads are roughly a tenth of input price) and latency, but every provider expresses it differently: Anthropic and Bedrock-Claude take inline cache_control markers; OpenAI / DeepSeek / Nova cache automatically with no markers; Gemini caches implicitly. CachePolicy hides that matrix.
from latent.agents import ReActAgent, CachePolicy
agent = ReActAgent(
name="support",
model="anthropic/claude-sonnet-4-5",
system_prompt=BIG_PROMPT,
cache_policy=CachePolicy(system=True, ttl="1h"),
)
Default is cache_policy=None (off) — existing agents are byte-for-byte unchanged.
CachePolicy¶
Declares what to cache and the TTL:
@dataclass(frozen=True, slots=True)
class CachePolicy:
system: bool = True # cache the stable system prefix
tools: bool = False # also cache the tool definitions
ttl: Literal["5m", "1h"] = "5m"
At most two cache markers are emitted (system + tools), always under every provider's breakpoint ceiling.
SystemContent — splitting static from per-turn content¶
If your system message has a stable prefix plus per-turn content (a timestamp, retrieved hints, a customer record), split them so only the stable part is cached. Override _system_content to return a SystemContent:
from latent.agents import SystemContent
class SupportAgent(ReActAgent):
def _system_content(self, messages):
return SystemContent(
static=self._stable_prompt, # cached
dynamic=self._per_turn_context(messages), # uncached, appended after
)
The cache marker is attached to static only, so per-turn content can never leak into the cached prefix and silently invalidate it. static always precedes dynamic on the wire — which also keeps the prefix stable for automatic-caching providers.
_system_content may return str (the whole string is the cacheable prefix), SystemContent (explicit split), or a raw list[dict] (legacy escape hatch — you own the markers, the policy is not applied).
Provider behavior¶
A single internal registry maps the model string to one decision — emit cache_control markers or not:
| Provider | Markers | 1h TTL |
|---|---|---|
anthropic/… |
yes | yes |
bedrock/…claude… / …anthropic… |
yes | only the Claude 4.5 line (Opus/Sonnet/Haiku 4.5) |
bedrock/…nova…, …llama… |
no (automatic) | — |
openai/…, gemini/…, vertex_ai/…, unknown |
no | — |
LiteLLM owns the wire-format translation (cache_control → Bedrock cachePoint); latent-py owns the decision of whether to emit. The split matters because LiteLLM does not reliably strip an unknown marker for a provider that rejects it.
Switching model= from anthropic/… to openai/… keeps working: markers are dropped, no crash, and OpenAI's automatic prefix caching takes over. No cache_control, provider name, or TTL syntax ever appears in your code.
Failure modes — degrade, never crash¶
| Situation | Behavior |
|---|---|
| Provider caches automatically / ignores markers | No markers emitted; ordering still optimal; no error |
| Unknown provider | No markers; no error |
| Content below the provider's minimum cacheable size | Provider succeeds with no caching; no error (no pre-check) |
ttl="1h" on a model that only supports 5m |
Degrades to 5m — never sends ttl:1h to a model that would reject it (Bedrock returns HTTP 400; Anthropic always honors 1h) |
| Per-turn content in the cached prefix | Structurally impossible — only SystemContent.static carries the marker |
Observability¶
Cache usage is reported on the existing Usage event via cache_creation_input_tokens and cache_read_input_tokens.
A zero-hit detector runs automatically: when a policy is active and the provider emits markers, the agent expects a cache read from the second LLM call onward (the static prefix is stable across turns and ReAct iterations). If reads stay at zero, it logs a one-time warning — the classic cause is a per-turn byte leaking into SystemContent.static, which invalidates the cache every call. The check is passive (warn, never block) and resets with reset().
Out of scope¶
Multi-turn history caching, Gemini's explicit cachedContents lifecycle, and per-region TTL are not covered in this release. Direct litellm.acompletion callers (outside an agent) are not affected — CachePolicy applies to ReActAgent and its subclasses.