Skip to main content
Response caching stores the answer to a request keyed by its content. When an identical request comes in again, the gateway returns the stored answer instead of calling the provider, so a cache hit costs nothing and returns immediately. This is distinct from Token Optimizer: caching skips the call on a repeat, Token Optimizer compresses the prompt on every call. The two stack
Caching is a gateway (data-plane) feature. The control plane shares the same Redis so the Caching page’s health check reports correctly, but the actual response caching happens where the LLM calls run

​
How it is wired in ForceAI

The stack ships a Redis service and a small config that turns caching on:
  • A pinned redis service in the compose (cache only, no persistence)
  • deploy/gateway/cache_config.yaml with litellm_settings.cache: true pointing at Redis, mounted into the gateway and backend via CONFIG_FILE_PATH
  • Database-stored models still load on top of the config, so nothing about routing changes
The config caches completion and embedding call types with a one-hour TTL:

​
Tune it

Edit deploy/gateway/cache_config.yaml and recreate the gateway (and backend) so the mounted config is re-read:
  • ttl sets how long a cached entry lives, in seconds
  • supported_call_types limits what gets cached; drop embedding/aembedding if you only want chat responses
  • Point host/port at an external Redis instead of the bundled one for a shared or managed cache

​
See cache hits

Open the dashboard Experimental -> Caching page. The health tile shows a healthy Redis cache, and the analytics show the hit ratio, cached responses, and cached tokens over a date range. Send the same prompt twice from the Playground and watch the cached-rows count go up

​
Verify by API

Two identical completions; the second is served from cache (same response id, no new provider call):
The two id values match on a hit. The cached call also records cache_hit=true in the spend logs, which is what the Caching page’s analytics read Check the cache backend directly:
A healthy cache returns {"status": "healthy", "cache_type": "redis"}
Caching runs after the pre-call hooks, so a cache hit still respects guardrails and access-group gating; it only avoids the provider round-trip. Two requests are the same entry only when their model, messages, and parameters (including temperature) match exactly