# zilliztech/gptcache architecture

> Architecture diagram of the GitHub repository zilliztech/gptcache, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zilliztech/gptcache
- Repository: https://github.com/zilliztech/gptcache

## Overview

1. GPTCache is a Python library for semantic caching of LLM and multimodal requests. The core idea is: intercept a model call, normalize/process the request, look for a semantically similar cached response, and either return the cache hit or forward to the upstream model and store the result.

2. The main runtime entry point is `gptcache.cache` from `gptcache/core.py` and `gptcache/client.py`, which exposes the initialize/use flow shown in the README. This is the orchestration layer that wires together embedding, storage, similarity evaluation, and request/response post-processing.

3. The request path is split into three conceptual stages under `gptcache/processor`: pre-processing, hit checking, and post-processing. Pre-processors shape the incoming prompt/context, `check_hit.py` decides whether cache lookup should proceed, and `processor/post.py` handles output shaping such as temperature-aware behavior.

4. Adapters under `gptcache/adapter` are the integration boundary to external model providers and SDKs. They present a unified interface over OpenAI, LangChain models, Llama.cpp, Dolly, Replicate, Diffusers, Stability SDK, and multimodal APIs, so the cache layer stays provider-agnostic.

5. Embedding generation lives in `gptcache/embedding`. This subsystem converts text, image, or audio inputs into vectors for semantic matching. It supports multiple backends such as OpenAI, ONNX, HuggingFace, SentenceTransformers/SBERT, Cohere, fastText, TIMM, ViT, UForm, and others. Disabling embeddings effectively turns GPTCache into a keyword/exact-match cache.

6. The cache data plane is managed by `gptcache/manager`. This is where cached objects are persisted, indexed, and retrieved. It is split into:
   - scalar data storage for request/metadata records,
   - vector data storage for embeddings and nearest-neighbor search,
   - object data storage for larger payloads/responses,
   - eviction management for cache lifecycle.
   The structure makes storage backends swappable without changing the orchestration layer.

7. Storage backends are a major architectural boundary. Scalar stores include SQLite, Redis, MongoDB, DynamoDB, and SQL-backed storage. Vector stores include FAISS, Milvus, PGVector, Chroma, Qdrant, Weaviate, HNSWLib, DocArray, Redis vector store, and USearch. Object storage includes local storage and S3. This makes the cache deployable from local/dev setups to distributed production systems.

8. Similarity decisions are implemented in `gptcache/similarity_evaluation`. These evaluators combine vector distances and/or secondary scoring methods to decide if a query is “close enough” to a cached request. The package includes distance-based evaluators, exact match, sequence match, ONNX-based classification, SBERT cross-encoder, K-reciprocal, Cohere rerank, and timing/threshold utilities.

9. Eviction and distribution are handled separately from similarity. `gptcache/manager/eviction` contains in-memory and Redis-backed distributed eviction logic, which is important for horizontal scaling. The repo also includes a dedicated `gptcache_server` package and Dockerfile, indicating a server mode for language-agnostic access to the cache over a process/network boundary.

10. The repository is organized as a product library plus supporting ecosystem:
   - `examples/` demonstrates adapter integrations, data managers, embeddings, processors, and benchmarking.
   - `docs/` provides usage, bootcamps, and deployment guidance, including horizontal scaling.
   - `tests/` contains extensive unit and integration coverage across adapters, embeddings, managers, processors, similarity evaluators, and session/client/core behavior.
   - GitHub Actions automate linting, unit tests, packaging, and release artifacts.
   This suggests the architecture is modular, backend-heavy, and optimized for composable cache pipelines rather than a single application UI.

Main entry points: [`gptcache/manager`](https://github.com/zilliztech/gptcache/tree/main/gptcache/manager) (Data Manager), [`gptcache/adapter/api.py`](https://github.com/zilliztech/gptcache/blob/main/gptcache/adapter/api.py) (Adapter API), [`gptcache/processor/check_hit.py`](https://github.com/zilliztech/gptcache/blob/main/gptcache/processor/check_hit.py) (Hit Check), [`gptcache/core.py`](https://github.com/zilliztech/gptcache/blob/main/gptcache/core.py) (Cache API)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Runtime"]
  node_core_api["Cache API<br/>orchestration<br/>[core.py]"]
  node_client["Client<br/>runtime facade<br/>[client.py]"]
  node_pre["Pre-process<br/>request shaping<br/>[pre.py]"]
  node_check_hit["Hit Check<br/>lookup gate<br/>[check_hit.py]"]
  node_post["Post-process<br/>response shaping<br/>[post.py]"]
  node_context["Context<br/>[context.py]"]
end

subgraph group_providers["Providers"]
  node_adapter_api["Adapter API<br/>integration boundary<br/>[api.py]"]
  node_openai_adapter["OpenAI<br/>provider adapter<br/>[openai.py]"]
  node_langchain_adapter["LangChain<br/>provider adapter"]
  node_media_adapters["Media Adapters<br/>provider adapters"]
end

subgraph group_data["Data Plane"]
  node_manager["Data Manager<br/>cache data plane"]
  node_scalar_store[("Scalar Store<br/>metadata storage")]
  node_vector_store[("Vector Store<br/>nn index")]
  node_object_store[("Object Store<br/>payload storage")]
  node_eviction["Eviction<br/>lifecycle control"]
end

subgraph group_scoring["Scoring"]
  node_embed["Embedding<br/>vectorization"]
  node_similarity["Similarity Eval<br/>match scoring"]
end

subgraph group_ops["Ops"]
  node_cache_server["Cache Server<br/>[server.py]"]
  node_docs["Docs<br/>documentation"]
  node_examples["Examples<br/>reference code"]
end

node_client -->|"uses"| node_core_api
node_core_api -->|"starts"| node_pre
node_pre -->|"builds"| node_context
node_pre -->|"hands off"| node_check_hit
node_check_hit -->|"vectorize"| node_embed
node_check_hit -->|"score"| node_similarity
node_similarity -->|"queries"| node_manager
node_manager -->|"reads/writes"| node_scalar_store
node_manager -->|"reads/writes"| node_vector_store
node_manager -->|"reads/writes"| node_object_store
node_manager -->|"governs"| node_eviction
node_check_hit -->|"on miss"| node_adapter_api
node_adapter_api -->|"implements"| node_openai_adapter
node_adapter_api -->|"implements"| node_langchain_adapter
node_adapter_api -->|"implements"| node_media_adapters
node_media_adapters -->|"returns to"| node_post
node_adapter_api -->|"returns to"| node_post
node_post -->|"stores hit/miss"| node_manager
node_core_api -->|"serves"| node_cache_server
node_examples -.->|"demonstrate"| node_core_api
node_docs -.->|"documents"| node_cache_server
node_docs -.->|"documents"| node_manager

click node_core_api "https://github.com/zilliztech/gptcache/blob/main/gptcache/core.py"
click node_client "https://github.com/zilliztech/gptcache/blob/main/gptcache/client.py"
click node_pre "https://github.com/zilliztech/gptcache/blob/main/gptcache/processor/pre.py"
click node_check_hit "https://github.com/zilliztech/gptcache/blob/main/gptcache/processor/check_hit.py"
click node_post "https://github.com/zilliztech/gptcache/blob/main/gptcache/processor/post.py"
click node_context "https://github.com/zilliztech/gptcache/blob/main/gptcache/processor/context/context.py"
click node_adapter_api "https://github.com/zilliztech/gptcache/blob/main/gptcache/adapter/api.py"
click node_openai_adapter "https://github.com/zilliztech/gptcache/blob/main/gptcache/adapter/openai.py"
click node_langchain_adapter "https://github.com/zilliztech/gptcache/blob/main/gptcache/adapter/langchain_models.py"
click node_media_adapters "https://github.com/zilliztech/gptcache/tree/main/gptcache/adapter"
click node_embed "https://github.com/zilliztech/gptcache/tree/main/gptcache/embedding"
click node_similarity "https://github.com/zilliztech/gptcache/tree/main/gptcache/similarity_evaluation"
click node_manager "https://github.com/zilliztech/gptcache/tree/main/gptcache/manager"
click node_scalar_store "https://github.com/zilliztech/gptcache/tree/main/gptcache/manager/scalar_data"
click node_vector_store "https://github.com/zilliztech/gptcache/tree/main/gptcache/manager/vector_data"
click node_object_store "https://github.com/zilliztech/gptcache/tree/main/gptcache/manager/object_data"
click node_eviction "https://github.com/zilliztech/gptcache/tree/main/gptcache/manager/eviction"
click node_cache_server "https://github.com/zilliztech/gptcache/blob/main/gptcache_server/server.py"
click node_docs "https://github.com/zilliztech/gptcache/tree/main/docs"
click node_examples "https://github.com/zilliztech/gptcache/tree/main/examples"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_core_api,node_client,node_pre,node_check_hit,node_post,node_context toneBlue
class node_adapter_api,node_openai_adapter,node_langchain_adapter,node_media_adapters toneAmber
class node_manager,node_scalar_store,node_vector_store,node_object_store,node_eviction toneMint
class node_embed,node_similarity toneRose
class node_cache_server,node_docs,node_examples toneIndigo
```

## Components

### Runtime

Core request interception and processing pipeline.

- [Cache API](https://github.com/zilliztech/gptcache/blob/main/gptcache/core.py): orchestration · Main library entry point that wires the cache workflow. · `gptcache/core.py`
- [Client](https://github.com/zilliztech/gptcache/blob/main/gptcache/client.py): runtime facade · User-facing client/session integration and cache invocation helpers. · `gptcache/client.py`
- [Pre-process](https://github.com/zilliztech/gptcache/blob/main/gptcache/processor/pre.py): request shaping · Normalizes prompts, context, and request inputs before lookup. · `gptcache/processor/pre.py`
- [Hit Check](https://github.com/zilliztech/gptcache/blob/main/gptcache/processor/check_hit.py): lookup gate · Decides whether a cache lookup should proceed and how the request is matched. · `gptcache/processor/check_hit.py`
- [Post-process](https://github.com/zilliztech/gptcache/blob/main/gptcache/processor/post.py): response shaping · Shapes returned responses, including temperature-aware handling. · `gptcache/processor/post.py`
- [Context](https://github.com/zilliztech/gptcache/blob/main/gptcache/processor/context/context.py): prompt context · Context composition utilities used by preprocessors. · `gptcache/processor/context/context.py`

### Providers

Integration layer for model and multimodal backends.

- [Adapter API](https://github.com/zilliztech/gptcache/blob/main/gptcache/adapter/api.py): integration boundary · Unified interface for provider-specific model calls. · `gptcache/adapter/api.py`
- [OpenAI](https://github.com/zilliztech/gptcache/blob/main/gptcache/adapter/openai.py): provider adapter · OpenAI-compatible model and multimodal adapter. · `gptcache/adapter/openai.py`
- [LangChain](https://github.com/zilliztech/gptcache/blob/main/gptcache/adapter/langchain_models.py): provider adapter · LangChain model integration adapter. · `gptcache/adapter/langchain_models.py`
- [Media Adapters](https://github.com/zilliztech/gptcache/tree/main/gptcache/adapter): provider adapters · Image and generation adapters for diffusion, stability, replicate, dolly, and llama.cpp. · `gptcache/adapter`

### Data Plane

Cache persistence, retrieval, and lifecycle management.

- [Data Manager](https://github.com/zilliztech/gptcache/tree/main/gptcache/manager): cache data plane · Coordinates scalar, vector, object, and eviction subsystems. · `gptcache/manager`
- [Scalar Store](https://github.com/zilliztech/gptcache/tree/main/gptcache/manager/scalar_data): metadata storage · Stores request metadata and cache records across SQL, Redis, MongoDB, and DynamoDB backends. · `gptcache/manager/scalar_data`
- [Vector Store](https://github.com/zilliztech/gptcache/tree/main/gptcache/manager/vector_data): nn index · Stores embeddings and supports nearest-neighbor search across multiple vector backends. · `gptcache/manager/vector_data`
- [Object Store](https://github.com/zilliztech/gptcache/tree/main/gptcache/manager/object_data): payload storage · Stores larger cached outputs and artifacts via local or S3 backends. · `gptcache/manager/object_data`
- [Eviction](https://github.com/zilliztech/gptcache/tree/main/gptcache/manager/eviction): lifecycle control · Handles cache retention, in-memory eviction, and distributed invalidation. · `gptcache/manager/eviction`

### Scoring

Embedding and similarity decision components.

- [Embedding](https://github.com/zilliztech/gptcache/tree/main/gptcache/embedding): vectorization · Converts text, image, or audio inputs into vectors for semantic matching. · `gptcache/embedding`
- [Similarity Eval](https://github.com/zilliztech/gptcache/tree/main/gptcache/similarity_evaluation): match scoring · Compares query and cached items to decide cache eligibility. · `gptcache/similarity_evaluation`

### Ops

Server, docs, examples, and validation ecosystem.

- [Cache Server](https://github.com/zilliztech/gptcache/blob/main/gptcache_server/server.py): service · Standalone server mode for network or language-agnostic cache access. · `gptcache_server/server.py`
- [Docs](https://github.com/zilliztech/gptcache/tree/main/docs): documentation · Usage, bootcamp, deployment, and scaling guidance. · `docs`
- [Examples](https://github.com/zilliztech/gptcache/tree/main/examples): reference code · Integration, manager, embedding, processor, and benchmark demonstrations. · `examples`

## Connections

- Client → Cache API: uses · Client forwards cache operations into the core orchestration layer.
- Cache API → Pre-process: starts · Core pipeline begins by preprocessing incoming requests.
- Pre-process → Context: builds · Pre-processing composes contextual prompt state.
- Pre-process → Hit Check: hands off · Normalized requests proceed to hit evaluation.
- Hit Check → Embedding: vectorize · Requests are embedded when semantic matching is enabled.
- Hit Check → Similarity Eval: score · Similarity evaluators determine whether a cached item is close enough.
- Similarity Eval → Data Manager: queries · Scoring consults the data plane for candidate cache entries.
- Data Manager → Scalar Store: reads/writes · Stores request metadata and cache records.
- Data Manager → Vector Store: reads/writes · Persists and searches embeddings.
- Data Manager → Object Store: reads/writes · Persists cached payloads and large outputs.
- Data Manager → Eviction: governs · Manages cache lifecycle and distributed eviction.
- Hit Check → Adapter API: on miss · A cache miss forwards the request to an upstream provider adapter.
- Adapter API → OpenAI: implements · Provider-specific OpenAI integrations.
- Adapter API → LangChain: implements · Provider-specific LangChain integrations.
- Adapter API → Media Adapters: implements · Media and generation backends share the same adapter boundary.
- Media Adapters → Post-process: returns to · Generated responses flow back through post-processing.
- Adapter API → Post-process: returns to · Upstream responses are shaped before being returned or stored.
- Post-process → Data Manager: stores hit/miss · Post-processing can persist responses and metadata for future reuse.
- Cache API → Cache Server: serves · Core cache logic can be exposed as a standalone service.
- Examples → Cache API: demonstrate · Examples illustrate end-to-end cache initialization and usage.
- Docs → Cache Server: documents · Docs cover deployment and horizontal scaling of server mode.
- Docs → Data Manager: documents · Documentation covers storage and backend configuration.
