How zilliztech/gptcache works
1. GPTCache is a Python library for semantic caching of LLM and multimodal requests. The core idea is: intercept a model call, normalize/process the request, look for a semantically similar cached response, and either return the cache hit or forward to the upstream model and store the result.
2. The main runtime entry point is gptcache.cache from gptcache/core.py and gptcache/client.py, which exposes the initialize/use flow shown in the README. This is the orchestration layer that wires together embedding, storage, similarity evaluation, and request/response post-processing.
3. The request path is split into three conceptual stages under gptcache/processor: pre-processing, hit checking, and post-processing. Pre-processors shape the incoming prompt/context, check_hit.py decides whether cache lookup should proceed, and processor/post.py handles output shaping such as temperature-aware behavior.
4. Adapters under gptcache/adapter are the integration boundary to external model providers and SDKs. They present a unified interface over OpenAI, LangChain models, Llama.cpp, Dolly, Replicate, Diffusers, Stability SDK, and multimodal APIs, so the cache layer stays provider-agnostic.
5. Embedding generation lives in gptcache/embedding. This subsystem converts text, image, or audio inputs into vectors for semantic matching. It supports multiple backends such as OpenAI, ONNX, HuggingFace, SentenceTransformers/SBERT, Cohere, fastText, TIMM, ViT, UForm, and others. Disabling embeddings effectively turns GPTCache into a keyword/exact-match cache.
6. The cache data plane is managed by gptcache/manager. This is where cached objects are persisted, indexed, and retrieved. It is split into:
- scalar data storage for request/metadata records,
- vector data storage for embeddings and nearest-neighbor search,
- object data storage for larger payloads/responses,
- eviction management for cache lifecycle.
The structure makes storage backends swappable without changing the orchestration layer.
7. Storage backends are a major architectural boundary. Scalar stores include SQLite, Redis, MongoDB, DynamoDB, and SQL-backed storage. Vector stores include FAISS, Milvus, PGVector, Chroma, Qdrant, Weaviate, HNSWLib, DocArray, Redis vector store, and USearch. Object storage includes local storage and S3. This makes the cache deployable from local/dev setups to distributed production systems.
8. Similarity decisions are implemented in gptcache/similarity_evaluation. These evaluators combine vector distances and/or secondary scoring methods to decide if a query is “close enough” to a cached request. The package includes distance-based evaluators, exact match, sequence match, ONNX-based classification, SBERT cross-encoder, K-reciprocal, Cohere rerank, and timing/threshold utilities.
9. Eviction and distribution are handled separately from similarity. gptcache/manager/eviction contains in-memory and Redis-backed distributed eviction logic, which is important for horizontal scaling. The repo also includes a dedicated gptcache_server package and Dockerfile, indicating a server mode for language-agnostic access to the cache over a process/network boundary.
10. The repository is organized as a product library plus supporting ecosystem:
- examples/ demonstrates adapter integrations, data managers, embeddings, processors, and benchmarking.
- docs/ provides usage, bootcamps, and deployment guidance, including horizontal scaling.
- tests/ contains extensive unit and integration coverage across adapters, embeddings, managers, processors, similarity evaluators, and session/client/core behavior.
- GitHub Actions automate linting, unit tests, packaging, and release artifacts.
This suggests the architecture is modular, backend-heavy, and optimized for composable cache pipelines rather than a single application UI.
Main entry points: manager Data Manager · api.py Adapter API · check_hit.py Hit Check · core.py Cache API
Components 20
Runtime
Core request interception and processing pipeline.
- Cache API · Main library entry point that wires the cache workflow.
gptcache/core.py - Client · User-facing client/session integration and cache invocation helpers.
gptcache/client.py - Pre-process · Normalizes prompts, context, and request inputs before lookup.
gptcache/processor/pre.py - Hit Check · Decides whether a cache lookup should proceed and how the request is matched.
gptcache/processor/check_hit.py - Post-process · Shapes returned responses, including temperature-aware handling.
gptcache/processor/post.py - Context · Context composition utilities used by preprocessors.
gptcache/processor/context/context.py
Providers
Integration layer for model and multimodal backends.
- Adapter API · Unified interface for provider-specific model calls.
gptcache/adapter/api.py - OpenAI · OpenAI-compatible model and multimodal adapter.
gptcache/adapter/openai.py - LangChain · LangChain model integration adapter.
gptcache/adapter/langchain_models.py - Media Adapters · Image and generation adapters for diffusion, stability, replicate, dolly, and llama.cpp.
gptcache/adapter
Data Plane
Cache persistence, retrieval, and lifecycle management.
- Data Manager · Coordinates scalar, vector, object, and eviction subsystems.
gptcache/manager - Scalar Store · Stores request metadata and cache records across SQL, Redis, MongoDB, and DynamoDB backends.
gptcache/manager/scalar_data - Vector Store · Stores embeddings and supports nearest-neighbor search across multiple vector backends.
gptcache/manager/vector_data - Object Store · Stores larger cached outputs and artifacts via local or S3 backends.
gptcache/manager/object_data - Eviction · Handles cache retention, in-memory eviction, and distributed invalidation.
gptcache/manager/eviction
Scoring
Embedding and similarity decision components.
- Embedding · Converts text, image, or audio inputs into vectors for semantic matching.
gptcache/embedding - Similarity Eval · Compares query and cached items to decide cache eligibility.
gptcache/similarity_evaluation
Ops
Server, docs, examples, and validation ecosystem.
- Cache Server · Standalone server mode for network or language-agnostic cache access.
gptcache_server/server.py - Docs · Usage, bootcamp, deployment, and scaling guidance.
docs - Examples · Integration, manager, embedding, processor, and benchmark demonstrations.
examples
Connections 22
- Client → Cache API: uses · Client forwards cache operations into the core orchestration layer.
- Cache API → Pre-process: starts · Core pipeline begins by preprocessing incoming requests.
- Pre-process → Context: builds · Pre-processing composes contextual prompt state.
- Pre-process → Hit Check: hands off · Normalized requests proceed to hit evaluation.
- Hit Check → Embedding: vectorize · Requests are embedded when semantic matching is enabled.
- Hit Check → Similarity Eval: score · Similarity evaluators determine whether a cached item is close enough.
- Similarity Eval → Data Manager: queries · Scoring consults the data plane for candidate cache entries.
- Data Manager → Scalar Store: reads/writes · Stores request metadata and cache records.
- Data Manager → Vector Store: reads/writes · Persists and searches embeddings.
- Data Manager → Object Store: reads/writes · Persists cached payloads and large outputs.
- Data Manager → Eviction: governs · Manages cache lifecycle and distributed eviction.
- Hit Check → Adapter API: on miss · A cache miss forwards the request to an upstream provider adapter.
- Adapter API → OpenAI: implements · Provider-specific OpenAI integrations.
- Adapter API → LangChain: implements · Provider-specific LangChain integrations.
- Adapter API → Media Adapters: implements · Media and generation backends share the same adapter boundary.
- Media Adapters → Post-process: returns to · Generated responses flow back through post-processing.
- Adapter API → Post-process: returns to · Upstream responses are shaped before being returned or stored.
- Post-process → Data Manager: stores hit/miss · Post-processing can persist responses and metadata for future reuse.
- Cache API → Cache Server: serves · Core cache logic can be exposed as a standalone service.
- Examples → Cache API: demonstrate · Examples illustrate end-to-end cache initialization and usage.
- Docs → Cache Server: documents · Docs cover deployment and horizontal scaling of server mode.
- Docs → Data Manager: documents · Documentation covers storage and backend configuration.