How gabrielchua/embedding-adapter works
Purpose and Public Entry Points
embedding-adapter is a Python library for fitting a post-processing adapter over existing embedding vectors, targeting retrieval, semantic search, and RAG use cases. Its primary API is:
• EmbeddingAdapter.fit(query_embeddings, document_embeddings, labels)
• EmbeddingAdapter.transform(new_embeddings)
• EmbeddingAdapter.patch() to wrap an existing embedding function.
Public package exports and core implementation are in:
• embedding_adapter/__init__.py
• embedding_adapter/embedding_adapter.py
Core Adapter Lifecycle
The main control flow is train-then-infer:
1. Call fit with query/document embedding pairs and relevance labels.
2. The adapter learns a transformation based on those labeled vectors.
3. Call transform on newly produced embeddings.
4. Optionally obtain a callable patch and apply it directly after an upstream embedding model function.
The adapter operates on vectors supplied by the caller; it does not appear to own text tokenization or base-model inference.
• embedding_adapter/embedding_adapter.py
Integration Boundary: Upstream Embedding Models
The library sits between an embedding provider and downstream retrieval infrastructure. Callers remain responsible for generating original embeddings, whether from hosted APIs such as OpenAI or local models such as gte-large. The patch interface provides the principal integration seam: it wraps a caller-provided embedding function and returns adapted vectors.
This keeps embedding-model selection, document storage, indexing, and retrieval systems outside this repository’s core boundary.
• embedding_adapter/embedding_adapter.py
• README.md
Label-Driven Training Data Boundary
Training requires three caller-owned inputs: query embeddings, document embeddings, and labels. Labels represent the relevance or supervision signal used to tune vector behavior for a domain. This separates adapter training from feedback collection, dataset construction, and embedding generation.
When real labels are unavailable, the repository provides a synthetic-label generation path rather than embedding that responsibility in the adapter training API.
• embedding_adapter/embedding_adapter.py
• embedding_adapter/utils.py
Synthetic Label Generation
LabelGenerator is a supporting component for generating labels through an external OpenAI-backed workflow. Its runtime dependency boundary is explicit: an OPENAI_API_KEY environment variable is required. This path is optional and distinct from the core vector adapter lifecycle.
External service and credential handling are therefore concentrated in the utility layer rather than the main adapter interface.
• embedding_adapter/utils.py
• README.md
Packaging and Runtime Distribution
The project is distributed as an installable Python package, including PyPI-oriented metadata and dependency declarations. Consumers install it with pip and import from embedding_adapter; the repository is library-first, with no indicated server, UI, or standalone application runtime.
• setup.py
• requirements.txt
• embedding_adapter/__init__.py
Main entry points: embedding_adapter.py EmbeddingAdapter · utils.py LabelGenerator · __init__.py Package Exports · setup.py Package Distribution
Components 15
Adapter Library
Core train-and-infer vector adaptation.
- Package Exports · Public import surface for the library.
embedding_adapter/__init__.py - EmbeddingAdapter · Fits and applies the learned embedding transformation.
embedding_adapter/embedding_adapter.py - fit · Learns a transformation from labeled vector pairs.
embedding_adapter/embedding_adapter.py - transform · Adapts newly generated embeddings.
embedding_adapter/embedding_adapter.py - patch · Wraps a caller-provided embedding function.
embedding_adapter/embedding_adapter.py
Optional Labeling
External synthetic-label workflow.
- LabelGenerator · Optionally generates synthetic supervision labels.
embedding_adapter/utils.py - OPENAI_API_KEY · Required by the OpenAI-backed labeling path.
Other components
- Library Consumer · Supplies vectors and integrates adapted output.
- Embedding Provider · Produces original query and document vectors.
- Labeled Vector Pairs · Query embeddings, document embeddings, and relevance labels.
- Adapted Embeddings · Vectors for downstream retrieval use.
- Retrieval Infrastructure · Indexing, storage, and search remain outside the library.
- OpenAI Service · Supports synthetic label generation.
- Package Distribution · Defines installable package metadata and dependencies.
setup.py - Runtime Dependencies
requirements.txt
Connections 17
- Library Consumer → Embedding Provider: requests embeddings
- Embedding Provider → Labeled Vector Pairs: produces training vectors
- Labeled Vector Pairs → fit: query/document vectors + labels
- Package Exports → EmbeddingAdapter: exports
- Library Consumer → Package Exports: imports
- fit → EmbeddingAdapter: stores learned transformation
- EmbeddingAdapter → transform: serves inference
- Embedding Provider → transform: new embeddings
- transform → Adapted Embeddings: transforms
- EmbeddingAdapter → patch: provides callable wrapper
- patch → Adapted Embeddings: adapts provider output
- Adapted Embeddings → Retrieval Infrastructure: indexes or searches
- LabelGenerator → Labeled Vector Pairs: optional synthetic labels
- OPENAI_API_KEY → LabelGenerator: authorizes
- LabelGenerator → OpenAI Service: calls
- Package Distribution → Runtime Dependencies: declares dependencies
- Package Distribution → Package Exports: distributes